diff --git a/_sources_indexrsttxt_cf07c1c2.txt b/_sources_indexrsttxt_cf07c1c2.txt
new file mode 100644
index 0000000000000000000000000000000000000000..bd34bd05c534bf555aa8e130d07ad9a2a9cbd475
--- /dev/null
+++ b/_sources_indexrsttxt_cf07c1c2.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/links/_sources/index.rst.txt#what-you-can-build
+Title: Overview - Pipecat
+==================================================
+
+Overview - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Get Started Overview Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Get Started Overview Installation & Setup Quickstart Core Concepts Next Steps & Examples Pipecat is an open source Python framework that handles the complex orchestration of AI services, network transport, audio processing, and multimodal interactions. “Multimodal” means you can use any combination of audio, video, images, and/or text in your interactions. And “real-time” means that things are happening quickly enough that it feels conversational—a “back-and-forth” with a bot, not submitting a query and waiting for results. ​ What You Can Build Voice Assistants Natural, real-time conversations with AI using speech recognition and synthesis Interactive Agents Personal coaches and meeting assistants that can understand context and provide guidance Multimodal Apps Applications that combine voice, video, images, and text for rich interactions Creative Tools Storytelling experiences and social companions that engage users Business Solutions Customer intake flows and support bots for automated business processes Complex Flows Structured conversations using Pipecat Flows for managing complex interactions ​ How It Works The flow of interactions in a Pipecat application is typically straightforward: The bot says something The user says something The bot says something The user says something This continues until the conversation naturally ends. While this flow seems simple, making it feel natural requires sophisticated real-time processing. ​ Real-time Processing Pipecat’s pipeline architecture handles both simple voice interactions and complex multimodal processing. Let’s look at how data flows through the system: Voice app Multimodal app 1 Send Audio Transmit and capture streamed audio from the user 2 Transcribe Speech Convert speech to text as the user is talking 3 Process with LLM Generate responses using a large language model 4 Convert to Speech Transform text responses into natural speech 5 Play Audio Stream the audio response back to the user 1 Send Audio Transmit and capture streamed audio from the user 2 Transcribe Speech Convert speech to text as the user is talking 3 Process with LLM Generate responses using a large language model 4 Convert to Speech Transform text responses into natural speech 5 Play Audio Stream the audio response back to the user 1 Send Audio and Video Transmit and capture audio, video, and image inputs simultaneously 2 Process Streams Handle multiple input streams in parallel 3 Model Processing Send combined inputs to multimodal models (like GPT-4V) 4 Generate Outputs Create various outputs (text, images, audio, etc.) 5 Coordinate Presentation Synchronize and present multiple output types In both cases, Pipecat: Processes responses as they stream in Handles multiple input/output modalities concurrently Manages resource allocation and synchronization Coordinates parallel processing tasks This architecture creates fluid, natural interactions without noticeable delays, whether you’re building a simple voice assistant or a complex multimodal application. Pipecat’s pipeline architecture is particularly valuable for managing the complexity of real-time, multimodal interactions, ensuring smooth data flow and proper synchronization regardless of the input/output types involved. Pipecat handles all this complexity for you, letting you focus on building your application rather than managing the underlying infrastructure. ​ Next Steps Ready to build your first Pipecat application? Installation & Setup Prepare your environment and install required dependencies Quickstart Build and run your first Pipecat application Core Concepts Learn about pipelines, frames, and real-time processing Use Cases Explore example implementations and patterns ​ Join Our Community Discord Community Connect with other developers, share your projects, and get support from the Pipecat team. Installation & Setup On this page What You Can Build How It Works Real-time Processing Next Steps Join Our Community Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/_sources_indexrsttxt_eb6b9580.txt b/_sources_indexrsttxt_eb6b9580.txt
new file mode 100644
index 0000000000000000000000000000000000000000..d3a9d003a1c46b11fc93eb07f4c25bf8b3c143a2
--- /dev/null
+++ b/_sources_indexrsttxt_eb6b9580.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/links/_sources/index.rst.txt
+Title: Overview - Pipecat
+==================================================
+
+Overview - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Get Started Overview Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Get Started Overview Installation & Setup Quickstart Core Concepts Next Steps & Examples Pipecat is an open source Python framework that handles the complex orchestration of AI services, network transport, audio processing, and multimodal interactions. “Multimodal” means you can use any combination of audio, video, images, and/or text in your interactions. And “real-time” means that things are happening quickly enough that it feels conversational—a “back-and-forth” with a bot, not submitting a query and waiting for results. ​ What You Can Build Voice Assistants Natural, real-time conversations with AI using speech recognition and synthesis Interactive Agents Personal coaches and meeting assistants that can understand context and provide guidance Multimodal Apps Applications that combine voice, video, images, and text for rich interactions Creative Tools Storytelling experiences and social companions that engage users Business Solutions Customer intake flows and support bots for automated business processes Complex Flows Structured conversations using Pipecat Flows for managing complex interactions ​ How It Works The flow of interactions in a Pipecat application is typically straightforward: The bot says something The user says something The bot says something The user says something This continues until the conversation naturally ends. While this flow seems simple, making it feel natural requires sophisticated real-time processing. ​ Real-time Processing Pipecat’s pipeline architecture handles both simple voice interactions and complex multimodal processing. Let’s look at how data flows through the system: Voice app Multimodal app 1 Send Audio Transmit and capture streamed audio from the user 2 Transcribe Speech Convert speech to text as the user is talking 3 Process with LLM Generate responses using a large language model 4 Convert to Speech Transform text responses into natural speech 5 Play Audio Stream the audio response back to the user 1 Send Audio Transmit and capture streamed audio from the user 2 Transcribe Speech Convert speech to text as the user is talking 3 Process with LLM Generate responses using a large language model 4 Convert to Speech Transform text responses into natural speech 5 Play Audio Stream the audio response back to the user 1 Send Audio and Video Transmit and capture audio, video, and image inputs simultaneously 2 Process Streams Handle multiple input streams in parallel 3 Model Processing Send combined inputs to multimodal models (like GPT-4V) 4 Generate Outputs Create various outputs (text, images, audio, etc.) 5 Coordinate Presentation Synchronize and present multiple output types In both cases, Pipecat: Processes responses as they stream in Handles multiple input/output modalities concurrently Manages resource allocation and synchronization Coordinates parallel processing tasks This architecture creates fluid, natural interactions without noticeable delays, whether you’re building a simple voice assistant or a complex multimodal application. Pipecat’s pipeline architecture is particularly valuable for managing the complexity of real-time, multimodal interactions, ensuring smooth data flow and proper synchronization regardless of the input/output types involved. Pipecat handles all this complexity for you, letting you focus on building your application rather than managing the underlying infrastructure. ​ Next Steps Ready to build your first Pipecat application? Installation & Setup Prepare your environment and install required dependencies Quickstart Build and run your first Pipecat application Core Concepts Learn about pipelines, frames, and real-time processing Use Cases Explore example implementations and patterns ​ Join Our Community Discord Community Connect with other developers, share your projects, and get support from the Pipecat team. Installation & Setup On this page What You Can Build How It Works Real-time Processing Next Steps Join Our Community Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/_sources_indexrsttxt_ef601814.txt b/_sources_indexrsttxt_ef601814.txt
new file mode 100644
index 0000000000000000000000000000000000000000..4cef9877f2676bb053bae8a0f1350e30001e747d
--- /dev/null
+++ b/_sources_indexrsttxt_ef601814.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/links/_sources/index.rst.txt#join-our-community
+Title: Overview - Pipecat
+==================================================
+
+Overview - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Get Started Overview Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Get Started Overview Installation & Setup Quickstart Core Concepts Next Steps & Examples Pipecat is an open source Python framework that handles the complex orchestration of AI services, network transport, audio processing, and multimodal interactions. “Multimodal” means you can use any combination of audio, video, images, and/or text in your interactions. And “real-time” means that things are happening quickly enough that it feels conversational—a “back-and-forth” with a bot, not submitting a query and waiting for results. ​ What You Can Build Voice Assistants Natural, real-time conversations with AI using speech recognition and synthesis Interactive Agents Personal coaches and meeting assistants that can understand context and provide guidance Multimodal Apps Applications that combine voice, video, images, and text for rich interactions Creative Tools Storytelling experiences and social companions that engage users Business Solutions Customer intake flows and support bots for automated business processes Complex Flows Structured conversations using Pipecat Flows for managing complex interactions ​ How It Works The flow of interactions in a Pipecat application is typically straightforward: The bot says something The user says something The bot says something The user says something This continues until the conversation naturally ends. While this flow seems simple, making it feel natural requires sophisticated real-time processing. ​ Real-time Processing Pipecat’s pipeline architecture handles both simple voice interactions and complex multimodal processing. Let’s look at how data flows through the system: Voice app Multimodal app 1 Send Audio Transmit and capture streamed audio from the user 2 Transcribe Speech Convert speech to text as the user is talking 3 Process with LLM Generate responses using a large language model 4 Convert to Speech Transform text responses into natural speech 5 Play Audio Stream the audio response back to the user 1 Send Audio Transmit and capture streamed audio from the user 2 Transcribe Speech Convert speech to text as the user is talking 3 Process with LLM Generate responses using a large language model 4 Convert to Speech Transform text responses into natural speech 5 Play Audio Stream the audio response back to the user 1 Send Audio and Video Transmit and capture audio, video, and image inputs simultaneously 2 Process Streams Handle multiple input streams in parallel 3 Model Processing Send combined inputs to multimodal models (like GPT-4V) 4 Generate Outputs Create various outputs (text, images, audio, etc.) 5 Coordinate Presentation Synchronize and present multiple output types In both cases, Pipecat: Processes responses as they stream in Handles multiple input/output modalities concurrently Manages resource allocation and synchronization Coordinates parallel processing tasks This architecture creates fluid, natural interactions without noticeable delays, whether you’re building a simple voice assistant or a complex multimodal application. Pipecat’s pipeline architecture is particularly valuable for managing the complexity of real-time, multimodal interactions, ensuring smooth data flow and proper synchronization regardless of the input/output types involved. Pipecat handles all this complexity for you, letting you focus on building your application rather than managing the underlying infrastructure. ​ Next Steps Ready to build your first Pipecat application? Installation & Setup Prepare your environment and install required dependencies Quickstart Build and run your first Pipecat application Core Concepts Learn about pipelines, frames, and real-time processing Use Cases Explore example implementations and patterns ​ Join Our Community Discord Community Connect with other developers, share your projects, and get support from the Pipecat team. Installation & Setup On this page What You Can Build How It Works Real-time Processing Next Steps Join Our Community Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/analytics_sentry_d30af7e0.txt b/analytics_sentry_d30af7e0.txt
new file mode 100644
index 0000000000000000000000000000000000000000..f29278fc9aa51b5f8b8b6e02ee051dceb912295d
--- /dev/null
+++ b/analytics_sentry_d30af7e0.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/analytics/sentry#installation
+Title: Sentry Metrics - Pipecat
+==================================================
+
+Sentry Metrics - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Analytics & Monitoring Sentry Metrics Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Sentry Metrics Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview SentryMetrics extends FrameProcessorMetrics to provide performance monitoring integration with Sentry. It tracks Time to First Byte (TTFB) and processing duration metrics for frame processors. ​ Installation To use Sentry metrics, install the Sentry SDK: Copy Ask AI pip install "pipecat-ai[sentry]" ​ Configuration Sentry must be initialized in your application before metrics will be collected: Copy Ask AI import sentry_sdk sentry_sdk.init( dsn = "your-sentry-dsn" , traces_sample_rate = 1.0 , ) ​ Usage Example Copy Ask AI import sentry_sdk from pipecat.services.openai.llm import OpenAILLMService from pipecat.services.elevenlabs.tts import ElevenLabsTTSService from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext from pipecat.processors.metrics.sentry import SentryMetrics from pipecat.transports.services.daily import DailyParams, DailyTransport async def create_metrics_pipeline (): sentry_sdk.init( dsn = "your-sentry-dsn" , traces_sample_rate = 1.0 , ) transport = DailyTransport( room_url, token, "Chatbot" , DailyParams( audio_out_enabled = True , audio_in_enabled = True , video_out_enabled = False , vad_analyzer = SileroVADAnalyzer(), transcription_enabled = True , ), ) tts = ElevenLabsTTSService( api_key = os.getenv( "ELEVENLABS_API_KEY" ), metrics = SentryMetrics(), ) llm = OpenAILLMService( api_key = os.getenv( "OPENAI_API_KEY" ), model = "gpt-4o" ), metrics = SentryMetrics(), ) messages = [ { "role" : "system" , "content" : "You are Chatbot, a friendly, helpful robot. Your goal is to demonstrate your capabilities in a succinct way. Your output will be converted to audio so don't include special characters in your answers. Respond to what the user said in a creative and helpful way, but keep your responses brief. Start by introducing yourself. Keep all your responses to 12 words or fewer." , }, ] context = OpenAILLMContext(messages) context_aggregator = llm.create_context_aggregator(context) # Use in pipeline pipeline = Pipeline([ transport.input(), context_aggregator.user(), llm, tts, transport.output(), context_aggregator.assistant(), ]) ​ Transaction Information Each transaction includes: Operation type ( ttfb or processing ) Description with processor name Start timestamp End timestamp Unique transaction ID ​ Fallback Behavior If Sentry is not available (not installed or not initialized): Warning logs are generated Metric methods execute without error No data is sent to Sentry ​ Notes Requires Sentry SDK to be installed and initialized Thread-safe metric collection Automatic transaction management Supports selective TTFB reporting Integrates with Sentry’s performance monitoring Provides detailed timing information Maintains timing data even when Sentry is unavailable Moondream Producer & Consumer Processors On this page Overview Installation Configuration Usage Example Transaction Information Fallback Behavior Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/android_api-reference_9763bcbe.txt b/android_api-reference_9763bcbe.txt
new file mode 100644
index 0000000000000000000000000000000000000000..211b20ac2bd072daaef656b394ec4c4550243064
--- /dev/null
+++ b/android_api-reference_9763bcbe.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/client/android/api-reference
+Title: All modules
+==================================================
+
+All modules All modules: pipecat-client-android Link copied to clipboard pipecat-transport-daily Link copied to clipboard pipecat-transport-gemini-live-websocket Link copied to clipboard pipecat-transport-openai-realtime-webrtc Link copied to clipboard © 2025 Copyright Generated by dokka
\ No newline at end of file
diff --git a/audio_audio-buffer-processor_94494faa.txt b/audio_audio-buffer-processor_94494faa.txt
new file mode 100644
index 0000000000000000000000000000000000000000..8e808a4eca9ea56b35ee75c57a4da39e68022581
--- /dev/null
+++ b/audio_audio-buffer-processor_94494faa.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/audio/audio-buffer-processor#start-recording
+Title: AudioBufferProcessor - Pipecat
+==================================================
+
+AudioBufferProcessor - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Audio Processing AudioBufferProcessor Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing AudioBufferProcessor KoalaFilter KrispFilter NoisereduceFilter SileroVADAnalyzer SoundfileMixer Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview The AudioBufferProcessor captures and buffers audio frames from both input (user) and output (bot) sources during conversations. It provides synchronized audio streams with configurable sample rates, supports both mono and stereo output, and offers flexible event handlers for various audio processing workflows. ​ Constructor Copy Ask AI AudioBufferProcessor( sample_rate = None , num_channels = 1 , buffer_size = 0 , enable_turn_audio = False , ** kwargs ) ​ Parameters ​ sample_rate Optional[int] default: "None" The desired output sample rate in Hz. If None , uses the transport’s sample rate from the StartFrame . ​ num_channels int default: "1" Number of output audio channels: 1 : Mono output (user and bot audio are mixed together) 2 : Stereo output (user audio on left channel, bot audio on right channel) ​ buffer_size int default: "0" Buffer size in bytes that triggers audio data events: 0 : Events only trigger when recording stops >0 : Events trigger whenever buffer reaches this size (useful for chunked processing) ​ enable_turn_audio bool default: "False" Whether to enable per-turn audio event handlers ( on_user_turn_audio_data and on_bot_turn_audio_data ). ​ Properties ​ sample_rate Copy Ask AI @ property def sample_rate ( self ) -> int The current sample rate of the audio processor in Hz. ​ num_channels Copy Ask AI @ property def num_channels ( self ) -> int The number of channels in the audio output (1 for mono, 2 for stereo). ​ Methods ​ start_recording() Copy Ask AI async def start_recording () Start recording audio from both user and bot sources. Initializes recording state and resets audio buffers. ​ stop_recording() Copy Ask AI async def stop_recording () Stop recording and trigger final audio data handlers with any remaining buffered audio. ​ has_audio() Copy Ask AI def has_audio () -> bool Check if both user and bot audio buffers contain data. Returns: True if both buffers contain audio data. ​ Event Handlers The processor supports multiple event handlers for different audio processing workflows. Register handlers using the @processor.event_handler() decorator. ​ on_audio_data Triggered when buffer_size is reached or recording stops, providing merged audio. Copy Ask AI @audiobuffer.event_handler ( "on_audio_data" ) async def on_audio_data ( buffer , audio : bytes , sample_rate : int , num_channels : int ): # Handle merged audio data pass Parameters: buffer : The AudioBufferProcessor instance audio : Merged audio data (format depends on num_channels setting) sample_rate : Sample rate in Hz num_channels : Number of channels (1 or 2) ​ on_track_audio_data Triggered alongside on_audio_data , providing separate user and bot audio tracks. Copy Ask AI @audiobuffer.event_handler ( "on_track_audio_data" ) async def on_track_audio_data ( buffer , user_audio : bytes , bot_audio : bytes , sample_rate : int , num_channels : int ): # Handle separate audio tracks pass Parameters: buffer : The AudioBufferProcessor instance user_audio : Raw user audio bytes (always mono) bot_audio : Raw bot audio bytes (always mono) sample_rate : Sample rate in Hz num_channels : Always 1 for individual tracks ​ on_user_turn_audio_data Triggered when a user speaking turn ends. Requires enable_turn_audio=True . Copy Ask AI @audiobuffer.event_handler ( "on_user_turn_audio_data" ) async def on_user_turn_audio_data ( buffer , audio : bytes , sample_rate : int , num_channels : int ): # Handle user turn audio pass Parameters: buffer : The AudioBufferProcessor instance audio : Audio data from the user’s speaking turn sample_rate : Sample rate in Hz num_channels : Always 1 (mono) ​ on_bot_turn_audio_data Triggered when a bot speaking turn ends. Requires enable_turn_audio=True . Copy Ask AI @audiobuffer.event_handler ( "on_bot_turn_audio_data" ) async def on_bot_turn_audio_data ( buffer , audio : bytes , sample_rate : int , num_channels : int ): # Handle bot turn audio pass Parameters: buffer : The AudioBufferProcessor instance audio : Audio data from the bot’s speaking turn sample_rate : Sample rate in Hz num_channels : Always 1 (mono) ​ Audio Processing Features Automatic resampling : Converts incoming audio to the specified sample rate Buffer synchronization : Aligns user and bot audio streams temporally Silence insertion : Fills gaps in non-continuous audio streams to maintain timing Turn tracking : Monitors speaking turns when enable_turn_audio=True ​ Integration Notes ​ STT Audio Passthrough If using an STT service in your pipeline, enable audio passthrough to make audio available to the AudioBufferProcessor: Copy Ask AI stt = DeepgramSTTService( api_key = os.getenv( "DEEPGRAM_API_KEY" ), audio_passthrough = True , ) audio_passthrough is enabled by default. ​ Pipeline Placement Add the AudioBufferProcessor after transport.output() to capture both user and bot audio: Copy Ask AI pipeline = Pipeline([ transport.input(), # ... other processors ... transport.output(), audiobuffer, # Place after audio output # ... remaining processors ... ]) UserIdleProcessor KoalaFilter On this page Overview Constructor Parameters Properties sample_rate num_channels Methods start_recording() stop_recording() has_audio() Event Handlers on_audio_data on_track_audio_data on_user_turn_audio_data on_bot_turn_audio_data Audio Processing Features Integration Notes STT Audio Passthrough Pipeline Placement Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/audio_audio-buffer-processor_9949bd98.txt b/audio_audio-buffer-processor_9949bd98.txt
new file mode 100644
index 0000000000000000000000000000000000000000..2aaea547f4aec050cc32e412674c7eae3b753bfc
--- /dev/null
+++ b/audio_audio-buffer-processor_9949bd98.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/audio/audio-buffer-processor#param-buffer-size
+Title: AudioBufferProcessor - Pipecat
+==================================================
+
+AudioBufferProcessor - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Audio Processing AudioBufferProcessor Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing AudioBufferProcessor KoalaFilter KrispFilter NoisereduceFilter SileroVADAnalyzer SoundfileMixer Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview The AudioBufferProcessor captures and buffers audio frames from both input (user) and output (bot) sources during conversations. It provides synchronized audio streams with configurable sample rates, supports both mono and stereo output, and offers flexible event handlers for various audio processing workflows. ​ Constructor Copy Ask AI AudioBufferProcessor( sample_rate = None , num_channels = 1 , buffer_size = 0 , enable_turn_audio = False , ** kwargs ) ​ Parameters ​ sample_rate Optional[int] default: "None" The desired output sample rate in Hz. If None , uses the transport’s sample rate from the StartFrame . ​ num_channels int default: "1" Number of output audio channels: 1 : Mono output (user and bot audio are mixed together) 2 : Stereo output (user audio on left channel, bot audio on right channel) ​ buffer_size int default: "0" Buffer size in bytes that triggers audio data events: 0 : Events only trigger when recording stops >0 : Events trigger whenever buffer reaches this size (useful for chunked processing) ​ enable_turn_audio bool default: "False" Whether to enable per-turn audio event handlers ( on_user_turn_audio_data and on_bot_turn_audio_data ). ​ Properties ​ sample_rate Copy Ask AI @ property def sample_rate ( self ) -> int The current sample rate of the audio processor in Hz. ​ num_channels Copy Ask AI @ property def num_channels ( self ) -> int The number of channels in the audio output (1 for mono, 2 for stereo). ​ Methods ​ start_recording() Copy Ask AI async def start_recording () Start recording audio from both user and bot sources. Initializes recording state and resets audio buffers. ​ stop_recording() Copy Ask AI async def stop_recording () Stop recording and trigger final audio data handlers with any remaining buffered audio. ​ has_audio() Copy Ask AI def has_audio () -> bool Check if both user and bot audio buffers contain data. Returns: True if both buffers contain audio data. ​ Event Handlers The processor supports multiple event handlers for different audio processing workflows. Register handlers using the @processor.event_handler() decorator. ​ on_audio_data Triggered when buffer_size is reached or recording stops, providing merged audio. Copy Ask AI @audiobuffer.event_handler ( "on_audio_data" ) async def on_audio_data ( buffer , audio : bytes , sample_rate : int , num_channels : int ): # Handle merged audio data pass Parameters: buffer : The AudioBufferProcessor instance audio : Merged audio data (format depends on num_channels setting) sample_rate : Sample rate in Hz num_channels : Number of channels (1 or 2) ​ on_track_audio_data Triggered alongside on_audio_data , providing separate user and bot audio tracks. Copy Ask AI @audiobuffer.event_handler ( "on_track_audio_data" ) async def on_track_audio_data ( buffer , user_audio : bytes , bot_audio : bytes , sample_rate : int , num_channels : int ): # Handle separate audio tracks pass Parameters: buffer : The AudioBufferProcessor instance user_audio : Raw user audio bytes (always mono) bot_audio : Raw bot audio bytes (always mono) sample_rate : Sample rate in Hz num_channels : Always 1 for individual tracks ​ on_user_turn_audio_data Triggered when a user speaking turn ends. Requires enable_turn_audio=True . Copy Ask AI @audiobuffer.event_handler ( "on_user_turn_audio_data" ) async def on_user_turn_audio_data ( buffer , audio : bytes , sample_rate : int , num_channels : int ): # Handle user turn audio pass Parameters: buffer : The AudioBufferProcessor instance audio : Audio data from the user’s speaking turn sample_rate : Sample rate in Hz num_channels : Always 1 (mono) ​ on_bot_turn_audio_data Triggered when a bot speaking turn ends. Requires enable_turn_audio=True . Copy Ask AI @audiobuffer.event_handler ( "on_bot_turn_audio_data" ) async def on_bot_turn_audio_data ( buffer , audio : bytes , sample_rate : int , num_channels : int ): # Handle bot turn audio pass Parameters: buffer : The AudioBufferProcessor instance audio : Audio data from the bot’s speaking turn sample_rate : Sample rate in Hz num_channels : Always 1 (mono) ​ Audio Processing Features Automatic resampling : Converts incoming audio to the specified sample rate Buffer synchronization : Aligns user and bot audio streams temporally Silence insertion : Fills gaps in non-continuous audio streams to maintain timing Turn tracking : Monitors speaking turns when enable_turn_audio=True ​ Integration Notes ​ STT Audio Passthrough If using an STT service in your pipeline, enable audio passthrough to make audio available to the AudioBufferProcessor: Copy Ask AI stt = DeepgramSTTService( api_key = os.getenv( "DEEPGRAM_API_KEY" ), audio_passthrough = True , ) audio_passthrough is enabled by default. ​ Pipeline Placement Add the AudioBufferProcessor after transport.output() to capture both user and bot audio: Copy Ask AI pipeline = Pipeline([ transport.input(), # ... other processors ... transport.output(), audiobuffer, # Place after audio output # ... remaining processors ... ]) UserIdleProcessor KoalaFilter On this page Overview Constructor Parameters Properties sample_rate num_channels Methods start_recording() stop_recording() has_audio() Event Handlers on_audio_data on_track_audio_data on_user_turn_audio_data on_bot_turn_audio_data Audio Processing Features Integration Notes STT Audio Passthrough Pipeline Placement Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/audio_audio-buffer-processor_a965e6e4.txt b/audio_audio-buffer-processor_a965e6e4.txt
new file mode 100644
index 0000000000000000000000000000000000000000..b1a9e6c24b0db7523a3c92758ad93c7cf7d8da77
--- /dev/null
+++ b/audio_audio-buffer-processor_a965e6e4.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/audio/audio-buffer-processor#pipeline-placement
+Title: AudioBufferProcessor - Pipecat
+==================================================
+
+AudioBufferProcessor - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Audio Processing AudioBufferProcessor Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing AudioBufferProcessor KoalaFilter KrispFilter NoisereduceFilter SileroVADAnalyzer SoundfileMixer Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview The AudioBufferProcessor captures and buffers audio frames from both input (user) and output (bot) sources during conversations. It provides synchronized audio streams with configurable sample rates, supports both mono and stereo output, and offers flexible event handlers for various audio processing workflows. ​ Constructor Copy Ask AI AudioBufferProcessor( sample_rate = None , num_channels = 1 , buffer_size = 0 , enable_turn_audio = False , ** kwargs ) ​ Parameters ​ sample_rate Optional[int] default: "None" The desired output sample rate in Hz. If None , uses the transport’s sample rate from the StartFrame . ​ num_channels int default: "1" Number of output audio channels: 1 : Mono output (user and bot audio are mixed together) 2 : Stereo output (user audio on left channel, bot audio on right channel) ​ buffer_size int default: "0" Buffer size in bytes that triggers audio data events: 0 : Events only trigger when recording stops >0 : Events trigger whenever buffer reaches this size (useful for chunked processing) ​ enable_turn_audio bool default: "False" Whether to enable per-turn audio event handlers ( on_user_turn_audio_data and on_bot_turn_audio_data ). ​ Properties ​ sample_rate Copy Ask AI @ property def sample_rate ( self ) -> int The current sample rate of the audio processor in Hz. ​ num_channels Copy Ask AI @ property def num_channels ( self ) -> int The number of channels in the audio output (1 for mono, 2 for stereo). ​ Methods ​ start_recording() Copy Ask AI async def start_recording () Start recording audio from both user and bot sources. Initializes recording state and resets audio buffers. ​ stop_recording() Copy Ask AI async def stop_recording () Stop recording and trigger final audio data handlers with any remaining buffered audio. ​ has_audio() Copy Ask AI def has_audio () -> bool Check if both user and bot audio buffers contain data. Returns: True if both buffers contain audio data. ​ Event Handlers The processor supports multiple event handlers for different audio processing workflows. Register handlers using the @processor.event_handler() decorator. ​ on_audio_data Triggered when buffer_size is reached or recording stops, providing merged audio. Copy Ask AI @audiobuffer.event_handler ( "on_audio_data" ) async def on_audio_data ( buffer , audio : bytes , sample_rate : int , num_channels : int ): # Handle merged audio data pass Parameters: buffer : The AudioBufferProcessor instance audio : Merged audio data (format depends on num_channels setting) sample_rate : Sample rate in Hz num_channels : Number of channels (1 or 2) ​ on_track_audio_data Triggered alongside on_audio_data , providing separate user and bot audio tracks. Copy Ask AI @audiobuffer.event_handler ( "on_track_audio_data" ) async def on_track_audio_data ( buffer , user_audio : bytes , bot_audio : bytes , sample_rate : int , num_channels : int ): # Handle separate audio tracks pass Parameters: buffer : The AudioBufferProcessor instance user_audio : Raw user audio bytes (always mono) bot_audio : Raw bot audio bytes (always mono) sample_rate : Sample rate in Hz num_channels : Always 1 for individual tracks ​ on_user_turn_audio_data Triggered when a user speaking turn ends. Requires enable_turn_audio=True . Copy Ask AI @audiobuffer.event_handler ( "on_user_turn_audio_data" ) async def on_user_turn_audio_data ( buffer , audio : bytes , sample_rate : int , num_channels : int ): # Handle user turn audio pass Parameters: buffer : The AudioBufferProcessor instance audio : Audio data from the user’s speaking turn sample_rate : Sample rate in Hz num_channels : Always 1 (mono) ​ on_bot_turn_audio_data Triggered when a bot speaking turn ends. Requires enable_turn_audio=True . Copy Ask AI @audiobuffer.event_handler ( "on_bot_turn_audio_data" ) async def on_bot_turn_audio_data ( buffer , audio : bytes , sample_rate : int , num_channels : int ): # Handle bot turn audio pass Parameters: buffer : The AudioBufferProcessor instance audio : Audio data from the bot’s speaking turn sample_rate : Sample rate in Hz num_channels : Always 1 (mono) ​ Audio Processing Features Automatic resampling : Converts incoming audio to the specified sample rate Buffer synchronization : Aligns user and bot audio streams temporally Silence insertion : Fills gaps in non-continuous audio streams to maintain timing Turn tracking : Monitors speaking turns when enable_turn_audio=True ​ Integration Notes ​ STT Audio Passthrough If using an STT service in your pipeline, enable audio passthrough to make audio available to the AudioBufferProcessor: Copy Ask AI stt = DeepgramSTTService( api_key = os.getenv( "DEEPGRAM_API_KEY" ), audio_passthrough = True , ) audio_passthrough is enabled by default. ​ Pipeline Placement Add the AudioBufferProcessor after transport.output() to capture both user and bot audio: Copy Ask AI pipeline = Pipeline([ transport.input(), # ... other processors ... transport.output(), audiobuffer, # Place after audio output # ... remaining processors ... ]) UserIdleProcessor KoalaFilter On this page Overview Constructor Parameters Properties sample_rate num_channels Methods start_recording() stop_recording() has_audio() Event Handlers on_audio_data on_track_audio_data on_user_turn_audio_data on_bot_turn_audio_data Audio Processing Features Integration Notes STT Audio Passthrough Pipeline Placement Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/audio_audio-buffer-processor_f9116c60.txt b/audio_audio-buffer-processor_f9116c60.txt
new file mode 100644
index 0000000000000000000000000000000000000000..6c12fe45d615bd3a33d78ee277c33a711248c9f5
--- /dev/null
+++ b/audio_audio-buffer-processor_f9116c60.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/audio/audio-buffer-processor#num-channels
+Title: AudioBufferProcessor - Pipecat
+==================================================
+
+AudioBufferProcessor - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Audio Processing AudioBufferProcessor Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing AudioBufferProcessor KoalaFilter KrispFilter NoisereduceFilter SileroVADAnalyzer SoundfileMixer Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview The AudioBufferProcessor captures and buffers audio frames from both input (user) and output (bot) sources during conversations. It provides synchronized audio streams with configurable sample rates, supports both mono and stereo output, and offers flexible event handlers for various audio processing workflows. ​ Constructor Copy Ask AI AudioBufferProcessor( sample_rate = None , num_channels = 1 , buffer_size = 0 , enable_turn_audio = False , ** kwargs ) ​ Parameters ​ sample_rate Optional[int] default: "None" The desired output sample rate in Hz. If None , uses the transport’s sample rate from the StartFrame . ​ num_channels int default: "1" Number of output audio channels: 1 : Mono output (user and bot audio are mixed together) 2 : Stereo output (user audio on left channel, bot audio on right channel) ​ buffer_size int default: "0" Buffer size in bytes that triggers audio data events: 0 : Events only trigger when recording stops >0 : Events trigger whenever buffer reaches this size (useful for chunked processing) ​ enable_turn_audio bool default: "False" Whether to enable per-turn audio event handlers ( on_user_turn_audio_data and on_bot_turn_audio_data ). ​ Properties ​ sample_rate Copy Ask AI @ property def sample_rate ( self ) -> int The current sample rate of the audio processor in Hz. ​ num_channels Copy Ask AI @ property def num_channels ( self ) -> int The number of channels in the audio output (1 for mono, 2 for stereo). ​ Methods ​ start_recording() Copy Ask AI async def start_recording () Start recording audio from both user and bot sources. Initializes recording state and resets audio buffers. ​ stop_recording() Copy Ask AI async def stop_recording () Stop recording and trigger final audio data handlers with any remaining buffered audio. ​ has_audio() Copy Ask AI def has_audio () -> bool Check if both user and bot audio buffers contain data. Returns: True if both buffers contain audio data. ​ Event Handlers The processor supports multiple event handlers for different audio processing workflows. Register handlers using the @processor.event_handler() decorator. ​ on_audio_data Triggered when buffer_size is reached or recording stops, providing merged audio. Copy Ask AI @audiobuffer.event_handler ( "on_audio_data" ) async def on_audio_data ( buffer , audio : bytes , sample_rate : int , num_channels : int ): # Handle merged audio data pass Parameters: buffer : The AudioBufferProcessor instance audio : Merged audio data (format depends on num_channels setting) sample_rate : Sample rate in Hz num_channels : Number of channels (1 or 2) ​ on_track_audio_data Triggered alongside on_audio_data , providing separate user and bot audio tracks. Copy Ask AI @audiobuffer.event_handler ( "on_track_audio_data" ) async def on_track_audio_data ( buffer , user_audio : bytes , bot_audio : bytes , sample_rate : int , num_channels : int ): # Handle separate audio tracks pass Parameters: buffer : The AudioBufferProcessor instance user_audio : Raw user audio bytes (always mono) bot_audio : Raw bot audio bytes (always mono) sample_rate : Sample rate in Hz num_channels : Always 1 for individual tracks ​ on_user_turn_audio_data Triggered when a user speaking turn ends. Requires enable_turn_audio=True . Copy Ask AI @audiobuffer.event_handler ( "on_user_turn_audio_data" ) async def on_user_turn_audio_data ( buffer , audio : bytes , sample_rate : int , num_channels : int ): # Handle user turn audio pass Parameters: buffer : The AudioBufferProcessor instance audio : Audio data from the user’s speaking turn sample_rate : Sample rate in Hz num_channels : Always 1 (mono) ​ on_bot_turn_audio_data Triggered when a bot speaking turn ends. Requires enable_turn_audio=True . Copy Ask AI @audiobuffer.event_handler ( "on_bot_turn_audio_data" ) async def on_bot_turn_audio_data ( buffer , audio : bytes , sample_rate : int , num_channels : int ): # Handle bot turn audio pass Parameters: buffer : The AudioBufferProcessor instance audio : Audio data from the bot’s speaking turn sample_rate : Sample rate in Hz num_channels : Always 1 (mono) ​ Audio Processing Features Automatic resampling : Converts incoming audio to the specified sample rate Buffer synchronization : Aligns user and bot audio streams temporally Silence insertion : Fills gaps in non-continuous audio streams to maintain timing Turn tracking : Monitors speaking turns when enable_turn_audio=True ​ Integration Notes ​ STT Audio Passthrough If using an STT service in your pipeline, enable audio passthrough to make audio available to the AudioBufferProcessor: Copy Ask AI stt = DeepgramSTTService( api_key = os.getenv( "DEEPGRAM_API_KEY" ), audio_passthrough = True , ) audio_passthrough is enabled by default. ​ Pipeline Placement Add the AudioBufferProcessor after transport.output() to capture both user and bot audio: Copy Ask AI pipeline = Pipeline([ transport.input(), # ... other processors ... transport.output(), audiobuffer, # Place after audio output # ... remaining processors ... ]) UserIdleProcessor KoalaFilter On this page Overview Constructor Parameters Properties sample_rate num_channels Methods start_recording() stop_recording() has_audio() Event Handlers on_audio_data on_track_audio_data on_user_turn_audio_data on_bot_turn_audio_data Audio Processing Features Integration Notes STT Audio Passthrough Pipeline Placement Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/audio_koala-filter_4cd6e2cf.txt b/audio_koala-filter_4cd6e2cf.txt
new file mode 100644
index 0000000000000000000000000000000000000000..7730261c21973f43e39fc55b4ee1c02bd66ca5e8
--- /dev/null
+++ b/audio_koala-filter_4cd6e2cf.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/audio/koala-filter
+Title: KoalaFilter - Pipecat
+==================================================
+
+KoalaFilter - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Audio Processing KoalaFilter Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing AudioBufferProcessor KoalaFilter KrispFilter NoisereduceFilter SileroVADAnalyzer SoundfileMixer Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview KoalaFilter is an audio processor that reduces background noise in real-time audio streams using Koala Noise Suppression technology from Picovoice. It inherits from BaseAudioFilter and processes audio frames to improve audio quality by removing unwanted noise. To use Koala, you need a Picovoice access key. Get started at Picovoice Console . ​ Installation The Koala filter requires additional dependencies: Copy Ask AI pip install "pipecat-ai[koala]" You’ll also need to set up your Koala access key as an environment variable: KOALA_ACCESS_KEY ​ Constructor Parameters ​ access_key str required Picovoice access key for using the Koala noise suppression service ​ Input Frames ​ FilterEnableFrame Frame Specific control frame to toggle filtering on/off Copy Ask AI from pipecat.frames.frames import FilterEnableFrame # Disable noise reduction await task.queue_frame(FilterEnableFrame( False )) # Re-enable noise reduction await task.queue_frame(FilterEnableFrame( True )) ​ Usage Example Copy Ask AI from pipecat.audio.filters.koala_filter import KoalaFilter transport = DailyTransport( room_url, token, "Respond bot" , DailyParams( audio_in_filter = KoalaFilter( access_key = os.getenv( "KOALA_ACCESS_KEY" )), # Enable Koala noise reduction audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer(), ), ) ​ Audio Flow ​ Notes Requires Picovoice access key Supports real-time audio processing Handles 16-bit PCM audio format Can be dynamically enabled/disabled Maintains audio quality while reducing noise Efficient processing for low latency Automatically handles audio frame buffering Sample rate must match Koala’s required sample rate AudioBufferProcessor KrispFilter On this page Overview Installation Constructor Parameters Input Frames Usage Example Audio Flow Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/audio_koala-filter_bef194f4.txt b/audio_koala-filter_bef194f4.txt
new file mode 100644
index 0000000000000000000000000000000000000000..2b84513326cfdd12b3f084d18ec04290da45166b
--- /dev/null
+++ b/audio_koala-filter_bef194f4.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/audio/koala-filter#audio-flow
+Title: KoalaFilter - Pipecat
+==================================================
+
+KoalaFilter - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Audio Processing KoalaFilter Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing AudioBufferProcessor KoalaFilter KrispFilter NoisereduceFilter SileroVADAnalyzer SoundfileMixer Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview KoalaFilter is an audio processor that reduces background noise in real-time audio streams using Koala Noise Suppression technology from Picovoice. It inherits from BaseAudioFilter and processes audio frames to improve audio quality by removing unwanted noise. To use Koala, you need a Picovoice access key. Get started at Picovoice Console . ​ Installation The Koala filter requires additional dependencies: Copy Ask AI pip install "pipecat-ai[koala]" You’ll also need to set up your Koala access key as an environment variable: KOALA_ACCESS_KEY ​ Constructor Parameters ​ access_key str required Picovoice access key for using the Koala noise suppression service ​ Input Frames ​ FilterEnableFrame Frame Specific control frame to toggle filtering on/off Copy Ask AI from pipecat.frames.frames import FilterEnableFrame # Disable noise reduction await task.queue_frame(FilterEnableFrame( False )) # Re-enable noise reduction await task.queue_frame(FilterEnableFrame( True )) ​ Usage Example Copy Ask AI from pipecat.audio.filters.koala_filter import KoalaFilter transport = DailyTransport( room_url, token, "Respond bot" , DailyParams( audio_in_filter = KoalaFilter( access_key = os.getenv( "KOALA_ACCESS_KEY" )), # Enable Koala noise reduction audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer(), ), ) ​ Audio Flow ​ Notes Requires Picovoice access key Supports real-time audio processing Handles 16-bit PCM audio format Can be dynamically enabled/disabled Maintains audio quality while reducing noise Efficient processing for low latency Automatically handles audio frame buffering Sample rate must match Koala’s required sample rate AudioBufferProcessor KrispFilter On this page Overview Installation Constructor Parameters Input Frames Usage Example Audio Flow Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/audio_koala-filter_e4ba46aa.txt b/audio_koala-filter_e4ba46aa.txt
new file mode 100644
index 0000000000000000000000000000000000000000..ba8728dd4e5bd4d4458ba9d9e22eb3508b7a5564
--- /dev/null
+++ b/audio_koala-filter_e4ba46aa.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/audio/koala-filter#notes
+Title: KoalaFilter - Pipecat
+==================================================
+
+KoalaFilter - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Audio Processing KoalaFilter Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing AudioBufferProcessor KoalaFilter KrispFilter NoisereduceFilter SileroVADAnalyzer SoundfileMixer Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview KoalaFilter is an audio processor that reduces background noise in real-time audio streams using Koala Noise Suppression technology from Picovoice. It inherits from BaseAudioFilter and processes audio frames to improve audio quality by removing unwanted noise. To use Koala, you need a Picovoice access key. Get started at Picovoice Console . ​ Installation The Koala filter requires additional dependencies: Copy Ask AI pip install "pipecat-ai[koala]" You’ll also need to set up your Koala access key as an environment variable: KOALA_ACCESS_KEY ​ Constructor Parameters ​ access_key str required Picovoice access key for using the Koala noise suppression service ​ Input Frames ​ FilterEnableFrame Frame Specific control frame to toggle filtering on/off Copy Ask AI from pipecat.frames.frames import FilterEnableFrame # Disable noise reduction await task.queue_frame(FilterEnableFrame( False )) # Re-enable noise reduction await task.queue_frame(FilterEnableFrame( True )) ​ Usage Example Copy Ask AI from pipecat.audio.filters.koala_filter import KoalaFilter transport = DailyTransport( room_url, token, "Respond bot" , DailyParams( audio_in_filter = KoalaFilter( access_key = os.getenv( "KOALA_ACCESS_KEY" )), # Enable Koala noise reduction audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer(), ), ) ​ Audio Flow ​ Notes Requires Picovoice access key Supports real-time audio processing Handles 16-bit PCM audio format Can be dynamically enabled/disabled Maintains audio quality while reducing noise Efficient processing for low latency Automatically handles audio frame buffering Sample rate must match Koala’s required sample rate AudioBufferProcessor KrispFilter On this page Overview Installation Constructor Parameters Input Frames Usage Example Audio Flow Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/audio_krisp-filter_06341763.txt b/audio_krisp-filter_06341763.txt
new file mode 100644
index 0000000000000000000000000000000000000000..61b447c56a0f3ccd766b91dd874085b95f1a14bb
--- /dev/null
+++ b/audio_krisp-filter_06341763.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/audio/krisp-filter#param-channels
+Title: KrispFilter - Pipecat
+==================================================
+
+KrispFilter - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Audio Processing KrispFilter Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing AudioBufferProcessor KoalaFilter KrispFilter NoisereduceFilter SileroVADAnalyzer SoundfileMixer Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview KrispFilter is an audio processor that reduces background noise in real-time audio streams using Krisp AI technology. It inherits from BaseAudioFilter and processes audio frames to improve audio quality by removing unwanted noise. To use Krisp, you need a Krisp SDK license. Get started at Krisp.ai . Looking for help getting started with Krisp and Pipecat? Checkout our Krisp noise cancellation guide . ​ Installation The Krisp filter requires additional dependencies: Copy Ask AI pip install "pipecat-ai[krisp]" ​ Environment Variables You need to provide the path to the Krisp model. This can either be done by setting the KRISP_MODEL_PATH environment variable or by setting the model_path in the constructor. ​ Constructor Parameters ​ sample_type str default: "PCM_16" Audio sample type format ​ channels int default: "1" Number of audio channels ​ model_path str default: "None" Path to the Krisp model file. You can set the model_path directly. Alternatively, you can set the KRISP_MODEL_PATH environment variable to the model file path. ​ Input Frames ​ FilterEnableFrame Frame Specific control frame to toggle filtering on/off Copy Ask AI from pipecat.frames.frames import FilterEnableFrame # Disable noise reduction await task.queue_frame(FilterEnableFrame( False )) # Re-enable noise reduction await task.queue_frame(FilterEnableFrame( True )) ​ Usage Example Copy Ask AI from pipecat.audio.filters.krisp_filter import KrispFilter transport = DailyTransport( room_url, token, "Respond bot" , DailyParams( audio_in_filter = KrispFilter(), # Enable Krisp noise reduction audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer(), ), ) ​ Audio Flow ​ Notes Requires Krisp SDK and model file to be available Supports real-time audio processing Supports additional features like background voice removal Handles PCM_16 audio format Thread-safe for pipeline processing Can be dynamically enabled/disabled Maintains audio quality while reducing noise Efficient processing for low latency KoalaFilter NoisereduceFilter On this page Overview Installation Environment Variables Constructor Parameters Input Frames Usage Example Audio Flow Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/audio_krisp-filter_2d4030bf.txt b/audio_krisp-filter_2d4030bf.txt
new file mode 100644
index 0000000000000000000000000000000000000000..28f2b1eccb4f6c893d426cb941bcfb92340ecab5
--- /dev/null
+++ b/audio_krisp-filter_2d4030bf.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/audio/krisp-filter#param-filter-enable-frame
+Title: KrispFilter - Pipecat
+==================================================
+
+KrispFilter - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Audio Processing KrispFilter Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing AudioBufferProcessor KoalaFilter KrispFilter NoisereduceFilter SileroVADAnalyzer SoundfileMixer Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview KrispFilter is an audio processor that reduces background noise in real-time audio streams using Krisp AI technology. It inherits from BaseAudioFilter and processes audio frames to improve audio quality by removing unwanted noise. To use Krisp, you need a Krisp SDK license. Get started at Krisp.ai . Looking for help getting started with Krisp and Pipecat? Checkout our Krisp noise cancellation guide . ​ Installation The Krisp filter requires additional dependencies: Copy Ask AI pip install "pipecat-ai[krisp]" ​ Environment Variables You need to provide the path to the Krisp model. This can either be done by setting the KRISP_MODEL_PATH environment variable or by setting the model_path in the constructor. ​ Constructor Parameters ​ sample_type str default: "PCM_16" Audio sample type format ​ channels int default: "1" Number of audio channels ​ model_path str default: "None" Path to the Krisp model file. You can set the model_path directly. Alternatively, you can set the KRISP_MODEL_PATH environment variable to the model file path. ​ Input Frames ​ FilterEnableFrame Frame Specific control frame to toggle filtering on/off Copy Ask AI from pipecat.frames.frames import FilterEnableFrame # Disable noise reduction await task.queue_frame(FilterEnableFrame( False )) # Re-enable noise reduction await task.queue_frame(FilterEnableFrame( True )) ​ Usage Example Copy Ask AI from pipecat.audio.filters.krisp_filter import KrispFilter transport = DailyTransport( room_url, token, "Respond bot" , DailyParams( audio_in_filter = KrispFilter(), # Enable Krisp noise reduction audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer(), ), ) ​ Audio Flow ​ Notes Requires Krisp SDK and model file to be available Supports real-time audio processing Supports additional features like background voice removal Handles PCM_16 audio format Thread-safe for pipeline processing Can be dynamically enabled/disabled Maintains audio quality while reducing noise Efficient processing for low latency KoalaFilter NoisereduceFilter On this page Overview Installation Environment Variables Constructor Parameters Input Frames Usage Example Audio Flow Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/audio_krisp-filter_db1f07bd.txt b/audio_krisp-filter_db1f07bd.txt
new file mode 100644
index 0000000000000000000000000000000000000000..2f4413136173c3022fa8e055eb4e5d1fec87cf0c
--- /dev/null
+++ b/audio_krisp-filter_db1f07bd.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/audio/krisp-filter#param-sample-type
+Title: KrispFilter - Pipecat
+==================================================
+
+KrispFilter - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Audio Processing KrispFilter Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing AudioBufferProcessor KoalaFilter KrispFilter NoisereduceFilter SileroVADAnalyzer SoundfileMixer Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview KrispFilter is an audio processor that reduces background noise in real-time audio streams using Krisp AI technology. It inherits from BaseAudioFilter and processes audio frames to improve audio quality by removing unwanted noise. To use Krisp, you need a Krisp SDK license. Get started at Krisp.ai . Looking for help getting started with Krisp and Pipecat? Checkout our Krisp noise cancellation guide . ​ Installation The Krisp filter requires additional dependencies: Copy Ask AI pip install "pipecat-ai[krisp]" ​ Environment Variables You need to provide the path to the Krisp model. This can either be done by setting the KRISP_MODEL_PATH environment variable or by setting the model_path in the constructor. ​ Constructor Parameters ​ sample_type str default: "PCM_16" Audio sample type format ​ channels int default: "1" Number of audio channels ​ model_path str default: "None" Path to the Krisp model file. You can set the model_path directly. Alternatively, you can set the KRISP_MODEL_PATH environment variable to the model file path. ​ Input Frames ​ FilterEnableFrame Frame Specific control frame to toggle filtering on/off Copy Ask AI from pipecat.frames.frames import FilterEnableFrame # Disable noise reduction await task.queue_frame(FilterEnableFrame( False )) # Re-enable noise reduction await task.queue_frame(FilterEnableFrame( True )) ​ Usage Example Copy Ask AI from pipecat.audio.filters.krisp_filter import KrispFilter transport = DailyTransport( room_url, token, "Respond bot" , DailyParams( audio_in_filter = KrispFilter(), # Enable Krisp noise reduction audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer(), ), ) ​ Audio Flow ​ Notes Requires Krisp SDK and model file to be available Supports real-time audio processing Supports additional features like background voice removal Handles PCM_16 audio format Thread-safe for pipeline processing Can be dynamically enabled/disabled Maintains audio quality while reducing noise Efficient processing for low latency KoalaFilter NoisereduceFilter On this page Overview Installation Environment Variables Constructor Parameters Input Frames Usage Example Audio Flow Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/audio_krisp-filter_e7d9e305.txt b/audio_krisp-filter_e7d9e305.txt
new file mode 100644
index 0000000000000000000000000000000000000000..a69bb519712814b7eb559f71f8eaa57cedb8c029
--- /dev/null
+++ b/audio_krisp-filter_e7d9e305.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/api-reference/utilities/audio/krisp-filter#next-steps
+Title: Overview - Pipecat
+==================================================
+
+Overview - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Get Started Overview Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Get Started Overview Installation & Setup Quickstart Core Concepts Next Steps & Examples Pipecat is an open source Python framework that handles the complex orchestration of AI services, network transport, audio processing, and multimodal interactions. “Multimodal” means you can use any combination of audio, video, images, and/or text in your interactions. And “real-time” means that things are happening quickly enough that it feels conversational—a “back-and-forth” with a bot, not submitting a query and waiting for results. ​ What You Can Build Voice Assistants Natural, real-time conversations with AI using speech recognition and synthesis Interactive Agents Personal coaches and meeting assistants that can understand context and provide guidance Multimodal Apps Applications that combine voice, video, images, and text for rich interactions Creative Tools Storytelling experiences and social companions that engage users Business Solutions Customer intake flows and support bots for automated business processes Complex Flows Structured conversations using Pipecat Flows for managing complex interactions ​ How It Works The flow of interactions in a Pipecat application is typically straightforward: The bot says something The user says something The bot says something The user says something This continues until the conversation naturally ends. While this flow seems simple, making it feel natural requires sophisticated real-time processing. ​ Real-time Processing Pipecat’s pipeline architecture handles both simple voice interactions and complex multimodal processing. Let’s look at how data flows through the system: Voice app Multimodal app 1 Send Audio Transmit and capture streamed audio from the user 2 Transcribe Speech Convert speech to text as the user is talking 3 Process with LLM Generate responses using a large language model 4 Convert to Speech Transform text responses into natural speech 5 Play Audio Stream the audio response back to the user 1 Send Audio Transmit and capture streamed audio from the user 2 Transcribe Speech Convert speech to text as the user is talking 3 Process with LLM Generate responses using a large language model 4 Convert to Speech Transform text responses into natural speech 5 Play Audio Stream the audio response back to the user 1 Send Audio and Video Transmit and capture audio, video, and image inputs simultaneously 2 Process Streams Handle multiple input streams in parallel 3 Model Processing Send combined inputs to multimodal models (like GPT-4V) 4 Generate Outputs Create various outputs (text, images, audio, etc.) 5 Coordinate Presentation Synchronize and present multiple output types In both cases, Pipecat: Processes responses as they stream in Handles multiple input/output modalities concurrently Manages resource allocation and synchronization Coordinates parallel processing tasks This architecture creates fluid, natural interactions without noticeable delays, whether you’re building a simple voice assistant or a complex multimodal application. Pipecat’s pipeline architecture is particularly valuable for managing the complexity of real-time, multimodal interactions, ensuring smooth data flow and proper synchronization regardless of the input/output types involved. Pipecat handles all this complexity for you, letting you focus on building your application rather than managing the underlying infrastructure. ​ Next Steps Ready to build your first Pipecat application? Installation & Setup Prepare your environment and install required dependencies Quickstart Build and run your first Pipecat application Core Concepts Learn about pipelines, frames, and real-time processing Use Cases Explore example implementations and patterns ​ Join Our Community Discord Community Connect with other developers, share your projects, and get support from the Pipecat team. Installation & Setup On this page What You Can Build How It Works Real-time Processing Next Steps Join Our Community Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/audio_krisp-filter_e84882be.txt b/audio_krisp-filter_e84882be.txt
new file mode 100644
index 0000000000000000000000000000000000000000..c2899fd6d0277319edf8d6bed35560ff9a1d6b98
--- /dev/null
+++ b/audio_krisp-filter_e84882be.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/audio/krisp-filter#installation
+Title: KrispFilter - Pipecat
+==================================================
+
+KrispFilter - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Audio Processing KrispFilter Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing AudioBufferProcessor KoalaFilter KrispFilter NoisereduceFilter SileroVADAnalyzer SoundfileMixer Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview KrispFilter is an audio processor that reduces background noise in real-time audio streams using Krisp AI technology. It inherits from BaseAudioFilter and processes audio frames to improve audio quality by removing unwanted noise. To use Krisp, you need a Krisp SDK license. Get started at Krisp.ai . Looking for help getting started with Krisp and Pipecat? Checkout our Krisp noise cancellation guide . ​ Installation The Krisp filter requires additional dependencies: Copy Ask AI pip install "pipecat-ai[krisp]" ​ Environment Variables You need to provide the path to the Krisp model. This can either be done by setting the KRISP_MODEL_PATH environment variable or by setting the model_path in the constructor. ​ Constructor Parameters ​ sample_type str default: "PCM_16" Audio sample type format ​ channels int default: "1" Number of audio channels ​ model_path str default: "None" Path to the Krisp model file. You can set the model_path directly. Alternatively, you can set the KRISP_MODEL_PATH environment variable to the model file path. ​ Input Frames ​ FilterEnableFrame Frame Specific control frame to toggle filtering on/off Copy Ask AI from pipecat.frames.frames import FilterEnableFrame # Disable noise reduction await task.queue_frame(FilterEnableFrame( False )) # Re-enable noise reduction await task.queue_frame(FilterEnableFrame( True )) ​ Usage Example Copy Ask AI from pipecat.audio.filters.krisp_filter import KrispFilter transport = DailyTransport( room_url, token, "Respond bot" , DailyParams( audio_in_filter = KrispFilter(), # Enable Krisp noise reduction audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer(), ), ) ​ Audio Flow ​ Notes Requires Krisp SDK and model file to be available Supports real-time audio processing Supports additional features like background voice removal Handles PCM_16 audio format Thread-safe for pipeline processing Can be dynamically enabled/disabled Maintains audio quality while reducing noise Efficient processing for low latency KoalaFilter NoisereduceFilter On this page Overview Installation Environment Variables Constructor Parameters Input Frames Usage Example Audio Flow Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/audio_noisereduce-filter_15677196.txt b/audio_noisereduce-filter_15677196.txt
new file mode 100644
index 0000000000000000000000000000000000000000..db3fedd26616240c54db2e4a0f1d661506f7d755
--- /dev/null
+++ b/audio_noisereduce-filter_15677196.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/audio/noisereduce-filter#usage-example
+Title: NoisereduceFilter - Pipecat
+==================================================
+
+NoisereduceFilter - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Audio Processing NoisereduceFilter Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing AudioBufferProcessor KoalaFilter KrispFilter NoisereduceFilter SileroVADAnalyzer SoundfileMixer Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview NoisereduceFilter is an audio processor that reduces background noise in real-time audio streams using the noisereduce library. It inherits from BaseAudioFilter and processes audio frames to improve audio quality by removing unwanted noise. ​ Installation The noisereduce filter requires additional dependencies: Copy Ask AI pip install "pipecat-ai[noisereduce]" ​ Constructor Parameters This filter has no configurable parameters in its constructor. ​ Input Frames ​ FilterEnableFrame Frame Specific control frame to toggle filtering on/off Copy Ask AI from pipecat.frames.frames import FilterEnableFrame # Disable noise reduction await task.queue_frame(FilterEnableFrame( False )) # Re-enable noise reduction await task.queue_frame(FilterEnableFrame( True )) ​ Usage Example Copy Ask AI from pipecat.audio.filters.noisereduce_filter import NoisereduceFilter transport = DailyTransport( room_url, token, "Respond bot" , DailyParams( audio_in_filter = NoisereduceFilter(), # Enable noise reduction audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer(), ), ) ​ Audio Flow ​ Notes Lightweight alternative to Krisp for noise reduction Supports real-time audio processing Handles PCM_16 audio format Thread-safe for pipeline processing Can be dynamically enabled/disabled No additional configuration required Uses statistical noise reduction techniques KrispFilter SileroVADAnalyzer On this page Overview Installation Constructor Parameters Input Frames Usage Example Audio Flow Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/audio_noisereduce-filter_45a37635.txt b/audio_noisereduce-filter_45a37635.txt
new file mode 100644
index 0000000000000000000000000000000000000000..7120990eb54d20ea3b052038d02d5c1274166e9b
--- /dev/null
+++ b/audio_noisereduce-filter_45a37635.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/audio/noisereduce-filter#constructor-parameters
+Title: NoisereduceFilter - Pipecat
+==================================================
+
+NoisereduceFilter - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Audio Processing NoisereduceFilter Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing AudioBufferProcessor KoalaFilter KrispFilter NoisereduceFilter SileroVADAnalyzer SoundfileMixer Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview NoisereduceFilter is an audio processor that reduces background noise in real-time audio streams using the noisereduce library. It inherits from BaseAudioFilter and processes audio frames to improve audio quality by removing unwanted noise. ​ Installation The noisereduce filter requires additional dependencies: Copy Ask AI pip install "pipecat-ai[noisereduce]" ​ Constructor Parameters This filter has no configurable parameters in its constructor. ​ Input Frames ​ FilterEnableFrame Frame Specific control frame to toggle filtering on/off Copy Ask AI from pipecat.frames.frames import FilterEnableFrame # Disable noise reduction await task.queue_frame(FilterEnableFrame( False )) # Re-enable noise reduction await task.queue_frame(FilterEnableFrame( True )) ​ Usage Example Copy Ask AI from pipecat.audio.filters.noisereduce_filter import NoisereduceFilter transport = DailyTransport( room_url, token, "Respond bot" , DailyParams( audio_in_filter = NoisereduceFilter(), # Enable noise reduction audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer(), ), ) ​ Audio Flow ​ Notes Lightweight alternative to Krisp for noise reduction Supports real-time audio processing Handles PCM_16 audio format Thread-safe for pipeline processing Can be dynamically enabled/disabled No additional configuration required Uses statistical noise reduction techniques KrispFilter SileroVADAnalyzer On this page Overview Installation Constructor Parameters Input Frames Usage Example Audio Flow Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/audio_noisereduce-filter_d0480795.txt b/audio_noisereduce-filter_d0480795.txt
new file mode 100644
index 0000000000000000000000000000000000000000..5bd039e3f74bac7b75f5e5342d106732e3265ab3
--- /dev/null
+++ b/audio_noisereduce-filter_d0480795.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/audio/noisereduce-filter#installation
+Title: NoisereduceFilter - Pipecat
+==================================================
+
+NoisereduceFilter - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Audio Processing NoisereduceFilter Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing AudioBufferProcessor KoalaFilter KrispFilter NoisereduceFilter SileroVADAnalyzer SoundfileMixer Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview NoisereduceFilter is an audio processor that reduces background noise in real-time audio streams using the noisereduce library. It inherits from BaseAudioFilter and processes audio frames to improve audio quality by removing unwanted noise. ​ Installation The noisereduce filter requires additional dependencies: Copy Ask AI pip install "pipecat-ai[noisereduce]" ​ Constructor Parameters This filter has no configurable parameters in its constructor. ​ Input Frames ​ FilterEnableFrame Frame Specific control frame to toggle filtering on/off Copy Ask AI from pipecat.frames.frames import FilterEnableFrame # Disable noise reduction await task.queue_frame(FilterEnableFrame( False )) # Re-enable noise reduction await task.queue_frame(FilterEnableFrame( True )) ​ Usage Example Copy Ask AI from pipecat.audio.filters.noisereduce_filter import NoisereduceFilter transport = DailyTransport( room_url, token, "Respond bot" , DailyParams( audio_in_filter = NoisereduceFilter(), # Enable noise reduction audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer(), ), ) ​ Audio Flow ​ Notes Lightweight alternative to Krisp for noise reduction Supports real-time audio processing Handles PCM_16 audio format Thread-safe for pipeline processing Can be dynamically enabled/disabled No additional configuration required Uses statistical noise reduction techniques KrispFilter SileroVADAnalyzer On this page Overview Installation Constructor Parameters Input Frames Usage Example Audio Flow Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/audio_silero-vad-analyzer_6edfde18.txt b/audio_silero-vad-analyzer_6edfde18.txt
new file mode 100644
index 0000000000000000000000000000000000000000..694178572e5a96bd7ad3d6261d52639186d79109
--- /dev/null
+++ b/audio_silero-vad-analyzer_6edfde18.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/audio/silero-vad-analyzer#constructor-parameters
+Title: SileroVADAnalyzer - Pipecat
+==================================================
+
+SileroVADAnalyzer - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Audio Processing SileroVADAnalyzer Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing AudioBufferProcessor KoalaFilter KrispFilter NoisereduceFilter SileroVADAnalyzer SoundfileMixer Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview SileroVADAnalyzer is a Voice Activity Detection (VAD) analyzer that uses the Silero VAD ONNX model to detect speech in audio streams. It provides high-accuracy speech detection with efficient processing using ONNX runtime. ​ Installation The Silero VAD analyzer requires additional dependencies: Copy Ask AI pip install "pipecat-ai[silero]" ​ Constructor Parameters ​ sample_rate int default: "None" Audio sample rate in Hz. Must be either 8000 or 16000. ​ params VADParams default: "VADParams()" Voice Activity Detection parameters object Show properties ​ confidence float default: "0.7" Confidence threshold for speech detection. Higher values make detection more strict. Must be between 0 and 1. ​ start_secs float default: "0.2" Time in seconds that speech must be detected before transitioning to SPEAKING state. ​ stop_secs float default: "0.8" Time in seconds of silence required before transitioning back to QUIET state. ​ min_volume float default: "0.6" Minimum audio volume threshold for speech detection. Must be between 0 and 1. ​ Usage Example Copy Ask AI transport = DailyTransport( room_url, token, "Respond bot" , DailyParams( audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer( params = VADParams( stop_secs = 0.5 )), ), ) ​ Technical Details ​ Sample Rate Requirements The analyzer supports two sample rates: 8000 Hz (256 samples per frame) 16000 Hz (512 samples per frame) Model Management Uses ONNX runtime for efficient inference Automatically resets model state every 5 seconds to manage memory Runs on CPU by default for consistent performance Includes built-in model file ​ Notes High-accuracy speech detection Efficient ONNX-based processing Automatic memory management Thread-safe for pipeline processing Built-in model file included CPU-optimized inference Supports 8kHz and 16kHz audio NoisereduceFilter SoundfileMixer On this page Overview Installation Constructor Parameters Usage Example Technical Details Sample Rate Requirements Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/audio_silero-vad-analyzer_9d32f47f.txt b/audio_silero-vad-analyzer_9d32f47f.txt
new file mode 100644
index 0000000000000000000000000000000000000000..33925fab1616093a5089e7d4d813fff3bd7a2e9d
--- /dev/null
+++ b/audio_silero-vad-analyzer_9d32f47f.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/audio/silero-vad-analyzer#sample-rate-requirements
+Title: SileroVADAnalyzer - Pipecat
+==================================================
+
+SileroVADAnalyzer - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Audio Processing SileroVADAnalyzer Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing AudioBufferProcessor KoalaFilter KrispFilter NoisereduceFilter SileroVADAnalyzer SoundfileMixer Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview SileroVADAnalyzer is a Voice Activity Detection (VAD) analyzer that uses the Silero VAD ONNX model to detect speech in audio streams. It provides high-accuracy speech detection with efficient processing using ONNX runtime. ​ Installation The Silero VAD analyzer requires additional dependencies: Copy Ask AI pip install "pipecat-ai[silero]" ​ Constructor Parameters ​ sample_rate int default: "None" Audio sample rate in Hz. Must be either 8000 or 16000. ​ params VADParams default: "VADParams()" Voice Activity Detection parameters object Show properties ​ confidence float default: "0.7" Confidence threshold for speech detection. Higher values make detection more strict. Must be between 0 and 1. ​ start_secs float default: "0.2" Time in seconds that speech must be detected before transitioning to SPEAKING state. ​ stop_secs float default: "0.8" Time in seconds of silence required before transitioning back to QUIET state. ​ min_volume float default: "0.6" Minimum audio volume threshold for speech detection. Must be between 0 and 1. ​ Usage Example Copy Ask AI transport = DailyTransport( room_url, token, "Respond bot" , DailyParams( audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer( params = VADParams( stop_secs = 0.5 )), ), ) ​ Technical Details ​ Sample Rate Requirements The analyzer supports two sample rates: 8000 Hz (256 samples per frame) 16000 Hz (512 samples per frame) Model Management Uses ONNX runtime for efficient inference Automatically resets model state every 5 seconds to manage memory Runs on CPU by default for consistent performance Includes built-in model file ​ Notes High-accuracy speech detection Efficient ONNX-based processing Automatic memory management Thread-safe for pipeline processing Built-in model file included CPU-optimized inference Supports 8kHz and 16kHz audio NoisereduceFilter SoundfileMixer On this page Overview Installation Constructor Parameters Usage Example Technical Details Sample Rate Requirements Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/audio_silero-vad-analyzer_a72e4ef6.txt b/audio_silero-vad-analyzer_a72e4ef6.txt
new file mode 100644
index 0000000000000000000000000000000000000000..18102dd3ce76518a0709c408a2e46c9dde9a8ffa
--- /dev/null
+++ b/audio_silero-vad-analyzer_a72e4ef6.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/audio/silero-vad-analyzer#technical-details
+Title: SileroVADAnalyzer - Pipecat
+==================================================
+
+SileroVADAnalyzer - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Audio Processing SileroVADAnalyzer Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing AudioBufferProcessor KoalaFilter KrispFilter NoisereduceFilter SileroVADAnalyzer SoundfileMixer Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview SileroVADAnalyzer is a Voice Activity Detection (VAD) analyzer that uses the Silero VAD ONNX model to detect speech in audio streams. It provides high-accuracy speech detection with efficient processing using ONNX runtime. ​ Installation The Silero VAD analyzer requires additional dependencies: Copy Ask AI pip install "pipecat-ai[silero]" ​ Constructor Parameters ​ sample_rate int default: "None" Audio sample rate in Hz. Must be either 8000 or 16000. ​ params VADParams default: "VADParams()" Voice Activity Detection parameters object Show properties ​ confidence float default: "0.7" Confidence threshold for speech detection. Higher values make detection more strict. Must be between 0 and 1. ​ start_secs float default: "0.2" Time in seconds that speech must be detected before transitioning to SPEAKING state. ​ stop_secs float default: "0.8" Time in seconds of silence required before transitioning back to QUIET state. ​ min_volume float default: "0.6" Minimum audio volume threshold for speech detection. Must be between 0 and 1. ​ Usage Example Copy Ask AI transport = DailyTransport( room_url, token, "Respond bot" , DailyParams( audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer( params = VADParams( stop_secs = 0.5 )), ), ) ​ Technical Details ​ Sample Rate Requirements The analyzer supports two sample rates: 8000 Hz (256 samples per frame) 16000 Hz (512 samples per frame) Model Management Uses ONNX runtime for efficient inference Automatically resets model state every 5 seconds to manage memory Runs on CPU by default for consistent performance Includes built-in model file ​ Notes High-accuracy speech detection Efficient ONNX-based processing Automatic memory management Thread-safe for pipeline processing Built-in model file included CPU-optimized inference Supports 8kHz and 16kHz audio NoisereduceFilter SoundfileMixer On this page Overview Installation Constructor Parameters Usage Example Technical Details Sample Rate Requirements Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/audio_silero-vad-analyzer_a747eddc.txt b/audio_silero-vad-analyzer_a747eddc.txt
new file mode 100644
index 0000000000000000000000000000000000000000..37d4fa3bd4577558f0d6b1996f84f2d57a601190
--- /dev/null
+++ b/audio_silero-vad-analyzer_a747eddc.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/audio/silero-vad-analyzer#usage-example
+Title: SileroVADAnalyzer - Pipecat
+==================================================
+
+SileroVADAnalyzer - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Audio Processing SileroVADAnalyzer Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing AudioBufferProcessor KoalaFilter KrispFilter NoisereduceFilter SileroVADAnalyzer SoundfileMixer Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview SileroVADAnalyzer is a Voice Activity Detection (VAD) analyzer that uses the Silero VAD ONNX model to detect speech in audio streams. It provides high-accuracy speech detection with efficient processing using ONNX runtime. ​ Installation The Silero VAD analyzer requires additional dependencies: Copy Ask AI pip install "pipecat-ai[silero]" ​ Constructor Parameters ​ sample_rate int default: "None" Audio sample rate in Hz. Must be either 8000 or 16000. ​ params VADParams default: "VADParams()" Voice Activity Detection parameters object Show properties ​ confidence float default: "0.7" Confidence threshold for speech detection. Higher values make detection more strict. Must be between 0 and 1. ​ start_secs float default: "0.2" Time in seconds that speech must be detected before transitioning to SPEAKING state. ​ stop_secs float default: "0.8" Time in seconds of silence required before transitioning back to QUIET state. ​ min_volume float default: "0.6" Minimum audio volume threshold for speech detection. Must be between 0 and 1. ​ Usage Example Copy Ask AI transport = DailyTransport( room_url, token, "Respond bot" , DailyParams( audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer( params = VADParams( stop_secs = 0.5 )), ), ) ​ Technical Details ​ Sample Rate Requirements The analyzer supports two sample rates: 8000 Hz (256 samples per frame) 16000 Hz (512 samples per frame) Model Management Uses ONNX runtime for efficient inference Automatically resets model state every 5 seconds to manage memory Runs on CPU by default for consistent performance Includes built-in model file ​ Notes High-accuracy speech detection Efficient ONNX-based processing Automatic memory management Thread-safe for pipeline processing Built-in model file included CPU-optimized inference Supports 8kHz and 16kHz audio NoisereduceFilter SoundfileMixer On this page Overview Installation Constructor Parameters Usage Example Technical Details Sample Rate Requirements Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/audio_soundfile-mixer_0e8c44c2.txt b/audio_soundfile-mixer_0e8c44c2.txt
new file mode 100644
index 0000000000000000000000000000000000000000..ace858e82827ba6a042286f130fbefb47eb62a6d
--- /dev/null
+++ b/audio_soundfile-mixer_0e8c44c2.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/audio/soundfile-mixer#overview
+Title: SoundfileMixer - Pipecat
+==================================================
+
+SoundfileMixer - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Audio Processing SoundfileMixer Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing AudioBufferProcessor KoalaFilter KrispFilter NoisereduceFilter SileroVADAnalyzer SoundfileMixer Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview SoundfileMixer is an audio mixer that combines incoming audio with audio from files. It supports multiple audio file formats through the soundfile library and can handle runtime volume adjustments and sound switching. ​ Installation The soundfile mixer requires additional dependencies: Copy Ask AI pip install "pipecat-ai[soundfile]" ​ Constructor Parameters ​ sound_files Mapping[str, str] required Dictionary mapping sound names to file paths. Files must be mono (single channel). ​ default_sound str required Name of the default sound to play (must be a key in sound_files). ​ volume float default: "0.4" Initial volume for the mixed sound. Values typically range from 0.0 to 1.0, but can go higher. ​ loop bool default: "true" Whether to loop the sound file when it reaches the end. ​ Control Frames ​ MixerUpdateSettingsFrame Frame Updates mixer settings at runtime Show properties ​ sound str Changes the current playing sound (must be a key in sound_files) ​ volume float Updates the mixing volume ​ loop bool Updates whether the sound should loop ​ MixerEnableFrame Frame Enables or disables the mixer Show properties ​ enable bool Whether mixing should be enabled ​ Usage Example Copy Ask AI # Initialize mixer with sound files mixer = SoundfileMixer( sound_files = { "office" : "office_ambience.wav" }, default_sound = "office" , volume = 2.0 , ) # Add to transport transport = DailyTransport( room_url, token, "Audio Bot" , DailyParams( audio_out_enabled = True , audio_out_mixer = mixer, ), ) # Control mixer at runtime await task.queue_frame(MixerUpdateSettingsFrame({ "volume" : 0.5 })) await task.queue_frame(MixerEnableFrame( False )) # Disable mixing await task.queue_frame(MixerEnableFrame( True )) # Enable mixing ​ Notes Supports any audio format that soundfile can read Automatically resamples audio files to match output sample rate Files must be mono (single channel) Thread-safe for pipeline processing Can dynamically switch between multiple sound files Volume can be adjusted in real-time Mixing can be enabled/disabled on demand SileroVADAnalyzer FrameFilter On this page Overview Installation Constructor Parameters Control Frames Usage Example Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/audio_soundfile-mixer_14467662.txt b/audio_soundfile-mixer_14467662.txt
new file mode 100644
index 0000000000000000000000000000000000000000..a674a51e0e5a739c1b0812671329679a68512a09
--- /dev/null
+++ b/audio_soundfile-mixer_14467662.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/audio/soundfile-mixer#param-loop
+Title: SoundfileMixer - Pipecat
+==================================================
+
+SoundfileMixer - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Audio Processing SoundfileMixer Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing AudioBufferProcessor KoalaFilter KrispFilter NoisereduceFilter SileroVADAnalyzer SoundfileMixer Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview SoundfileMixer is an audio mixer that combines incoming audio with audio from files. It supports multiple audio file formats through the soundfile library and can handle runtime volume adjustments and sound switching. ​ Installation The soundfile mixer requires additional dependencies: Copy Ask AI pip install "pipecat-ai[soundfile]" ​ Constructor Parameters ​ sound_files Mapping[str, str] required Dictionary mapping sound names to file paths. Files must be mono (single channel). ​ default_sound str required Name of the default sound to play (must be a key in sound_files). ​ volume float default: "0.4" Initial volume for the mixed sound. Values typically range from 0.0 to 1.0, but can go higher. ​ loop bool default: "true" Whether to loop the sound file when it reaches the end. ​ Control Frames ​ MixerUpdateSettingsFrame Frame Updates mixer settings at runtime Show properties ​ sound str Changes the current playing sound (must be a key in sound_files) ​ volume float Updates the mixing volume ​ loop bool Updates whether the sound should loop ​ MixerEnableFrame Frame Enables or disables the mixer Show properties ​ enable bool Whether mixing should be enabled ​ Usage Example Copy Ask AI # Initialize mixer with sound files mixer = SoundfileMixer( sound_files = { "office" : "office_ambience.wav" }, default_sound = "office" , volume = 2.0 , ) # Add to transport transport = DailyTransport( room_url, token, "Audio Bot" , DailyParams( audio_out_enabled = True , audio_out_mixer = mixer, ), ) # Control mixer at runtime await task.queue_frame(MixerUpdateSettingsFrame({ "volume" : 0.5 })) await task.queue_frame(MixerEnableFrame( False )) # Disable mixing await task.queue_frame(MixerEnableFrame( True )) # Enable mixing ​ Notes Supports any audio format that soundfile can read Automatically resamples audio files to match output sample rate Files must be mono (single channel) Thread-safe for pipeline processing Can dynamically switch between multiple sound files Volume can be adjusted in real-time Mixing can be enabled/disabled on demand SileroVADAnalyzer FrameFilter On this page Overview Installation Constructor Parameters Control Frames Usage Example Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/audio_soundfile-mixer_1684ce92.txt b/audio_soundfile-mixer_1684ce92.txt
new file mode 100644
index 0000000000000000000000000000000000000000..bc2449187f4c34bce16066ad30c5e6c8afe47c39
--- /dev/null
+++ b/audio_soundfile-mixer_1684ce92.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/audio/soundfile-mixer#param-sound-files
+Title: SoundfileMixer - Pipecat
+==================================================
+
+SoundfileMixer - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Audio Processing SoundfileMixer Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing AudioBufferProcessor KoalaFilter KrispFilter NoisereduceFilter SileroVADAnalyzer SoundfileMixer Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview SoundfileMixer is an audio mixer that combines incoming audio with audio from files. It supports multiple audio file formats through the soundfile library and can handle runtime volume adjustments and sound switching. ​ Installation The soundfile mixer requires additional dependencies: Copy Ask AI pip install "pipecat-ai[soundfile]" ​ Constructor Parameters ​ sound_files Mapping[str, str] required Dictionary mapping sound names to file paths. Files must be mono (single channel). ​ default_sound str required Name of the default sound to play (must be a key in sound_files). ​ volume float default: "0.4" Initial volume for the mixed sound. Values typically range from 0.0 to 1.0, but can go higher. ​ loop bool default: "true" Whether to loop the sound file when it reaches the end. ​ Control Frames ​ MixerUpdateSettingsFrame Frame Updates mixer settings at runtime Show properties ​ sound str Changes the current playing sound (must be a key in sound_files) ​ volume float Updates the mixing volume ​ loop bool Updates whether the sound should loop ​ MixerEnableFrame Frame Enables or disables the mixer Show properties ​ enable bool Whether mixing should be enabled ​ Usage Example Copy Ask AI # Initialize mixer with sound files mixer = SoundfileMixer( sound_files = { "office" : "office_ambience.wav" }, default_sound = "office" , volume = 2.0 , ) # Add to transport transport = DailyTransport( room_url, token, "Audio Bot" , DailyParams( audio_out_enabled = True , audio_out_mixer = mixer, ), ) # Control mixer at runtime await task.queue_frame(MixerUpdateSettingsFrame({ "volume" : 0.5 })) await task.queue_frame(MixerEnableFrame( False )) # Disable mixing await task.queue_frame(MixerEnableFrame( True )) # Enable mixing ​ Notes Supports any audio format that soundfile can read Automatically resamples audio files to match output sample rate Files must be mono (single channel) Thread-safe for pipeline processing Can dynamically switch between multiple sound files Volume can be adjusted in real-time Mixing can be enabled/disabled on demand SileroVADAnalyzer FrameFilter On this page Overview Installation Constructor Parameters Control Frames Usage Example Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/audio_soundfile-mixer_49840d70.txt b/audio_soundfile-mixer_49840d70.txt
new file mode 100644
index 0000000000000000000000000000000000000000..570c73b91d49f3b12212d7634890b20e5fe200ac
--- /dev/null
+++ b/audio_soundfile-mixer_49840d70.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/audio/soundfile-mixer#param-enable
+Title: SoundfileMixer - Pipecat
+==================================================
+
+SoundfileMixer - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Audio Processing SoundfileMixer Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing AudioBufferProcessor KoalaFilter KrispFilter NoisereduceFilter SileroVADAnalyzer SoundfileMixer Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview SoundfileMixer is an audio mixer that combines incoming audio with audio from files. It supports multiple audio file formats through the soundfile library and can handle runtime volume adjustments and sound switching. ​ Installation The soundfile mixer requires additional dependencies: Copy Ask AI pip install "pipecat-ai[soundfile]" ​ Constructor Parameters ​ sound_files Mapping[str, str] required Dictionary mapping sound names to file paths. Files must be mono (single channel). ​ default_sound str required Name of the default sound to play (must be a key in sound_files). ​ volume float default: "0.4" Initial volume for the mixed sound. Values typically range from 0.0 to 1.0, but can go higher. ​ loop bool default: "true" Whether to loop the sound file when it reaches the end. ​ Control Frames ​ MixerUpdateSettingsFrame Frame Updates mixer settings at runtime Show properties ​ sound str Changes the current playing sound (must be a key in sound_files) ​ volume float Updates the mixing volume ​ loop bool Updates whether the sound should loop ​ MixerEnableFrame Frame Enables or disables the mixer Show properties ​ enable bool Whether mixing should be enabled ​ Usage Example Copy Ask AI # Initialize mixer with sound files mixer = SoundfileMixer( sound_files = { "office" : "office_ambience.wav" }, default_sound = "office" , volume = 2.0 , ) # Add to transport transport = DailyTransport( room_url, token, "Audio Bot" , DailyParams( audio_out_enabled = True , audio_out_mixer = mixer, ), ) # Control mixer at runtime await task.queue_frame(MixerUpdateSettingsFrame({ "volume" : 0.5 })) await task.queue_frame(MixerEnableFrame( False )) # Disable mixing await task.queue_frame(MixerEnableFrame( True )) # Enable mixing ​ Notes Supports any audio format that soundfile can read Automatically resamples audio files to match output sample rate Files must be mono (single channel) Thread-safe for pipeline processing Can dynamically switch between multiple sound files Volume can be adjusted in real-time Mixing can be enabled/disabled on demand SileroVADAnalyzer FrameFilter On this page Overview Installation Constructor Parameters Control Frames Usage Example Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/audio_soundfile-mixer_5aa2a166.txt b/audio_soundfile-mixer_5aa2a166.txt
new file mode 100644
index 0000000000000000000000000000000000000000..1dab5663e7ab45c2b625e017dde27f2db1f06c65
--- /dev/null
+++ b/audio_soundfile-mixer_5aa2a166.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/audio/soundfile-mixer#param-loop-1
+Title: SoundfileMixer - Pipecat
+==================================================
+
+SoundfileMixer - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Audio Processing SoundfileMixer Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing AudioBufferProcessor KoalaFilter KrispFilter NoisereduceFilter SileroVADAnalyzer SoundfileMixer Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview SoundfileMixer is an audio mixer that combines incoming audio with audio from files. It supports multiple audio file formats through the soundfile library and can handle runtime volume adjustments and sound switching. ​ Installation The soundfile mixer requires additional dependencies: Copy Ask AI pip install "pipecat-ai[soundfile]" ​ Constructor Parameters ​ sound_files Mapping[str, str] required Dictionary mapping sound names to file paths. Files must be mono (single channel). ​ default_sound str required Name of the default sound to play (must be a key in sound_files). ​ volume float default: "0.4" Initial volume for the mixed sound. Values typically range from 0.0 to 1.0, but can go higher. ​ loop bool default: "true" Whether to loop the sound file when it reaches the end. ​ Control Frames ​ MixerUpdateSettingsFrame Frame Updates mixer settings at runtime Show properties ​ sound str Changes the current playing sound (must be a key in sound_files) ​ volume float Updates the mixing volume ​ loop bool Updates whether the sound should loop ​ MixerEnableFrame Frame Enables or disables the mixer Show properties ​ enable bool Whether mixing should be enabled ​ Usage Example Copy Ask AI # Initialize mixer with sound files mixer = SoundfileMixer( sound_files = { "office" : "office_ambience.wav" }, default_sound = "office" , volume = 2.0 , ) # Add to transport transport = DailyTransport( room_url, token, "Audio Bot" , DailyParams( audio_out_enabled = True , audio_out_mixer = mixer, ), ) # Control mixer at runtime await task.queue_frame(MixerUpdateSettingsFrame({ "volume" : 0.5 })) await task.queue_frame(MixerEnableFrame( False )) # Disable mixing await task.queue_frame(MixerEnableFrame( True )) # Enable mixing ​ Notes Supports any audio format that soundfile can read Automatically resamples audio files to match output sample rate Files must be mono (single channel) Thread-safe for pipeline processing Can dynamically switch between multiple sound files Volume can be adjusted in real-time Mixing can be enabled/disabled on demand SileroVADAnalyzer FrameFilter On this page Overview Installation Constructor Parameters Control Frames Usage Example Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/audio_soundfile-mixer_d2c38853.txt b/audio_soundfile-mixer_d2c38853.txt
new file mode 100644
index 0000000000000000000000000000000000000000..53697a49ea4bb4a6c9e7361d298b5f87b8881bbb
--- /dev/null
+++ b/audio_soundfile-mixer_d2c38853.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/audio/soundfile-mixer#constructor-parameters
+Title: SoundfileMixer - Pipecat
+==================================================
+
+SoundfileMixer - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Audio Processing SoundfileMixer Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing AudioBufferProcessor KoalaFilter KrispFilter NoisereduceFilter SileroVADAnalyzer SoundfileMixer Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview SoundfileMixer is an audio mixer that combines incoming audio with audio from files. It supports multiple audio file formats through the soundfile library and can handle runtime volume adjustments and sound switching. ​ Installation The soundfile mixer requires additional dependencies: Copy Ask AI pip install "pipecat-ai[soundfile]" ​ Constructor Parameters ​ sound_files Mapping[str, str] required Dictionary mapping sound names to file paths. Files must be mono (single channel). ​ default_sound str required Name of the default sound to play (must be a key in sound_files). ​ volume float default: "0.4" Initial volume for the mixed sound. Values typically range from 0.0 to 1.0, but can go higher. ​ loop bool default: "true" Whether to loop the sound file when it reaches the end. ​ Control Frames ​ MixerUpdateSettingsFrame Frame Updates mixer settings at runtime Show properties ​ sound str Changes the current playing sound (must be a key in sound_files) ​ volume float Updates the mixing volume ​ loop bool Updates whether the sound should loop ​ MixerEnableFrame Frame Enables or disables the mixer Show properties ​ enable bool Whether mixing should be enabled ​ Usage Example Copy Ask AI # Initialize mixer with sound files mixer = SoundfileMixer( sound_files = { "office" : "office_ambience.wav" }, default_sound = "office" , volume = 2.0 , ) # Add to transport transport = DailyTransport( room_url, token, "Audio Bot" , DailyParams( audio_out_enabled = True , audio_out_mixer = mixer, ), ) # Control mixer at runtime await task.queue_frame(MixerUpdateSettingsFrame({ "volume" : 0.5 })) await task.queue_frame(MixerEnableFrame( False )) # Disable mixing await task.queue_frame(MixerEnableFrame( True )) # Enable mixing ​ Notes Supports any audio format that soundfile can read Automatically resamples audio files to match output sample rate Files must be mono (single channel) Thread-safe for pipeline processing Can dynamically switch between multiple sound files Volume can be adjusted in real-time Mixing can be enabled/disabled on demand SileroVADAnalyzer FrameFilter On this page Overview Installation Constructor Parameters Control Frames Usage Example Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/base-classes_media_af1c223c.txt b/base-classes_media_af1c223c.txt
new file mode 100644
index 0000000000000000000000000000000000000000..017679ef8769a947b6606cbab0d5ccae26273331
--- /dev/null
+++ b/base-classes_media_af1c223c.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/base-classes/media#join-our-community
+Title: Overview - Pipecat
+==================================================
+
+Overview - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Get Started Overview Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Get Started Overview Installation & Setup Quickstart Core Concepts Next Steps & Examples Pipecat is an open source Python framework that handles the complex orchestration of AI services, network transport, audio processing, and multimodal interactions. “Multimodal” means you can use any combination of audio, video, images, and/or text in your interactions. And “real-time” means that things are happening quickly enough that it feels conversational—a “back-and-forth” with a bot, not submitting a query and waiting for results. ​ What You Can Build Voice Assistants Natural, real-time conversations with AI using speech recognition and synthesis Interactive Agents Personal coaches and meeting assistants that can understand context and provide guidance Multimodal Apps Applications that combine voice, video, images, and text for rich interactions Creative Tools Storytelling experiences and social companions that engage users Business Solutions Customer intake flows and support bots for automated business processes Complex Flows Structured conversations using Pipecat Flows for managing complex interactions ​ How It Works The flow of interactions in a Pipecat application is typically straightforward: The bot says something The user says something The bot says something The user says something This continues until the conversation naturally ends. While this flow seems simple, making it feel natural requires sophisticated real-time processing. ​ Real-time Processing Pipecat’s pipeline architecture handles both simple voice interactions and complex multimodal processing. Let’s look at how data flows through the system: Voice app Multimodal app 1 Send Audio Transmit and capture streamed audio from the user 2 Transcribe Speech Convert speech to text as the user is talking 3 Process with LLM Generate responses using a large language model 4 Convert to Speech Transform text responses into natural speech 5 Play Audio Stream the audio response back to the user 1 Send Audio Transmit and capture streamed audio from the user 2 Transcribe Speech Convert speech to text as the user is talking 3 Process with LLM Generate responses using a large language model 4 Convert to Speech Transform text responses into natural speech 5 Play Audio Stream the audio response back to the user 1 Send Audio and Video Transmit and capture audio, video, and image inputs simultaneously 2 Process Streams Handle multiple input streams in parallel 3 Model Processing Send combined inputs to multimodal models (like GPT-4V) 4 Generate Outputs Create various outputs (text, images, audio, etc.) 5 Coordinate Presentation Synchronize and present multiple output types In both cases, Pipecat: Processes responses as they stream in Handles multiple input/output modalities concurrently Manages resource allocation and synchronization Coordinates parallel processing tasks This architecture creates fluid, natural interactions without noticeable delays, whether you’re building a simple voice assistant or a complex multimodal application. Pipecat’s pipeline architecture is particularly valuable for managing the complexity of real-time, multimodal interactions, ensuring smooth data flow and proper synchronization regardless of the input/output types involved. Pipecat handles all this complexity for you, letting you focus on building your application rather than managing the underlying infrastructure. ​ Next Steps Ready to build your first Pipecat application? Installation & Setup Prepare your environment and install required dependencies Quickstart Build and run your first Pipecat application Core Concepts Learn about pipelines, frames, and real-time processing Use Cases Explore example implementations and patterns ​ Join Our Community Discord Community Connect with other developers, share your projects, and get support from the Pipecat team. Installation & Setup On this page What You Can Build How It Works Real-time Processing Next Steps Join Our Community Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/base-classes_media_db6c6ed3.txt b/base-classes_media_db6c6ed3.txt
new file mode 100644
index 0000000000000000000000000000000000000000..002649bb469432b81b8ee7b796c382c1a492c546
--- /dev/null
+++ b/base-classes_media_db6c6ed3.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/base-classes/media#methods
+Title: Overview - Pipecat
+==================================================
+
+Overview - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Get Started Overview Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Get Started Overview Installation & Setup Quickstart Core Concepts Next Steps & Examples Pipecat is an open source Python framework that handles the complex orchestration of AI services, network transport, audio processing, and multimodal interactions. “Multimodal” means you can use any combination of audio, video, images, and/or text in your interactions. And “real-time” means that things are happening quickly enough that it feels conversational—a “back-and-forth” with a bot, not submitting a query and waiting for results. ​ What You Can Build Voice Assistants Natural, real-time conversations with AI using speech recognition and synthesis Interactive Agents Personal coaches and meeting assistants that can understand context and provide guidance Multimodal Apps Applications that combine voice, video, images, and text for rich interactions Creative Tools Storytelling experiences and social companions that engage users Business Solutions Customer intake flows and support bots for automated business processes Complex Flows Structured conversations using Pipecat Flows for managing complex interactions ​ How It Works The flow of interactions in a Pipecat application is typically straightforward: The bot says something The user says something The bot says something The user says something This continues until the conversation naturally ends. While this flow seems simple, making it feel natural requires sophisticated real-time processing. ​ Real-time Processing Pipecat’s pipeline architecture handles both simple voice interactions and complex multimodal processing. Let’s look at how data flows through the system: Voice app Multimodal app 1 Send Audio Transmit and capture streamed audio from the user 2 Transcribe Speech Convert speech to text as the user is talking 3 Process with LLM Generate responses using a large language model 4 Convert to Speech Transform text responses into natural speech 5 Play Audio Stream the audio response back to the user 1 Send Audio Transmit and capture streamed audio from the user 2 Transcribe Speech Convert speech to text as the user is talking 3 Process with LLM Generate responses using a large language model 4 Convert to Speech Transform text responses into natural speech 5 Play Audio Stream the audio response back to the user 1 Send Audio and Video Transmit and capture audio, video, and image inputs simultaneously 2 Process Streams Handle multiple input streams in parallel 3 Model Processing Send combined inputs to multimodal models (like GPT-4V) 4 Generate Outputs Create various outputs (text, images, audio, etc.) 5 Coordinate Presentation Synchronize and present multiple output types In both cases, Pipecat: Processes responses as they stream in Handles multiple input/output modalities concurrently Manages resource allocation and synchronization Coordinates parallel processing tasks This architecture creates fluid, natural interactions without noticeable delays, whether you’re building a simple voice assistant or a complex multimodal application. Pipecat’s pipeline architecture is particularly valuable for managing the complexity of real-time, multimodal interactions, ensuring smooth data flow and proper synchronization regardless of the input/output types involved. Pipecat handles all this complexity for you, letting you focus on building your application rather than managing the underlying infrastructure. ​ Next Steps Ready to build your first Pipecat application? Installation & Setup Prepare your environment and install required dependencies Quickstart Build and run your first Pipecat application Core Concepts Learn about pipelines, frames, and real-time processing Use Cases Explore example implementations and patterns ​ Join Our Community Discord Community Connect with other developers, share your projects, and get support from the Pipecat team. Installation & Setup On this page What You Can Build How It Works Real-time Processing Next Steps Join Our Community Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/base-classes_speech_afdfa0be.txt b/base-classes_speech_afdfa0be.txt
new file mode 100644
index 0000000000000000000000000000000000000000..6ebd885e7fa87f9d1c0680bd81927c24cc6006b3
--- /dev/null
+++ b/base-classes_speech_afdfa0be.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/base-classes/speech#real-time-processing
+Title: Overview - Pipecat
+==================================================
+
+Overview - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Get Started Overview Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Get Started Overview Installation & Setup Quickstart Core Concepts Next Steps & Examples Pipecat is an open source Python framework that handles the complex orchestration of AI services, network transport, audio processing, and multimodal interactions. “Multimodal” means you can use any combination of audio, video, images, and/or text in your interactions. And “real-time” means that things are happening quickly enough that it feels conversational—a “back-and-forth” with a bot, not submitting a query and waiting for results. ​ What You Can Build Voice Assistants Natural, real-time conversations with AI using speech recognition and synthesis Interactive Agents Personal coaches and meeting assistants that can understand context and provide guidance Multimodal Apps Applications that combine voice, video, images, and text for rich interactions Creative Tools Storytelling experiences and social companions that engage users Business Solutions Customer intake flows and support bots for automated business processes Complex Flows Structured conversations using Pipecat Flows for managing complex interactions ​ How It Works The flow of interactions in a Pipecat application is typically straightforward: The bot says something The user says something The bot says something The user says something This continues until the conversation naturally ends. While this flow seems simple, making it feel natural requires sophisticated real-time processing. ​ Real-time Processing Pipecat’s pipeline architecture handles both simple voice interactions and complex multimodal processing. Let’s look at how data flows through the system: Voice app Multimodal app 1 Send Audio Transmit and capture streamed audio from the user 2 Transcribe Speech Convert speech to text as the user is talking 3 Process with LLM Generate responses using a large language model 4 Convert to Speech Transform text responses into natural speech 5 Play Audio Stream the audio response back to the user 1 Send Audio Transmit and capture streamed audio from the user 2 Transcribe Speech Convert speech to text as the user is talking 3 Process with LLM Generate responses using a large language model 4 Convert to Speech Transform text responses into natural speech 5 Play Audio Stream the audio response back to the user 1 Send Audio and Video Transmit and capture audio, video, and image inputs simultaneously 2 Process Streams Handle multiple input streams in parallel 3 Model Processing Send combined inputs to multimodal models (like GPT-4V) 4 Generate Outputs Create various outputs (text, images, audio, etc.) 5 Coordinate Presentation Synchronize and present multiple output types In both cases, Pipecat: Processes responses as they stream in Handles multiple input/output modalities concurrently Manages resource allocation and synchronization Coordinates parallel processing tasks This architecture creates fluid, natural interactions without noticeable delays, whether you’re building a simple voice assistant or a complex multimodal application. Pipecat’s pipeline architecture is particularly valuable for managing the complexity of real-time, multimodal interactions, ensuring smooth data flow and proper synchronization regardless of the input/output types involved. Pipecat handles all this complexity for you, letting you focus on building your application rather than managing the underlying infrastructure. ​ Next Steps Ready to build your first Pipecat application? Installation & Setup Prepare your environment and install required dependencies Quickstart Build and run your first Pipecat application Core Concepts Learn about pipelines, frames, and real-time processing Use Cases Explore example implementations and patterns ​ Join Our Community Discord Community Connect with other developers, share your projects, and get support from the Pipecat team. Installation & Setup On this page What You Can Build How It Works Real-time Processing Next Steps Join Our Community Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/base-classes_text_1252cc3e.txt b/base-classes_text_1252cc3e.txt
new file mode 100644
index 0000000000000000000000000000000000000000..8cfa341a133dba67a11623a071a4f97e43a60227
--- /dev/null
+++ b/base-classes_text_1252cc3e.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/base-classes/text#real-time-processing
+Title: Overview - Pipecat
+==================================================
+
+Overview - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Get Started Overview Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Get Started Overview Installation & Setup Quickstart Core Concepts Next Steps & Examples Pipecat is an open source Python framework that handles the complex orchestration of AI services, network transport, audio processing, and multimodal interactions. “Multimodal” means you can use any combination of audio, video, images, and/or text in your interactions. And “real-time” means that things are happening quickly enough that it feels conversational—a “back-and-forth” with a bot, not submitting a query and waiting for results. ​ What You Can Build Voice Assistants Natural, real-time conversations with AI using speech recognition and synthesis Interactive Agents Personal coaches and meeting assistants that can understand context and provide guidance Multimodal Apps Applications that combine voice, video, images, and text for rich interactions Creative Tools Storytelling experiences and social companions that engage users Business Solutions Customer intake flows and support bots for automated business processes Complex Flows Structured conversations using Pipecat Flows for managing complex interactions ​ How It Works The flow of interactions in a Pipecat application is typically straightforward: The bot says something The user says something The bot says something The user says something This continues until the conversation naturally ends. While this flow seems simple, making it feel natural requires sophisticated real-time processing. ​ Real-time Processing Pipecat’s pipeline architecture handles both simple voice interactions and complex multimodal processing. Let’s look at how data flows through the system: Voice app Multimodal app 1 Send Audio Transmit and capture streamed audio from the user 2 Transcribe Speech Convert speech to text as the user is talking 3 Process with LLM Generate responses using a large language model 4 Convert to Speech Transform text responses into natural speech 5 Play Audio Stream the audio response back to the user 1 Send Audio Transmit and capture streamed audio from the user 2 Transcribe Speech Convert speech to text as the user is talking 3 Process with LLM Generate responses using a large language model 4 Convert to Speech Transform text responses into natural speech 5 Play Audio Stream the audio response back to the user 1 Send Audio and Video Transmit and capture audio, video, and image inputs simultaneously 2 Process Streams Handle multiple input streams in parallel 3 Model Processing Send combined inputs to multimodal models (like GPT-4V) 4 Generate Outputs Create various outputs (text, images, audio, etc.) 5 Coordinate Presentation Synchronize and present multiple output types In both cases, Pipecat: Processes responses as they stream in Handles multiple input/output modalities concurrently Manages resource allocation and synchronization Coordinates parallel processing tasks This architecture creates fluid, natural interactions without noticeable delays, whether you’re building a simple voice assistant or a complex multimodal application. Pipecat’s pipeline architecture is particularly valuable for managing the complexity of real-time, multimodal interactions, ensuring smooth data flow and proper synchronization regardless of the input/output types involved. Pipecat handles all this complexity for you, letting you focus on building your application rather than managing the underlying infrastructure. ​ Next Steps Ready to build your first Pipecat application? Installation & Setup Prepare your environment and install required dependencies Quickstart Build and run your first Pipecat application Core Concepts Learn about pipelines, frames, and real-time processing Use Cases Explore example implementations and patterns ​ Join Our Community Discord Community Connect with other developers, share your projects, and get support from the Pipecat team. Installation & Setup On this page What You Can Build How It Works Real-time Processing Next Steps Join Our Community Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/base-classes_text_96b7f57c.txt b/base-classes_text_96b7f57c.txt
new file mode 100644
index 0000000000000000000000000000000000000000..a5ab6a250d2e5c5f8fd0ce71fe5bf254c275fc2a
--- /dev/null
+++ b/base-classes_text_96b7f57c.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/base-classes/text#join-our-community
+Title: Overview - Pipecat
+==================================================
+
+Overview - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Get Started Overview Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Get Started Overview Installation & Setup Quickstart Core Concepts Next Steps & Examples Pipecat is an open source Python framework that handles the complex orchestration of AI services, network transport, audio processing, and multimodal interactions. “Multimodal” means you can use any combination of audio, video, images, and/or text in your interactions. And “real-time” means that things are happening quickly enough that it feels conversational—a “back-and-forth” with a bot, not submitting a query and waiting for results. ​ What You Can Build Voice Assistants Natural, real-time conversations with AI using speech recognition and synthesis Interactive Agents Personal coaches and meeting assistants that can understand context and provide guidance Multimodal Apps Applications that combine voice, video, images, and text for rich interactions Creative Tools Storytelling experiences and social companions that engage users Business Solutions Customer intake flows and support bots for automated business processes Complex Flows Structured conversations using Pipecat Flows for managing complex interactions ​ How It Works The flow of interactions in a Pipecat application is typically straightforward: The bot says something The user says something The bot says something The user says something This continues until the conversation naturally ends. While this flow seems simple, making it feel natural requires sophisticated real-time processing. ​ Real-time Processing Pipecat’s pipeline architecture handles both simple voice interactions and complex multimodal processing. Let’s look at how data flows through the system: Voice app Multimodal app 1 Send Audio Transmit and capture streamed audio from the user 2 Transcribe Speech Convert speech to text as the user is talking 3 Process with LLM Generate responses using a large language model 4 Convert to Speech Transform text responses into natural speech 5 Play Audio Stream the audio response back to the user 1 Send Audio Transmit and capture streamed audio from the user 2 Transcribe Speech Convert speech to text as the user is talking 3 Process with LLM Generate responses using a large language model 4 Convert to Speech Transform text responses into natural speech 5 Play Audio Stream the audio response back to the user 1 Send Audio and Video Transmit and capture audio, video, and image inputs simultaneously 2 Process Streams Handle multiple input streams in parallel 3 Model Processing Send combined inputs to multimodal models (like GPT-4V) 4 Generate Outputs Create various outputs (text, images, audio, etc.) 5 Coordinate Presentation Synchronize and present multiple output types In both cases, Pipecat: Processes responses as they stream in Handles multiple input/output modalities concurrently Manages resource allocation and synchronization Coordinates parallel processing tasks This architecture creates fluid, natural interactions without noticeable delays, whether you’re building a simple voice assistant or a complex multimodal application. Pipecat’s pipeline architecture is particularly valuable for managing the complexity of real-time, multimodal interactions, ensuring smooth data flow and proper synchronization regardless of the input/output types involved. Pipecat handles all this complexity for you, letting you focus on building your application rather than managing the underlying infrastructure. ​ Next Steps Ready to build your first Pipecat application? Installation & Setup Prepare your environment and install required dependencies Quickstart Build and run your first Pipecat application Core Concepts Learn about pipelines, frames, and real-time processing Use Cases Explore example implementations and patterns ​ Join Our Community Discord Community Connect with other developers, share your projects, and get support from the Pipecat team. Installation & Setup On this page What You Can Build How It Works Real-time Processing Next Steps Join Our Community Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/c_introduction_82958e9e.txt b/c_introduction_82958e9e.txt
new file mode 100644
index 0000000000000000000000000000000000000000..2b9fe07ff4ae88500f8553f84346ac69110fea03
--- /dev/null
+++ b/c_introduction_82958e9e.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/client/c++/introduction#libcurl
+Title: SDK Introduction - Pipecat
+==================================================
+
+SDK Introduction - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation C++ SDK SDK Introduction Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Client SDKs The RTVI Standard RTVIClient Migration Guide Javascript SDK SDK Introduction API Reference Transport packages React SDK SDK Introduction API Reference React Native SDK SDK Introduction API Reference iOS SDK SDK Introduction API Reference Transport packages Android SDK SDK Introduction API Reference Transport packages C++ SDK SDK Introduction Daily WebRTC Transport The Pipecat C++ SDK provides a native implementation for building voice and multimodal AI applications. It supports: Linux ( x86_64 and aarch64 ) macOS ( aarch64 ) Windows ( x86_64 ) ​ Dependencies ​ libcurl The SDK uses libcurl for HTTP requests. Linux macOS Windows Copy Ask AI sudo apt-get install libcurl4-openssl-dev Copy Ask AI sudo apt-get install libcurl4-openssl-dev On macOS libcurl is already included so there is nothing to install. On Windows we use vcpkg to install dependencies. You need to set it up following one of the tutorials . The libcurl dependency will be automatically downloaded when building. ​ Installation Build the SDK using CMake: Linux/macOS Windows Copy Ask AI cmake . -G Ninja -Bbuild -DCMAKE_BUILD_TYPE=Release ninja -C build Copy Ask AI cmake . -G Ninja -Bbuild -DCMAKE_BUILD_TYPE=Release ninja -C build Copy Ask AI # Initialize Visual Studio environment "C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Auxiliary\Build\vcvarsall.bat" amd64 # Configure and build cmake . -Bbuild --preset vcpkg cmake --buildbuild --config Release ​ Cross-compilation For Linux aarch64: Copy Ask AI cmake . -G Ninja -Bbuild -DCMAKE_TOOLCHAIN_FILE=aarch64-linux-toolchain.cmake -DCMAKE_BUILD_TYPE=Release ninja -C build ​ Documentation API Reference Complete SDK API documentation Daily Transport WebRTC implementation using Daily Small WebRTC Transport Daily WebRTC Transport On this page Dependencies libcurl Installation Cross-compilation Documentation Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/client_introduction_88cc8f09.txt b/client_introduction_88cc8f09.txt
new file mode 100644
index 0000000000000000000000000000000000000000..a8ce72adfffcc14224dc110a346eb5c0f45f4e85
--- /dev/null
+++ b/client_introduction_88cc8f09.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/client/introduction#rtvimessage
+Title: Client SDKs - Pipecat
+==================================================
+
+Client SDKs - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Client SDKs Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Client SDKs The RTVI Standard RTVIClient Migration Guide Javascript SDK SDK Introduction API Reference Transport packages React SDK SDK Introduction API Reference React Native SDK SDK Introduction API Reference iOS SDK SDK Introduction API Reference Transport packages Android SDK SDK Introduction API Reference Transport packages C++ SDK SDK Introduction Daily WebRTC Transport The Client SDKs are currently in transition to a new, simpler API design. The js and react libraries have already been deployed with these changes. Their corresponding documentation along with this top-level documentation has been updated to reflect the latest changes. For transitioning to the new API, please refer to the migration guide . Note that React Native, iOS, and Android SDKs are still in the process of being updated and their documentation will be updated once the new versions are released. If you have any questions or need assistance, please reach out to us on Discord . Pipecat provides client SDKs for multiple platforms, all implementing the RTVI (Real-Time Voice and Video Inference) standard. These SDKs make it easy to build real-time AI applications that can handle voice, video, and text interactions. Javascript Pipecat JS SDK React Pipecat React SDK React Native Pipecat React Native SDK Swift Pipecat iOS SDK Kotlin Pipecat Android SDK C++ Pipecat C++ SDK ​ Core Functionality All Pipecat client SDKs provide: Media Management Handle device inputs and media streams for audio and video Bot Integration Configure and communicate with your Pipecat bot Session Management Manage connection state and error handling ​ Core Types ​ PipecatClient The main class for interacting with Pipecat bots. It is the primary type you will interact with. ​ Transport The PipecatClient wraps a Transport, which defines and provides the underlying connection mechanism (e.g., WebSocket, WebRTC). Your Pipecat pipeline will contain a corresponding transport. ​ RTVIMessage Represents a message sent to or received from a Pipecat bot. ​ Simple Usage Examples Connecting to a Bot Custom Messaging Establish ongoing connections via WebSocket or WebRTC for: Live voice conversations Real-time video processing Continuous interactions javascript react Copy Ask AI // Example: Establishing a real-time connection import { RTVIEvent , RTVIMessage , PipecatClient } from "@pipecat-ai/client-js" ; import { DailyTransport } from "@pipecat-ai/daily-transport" ; const pcClient = new PipecatClient ({ transport: new DailyTransport (), enableMic: true , enableCam: false , enableScreenShare: false , callbacks: { onBotConnected : () => { console . log ( "[CALLBACK] Bot connected" ); }, onBotDisconnected : () => { console . log ( "[CALLBACK] Bot disconnected" ); }, onBotReady : () => { console . log ( "[CALLBACK] Bot ready to chat!" ); }, }, }); try { // Below, we use a REST endpoint to fetch connection credentials for our // Daily Transport. Alternatively, you could provide those credentials // directly to `connect()`. await pcClient . connect ({ endpoint: "https://your-connect-end-point-here/connect" , }); } catch ( e ) { console . error ( e . message ); } // Events (alternative approach to constructor-provided callbacks) pcClient . on ( RTVIEvent . Connected , () => { console . log ( "[EVENT] User connected" ); }); pcClient . on ( RTVIEvent . Disconnected , () => { console . log ( "[EVENT] User disconnected" ); }); Establish ongoing connections via WebSocket or WebRTC for: Live voice conversations Real-time video processing Continuous interactions javascript react Copy Ask AI // Example: Establishing a real-time connection import { RTVIEvent , RTVIMessage , PipecatClient } from "@pipecat-ai/client-js" ; import { DailyTransport } from "@pipecat-ai/daily-transport" ; const pcClient = new PipecatClient ({ transport: new DailyTransport (), enableMic: true , enableCam: false , enableScreenShare: false , callbacks: { onBotConnected : () => { console . log ( "[CALLBACK] Bot connected" ); }, onBotDisconnected : () => { console . log ( "[CALLBACK] Bot disconnected" ); }, onBotReady : () => { console . log ( "[CALLBACK] Bot ready to chat!" ); }, }, }); try { // Below, we use a REST endpoint to fetch connection credentials for our // Daily Transport. Alternatively, you could provide those credentials // directly to `connect()`. await pcClient . connect ({ endpoint: "https://your-connect-end-point-here/connect" , }); } catch ( e ) { console . error ( e . message ); } // Events (alternative approach to constructor-provided callbacks) pcClient . on ( RTVIEvent . Connected , () => { console . log ( "[EVENT] User connected" ); }); pcClient . on ( RTVIEvent . Disconnected , () => { console . log ( "[EVENT] User disconnected" ); }); Send custom messages and handle responses from your bot. This is useful for: Running server-side functionality Triggering specific bot actions Querying the server Responding to server requests javascript react Copy Ask AI import { PipecatClient } from "@pipecat-ai/client-js" ; const pcClient = new PipecatClient ({ transport: new DailyTransport (), callbacks: { onBotConnected : () => { pcClient . sendClientRequest ( 'get-language' ) . then (( response ) => { console . log ( "[CALLBACK] Bot using language:" , response ); if ( response !== preferredLanguage ) { pcClient . sendClientMessage ( 'set-language' , { language: preferredLanguage }); } }) . catch (( error ) => { console . error ( "[CALLBACK] Error getting language:" , error ); }); }, onServerMessage : ( message ) => { console . log ( "[CALLBACK] Received message from server:" , message ); }, }, }); await pcClient . connect ({ url: "https://your-daily-room-url" , token: "your-daily-token" }); ​ About RTVI Pipecat’s client SDKs implement the RTVI (Real-Time Voice and Video Inference) standard, an open specification for real-time AI inference. This means: Your code can work with any RTVI-compatible inference service You get battle-tested tooling for real-time multimedia handling You can easily set up development and testing environments ​ Next Steps Get started by trying out examples: Simple Chatbot Example Complete client-server example with both bot backend (Python) and frontend implementation (JS, React, React Native, iOS, and Android). More Examples Explore our full collection of example applications and implementations across different platforms and use cases. The RTVI Standard On this page Core Functionality Core Types PipecatClient Transport RTVIMessage Simple Usage Examples About RTVI Next Steps Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/client_rtvi-standard_37662bce.txt b/client_rtvi-standard_37662bce.txt
new file mode 100644
index 0000000000000000000000000000000000000000..e02121bf3a46550dc80f9bd838d676aabb70d85d
--- /dev/null
+++ b/client_rtvi-standard_37662bce.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/client/rtvi-standard#error-response-%F0%9F%A4%96
+Title: The RTVI Standard - Pipecat
+==================================================
+
+The RTVI Standard - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation The RTVI Standard Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Client SDKs The RTVI Standard RTVIClient Migration Guide Javascript SDK SDK Introduction API Reference Transport packages React SDK SDK Introduction API Reference React Native SDK SDK Introduction API Reference iOS SDK SDK Introduction API Reference Transport packages Android SDK SDK Introduction API Reference Transport packages C++ SDK SDK Introduction Daily WebRTC Transport The RTVI (Real-Time Voice and Video Inference) standard defines a set of message types and structures sent between clients and servers. It is designed to facilitate real-time interactions between clients and AI applications that require voice, video, and text communication. It provides a consistent framework for building applications that can communicate with AI models and the backends running those models in real-time. This page documents version 1.0 of the RTVI standard, released in June 2025. ​ Key Features Connection Management RTVI provides a flexible connection model that allows clients to connect to AI services and coordinate state. Transcriptions The standard includes built-in support for real-time transcription of audio streams. Client-Server Messaging The standard defines a messaging protocol for sending and receiving messages between clients and servers, allowing for efficient communication of requests and responses. Advanced LLM Interactions The standard supports advanced interactions with large language models (LLMs), including context management, function call handline, and search results. Service-Specific Insights RTVI supports events to provide insight into the input/output and state for typical services that exist in speech-to-speech workflows. Metrics and Monitoring RTVI provides mechanisms for collecting metrics and monitoring the performance of server-side services. ​ Terms Client : The front-end application or user interface that interacts with the RTVI server. Server : The backend-end service that runs the AI framework and processes requests from the client. User : The end user interacting with the client application. Bot : The AI interacting with the user, technically an amalgamation of a large language model (LLM) and a text-to-speech (TTS) service. ​ RTVI Message Format The messages defined as part of the RTVI protocol adhere to the following format: Copy Ask AI { "id" : string , "label" : "rtvi-ai" , "type" : string , "data" : unknown } ​ id string A unique identifier for the message, used to correlate requests and responses. ​ label string default: "rtvi-ai" required A label that identifies this message as an RTVI message. This field is required and should always be set to 'rtvi-ai' . ​ type string required The type of message being sent. This field is required and should be set to one of the predefined RTVI message types listed below. ​ data unknown The payload of the message, which can be any data structure relevant to the message type. ​ RTVI Message Types Following the above format, this section describes the various message types defined by the RTVI standard. Each message type has a specific purpose and structure, allowing for clear communication between clients and servers. Each message type below includes either a 🤖 or 🏄 emoji to denote whether the message is sent from the bot (🤖) or client (🏄). ​ Connection Management ​ client-ready 🏄 Indicates that the client is ready to receive messages and interact with the server. Typically sent after the transport media channels have connected. type : 'client-ready' data : version : string The version of the RTVI standard being used. This is useful for ensuring compatibility between client and server implementations. about : AboutClient Object An object containing information about the client, such as its rtvi-version, client library, and any other relevant metadata. The AboutClient object follows this structure: Show AboutClient ​ library string required ​ library_version string ​ platform string ​ platform_version string ​ platform_details any Any platform-specific details that may be relevant to the server. This could include information about the browser, operating system, or any other environment-specific data needed by the server. This field is optional and open-ended, so please be mindful of the data you include here and any security concerns that may arise from exposing sensitive or personal-identifiable information. ​ bot-ready 🤖 Indicates that the bot is ready to receive messages and interact with the client. Typically send after the transport media channels have connected. type : 'bot-ready' data : version : string The version of the RTVI standard being used. This is useful for ensuring compatibility between client and server implementations. about : any (Optional) An object containing information about the server or bot. It’s structure and value are both undefined by default. This provides flexibility to include any relevant metadata your client may need to know about the server at connection time, without any built-in security concerns. Please be mindful of the data you include here and any security concerns that may arise from exposing sensitive information. ​ disconnect-bot 🏄 Indicates that the client wishes to disconnect from the bot. Typically used when the client is shutting down or no longer needs to interact with the bot. Note: Disconnets should happen automatically when either the client or bot disconnects from the transport, so this message is intended for the case where a client may want to remain connected to the transport but no longer wishes to interact with the bot. type : 'disconnect-bot' data : undefined ​ error 🤖 Indicates an error occurred during bot initialization or runtime. type : 'error' data : message : string Description of the error. fatal : boolean Indicates if the error is fatal to the session. ​ Transcription ​ user-started-speaking 🤖 Emitted when the user begins speaking type : 'user-started-speaking' data : None ​ user-stopped-speaking 🤖 Emitted when the user stops speaking type : 'user-stopped-speaking' data : None ​ bot-started-speaking 🤖 Emitted when the bot begins speaking type : 'bot-started-speaking' data : None ​ bot-stopped-speaking 🤖 Emitted when the bot stops speaking type : 'bot-stopped-speaking' data : None ​ user-transcription 🤖 Real-time transcription of user speech, including both partial and final results. type : 'user-transcription' data : text : string The transcribed text of the user. final : boolean Indicates if this is a final transcription or a partial result. timestamp : string The timestamp when the transcription was generated. user_id : string Identifier for the user who spoke. ​ bot-transcription 🤖 Transcription of the bot’s speech. Note: This protocol currently does not match the user transcription format to support real-time timestamping for bot transcriptions. Rather, the event is typically sent for each sentence of the bot’s response. This difference is currently due to limitations in TTS services which mostly do not support (or support well), accurate timing information. If/when this changes, this protocol may be updated to include the necessary timing information. For now, if you want to attempt real-time transcription to match your bot’s speaking, you can try using the bot-tts-text message type. type : 'bot-transcription' data : text : string The transcribed text from the bot, typically aggregated at a per-sentence level. ​ Client-Server Messaging ​ server-message 🤖 An arbitrary message sent from the server to the client. This can be used for custom interactions or commands. This message may be coupled with the client-message message type to handle responses from the client. type : 'server-message' data : any The data can be any JSON-serializable object, formatted according to your own specifications. ​ client-message 🏄 An arbitrary message sent from the client to the server. This can be used for custom interactions or commands. This message may be coupled with the server-response message type to handle responses from the server. type : 'client-message' data : t : string d : unknown (optional) The data payload should contain a t field indicating the type of message and an optional d field containing any custom, corresponding data needed for the message. ​ server-response 🤖 An message sent from the server to the client in response to a client-message . IMPORTANT : The id should match the id of the original client-message to correlate the response with the request. type : 'client-message' data : t : string d : unknown (optional) The data payload should contain a t field indicating the type of message and an optional d field containing any custom, corresponding data needed for the message. ​ error-response 🤖 Error response to a specific client message. IMPORTANT : The id should match the id of the original client-message to correlate the response with the request. type : 'error-response' data : error : string ​ Advanced LLM Interactions ​ append-to-context 🏄 A message sent from the client to the server to append data to the context of the current llm conversation. This is useful for providing text-based content for the user or augmenting the context for the assistant. type : 'append-to-context' data : role : "user" | "assistant" The role the context should be appended to. Currently only supports "user" and "assistant" . content : unknown The content to append to the context. This can be any data structure the llm understand. run_immediately : boolean (optional) Indicates whether the context should be run immediately after appending. Defaults to false . If set to false , the context will be appended but not executed until the next llm run. ​ llm-function-call 🤖 A function call request from the LLM, sent from the bot to the client. Note that for most cases, an LLM function call will be handled completely server-side. However, in the event that the call requires input from the client or the client needs to be aware of the function call, this message/response schema is required. type : 'llm-function-call' data : function_name : string Name of the function to be called. tool_call_id : string Unique identifier for this function call. args : Record<string, unknown> Arguments to be passed to the function. ​ llm-function-call-result 🏄 The result of the function call requested by the LLM, returned from the client. type : 'llm-function-call-result' data : function_name : string Name of the called function. tool_call_id : string Identifier matching the original function call. args : Record<string, unknown> Arguments that were passed to the function. result : Record<string, unknown> | string The result returned by the function. ​ bot-llm-search-response 🤖 Search results from the LLM’s knowledge base. Currently, Google Gemini is the only LLM that supports built-in search. However, we expect other LLMs to follow suite, which is why this message type is defined as part of the RTVI standard. As more LLMs add support for this feature, the format of this message type may evolve to accommodate discrepancies. type : 'bot-llm-search-response' data : search_result : string (optional) Raw search result text. rendered_content : string (optional) Formatted version of the search results. origins : Array<Origin Object> Source information and confidence scores for search results. The Origin Object follows this structure: Copy Ask AI { "site_uri" : string (optional) , "site_title" : string (optional) , "results" : Array< { "text" : string , "confidence" : number [] } > } Example: Copy Ask AI "id" : undefined "label" : "rtvi-ai" "type" : "bot-llm-search-response" "data" : { "origins" : [ { "results" : [ { "confidence" : [ 0.9881149530410768 ], "text" : "* Juneteenth: A Freedom Celebration is scheduled for June 18th from 12:00 pm to 2:00 pm." }, { "confidence" : [ 0.9692034721374512 ], "ext" : "* A Juneteenth celebration at Fort Negley Park will take place on June 19th from 5:00 pm to 9:30 pm." } ], "site_title" : "vanderbilt.edu" , "site_uri" : "https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQHwif83VK9KAzrbMSGSBsKwL8vWfSfC9pgEWYKmStHyqiRoV1oe8j1S0nbwRg_iWgqAr9wUkiegu3ATC8Ll-cuE-vpzwElRHiJ2KgRYcqnOQMoOeokVpWqi" }, { "results" : [ { "confidence" : [ 0.6554043292999268 ], "text" : "In addition to these events, Vanderbilt University is a large research institution with ongoing activities across many fields." } ], "site_title" : "wikipedia.org" , "site_uri" : "https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQESbF-ijx78QbaglrhflHCUWdPTD4M6tYOQigW5hgsHNctRlAHu9ktfPmJx7DfoP5QicE0y-OQY1cRl9w4Id0btiFgLYSKIm2-SPtOHXeNrAlgA7mBnclaGrD7rgnLIbrjl8DgUEJrrvT0CKzuo" }], "rendered_content" : "<style> \n .container ... </div> \n </div> \n " , "search_result" : "Several events are happening at Vanderbilt University: \n\n * Juneteenth: A Freedom Celebration is scheduled for June 18th from 12:00 pm to 2:00 pm. \n * A Juneteenth celebration at Fort Negley Park will take place on June 19th from 5:00 pm to 9:30 pm. \n\n In addition to these events, Vanderbilt University is a large research institution with ongoing activities across many fields. For the most recent news, you should check Vanderbilt's official news website. \n " } ​ Service-Specific Insights ​ bot-llm-started 🤖 Indicates LLM processing has begun type : bot-llm-started data : None ​ bot-llm-stopped 🤖 Indicates LLM processing has completed type : bot-llm-stopped data : None ​ user-llm-text 🤖 Aggregated user input text that is sent to the LLM. type : 'user-llm-text' data : text : string The user’s input text to be processed by the LLM. ​ bot-llm-text 🤖 Individual tokens streamed from the LLM as they are generated. type : 'bot-llm-text' data : text : string The token text from the LLM. ​ bot-tts-started 🤖 Indicates text-to-speech (TTS) processing has begun. type : 'bot-tts-started' data : None ​ bot-tts-stopped 🤖 Indicates text-to-speech (TTS) processing has completed. type : 'bot-tts-stopped' data : None ​ bot-tts-text 🤖 The per-token text output of the text-to-speech (TTS) service (what the TTS actually says). type : 'bot-tts-text' data : text : string The text representation of the generated bot speech. ​ Metrics and Monitoring ​ metrics 🤖 Performance metrics for various processing stages and services. Each message will contain entries for one or more of the metrics types: processing , ttfb , characters . type : 'metrics' data : processing : [See Below] (optional) Processing time metrics. ttfb : [See Below] (optional) Time to first byte metrics. characters : [See Below] (optional) Character processing metrics. For each metric type, the data structure is an array of objects with the following structure: processor : string The name of the processor or service that generated the metric. value : number The value of the metric, typically in milliseconds or character count. model : string (optional) The model of the service that generated the metric, if applicable. Example: Copy Ask AI { "type" : "metrics" , "data" : { "processing" : [ { "model" : "eleven_flash_v2_5" , "processor" : "ElevenLabsTTSService#0" , "value" : 0.0005140304565429688 } ], "ttfb" : [ { "model" : "eleven_flash_v2_5" , "processor" : "ElevenLabsTTSService#0" , "value" : 0.1573178768157959 } ], "characters" : [ { "model" : "eleven_flash_v2_5" , "processor" : "ElevenLabsTTSService#0" , "value" : 43 } ] } } Client SDKs RTVIClient Migration Guide On this page Key Features Terms RTVI Message Format RTVI Message Types Connection Management client-ready 🏄 bot-ready 🤖 disconnect-bot 🏄 error 🤖 Transcription user-started-speaking 🤖 user-stopped-speaking 🤖 bot-started-speaking 🤖 bot-stopped-speaking 🤖 user-transcription 🤖 bot-transcription 🤖 Client-Server Messaging server-message 🤖 client-message 🏄 server-response 🤖 error-response 🤖 Advanced LLM Interactions append-to-context 🏄 llm-function-call 🤖 llm-function-call-result 🏄 bot-llm-search-response 🤖 Service-Specific Insights bot-llm-started 🤖 bot-llm-stopped 🤖 user-llm-text 🤖 bot-llm-text 🤖 bot-tts-started 🤖 bot-tts-stopped 🤖 bot-tts-text 🤖 Metrics and Monitoring metrics 🤖 Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/client_rtvi-standard_3cedd675.txt b/client_rtvi-standard_3cedd675.txt
new file mode 100644
index 0000000000000000000000000000000000000000..9cb5b7d7892cf045f5be258520e453ea6c8610b1
--- /dev/null
+++ b/client_rtvi-standard_3cedd675.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/client/rtvi-standard#user-llm-text-%F0%9F%A4%96
+Title: The RTVI Standard - Pipecat
+==================================================
+
+The RTVI Standard - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation The RTVI Standard Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Client SDKs The RTVI Standard RTVIClient Migration Guide Javascript SDK SDK Introduction API Reference Transport packages React SDK SDK Introduction API Reference React Native SDK SDK Introduction API Reference iOS SDK SDK Introduction API Reference Transport packages Android SDK SDK Introduction API Reference Transport packages C++ SDK SDK Introduction Daily WebRTC Transport The RTVI (Real-Time Voice and Video Inference) standard defines a set of message types and structures sent between clients and servers. It is designed to facilitate real-time interactions between clients and AI applications that require voice, video, and text communication. It provides a consistent framework for building applications that can communicate with AI models and the backends running those models in real-time. This page documents version 1.0 of the RTVI standard, released in June 2025. ​ Key Features Connection Management RTVI provides a flexible connection model that allows clients to connect to AI services and coordinate state. Transcriptions The standard includes built-in support for real-time transcription of audio streams. Client-Server Messaging The standard defines a messaging protocol for sending and receiving messages between clients and servers, allowing for efficient communication of requests and responses. Advanced LLM Interactions The standard supports advanced interactions with large language models (LLMs), including context management, function call handline, and search results. Service-Specific Insights RTVI supports events to provide insight into the input/output and state for typical services that exist in speech-to-speech workflows. Metrics and Monitoring RTVI provides mechanisms for collecting metrics and monitoring the performance of server-side services. ​ Terms Client : The front-end application or user interface that interacts with the RTVI server. Server : The backend-end service that runs the AI framework and processes requests from the client. User : The end user interacting with the client application. Bot : The AI interacting with the user, technically an amalgamation of a large language model (LLM) and a text-to-speech (TTS) service. ​ RTVI Message Format The messages defined as part of the RTVI protocol adhere to the following format: Copy Ask AI { "id" : string , "label" : "rtvi-ai" , "type" : string , "data" : unknown } ​ id string A unique identifier for the message, used to correlate requests and responses. ​ label string default: "rtvi-ai" required A label that identifies this message as an RTVI message. This field is required and should always be set to 'rtvi-ai' . ​ type string required The type of message being sent. This field is required and should be set to one of the predefined RTVI message types listed below. ​ data unknown The payload of the message, which can be any data structure relevant to the message type. ​ RTVI Message Types Following the above format, this section describes the various message types defined by the RTVI standard. Each message type has a specific purpose and structure, allowing for clear communication between clients and servers. Each message type below includes either a 🤖 or 🏄 emoji to denote whether the message is sent from the bot (🤖) or client (🏄). ​ Connection Management ​ client-ready 🏄 Indicates that the client is ready to receive messages and interact with the server. Typically sent after the transport media channels have connected. type : 'client-ready' data : version : string The version of the RTVI standard being used. This is useful for ensuring compatibility between client and server implementations. about : AboutClient Object An object containing information about the client, such as its rtvi-version, client library, and any other relevant metadata. The AboutClient object follows this structure: Show AboutClient ​ library string required ​ library_version string ​ platform string ​ platform_version string ​ platform_details any Any platform-specific details that may be relevant to the server. This could include information about the browser, operating system, or any other environment-specific data needed by the server. This field is optional and open-ended, so please be mindful of the data you include here and any security concerns that may arise from exposing sensitive or personal-identifiable information. ​ bot-ready 🤖 Indicates that the bot is ready to receive messages and interact with the client. Typically send after the transport media channels have connected. type : 'bot-ready' data : version : string The version of the RTVI standard being used. This is useful for ensuring compatibility between client and server implementations. about : any (Optional) An object containing information about the server or bot. It’s structure and value are both undefined by default. This provides flexibility to include any relevant metadata your client may need to know about the server at connection time, without any built-in security concerns. Please be mindful of the data you include here and any security concerns that may arise from exposing sensitive information. ​ disconnect-bot 🏄 Indicates that the client wishes to disconnect from the bot. Typically used when the client is shutting down or no longer needs to interact with the bot. Note: Disconnets should happen automatically when either the client or bot disconnects from the transport, so this message is intended for the case where a client may want to remain connected to the transport but no longer wishes to interact with the bot. type : 'disconnect-bot' data : undefined ​ error 🤖 Indicates an error occurred during bot initialization or runtime. type : 'error' data : message : string Description of the error. fatal : boolean Indicates if the error is fatal to the session. ​ Transcription ​ user-started-speaking 🤖 Emitted when the user begins speaking type : 'user-started-speaking' data : None ​ user-stopped-speaking 🤖 Emitted when the user stops speaking type : 'user-stopped-speaking' data : None ​ bot-started-speaking 🤖 Emitted when the bot begins speaking type : 'bot-started-speaking' data : None ​ bot-stopped-speaking 🤖 Emitted when the bot stops speaking type : 'bot-stopped-speaking' data : None ​ user-transcription 🤖 Real-time transcription of user speech, including both partial and final results. type : 'user-transcription' data : text : string The transcribed text of the user. final : boolean Indicates if this is a final transcription or a partial result. timestamp : string The timestamp when the transcription was generated. user_id : string Identifier for the user who spoke. ​ bot-transcription 🤖 Transcription of the bot’s speech. Note: This protocol currently does not match the user transcription format to support real-time timestamping for bot transcriptions. Rather, the event is typically sent for each sentence of the bot’s response. This difference is currently due to limitations in TTS services which mostly do not support (or support well), accurate timing information. If/when this changes, this protocol may be updated to include the necessary timing information. For now, if you want to attempt real-time transcription to match your bot’s speaking, you can try using the bot-tts-text message type. type : 'bot-transcription' data : text : string The transcribed text from the bot, typically aggregated at a per-sentence level. ​ Client-Server Messaging ​ server-message 🤖 An arbitrary message sent from the server to the client. This can be used for custom interactions or commands. This message may be coupled with the client-message message type to handle responses from the client. type : 'server-message' data : any The data can be any JSON-serializable object, formatted according to your own specifications. ​ client-message 🏄 An arbitrary message sent from the client to the server. This can be used for custom interactions or commands. This message may be coupled with the server-response message type to handle responses from the server. type : 'client-message' data : t : string d : unknown (optional) The data payload should contain a t field indicating the type of message and an optional d field containing any custom, corresponding data needed for the message. ​ server-response 🤖 An message sent from the server to the client in response to a client-message . IMPORTANT : The id should match the id of the original client-message to correlate the response with the request. type : 'client-message' data : t : string d : unknown (optional) The data payload should contain a t field indicating the type of message and an optional d field containing any custom, corresponding data needed for the message. ​ error-response 🤖 Error response to a specific client message. IMPORTANT : The id should match the id of the original client-message to correlate the response with the request. type : 'error-response' data : error : string ​ Advanced LLM Interactions ​ append-to-context 🏄 A message sent from the client to the server to append data to the context of the current llm conversation. This is useful for providing text-based content for the user or augmenting the context for the assistant. type : 'append-to-context' data : role : "user" | "assistant" The role the context should be appended to. Currently only supports "user" and "assistant" . content : unknown The content to append to the context. This can be any data structure the llm understand. run_immediately : boolean (optional) Indicates whether the context should be run immediately after appending. Defaults to false . If set to false , the context will be appended but not executed until the next llm run. ​ llm-function-call 🤖 A function call request from the LLM, sent from the bot to the client. Note that for most cases, an LLM function call will be handled completely server-side. However, in the event that the call requires input from the client or the client needs to be aware of the function call, this message/response schema is required. type : 'llm-function-call' data : function_name : string Name of the function to be called. tool_call_id : string Unique identifier for this function call. args : Record<string, unknown> Arguments to be passed to the function. ​ llm-function-call-result 🏄 The result of the function call requested by the LLM, returned from the client. type : 'llm-function-call-result' data : function_name : string Name of the called function. tool_call_id : string Identifier matching the original function call. args : Record<string, unknown> Arguments that were passed to the function. result : Record<string, unknown> | string The result returned by the function. ​ bot-llm-search-response 🤖 Search results from the LLM’s knowledge base. Currently, Google Gemini is the only LLM that supports built-in search. However, we expect other LLMs to follow suite, which is why this message type is defined as part of the RTVI standard. As more LLMs add support for this feature, the format of this message type may evolve to accommodate discrepancies. type : 'bot-llm-search-response' data : search_result : string (optional) Raw search result text. rendered_content : string (optional) Formatted version of the search results. origins : Array<Origin Object> Source information and confidence scores for search results. The Origin Object follows this structure: Copy Ask AI { "site_uri" : string (optional) , "site_title" : string (optional) , "results" : Array< { "text" : string , "confidence" : number [] } > } Example: Copy Ask AI "id" : undefined "label" : "rtvi-ai" "type" : "bot-llm-search-response" "data" : { "origins" : [ { "results" : [ { "confidence" : [ 0.9881149530410768 ], "text" : "* Juneteenth: A Freedom Celebration is scheduled for June 18th from 12:00 pm to 2:00 pm." }, { "confidence" : [ 0.9692034721374512 ], "ext" : "* A Juneteenth celebration at Fort Negley Park will take place on June 19th from 5:00 pm to 9:30 pm." } ], "site_title" : "vanderbilt.edu" , "site_uri" : "https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQHwif83VK9KAzrbMSGSBsKwL8vWfSfC9pgEWYKmStHyqiRoV1oe8j1S0nbwRg_iWgqAr9wUkiegu3ATC8Ll-cuE-vpzwElRHiJ2KgRYcqnOQMoOeokVpWqi" }, { "results" : [ { "confidence" : [ 0.6554043292999268 ], "text" : "In addition to these events, Vanderbilt University is a large research institution with ongoing activities across many fields." } ], "site_title" : "wikipedia.org" , "site_uri" : "https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQESbF-ijx78QbaglrhflHCUWdPTD4M6tYOQigW5hgsHNctRlAHu9ktfPmJx7DfoP5QicE0y-OQY1cRl9w4Id0btiFgLYSKIm2-SPtOHXeNrAlgA7mBnclaGrD7rgnLIbrjl8DgUEJrrvT0CKzuo" }], "rendered_content" : "<style> \n .container ... </div> \n </div> \n " , "search_result" : "Several events are happening at Vanderbilt University: \n\n * Juneteenth: A Freedom Celebration is scheduled for June 18th from 12:00 pm to 2:00 pm. \n * A Juneteenth celebration at Fort Negley Park will take place on June 19th from 5:00 pm to 9:30 pm. \n\n In addition to these events, Vanderbilt University is a large research institution with ongoing activities across many fields. For the most recent news, you should check Vanderbilt's official news website. \n " } ​ Service-Specific Insights ​ bot-llm-started 🤖 Indicates LLM processing has begun type : bot-llm-started data : None ​ bot-llm-stopped 🤖 Indicates LLM processing has completed type : bot-llm-stopped data : None ​ user-llm-text 🤖 Aggregated user input text that is sent to the LLM. type : 'user-llm-text' data : text : string The user’s input text to be processed by the LLM. ​ bot-llm-text 🤖 Individual tokens streamed from the LLM as they are generated. type : 'bot-llm-text' data : text : string The token text from the LLM. ​ bot-tts-started 🤖 Indicates text-to-speech (TTS) processing has begun. type : 'bot-tts-started' data : None ​ bot-tts-stopped 🤖 Indicates text-to-speech (TTS) processing has completed. type : 'bot-tts-stopped' data : None ​ bot-tts-text 🤖 The per-token text output of the text-to-speech (TTS) service (what the TTS actually says). type : 'bot-tts-text' data : text : string The text representation of the generated bot speech. ​ Metrics and Monitoring ​ metrics 🤖 Performance metrics for various processing stages and services. Each message will contain entries for one or more of the metrics types: processing , ttfb , characters . type : 'metrics' data : processing : [See Below] (optional) Processing time metrics. ttfb : [See Below] (optional) Time to first byte metrics. characters : [See Below] (optional) Character processing metrics. For each metric type, the data structure is an array of objects with the following structure: processor : string The name of the processor or service that generated the metric. value : number The value of the metric, typically in milliseconds or character count. model : string (optional) The model of the service that generated the metric, if applicable. Example: Copy Ask AI { "type" : "metrics" , "data" : { "processing" : [ { "model" : "eleven_flash_v2_5" , "processor" : "ElevenLabsTTSService#0" , "value" : 0.0005140304565429688 } ], "ttfb" : [ { "model" : "eleven_flash_v2_5" , "processor" : "ElevenLabsTTSService#0" , "value" : 0.1573178768157959 } ], "characters" : [ { "model" : "eleven_flash_v2_5" , "processor" : "ElevenLabsTTSService#0" , "value" : 43 } ] } } Client SDKs RTVIClient Migration Guide On this page Key Features Terms RTVI Message Format RTVI Message Types Connection Management client-ready 🏄 bot-ready 🤖 disconnect-bot 🏄 error 🤖 Transcription user-started-speaking 🤖 user-stopped-speaking 🤖 bot-started-speaking 🤖 bot-stopped-speaking 🤖 user-transcription 🤖 bot-transcription 🤖 Client-Server Messaging server-message 🤖 client-message 🏄 server-response 🤖 error-response 🤖 Advanced LLM Interactions append-to-context 🏄 llm-function-call 🤖 llm-function-call-result 🏄 bot-llm-search-response 🤖 Service-Specific Insights bot-llm-started 🤖 bot-llm-stopped 🤖 user-llm-text 🤖 bot-llm-text 🤖 bot-tts-started 🤖 bot-tts-stopped 🤖 bot-tts-text 🤖 Metrics and Monitoring metrics 🤖 Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/client_rtvi-standard_42f1d183.txt b/client_rtvi-standard_42f1d183.txt
new file mode 100644
index 0000000000000000000000000000000000000000..214cc425f5b84135b8f29f8bfd33d3df26125cd7
--- /dev/null
+++ b/client_rtvi-standard_42f1d183.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/client/rtvi-standard#connection-management
+Title: The RTVI Standard - Pipecat
+==================================================
+
+The RTVI Standard - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation The RTVI Standard Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Client SDKs The RTVI Standard RTVIClient Migration Guide Javascript SDK SDK Introduction API Reference Transport packages React SDK SDK Introduction API Reference React Native SDK SDK Introduction API Reference iOS SDK SDK Introduction API Reference Transport packages Android SDK SDK Introduction API Reference Transport packages C++ SDK SDK Introduction Daily WebRTC Transport The RTVI (Real-Time Voice and Video Inference) standard defines a set of message types and structures sent between clients and servers. It is designed to facilitate real-time interactions between clients and AI applications that require voice, video, and text communication. It provides a consistent framework for building applications that can communicate with AI models and the backends running those models in real-time. This page documents version 1.0 of the RTVI standard, released in June 2025. ​ Key Features Connection Management RTVI provides a flexible connection model that allows clients to connect to AI services and coordinate state. Transcriptions The standard includes built-in support for real-time transcription of audio streams. Client-Server Messaging The standard defines a messaging protocol for sending and receiving messages between clients and servers, allowing for efficient communication of requests and responses. Advanced LLM Interactions The standard supports advanced interactions with large language models (LLMs), including context management, function call handline, and search results. Service-Specific Insights RTVI supports events to provide insight into the input/output and state for typical services that exist in speech-to-speech workflows. Metrics and Monitoring RTVI provides mechanisms for collecting metrics and monitoring the performance of server-side services. ​ Terms Client : The front-end application or user interface that interacts with the RTVI server. Server : The backend-end service that runs the AI framework and processes requests from the client. User : The end user interacting with the client application. Bot : The AI interacting with the user, technically an amalgamation of a large language model (LLM) and a text-to-speech (TTS) service. ​ RTVI Message Format The messages defined as part of the RTVI protocol adhere to the following format: Copy Ask AI { "id" : string , "label" : "rtvi-ai" , "type" : string , "data" : unknown } ​ id string A unique identifier for the message, used to correlate requests and responses. ​ label string default: "rtvi-ai" required A label that identifies this message as an RTVI message. This field is required and should always be set to 'rtvi-ai' . ​ type string required The type of message being sent. This field is required and should be set to one of the predefined RTVI message types listed below. ​ data unknown The payload of the message, which can be any data structure relevant to the message type. ​ RTVI Message Types Following the above format, this section describes the various message types defined by the RTVI standard. Each message type has a specific purpose and structure, allowing for clear communication between clients and servers. Each message type below includes either a 🤖 or 🏄 emoji to denote whether the message is sent from the bot (🤖) or client (🏄). ​ Connection Management ​ client-ready 🏄 Indicates that the client is ready to receive messages and interact with the server. Typically sent after the transport media channels have connected. type : 'client-ready' data : version : string The version of the RTVI standard being used. This is useful for ensuring compatibility between client and server implementations. about : AboutClient Object An object containing information about the client, such as its rtvi-version, client library, and any other relevant metadata. The AboutClient object follows this structure: Show AboutClient ​ library string required ​ library_version string ​ platform string ​ platform_version string ​ platform_details any Any platform-specific details that may be relevant to the server. This could include information about the browser, operating system, or any other environment-specific data needed by the server. This field is optional and open-ended, so please be mindful of the data you include here and any security concerns that may arise from exposing sensitive or personal-identifiable information. ​ bot-ready 🤖 Indicates that the bot is ready to receive messages and interact with the client. Typically send after the transport media channels have connected. type : 'bot-ready' data : version : string The version of the RTVI standard being used. This is useful for ensuring compatibility between client and server implementations. about : any (Optional) An object containing information about the server or bot. It’s structure and value are both undefined by default. This provides flexibility to include any relevant metadata your client may need to know about the server at connection time, without any built-in security concerns. Please be mindful of the data you include here and any security concerns that may arise from exposing sensitive information. ​ disconnect-bot 🏄 Indicates that the client wishes to disconnect from the bot. Typically used when the client is shutting down or no longer needs to interact with the bot. Note: Disconnets should happen automatically when either the client or bot disconnects from the transport, so this message is intended for the case where a client may want to remain connected to the transport but no longer wishes to interact with the bot. type : 'disconnect-bot' data : undefined ​ error 🤖 Indicates an error occurred during bot initialization or runtime. type : 'error' data : message : string Description of the error. fatal : boolean Indicates if the error is fatal to the session. ​ Transcription ​ user-started-speaking 🤖 Emitted when the user begins speaking type : 'user-started-speaking' data : None ​ user-stopped-speaking 🤖 Emitted when the user stops speaking type : 'user-stopped-speaking' data : None ​ bot-started-speaking 🤖 Emitted when the bot begins speaking type : 'bot-started-speaking' data : None ​ bot-stopped-speaking 🤖 Emitted when the bot stops speaking type : 'bot-stopped-speaking' data : None ​ user-transcription 🤖 Real-time transcription of user speech, including both partial and final results. type : 'user-transcription' data : text : string The transcribed text of the user. final : boolean Indicates if this is a final transcription or a partial result. timestamp : string The timestamp when the transcription was generated. user_id : string Identifier for the user who spoke. ​ bot-transcription 🤖 Transcription of the bot’s speech. Note: This protocol currently does not match the user transcription format to support real-time timestamping for bot transcriptions. Rather, the event is typically sent for each sentence of the bot’s response. This difference is currently due to limitations in TTS services which mostly do not support (or support well), accurate timing information. If/when this changes, this protocol may be updated to include the necessary timing information. For now, if you want to attempt real-time transcription to match your bot’s speaking, you can try using the bot-tts-text message type. type : 'bot-transcription' data : text : string The transcribed text from the bot, typically aggregated at a per-sentence level. ​ Client-Server Messaging ​ server-message 🤖 An arbitrary message sent from the server to the client. This can be used for custom interactions or commands. This message may be coupled with the client-message message type to handle responses from the client. type : 'server-message' data : any The data can be any JSON-serializable object, formatted according to your own specifications. ​ client-message 🏄 An arbitrary message sent from the client to the server. This can be used for custom interactions or commands. This message may be coupled with the server-response message type to handle responses from the server. type : 'client-message' data : t : string d : unknown (optional) The data payload should contain a t field indicating the type of message and an optional d field containing any custom, corresponding data needed for the message. ​ server-response 🤖 An message sent from the server to the client in response to a client-message . IMPORTANT : The id should match the id of the original client-message to correlate the response with the request. type : 'client-message' data : t : string d : unknown (optional) The data payload should contain a t field indicating the type of message and an optional d field containing any custom, corresponding data needed for the message. ​ error-response 🤖 Error response to a specific client message. IMPORTANT : The id should match the id of the original client-message to correlate the response with the request. type : 'error-response' data : error : string ​ Advanced LLM Interactions ​ append-to-context 🏄 A message sent from the client to the server to append data to the context of the current llm conversation. This is useful for providing text-based content for the user or augmenting the context for the assistant. type : 'append-to-context' data : role : "user" | "assistant" The role the context should be appended to. Currently only supports "user" and "assistant" . content : unknown The content to append to the context. This can be any data structure the llm understand. run_immediately : boolean (optional) Indicates whether the context should be run immediately after appending. Defaults to false . If set to false , the context will be appended but not executed until the next llm run. ​ llm-function-call 🤖 A function call request from the LLM, sent from the bot to the client. Note that for most cases, an LLM function call will be handled completely server-side. However, in the event that the call requires input from the client or the client needs to be aware of the function call, this message/response schema is required. type : 'llm-function-call' data : function_name : string Name of the function to be called. tool_call_id : string Unique identifier for this function call. args : Record<string, unknown> Arguments to be passed to the function. ​ llm-function-call-result 🏄 The result of the function call requested by the LLM, returned from the client. type : 'llm-function-call-result' data : function_name : string Name of the called function. tool_call_id : string Identifier matching the original function call. args : Record<string, unknown> Arguments that were passed to the function. result : Record<string, unknown> | string The result returned by the function. ​ bot-llm-search-response 🤖 Search results from the LLM’s knowledge base. Currently, Google Gemini is the only LLM that supports built-in search. However, we expect other LLMs to follow suite, which is why this message type is defined as part of the RTVI standard. As more LLMs add support for this feature, the format of this message type may evolve to accommodate discrepancies. type : 'bot-llm-search-response' data : search_result : string (optional) Raw search result text. rendered_content : string (optional) Formatted version of the search results. origins : Array<Origin Object> Source information and confidence scores for search results. The Origin Object follows this structure: Copy Ask AI { "site_uri" : string (optional) , "site_title" : string (optional) , "results" : Array< { "text" : string , "confidence" : number [] } > } Example: Copy Ask AI "id" : undefined "label" : "rtvi-ai" "type" : "bot-llm-search-response" "data" : { "origins" : [ { "results" : [ { "confidence" : [ 0.9881149530410768 ], "text" : "* Juneteenth: A Freedom Celebration is scheduled for June 18th from 12:00 pm to 2:00 pm." }, { "confidence" : [ 0.9692034721374512 ], "ext" : "* A Juneteenth celebration at Fort Negley Park will take place on June 19th from 5:00 pm to 9:30 pm." } ], "site_title" : "vanderbilt.edu" , "site_uri" : "https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQHwif83VK9KAzrbMSGSBsKwL8vWfSfC9pgEWYKmStHyqiRoV1oe8j1S0nbwRg_iWgqAr9wUkiegu3ATC8Ll-cuE-vpzwElRHiJ2KgRYcqnOQMoOeokVpWqi" }, { "results" : [ { "confidence" : [ 0.6554043292999268 ], "text" : "In addition to these events, Vanderbilt University is a large research institution with ongoing activities across many fields." } ], "site_title" : "wikipedia.org" , "site_uri" : "https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQESbF-ijx78QbaglrhflHCUWdPTD4M6tYOQigW5hgsHNctRlAHu9ktfPmJx7DfoP5QicE0y-OQY1cRl9w4Id0btiFgLYSKIm2-SPtOHXeNrAlgA7mBnclaGrD7rgnLIbrjl8DgUEJrrvT0CKzuo" }], "rendered_content" : "<style> \n .container ... </div> \n </div> \n " , "search_result" : "Several events are happening at Vanderbilt University: \n\n * Juneteenth: A Freedom Celebration is scheduled for June 18th from 12:00 pm to 2:00 pm. \n * A Juneteenth celebration at Fort Negley Park will take place on June 19th from 5:00 pm to 9:30 pm. \n\n In addition to these events, Vanderbilt University is a large research institution with ongoing activities across many fields. For the most recent news, you should check Vanderbilt's official news website. \n " } ​ Service-Specific Insights ​ bot-llm-started 🤖 Indicates LLM processing has begun type : bot-llm-started data : None ​ bot-llm-stopped 🤖 Indicates LLM processing has completed type : bot-llm-stopped data : None ​ user-llm-text 🤖 Aggregated user input text that is sent to the LLM. type : 'user-llm-text' data : text : string The user’s input text to be processed by the LLM. ​ bot-llm-text 🤖 Individual tokens streamed from the LLM as they are generated. type : 'bot-llm-text' data : text : string The token text from the LLM. ​ bot-tts-started 🤖 Indicates text-to-speech (TTS) processing has begun. type : 'bot-tts-started' data : None ​ bot-tts-stopped 🤖 Indicates text-to-speech (TTS) processing has completed. type : 'bot-tts-stopped' data : None ​ bot-tts-text 🤖 The per-token text output of the text-to-speech (TTS) service (what the TTS actually says). type : 'bot-tts-text' data : text : string The text representation of the generated bot speech. ​ Metrics and Monitoring ​ metrics 🤖 Performance metrics for various processing stages and services. Each message will contain entries for one or more of the metrics types: processing , ttfb , characters . type : 'metrics' data : processing : [See Below] (optional) Processing time metrics. ttfb : [See Below] (optional) Time to first byte metrics. characters : [See Below] (optional) Character processing metrics. For each metric type, the data structure is an array of objects with the following structure: processor : string The name of the processor or service that generated the metric. value : number The value of the metric, typically in milliseconds or character count. model : string (optional) The model of the service that generated the metric, if applicable. Example: Copy Ask AI { "type" : "metrics" , "data" : { "processing" : [ { "model" : "eleven_flash_v2_5" , "processor" : "ElevenLabsTTSService#0" , "value" : 0.0005140304565429688 } ], "ttfb" : [ { "model" : "eleven_flash_v2_5" , "processor" : "ElevenLabsTTSService#0" , "value" : 0.1573178768157959 } ], "characters" : [ { "model" : "eleven_flash_v2_5" , "processor" : "ElevenLabsTTSService#0" , "value" : 43 } ] } } Client SDKs RTVIClient Migration Guide On this page Key Features Terms RTVI Message Format RTVI Message Types Connection Management client-ready 🏄 bot-ready 🤖 disconnect-bot 🏄 error 🤖 Transcription user-started-speaking 🤖 user-stopped-speaking 🤖 bot-started-speaking 🤖 bot-stopped-speaking 🤖 user-transcription 🤖 bot-transcription 🤖 Client-Server Messaging server-message 🤖 client-message 🏄 server-response 🤖 error-response 🤖 Advanced LLM Interactions append-to-context 🏄 llm-function-call 🤖 llm-function-call-result 🏄 bot-llm-search-response 🤖 Service-Specific Insights bot-llm-started 🤖 bot-llm-stopped 🤖 user-llm-text 🤖 bot-llm-text 🤖 bot-tts-started 🤖 bot-tts-stopped 🤖 bot-tts-text 🤖 Metrics and Monitoring metrics 🤖 Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/client_rtvi-standard_453c385a.txt b/client_rtvi-standard_453c385a.txt
new file mode 100644
index 0000000000000000000000000000000000000000..fef3c5041ad66b1a6bbd2270f9b582a50a5027ce
--- /dev/null
+++ b/client_rtvi-standard_453c385a.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/client/rtvi-standard
+Title: The RTVI Standard - Pipecat
+==================================================
+
+The RTVI Standard - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation The RTVI Standard Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Client SDKs The RTVI Standard RTVIClient Migration Guide Javascript SDK SDK Introduction API Reference Transport packages React SDK SDK Introduction API Reference React Native SDK SDK Introduction API Reference iOS SDK SDK Introduction API Reference Transport packages Android SDK SDK Introduction API Reference Transport packages C++ SDK SDK Introduction Daily WebRTC Transport The RTVI (Real-Time Voice and Video Inference) standard defines a set of message types and structures sent between clients and servers. It is designed to facilitate real-time interactions between clients and AI applications that require voice, video, and text communication. It provides a consistent framework for building applications that can communicate with AI models and the backends running those models in real-time. This page documents version 1.0 of the RTVI standard, released in June 2025. ​ Key Features Connection Management RTVI provides a flexible connection model that allows clients to connect to AI services and coordinate state. Transcriptions The standard includes built-in support for real-time transcription of audio streams. Client-Server Messaging The standard defines a messaging protocol for sending and receiving messages between clients and servers, allowing for efficient communication of requests and responses. Advanced LLM Interactions The standard supports advanced interactions with large language models (LLMs), including context management, function call handline, and search results. Service-Specific Insights RTVI supports events to provide insight into the input/output and state for typical services that exist in speech-to-speech workflows. Metrics and Monitoring RTVI provides mechanisms for collecting metrics and monitoring the performance of server-side services. ​ Terms Client : The front-end application or user interface that interacts with the RTVI server. Server : The backend-end service that runs the AI framework and processes requests from the client. User : The end user interacting with the client application. Bot : The AI interacting with the user, technically an amalgamation of a large language model (LLM) and a text-to-speech (TTS) service. ​ RTVI Message Format The messages defined as part of the RTVI protocol adhere to the following format: Copy Ask AI { "id" : string , "label" : "rtvi-ai" , "type" : string , "data" : unknown } ​ id string A unique identifier for the message, used to correlate requests and responses. ​ label string default: "rtvi-ai" required A label that identifies this message as an RTVI message. This field is required and should always be set to 'rtvi-ai' . ​ type string required The type of message being sent. This field is required and should be set to one of the predefined RTVI message types listed below. ​ data unknown The payload of the message, which can be any data structure relevant to the message type. ​ RTVI Message Types Following the above format, this section describes the various message types defined by the RTVI standard. Each message type has a specific purpose and structure, allowing for clear communication between clients and servers. Each message type below includes either a 🤖 or 🏄 emoji to denote whether the message is sent from the bot (🤖) or client (🏄). ​ Connection Management ​ client-ready 🏄 Indicates that the client is ready to receive messages and interact with the server. Typically sent after the transport media channels have connected. type : 'client-ready' data : version : string The version of the RTVI standard being used. This is useful for ensuring compatibility between client and server implementations. about : AboutClient Object An object containing information about the client, such as its rtvi-version, client library, and any other relevant metadata. The AboutClient object follows this structure: Show AboutClient ​ library string required ​ library_version string ​ platform string ​ platform_version string ​ platform_details any Any platform-specific details that may be relevant to the server. This could include information about the browser, operating system, or any other environment-specific data needed by the server. This field is optional and open-ended, so please be mindful of the data you include here and any security concerns that may arise from exposing sensitive or personal-identifiable information. ​ bot-ready 🤖 Indicates that the bot is ready to receive messages and interact with the client. Typically send after the transport media channels have connected. type : 'bot-ready' data : version : string The version of the RTVI standard being used. This is useful for ensuring compatibility between client and server implementations. about : any (Optional) An object containing information about the server or bot. It’s structure and value are both undefined by default. This provides flexibility to include any relevant metadata your client may need to know about the server at connection time, without any built-in security concerns. Please be mindful of the data you include here and any security concerns that may arise from exposing sensitive information. ​ disconnect-bot 🏄 Indicates that the client wishes to disconnect from the bot. Typically used when the client is shutting down or no longer needs to interact with the bot. Note: Disconnets should happen automatically when either the client or bot disconnects from the transport, so this message is intended for the case where a client may want to remain connected to the transport but no longer wishes to interact with the bot. type : 'disconnect-bot' data : undefined ​ error 🤖 Indicates an error occurred during bot initialization or runtime. type : 'error' data : message : string Description of the error. fatal : boolean Indicates if the error is fatal to the session. ​ Transcription ​ user-started-speaking 🤖 Emitted when the user begins speaking type : 'user-started-speaking' data : None ​ user-stopped-speaking 🤖 Emitted when the user stops speaking type : 'user-stopped-speaking' data : None ​ bot-started-speaking 🤖 Emitted when the bot begins speaking type : 'bot-started-speaking' data : None ​ bot-stopped-speaking 🤖 Emitted when the bot stops speaking type : 'bot-stopped-speaking' data : None ​ user-transcription 🤖 Real-time transcription of user speech, including both partial and final results. type : 'user-transcription' data : text : string The transcribed text of the user. final : boolean Indicates if this is a final transcription or a partial result. timestamp : string The timestamp when the transcription was generated. user_id : string Identifier for the user who spoke. ​ bot-transcription 🤖 Transcription of the bot’s speech. Note: This protocol currently does not match the user transcription format to support real-time timestamping for bot transcriptions. Rather, the event is typically sent for each sentence of the bot’s response. This difference is currently due to limitations in TTS services which mostly do not support (or support well), accurate timing information. If/when this changes, this protocol may be updated to include the necessary timing information. For now, if you want to attempt real-time transcription to match your bot’s speaking, you can try using the bot-tts-text message type. type : 'bot-transcription' data : text : string The transcribed text from the bot, typically aggregated at a per-sentence level. ​ Client-Server Messaging ​ server-message 🤖 An arbitrary message sent from the server to the client. This can be used for custom interactions or commands. This message may be coupled with the client-message message type to handle responses from the client. type : 'server-message' data : any The data can be any JSON-serializable object, formatted according to your own specifications. ​ client-message 🏄 An arbitrary message sent from the client to the server. This can be used for custom interactions or commands. This message may be coupled with the server-response message type to handle responses from the server. type : 'client-message' data : t : string d : unknown (optional) The data payload should contain a t field indicating the type of message and an optional d field containing any custom, corresponding data needed for the message. ​ server-response 🤖 An message sent from the server to the client in response to a client-message . IMPORTANT : The id should match the id of the original client-message to correlate the response with the request. type : 'client-message' data : t : string d : unknown (optional) The data payload should contain a t field indicating the type of message and an optional d field containing any custom, corresponding data needed for the message. ​ error-response 🤖 Error response to a specific client message. IMPORTANT : The id should match the id of the original client-message to correlate the response with the request. type : 'error-response' data : error : string ​ Advanced LLM Interactions ​ append-to-context 🏄 A message sent from the client to the server to append data to the context of the current llm conversation. This is useful for providing text-based content for the user or augmenting the context for the assistant. type : 'append-to-context' data : role : "user" | "assistant" The role the context should be appended to. Currently only supports "user" and "assistant" . content : unknown The content to append to the context. This can be any data structure the llm understand. run_immediately : boolean (optional) Indicates whether the context should be run immediately after appending. Defaults to false . If set to false , the context will be appended but not executed until the next llm run. ​ llm-function-call 🤖 A function call request from the LLM, sent from the bot to the client. Note that for most cases, an LLM function call will be handled completely server-side. However, in the event that the call requires input from the client or the client needs to be aware of the function call, this message/response schema is required. type : 'llm-function-call' data : function_name : string Name of the function to be called. tool_call_id : string Unique identifier for this function call. args : Record<string, unknown> Arguments to be passed to the function. ​ llm-function-call-result 🏄 The result of the function call requested by the LLM, returned from the client. type : 'llm-function-call-result' data : function_name : string Name of the called function. tool_call_id : string Identifier matching the original function call. args : Record<string, unknown> Arguments that were passed to the function. result : Record<string, unknown> | string The result returned by the function. ​ bot-llm-search-response 🤖 Search results from the LLM’s knowledge base. Currently, Google Gemini is the only LLM that supports built-in search. However, we expect other LLMs to follow suite, which is why this message type is defined as part of the RTVI standard. As more LLMs add support for this feature, the format of this message type may evolve to accommodate discrepancies. type : 'bot-llm-search-response' data : search_result : string (optional) Raw search result text. rendered_content : string (optional) Formatted version of the search results. origins : Array<Origin Object> Source information and confidence scores for search results. The Origin Object follows this structure: Copy Ask AI { "site_uri" : string (optional) , "site_title" : string (optional) , "results" : Array< { "text" : string , "confidence" : number [] } > } Example: Copy Ask AI "id" : undefined "label" : "rtvi-ai" "type" : "bot-llm-search-response" "data" : { "origins" : [ { "results" : [ { "confidence" : [ 0.9881149530410768 ], "text" : "* Juneteenth: A Freedom Celebration is scheduled for June 18th from 12:00 pm to 2:00 pm." }, { "confidence" : [ 0.9692034721374512 ], "ext" : "* A Juneteenth celebration at Fort Negley Park will take place on June 19th from 5:00 pm to 9:30 pm." } ], "site_title" : "vanderbilt.edu" , "site_uri" : "https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQHwif83VK9KAzrbMSGSBsKwL8vWfSfC9pgEWYKmStHyqiRoV1oe8j1S0nbwRg_iWgqAr9wUkiegu3ATC8Ll-cuE-vpzwElRHiJ2KgRYcqnOQMoOeokVpWqi" }, { "results" : [ { "confidence" : [ 0.6554043292999268 ], "text" : "In addition to these events, Vanderbilt University is a large research institution with ongoing activities across many fields." } ], "site_title" : "wikipedia.org" , "site_uri" : "https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQESbF-ijx78QbaglrhflHCUWdPTD4M6tYOQigW5hgsHNctRlAHu9ktfPmJx7DfoP5QicE0y-OQY1cRl9w4Id0btiFgLYSKIm2-SPtOHXeNrAlgA7mBnclaGrD7rgnLIbrjl8DgUEJrrvT0CKzuo" }], "rendered_content" : "<style> \n .container ... </div> \n </div> \n " , "search_result" : "Several events are happening at Vanderbilt University: \n\n * Juneteenth: A Freedom Celebration is scheduled for June 18th from 12:00 pm to 2:00 pm. \n * A Juneteenth celebration at Fort Negley Park will take place on June 19th from 5:00 pm to 9:30 pm. \n\n In addition to these events, Vanderbilt University is a large research institution with ongoing activities across many fields. For the most recent news, you should check Vanderbilt's official news website. \n " } ​ Service-Specific Insights ​ bot-llm-started 🤖 Indicates LLM processing has begun type : bot-llm-started data : None ​ bot-llm-stopped 🤖 Indicates LLM processing has completed type : bot-llm-stopped data : None ​ user-llm-text 🤖 Aggregated user input text that is sent to the LLM. type : 'user-llm-text' data : text : string The user’s input text to be processed by the LLM. ​ bot-llm-text 🤖 Individual tokens streamed from the LLM as they are generated. type : 'bot-llm-text' data : text : string The token text from the LLM. ​ bot-tts-started 🤖 Indicates text-to-speech (TTS) processing has begun. type : 'bot-tts-started' data : None ​ bot-tts-stopped 🤖 Indicates text-to-speech (TTS) processing has completed. type : 'bot-tts-stopped' data : None ​ bot-tts-text 🤖 The per-token text output of the text-to-speech (TTS) service (what the TTS actually says). type : 'bot-tts-text' data : text : string The text representation of the generated bot speech. ​ Metrics and Monitoring ​ metrics 🤖 Performance metrics for various processing stages and services. Each message will contain entries for one or more of the metrics types: processing , ttfb , characters . type : 'metrics' data : processing : [See Below] (optional) Processing time metrics. ttfb : [See Below] (optional) Time to first byte metrics. characters : [See Below] (optional) Character processing metrics. For each metric type, the data structure is an array of objects with the following structure: processor : string The name of the processor or service that generated the metric. value : number The value of the metric, typically in milliseconds or character count. model : string (optional) The model of the service that generated the metric, if applicable. Example: Copy Ask AI { "type" : "metrics" , "data" : { "processing" : [ { "model" : "eleven_flash_v2_5" , "processor" : "ElevenLabsTTSService#0" , "value" : 0.0005140304565429688 } ], "ttfb" : [ { "model" : "eleven_flash_v2_5" , "processor" : "ElevenLabsTTSService#0" , "value" : 0.1573178768157959 } ], "characters" : [ { "model" : "eleven_flash_v2_5" , "processor" : "ElevenLabsTTSService#0" , "value" : 43 } ] } } Client SDKs RTVIClient Migration Guide On this page Key Features Terms RTVI Message Format RTVI Message Types Connection Management client-ready 🏄 bot-ready 🤖 disconnect-bot 🏄 error 🤖 Transcription user-started-speaking 🤖 user-stopped-speaking 🤖 bot-started-speaking 🤖 bot-stopped-speaking 🤖 user-transcription 🤖 bot-transcription 🤖 Client-Server Messaging server-message 🤖 client-message 🏄 server-response 🤖 error-response 🤖 Advanced LLM Interactions append-to-context 🏄 llm-function-call 🤖 llm-function-call-result 🏄 bot-llm-search-response 🤖 Service-Specific Insights bot-llm-started 🤖 bot-llm-stopped 🤖 user-llm-text 🤖 bot-llm-text 🤖 bot-tts-started 🤖 bot-tts-stopped 🤖 bot-tts-text 🤖 Metrics and Monitoring metrics 🤖 Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/client_rtvi-standard_57732c98.txt b/client_rtvi-standard_57732c98.txt
new file mode 100644
index 0000000000000000000000000000000000000000..87a6e163248f090a80363474f9187c8d130bcbc6
--- /dev/null
+++ b/client_rtvi-standard_57732c98.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/client/rtvi-standard#bot-tts-stopped-%F0%9F%A4%96
+Title: The RTVI Standard - Pipecat
+==================================================
+
+The RTVI Standard - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation The RTVI Standard Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Client SDKs The RTVI Standard RTVIClient Migration Guide Javascript SDK SDK Introduction API Reference Transport packages React SDK SDK Introduction API Reference React Native SDK SDK Introduction API Reference iOS SDK SDK Introduction API Reference Transport packages Android SDK SDK Introduction API Reference Transport packages C++ SDK SDK Introduction Daily WebRTC Transport The RTVI (Real-Time Voice and Video Inference) standard defines a set of message types and structures sent between clients and servers. It is designed to facilitate real-time interactions between clients and AI applications that require voice, video, and text communication. It provides a consistent framework for building applications that can communicate with AI models and the backends running those models in real-time. This page documents version 1.0 of the RTVI standard, released in June 2025. ​ Key Features Connection Management RTVI provides a flexible connection model that allows clients to connect to AI services and coordinate state. Transcriptions The standard includes built-in support for real-time transcription of audio streams. Client-Server Messaging The standard defines a messaging protocol for sending and receiving messages between clients and servers, allowing for efficient communication of requests and responses. Advanced LLM Interactions The standard supports advanced interactions with large language models (LLMs), including context management, function call handline, and search results. Service-Specific Insights RTVI supports events to provide insight into the input/output and state for typical services that exist in speech-to-speech workflows. Metrics and Monitoring RTVI provides mechanisms for collecting metrics and monitoring the performance of server-side services. ​ Terms Client : The front-end application or user interface that interacts with the RTVI server. Server : The backend-end service that runs the AI framework and processes requests from the client. User : The end user interacting with the client application. Bot : The AI interacting with the user, technically an amalgamation of a large language model (LLM) and a text-to-speech (TTS) service. ​ RTVI Message Format The messages defined as part of the RTVI protocol adhere to the following format: Copy Ask AI { "id" : string , "label" : "rtvi-ai" , "type" : string , "data" : unknown } ​ id string A unique identifier for the message, used to correlate requests and responses. ​ label string default: "rtvi-ai" required A label that identifies this message as an RTVI message. This field is required and should always be set to 'rtvi-ai' . ​ type string required The type of message being sent. This field is required and should be set to one of the predefined RTVI message types listed below. ​ data unknown The payload of the message, which can be any data structure relevant to the message type. ​ RTVI Message Types Following the above format, this section describes the various message types defined by the RTVI standard. Each message type has a specific purpose and structure, allowing for clear communication between clients and servers. Each message type below includes either a 🤖 or 🏄 emoji to denote whether the message is sent from the bot (🤖) or client (🏄). ​ Connection Management ​ client-ready 🏄 Indicates that the client is ready to receive messages and interact with the server. Typically sent after the transport media channels have connected. type : 'client-ready' data : version : string The version of the RTVI standard being used. This is useful for ensuring compatibility between client and server implementations. about : AboutClient Object An object containing information about the client, such as its rtvi-version, client library, and any other relevant metadata. The AboutClient object follows this structure: Show AboutClient ​ library string required ​ library_version string ​ platform string ​ platform_version string ​ platform_details any Any platform-specific details that may be relevant to the server. This could include information about the browser, operating system, or any other environment-specific data needed by the server. This field is optional and open-ended, so please be mindful of the data you include here and any security concerns that may arise from exposing sensitive or personal-identifiable information. ​ bot-ready 🤖 Indicates that the bot is ready to receive messages and interact with the client. Typically send after the transport media channels have connected. type : 'bot-ready' data : version : string The version of the RTVI standard being used. This is useful for ensuring compatibility between client and server implementations. about : any (Optional) An object containing information about the server or bot. It’s structure and value are both undefined by default. This provides flexibility to include any relevant metadata your client may need to know about the server at connection time, without any built-in security concerns. Please be mindful of the data you include here and any security concerns that may arise from exposing sensitive information. ​ disconnect-bot 🏄 Indicates that the client wishes to disconnect from the bot. Typically used when the client is shutting down or no longer needs to interact with the bot. Note: Disconnets should happen automatically when either the client or bot disconnects from the transport, so this message is intended for the case where a client may want to remain connected to the transport but no longer wishes to interact with the bot. type : 'disconnect-bot' data : undefined ​ error 🤖 Indicates an error occurred during bot initialization or runtime. type : 'error' data : message : string Description of the error. fatal : boolean Indicates if the error is fatal to the session. ​ Transcription ​ user-started-speaking 🤖 Emitted when the user begins speaking type : 'user-started-speaking' data : None ​ user-stopped-speaking 🤖 Emitted when the user stops speaking type : 'user-stopped-speaking' data : None ​ bot-started-speaking 🤖 Emitted when the bot begins speaking type : 'bot-started-speaking' data : None ​ bot-stopped-speaking 🤖 Emitted when the bot stops speaking type : 'bot-stopped-speaking' data : None ​ user-transcription 🤖 Real-time transcription of user speech, including both partial and final results. type : 'user-transcription' data : text : string The transcribed text of the user. final : boolean Indicates if this is a final transcription or a partial result. timestamp : string The timestamp when the transcription was generated. user_id : string Identifier for the user who spoke. ​ bot-transcription 🤖 Transcription of the bot’s speech. Note: This protocol currently does not match the user transcription format to support real-time timestamping for bot transcriptions. Rather, the event is typically sent for each sentence of the bot’s response. This difference is currently due to limitations in TTS services which mostly do not support (or support well), accurate timing information. If/when this changes, this protocol may be updated to include the necessary timing information. For now, if you want to attempt real-time transcription to match your bot’s speaking, you can try using the bot-tts-text message type. type : 'bot-transcription' data : text : string The transcribed text from the bot, typically aggregated at a per-sentence level. ​ Client-Server Messaging ​ server-message 🤖 An arbitrary message sent from the server to the client. This can be used for custom interactions or commands. This message may be coupled with the client-message message type to handle responses from the client. type : 'server-message' data : any The data can be any JSON-serializable object, formatted according to your own specifications. ​ client-message 🏄 An arbitrary message sent from the client to the server. This can be used for custom interactions or commands. This message may be coupled with the server-response message type to handle responses from the server. type : 'client-message' data : t : string d : unknown (optional) The data payload should contain a t field indicating the type of message and an optional d field containing any custom, corresponding data needed for the message. ​ server-response 🤖 An message sent from the server to the client in response to a client-message . IMPORTANT : The id should match the id of the original client-message to correlate the response with the request. type : 'client-message' data : t : string d : unknown (optional) The data payload should contain a t field indicating the type of message and an optional d field containing any custom, corresponding data needed for the message. ​ error-response 🤖 Error response to a specific client message. IMPORTANT : The id should match the id of the original client-message to correlate the response with the request. type : 'error-response' data : error : string ​ Advanced LLM Interactions ​ append-to-context 🏄 A message sent from the client to the server to append data to the context of the current llm conversation. This is useful for providing text-based content for the user or augmenting the context for the assistant. type : 'append-to-context' data : role : "user" | "assistant" The role the context should be appended to. Currently only supports "user" and "assistant" . content : unknown The content to append to the context. This can be any data structure the llm understand. run_immediately : boolean (optional) Indicates whether the context should be run immediately after appending. Defaults to false . If set to false , the context will be appended but not executed until the next llm run. ​ llm-function-call 🤖 A function call request from the LLM, sent from the bot to the client. Note that for most cases, an LLM function call will be handled completely server-side. However, in the event that the call requires input from the client or the client needs to be aware of the function call, this message/response schema is required. type : 'llm-function-call' data : function_name : string Name of the function to be called. tool_call_id : string Unique identifier for this function call. args : Record<string, unknown> Arguments to be passed to the function. ​ llm-function-call-result 🏄 The result of the function call requested by the LLM, returned from the client. type : 'llm-function-call-result' data : function_name : string Name of the called function. tool_call_id : string Identifier matching the original function call. args : Record<string, unknown> Arguments that were passed to the function. result : Record<string, unknown> | string The result returned by the function. ​ bot-llm-search-response 🤖 Search results from the LLM’s knowledge base. Currently, Google Gemini is the only LLM that supports built-in search. However, we expect other LLMs to follow suite, which is why this message type is defined as part of the RTVI standard. As more LLMs add support for this feature, the format of this message type may evolve to accommodate discrepancies. type : 'bot-llm-search-response' data : search_result : string (optional) Raw search result text. rendered_content : string (optional) Formatted version of the search results. origins : Array<Origin Object> Source information and confidence scores for search results. The Origin Object follows this structure: Copy Ask AI { "site_uri" : string (optional) , "site_title" : string (optional) , "results" : Array< { "text" : string , "confidence" : number [] } > } Example: Copy Ask AI "id" : undefined "label" : "rtvi-ai" "type" : "bot-llm-search-response" "data" : { "origins" : [ { "results" : [ { "confidence" : [ 0.9881149530410768 ], "text" : "* Juneteenth: A Freedom Celebration is scheduled for June 18th from 12:00 pm to 2:00 pm." }, { "confidence" : [ 0.9692034721374512 ], "ext" : "* A Juneteenth celebration at Fort Negley Park will take place on June 19th from 5:00 pm to 9:30 pm." } ], "site_title" : "vanderbilt.edu" , "site_uri" : "https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQHwif83VK9KAzrbMSGSBsKwL8vWfSfC9pgEWYKmStHyqiRoV1oe8j1S0nbwRg_iWgqAr9wUkiegu3ATC8Ll-cuE-vpzwElRHiJ2KgRYcqnOQMoOeokVpWqi" }, { "results" : [ { "confidence" : [ 0.6554043292999268 ], "text" : "In addition to these events, Vanderbilt University is a large research institution with ongoing activities across many fields." } ], "site_title" : "wikipedia.org" , "site_uri" : "https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQESbF-ijx78QbaglrhflHCUWdPTD4M6tYOQigW5hgsHNctRlAHu9ktfPmJx7DfoP5QicE0y-OQY1cRl9w4Id0btiFgLYSKIm2-SPtOHXeNrAlgA7mBnclaGrD7rgnLIbrjl8DgUEJrrvT0CKzuo" }], "rendered_content" : "<style> \n .container ... </div> \n </div> \n " , "search_result" : "Several events are happening at Vanderbilt University: \n\n * Juneteenth: A Freedom Celebration is scheduled for June 18th from 12:00 pm to 2:00 pm. \n * A Juneteenth celebration at Fort Negley Park will take place on June 19th from 5:00 pm to 9:30 pm. \n\n In addition to these events, Vanderbilt University is a large research institution with ongoing activities across many fields. For the most recent news, you should check Vanderbilt's official news website. \n " } ​ Service-Specific Insights ​ bot-llm-started 🤖 Indicates LLM processing has begun type : bot-llm-started data : None ​ bot-llm-stopped 🤖 Indicates LLM processing has completed type : bot-llm-stopped data : None ​ user-llm-text 🤖 Aggregated user input text that is sent to the LLM. type : 'user-llm-text' data : text : string The user’s input text to be processed by the LLM. ​ bot-llm-text 🤖 Individual tokens streamed from the LLM as they are generated. type : 'bot-llm-text' data : text : string The token text from the LLM. ​ bot-tts-started 🤖 Indicates text-to-speech (TTS) processing has begun. type : 'bot-tts-started' data : None ​ bot-tts-stopped 🤖 Indicates text-to-speech (TTS) processing has completed. type : 'bot-tts-stopped' data : None ​ bot-tts-text 🤖 The per-token text output of the text-to-speech (TTS) service (what the TTS actually says). type : 'bot-tts-text' data : text : string The text representation of the generated bot speech. ​ Metrics and Monitoring ​ metrics 🤖 Performance metrics for various processing stages and services. Each message will contain entries for one or more of the metrics types: processing , ttfb , characters . type : 'metrics' data : processing : [See Below] (optional) Processing time metrics. ttfb : [See Below] (optional) Time to first byte metrics. characters : [See Below] (optional) Character processing metrics. For each metric type, the data structure is an array of objects with the following structure: processor : string The name of the processor or service that generated the metric. value : number The value of the metric, typically in milliseconds or character count. model : string (optional) The model of the service that generated the metric, if applicable. Example: Copy Ask AI { "type" : "metrics" , "data" : { "processing" : [ { "model" : "eleven_flash_v2_5" , "processor" : "ElevenLabsTTSService#0" , "value" : 0.0005140304565429688 } ], "ttfb" : [ { "model" : "eleven_flash_v2_5" , "processor" : "ElevenLabsTTSService#0" , "value" : 0.1573178768157959 } ], "characters" : [ { "model" : "eleven_flash_v2_5" , "processor" : "ElevenLabsTTSService#0" , "value" : 43 } ] } } Client SDKs RTVIClient Migration Guide On this page Key Features Terms RTVI Message Format RTVI Message Types Connection Management client-ready 🏄 bot-ready 🤖 disconnect-bot 🏄 error 🤖 Transcription user-started-speaking 🤖 user-stopped-speaking 🤖 bot-started-speaking 🤖 bot-stopped-speaking 🤖 user-transcription 🤖 bot-transcription 🤖 Client-Server Messaging server-message 🤖 client-message 🏄 server-response 🤖 error-response 🤖 Advanced LLM Interactions append-to-context 🏄 llm-function-call 🤖 llm-function-call-result 🏄 bot-llm-search-response 🤖 Service-Specific Insights bot-llm-started 🤖 bot-llm-stopped 🤖 user-llm-text 🤖 bot-llm-text 🤖 bot-tts-started 🤖 bot-tts-stopped 🤖 bot-tts-text 🤖 Metrics and Monitoring metrics 🤖 Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/client_rtvi-standard_6fd8fcf8.txt b/client_rtvi-standard_6fd8fcf8.txt
new file mode 100644
index 0000000000000000000000000000000000000000..fb00f41eada730a2af5bd100c11d4fb1565a31e8
--- /dev/null
+++ b/client_rtvi-standard_6fd8fcf8.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/client/rtvi-standard#param-type
+Title: The RTVI Standard - Pipecat
+==================================================
+
+The RTVI Standard - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation The RTVI Standard Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Client SDKs The RTVI Standard RTVIClient Migration Guide Javascript SDK SDK Introduction API Reference Transport packages React SDK SDK Introduction API Reference React Native SDK SDK Introduction API Reference iOS SDK SDK Introduction API Reference Transport packages Android SDK SDK Introduction API Reference Transport packages C++ SDK SDK Introduction Daily WebRTC Transport The RTVI (Real-Time Voice and Video Inference) standard defines a set of message types and structures sent between clients and servers. It is designed to facilitate real-time interactions between clients and AI applications that require voice, video, and text communication. It provides a consistent framework for building applications that can communicate with AI models and the backends running those models in real-time. This page documents version 1.0 of the RTVI standard, released in June 2025. ​ Key Features Connection Management RTVI provides a flexible connection model that allows clients to connect to AI services and coordinate state. Transcriptions The standard includes built-in support for real-time transcription of audio streams. Client-Server Messaging The standard defines a messaging protocol for sending and receiving messages between clients and servers, allowing for efficient communication of requests and responses. Advanced LLM Interactions The standard supports advanced interactions with large language models (LLMs), including context management, function call handline, and search results. Service-Specific Insights RTVI supports events to provide insight into the input/output and state for typical services that exist in speech-to-speech workflows. Metrics and Monitoring RTVI provides mechanisms for collecting metrics and monitoring the performance of server-side services. ​ Terms Client : The front-end application or user interface that interacts with the RTVI server. Server : The backend-end service that runs the AI framework and processes requests from the client. User : The end user interacting with the client application. Bot : The AI interacting with the user, technically an amalgamation of a large language model (LLM) and a text-to-speech (TTS) service. ​ RTVI Message Format The messages defined as part of the RTVI protocol adhere to the following format: Copy Ask AI { "id" : string , "label" : "rtvi-ai" , "type" : string , "data" : unknown } ​ id string A unique identifier for the message, used to correlate requests and responses. ​ label string default: "rtvi-ai" required A label that identifies this message as an RTVI message. This field is required and should always be set to 'rtvi-ai' . ​ type string required The type of message being sent. This field is required and should be set to one of the predefined RTVI message types listed below. ​ data unknown The payload of the message, which can be any data structure relevant to the message type. ​ RTVI Message Types Following the above format, this section describes the various message types defined by the RTVI standard. Each message type has a specific purpose and structure, allowing for clear communication between clients and servers. Each message type below includes either a 🤖 or 🏄 emoji to denote whether the message is sent from the bot (🤖) or client (🏄). ​ Connection Management ​ client-ready 🏄 Indicates that the client is ready to receive messages and interact with the server. Typically sent after the transport media channels have connected. type : 'client-ready' data : version : string The version of the RTVI standard being used. This is useful for ensuring compatibility between client and server implementations. about : AboutClient Object An object containing information about the client, such as its rtvi-version, client library, and any other relevant metadata. The AboutClient object follows this structure: Show AboutClient ​ library string required ​ library_version string ​ platform string ​ platform_version string ​ platform_details any Any platform-specific details that may be relevant to the server. This could include information about the browser, operating system, or any other environment-specific data needed by the server. This field is optional and open-ended, so please be mindful of the data you include here and any security concerns that may arise from exposing sensitive or personal-identifiable information. ​ bot-ready 🤖 Indicates that the bot is ready to receive messages and interact with the client. Typically send after the transport media channels have connected. type : 'bot-ready' data : version : string The version of the RTVI standard being used. This is useful for ensuring compatibility between client and server implementations. about : any (Optional) An object containing information about the server or bot. It’s structure and value are both undefined by default. This provides flexibility to include any relevant metadata your client may need to know about the server at connection time, without any built-in security concerns. Please be mindful of the data you include here and any security concerns that may arise from exposing sensitive information. ​ disconnect-bot 🏄 Indicates that the client wishes to disconnect from the bot. Typically used when the client is shutting down or no longer needs to interact with the bot. Note: Disconnets should happen automatically when either the client or bot disconnects from the transport, so this message is intended for the case where a client may want to remain connected to the transport but no longer wishes to interact with the bot. type : 'disconnect-bot' data : undefined ​ error 🤖 Indicates an error occurred during bot initialization or runtime. type : 'error' data : message : string Description of the error. fatal : boolean Indicates if the error is fatal to the session. ​ Transcription ​ user-started-speaking 🤖 Emitted when the user begins speaking type : 'user-started-speaking' data : None ​ user-stopped-speaking 🤖 Emitted when the user stops speaking type : 'user-stopped-speaking' data : None ​ bot-started-speaking 🤖 Emitted when the bot begins speaking type : 'bot-started-speaking' data : None ​ bot-stopped-speaking 🤖 Emitted when the bot stops speaking type : 'bot-stopped-speaking' data : None ​ user-transcription 🤖 Real-time transcription of user speech, including both partial and final results. type : 'user-transcription' data : text : string The transcribed text of the user. final : boolean Indicates if this is a final transcription or a partial result. timestamp : string The timestamp when the transcription was generated. user_id : string Identifier for the user who spoke. ​ bot-transcription 🤖 Transcription of the bot’s speech. Note: This protocol currently does not match the user transcription format to support real-time timestamping for bot transcriptions. Rather, the event is typically sent for each sentence of the bot’s response. This difference is currently due to limitations in TTS services which mostly do not support (or support well), accurate timing information. If/when this changes, this protocol may be updated to include the necessary timing information. For now, if you want to attempt real-time transcription to match your bot’s speaking, you can try using the bot-tts-text message type. type : 'bot-transcription' data : text : string The transcribed text from the bot, typically aggregated at a per-sentence level. ​ Client-Server Messaging ​ server-message 🤖 An arbitrary message sent from the server to the client. This can be used for custom interactions or commands. This message may be coupled with the client-message message type to handle responses from the client. type : 'server-message' data : any The data can be any JSON-serializable object, formatted according to your own specifications. ​ client-message 🏄 An arbitrary message sent from the client to the server. This can be used for custom interactions or commands. This message may be coupled with the server-response message type to handle responses from the server. type : 'client-message' data : t : string d : unknown (optional) The data payload should contain a t field indicating the type of message and an optional d field containing any custom, corresponding data needed for the message. ​ server-response 🤖 An message sent from the server to the client in response to a client-message . IMPORTANT : The id should match the id of the original client-message to correlate the response with the request. type : 'client-message' data : t : string d : unknown (optional) The data payload should contain a t field indicating the type of message and an optional d field containing any custom, corresponding data needed for the message. ​ error-response 🤖 Error response to a specific client message. IMPORTANT : The id should match the id of the original client-message to correlate the response with the request. type : 'error-response' data : error : string ​ Advanced LLM Interactions ​ append-to-context 🏄 A message sent from the client to the server to append data to the context of the current llm conversation. This is useful for providing text-based content for the user or augmenting the context for the assistant. type : 'append-to-context' data : role : "user" | "assistant" The role the context should be appended to. Currently only supports "user" and "assistant" . content : unknown The content to append to the context. This can be any data structure the llm understand. run_immediately : boolean (optional) Indicates whether the context should be run immediately after appending. Defaults to false . If set to false , the context will be appended but not executed until the next llm run. ​ llm-function-call 🤖 A function call request from the LLM, sent from the bot to the client. Note that for most cases, an LLM function call will be handled completely server-side. However, in the event that the call requires input from the client or the client needs to be aware of the function call, this message/response schema is required. type : 'llm-function-call' data : function_name : string Name of the function to be called. tool_call_id : string Unique identifier for this function call. args : Record<string, unknown> Arguments to be passed to the function. ​ llm-function-call-result 🏄 The result of the function call requested by the LLM, returned from the client. type : 'llm-function-call-result' data : function_name : string Name of the called function. tool_call_id : string Identifier matching the original function call. args : Record<string, unknown> Arguments that were passed to the function. result : Record<string, unknown> | string The result returned by the function. ​ bot-llm-search-response 🤖 Search results from the LLM’s knowledge base. Currently, Google Gemini is the only LLM that supports built-in search. However, we expect other LLMs to follow suite, which is why this message type is defined as part of the RTVI standard. As more LLMs add support for this feature, the format of this message type may evolve to accommodate discrepancies. type : 'bot-llm-search-response' data : search_result : string (optional) Raw search result text. rendered_content : string (optional) Formatted version of the search results. origins : Array<Origin Object> Source information and confidence scores for search results. The Origin Object follows this structure: Copy Ask AI { "site_uri" : string (optional) , "site_title" : string (optional) , "results" : Array< { "text" : string , "confidence" : number [] } > } Example: Copy Ask AI "id" : undefined "label" : "rtvi-ai" "type" : "bot-llm-search-response" "data" : { "origins" : [ { "results" : [ { "confidence" : [ 0.9881149530410768 ], "text" : "* Juneteenth: A Freedom Celebration is scheduled for June 18th from 12:00 pm to 2:00 pm." }, { "confidence" : [ 0.9692034721374512 ], "ext" : "* A Juneteenth celebration at Fort Negley Park will take place on June 19th from 5:00 pm to 9:30 pm." } ], "site_title" : "vanderbilt.edu" , "site_uri" : "https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQHwif83VK9KAzrbMSGSBsKwL8vWfSfC9pgEWYKmStHyqiRoV1oe8j1S0nbwRg_iWgqAr9wUkiegu3ATC8Ll-cuE-vpzwElRHiJ2KgRYcqnOQMoOeokVpWqi" }, { "results" : [ { "confidence" : [ 0.6554043292999268 ], "text" : "In addition to these events, Vanderbilt University is a large research institution with ongoing activities across many fields." } ], "site_title" : "wikipedia.org" , "site_uri" : "https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQESbF-ijx78QbaglrhflHCUWdPTD4M6tYOQigW5hgsHNctRlAHu9ktfPmJx7DfoP5QicE0y-OQY1cRl9w4Id0btiFgLYSKIm2-SPtOHXeNrAlgA7mBnclaGrD7rgnLIbrjl8DgUEJrrvT0CKzuo" }], "rendered_content" : "<style> \n .container ... </div> \n </div> \n " , "search_result" : "Several events are happening at Vanderbilt University: \n\n * Juneteenth: A Freedom Celebration is scheduled for June 18th from 12:00 pm to 2:00 pm. \n * A Juneteenth celebration at Fort Negley Park will take place on June 19th from 5:00 pm to 9:30 pm. \n\n In addition to these events, Vanderbilt University is a large research institution with ongoing activities across many fields. For the most recent news, you should check Vanderbilt's official news website. \n " } ​ Service-Specific Insights ​ bot-llm-started 🤖 Indicates LLM processing has begun type : bot-llm-started data : None ​ bot-llm-stopped 🤖 Indicates LLM processing has completed type : bot-llm-stopped data : None ​ user-llm-text 🤖 Aggregated user input text that is sent to the LLM. type : 'user-llm-text' data : text : string The user’s input text to be processed by the LLM. ​ bot-llm-text 🤖 Individual tokens streamed from the LLM as they are generated. type : 'bot-llm-text' data : text : string The token text from the LLM. ​ bot-tts-started 🤖 Indicates text-to-speech (TTS) processing has begun. type : 'bot-tts-started' data : None ​ bot-tts-stopped 🤖 Indicates text-to-speech (TTS) processing has completed. type : 'bot-tts-stopped' data : None ​ bot-tts-text 🤖 The per-token text output of the text-to-speech (TTS) service (what the TTS actually says). type : 'bot-tts-text' data : text : string The text representation of the generated bot speech. ​ Metrics and Monitoring ​ metrics 🤖 Performance metrics for various processing stages and services. Each message will contain entries for one or more of the metrics types: processing , ttfb , characters . type : 'metrics' data : processing : [See Below] (optional) Processing time metrics. ttfb : [See Below] (optional) Time to first byte metrics. characters : [See Below] (optional) Character processing metrics. For each metric type, the data structure is an array of objects with the following structure: processor : string The name of the processor or service that generated the metric. value : number The value of the metric, typically in milliseconds or character count. model : string (optional) The model of the service that generated the metric, if applicable. Example: Copy Ask AI { "type" : "metrics" , "data" : { "processing" : [ { "model" : "eleven_flash_v2_5" , "processor" : "ElevenLabsTTSService#0" , "value" : 0.0005140304565429688 } ], "ttfb" : [ { "model" : "eleven_flash_v2_5" , "processor" : "ElevenLabsTTSService#0" , "value" : 0.1573178768157959 } ], "characters" : [ { "model" : "eleven_flash_v2_5" , "processor" : "ElevenLabsTTSService#0" , "value" : 43 } ] } } Client SDKs RTVIClient Migration Guide On this page Key Features Terms RTVI Message Format RTVI Message Types Connection Management client-ready 🏄 bot-ready 🤖 disconnect-bot 🏄 error 🤖 Transcription user-started-speaking 🤖 user-stopped-speaking 🤖 bot-started-speaking 🤖 bot-stopped-speaking 🤖 user-transcription 🤖 bot-transcription 🤖 Client-Server Messaging server-message 🤖 client-message 🏄 server-response 🤖 error-response 🤖 Advanced LLM Interactions append-to-context 🏄 llm-function-call 🤖 llm-function-call-result 🏄 bot-llm-search-response 🤖 Service-Specific Insights bot-llm-started 🤖 bot-llm-stopped 🤖 user-llm-text 🤖 bot-llm-text 🤖 bot-tts-started 🤖 bot-tts-stopped 🤖 bot-tts-text 🤖 Metrics and Monitoring metrics 🤖 Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/client_rtvi-standard_77ebb9a6.txt b/client_rtvi-standard_77ebb9a6.txt
new file mode 100644
index 0000000000000000000000000000000000000000..10690723cb91cc6a92d98bb293530882ad48d682
--- /dev/null
+++ b/client_rtvi-standard_77ebb9a6.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/client/rtvi-standard#server-message-%F0%9F%A4%96
+Title: The RTVI Standard - Pipecat
+==================================================
+
+The RTVI Standard - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation The RTVI Standard Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Client SDKs The RTVI Standard RTVIClient Migration Guide Javascript SDK SDK Introduction API Reference Transport packages React SDK SDK Introduction API Reference React Native SDK SDK Introduction API Reference iOS SDK SDK Introduction API Reference Transport packages Android SDK SDK Introduction API Reference Transport packages C++ SDK SDK Introduction Daily WebRTC Transport The RTVI (Real-Time Voice and Video Inference) standard defines a set of message types and structures sent between clients and servers. It is designed to facilitate real-time interactions between clients and AI applications that require voice, video, and text communication. It provides a consistent framework for building applications that can communicate with AI models and the backends running those models in real-time. This page documents version 1.0 of the RTVI standard, released in June 2025. ​ Key Features Connection Management RTVI provides a flexible connection model that allows clients to connect to AI services and coordinate state. Transcriptions The standard includes built-in support for real-time transcription of audio streams. Client-Server Messaging The standard defines a messaging protocol for sending and receiving messages between clients and servers, allowing for efficient communication of requests and responses. Advanced LLM Interactions The standard supports advanced interactions with large language models (LLMs), including context management, function call handline, and search results. Service-Specific Insights RTVI supports events to provide insight into the input/output and state for typical services that exist in speech-to-speech workflows. Metrics and Monitoring RTVI provides mechanisms for collecting metrics and monitoring the performance of server-side services. ​ Terms Client : The front-end application or user interface that interacts with the RTVI server. Server : The backend-end service that runs the AI framework and processes requests from the client. User : The end user interacting with the client application. Bot : The AI interacting with the user, technically an amalgamation of a large language model (LLM) and a text-to-speech (TTS) service. ​ RTVI Message Format The messages defined as part of the RTVI protocol adhere to the following format: Copy Ask AI { "id" : string , "label" : "rtvi-ai" , "type" : string , "data" : unknown } ​ id string A unique identifier for the message, used to correlate requests and responses. ​ label string default: "rtvi-ai" required A label that identifies this message as an RTVI message. This field is required and should always be set to 'rtvi-ai' . ​ type string required The type of message being sent. This field is required and should be set to one of the predefined RTVI message types listed below. ​ data unknown The payload of the message, which can be any data structure relevant to the message type. ​ RTVI Message Types Following the above format, this section describes the various message types defined by the RTVI standard. Each message type has a specific purpose and structure, allowing for clear communication between clients and servers. Each message type below includes either a 🤖 or 🏄 emoji to denote whether the message is sent from the bot (🤖) or client (🏄). ​ Connection Management ​ client-ready 🏄 Indicates that the client is ready to receive messages and interact with the server. Typically sent after the transport media channels have connected. type : 'client-ready' data : version : string The version of the RTVI standard being used. This is useful for ensuring compatibility between client and server implementations. about : AboutClient Object An object containing information about the client, such as its rtvi-version, client library, and any other relevant metadata. The AboutClient object follows this structure: Show AboutClient ​ library string required ​ library_version string ​ platform string ​ platform_version string ​ platform_details any Any platform-specific details that may be relevant to the server. This could include information about the browser, operating system, or any other environment-specific data needed by the server. This field is optional and open-ended, so please be mindful of the data you include here and any security concerns that may arise from exposing sensitive or personal-identifiable information. ​ bot-ready 🤖 Indicates that the bot is ready to receive messages and interact with the client. Typically send after the transport media channels have connected. type : 'bot-ready' data : version : string The version of the RTVI standard being used. This is useful for ensuring compatibility between client and server implementations. about : any (Optional) An object containing information about the server or bot. It’s structure and value are both undefined by default. This provides flexibility to include any relevant metadata your client may need to know about the server at connection time, without any built-in security concerns. Please be mindful of the data you include here and any security concerns that may arise from exposing sensitive information. ​ disconnect-bot 🏄 Indicates that the client wishes to disconnect from the bot. Typically used when the client is shutting down or no longer needs to interact with the bot. Note: Disconnets should happen automatically when either the client or bot disconnects from the transport, so this message is intended for the case where a client may want to remain connected to the transport but no longer wishes to interact with the bot. type : 'disconnect-bot' data : undefined ​ error 🤖 Indicates an error occurred during bot initialization or runtime. type : 'error' data : message : string Description of the error. fatal : boolean Indicates if the error is fatal to the session. ​ Transcription ​ user-started-speaking 🤖 Emitted when the user begins speaking type : 'user-started-speaking' data : None ​ user-stopped-speaking 🤖 Emitted when the user stops speaking type : 'user-stopped-speaking' data : None ​ bot-started-speaking 🤖 Emitted when the bot begins speaking type : 'bot-started-speaking' data : None ​ bot-stopped-speaking 🤖 Emitted when the bot stops speaking type : 'bot-stopped-speaking' data : None ​ user-transcription 🤖 Real-time transcription of user speech, including both partial and final results. type : 'user-transcription' data : text : string The transcribed text of the user. final : boolean Indicates if this is a final transcription or a partial result. timestamp : string The timestamp when the transcription was generated. user_id : string Identifier for the user who spoke. ​ bot-transcription 🤖 Transcription of the bot’s speech. Note: This protocol currently does not match the user transcription format to support real-time timestamping for bot transcriptions. Rather, the event is typically sent for each sentence of the bot’s response. This difference is currently due to limitations in TTS services which mostly do not support (or support well), accurate timing information. If/when this changes, this protocol may be updated to include the necessary timing information. For now, if you want to attempt real-time transcription to match your bot’s speaking, you can try using the bot-tts-text message type. type : 'bot-transcription' data : text : string The transcribed text from the bot, typically aggregated at a per-sentence level. ​ Client-Server Messaging ​ server-message 🤖 An arbitrary message sent from the server to the client. This can be used for custom interactions or commands. This message may be coupled with the client-message message type to handle responses from the client. type : 'server-message' data : any The data can be any JSON-serializable object, formatted according to your own specifications. ​ client-message 🏄 An arbitrary message sent from the client to the server. This can be used for custom interactions or commands. This message may be coupled with the server-response message type to handle responses from the server. type : 'client-message' data : t : string d : unknown (optional) The data payload should contain a t field indicating the type of message and an optional d field containing any custom, corresponding data needed for the message. ​ server-response 🤖 An message sent from the server to the client in response to a client-message . IMPORTANT : The id should match the id of the original client-message to correlate the response with the request. type : 'client-message' data : t : string d : unknown (optional) The data payload should contain a t field indicating the type of message and an optional d field containing any custom, corresponding data needed for the message. ​ error-response 🤖 Error response to a specific client message. IMPORTANT : The id should match the id of the original client-message to correlate the response with the request. type : 'error-response' data : error : string ​ Advanced LLM Interactions ​ append-to-context 🏄 A message sent from the client to the server to append data to the context of the current llm conversation. This is useful for providing text-based content for the user or augmenting the context for the assistant. type : 'append-to-context' data : role : "user" | "assistant" The role the context should be appended to. Currently only supports "user" and "assistant" . content : unknown The content to append to the context. This can be any data structure the llm understand. run_immediately : boolean (optional) Indicates whether the context should be run immediately after appending. Defaults to false . If set to false , the context will be appended but not executed until the next llm run. ​ llm-function-call 🤖 A function call request from the LLM, sent from the bot to the client. Note that for most cases, an LLM function call will be handled completely server-side. However, in the event that the call requires input from the client or the client needs to be aware of the function call, this message/response schema is required. type : 'llm-function-call' data : function_name : string Name of the function to be called. tool_call_id : string Unique identifier for this function call. args : Record<string, unknown> Arguments to be passed to the function. ​ llm-function-call-result 🏄 The result of the function call requested by the LLM, returned from the client. type : 'llm-function-call-result' data : function_name : string Name of the called function. tool_call_id : string Identifier matching the original function call. args : Record<string, unknown> Arguments that were passed to the function. result : Record<string, unknown> | string The result returned by the function. ​ bot-llm-search-response 🤖 Search results from the LLM’s knowledge base. Currently, Google Gemini is the only LLM that supports built-in search. However, we expect other LLMs to follow suite, which is why this message type is defined as part of the RTVI standard. As more LLMs add support for this feature, the format of this message type may evolve to accommodate discrepancies. type : 'bot-llm-search-response' data : search_result : string (optional) Raw search result text. rendered_content : string (optional) Formatted version of the search results. origins : Array<Origin Object> Source information and confidence scores for search results. The Origin Object follows this structure: Copy Ask AI { "site_uri" : string (optional) , "site_title" : string (optional) , "results" : Array< { "text" : string , "confidence" : number [] } > } Example: Copy Ask AI "id" : undefined "label" : "rtvi-ai" "type" : "bot-llm-search-response" "data" : { "origins" : [ { "results" : [ { "confidence" : [ 0.9881149530410768 ], "text" : "* Juneteenth: A Freedom Celebration is scheduled for June 18th from 12:00 pm to 2:00 pm." }, { "confidence" : [ 0.9692034721374512 ], "ext" : "* A Juneteenth celebration at Fort Negley Park will take place on June 19th from 5:00 pm to 9:30 pm." } ], "site_title" : "vanderbilt.edu" , "site_uri" : "https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQHwif83VK9KAzrbMSGSBsKwL8vWfSfC9pgEWYKmStHyqiRoV1oe8j1S0nbwRg_iWgqAr9wUkiegu3ATC8Ll-cuE-vpzwElRHiJ2KgRYcqnOQMoOeokVpWqi" }, { "results" : [ { "confidence" : [ 0.6554043292999268 ], "text" : "In addition to these events, Vanderbilt University is a large research institution with ongoing activities across many fields." } ], "site_title" : "wikipedia.org" , "site_uri" : "https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQESbF-ijx78QbaglrhflHCUWdPTD4M6tYOQigW5hgsHNctRlAHu9ktfPmJx7DfoP5QicE0y-OQY1cRl9w4Id0btiFgLYSKIm2-SPtOHXeNrAlgA7mBnclaGrD7rgnLIbrjl8DgUEJrrvT0CKzuo" }], "rendered_content" : "<style> \n .container ... </div> \n </div> \n " , "search_result" : "Several events are happening at Vanderbilt University: \n\n * Juneteenth: A Freedom Celebration is scheduled for June 18th from 12:00 pm to 2:00 pm. \n * A Juneteenth celebration at Fort Negley Park will take place on June 19th from 5:00 pm to 9:30 pm. \n\n In addition to these events, Vanderbilt University is a large research institution with ongoing activities across many fields. For the most recent news, you should check Vanderbilt's official news website. \n " } ​ Service-Specific Insights ​ bot-llm-started 🤖 Indicates LLM processing has begun type : bot-llm-started data : None ​ bot-llm-stopped 🤖 Indicates LLM processing has completed type : bot-llm-stopped data : None ​ user-llm-text 🤖 Aggregated user input text that is sent to the LLM. type : 'user-llm-text' data : text : string The user’s input text to be processed by the LLM. ​ bot-llm-text 🤖 Individual tokens streamed from the LLM as they are generated. type : 'bot-llm-text' data : text : string The token text from the LLM. ​ bot-tts-started 🤖 Indicates text-to-speech (TTS) processing has begun. type : 'bot-tts-started' data : None ​ bot-tts-stopped 🤖 Indicates text-to-speech (TTS) processing has completed. type : 'bot-tts-stopped' data : None ​ bot-tts-text 🤖 The per-token text output of the text-to-speech (TTS) service (what the TTS actually says). type : 'bot-tts-text' data : text : string The text representation of the generated bot speech. ​ Metrics and Monitoring ​ metrics 🤖 Performance metrics for various processing stages and services. Each message will contain entries for one or more of the metrics types: processing , ttfb , characters . type : 'metrics' data : processing : [See Below] (optional) Processing time metrics. ttfb : [See Below] (optional) Time to first byte metrics. characters : [See Below] (optional) Character processing metrics. For each metric type, the data structure is an array of objects with the following structure: processor : string The name of the processor or service that generated the metric. value : number The value of the metric, typically in milliseconds or character count. model : string (optional) The model of the service that generated the metric, if applicable. Example: Copy Ask AI { "type" : "metrics" , "data" : { "processing" : [ { "model" : "eleven_flash_v2_5" , "processor" : "ElevenLabsTTSService#0" , "value" : 0.0005140304565429688 } ], "ttfb" : [ { "model" : "eleven_flash_v2_5" , "processor" : "ElevenLabsTTSService#0" , "value" : 0.1573178768157959 } ], "characters" : [ { "model" : "eleven_flash_v2_5" , "processor" : "ElevenLabsTTSService#0" , "value" : 43 } ] } } Client SDKs RTVIClient Migration Guide On this page Key Features Terms RTVI Message Format RTVI Message Types Connection Management client-ready 🏄 bot-ready 🤖 disconnect-bot 🏄 error 🤖 Transcription user-started-speaking 🤖 user-stopped-speaking 🤖 bot-started-speaking 🤖 bot-stopped-speaking 🤖 user-transcription 🤖 bot-transcription 🤖 Client-Server Messaging server-message 🤖 client-message 🏄 server-response 🤖 error-response 🤖 Advanced LLM Interactions append-to-context 🏄 llm-function-call 🤖 llm-function-call-result 🏄 bot-llm-search-response 🤖 Service-Specific Insights bot-llm-started 🤖 bot-llm-stopped 🤖 user-llm-text 🤖 bot-llm-text 🤖 bot-tts-started 🤖 bot-tts-stopped 🤖 bot-tts-text 🤖 Metrics and Monitoring metrics 🤖 Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/client_rtvi-standard_88d2e3f4.txt b/client_rtvi-standard_88d2e3f4.txt
new file mode 100644
index 0000000000000000000000000000000000000000..f7775c873a62ed088574af0669eb1fb0f842a074
--- /dev/null
+++ b/client_rtvi-standard_88d2e3f4.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/client/rtvi-standard#param-platform
+Title: The RTVI Standard - Pipecat
+==================================================
+
+The RTVI Standard - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation The RTVI Standard Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Client SDKs The RTVI Standard RTVIClient Migration Guide Javascript SDK SDK Introduction API Reference Transport packages React SDK SDK Introduction API Reference React Native SDK SDK Introduction API Reference iOS SDK SDK Introduction API Reference Transport packages Android SDK SDK Introduction API Reference Transport packages C++ SDK SDK Introduction Daily WebRTC Transport The RTVI (Real-Time Voice and Video Inference) standard defines a set of message types and structures sent between clients and servers. It is designed to facilitate real-time interactions between clients and AI applications that require voice, video, and text communication. It provides a consistent framework for building applications that can communicate with AI models and the backends running those models in real-time. This page documents version 1.0 of the RTVI standard, released in June 2025. ​ Key Features Connection Management RTVI provides a flexible connection model that allows clients to connect to AI services and coordinate state. Transcriptions The standard includes built-in support for real-time transcription of audio streams. Client-Server Messaging The standard defines a messaging protocol for sending and receiving messages between clients and servers, allowing for efficient communication of requests and responses. Advanced LLM Interactions The standard supports advanced interactions with large language models (LLMs), including context management, function call handline, and search results. Service-Specific Insights RTVI supports events to provide insight into the input/output and state for typical services that exist in speech-to-speech workflows. Metrics and Monitoring RTVI provides mechanisms for collecting metrics and monitoring the performance of server-side services. ​ Terms Client : The front-end application or user interface that interacts with the RTVI server. Server : The backend-end service that runs the AI framework and processes requests from the client. User : The end user interacting with the client application. Bot : The AI interacting with the user, technically an amalgamation of a large language model (LLM) and a text-to-speech (TTS) service. ​ RTVI Message Format The messages defined as part of the RTVI protocol adhere to the following format: Copy Ask AI { "id" : string , "label" : "rtvi-ai" , "type" : string , "data" : unknown } ​ id string A unique identifier for the message, used to correlate requests and responses. ​ label string default: "rtvi-ai" required A label that identifies this message as an RTVI message. This field is required and should always be set to 'rtvi-ai' . ​ type string required The type of message being sent. This field is required and should be set to one of the predefined RTVI message types listed below. ​ data unknown The payload of the message, which can be any data structure relevant to the message type. ​ RTVI Message Types Following the above format, this section describes the various message types defined by the RTVI standard. Each message type has a specific purpose and structure, allowing for clear communication between clients and servers. Each message type below includes either a 🤖 or 🏄 emoji to denote whether the message is sent from the bot (🤖) or client (🏄). ​ Connection Management ​ client-ready 🏄 Indicates that the client is ready to receive messages and interact with the server. Typically sent after the transport media channels have connected. type : 'client-ready' data : version : string The version of the RTVI standard being used. This is useful for ensuring compatibility between client and server implementations. about : AboutClient Object An object containing information about the client, such as its rtvi-version, client library, and any other relevant metadata. The AboutClient object follows this structure: Show AboutClient ​ library string required ​ library_version string ​ platform string ​ platform_version string ​ platform_details any Any platform-specific details that may be relevant to the server. This could include information about the browser, operating system, or any other environment-specific data needed by the server. This field is optional and open-ended, so please be mindful of the data you include here and any security concerns that may arise from exposing sensitive or personal-identifiable information. ​ bot-ready 🤖 Indicates that the bot is ready to receive messages and interact with the client. Typically send after the transport media channels have connected. type : 'bot-ready' data : version : string The version of the RTVI standard being used. This is useful for ensuring compatibility between client and server implementations. about : any (Optional) An object containing information about the server or bot. It’s structure and value are both undefined by default. This provides flexibility to include any relevant metadata your client may need to know about the server at connection time, without any built-in security concerns. Please be mindful of the data you include here and any security concerns that may arise from exposing sensitive information. ​ disconnect-bot 🏄 Indicates that the client wishes to disconnect from the bot. Typically used when the client is shutting down or no longer needs to interact with the bot. Note: Disconnets should happen automatically when either the client or bot disconnects from the transport, so this message is intended for the case where a client may want to remain connected to the transport but no longer wishes to interact with the bot. type : 'disconnect-bot' data : undefined ​ error 🤖 Indicates an error occurred during bot initialization or runtime. type : 'error' data : message : string Description of the error. fatal : boolean Indicates if the error is fatal to the session. ​ Transcription ​ user-started-speaking 🤖 Emitted when the user begins speaking type : 'user-started-speaking' data : None ​ user-stopped-speaking 🤖 Emitted when the user stops speaking type : 'user-stopped-speaking' data : None ​ bot-started-speaking 🤖 Emitted when the bot begins speaking type : 'bot-started-speaking' data : None ​ bot-stopped-speaking 🤖 Emitted when the bot stops speaking type : 'bot-stopped-speaking' data : None ​ user-transcription 🤖 Real-time transcription of user speech, including both partial and final results. type : 'user-transcription' data : text : string The transcribed text of the user. final : boolean Indicates if this is a final transcription or a partial result. timestamp : string The timestamp when the transcription was generated. user_id : string Identifier for the user who spoke. ​ bot-transcription 🤖 Transcription of the bot’s speech. Note: This protocol currently does not match the user transcription format to support real-time timestamping for bot transcriptions. Rather, the event is typically sent for each sentence of the bot’s response. This difference is currently due to limitations in TTS services which mostly do not support (or support well), accurate timing information. If/when this changes, this protocol may be updated to include the necessary timing information. For now, if you want to attempt real-time transcription to match your bot’s speaking, you can try using the bot-tts-text message type. type : 'bot-transcription' data : text : string The transcribed text from the bot, typically aggregated at a per-sentence level. ​ Client-Server Messaging ​ server-message 🤖 An arbitrary message sent from the server to the client. This can be used for custom interactions or commands. This message may be coupled with the client-message message type to handle responses from the client. type : 'server-message' data : any The data can be any JSON-serializable object, formatted according to your own specifications. ​ client-message 🏄 An arbitrary message sent from the client to the server. This can be used for custom interactions or commands. This message may be coupled with the server-response message type to handle responses from the server. type : 'client-message' data : t : string d : unknown (optional) The data payload should contain a t field indicating the type of message and an optional d field containing any custom, corresponding data needed for the message. ​ server-response 🤖 An message sent from the server to the client in response to a client-message . IMPORTANT : The id should match the id of the original client-message to correlate the response with the request. type : 'client-message' data : t : string d : unknown (optional) The data payload should contain a t field indicating the type of message and an optional d field containing any custom, corresponding data needed for the message. ​ error-response 🤖 Error response to a specific client message. IMPORTANT : The id should match the id of the original client-message to correlate the response with the request. type : 'error-response' data : error : string ​ Advanced LLM Interactions ​ append-to-context 🏄 A message sent from the client to the server to append data to the context of the current llm conversation. This is useful for providing text-based content for the user or augmenting the context for the assistant. type : 'append-to-context' data : role : "user" | "assistant" The role the context should be appended to. Currently only supports "user" and "assistant" . content : unknown The content to append to the context. This can be any data structure the llm understand. run_immediately : boolean (optional) Indicates whether the context should be run immediately after appending. Defaults to false . If set to false , the context will be appended but not executed until the next llm run. ​ llm-function-call 🤖 A function call request from the LLM, sent from the bot to the client. Note that for most cases, an LLM function call will be handled completely server-side. However, in the event that the call requires input from the client or the client needs to be aware of the function call, this message/response schema is required. type : 'llm-function-call' data : function_name : string Name of the function to be called. tool_call_id : string Unique identifier for this function call. args : Record<string, unknown> Arguments to be passed to the function. ​ llm-function-call-result 🏄 The result of the function call requested by the LLM, returned from the client. type : 'llm-function-call-result' data : function_name : string Name of the called function. tool_call_id : string Identifier matching the original function call. args : Record<string, unknown> Arguments that were passed to the function. result : Record<string, unknown> | string The result returned by the function. ​ bot-llm-search-response 🤖 Search results from the LLM’s knowledge base. Currently, Google Gemini is the only LLM that supports built-in search. However, we expect other LLMs to follow suite, which is why this message type is defined as part of the RTVI standard. As more LLMs add support for this feature, the format of this message type may evolve to accommodate discrepancies. type : 'bot-llm-search-response' data : search_result : string (optional) Raw search result text. rendered_content : string (optional) Formatted version of the search results. origins : Array<Origin Object> Source information and confidence scores for search results. The Origin Object follows this structure: Copy Ask AI { "site_uri" : string (optional) , "site_title" : string (optional) , "results" : Array< { "text" : string , "confidence" : number [] } > } Example: Copy Ask AI "id" : undefined "label" : "rtvi-ai" "type" : "bot-llm-search-response" "data" : { "origins" : [ { "results" : [ { "confidence" : [ 0.9881149530410768 ], "text" : "* Juneteenth: A Freedom Celebration is scheduled for June 18th from 12:00 pm to 2:00 pm." }, { "confidence" : [ 0.9692034721374512 ], "ext" : "* A Juneteenth celebration at Fort Negley Park will take place on June 19th from 5:00 pm to 9:30 pm." } ], "site_title" : "vanderbilt.edu" , "site_uri" : "https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQHwif83VK9KAzrbMSGSBsKwL8vWfSfC9pgEWYKmStHyqiRoV1oe8j1S0nbwRg_iWgqAr9wUkiegu3ATC8Ll-cuE-vpzwElRHiJ2KgRYcqnOQMoOeokVpWqi" }, { "results" : [ { "confidence" : [ 0.6554043292999268 ], "text" : "In addition to these events, Vanderbilt University is a large research institution with ongoing activities across many fields." } ], "site_title" : "wikipedia.org" , "site_uri" : "https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQESbF-ijx78QbaglrhflHCUWdPTD4M6tYOQigW5hgsHNctRlAHu9ktfPmJx7DfoP5QicE0y-OQY1cRl9w4Id0btiFgLYSKIm2-SPtOHXeNrAlgA7mBnclaGrD7rgnLIbrjl8DgUEJrrvT0CKzuo" }], "rendered_content" : "<style> \n .container ... </div> \n </div> \n " , "search_result" : "Several events are happening at Vanderbilt University: \n\n * Juneteenth: A Freedom Celebration is scheduled for June 18th from 12:00 pm to 2:00 pm. \n * A Juneteenth celebration at Fort Negley Park will take place on June 19th from 5:00 pm to 9:30 pm. \n\n In addition to these events, Vanderbilt University is a large research institution with ongoing activities across many fields. For the most recent news, you should check Vanderbilt's official news website. \n " } ​ Service-Specific Insights ​ bot-llm-started 🤖 Indicates LLM processing has begun type : bot-llm-started data : None ​ bot-llm-stopped 🤖 Indicates LLM processing has completed type : bot-llm-stopped data : None ​ user-llm-text 🤖 Aggregated user input text that is sent to the LLM. type : 'user-llm-text' data : text : string The user’s input text to be processed by the LLM. ​ bot-llm-text 🤖 Individual tokens streamed from the LLM as they are generated. type : 'bot-llm-text' data : text : string The token text from the LLM. ​ bot-tts-started 🤖 Indicates text-to-speech (TTS) processing has begun. type : 'bot-tts-started' data : None ​ bot-tts-stopped 🤖 Indicates text-to-speech (TTS) processing has completed. type : 'bot-tts-stopped' data : None ​ bot-tts-text 🤖 The per-token text output of the text-to-speech (TTS) service (what the TTS actually says). type : 'bot-tts-text' data : text : string The text representation of the generated bot speech. ​ Metrics and Monitoring ​ metrics 🤖 Performance metrics for various processing stages and services. Each message will contain entries for one or more of the metrics types: processing , ttfb , characters . type : 'metrics' data : processing : [See Below] (optional) Processing time metrics. ttfb : [See Below] (optional) Time to first byte metrics. characters : [See Below] (optional) Character processing metrics. For each metric type, the data structure is an array of objects with the following structure: processor : string The name of the processor or service that generated the metric. value : number The value of the metric, typically in milliseconds or character count. model : string (optional) The model of the service that generated the metric, if applicable. Example: Copy Ask AI { "type" : "metrics" , "data" : { "processing" : [ { "model" : "eleven_flash_v2_5" , "processor" : "ElevenLabsTTSService#0" , "value" : 0.0005140304565429688 } ], "ttfb" : [ { "model" : "eleven_flash_v2_5" , "processor" : "ElevenLabsTTSService#0" , "value" : 0.1573178768157959 } ], "characters" : [ { "model" : "eleven_flash_v2_5" , "processor" : "ElevenLabsTTSService#0" , "value" : 43 } ] } } Client SDKs RTVIClient Migration Guide On this page Key Features Terms RTVI Message Format RTVI Message Types Connection Management client-ready 🏄 bot-ready 🤖 disconnect-bot 🏄 error 🤖 Transcription user-started-speaking 🤖 user-stopped-speaking 🤖 bot-started-speaking 🤖 bot-stopped-speaking 🤖 user-transcription 🤖 bot-transcription 🤖 Client-Server Messaging server-message 🤖 client-message 🏄 server-response 🤖 error-response 🤖 Advanced LLM Interactions append-to-context 🏄 llm-function-call 🤖 llm-function-call-result 🏄 bot-llm-search-response 🤖 Service-Specific Insights bot-llm-started 🤖 bot-llm-stopped 🤖 user-llm-text 🤖 bot-llm-text 🤖 bot-tts-started 🤖 bot-tts-stopped 🤖 bot-tts-text 🤖 Metrics and Monitoring metrics 🤖 Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/client_rtvi-standard_a8d1a6ce.txt b/client_rtvi-standard_a8d1a6ce.txt
new file mode 100644
index 0000000000000000000000000000000000000000..2bce87bcbe87a619f371971864a4da893a636058
--- /dev/null
+++ b/client_rtvi-standard_a8d1a6ce.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/client/rtvi-standard#metrics-%F0%9F%A4%96
+Title: The RTVI Standard - Pipecat
+==================================================
+
+The RTVI Standard - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation The RTVI Standard Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Client SDKs The RTVI Standard RTVIClient Migration Guide Javascript SDK SDK Introduction API Reference Transport packages React SDK SDK Introduction API Reference React Native SDK SDK Introduction API Reference iOS SDK SDK Introduction API Reference Transport packages Android SDK SDK Introduction API Reference Transport packages C++ SDK SDK Introduction Daily WebRTC Transport The RTVI (Real-Time Voice and Video Inference) standard defines a set of message types and structures sent between clients and servers. It is designed to facilitate real-time interactions between clients and AI applications that require voice, video, and text communication. It provides a consistent framework for building applications that can communicate with AI models and the backends running those models in real-time. This page documents version 1.0 of the RTVI standard, released in June 2025. ​ Key Features Connection Management RTVI provides a flexible connection model that allows clients to connect to AI services and coordinate state. Transcriptions The standard includes built-in support for real-time transcription of audio streams. Client-Server Messaging The standard defines a messaging protocol for sending and receiving messages between clients and servers, allowing for efficient communication of requests and responses. Advanced LLM Interactions The standard supports advanced interactions with large language models (LLMs), including context management, function call handline, and search results. Service-Specific Insights RTVI supports events to provide insight into the input/output and state for typical services that exist in speech-to-speech workflows. Metrics and Monitoring RTVI provides mechanisms for collecting metrics and monitoring the performance of server-side services. ​ Terms Client : The front-end application or user interface that interacts with the RTVI server. Server : The backend-end service that runs the AI framework and processes requests from the client. User : The end user interacting with the client application. Bot : The AI interacting with the user, technically an amalgamation of a large language model (LLM) and a text-to-speech (TTS) service. ​ RTVI Message Format The messages defined as part of the RTVI protocol adhere to the following format: Copy Ask AI { "id" : string , "label" : "rtvi-ai" , "type" : string , "data" : unknown } ​ id string A unique identifier for the message, used to correlate requests and responses. ​ label string default: "rtvi-ai" required A label that identifies this message as an RTVI message. This field is required and should always be set to 'rtvi-ai' . ​ type string required The type of message being sent. This field is required and should be set to one of the predefined RTVI message types listed below. ​ data unknown The payload of the message, which can be any data structure relevant to the message type. ​ RTVI Message Types Following the above format, this section describes the various message types defined by the RTVI standard. Each message type has a specific purpose and structure, allowing for clear communication between clients and servers. Each message type below includes either a 🤖 or 🏄 emoji to denote whether the message is sent from the bot (🤖) or client (🏄). ​ Connection Management ​ client-ready 🏄 Indicates that the client is ready to receive messages and interact with the server. Typically sent after the transport media channels have connected. type : 'client-ready' data : version : string The version of the RTVI standard being used. This is useful for ensuring compatibility between client and server implementations. about : AboutClient Object An object containing information about the client, such as its rtvi-version, client library, and any other relevant metadata. The AboutClient object follows this structure: Show AboutClient ​ library string required ​ library_version string ​ platform string ​ platform_version string ​ platform_details any Any platform-specific details that may be relevant to the server. This could include information about the browser, operating system, or any other environment-specific data needed by the server. This field is optional and open-ended, so please be mindful of the data you include here and any security concerns that may arise from exposing sensitive or personal-identifiable information. ​ bot-ready 🤖 Indicates that the bot is ready to receive messages and interact with the client. Typically send after the transport media channels have connected. type : 'bot-ready' data : version : string The version of the RTVI standard being used. This is useful for ensuring compatibility between client and server implementations. about : any (Optional) An object containing information about the server or bot. It’s structure and value are both undefined by default. This provides flexibility to include any relevant metadata your client may need to know about the server at connection time, without any built-in security concerns. Please be mindful of the data you include here and any security concerns that may arise from exposing sensitive information. ​ disconnect-bot 🏄 Indicates that the client wishes to disconnect from the bot. Typically used when the client is shutting down or no longer needs to interact with the bot. Note: Disconnets should happen automatically when either the client or bot disconnects from the transport, so this message is intended for the case where a client may want to remain connected to the transport but no longer wishes to interact with the bot. type : 'disconnect-bot' data : undefined ​ error 🤖 Indicates an error occurred during bot initialization or runtime. type : 'error' data : message : string Description of the error. fatal : boolean Indicates if the error is fatal to the session. ​ Transcription ​ user-started-speaking 🤖 Emitted when the user begins speaking type : 'user-started-speaking' data : None ​ user-stopped-speaking 🤖 Emitted when the user stops speaking type : 'user-stopped-speaking' data : None ​ bot-started-speaking 🤖 Emitted when the bot begins speaking type : 'bot-started-speaking' data : None ​ bot-stopped-speaking 🤖 Emitted when the bot stops speaking type : 'bot-stopped-speaking' data : None ​ user-transcription 🤖 Real-time transcription of user speech, including both partial and final results. type : 'user-transcription' data : text : string The transcribed text of the user. final : boolean Indicates if this is a final transcription or a partial result. timestamp : string The timestamp when the transcription was generated. user_id : string Identifier for the user who spoke. ​ bot-transcription 🤖 Transcription of the bot’s speech. Note: This protocol currently does not match the user transcription format to support real-time timestamping for bot transcriptions. Rather, the event is typically sent for each sentence of the bot’s response. This difference is currently due to limitations in TTS services which mostly do not support (or support well), accurate timing information. If/when this changes, this protocol may be updated to include the necessary timing information. For now, if you want to attempt real-time transcription to match your bot’s speaking, you can try using the bot-tts-text message type. type : 'bot-transcription' data : text : string The transcribed text from the bot, typically aggregated at a per-sentence level. ​ Client-Server Messaging ​ server-message 🤖 An arbitrary message sent from the server to the client. This can be used for custom interactions or commands. This message may be coupled with the client-message message type to handle responses from the client. type : 'server-message' data : any The data can be any JSON-serializable object, formatted according to your own specifications. ​ client-message 🏄 An arbitrary message sent from the client to the server. This can be used for custom interactions or commands. This message may be coupled with the server-response message type to handle responses from the server. type : 'client-message' data : t : string d : unknown (optional) The data payload should contain a t field indicating the type of message and an optional d field containing any custom, corresponding data needed for the message. ​ server-response 🤖 An message sent from the server to the client in response to a client-message . IMPORTANT : The id should match the id of the original client-message to correlate the response with the request. type : 'client-message' data : t : string d : unknown (optional) The data payload should contain a t field indicating the type of message and an optional d field containing any custom, corresponding data needed for the message. ​ error-response 🤖 Error response to a specific client message. IMPORTANT : The id should match the id of the original client-message to correlate the response with the request. type : 'error-response' data : error : string ​ Advanced LLM Interactions ​ append-to-context 🏄 A message sent from the client to the server to append data to the context of the current llm conversation. This is useful for providing text-based content for the user or augmenting the context for the assistant. type : 'append-to-context' data : role : "user" | "assistant" The role the context should be appended to. Currently only supports "user" and "assistant" . content : unknown The content to append to the context. This can be any data structure the llm understand. run_immediately : boolean (optional) Indicates whether the context should be run immediately after appending. Defaults to false . If set to false , the context will be appended but not executed until the next llm run. ​ llm-function-call 🤖 A function call request from the LLM, sent from the bot to the client. Note that for most cases, an LLM function call will be handled completely server-side. However, in the event that the call requires input from the client or the client needs to be aware of the function call, this message/response schema is required. type : 'llm-function-call' data : function_name : string Name of the function to be called. tool_call_id : string Unique identifier for this function call. args : Record<string, unknown> Arguments to be passed to the function. ​ llm-function-call-result 🏄 The result of the function call requested by the LLM, returned from the client. type : 'llm-function-call-result' data : function_name : string Name of the called function. tool_call_id : string Identifier matching the original function call. args : Record<string, unknown> Arguments that were passed to the function. result : Record<string, unknown> | string The result returned by the function. ​ bot-llm-search-response 🤖 Search results from the LLM’s knowledge base. Currently, Google Gemini is the only LLM that supports built-in search. However, we expect other LLMs to follow suite, which is why this message type is defined as part of the RTVI standard. As more LLMs add support for this feature, the format of this message type may evolve to accommodate discrepancies. type : 'bot-llm-search-response' data : search_result : string (optional) Raw search result text. rendered_content : string (optional) Formatted version of the search results. origins : Array<Origin Object> Source information and confidence scores for search results. The Origin Object follows this structure: Copy Ask AI { "site_uri" : string (optional) , "site_title" : string (optional) , "results" : Array< { "text" : string , "confidence" : number [] } > } Example: Copy Ask AI "id" : undefined "label" : "rtvi-ai" "type" : "bot-llm-search-response" "data" : { "origins" : [ { "results" : [ { "confidence" : [ 0.9881149530410768 ], "text" : "* Juneteenth: A Freedom Celebration is scheduled for June 18th from 12:00 pm to 2:00 pm." }, { "confidence" : [ 0.9692034721374512 ], "ext" : "* A Juneteenth celebration at Fort Negley Park will take place on June 19th from 5:00 pm to 9:30 pm." } ], "site_title" : "vanderbilt.edu" , "site_uri" : "https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQHwif83VK9KAzrbMSGSBsKwL8vWfSfC9pgEWYKmStHyqiRoV1oe8j1S0nbwRg_iWgqAr9wUkiegu3ATC8Ll-cuE-vpzwElRHiJ2KgRYcqnOQMoOeokVpWqi" }, { "results" : [ { "confidence" : [ 0.6554043292999268 ], "text" : "In addition to these events, Vanderbilt University is a large research institution with ongoing activities across many fields." } ], "site_title" : "wikipedia.org" , "site_uri" : "https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQESbF-ijx78QbaglrhflHCUWdPTD4M6tYOQigW5hgsHNctRlAHu9ktfPmJx7DfoP5QicE0y-OQY1cRl9w4Id0btiFgLYSKIm2-SPtOHXeNrAlgA7mBnclaGrD7rgnLIbrjl8DgUEJrrvT0CKzuo" }], "rendered_content" : "<style> \n .container ... </div> \n </div> \n " , "search_result" : "Several events are happening at Vanderbilt University: \n\n * Juneteenth: A Freedom Celebration is scheduled for June 18th from 12:00 pm to 2:00 pm. \n * A Juneteenth celebration at Fort Negley Park will take place on June 19th from 5:00 pm to 9:30 pm. \n\n In addition to these events, Vanderbilt University is a large research institution with ongoing activities across many fields. For the most recent news, you should check Vanderbilt's official news website. \n " } ​ Service-Specific Insights ​ bot-llm-started 🤖 Indicates LLM processing has begun type : bot-llm-started data : None ​ bot-llm-stopped 🤖 Indicates LLM processing has completed type : bot-llm-stopped data : None ​ user-llm-text 🤖 Aggregated user input text that is sent to the LLM. type : 'user-llm-text' data : text : string The user’s input text to be processed by the LLM. ​ bot-llm-text 🤖 Individual tokens streamed from the LLM as they are generated. type : 'bot-llm-text' data : text : string The token text from the LLM. ​ bot-tts-started 🤖 Indicates text-to-speech (TTS) processing has begun. type : 'bot-tts-started' data : None ​ bot-tts-stopped 🤖 Indicates text-to-speech (TTS) processing has completed. type : 'bot-tts-stopped' data : None ​ bot-tts-text 🤖 The per-token text output of the text-to-speech (TTS) service (what the TTS actually says). type : 'bot-tts-text' data : text : string The text representation of the generated bot speech. ​ Metrics and Monitoring ​ metrics 🤖 Performance metrics for various processing stages and services. Each message will contain entries for one or more of the metrics types: processing , ttfb , characters . type : 'metrics' data : processing : [See Below] (optional) Processing time metrics. ttfb : [See Below] (optional) Time to first byte metrics. characters : [See Below] (optional) Character processing metrics. For each metric type, the data structure is an array of objects with the following structure: processor : string The name of the processor or service that generated the metric. value : number The value of the metric, typically in milliseconds or character count. model : string (optional) The model of the service that generated the metric, if applicable. Example: Copy Ask AI { "type" : "metrics" , "data" : { "processing" : [ { "model" : "eleven_flash_v2_5" , "processor" : "ElevenLabsTTSService#0" , "value" : 0.0005140304565429688 } ], "ttfb" : [ { "model" : "eleven_flash_v2_5" , "processor" : "ElevenLabsTTSService#0" , "value" : 0.1573178768157959 } ], "characters" : [ { "model" : "eleven_flash_v2_5" , "processor" : "ElevenLabsTTSService#0" , "value" : 43 } ] } } Client SDKs RTVIClient Migration Guide On this page Key Features Terms RTVI Message Format RTVI Message Types Connection Management client-ready 🏄 bot-ready 🤖 disconnect-bot 🏄 error 🤖 Transcription user-started-speaking 🤖 user-stopped-speaking 🤖 bot-started-speaking 🤖 bot-stopped-speaking 🤖 user-transcription 🤖 bot-transcription 🤖 Client-Server Messaging server-message 🤖 client-message 🏄 server-response 🤖 error-response 🤖 Advanced LLM Interactions append-to-context 🏄 llm-function-call 🤖 llm-function-call-result 🏄 bot-llm-search-response 🤖 Service-Specific Insights bot-llm-started 🤖 bot-llm-stopped 🤖 user-llm-text 🤖 bot-llm-text 🤖 bot-tts-started 🤖 bot-tts-stopped 🤖 bot-tts-text 🤖 Metrics and Monitoring metrics 🤖 Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/client_rtvi-standard_cf6f0340.txt b/client_rtvi-standard_cf6f0340.txt
new file mode 100644
index 0000000000000000000000000000000000000000..051d448101607b453ff66085a6f1265d7385b59e
--- /dev/null
+++ b/client_rtvi-standard_cf6f0340.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/client/rtvi-standard#bot-ready-%F0%9F%A4%96
+Title: The RTVI Standard - Pipecat
+==================================================
+
+The RTVI Standard - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation The RTVI Standard Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Client SDKs The RTVI Standard RTVIClient Migration Guide Javascript SDK SDK Introduction API Reference Transport packages React SDK SDK Introduction API Reference React Native SDK SDK Introduction API Reference iOS SDK SDK Introduction API Reference Transport packages Android SDK SDK Introduction API Reference Transport packages C++ SDK SDK Introduction Daily WebRTC Transport The RTVI (Real-Time Voice and Video Inference) standard defines a set of message types and structures sent between clients and servers. It is designed to facilitate real-time interactions between clients and AI applications that require voice, video, and text communication. It provides a consistent framework for building applications that can communicate with AI models and the backends running those models in real-time. This page documents version 1.0 of the RTVI standard, released in June 2025. ​ Key Features Connection Management RTVI provides a flexible connection model that allows clients to connect to AI services and coordinate state. Transcriptions The standard includes built-in support for real-time transcription of audio streams. Client-Server Messaging The standard defines a messaging protocol for sending and receiving messages between clients and servers, allowing for efficient communication of requests and responses. Advanced LLM Interactions The standard supports advanced interactions with large language models (LLMs), including context management, function call handline, and search results. Service-Specific Insights RTVI supports events to provide insight into the input/output and state for typical services that exist in speech-to-speech workflows. Metrics and Monitoring RTVI provides mechanisms for collecting metrics and monitoring the performance of server-side services. ​ Terms Client : The front-end application or user interface that interacts with the RTVI server. Server : The backend-end service that runs the AI framework and processes requests from the client. User : The end user interacting with the client application. Bot : The AI interacting with the user, technically an amalgamation of a large language model (LLM) and a text-to-speech (TTS) service. ​ RTVI Message Format The messages defined as part of the RTVI protocol adhere to the following format: Copy Ask AI { "id" : string , "label" : "rtvi-ai" , "type" : string , "data" : unknown } ​ id string A unique identifier for the message, used to correlate requests and responses. ​ label string default: "rtvi-ai" required A label that identifies this message as an RTVI message. This field is required and should always be set to 'rtvi-ai' . ​ type string required The type of message being sent. This field is required and should be set to one of the predefined RTVI message types listed below. ​ data unknown The payload of the message, which can be any data structure relevant to the message type. ​ RTVI Message Types Following the above format, this section describes the various message types defined by the RTVI standard. Each message type has a specific purpose and structure, allowing for clear communication between clients and servers. Each message type below includes either a 🤖 or 🏄 emoji to denote whether the message is sent from the bot (🤖) or client (🏄). ​ Connection Management ​ client-ready 🏄 Indicates that the client is ready to receive messages and interact with the server. Typically sent after the transport media channels have connected. type : 'client-ready' data : version : string The version of the RTVI standard being used. This is useful for ensuring compatibility between client and server implementations. about : AboutClient Object An object containing information about the client, such as its rtvi-version, client library, and any other relevant metadata. The AboutClient object follows this structure: Show AboutClient ​ library string required ​ library_version string ​ platform string ​ platform_version string ​ platform_details any Any platform-specific details that may be relevant to the server. This could include information about the browser, operating system, or any other environment-specific data needed by the server. This field is optional and open-ended, so please be mindful of the data you include here and any security concerns that may arise from exposing sensitive or personal-identifiable information. ​ bot-ready 🤖 Indicates that the bot is ready to receive messages and interact with the client. Typically send after the transport media channels have connected. type : 'bot-ready' data : version : string The version of the RTVI standard being used. This is useful for ensuring compatibility between client and server implementations. about : any (Optional) An object containing information about the server or bot. It’s structure and value are both undefined by default. This provides flexibility to include any relevant metadata your client may need to know about the server at connection time, without any built-in security concerns. Please be mindful of the data you include here and any security concerns that may arise from exposing sensitive information. ​ disconnect-bot 🏄 Indicates that the client wishes to disconnect from the bot. Typically used when the client is shutting down or no longer needs to interact with the bot. Note: Disconnets should happen automatically when either the client or bot disconnects from the transport, so this message is intended for the case where a client may want to remain connected to the transport but no longer wishes to interact with the bot. type : 'disconnect-bot' data : undefined ​ error 🤖 Indicates an error occurred during bot initialization or runtime. type : 'error' data : message : string Description of the error. fatal : boolean Indicates if the error is fatal to the session. ​ Transcription ​ user-started-speaking 🤖 Emitted when the user begins speaking type : 'user-started-speaking' data : None ​ user-stopped-speaking 🤖 Emitted when the user stops speaking type : 'user-stopped-speaking' data : None ​ bot-started-speaking 🤖 Emitted when the bot begins speaking type : 'bot-started-speaking' data : None ​ bot-stopped-speaking 🤖 Emitted when the bot stops speaking type : 'bot-stopped-speaking' data : None ​ user-transcription 🤖 Real-time transcription of user speech, including both partial and final results. type : 'user-transcription' data : text : string The transcribed text of the user. final : boolean Indicates if this is a final transcription or a partial result. timestamp : string The timestamp when the transcription was generated. user_id : string Identifier for the user who spoke. ​ bot-transcription 🤖 Transcription of the bot’s speech. Note: This protocol currently does not match the user transcription format to support real-time timestamping for bot transcriptions. Rather, the event is typically sent for each sentence of the bot’s response. This difference is currently due to limitations in TTS services which mostly do not support (or support well), accurate timing information. If/when this changes, this protocol may be updated to include the necessary timing information. For now, if you want to attempt real-time transcription to match your bot’s speaking, you can try using the bot-tts-text message type. type : 'bot-transcription' data : text : string The transcribed text from the bot, typically aggregated at a per-sentence level. ​ Client-Server Messaging ​ server-message 🤖 An arbitrary message sent from the server to the client. This can be used for custom interactions or commands. This message may be coupled with the client-message message type to handle responses from the client. type : 'server-message' data : any The data can be any JSON-serializable object, formatted according to your own specifications. ​ client-message 🏄 An arbitrary message sent from the client to the server. This can be used for custom interactions or commands. This message may be coupled with the server-response message type to handle responses from the server. type : 'client-message' data : t : string d : unknown (optional) The data payload should contain a t field indicating the type of message and an optional d field containing any custom, corresponding data needed for the message. ​ server-response 🤖 An message sent from the server to the client in response to a client-message . IMPORTANT : The id should match the id of the original client-message to correlate the response with the request. type : 'client-message' data : t : string d : unknown (optional) The data payload should contain a t field indicating the type of message and an optional d field containing any custom, corresponding data needed for the message. ​ error-response 🤖 Error response to a specific client message. IMPORTANT : The id should match the id of the original client-message to correlate the response with the request. type : 'error-response' data : error : string ​ Advanced LLM Interactions ​ append-to-context 🏄 A message sent from the client to the server to append data to the context of the current llm conversation. This is useful for providing text-based content for the user or augmenting the context for the assistant. type : 'append-to-context' data : role : "user" | "assistant" The role the context should be appended to. Currently only supports "user" and "assistant" . content : unknown The content to append to the context. This can be any data structure the llm understand. run_immediately : boolean (optional) Indicates whether the context should be run immediately after appending. Defaults to false . If set to false , the context will be appended but not executed until the next llm run. ​ llm-function-call 🤖 A function call request from the LLM, sent from the bot to the client. Note that for most cases, an LLM function call will be handled completely server-side. However, in the event that the call requires input from the client or the client needs to be aware of the function call, this message/response schema is required. type : 'llm-function-call' data : function_name : string Name of the function to be called. tool_call_id : string Unique identifier for this function call. args : Record<string, unknown> Arguments to be passed to the function. ​ llm-function-call-result 🏄 The result of the function call requested by the LLM, returned from the client. type : 'llm-function-call-result' data : function_name : string Name of the called function. tool_call_id : string Identifier matching the original function call. args : Record<string, unknown> Arguments that were passed to the function. result : Record<string, unknown> | string The result returned by the function. ​ bot-llm-search-response 🤖 Search results from the LLM’s knowledge base. Currently, Google Gemini is the only LLM that supports built-in search. However, we expect other LLMs to follow suite, which is why this message type is defined as part of the RTVI standard. As more LLMs add support for this feature, the format of this message type may evolve to accommodate discrepancies. type : 'bot-llm-search-response' data : search_result : string (optional) Raw search result text. rendered_content : string (optional) Formatted version of the search results. origins : Array<Origin Object> Source information and confidence scores for search results. The Origin Object follows this structure: Copy Ask AI { "site_uri" : string (optional) , "site_title" : string (optional) , "results" : Array< { "text" : string , "confidence" : number [] } > } Example: Copy Ask AI "id" : undefined "label" : "rtvi-ai" "type" : "bot-llm-search-response" "data" : { "origins" : [ { "results" : [ { "confidence" : [ 0.9881149530410768 ], "text" : "* Juneteenth: A Freedom Celebration is scheduled for June 18th from 12:00 pm to 2:00 pm." }, { "confidence" : [ 0.9692034721374512 ], "ext" : "* A Juneteenth celebration at Fort Negley Park will take place on June 19th from 5:00 pm to 9:30 pm." } ], "site_title" : "vanderbilt.edu" , "site_uri" : "https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQHwif83VK9KAzrbMSGSBsKwL8vWfSfC9pgEWYKmStHyqiRoV1oe8j1S0nbwRg_iWgqAr9wUkiegu3ATC8Ll-cuE-vpzwElRHiJ2KgRYcqnOQMoOeokVpWqi" }, { "results" : [ { "confidence" : [ 0.6554043292999268 ], "text" : "In addition to these events, Vanderbilt University is a large research institution with ongoing activities across many fields." } ], "site_title" : "wikipedia.org" , "site_uri" : "https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQESbF-ijx78QbaglrhflHCUWdPTD4M6tYOQigW5hgsHNctRlAHu9ktfPmJx7DfoP5QicE0y-OQY1cRl9w4Id0btiFgLYSKIm2-SPtOHXeNrAlgA7mBnclaGrD7rgnLIbrjl8DgUEJrrvT0CKzuo" }], "rendered_content" : "<style> \n .container ... </div> \n </div> \n " , "search_result" : "Several events are happening at Vanderbilt University: \n\n * Juneteenth: A Freedom Celebration is scheduled for June 18th from 12:00 pm to 2:00 pm. \n * A Juneteenth celebration at Fort Negley Park will take place on June 19th from 5:00 pm to 9:30 pm. \n\n In addition to these events, Vanderbilt University is a large research institution with ongoing activities across many fields. For the most recent news, you should check Vanderbilt's official news website. \n " } ​ Service-Specific Insights ​ bot-llm-started 🤖 Indicates LLM processing has begun type : bot-llm-started data : None ​ bot-llm-stopped 🤖 Indicates LLM processing has completed type : bot-llm-stopped data : None ​ user-llm-text 🤖 Aggregated user input text that is sent to the LLM. type : 'user-llm-text' data : text : string The user’s input text to be processed by the LLM. ​ bot-llm-text 🤖 Individual tokens streamed from the LLM as they are generated. type : 'bot-llm-text' data : text : string The token text from the LLM. ​ bot-tts-started 🤖 Indicates text-to-speech (TTS) processing has begun. type : 'bot-tts-started' data : None ​ bot-tts-stopped 🤖 Indicates text-to-speech (TTS) processing has completed. type : 'bot-tts-stopped' data : None ​ bot-tts-text 🤖 The per-token text output of the text-to-speech (TTS) service (what the TTS actually says). type : 'bot-tts-text' data : text : string The text representation of the generated bot speech. ​ Metrics and Monitoring ​ metrics 🤖 Performance metrics for various processing stages and services. Each message will contain entries for one or more of the metrics types: processing , ttfb , characters . type : 'metrics' data : processing : [See Below] (optional) Processing time metrics. ttfb : [See Below] (optional) Time to first byte metrics. characters : [See Below] (optional) Character processing metrics. For each metric type, the data structure is an array of objects with the following structure: processor : string The name of the processor or service that generated the metric. value : number The value of the metric, typically in milliseconds or character count. model : string (optional) The model of the service that generated the metric, if applicable. Example: Copy Ask AI { "type" : "metrics" , "data" : { "processing" : [ { "model" : "eleven_flash_v2_5" , "processor" : "ElevenLabsTTSService#0" , "value" : 0.0005140304565429688 } ], "ttfb" : [ { "model" : "eleven_flash_v2_5" , "processor" : "ElevenLabsTTSService#0" , "value" : 0.1573178768157959 } ], "characters" : [ { "model" : "eleven_flash_v2_5" , "processor" : "ElevenLabsTTSService#0" , "value" : 43 } ] } } Client SDKs RTVIClient Migration Guide On this page Key Features Terms RTVI Message Format RTVI Message Types Connection Management client-ready 🏄 bot-ready 🤖 disconnect-bot 🏄 error 🤖 Transcription user-started-speaking 🤖 user-stopped-speaking 🤖 bot-started-speaking 🤖 bot-stopped-speaking 🤖 user-transcription 🤖 bot-transcription 🤖 Client-Server Messaging server-message 🤖 client-message 🏄 server-response 🤖 error-response 🤖 Advanced LLM Interactions append-to-context 🏄 llm-function-call 🤖 llm-function-call-result 🏄 bot-llm-search-response 🤖 Service-Specific Insights bot-llm-started 🤖 bot-llm-stopped 🤖 user-llm-text 🤖 bot-llm-text 🤖 bot-tts-started 🤖 bot-tts-stopped 🤖 bot-tts-text 🤖 Metrics and Monitoring metrics 🤖 Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/client_rtvi-standard_d1d214a8.txt b/client_rtvi-standard_d1d214a8.txt
new file mode 100644
index 0000000000000000000000000000000000000000..cdb30ac12444aa7d451483651ca49449c9016861
--- /dev/null
+++ b/client_rtvi-standard_d1d214a8.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/client/rtvi-standard#bot-llm-started-%F0%9F%A4%96
+Title: The RTVI Standard - Pipecat
+==================================================
+
+The RTVI Standard - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation The RTVI Standard Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Client SDKs The RTVI Standard RTVIClient Migration Guide Javascript SDK SDK Introduction API Reference Transport packages React SDK SDK Introduction API Reference React Native SDK SDK Introduction API Reference iOS SDK SDK Introduction API Reference Transport packages Android SDK SDK Introduction API Reference Transport packages C++ SDK SDK Introduction Daily WebRTC Transport The RTVI (Real-Time Voice and Video Inference) standard defines a set of message types and structures sent between clients and servers. It is designed to facilitate real-time interactions between clients and AI applications that require voice, video, and text communication. It provides a consistent framework for building applications that can communicate with AI models and the backends running those models in real-time. This page documents version 1.0 of the RTVI standard, released in June 2025. ​ Key Features Connection Management RTVI provides a flexible connection model that allows clients to connect to AI services and coordinate state. Transcriptions The standard includes built-in support for real-time transcription of audio streams. Client-Server Messaging The standard defines a messaging protocol for sending and receiving messages between clients and servers, allowing for efficient communication of requests and responses. Advanced LLM Interactions The standard supports advanced interactions with large language models (LLMs), including context management, function call handline, and search results. Service-Specific Insights RTVI supports events to provide insight into the input/output and state for typical services that exist in speech-to-speech workflows. Metrics and Monitoring RTVI provides mechanisms for collecting metrics and monitoring the performance of server-side services. ​ Terms Client : The front-end application or user interface that interacts with the RTVI server. Server : The backend-end service that runs the AI framework and processes requests from the client. User : The end user interacting with the client application. Bot : The AI interacting with the user, technically an amalgamation of a large language model (LLM) and a text-to-speech (TTS) service. ​ RTVI Message Format The messages defined as part of the RTVI protocol adhere to the following format: Copy Ask AI { "id" : string , "label" : "rtvi-ai" , "type" : string , "data" : unknown } ​ id string A unique identifier for the message, used to correlate requests and responses. ​ label string default: "rtvi-ai" required A label that identifies this message as an RTVI message. This field is required and should always be set to 'rtvi-ai' . ​ type string required The type of message being sent. This field is required and should be set to one of the predefined RTVI message types listed below. ​ data unknown The payload of the message, which can be any data structure relevant to the message type. ​ RTVI Message Types Following the above format, this section describes the various message types defined by the RTVI standard. Each message type has a specific purpose and structure, allowing for clear communication between clients and servers. Each message type below includes either a 🤖 or 🏄 emoji to denote whether the message is sent from the bot (🤖) or client (🏄). ​ Connection Management ​ client-ready 🏄 Indicates that the client is ready to receive messages and interact with the server. Typically sent after the transport media channels have connected. type : 'client-ready' data : version : string The version of the RTVI standard being used. This is useful for ensuring compatibility between client and server implementations. about : AboutClient Object An object containing information about the client, such as its rtvi-version, client library, and any other relevant metadata. The AboutClient object follows this structure: Show AboutClient ​ library string required ​ library_version string ​ platform string ​ platform_version string ​ platform_details any Any platform-specific details that may be relevant to the server. This could include information about the browser, operating system, or any other environment-specific data needed by the server. This field is optional and open-ended, so please be mindful of the data you include here and any security concerns that may arise from exposing sensitive or personal-identifiable information. ​ bot-ready 🤖 Indicates that the bot is ready to receive messages and interact with the client. Typically send after the transport media channels have connected. type : 'bot-ready' data : version : string The version of the RTVI standard being used. This is useful for ensuring compatibility between client and server implementations. about : any (Optional) An object containing information about the server or bot. It’s structure and value are both undefined by default. This provides flexibility to include any relevant metadata your client may need to know about the server at connection time, without any built-in security concerns. Please be mindful of the data you include here and any security concerns that may arise from exposing sensitive information. ​ disconnect-bot 🏄 Indicates that the client wishes to disconnect from the bot. Typically used when the client is shutting down or no longer needs to interact with the bot. Note: Disconnets should happen automatically when either the client or bot disconnects from the transport, so this message is intended for the case where a client may want to remain connected to the transport but no longer wishes to interact with the bot. type : 'disconnect-bot' data : undefined ​ error 🤖 Indicates an error occurred during bot initialization or runtime. type : 'error' data : message : string Description of the error. fatal : boolean Indicates if the error is fatal to the session. ​ Transcription ​ user-started-speaking 🤖 Emitted when the user begins speaking type : 'user-started-speaking' data : None ​ user-stopped-speaking 🤖 Emitted when the user stops speaking type : 'user-stopped-speaking' data : None ​ bot-started-speaking 🤖 Emitted when the bot begins speaking type : 'bot-started-speaking' data : None ​ bot-stopped-speaking 🤖 Emitted when the bot stops speaking type : 'bot-stopped-speaking' data : None ​ user-transcription 🤖 Real-time transcription of user speech, including both partial and final results. type : 'user-transcription' data : text : string The transcribed text of the user. final : boolean Indicates if this is a final transcription or a partial result. timestamp : string The timestamp when the transcription was generated. user_id : string Identifier for the user who spoke. ​ bot-transcription 🤖 Transcription of the bot’s speech. Note: This protocol currently does not match the user transcription format to support real-time timestamping for bot transcriptions. Rather, the event is typically sent for each sentence of the bot’s response. This difference is currently due to limitations in TTS services which mostly do not support (or support well), accurate timing information. If/when this changes, this protocol may be updated to include the necessary timing information. For now, if you want to attempt real-time transcription to match your bot’s speaking, you can try using the bot-tts-text message type. type : 'bot-transcription' data : text : string The transcribed text from the bot, typically aggregated at a per-sentence level. ​ Client-Server Messaging ​ server-message 🤖 An arbitrary message sent from the server to the client. This can be used for custom interactions or commands. This message may be coupled with the client-message message type to handle responses from the client. type : 'server-message' data : any The data can be any JSON-serializable object, formatted according to your own specifications. ​ client-message 🏄 An arbitrary message sent from the client to the server. This can be used for custom interactions or commands. This message may be coupled with the server-response message type to handle responses from the server. type : 'client-message' data : t : string d : unknown (optional) The data payload should contain a t field indicating the type of message and an optional d field containing any custom, corresponding data needed for the message. ​ server-response 🤖 An message sent from the server to the client in response to a client-message . IMPORTANT : The id should match the id of the original client-message to correlate the response with the request. type : 'client-message' data : t : string d : unknown (optional) The data payload should contain a t field indicating the type of message and an optional d field containing any custom, corresponding data needed for the message. ​ error-response 🤖 Error response to a specific client message. IMPORTANT : The id should match the id of the original client-message to correlate the response with the request. type : 'error-response' data : error : string ​ Advanced LLM Interactions ​ append-to-context 🏄 A message sent from the client to the server to append data to the context of the current llm conversation. This is useful for providing text-based content for the user or augmenting the context for the assistant. type : 'append-to-context' data : role : "user" | "assistant" The role the context should be appended to. Currently only supports "user" and "assistant" . content : unknown The content to append to the context. This can be any data structure the llm understand. run_immediately : boolean (optional) Indicates whether the context should be run immediately after appending. Defaults to false . If set to false , the context will be appended but not executed until the next llm run. ​ llm-function-call 🤖 A function call request from the LLM, sent from the bot to the client. Note that for most cases, an LLM function call will be handled completely server-side. However, in the event that the call requires input from the client or the client needs to be aware of the function call, this message/response schema is required. type : 'llm-function-call' data : function_name : string Name of the function to be called. tool_call_id : string Unique identifier for this function call. args : Record<string, unknown> Arguments to be passed to the function. ​ llm-function-call-result 🏄 The result of the function call requested by the LLM, returned from the client. type : 'llm-function-call-result' data : function_name : string Name of the called function. tool_call_id : string Identifier matching the original function call. args : Record<string, unknown> Arguments that were passed to the function. result : Record<string, unknown> | string The result returned by the function. ​ bot-llm-search-response 🤖 Search results from the LLM’s knowledge base. Currently, Google Gemini is the only LLM that supports built-in search. However, we expect other LLMs to follow suite, which is why this message type is defined as part of the RTVI standard. As more LLMs add support for this feature, the format of this message type may evolve to accommodate discrepancies. type : 'bot-llm-search-response' data : search_result : string (optional) Raw search result text. rendered_content : string (optional) Formatted version of the search results. origins : Array<Origin Object> Source information and confidence scores for search results. The Origin Object follows this structure: Copy Ask AI { "site_uri" : string (optional) , "site_title" : string (optional) , "results" : Array< { "text" : string , "confidence" : number [] } > } Example: Copy Ask AI "id" : undefined "label" : "rtvi-ai" "type" : "bot-llm-search-response" "data" : { "origins" : [ { "results" : [ { "confidence" : [ 0.9881149530410768 ], "text" : "* Juneteenth: A Freedom Celebration is scheduled for June 18th from 12:00 pm to 2:00 pm." }, { "confidence" : [ 0.9692034721374512 ], "ext" : "* A Juneteenth celebration at Fort Negley Park will take place on June 19th from 5:00 pm to 9:30 pm." } ], "site_title" : "vanderbilt.edu" , "site_uri" : "https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQHwif83VK9KAzrbMSGSBsKwL8vWfSfC9pgEWYKmStHyqiRoV1oe8j1S0nbwRg_iWgqAr9wUkiegu3ATC8Ll-cuE-vpzwElRHiJ2KgRYcqnOQMoOeokVpWqi" }, { "results" : [ { "confidence" : [ 0.6554043292999268 ], "text" : "In addition to these events, Vanderbilt University is a large research institution with ongoing activities across many fields." } ], "site_title" : "wikipedia.org" , "site_uri" : "https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQESbF-ijx78QbaglrhflHCUWdPTD4M6tYOQigW5hgsHNctRlAHu9ktfPmJx7DfoP5QicE0y-OQY1cRl9w4Id0btiFgLYSKIm2-SPtOHXeNrAlgA7mBnclaGrD7rgnLIbrjl8DgUEJrrvT0CKzuo" }], "rendered_content" : "<style> \n .container ... </div> \n </div> \n " , "search_result" : "Several events are happening at Vanderbilt University: \n\n * Juneteenth: A Freedom Celebration is scheduled for June 18th from 12:00 pm to 2:00 pm. \n * A Juneteenth celebration at Fort Negley Park will take place on June 19th from 5:00 pm to 9:30 pm. \n\n In addition to these events, Vanderbilt University is a large research institution with ongoing activities across many fields. For the most recent news, you should check Vanderbilt's official news website. \n " } ​ Service-Specific Insights ​ bot-llm-started 🤖 Indicates LLM processing has begun type : bot-llm-started data : None ​ bot-llm-stopped 🤖 Indicates LLM processing has completed type : bot-llm-stopped data : None ​ user-llm-text 🤖 Aggregated user input text that is sent to the LLM. type : 'user-llm-text' data : text : string The user’s input text to be processed by the LLM. ​ bot-llm-text 🤖 Individual tokens streamed from the LLM as they are generated. type : 'bot-llm-text' data : text : string The token text from the LLM. ​ bot-tts-started 🤖 Indicates text-to-speech (TTS) processing has begun. type : 'bot-tts-started' data : None ​ bot-tts-stopped 🤖 Indicates text-to-speech (TTS) processing has completed. type : 'bot-tts-stopped' data : None ​ bot-tts-text 🤖 The per-token text output of the text-to-speech (TTS) service (what the TTS actually says). type : 'bot-tts-text' data : text : string The text representation of the generated bot speech. ​ Metrics and Monitoring ​ metrics 🤖 Performance metrics for various processing stages and services. Each message will contain entries for one or more of the metrics types: processing , ttfb , characters . type : 'metrics' data : processing : [See Below] (optional) Processing time metrics. ttfb : [See Below] (optional) Time to first byte metrics. characters : [See Below] (optional) Character processing metrics. For each metric type, the data structure is an array of objects with the following structure: processor : string The name of the processor or service that generated the metric. value : number The value of the metric, typically in milliseconds or character count. model : string (optional) The model of the service that generated the metric, if applicable. Example: Copy Ask AI { "type" : "metrics" , "data" : { "processing" : [ { "model" : "eleven_flash_v2_5" , "processor" : "ElevenLabsTTSService#0" , "value" : 0.0005140304565429688 } ], "ttfb" : [ { "model" : "eleven_flash_v2_5" , "processor" : "ElevenLabsTTSService#0" , "value" : 0.1573178768157959 } ], "characters" : [ { "model" : "eleven_flash_v2_5" , "processor" : "ElevenLabsTTSService#0" , "value" : 43 } ] } } Client SDKs RTVIClient Migration Guide On this page Key Features Terms RTVI Message Format RTVI Message Types Connection Management client-ready 🏄 bot-ready 🤖 disconnect-bot 🏄 error 🤖 Transcription user-started-speaking 🤖 user-stopped-speaking 🤖 bot-started-speaking 🤖 bot-stopped-speaking 🤖 user-transcription 🤖 bot-transcription 🤖 Client-Server Messaging server-message 🤖 client-message 🏄 server-response 🤖 error-response 🤖 Advanced LLM Interactions append-to-context 🏄 llm-function-call 🤖 llm-function-call-result 🏄 bot-llm-search-response 🤖 Service-Specific Insights bot-llm-started 🤖 bot-llm-stopped 🤖 user-llm-text 🤖 bot-llm-text 🤖 bot-tts-started 🤖 bot-tts-stopped 🤖 bot-tts-text 🤖 Metrics and Monitoring metrics 🤖 Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/client_rtvi-standard_d7bdeedf.txt b/client_rtvi-standard_d7bdeedf.txt
new file mode 100644
index 0000000000000000000000000000000000000000..20b230907e3ad627bfc60199b04f1f5c4d0ae1f1
--- /dev/null
+++ b/client_rtvi-standard_d7bdeedf.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/client/rtvi-standard#bot-llm-text-%F0%9F%A4%96
+Title: The RTVI Standard - Pipecat
+==================================================
+
+The RTVI Standard - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation The RTVI Standard Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Client SDKs The RTVI Standard RTVIClient Migration Guide Javascript SDK SDK Introduction API Reference Transport packages React SDK SDK Introduction API Reference React Native SDK SDK Introduction API Reference iOS SDK SDK Introduction API Reference Transport packages Android SDK SDK Introduction API Reference Transport packages C++ SDK SDK Introduction Daily WebRTC Transport The RTVI (Real-Time Voice and Video Inference) standard defines a set of message types and structures sent between clients and servers. It is designed to facilitate real-time interactions between clients and AI applications that require voice, video, and text communication. It provides a consistent framework for building applications that can communicate with AI models and the backends running those models in real-time. This page documents version 1.0 of the RTVI standard, released in June 2025. ​ Key Features Connection Management RTVI provides a flexible connection model that allows clients to connect to AI services and coordinate state. Transcriptions The standard includes built-in support for real-time transcription of audio streams. Client-Server Messaging The standard defines a messaging protocol for sending and receiving messages between clients and servers, allowing for efficient communication of requests and responses. Advanced LLM Interactions The standard supports advanced interactions with large language models (LLMs), including context management, function call handline, and search results. Service-Specific Insights RTVI supports events to provide insight into the input/output and state for typical services that exist in speech-to-speech workflows. Metrics and Monitoring RTVI provides mechanisms for collecting metrics and monitoring the performance of server-side services. ​ Terms Client : The front-end application or user interface that interacts with the RTVI server. Server : The backend-end service that runs the AI framework and processes requests from the client. User : The end user interacting with the client application. Bot : The AI interacting with the user, technically an amalgamation of a large language model (LLM) and a text-to-speech (TTS) service. ​ RTVI Message Format The messages defined as part of the RTVI protocol adhere to the following format: Copy Ask AI { "id" : string , "label" : "rtvi-ai" , "type" : string , "data" : unknown } ​ id string A unique identifier for the message, used to correlate requests and responses. ​ label string default: "rtvi-ai" required A label that identifies this message as an RTVI message. This field is required and should always be set to 'rtvi-ai' . ​ type string required The type of message being sent. This field is required and should be set to one of the predefined RTVI message types listed below. ​ data unknown The payload of the message, which can be any data structure relevant to the message type. ​ RTVI Message Types Following the above format, this section describes the various message types defined by the RTVI standard. Each message type has a specific purpose and structure, allowing for clear communication between clients and servers. Each message type below includes either a 🤖 or 🏄 emoji to denote whether the message is sent from the bot (🤖) or client (🏄). ​ Connection Management ​ client-ready 🏄 Indicates that the client is ready to receive messages and interact with the server. Typically sent after the transport media channels have connected. type : 'client-ready' data : version : string The version of the RTVI standard being used. This is useful for ensuring compatibility between client and server implementations. about : AboutClient Object An object containing information about the client, such as its rtvi-version, client library, and any other relevant metadata. The AboutClient object follows this structure: Show AboutClient ​ library string required ​ library_version string ​ platform string ​ platform_version string ​ platform_details any Any platform-specific details that may be relevant to the server. This could include information about the browser, operating system, or any other environment-specific data needed by the server. This field is optional and open-ended, so please be mindful of the data you include here and any security concerns that may arise from exposing sensitive or personal-identifiable information. ​ bot-ready 🤖 Indicates that the bot is ready to receive messages and interact with the client. Typically send after the transport media channels have connected. type : 'bot-ready' data : version : string The version of the RTVI standard being used. This is useful for ensuring compatibility between client and server implementations. about : any (Optional) An object containing information about the server or bot. It’s structure and value are both undefined by default. This provides flexibility to include any relevant metadata your client may need to know about the server at connection time, without any built-in security concerns. Please be mindful of the data you include here and any security concerns that may arise from exposing sensitive information. ​ disconnect-bot 🏄 Indicates that the client wishes to disconnect from the bot. Typically used when the client is shutting down or no longer needs to interact with the bot. Note: Disconnets should happen automatically when either the client or bot disconnects from the transport, so this message is intended for the case where a client may want to remain connected to the transport but no longer wishes to interact with the bot. type : 'disconnect-bot' data : undefined ​ error 🤖 Indicates an error occurred during bot initialization or runtime. type : 'error' data : message : string Description of the error. fatal : boolean Indicates if the error is fatal to the session. ​ Transcription ​ user-started-speaking 🤖 Emitted when the user begins speaking type : 'user-started-speaking' data : None ​ user-stopped-speaking 🤖 Emitted when the user stops speaking type : 'user-stopped-speaking' data : None ​ bot-started-speaking 🤖 Emitted when the bot begins speaking type : 'bot-started-speaking' data : None ​ bot-stopped-speaking 🤖 Emitted when the bot stops speaking type : 'bot-stopped-speaking' data : None ​ user-transcription 🤖 Real-time transcription of user speech, including both partial and final results. type : 'user-transcription' data : text : string The transcribed text of the user. final : boolean Indicates if this is a final transcription or a partial result. timestamp : string The timestamp when the transcription was generated. user_id : string Identifier for the user who spoke. ​ bot-transcription 🤖 Transcription of the bot’s speech. Note: This protocol currently does not match the user transcription format to support real-time timestamping for bot transcriptions. Rather, the event is typically sent for each sentence of the bot’s response. This difference is currently due to limitations in TTS services which mostly do not support (or support well), accurate timing information. If/when this changes, this protocol may be updated to include the necessary timing information. For now, if you want to attempt real-time transcription to match your bot’s speaking, you can try using the bot-tts-text message type. type : 'bot-transcription' data : text : string The transcribed text from the bot, typically aggregated at a per-sentence level. ​ Client-Server Messaging ​ server-message 🤖 An arbitrary message sent from the server to the client. This can be used for custom interactions or commands. This message may be coupled with the client-message message type to handle responses from the client. type : 'server-message' data : any The data can be any JSON-serializable object, formatted according to your own specifications. ​ client-message 🏄 An arbitrary message sent from the client to the server. This can be used for custom interactions or commands. This message may be coupled with the server-response message type to handle responses from the server. type : 'client-message' data : t : string d : unknown (optional) The data payload should contain a t field indicating the type of message and an optional d field containing any custom, corresponding data needed for the message. ​ server-response 🤖 An message sent from the server to the client in response to a client-message . IMPORTANT : The id should match the id of the original client-message to correlate the response with the request. type : 'client-message' data : t : string d : unknown (optional) The data payload should contain a t field indicating the type of message and an optional d field containing any custom, corresponding data needed for the message. ​ error-response 🤖 Error response to a specific client message. IMPORTANT : The id should match the id of the original client-message to correlate the response with the request. type : 'error-response' data : error : string ​ Advanced LLM Interactions ​ append-to-context 🏄 A message sent from the client to the server to append data to the context of the current llm conversation. This is useful for providing text-based content for the user or augmenting the context for the assistant. type : 'append-to-context' data : role : "user" | "assistant" The role the context should be appended to. Currently only supports "user" and "assistant" . content : unknown The content to append to the context. This can be any data structure the llm understand. run_immediately : boolean (optional) Indicates whether the context should be run immediately after appending. Defaults to false . If set to false , the context will be appended but not executed until the next llm run. ​ llm-function-call 🤖 A function call request from the LLM, sent from the bot to the client. Note that for most cases, an LLM function call will be handled completely server-side. However, in the event that the call requires input from the client or the client needs to be aware of the function call, this message/response schema is required. type : 'llm-function-call' data : function_name : string Name of the function to be called. tool_call_id : string Unique identifier for this function call. args : Record<string, unknown> Arguments to be passed to the function. ​ llm-function-call-result 🏄 The result of the function call requested by the LLM, returned from the client. type : 'llm-function-call-result' data : function_name : string Name of the called function. tool_call_id : string Identifier matching the original function call. args : Record<string, unknown> Arguments that were passed to the function. result : Record<string, unknown> | string The result returned by the function. ​ bot-llm-search-response 🤖 Search results from the LLM’s knowledge base. Currently, Google Gemini is the only LLM that supports built-in search. However, we expect other LLMs to follow suite, which is why this message type is defined as part of the RTVI standard. As more LLMs add support for this feature, the format of this message type may evolve to accommodate discrepancies. type : 'bot-llm-search-response' data : search_result : string (optional) Raw search result text. rendered_content : string (optional) Formatted version of the search results. origins : Array<Origin Object> Source information and confidence scores for search results. The Origin Object follows this structure: Copy Ask AI { "site_uri" : string (optional) , "site_title" : string (optional) , "results" : Array< { "text" : string , "confidence" : number [] } > } Example: Copy Ask AI "id" : undefined "label" : "rtvi-ai" "type" : "bot-llm-search-response" "data" : { "origins" : [ { "results" : [ { "confidence" : [ 0.9881149530410768 ], "text" : "* Juneteenth: A Freedom Celebration is scheduled for June 18th from 12:00 pm to 2:00 pm." }, { "confidence" : [ 0.9692034721374512 ], "ext" : "* A Juneteenth celebration at Fort Negley Park will take place on June 19th from 5:00 pm to 9:30 pm." } ], "site_title" : "vanderbilt.edu" , "site_uri" : "https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQHwif83VK9KAzrbMSGSBsKwL8vWfSfC9pgEWYKmStHyqiRoV1oe8j1S0nbwRg_iWgqAr9wUkiegu3ATC8Ll-cuE-vpzwElRHiJ2KgRYcqnOQMoOeokVpWqi" }, { "results" : [ { "confidence" : [ 0.6554043292999268 ], "text" : "In addition to these events, Vanderbilt University is a large research institution with ongoing activities across many fields." } ], "site_title" : "wikipedia.org" , "site_uri" : "https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQESbF-ijx78QbaglrhflHCUWdPTD4M6tYOQigW5hgsHNctRlAHu9ktfPmJx7DfoP5QicE0y-OQY1cRl9w4Id0btiFgLYSKIm2-SPtOHXeNrAlgA7mBnclaGrD7rgnLIbrjl8DgUEJrrvT0CKzuo" }], "rendered_content" : "<style> \n .container ... </div> \n </div> \n " , "search_result" : "Several events are happening at Vanderbilt University: \n\n * Juneteenth: A Freedom Celebration is scheduled for June 18th from 12:00 pm to 2:00 pm. \n * A Juneteenth celebration at Fort Negley Park will take place on June 19th from 5:00 pm to 9:30 pm. \n\n In addition to these events, Vanderbilt University is a large research institution with ongoing activities across many fields. For the most recent news, you should check Vanderbilt's official news website. \n " } ​ Service-Specific Insights ​ bot-llm-started 🤖 Indicates LLM processing has begun type : bot-llm-started data : None ​ bot-llm-stopped 🤖 Indicates LLM processing has completed type : bot-llm-stopped data : None ​ user-llm-text 🤖 Aggregated user input text that is sent to the LLM. type : 'user-llm-text' data : text : string The user’s input text to be processed by the LLM. ​ bot-llm-text 🤖 Individual tokens streamed from the LLM as they are generated. type : 'bot-llm-text' data : text : string The token text from the LLM. ​ bot-tts-started 🤖 Indicates text-to-speech (TTS) processing has begun. type : 'bot-tts-started' data : None ​ bot-tts-stopped 🤖 Indicates text-to-speech (TTS) processing has completed. type : 'bot-tts-stopped' data : None ​ bot-tts-text 🤖 The per-token text output of the text-to-speech (TTS) service (what the TTS actually says). type : 'bot-tts-text' data : text : string The text representation of the generated bot speech. ​ Metrics and Monitoring ​ metrics 🤖 Performance metrics for various processing stages and services. Each message will contain entries for one or more of the metrics types: processing , ttfb , characters . type : 'metrics' data : processing : [See Below] (optional) Processing time metrics. ttfb : [See Below] (optional) Time to first byte metrics. characters : [See Below] (optional) Character processing metrics. For each metric type, the data structure is an array of objects with the following structure: processor : string The name of the processor or service that generated the metric. value : number The value of the metric, typically in milliseconds or character count. model : string (optional) The model of the service that generated the metric, if applicable. Example: Copy Ask AI { "type" : "metrics" , "data" : { "processing" : [ { "model" : "eleven_flash_v2_5" , "processor" : "ElevenLabsTTSService#0" , "value" : 0.0005140304565429688 } ], "ttfb" : [ { "model" : "eleven_flash_v2_5" , "processor" : "ElevenLabsTTSService#0" , "value" : 0.1573178768157959 } ], "characters" : [ { "model" : "eleven_flash_v2_5" , "processor" : "ElevenLabsTTSService#0" , "value" : 43 } ] } } Client SDKs RTVIClient Migration Guide On this page Key Features Terms RTVI Message Format RTVI Message Types Connection Management client-ready 🏄 bot-ready 🤖 disconnect-bot 🏄 error 🤖 Transcription user-started-speaking 🤖 user-stopped-speaking 🤖 bot-started-speaking 🤖 bot-stopped-speaking 🤖 user-transcription 🤖 bot-transcription 🤖 Client-Server Messaging server-message 🤖 client-message 🏄 server-response 🤖 error-response 🤖 Advanced LLM Interactions append-to-context 🏄 llm-function-call 🤖 llm-function-call-result 🏄 bot-llm-search-response 🤖 Service-Specific Insights bot-llm-started 🤖 bot-llm-stopped 🤖 user-llm-text 🤖 bot-llm-text 🤖 bot-tts-started 🤖 bot-tts-stopped 🤖 bot-tts-text 🤖 Metrics and Monitoring metrics 🤖 Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/client_rtvi-standard_e1d7703a.txt b/client_rtvi-standard_e1d7703a.txt
new file mode 100644
index 0000000000000000000000000000000000000000..6d40e0169cee5e280c119ee66de59df6dc266a07
--- /dev/null
+++ b/client_rtvi-standard_e1d7703a.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/client/rtvi-standard#error-%F0%9F%A4%96
+Title: The RTVI Standard - Pipecat
+==================================================
+
+The RTVI Standard - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation The RTVI Standard Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Client SDKs The RTVI Standard RTVIClient Migration Guide Javascript SDK SDK Introduction API Reference Transport packages React SDK SDK Introduction API Reference React Native SDK SDK Introduction API Reference iOS SDK SDK Introduction API Reference Transport packages Android SDK SDK Introduction API Reference Transport packages C++ SDK SDK Introduction Daily WebRTC Transport The RTVI (Real-Time Voice and Video Inference) standard defines a set of message types and structures sent between clients and servers. It is designed to facilitate real-time interactions between clients and AI applications that require voice, video, and text communication. It provides a consistent framework for building applications that can communicate with AI models and the backends running those models in real-time. This page documents version 1.0 of the RTVI standard, released in June 2025. ​ Key Features Connection Management RTVI provides a flexible connection model that allows clients to connect to AI services and coordinate state. Transcriptions The standard includes built-in support for real-time transcription of audio streams. Client-Server Messaging The standard defines a messaging protocol for sending and receiving messages between clients and servers, allowing for efficient communication of requests and responses. Advanced LLM Interactions The standard supports advanced interactions with large language models (LLMs), including context management, function call handline, and search results. Service-Specific Insights RTVI supports events to provide insight into the input/output and state for typical services that exist in speech-to-speech workflows. Metrics and Monitoring RTVI provides mechanisms for collecting metrics and monitoring the performance of server-side services. ​ Terms Client : The front-end application or user interface that interacts with the RTVI server. Server : The backend-end service that runs the AI framework and processes requests from the client. User : The end user interacting with the client application. Bot : The AI interacting with the user, technically an amalgamation of a large language model (LLM) and a text-to-speech (TTS) service. ​ RTVI Message Format The messages defined as part of the RTVI protocol adhere to the following format: Copy Ask AI { "id" : string , "label" : "rtvi-ai" , "type" : string , "data" : unknown } ​ id string A unique identifier for the message, used to correlate requests and responses. ​ label string default: "rtvi-ai" required A label that identifies this message as an RTVI message. This field is required and should always be set to 'rtvi-ai' . ​ type string required The type of message being sent. This field is required and should be set to one of the predefined RTVI message types listed below. ​ data unknown The payload of the message, which can be any data structure relevant to the message type. ​ RTVI Message Types Following the above format, this section describes the various message types defined by the RTVI standard. Each message type has a specific purpose and structure, allowing for clear communication between clients and servers. Each message type below includes either a 🤖 or 🏄 emoji to denote whether the message is sent from the bot (🤖) or client (🏄). ​ Connection Management ​ client-ready 🏄 Indicates that the client is ready to receive messages and interact with the server. Typically sent after the transport media channels have connected. type : 'client-ready' data : version : string The version of the RTVI standard being used. This is useful for ensuring compatibility between client and server implementations. about : AboutClient Object An object containing information about the client, such as its rtvi-version, client library, and any other relevant metadata. The AboutClient object follows this structure: Show AboutClient ​ library string required ​ library_version string ​ platform string ​ platform_version string ​ platform_details any Any platform-specific details that may be relevant to the server. This could include information about the browser, operating system, or any other environment-specific data needed by the server. This field is optional and open-ended, so please be mindful of the data you include here and any security concerns that may arise from exposing sensitive or personal-identifiable information. ​ bot-ready 🤖 Indicates that the bot is ready to receive messages and interact with the client. Typically send after the transport media channels have connected. type : 'bot-ready' data : version : string The version of the RTVI standard being used. This is useful for ensuring compatibility between client and server implementations. about : any (Optional) An object containing information about the server or bot. It’s structure and value are both undefined by default. This provides flexibility to include any relevant metadata your client may need to know about the server at connection time, without any built-in security concerns. Please be mindful of the data you include here and any security concerns that may arise from exposing sensitive information. ​ disconnect-bot 🏄 Indicates that the client wishes to disconnect from the bot. Typically used when the client is shutting down or no longer needs to interact with the bot. Note: Disconnets should happen automatically when either the client or bot disconnects from the transport, so this message is intended for the case where a client may want to remain connected to the transport but no longer wishes to interact with the bot. type : 'disconnect-bot' data : undefined ​ error 🤖 Indicates an error occurred during bot initialization or runtime. type : 'error' data : message : string Description of the error. fatal : boolean Indicates if the error is fatal to the session. ​ Transcription ​ user-started-speaking 🤖 Emitted when the user begins speaking type : 'user-started-speaking' data : None ​ user-stopped-speaking 🤖 Emitted when the user stops speaking type : 'user-stopped-speaking' data : None ​ bot-started-speaking 🤖 Emitted when the bot begins speaking type : 'bot-started-speaking' data : None ​ bot-stopped-speaking 🤖 Emitted when the bot stops speaking type : 'bot-stopped-speaking' data : None ​ user-transcription 🤖 Real-time transcription of user speech, including both partial and final results. type : 'user-transcription' data : text : string The transcribed text of the user. final : boolean Indicates if this is a final transcription or a partial result. timestamp : string The timestamp when the transcription was generated. user_id : string Identifier for the user who spoke. ​ bot-transcription 🤖 Transcription of the bot’s speech. Note: This protocol currently does not match the user transcription format to support real-time timestamping for bot transcriptions. Rather, the event is typically sent for each sentence of the bot’s response. This difference is currently due to limitations in TTS services which mostly do not support (or support well), accurate timing information. If/when this changes, this protocol may be updated to include the necessary timing information. For now, if you want to attempt real-time transcription to match your bot’s speaking, you can try using the bot-tts-text message type. type : 'bot-transcription' data : text : string The transcribed text from the bot, typically aggregated at a per-sentence level. ​ Client-Server Messaging ​ server-message 🤖 An arbitrary message sent from the server to the client. This can be used for custom interactions or commands. This message may be coupled with the client-message message type to handle responses from the client. type : 'server-message' data : any The data can be any JSON-serializable object, formatted according to your own specifications. ​ client-message 🏄 An arbitrary message sent from the client to the server. This can be used for custom interactions or commands. This message may be coupled with the server-response message type to handle responses from the server. type : 'client-message' data : t : string d : unknown (optional) The data payload should contain a t field indicating the type of message and an optional d field containing any custom, corresponding data needed for the message. ​ server-response 🤖 An message sent from the server to the client in response to a client-message . IMPORTANT : The id should match the id of the original client-message to correlate the response with the request. type : 'client-message' data : t : string d : unknown (optional) The data payload should contain a t field indicating the type of message and an optional d field containing any custom, corresponding data needed for the message. ​ error-response 🤖 Error response to a specific client message. IMPORTANT : The id should match the id of the original client-message to correlate the response with the request. type : 'error-response' data : error : string ​ Advanced LLM Interactions ​ append-to-context 🏄 A message sent from the client to the server to append data to the context of the current llm conversation. This is useful for providing text-based content for the user or augmenting the context for the assistant. type : 'append-to-context' data : role : "user" | "assistant" The role the context should be appended to. Currently only supports "user" and "assistant" . content : unknown The content to append to the context. This can be any data structure the llm understand. run_immediately : boolean (optional) Indicates whether the context should be run immediately after appending. Defaults to false . If set to false , the context will be appended but not executed until the next llm run. ​ llm-function-call 🤖 A function call request from the LLM, sent from the bot to the client. Note that for most cases, an LLM function call will be handled completely server-side. However, in the event that the call requires input from the client or the client needs to be aware of the function call, this message/response schema is required. type : 'llm-function-call' data : function_name : string Name of the function to be called. tool_call_id : string Unique identifier for this function call. args : Record<string, unknown> Arguments to be passed to the function. ​ llm-function-call-result 🏄 The result of the function call requested by the LLM, returned from the client. type : 'llm-function-call-result' data : function_name : string Name of the called function. tool_call_id : string Identifier matching the original function call. args : Record<string, unknown> Arguments that were passed to the function. result : Record<string, unknown> | string The result returned by the function. ​ bot-llm-search-response 🤖 Search results from the LLM’s knowledge base. Currently, Google Gemini is the only LLM that supports built-in search. However, we expect other LLMs to follow suite, which is why this message type is defined as part of the RTVI standard. As more LLMs add support for this feature, the format of this message type may evolve to accommodate discrepancies. type : 'bot-llm-search-response' data : search_result : string (optional) Raw search result text. rendered_content : string (optional) Formatted version of the search results. origins : Array<Origin Object> Source information and confidence scores for search results. The Origin Object follows this structure: Copy Ask AI { "site_uri" : string (optional) , "site_title" : string (optional) , "results" : Array< { "text" : string , "confidence" : number [] } > } Example: Copy Ask AI "id" : undefined "label" : "rtvi-ai" "type" : "bot-llm-search-response" "data" : { "origins" : [ { "results" : [ { "confidence" : [ 0.9881149530410768 ], "text" : "* Juneteenth: A Freedom Celebration is scheduled for June 18th from 12:00 pm to 2:00 pm." }, { "confidence" : [ 0.9692034721374512 ], "ext" : "* A Juneteenth celebration at Fort Negley Park will take place on June 19th from 5:00 pm to 9:30 pm." } ], "site_title" : "vanderbilt.edu" , "site_uri" : "https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQHwif83VK9KAzrbMSGSBsKwL8vWfSfC9pgEWYKmStHyqiRoV1oe8j1S0nbwRg_iWgqAr9wUkiegu3ATC8Ll-cuE-vpzwElRHiJ2KgRYcqnOQMoOeokVpWqi" }, { "results" : [ { "confidence" : [ 0.6554043292999268 ], "text" : "In addition to these events, Vanderbilt University is a large research institution with ongoing activities across many fields." } ], "site_title" : "wikipedia.org" , "site_uri" : "https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQESbF-ijx78QbaglrhflHCUWdPTD4M6tYOQigW5hgsHNctRlAHu9ktfPmJx7DfoP5QicE0y-OQY1cRl9w4Id0btiFgLYSKIm2-SPtOHXeNrAlgA7mBnclaGrD7rgnLIbrjl8DgUEJrrvT0CKzuo" }], "rendered_content" : "<style> \n .container ... </div> \n </div> \n " , "search_result" : "Several events are happening at Vanderbilt University: \n\n * Juneteenth: A Freedom Celebration is scheduled for June 18th from 12:00 pm to 2:00 pm. \n * A Juneteenth celebration at Fort Negley Park will take place on June 19th from 5:00 pm to 9:30 pm. \n\n In addition to these events, Vanderbilt University is a large research institution with ongoing activities across many fields. For the most recent news, you should check Vanderbilt's official news website. \n " } ​ Service-Specific Insights ​ bot-llm-started 🤖 Indicates LLM processing has begun type : bot-llm-started data : None ​ bot-llm-stopped 🤖 Indicates LLM processing has completed type : bot-llm-stopped data : None ​ user-llm-text 🤖 Aggregated user input text that is sent to the LLM. type : 'user-llm-text' data : text : string The user’s input text to be processed by the LLM. ​ bot-llm-text 🤖 Individual tokens streamed from the LLM as they are generated. type : 'bot-llm-text' data : text : string The token text from the LLM. ​ bot-tts-started 🤖 Indicates text-to-speech (TTS) processing has begun. type : 'bot-tts-started' data : None ​ bot-tts-stopped 🤖 Indicates text-to-speech (TTS) processing has completed. type : 'bot-tts-stopped' data : None ​ bot-tts-text 🤖 The per-token text output of the text-to-speech (TTS) service (what the TTS actually says). type : 'bot-tts-text' data : text : string The text representation of the generated bot speech. ​ Metrics and Monitoring ​ metrics 🤖 Performance metrics for various processing stages and services. Each message will contain entries for one or more of the metrics types: processing , ttfb , characters . type : 'metrics' data : processing : [See Below] (optional) Processing time metrics. ttfb : [See Below] (optional) Time to first byte metrics. characters : [See Below] (optional) Character processing metrics. For each metric type, the data structure is an array of objects with the following structure: processor : string The name of the processor or service that generated the metric. value : number The value of the metric, typically in milliseconds or character count. model : string (optional) The model of the service that generated the metric, if applicable. Example: Copy Ask AI { "type" : "metrics" , "data" : { "processing" : [ { "model" : "eleven_flash_v2_5" , "processor" : "ElevenLabsTTSService#0" , "value" : 0.0005140304565429688 } ], "ttfb" : [ { "model" : "eleven_flash_v2_5" , "processor" : "ElevenLabsTTSService#0" , "value" : 0.1573178768157959 } ], "characters" : [ { "model" : "eleven_flash_v2_5" , "processor" : "ElevenLabsTTSService#0" , "value" : 43 } ] } } Client SDKs RTVIClient Migration Guide On this page Key Features Terms RTVI Message Format RTVI Message Types Connection Management client-ready 🏄 bot-ready 🤖 disconnect-bot 🏄 error 🤖 Transcription user-started-speaking 🤖 user-stopped-speaking 🤖 bot-started-speaking 🤖 bot-stopped-speaking 🤖 user-transcription 🤖 bot-transcription 🤖 Client-Server Messaging server-message 🤖 client-message 🏄 server-response 🤖 error-response 🤖 Advanced LLM Interactions append-to-context 🏄 llm-function-call 🤖 llm-function-call-result 🏄 bot-llm-search-response 🤖 Service-Specific Insights bot-llm-started 🤖 bot-llm-stopped 🤖 user-llm-text 🤖 bot-llm-text 🤖 bot-tts-started 🤖 bot-tts-stopped 🤖 bot-tts-text 🤖 Metrics and Monitoring metrics 🤖 Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/client_rtvi-standard_e2097772.txt b/client_rtvi-standard_e2097772.txt
new file mode 100644
index 0000000000000000000000000000000000000000..b714dc8519faf6a943c731c5a1e1686c62cd2767
--- /dev/null
+++ b/client_rtvi-standard_e2097772.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/client/rtvi-standard#append-to-context-%F0%9F%8F%84
+Title: The RTVI Standard - Pipecat
+==================================================
+
+The RTVI Standard - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation The RTVI Standard Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Client SDKs The RTVI Standard RTVIClient Migration Guide Javascript SDK SDK Introduction API Reference Transport packages React SDK SDK Introduction API Reference React Native SDK SDK Introduction API Reference iOS SDK SDK Introduction API Reference Transport packages Android SDK SDK Introduction API Reference Transport packages C++ SDK SDK Introduction Daily WebRTC Transport The RTVI (Real-Time Voice and Video Inference) standard defines a set of message types and structures sent between clients and servers. It is designed to facilitate real-time interactions between clients and AI applications that require voice, video, and text communication. It provides a consistent framework for building applications that can communicate with AI models and the backends running those models in real-time. This page documents version 1.0 of the RTVI standard, released in June 2025. ​ Key Features Connection Management RTVI provides a flexible connection model that allows clients to connect to AI services and coordinate state. Transcriptions The standard includes built-in support for real-time transcription of audio streams. Client-Server Messaging The standard defines a messaging protocol for sending and receiving messages between clients and servers, allowing for efficient communication of requests and responses. Advanced LLM Interactions The standard supports advanced interactions with large language models (LLMs), including context management, function call handline, and search results. Service-Specific Insights RTVI supports events to provide insight into the input/output and state for typical services that exist in speech-to-speech workflows. Metrics and Monitoring RTVI provides mechanisms for collecting metrics and monitoring the performance of server-side services. ​ Terms Client : The front-end application or user interface that interacts with the RTVI server. Server : The backend-end service that runs the AI framework and processes requests from the client. User : The end user interacting with the client application. Bot : The AI interacting with the user, technically an amalgamation of a large language model (LLM) and a text-to-speech (TTS) service. ​ RTVI Message Format The messages defined as part of the RTVI protocol adhere to the following format: Copy Ask AI { "id" : string , "label" : "rtvi-ai" , "type" : string , "data" : unknown } ​ id string A unique identifier for the message, used to correlate requests and responses. ​ label string default: "rtvi-ai" required A label that identifies this message as an RTVI message. This field is required and should always be set to 'rtvi-ai' . ​ type string required The type of message being sent. This field is required and should be set to one of the predefined RTVI message types listed below. ​ data unknown The payload of the message, which can be any data structure relevant to the message type. ​ RTVI Message Types Following the above format, this section describes the various message types defined by the RTVI standard. Each message type has a specific purpose and structure, allowing for clear communication between clients and servers. Each message type below includes either a 🤖 or 🏄 emoji to denote whether the message is sent from the bot (🤖) or client (🏄). ​ Connection Management ​ client-ready 🏄 Indicates that the client is ready to receive messages and interact with the server. Typically sent after the transport media channels have connected. type : 'client-ready' data : version : string The version of the RTVI standard being used. This is useful for ensuring compatibility between client and server implementations. about : AboutClient Object An object containing information about the client, such as its rtvi-version, client library, and any other relevant metadata. The AboutClient object follows this structure: Show AboutClient ​ library string required ​ library_version string ​ platform string ​ platform_version string ​ platform_details any Any platform-specific details that may be relevant to the server. This could include information about the browser, operating system, or any other environment-specific data needed by the server. This field is optional and open-ended, so please be mindful of the data you include here and any security concerns that may arise from exposing sensitive or personal-identifiable information. ​ bot-ready 🤖 Indicates that the bot is ready to receive messages and interact with the client. Typically send after the transport media channels have connected. type : 'bot-ready' data : version : string The version of the RTVI standard being used. This is useful for ensuring compatibility between client and server implementations. about : any (Optional) An object containing information about the server or bot. It’s structure and value are both undefined by default. This provides flexibility to include any relevant metadata your client may need to know about the server at connection time, without any built-in security concerns. Please be mindful of the data you include here and any security concerns that may arise from exposing sensitive information. ​ disconnect-bot 🏄 Indicates that the client wishes to disconnect from the bot. Typically used when the client is shutting down or no longer needs to interact with the bot. Note: Disconnets should happen automatically when either the client or bot disconnects from the transport, so this message is intended for the case where a client may want to remain connected to the transport but no longer wishes to interact with the bot. type : 'disconnect-bot' data : undefined ​ error 🤖 Indicates an error occurred during bot initialization or runtime. type : 'error' data : message : string Description of the error. fatal : boolean Indicates if the error is fatal to the session. ​ Transcription ​ user-started-speaking 🤖 Emitted when the user begins speaking type : 'user-started-speaking' data : None ​ user-stopped-speaking 🤖 Emitted when the user stops speaking type : 'user-stopped-speaking' data : None ​ bot-started-speaking 🤖 Emitted when the bot begins speaking type : 'bot-started-speaking' data : None ​ bot-stopped-speaking 🤖 Emitted when the bot stops speaking type : 'bot-stopped-speaking' data : None ​ user-transcription 🤖 Real-time transcription of user speech, including both partial and final results. type : 'user-transcription' data : text : string The transcribed text of the user. final : boolean Indicates if this is a final transcription or a partial result. timestamp : string The timestamp when the transcription was generated. user_id : string Identifier for the user who spoke. ​ bot-transcription 🤖 Transcription of the bot’s speech. Note: This protocol currently does not match the user transcription format to support real-time timestamping for bot transcriptions. Rather, the event is typically sent for each sentence of the bot’s response. This difference is currently due to limitations in TTS services which mostly do not support (or support well), accurate timing information. If/when this changes, this protocol may be updated to include the necessary timing information. For now, if you want to attempt real-time transcription to match your bot’s speaking, you can try using the bot-tts-text message type. type : 'bot-transcription' data : text : string The transcribed text from the bot, typically aggregated at a per-sentence level. ​ Client-Server Messaging ​ server-message 🤖 An arbitrary message sent from the server to the client. This can be used for custom interactions or commands. This message may be coupled with the client-message message type to handle responses from the client. type : 'server-message' data : any The data can be any JSON-serializable object, formatted according to your own specifications. ​ client-message 🏄 An arbitrary message sent from the client to the server. This can be used for custom interactions or commands. This message may be coupled with the server-response message type to handle responses from the server. type : 'client-message' data : t : string d : unknown (optional) The data payload should contain a t field indicating the type of message and an optional d field containing any custom, corresponding data needed for the message. ​ server-response 🤖 An message sent from the server to the client in response to a client-message . IMPORTANT : The id should match the id of the original client-message to correlate the response with the request. type : 'client-message' data : t : string d : unknown (optional) The data payload should contain a t field indicating the type of message and an optional d field containing any custom, corresponding data needed for the message. ​ error-response 🤖 Error response to a specific client message. IMPORTANT : The id should match the id of the original client-message to correlate the response with the request. type : 'error-response' data : error : string ​ Advanced LLM Interactions ​ append-to-context 🏄 A message sent from the client to the server to append data to the context of the current llm conversation. This is useful for providing text-based content for the user or augmenting the context for the assistant. type : 'append-to-context' data : role : "user" | "assistant" The role the context should be appended to. Currently only supports "user" and "assistant" . content : unknown The content to append to the context. This can be any data structure the llm understand. run_immediately : boolean (optional) Indicates whether the context should be run immediately after appending. Defaults to false . If set to false , the context will be appended but not executed until the next llm run. ​ llm-function-call 🤖 A function call request from the LLM, sent from the bot to the client. Note that for most cases, an LLM function call will be handled completely server-side. However, in the event that the call requires input from the client or the client needs to be aware of the function call, this message/response schema is required. type : 'llm-function-call' data : function_name : string Name of the function to be called. tool_call_id : string Unique identifier for this function call. args : Record<string, unknown> Arguments to be passed to the function. ​ llm-function-call-result 🏄 The result of the function call requested by the LLM, returned from the client. type : 'llm-function-call-result' data : function_name : string Name of the called function. tool_call_id : string Identifier matching the original function call. args : Record<string, unknown> Arguments that were passed to the function. result : Record<string, unknown> | string The result returned by the function. ​ bot-llm-search-response 🤖 Search results from the LLM’s knowledge base. Currently, Google Gemini is the only LLM that supports built-in search. However, we expect other LLMs to follow suite, which is why this message type is defined as part of the RTVI standard. As more LLMs add support for this feature, the format of this message type may evolve to accommodate discrepancies. type : 'bot-llm-search-response' data : search_result : string (optional) Raw search result text. rendered_content : string (optional) Formatted version of the search results. origins : Array<Origin Object> Source information and confidence scores for search results. The Origin Object follows this structure: Copy Ask AI { "site_uri" : string (optional) , "site_title" : string (optional) , "results" : Array< { "text" : string , "confidence" : number [] } > } Example: Copy Ask AI "id" : undefined "label" : "rtvi-ai" "type" : "bot-llm-search-response" "data" : { "origins" : [ { "results" : [ { "confidence" : [ 0.9881149530410768 ], "text" : "* Juneteenth: A Freedom Celebration is scheduled for June 18th from 12:00 pm to 2:00 pm." }, { "confidence" : [ 0.9692034721374512 ], "ext" : "* A Juneteenth celebration at Fort Negley Park will take place on June 19th from 5:00 pm to 9:30 pm." } ], "site_title" : "vanderbilt.edu" , "site_uri" : "https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQHwif83VK9KAzrbMSGSBsKwL8vWfSfC9pgEWYKmStHyqiRoV1oe8j1S0nbwRg_iWgqAr9wUkiegu3ATC8Ll-cuE-vpzwElRHiJ2KgRYcqnOQMoOeokVpWqi" }, { "results" : [ { "confidence" : [ 0.6554043292999268 ], "text" : "In addition to these events, Vanderbilt University is a large research institution with ongoing activities across many fields." } ], "site_title" : "wikipedia.org" , "site_uri" : "https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQESbF-ijx78QbaglrhflHCUWdPTD4M6tYOQigW5hgsHNctRlAHu9ktfPmJx7DfoP5QicE0y-OQY1cRl9w4Id0btiFgLYSKIm2-SPtOHXeNrAlgA7mBnclaGrD7rgnLIbrjl8DgUEJrrvT0CKzuo" }], "rendered_content" : "<style> \n .container ... </div> \n </div> \n " , "search_result" : "Several events are happening at Vanderbilt University: \n\n * Juneteenth: A Freedom Celebration is scheduled for June 18th from 12:00 pm to 2:00 pm. \n * A Juneteenth celebration at Fort Negley Park will take place on June 19th from 5:00 pm to 9:30 pm. \n\n In addition to these events, Vanderbilt University is a large research institution with ongoing activities across many fields. For the most recent news, you should check Vanderbilt's official news website. \n " } ​ Service-Specific Insights ​ bot-llm-started 🤖 Indicates LLM processing has begun type : bot-llm-started data : None ​ bot-llm-stopped 🤖 Indicates LLM processing has completed type : bot-llm-stopped data : None ​ user-llm-text 🤖 Aggregated user input text that is sent to the LLM. type : 'user-llm-text' data : text : string The user’s input text to be processed by the LLM. ​ bot-llm-text 🤖 Individual tokens streamed from the LLM as they are generated. type : 'bot-llm-text' data : text : string The token text from the LLM. ​ bot-tts-started 🤖 Indicates text-to-speech (TTS) processing has begun. type : 'bot-tts-started' data : None ​ bot-tts-stopped 🤖 Indicates text-to-speech (TTS) processing has completed. type : 'bot-tts-stopped' data : None ​ bot-tts-text 🤖 The per-token text output of the text-to-speech (TTS) service (what the TTS actually says). type : 'bot-tts-text' data : text : string The text representation of the generated bot speech. ​ Metrics and Monitoring ​ metrics 🤖 Performance metrics for various processing stages and services. Each message will contain entries for one or more of the metrics types: processing , ttfb , characters . type : 'metrics' data : processing : [See Below] (optional) Processing time metrics. ttfb : [See Below] (optional) Time to first byte metrics. characters : [See Below] (optional) Character processing metrics. For each metric type, the data structure is an array of objects with the following structure: processor : string The name of the processor or service that generated the metric. value : number The value of the metric, typically in milliseconds or character count. model : string (optional) The model of the service that generated the metric, if applicable. Example: Copy Ask AI { "type" : "metrics" , "data" : { "processing" : [ { "model" : "eleven_flash_v2_5" , "processor" : "ElevenLabsTTSService#0" , "value" : 0.0005140304565429688 } ], "ttfb" : [ { "model" : "eleven_flash_v2_5" , "processor" : "ElevenLabsTTSService#0" , "value" : 0.1573178768157959 } ], "characters" : [ { "model" : "eleven_flash_v2_5" , "processor" : "ElevenLabsTTSService#0" , "value" : 43 } ] } } Client SDKs RTVIClient Migration Guide On this page Key Features Terms RTVI Message Format RTVI Message Types Connection Management client-ready 🏄 bot-ready 🤖 disconnect-bot 🏄 error 🤖 Transcription user-started-speaking 🤖 user-stopped-speaking 🤖 bot-started-speaking 🤖 bot-stopped-speaking 🤖 user-transcription 🤖 bot-transcription 🤖 Client-Server Messaging server-message 🤖 client-message 🏄 server-response 🤖 error-response 🤖 Advanced LLM Interactions append-to-context 🏄 llm-function-call 🤖 llm-function-call-result 🏄 bot-llm-search-response 🤖 Service-Specific Insights bot-llm-started 🤖 bot-llm-stopped 🤖 user-llm-text 🤖 bot-llm-text 🤖 bot-tts-started 🤖 bot-tts-stopped 🤖 bot-tts-text 🤖 Metrics and Monitoring metrics 🤖 Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/client_rtvi-standard_eb35f846.txt b/client_rtvi-standard_eb35f846.txt
new file mode 100644
index 0000000000000000000000000000000000000000..f2965784af1ae5fe62e2a9bb278bf18c517ad172
--- /dev/null
+++ b/client_rtvi-standard_eb35f846.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/client/rtvi-standard#param-library
+Title: The RTVI Standard - Pipecat
+==================================================
+
+The RTVI Standard - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation The RTVI Standard Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Client SDKs The RTVI Standard RTVIClient Migration Guide Javascript SDK SDK Introduction API Reference Transport packages React SDK SDK Introduction API Reference React Native SDK SDK Introduction API Reference iOS SDK SDK Introduction API Reference Transport packages Android SDK SDK Introduction API Reference Transport packages C++ SDK SDK Introduction Daily WebRTC Transport The RTVI (Real-Time Voice and Video Inference) standard defines a set of message types and structures sent between clients and servers. It is designed to facilitate real-time interactions between clients and AI applications that require voice, video, and text communication. It provides a consistent framework for building applications that can communicate with AI models and the backends running those models in real-time. This page documents version 1.0 of the RTVI standard, released in June 2025. ​ Key Features Connection Management RTVI provides a flexible connection model that allows clients to connect to AI services and coordinate state. Transcriptions The standard includes built-in support for real-time transcription of audio streams. Client-Server Messaging The standard defines a messaging protocol for sending and receiving messages between clients and servers, allowing for efficient communication of requests and responses. Advanced LLM Interactions The standard supports advanced interactions with large language models (LLMs), including context management, function call handline, and search results. Service-Specific Insights RTVI supports events to provide insight into the input/output and state for typical services that exist in speech-to-speech workflows. Metrics and Monitoring RTVI provides mechanisms for collecting metrics and monitoring the performance of server-side services. ​ Terms Client : The front-end application or user interface that interacts with the RTVI server. Server : The backend-end service that runs the AI framework and processes requests from the client. User : The end user interacting with the client application. Bot : The AI interacting with the user, technically an amalgamation of a large language model (LLM) and a text-to-speech (TTS) service. ​ RTVI Message Format The messages defined as part of the RTVI protocol adhere to the following format: Copy Ask AI { "id" : string , "label" : "rtvi-ai" , "type" : string , "data" : unknown } ​ id string A unique identifier for the message, used to correlate requests and responses. ​ label string default: "rtvi-ai" required A label that identifies this message as an RTVI message. This field is required and should always be set to 'rtvi-ai' . ​ type string required The type of message being sent. This field is required and should be set to one of the predefined RTVI message types listed below. ​ data unknown The payload of the message, which can be any data structure relevant to the message type. ​ RTVI Message Types Following the above format, this section describes the various message types defined by the RTVI standard. Each message type has a specific purpose and structure, allowing for clear communication between clients and servers. Each message type below includes either a 🤖 or 🏄 emoji to denote whether the message is sent from the bot (🤖) or client (🏄). ​ Connection Management ​ client-ready 🏄 Indicates that the client is ready to receive messages and interact with the server. Typically sent after the transport media channels have connected. type : 'client-ready' data : version : string The version of the RTVI standard being used. This is useful for ensuring compatibility between client and server implementations. about : AboutClient Object An object containing information about the client, such as its rtvi-version, client library, and any other relevant metadata. The AboutClient object follows this structure: Show AboutClient ​ library string required ​ library_version string ​ platform string ​ platform_version string ​ platform_details any Any platform-specific details that may be relevant to the server. This could include information about the browser, operating system, or any other environment-specific data needed by the server. This field is optional and open-ended, so please be mindful of the data you include here and any security concerns that may arise from exposing sensitive or personal-identifiable information. ​ bot-ready 🤖 Indicates that the bot is ready to receive messages and interact with the client. Typically send after the transport media channels have connected. type : 'bot-ready' data : version : string The version of the RTVI standard being used. This is useful for ensuring compatibility between client and server implementations. about : any (Optional) An object containing information about the server or bot. It’s structure and value are both undefined by default. This provides flexibility to include any relevant metadata your client may need to know about the server at connection time, without any built-in security concerns. Please be mindful of the data you include here and any security concerns that may arise from exposing sensitive information. ​ disconnect-bot 🏄 Indicates that the client wishes to disconnect from the bot. Typically used when the client is shutting down or no longer needs to interact with the bot. Note: Disconnets should happen automatically when either the client or bot disconnects from the transport, so this message is intended for the case where a client may want to remain connected to the transport but no longer wishes to interact with the bot. type : 'disconnect-bot' data : undefined ​ error 🤖 Indicates an error occurred during bot initialization or runtime. type : 'error' data : message : string Description of the error. fatal : boolean Indicates if the error is fatal to the session. ​ Transcription ​ user-started-speaking 🤖 Emitted when the user begins speaking type : 'user-started-speaking' data : None ​ user-stopped-speaking 🤖 Emitted when the user stops speaking type : 'user-stopped-speaking' data : None ​ bot-started-speaking 🤖 Emitted when the bot begins speaking type : 'bot-started-speaking' data : None ​ bot-stopped-speaking 🤖 Emitted when the bot stops speaking type : 'bot-stopped-speaking' data : None ​ user-transcription 🤖 Real-time transcription of user speech, including both partial and final results. type : 'user-transcription' data : text : string The transcribed text of the user. final : boolean Indicates if this is a final transcription or a partial result. timestamp : string The timestamp when the transcription was generated. user_id : string Identifier for the user who spoke. ​ bot-transcription 🤖 Transcription of the bot’s speech. Note: This protocol currently does not match the user transcription format to support real-time timestamping for bot transcriptions. Rather, the event is typically sent for each sentence of the bot’s response. This difference is currently due to limitations in TTS services which mostly do not support (or support well), accurate timing information. If/when this changes, this protocol may be updated to include the necessary timing information. For now, if you want to attempt real-time transcription to match your bot’s speaking, you can try using the bot-tts-text message type. type : 'bot-transcription' data : text : string The transcribed text from the bot, typically aggregated at a per-sentence level. ​ Client-Server Messaging ​ server-message 🤖 An arbitrary message sent from the server to the client. This can be used for custom interactions or commands. This message may be coupled with the client-message message type to handle responses from the client. type : 'server-message' data : any The data can be any JSON-serializable object, formatted according to your own specifications. ​ client-message 🏄 An arbitrary message sent from the client to the server. This can be used for custom interactions or commands. This message may be coupled with the server-response message type to handle responses from the server. type : 'client-message' data : t : string d : unknown (optional) The data payload should contain a t field indicating the type of message and an optional d field containing any custom, corresponding data needed for the message. ​ server-response 🤖 An message sent from the server to the client in response to a client-message . IMPORTANT : The id should match the id of the original client-message to correlate the response with the request. type : 'client-message' data : t : string d : unknown (optional) The data payload should contain a t field indicating the type of message and an optional d field containing any custom, corresponding data needed for the message. ​ error-response 🤖 Error response to a specific client message. IMPORTANT : The id should match the id of the original client-message to correlate the response with the request. type : 'error-response' data : error : string ​ Advanced LLM Interactions ​ append-to-context 🏄 A message sent from the client to the server to append data to the context of the current llm conversation. This is useful for providing text-based content for the user or augmenting the context for the assistant. type : 'append-to-context' data : role : "user" | "assistant" The role the context should be appended to. Currently only supports "user" and "assistant" . content : unknown The content to append to the context. This can be any data structure the llm understand. run_immediately : boolean (optional) Indicates whether the context should be run immediately after appending. Defaults to false . If set to false , the context will be appended but not executed until the next llm run. ​ llm-function-call 🤖 A function call request from the LLM, sent from the bot to the client. Note that for most cases, an LLM function call will be handled completely server-side. However, in the event that the call requires input from the client or the client needs to be aware of the function call, this message/response schema is required. type : 'llm-function-call' data : function_name : string Name of the function to be called. tool_call_id : string Unique identifier for this function call. args : Record<string, unknown> Arguments to be passed to the function. ​ llm-function-call-result 🏄 The result of the function call requested by the LLM, returned from the client. type : 'llm-function-call-result' data : function_name : string Name of the called function. tool_call_id : string Identifier matching the original function call. args : Record<string, unknown> Arguments that were passed to the function. result : Record<string, unknown> | string The result returned by the function. ​ bot-llm-search-response 🤖 Search results from the LLM’s knowledge base. Currently, Google Gemini is the only LLM that supports built-in search. However, we expect other LLMs to follow suite, which is why this message type is defined as part of the RTVI standard. As more LLMs add support for this feature, the format of this message type may evolve to accommodate discrepancies. type : 'bot-llm-search-response' data : search_result : string (optional) Raw search result text. rendered_content : string (optional) Formatted version of the search results. origins : Array<Origin Object> Source information and confidence scores for search results. The Origin Object follows this structure: Copy Ask AI { "site_uri" : string (optional) , "site_title" : string (optional) , "results" : Array< { "text" : string , "confidence" : number [] } > } Example: Copy Ask AI "id" : undefined "label" : "rtvi-ai" "type" : "bot-llm-search-response" "data" : { "origins" : [ { "results" : [ { "confidence" : [ 0.9881149530410768 ], "text" : "* Juneteenth: A Freedom Celebration is scheduled for June 18th from 12:00 pm to 2:00 pm." }, { "confidence" : [ 0.9692034721374512 ], "ext" : "* A Juneteenth celebration at Fort Negley Park will take place on June 19th from 5:00 pm to 9:30 pm." } ], "site_title" : "vanderbilt.edu" , "site_uri" : "https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQHwif83VK9KAzrbMSGSBsKwL8vWfSfC9pgEWYKmStHyqiRoV1oe8j1S0nbwRg_iWgqAr9wUkiegu3ATC8Ll-cuE-vpzwElRHiJ2KgRYcqnOQMoOeokVpWqi" }, { "results" : [ { "confidence" : [ 0.6554043292999268 ], "text" : "In addition to these events, Vanderbilt University is a large research institution with ongoing activities across many fields." } ], "site_title" : "wikipedia.org" , "site_uri" : "https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQESbF-ijx78QbaglrhflHCUWdPTD4M6tYOQigW5hgsHNctRlAHu9ktfPmJx7DfoP5QicE0y-OQY1cRl9w4Id0btiFgLYSKIm2-SPtOHXeNrAlgA7mBnclaGrD7rgnLIbrjl8DgUEJrrvT0CKzuo" }], "rendered_content" : "<style> \n .container ... </div> \n </div> \n " , "search_result" : "Several events are happening at Vanderbilt University: \n\n * Juneteenth: A Freedom Celebration is scheduled for June 18th from 12:00 pm to 2:00 pm. \n * A Juneteenth celebration at Fort Negley Park will take place on June 19th from 5:00 pm to 9:30 pm. \n\n In addition to these events, Vanderbilt University is a large research institution with ongoing activities across many fields. For the most recent news, you should check Vanderbilt's official news website. \n " } ​ Service-Specific Insights ​ bot-llm-started 🤖 Indicates LLM processing has begun type : bot-llm-started data : None ​ bot-llm-stopped 🤖 Indicates LLM processing has completed type : bot-llm-stopped data : None ​ user-llm-text 🤖 Aggregated user input text that is sent to the LLM. type : 'user-llm-text' data : text : string The user’s input text to be processed by the LLM. ​ bot-llm-text 🤖 Individual tokens streamed from the LLM as they are generated. type : 'bot-llm-text' data : text : string The token text from the LLM. ​ bot-tts-started 🤖 Indicates text-to-speech (TTS) processing has begun. type : 'bot-tts-started' data : None ​ bot-tts-stopped 🤖 Indicates text-to-speech (TTS) processing has completed. type : 'bot-tts-stopped' data : None ​ bot-tts-text 🤖 The per-token text output of the text-to-speech (TTS) service (what the TTS actually says). type : 'bot-tts-text' data : text : string The text representation of the generated bot speech. ​ Metrics and Monitoring ​ metrics 🤖 Performance metrics for various processing stages and services. Each message will contain entries for one or more of the metrics types: processing , ttfb , characters . type : 'metrics' data : processing : [See Below] (optional) Processing time metrics. ttfb : [See Below] (optional) Time to first byte metrics. characters : [See Below] (optional) Character processing metrics. For each metric type, the data structure is an array of objects with the following structure: processor : string The name of the processor or service that generated the metric. value : number The value of the metric, typically in milliseconds or character count. model : string (optional) The model of the service that generated the metric, if applicable. Example: Copy Ask AI { "type" : "metrics" , "data" : { "processing" : [ { "model" : "eleven_flash_v2_5" , "processor" : "ElevenLabsTTSService#0" , "value" : 0.0005140304565429688 } ], "ttfb" : [ { "model" : "eleven_flash_v2_5" , "processor" : "ElevenLabsTTSService#0" , "value" : 0.1573178768157959 } ], "characters" : [ { "model" : "eleven_flash_v2_5" , "processor" : "ElevenLabsTTSService#0" , "value" : 43 } ] } } Client SDKs RTVIClient Migration Guide On this page Key Features Terms RTVI Message Format RTVI Message Types Connection Management client-ready 🏄 bot-ready 🤖 disconnect-bot 🏄 error 🤖 Transcription user-started-speaking 🤖 user-stopped-speaking 🤖 bot-started-speaking 🤖 bot-stopped-speaking 🤖 user-transcription 🤖 bot-transcription 🤖 Client-Server Messaging server-message 🤖 client-message 🏄 server-response 🤖 error-response 🤖 Advanced LLM Interactions append-to-context 🏄 llm-function-call 🤖 llm-function-call-result 🏄 bot-llm-search-response 🤖 Service-Specific Insights bot-llm-started 🤖 bot-llm-stopped 🤖 user-llm-text 🤖 bot-llm-text 🤖 bot-tts-started 🤖 bot-tts-stopped 🤖 bot-tts-text 🤖 Metrics and Monitoring metrics 🤖 Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/client_rtvi-standard_edc3510d.txt b/client_rtvi-standard_edc3510d.txt
new file mode 100644
index 0000000000000000000000000000000000000000..98e64f9783d3851818ff3e4631c90c8bbc6c2a76
--- /dev/null
+++ b/client_rtvi-standard_edc3510d.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/client/rtvi-standard#param-platform-details
+Title: The RTVI Standard - Pipecat
+==================================================
+
+The RTVI Standard - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation The RTVI Standard Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Client SDKs The RTVI Standard RTVIClient Migration Guide Javascript SDK SDK Introduction API Reference Transport packages React SDK SDK Introduction API Reference React Native SDK SDK Introduction API Reference iOS SDK SDK Introduction API Reference Transport packages Android SDK SDK Introduction API Reference Transport packages C++ SDK SDK Introduction Daily WebRTC Transport The RTVI (Real-Time Voice and Video Inference) standard defines a set of message types and structures sent between clients and servers. It is designed to facilitate real-time interactions between clients and AI applications that require voice, video, and text communication. It provides a consistent framework for building applications that can communicate with AI models and the backends running those models in real-time. This page documents version 1.0 of the RTVI standard, released in June 2025. ​ Key Features Connection Management RTVI provides a flexible connection model that allows clients to connect to AI services and coordinate state. Transcriptions The standard includes built-in support for real-time transcription of audio streams. Client-Server Messaging The standard defines a messaging protocol for sending and receiving messages between clients and servers, allowing for efficient communication of requests and responses. Advanced LLM Interactions The standard supports advanced interactions with large language models (LLMs), including context management, function call handline, and search results. Service-Specific Insights RTVI supports events to provide insight into the input/output and state for typical services that exist in speech-to-speech workflows. Metrics and Monitoring RTVI provides mechanisms for collecting metrics and monitoring the performance of server-side services. ​ Terms Client : The front-end application or user interface that interacts with the RTVI server. Server : The backend-end service that runs the AI framework and processes requests from the client. User : The end user interacting with the client application. Bot : The AI interacting with the user, technically an amalgamation of a large language model (LLM) and a text-to-speech (TTS) service. ​ RTVI Message Format The messages defined as part of the RTVI protocol adhere to the following format: Copy Ask AI { "id" : string , "label" : "rtvi-ai" , "type" : string , "data" : unknown } ​ id string A unique identifier for the message, used to correlate requests and responses. ​ label string default: "rtvi-ai" required A label that identifies this message as an RTVI message. This field is required and should always be set to 'rtvi-ai' . ​ type string required The type of message being sent. This field is required and should be set to one of the predefined RTVI message types listed below. ​ data unknown The payload of the message, which can be any data structure relevant to the message type. ​ RTVI Message Types Following the above format, this section describes the various message types defined by the RTVI standard. Each message type has a specific purpose and structure, allowing for clear communication between clients and servers. Each message type below includes either a 🤖 or 🏄 emoji to denote whether the message is sent from the bot (🤖) or client (🏄). ​ Connection Management ​ client-ready 🏄 Indicates that the client is ready to receive messages and interact with the server. Typically sent after the transport media channels have connected. type : 'client-ready' data : version : string The version of the RTVI standard being used. This is useful for ensuring compatibility between client and server implementations. about : AboutClient Object An object containing information about the client, such as its rtvi-version, client library, and any other relevant metadata. The AboutClient object follows this structure: Show AboutClient ​ library string required ​ library_version string ​ platform string ​ platform_version string ​ platform_details any Any platform-specific details that may be relevant to the server. This could include information about the browser, operating system, or any other environment-specific data needed by the server. This field is optional and open-ended, so please be mindful of the data you include here and any security concerns that may arise from exposing sensitive or personal-identifiable information. ​ bot-ready 🤖 Indicates that the bot is ready to receive messages and interact with the client. Typically send after the transport media channels have connected. type : 'bot-ready' data : version : string The version of the RTVI standard being used. This is useful for ensuring compatibility between client and server implementations. about : any (Optional) An object containing information about the server or bot. It’s structure and value are both undefined by default. This provides flexibility to include any relevant metadata your client may need to know about the server at connection time, without any built-in security concerns. Please be mindful of the data you include here and any security concerns that may arise from exposing sensitive information. ​ disconnect-bot 🏄 Indicates that the client wishes to disconnect from the bot. Typically used when the client is shutting down or no longer needs to interact with the bot. Note: Disconnets should happen automatically when either the client or bot disconnects from the transport, so this message is intended for the case where a client may want to remain connected to the transport but no longer wishes to interact with the bot. type : 'disconnect-bot' data : undefined ​ error 🤖 Indicates an error occurred during bot initialization or runtime. type : 'error' data : message : string Description of the error. fatal : boolean Indicates if the error is fatal to the session. ​ Transcription ​ user-started-speaking 🤖 Emitted when the user begins speaking type : 'user-started-speaking' data : None ​ user-stopped-speaking 🤖 Emitted when the user stops speaking type : 'user-stopped-speaking' data : None ​ bot-started-speaking 🤖 Emitted when the bot begins speaking type : 'bot-started-speaking' data : None ​ bot-stopped-speaking 🤖 Emitted when the bot stops speaking type : 'bot-stopped-speaking' data : None ​ user-transcription 🤖 Real-time transcription of user speech, including both partial and final results. type : 'user-transcription' data : text : string The transcribed text of the user. final : boolean Indicates if this is a final transcription or a partial result. timestamp : string The timestamp when the transcription was generated. user_id : string Identifier for the user who spoke. ​ bot-transcription 🤖 Transcription of the bot’s speech. Note: This protocol currently does not match the user transcription format to support real-time timestamping for bot transcriptions. Rather, the event is typically sent for each sentence of the bot’s response. This difference is currently due to limitations in TTS services which mostly do not support (or support well), accurate timing information. If/when this changes, this protocol may be updated to include the necessary timing information. For now, if you want to attempt real-time transcription to match your bot’s speaking, you can try using the bot-tts-text message type. type : 'bot-transcription' data : text : string The transcribed text from the bot, typically aggregated at a per-sentence level. ​ Client-Server Messaging ​ server-message 🤖 An arbitrary message sent from the server to the client. This can be used for custom interactions or commands. This message may be coupled with the client-message message type to handle responses from the client. type : 'server-message' data : any The data can be any JSON-serializable object, formatted according to your own specifications. ​ client-message 🏄 An arbitrary message sent from the client to the server. This can be used for custom interactions or commands. This message may be coupled with the server-response message type to handle responses from the server. type : 'client-message' data : t : string d : unknown (optional) The data payload should contain a t field indicating the type of message and an optional d field containing any custom, corresponding data needed for the message. ​ server-response 🤖 An message sent from the server to the client in response to a client-message . IMPORTANT : The id should match the id of the original client-message to correlate the response with the request. type : 'client-message' data : t : string d : unknown (optional) The data payload should contain a t field indicating the type of message and an optional d field containing any custom, corresponding data needed for the message. ​ error-response 🤖 Error response to a specific client message. IMPORTANT : The id should match the id of the original client-message to correlate the response with the request. type : 'error-response' data : error : string ​ Advanced LLM Interactions ​ append-to-context 🏄 A message sent from the client to the server to append data to the context of the current llm conversation. This is useful for providing text-based content for the user or augmenting the context for the assistant. type : 'append-to-context' data : role : "user" | "assistant" The role the context should be appended to. Currently only supports "user" and "assistant" . content : unknown The content to append to the context. This can be any data structure the llm understand. run_immediately : boolean (optional) Indicates whether the context should be run immediately after appending. Defaults to false . If set to false , the context will be appended but not executed until the next llm run. ​ llm-function-call 🤖 A function call request from the LLM, sent from the bot to the client. Note that for most cases, an LLM function call will be handled completely server-side. However, in the event that the call requires input from the client or the client needs to be aware of the function call, this message/response schema is required. type : 'llm-function-call' data : function_name : string Name of the function to be called. tool_call_id : string Unique identifier for this function call. args : Record<string, unknown> Arguments to be passed to the function. ​ llm-function-call-result 🏄 The result of the function call requested by the LLM, returned from the client. type : 'llm-function-call-result' data : function_name : string Name of the called function. tool_call_id : string Identifier matching the original function call. args : Record<string, unknown> Arguments that were passed to the function. result : Record<string, unknown> | string The result returned by the function. ​ bot-llm-search-response 🤖 Search results from the LLM’s knowledge base. Currently, Google Gemini is the only LLM that supports built-in search. However, we expect other LLMs to follow suite, which is why this message type is defined as part of the RTVI standard. As more LLMs add support for this feature, the format of this message type may evolve to accommodate discrepancies. type : 'bot-llm-search-response' data : search_result : string (optional) Raw search result text. rendered_content : string (optional) Formatted version of the search results. origins : Array<Origin Object> Source information and confidence scores for search results. The Origin Object follows this structure: Copy Ask AI { "site_uri" : string (optional) , "site_title" : string (optional) , "results" : Array< { "text" : string , "confidence" : number [] } > } Example: Copy Ask AI "id" : undefined "label" : "rtvi-ai" "type" : "bot-llm-search-response" "data" : { "origins" : [ { "results" : [ { "confidence" : [ 0.9881149530410768 ], "text" : "* Juneteenth: A Freedom Celebration is scheduled for June 18th from 12:00 pm to 2:00 pm." }, { "confidence" : [ 0.9692034721374512 ], "ext" : "* A Juneteenth celebration at Fort Negley Park will take place on June 19th from 5:00 pm to 9:30 pm." } ], "site_title" : "vanderbilt.edu" , "site_uri" : "https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQHwif83VK9KAzrbMSGSBsKwL8vWfSfC9pgEWYKmStHyqiRoV1oe8j1S0nbwRg_iWgqAr9wUkiegu3ATC8Ll-cuE-vpzwElRHiJ2KgRYcqnOQMoOeokVpWqi" }, { "results" : [ { "confidence" : [ 0.6554043292999268 ], "text" : "In addition to these events, Vanderbilt University is a large research institution with ongoing activities across many fields." } ], "site_title" : "wikipedia.org" , "site_uri" : "https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQESbF-ijx78QbaglrhflHCUWdPTD4M6tYOQigW5hgsHNctRlAHu9ktfPmJx7DfoP5QicE0y-OQY1cRl9w4Id0btiFgLYSKIm2-SPtOHXeNrAlgA7mBnclaGrD7rgnLIbrjl8DgUEJrrvT0CKzuo" }], "rendered_content" : "<style> \n .container ... </div> \n </div> \n " , "search_result" : "Several events are happening at Vanderbilt University: \n\n * Juneteenth: A Freedom Celebration is scheduled for June 18th from 12:00 pm to 2:00 pm. \n * A Juneteenth celebration at Fort Negley Park will take place on June 19th from 5:00 pm to 9:30 pm. \n\n In addition to these events, Vanderbilt University is a large research institution with ongoing activities across many fields. For the most recent news, you should check Vanderbilt's official news website. \n " } ​ Service-Specific Insights ​ bot-llm-started 🤖 Indicates LLM processing has begun type : bot-llm-started data : None ​ bot-llm-stopped 🤖 Indicates LLM processing has completed type : bot-llm-stopped data : None ​ user-llm-text 🤖 Aggregated user input text that is sent to the LLM. type : 'user-llm-text' data : text : string The user’s input text to be processed by the LLM. ​ bot-llm-text 🤖 Individual tokens streamed from the LLM as they are generated. type : 'bot-llm-text' data : text : string The token text from the LLM. ​ bot-tts-started 🤖 Indicates text-to-speech (TTS) processing has begun. type : 'bot-tts-started' data : None ​ bot-tts-stopped 🤖 Indicates text-to-speech (TTS) processing has completed. type : 'bot-tts-stopped' data : None ​ bot-tts-text 🤖 The per-token text output of the text-to-speech (TTS) service (what the TTS actually says). type : 'bot-tts-text' data : text : string The text representation of the generated bot speech. ​ Metrics and Monitoring ​ metrics 🤖 Performance metrics for various processing stages and services. Each message will contain entries for one or more of the metrics types: processing , ttfb , characters . type : 'metrics' data : processing : [See Below] (optional) Processing time metrics. ttfb : [See Below] (optional) Time to first byte metrics. characters : [See Below] (optional) Character processing metrics. For each metric type, the data structure is an array of objects with the following structure: processor : string The name of the processor or service that generated the metric. value : number The value of the metric, typically in milliseconds or character count. model : string (optional) The model of the service that generated the metric, if applicable. Example: Copy Ask AI { "type" : "metrics" , "data" : { "processing" : [ { "model" : "eleven_flash_v2_5" , "processor" : "ElevenLabsTTSService#0" , "value" : 0.0005140304565429688 } ], "ttfb" : [ { "model" : "eleven_flash_v2_5" , "processor" : "ElevenLabsTTSService#0" , "value" : 0.1573178768157959 } ], "characters" : [ { "model" : "eleven_flash_v2_5" , "processor" : "ElevenLabsTTSService#0" , "value" : 43 } ] } } Client SDKs RTVIClient Migration Guide On this page Key Features Terms RTVI Message Format RTVI Message Types Connection Management client-ready 🏄 bot-ready 🤖 disconnect-bot 🏄 error 🤖 Transcription user-started-speaking 🤖 user-stopped-speaking 🤖 bot-started-speaking 🤖 bot-stopped-speaking 🤖 user-transcription 🤖 bot-transcription 🤖 Client-Server Messaging server-message 🤖 client-message 🏄 server-response 🤖 error-response 🤖 Advanced LLM Interactions append-to-context 🏄 llm-function-call 🤖 llm-function-call-result 🏄 bot-llm-search-response 🤖 Service-Specific Insights bot-llm-started 🤖 bot-llm-stopped 🤖 user-llm-text 🤖 bot-llm-text 🤖 bot-tts-started 🤖 bot-tts-stopped 🤖 bot-tts-text 🤖 Metrics and Monitoring metrics 🤖 Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/daily_rest-helpers_009954fb.txt b/daily_rest-helpers_009954fb.txt
new file mode 100644
index 0000000000000000000000000000000000000000..4e4dd219b15d39c20258713107fd752ef8c98f2b
--- /dev/null
+++ b/daily_rest-helpers_009954fb.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/daily/rest-helpers#get-name-from-url
+Title: Daily REST Helper - Pipecat
+==================================================
+
+Daily REST Helper - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Service Utilities Daily REST Helper Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Daily REST Helper Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline Daily REST API Documentation For complete Daily REST API reference and additional details ​ Classes ​ DailyRoomSipParams Configuration for SIP (Session Initiation Protocol) parameters. ​ display_name string default: "sw-sip-dialin" Display name for the SIP endpoint ​ video boolean default: false Whether video is enabled for SIP ​ sip_mode string default: "dial-in" SIP connection mode ​ num_endpoints integer default: 1 Number of SIP endpoints Copy Ask AI from pipecat.transports.services.helpers.daily_rest import DailyRoomSipParams sip_params = DailyRoomSipParams( display_name = "conference-line" , video = True , num_endpoints = 2 ) ​ RecordingsBucketConfig Configuration for storing Daily recordings in a custom S3 bucket. ​ bucket_name string required Name of the S3 bucket for storing recordings ​ bucket_region string required AWS region where the S3 bucket is located ​ assume_role_arn string required ARN of the IAM role to assume for S3 access ​ allow_api_access boolean default: false Whether to allow API access to the recordings Copy Ask AI from pipecat.transports.services.helpers.daily_rest import RecordingsBucketConfig bucket_config = RecordingsBucketConfig( bucket_name = "my-recordings-bucket" , bucket_region = "us-west-2" , assume_role_arn = "arn:aws:iam::123456789012:role/DailyRecordingsRole" , allow_api_access = True ) ​ DailyRoomProperties Properties that configure a Daily room’s behavior and features. ​ exp float Room expiration time as Unix timestamp (e.g., time.time() + 300 for 5 minutes) ​ enable_chat boolean default: false Whether chat is enabled in the room ​ enable_prejoin_ui boolean default: false Whether the prejoin lobby UI is enabled ​ enable_emoji_reactions boolean default: false Whether emoji reactions are enabled ​ eject_at_room_exp boolean default: false Whether to eject participants when room expires ​ enable_dialout boolean Whether dial-out is enabled ​ enable_recording string Recording settings (“cloud”, “local”, or “raw-tracks”) ​ geo string Geographic region for room ​ max_participants number Maximum number of participants allowed in the room ​ recordings_bucket RecordingsBucketConfig Configuration for custom S3 bucket recordings ​ sip DailyRoomSipParams SIP configuration parameters ​ sip_uri dict SIP URI configuration (returned by Daily) ​ start_video_off boolean default: false Whether the camera video is turned off by default The class also includes a sip_endpoint property that returns the SIP endpoint URI if available. Copy Ask AI import time from pipecat.transports.services.helpers.daily_rest import ( DailyRoomProperties, DailyRoomSipParams, RecordingsBucketConfig, ) properties = DailyRoomProperties( exp = time.time() + 3600 , # 1 hour from now enable_chat = True , enable_emoji_reactions = True , enable_recording = "cloud" , geo = "us-west" , max_participants = 50 , sip = DailyRoomSipParams( display_name = "conference" ), recordings_bucket = RecordingsBucketConfig( bucket_name = "my-bucket" , bucket_region = "us-west-2" , assume_role_arn = "arn:aws:iam::123456789012:role/DailyRole" ) ) # Access SIP endpoint if available if properties.sip_endpoint: print ( f "SIP endpoint: { properties.sip_endpoint } " ) ​ DailyRoomParams Parameters for creating a new Daily room. ​ name string Room name (if not provided, one will be generated) ​ privacy string default: "public" Room privacy setting (“private” or “public”) ​ properties DailyRoomProperties Room configuration properties Copy Ask AI import time from pipecat.transports.services.helpers.daily_rest import ( DailyRoomParams, DailyRoomProperties, ) params = DailyRoomParams( name = "team-meeting" , privacy = "private" , properties = DailyRoomProperties( enable_chat = True , exp = time.time() + 7200 # 2 hours from now ) ) ​ DailyRoomObject Response object representing a Daily room. ​ id string Unique room identifier ​ name string Room name ​ api_created boolean Whether the room was created via API ​ privacy string Room privacy setting ​ url string Complete room URL ​ created_at string Room creation timestamp in ISO 8601 format ​ config DailyRoomProperties Room configuration Copy Ask AI from pipecat.transports.services.helpers.daily_rest import ( DailyRoomObject, DailyRoomProperties, ) # Example of what a DailyRoomObject looks like when received room = DailyRoomObject( id = "abc123" , name = "team-meeting" , api_created = True , privacy = "private" , url = "https://your-domain.daily.co/team-meeting" , created_at = "2024-01-20T10:00:00.000Z" , config = DailyRoomProperties( enable_chat = True , exp = 1705743600 ) ) ​ DailyMeetingTokenProperties Properties for configuring a Daily meeting token. ​ room_name string The room this token is valid for. If not set, token is valid for all rooms. ​ eject_at_token_exp boolean Whether to eject user when token expires ​ eject_after_elapsed integer Eject user after this many seconds ​ nbf integer “Not before” timestamp - users cannot join before this time ​ exp integer Expiration timestamp - users cannot join after this time ​ is_owner boolean Whether token grants owner privileges ​ user_name string User’s display name in the meeting ​ user_id string Unique identifier for the user (36 char limit) ​ enable_screenshare boolean Whether user can share their screen ​ start_video_off boolean Whether to join with video off ​ start_audio_off boolean Whether to join with audio off ​ enable_recording string Recording settings (“cloud”, “local”, or “raw-tracks”) ​ enable_prejoin_ui boolean Whether to show prejoin UI ​ start_cloud_recording boolean Whether to start cloud recording when user joins ​ permissions dict Initial default permissions for a non-meeting-owner participant ​ DailyMeetingTokenParams Parameters for creating a Daily meeting token. ​ properties DailyMeetingTokenProperties Token configuration properties Copy Ask AI from pipecat.transports.services.helpers.daily_rest import ( DailyMeetingTokenParams, DailyMeetingTokenProperties, ) token_params = DailyMeetingTokenParams( properties = DailyMeetingTokenProperties( user_name = "John Doe" , enable_screenshare = True , start_video_off = True , permissions = { "canSend" : [ "video" , "audio" ]} ) ) ​ Initialize DailyRESTHelper Create a new instance of the Daily REST helper. ​ daily_api_key string required Your Daily API key ​ daily_api_url string default: "https://api.daily.co/v1" The Daily API base URL ​ aiohttp_session aiohttp.ClientSession required An aiohttp client session for making HTTP requests Copy Ask AI helper = DailyRESTHelper( daily_api_key = "your-api-key" , aiohttp_session = session ) ​ Create Room Creates a new Daily room with specified parameters. ​ params DailyRoomParams required Room configuration parameters including name, privacy, and properties Copy Ask AI # Create a room that expires in 1 hour params = DailyRoomParams( name = "my-room" , privacy = "private" , properties = DailyRoomProperties( exp = time.time() + 3600 , enable_chat = True ) ) room = await helper.create_room(params) print ( f "Room URL: { room.url } " ) ​ Get Room From URL Retrieves room information using a Daily room URL. ​ room_url string required The complete Daily room URL Copy Ask AI room = await helper.get_room_from_url( "https://your-domain.daily.co/my-room" ) print ( f "Room name: { room.name } " ) ​ Get Token Generates a meeting token for a specific room. ​ room_url string required The complete Daily room URL ​ expiry_time float default: "3600" Token expiration time in seconds ​ eject_at_token_exp bool default: "False" Whether to eject user when token expires ​ owner bool default: "True" Whether the token should have owner privileges (overrides any setting in params) ​ params DailyMeetingTokenParams Additional token configuration. Note that room_name , exp , eject_at_token_exp , and is_owner will be set based on the other function parameters. Copy Ask AI # Basic token generation token = await helper.get_token( room_url = "https://your-domain.daily.co/my-room" , expiry_time = 1800 , # 30 minutes owner = True , eject_at_token_exp = True ) # Advanced token generation with additional properties token_params = DailyMeetingTokenParams( properties = DailyMeetingTokenProperties( user_name = "John Doe" , start_video_off = True ) ) token = await helper.get_token( room_url = "https://your-domain.daily.co/my-room" , expiry_time = 1800 , owner = False , eject_at_token_exp = True , params = token_params ) ​ Delete Room By URL Deletes a room using its URL. ​ room_url string required The complete Daily room URL Copy Ask AI success = await helper.delete_room_by_url( "https://your-domain.daily.co/my-room" ) if success: print ( "Room deleted successfully" ) ​ Delete Room By Name Deletes a room using its name. ​ room_name string required The name of the Daily room Copy Ask AI success = await helper.delete_room_by_name( "my-room" ) if success: print ( "Room deleted successfully" ) ​ Get Name From URL Extracts the room name from a Daily room URL. ​ room_url string required The complete Daily room URL Copy Ask AI room_name = helper.get_name_from_url( "https://your-domain.daily.co/my-room" ) print ( f "Room name: { room_name } " ) # Outputs: "my-room" Turn Tracking Observer Smart Turn Overview On this page Classes DailyRoomSipParams RecordingsBucketConfig DailyRoomProperties DailyRoomParams DailyRoomObject DailyMeetingTokenProperties DailyMeetingTokenParams Initialize DailyRESTHelper Create Room Get Room From URL Get Token Delete Room By URL Delete Room By Name Get Name From URL Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/daily_rest-helpers_00a6c2c2.txt b/daily_rest-helpers_00a6c2c2.txt
new file mode 100644
index 0000000000000000000000000000000000000000..673cb3779fa5a95f6883fafa637c4f56137ffee8
--- /dev/null
+++ b/daily_rest-helpers_00a6c2c2.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/daily/rest-helpers#dailyroomproperties
+Title: Daily REST Helper - Pipecat
+==================================================
+
+Daily REST Helper - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Service Utilities Daily REST Helper Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Daily REST Helper Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline Daily REST API Documentation For complete Daily REST API reference and additional details ​ Classes ​ DailyRoomSipParams Configuration for SIP (Session Initiation Protocol) parameters. ​ display_name string default: "sw-sip-dialin" Display name for the SIP endpoint ​ video boolean default: false Whether video is enabled for SIP ​ sip_mode string default: "dial-in" SIP connection mode ​ num_endpoints integer default: 1 Number of SIP endpoints Copy Ask AI from pipecat.transports.services.helpers.daily_rest import DailyRoomSipParams sip_params = DailyRoomSipParams( display_name = "conference-line" , video = True , num_endpoints = 2 ) ​ RecordingsBucketConfig Configuration for storing Daily recordings in a custom S3 bucket. ​ bucket_name string required Name of the S3 bucket for storing recordings ​ bucket_region string required AWS region where the S3 bucket is located ​ assume_role_arn string required ARN of the IAM role to assume for S3 access ​ allow_api_access boolean default: false Whether to allow API access to the recordings Copy Ask AI from pipecat.transports.services.helpers.daily_rest import RecordingsBucketConfig bucket_config = RecordingsBucketConfig( bucket_name = "my-recordings-bucket" , bucket_region = "us-west-2" , assume_role_arn = "arn:aws:iam::123456789012:role/DailyRecordingsRole" , allow_api_access = True ) ​ DailyRoomProperties Properties that configure a Daily room’s behavior and features. ​ exp float Room expiration time as Unix timestamp (e.g., time.time() + 300 for 5 minutes) ​ enable_chat boolean default: false Whether chat is enabled in the room ​ enable_prejoin_ui boolean default: false Whether the prejoin lobby UI is enabled ​ enable_emoji_reactions boolean default: false Whether emoji reactions are enabled ​ eject_at_room_exp boolean default: false Whether to eject participants when room expires ​ enable_dialout boolean Whether dial-out is enabled ​ enable_recording string Recording settings (“cloud”, “local”, or “raw-tracks”) ​ geo string Geographic region for room ​ max_participants number Maximum number of participants allowed in the room ​ recordings_bucket RecordingsBucketConfig Configuration for custom S3 bucket recordings ​ sip DailyRoomSipParams SIP configuration parameters ​ sip_uri dict SIP URI configuration (returned by Daily) ​ start_video_off boolean default: false Whether the camera video is turned off by default The class also includes a sip_endpoint property that returns the SIP endpoint URI if available. Copy Ask AI import time from pipecat.transports.services.helpers.daily_rest import ( DailyRoomProperties, DailyRoomSipParams, RecordingsBucketConfig, ) properties = DailyRoomProperties( exp = time.time() + 3600 , # 1 hour from now enable_chat = True , enable_emoji_reactions = True , enable_recording = "cloud" , geo = "us-west" , max_participants = 50 , sip = DailyRoomSipParams( display_name = "conference" ), recordings_bucket = RecordingsBucketConfig( bucket_name = "my-bucket" , bucket_region = "us-west-2" , assume_role_arn = "arn:aws:iam::123456789012:role/DailyRole" ) ) # Access SIP endpoint if available if properties.sip_endpoint: print ( f "SIP endpoint: { properties.sip_endpoint } " ) ​ DailyRoomParams Parameters for creating a new Daily room. ​ name string Room name (if not provided, one will be generated) ​ privacy string default: "public" Room privacy setting (“private” or “public”) ​ properties DailyRoomProperties Room configuration properties Copy Ask AI import time from pipecat.transports.services.helpers.daily_rest import ( DailyRoomParams, DailyRoomProperties, ) params = DailyRoomParams( name = "team-meeting" , privacy = "private" , properties = DailyRoomProperties( enable_chat = True , exp = time.time() + 7200 # 2 hours from now ) ) ​ DailyRoomObject Response object representing a Daily room. ​ id string Unique room identifier ​ name string Room name ​ api_created boolean Whether the room was created via API ​ privacy string Room privacy setting ​ url string Complete room URL ​ created_at string Room creation timestamp in ISO 8601 format ​ config DailyRoomProperties Room configuration Copy Ask AI from pipecat.transports.services.helpers.daily_rest import ( DailyRoomObject, DailyRoomProperties, ) # Example of what a DailyRoomObject looks like when received room = DailyRoomObject( id = "abc123" , name = "team-meeting" , api_created = True , privacy = "private" , url = "https://your-domain.daily.co/team-meeting" , created_at = "2024-01-20T10:00:00.000Z" , config = DailyRoomProperties( enable_chat = True , exp = 1705743600 ) ) ​ DailyMeetingTokenProperties Properties for configuring a Daily meeting token. ​ room_name string The room this token is valid for. If not set, token is valid for all rooms. ​ eject_at_token_exp boolean Whether to eject user when token expires ​ eject_after_elapsed integer Eject user after this many seconds ​ nbf integer “Not before” timestamp - users cannot join before this time ​ exp integer Expiration timestamp - users cannot join after this time ​ is_owner boolean Whether token grants owner privileges ​ user_name string User’s display name in the meeting ​ user_id string Unique identifier for the user (36 char limit) ​ enable_screenshare boolean Whether user can share their screen ​ start_video_off boolean Whether to join with video off ​ start_audio_off boolean Whether to join with audio off ​ enable_recording string Recording settings (“cloud”, “local”, or “raw-tracks”) ​ enable_prejoin_ui boolean Whether to show prejoin UI ​ start_cloud_recording boolean Whether to start cloud recording when user joins ​ permissions dict Initial default permissions for a non-meeting-owner participant ​ DailyMeetingTokenParams Parameters for creating a Daily meeting token. ​ properties DailyMeetingTokenProperties Token configuration properties Copy Ask AI from pipecat.transports.services.helpers.daily_rest import ( DailyMeetingTokenParams, DailyMeetingTokenProperties, ) token_params = DailyMeetingTokenParams( properties = DailyMeetingTokenProperties( user_name = "John Doe" , enable_screenshare = True , start_video_off = True , permissions = { "canSend" : [ "video" , "audio" ]} ) ) ​ Initialize DailyRESTHelper Create a new instance of the Daily REST helper. ​ daily_api_key string required Your Daily API key ​ daily_api_url string default: "https://api.daily.co/v1" The Daily API base URL ​ aiohttp_session aiohttp.ClientSession required An aiohttp client session for making HTTP requests Copy Ask AI helper = DailyRESTHelper( daily_api_key = "your-api-key" , aiohttp_session = session ) ​ Create Room Creates a new Daily room with specified parameters. ​ params DailyRoomParams required Room configuration parameters including name, privacy, and properties Copy Ask AI # Create a room that expires in 1 hour params = DailyRoomParams( name = "my-room" , privacy = "private" , properties = DailyRoomProperties( exp = time.time() + 3600 , enable_chat = True ) ) room = await helper.create_room(params) print ( f "Room URL: { room.url } " ) ​ Get Room From URL Retrieves room information using a Daily room URL. ​ room_url string required The complete Daily room URL Copy Ask AI room = await helper.get_room_from_url( "https://your-domain.daily.co/my-room" ) print ( f "Room name: { room.name } " ) ​ Get Token Generates a meeting token for a specific room. ​ room_url string required The complete Daily room URL ​ expiry_time float default: "3600" Token expiration time in seconds ​ eject_at_token_exp bool default: "False" Whether to eject user when token expires ​ owner bool default: "True" Whether the token should have owner privileges (overrides any setting in params) ​ params DailyMeetingTokenParams Additional token configuration. Note that room_name , exp , eject_at_token_exp , and is_owner will be set based on the other function parameters. Copy Ask AI # Basic token generation token = await helper.get_token( room_url = "https://your-domain.daily.co/my-room" , expiry_time = 1800 , # 30 minutes owner = True , eject_at_token_exp = True ) # Advanced token generation with additional properties token_params = DailyMeetingTokenParams( properties = DailyMeetingTokenProperties( user_name = "John Doe" , start_video_off = True ) ) token = await helper.get_token( room_url = "https://your-domain.daily.co/my-room" , expiry_time = 1800 , owner = False , eject_at_token_exp = True , params = token_params ) ​ Delete Room By URL Deletes a room using its URL. ​ room_url string required The complete Daily room URL Copy Ask AI success = await helper.delete_room_by_url( "https://your-domain.daily.co/my-room" ) if success: print ( "Room deleted successfully" ) ​ Delete Room By Name Deletes a room using its name. ​ room_name string required The name of the Daily room Copy Ask AI success = await helper.delete_room_by_name( "my-room" ) if success: print ( "Room deleted successfully" ) ​ Get Name From URL Extracts the room name from a Daily room URL. ​ room_url string required The complete Daily room URL Copy Ask AI room_name = helper.get_name_from_url( "https://your-domain.daily.co/my-room" ) print ( f "Room name: { room_name } " ) # Outputs: "my-room" Turn Tracking Observer Smart Turn Overview On this page Classes DailyRoomSipParams RecordingsBucketConfig DailyRoomProperties DailyRoomParams DailyRoomObject DailyMeetingTokenProperties DailyMeetingTokenParams Initialize DailyRESTHelper Create Room Get Room From URL Get Token Delete Room By URL Delete Room By Name Get Name From URL Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/daily_rest-helpers_02ed9fd9.txt b/daily_rest-helpers_02ed9fd9.txt
new file mode 100644
index 0000000000000000000000000000000000000000..fa54714f129280aeb51d1c763389fe5189536e32
--- /dev/null
+++ b/daily_rest-helpers_02ed9fd9.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/daily/rest-helpers#param-user-name
+Title: Daily REST Helper - Pipecat
+==================================================
+
+Daily REST Helper - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Service Utilities Daily REST Helper Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Daily REST Helper Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline Daily REST API Documentation For complete Daily REST API reference and additional details ​ Classes ​ DailyRoomSipParams Configuration for SIP (Session Initiation Protocol) parameters. ​ display_name string default: "sw-sip-dialin" Display name for the SIP endpoint ​ video boolean default: false Whether video is enabled for SIP ​ sip_mode string default: "dial-in" SIP connection mode ​ num_endpoints integer default: 1 Number of SIP endpoints Copy Ask AI from pipecat.transports.services.helpers.daily_rest import DailyRoomSipParams sip_params = DailyRoomSipParams( display_name = "conference-line" , video = True , num_endpoints = 2 ) ​ RecordingsBucketConfig Configuration for storing Daily recordings in a custom S3 bucket. ​ bucket_name string required Name of the S3 bucket for storing recordings ​ bucket_region string required AWS region where the S3 bucket is located ​ assume_role_arn string required ARN of the IAM role to assume for S3 access ​ allow_api_access boolean default: false Whether to allow API access to the recordings Copy Ask AI from pipecat.transports.services.helpers.daily_rest import RecordingsBucketConfig bucket_config = RecordingsBucketConfig( bucket_name = "my-recordings-bucket" , bucket_region = "us-west-2" , assume_role_arn = "arn:aws:iam::123456789012:role/DailyRecordingsRole" , allow_api_access = True ) ​ DailyRoomProperties Properties that configure a Daily room’s behavior and features. ​ exp float Room expiration time as Unix timestamp (e.g., time.time() + 300 for 5 minutes) ​ enable_chat boolean default: false Whether chat is enabled in the room ​ enable_prejoin_ui boolean default: false Whether the prejoin lobby UI is enabled ​ enable_emoji_reactions boolean default: false Whether emoji reactions are enabled ​ eject_at_room_exp boolean default: false Whether to eject participants when room expires ​ enable_dialout boolean Whether dial-out is enabled ​ enable_recording string Recording settings (“cloud”, “local”, or “raw-tracks”) ​ geo string Geographic region for room ​ max_participants number Maximum number of participants allowed in the room ​ recordings_bucket RecordingsBucketConfig Configuration for custom S3 bucket recordings ​ sip DailyRoomSipParams SIP configuration parameters ​ sip_uri dict SIP URI configuration (returned by Daily) ​ start_video_off boolean default: false Whether the camera video is turned off by default The class also includes a sip_endpoint property that returns the SIP endpoint URI if available. Copy Ask AI import time from pipecat.transports.services.helpers.daily_rest import ( DailyRoomProperties, DailyRoomSipParams, RecordingsBucketConfig, ) properties = DailyRoomProperties( exp = time.time() + 3600 , # 1 hour from now enable_chat = True , enable_emoji_reactions = True , enable_recording = "cloud" , geo = "us-west" , max_participants = 50 , sip = DailyRoomSipParams( display_name = "conference" ), recordings_bucket = RecordingsBucketConfig( bucket_name = "my-bucket" , bucket_region = "us-west-2" , assume_role_arn = "arn:aws:iam::123456789012:role/DailyRole" ) ) # Access SIP endpoint if available if properties.sip_endpoint: print ( f "SIP endpoint: { properties.sip_endpoint } " ) ​ DailyRoomParams Parameters for creating a new Daily room. ​ name string Room name (if not provided, one will be generated) ​ privacy string default: "public" Room privacy setting (“private” or “public”) ​ properties DailyRoomProperties Room configuration properties Copy Ask AI import time from pipecat.transports.services.helpers.daily_rest import ( DailyRoomParams, DailyRoomProperties, ) params = DailyRoomParams( name = "team-meeting" , privacy = "private" , properties = DailyRoomProperties( enable_chat = True , exp = time.time() + 7200 # 2 hours from now ) ) ​ DailyRoomObject Response object representing a Daily room. ​ id string Unique room identifier ​ name string Room name ​ api_created boolean Whether the room was created via API ​ privacy string Room privacy setting ​ url string Complete room URL ​ created_at string Room creation timestamp in ISO 8601 format ​ config DailyRoomProperties Room configuration Copy Ask AI from pipecat.transports.services.helpers.daily_rest import ( DailyRoomObject, DailyRoomProperties, ) # Example of what a DailyRoomObject looks like when received room = DailyRoomObject( id = "abc123" , name = "team-meeting" , api_created = True , privacy = "private" , url = "https://your-domain.daily.co/team-meeting" , created_at = "2024-01-20T10:00:00.000Z" , config = DailyRoomProperties( enable_chat = True , exp = 1705743600 ) ) ​ DailyMeetingTokenProperties Properties for configuring a Daily meeting token. ​ room_name string The room this token is valid for. If not set, token is valid for all rooms. ​ eject_at_token_exp boolean Whether to eject user when token expires ​ eject_after_elapsed integer Eject user after this many seconds ​ nbf integer “Not before” timestamp - users cannot join before this time ​ exp integer Expiration timestamp - users cannot join after this time ​ is_owner boolean Whether token grants owner privileges ​ user_name string User’s display name in the meeting ​ user_id string Unique identifier for the user (36 char limit) ​ enable_screenshare boolean Whether user can share their screen ​ start_video_off boolean Whether to join with video off ​ start_audio_off boolean Whether to join with audio off ​ enable_recording string Recording settings (“cloud”, “local”, or “raw-tracks”) ​ enable_prejoin_ui boolean Whether to show prejoin UI ​ start_cloud_recording boolean Whether to start cloud recording when user joins ​ permissions dict Initial default permissions for a non-meeting-owner participant ​ DailyMeetingTokenParams Parameters for creating a Daily meeting token. ​ properties DailyMeetingTokenProperties Token configuration properties Copy Ask AI from pipecat.transports.services.helpers.daily_rest import ( DailyMeetingTokenParams, DailyMeetingTokenProperties, ) token_params = DailyMeetingTokenParams( properties = DailyMeetingTokenProperties( user_name = "John Doe" , enable_screenshare = True , start_video_off = True , permissions = { "canSend" : [ "video" , "audio" ]} ) ) ​ Initialize DailyRESTHelper Create a new instance of the Daily REST helper. ​ daily_api_key string required Your Daily API key ​ daily_api_url string default: "https://api.daily.co/v1" The Daily API base URL ​ aiohttp_session aiohttp.ClientSession required An aiohttp client session for making HTTP requests Copy Ask AI helper = DailyRESTHelper( daily_api_key = "your-api-key" , aiohttp_session = session ) ​ Create Room Creates a new Daily room with specified parameters. ​ params DailyRoomParams required Room configuration parameters including name, privacy, and properties Copy Ask AI # Create a room that expires in 1 hour params = DailyRoomParams( name = "my-room" , privacy = "private" , properties = DailyRoomProperties( exp = time.time() + 3600 , enable_chat = True ) ) room = await helper.create_room(params) print ( f "Room URL: { room.url } " ) ​ Get Room From URL Retrieves room information using a Daily room URL. ​ room_url string required The complete Daily room URL Copy Ask AI room = await helper.get_room_from_url( "https://your-domain.daily.co/my-room" ) print ( f "Room name: { room.name } " ) ​ Get Token Generates a meeting token for a specific room. ​ room_url string required The complete Daily room URL ​ expiry_time float default: "3600" Token expiration time in seconds ​ eject_at_token_exp bool default: "False" Whether to eject user when token expires ​ owner bool default: "True" Whether the token should have owner privileges (overrides any setting in params) ​ params DailyMeetingTokenParams Additional token configuration. Note that room_name , exp , eject_at_token_exp , and is_owner will be set based on the other function parameters. Copy Ask AI # Basic token generation token = await helper.get_token( room_url = "https://your-domain.daily.co/my-room" , expiry_time = 1800 , # 30 minutes owner = True , eject_at_token_exp = True ) # Advanced token generation with additional properties token_params = DailyMeetingTokenParams( properties = DailyMeetingTokenProperties( user_name = "John Doe" , start_video_off = True ) ) token = await helper.get_token( room_url = "https://your-domain.daily.co/my-room" , expiry_time = 1800 , owner = False , eject_at_token_exp = True , params = token_params ) ​ Delete Room By URL Deletes a room using its URL. ​ room_url string required The complete Daily room URL Copy Ask AI success = await helper.delete_room_by_url( "https://your-domain.daily.co/my-room" ) if success: print ( "Room deleted successfully" ) ​ Delete Room By Name Deletes a room using its name. ​ room_name string required The name of the Daily room Copy Ask AI success = await helper.delete_room_by_name( "my-room" ) if success: print ( "Room deleted successfully" ) ​ Get Name From URL Extracts the room name from a Daily room URL. ​ room_url string required The complete Daily room URL Copy Ask AI room_name = helper.get_name_from_url( "https://your-domain.daily.co/my-room" ) print ( f "Room name: { room_name } " ) # Outputs: "my-room" Turn Tracking Observer Smart Turn Overview On this page Classes DailyRoomSipParams RecordingsBucketConfig DailyRoomProperties DailyRoomParams DailyRoomObject DailyMeetingTokenProperties DailyMeetingTokenParams Initialize DailyRESTHelper Create Room Get Room From URL Get Token Delete Room By URL Delete Room By Name Get Name From URL Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/daily_rest-helpers_0e01defa.txt b/daily_rest-helpers_0e01defa.txt
new file mode 100644
index 0000000000000000000000000000000000000000..c7079dd845e94a4d6fad38a4ddf34ff317fdd923
--- /dev/null
+++ b/daily_rest-helpers_0e01defa.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/daily/rest-helpers#dailymeetingtokenparams
+Title: Daily REST Helper - Pipecat
+==================================================
+
+Daily REST Helper - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Service Utilities Daily REST Helper Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Daily REST Helper Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline Daily REST API Documentation For complete Daily REST API reference and additional details ​ Classes ​ DailyRoomSipParams Configuration for SIP (Session Initiation Protocol) parameters. ​ display_name string default: "sw-sip-dialin" Display name for the SIP endpoint ​ video boolean default: false Whether video is enabled for SIP ​ sip_mode string default: "dial-in" SIP connection mode ​ num_endpoints integer default: 1 Number of SIP endpoints Copy Ask AI from pipecat.transports.services.helpers.daily_rest import DailyRoomSipParams sip_params = DailyRoomSipParams( display_name = "conference-line" , video = True , num_endpoints = 2 ) ​ RecordingsBucketConfig Configuration for storing Daily recordings in a custom S3 bucket. ​ bucket_name string required Name of the S3 bucket for storing recordings ​ bucket_region string required AWS region where the S3 bucket is located ​ assume_role_arn string required ARN of the IAM role to assume for S3 access ​ allow_api_access boolean default: false Whether to allow API access to the recordings Copy Ask AI from pipecat.transports.services.helpers.daily_rest import RecordingsBucketConfig bucket_config = RecordingsBucketConfig( bucket_name = "my-recordings-bucket" , bucket_region = "us-west-2" , assume_role_arn = "arn:aws:iam::123456789012:role/DailyRecordingsRole" , allow_api_access = True ) ​ DailyRoomProperties Properties that configure a Daily room’s behavior and features. ​ exp float Room expiration time as Unix timestamp (e.g., time.time() + 300 for 5 minutes) ​ enable_chat boolean default: false Whether chat is enabled in the room ​ enable_prejoin_ui boolean default: false Whether the prejoin lobby UI is enabled ​ enable_emoji_reactions boolean default: false Whether emoji reactions are enabled ​ eject_at_room_exp boolean default: false Whether to eject participants when room expires ​ enable_dialout boolean Whether dial-out is enabled ​ enable_recording string Recording settings (“cloud”, “local”, or “raw-tracks”) ​ geo string Geographic region for room ​ max_participants number Maximum number of participants allowed in the room ​ recordings_bucket RecordingsBucketConfig Configuration for custom S3 bucket recordings ​ sip DailyRoomSipParams SIP configuration parameters ​ sip_uri dict SIP URI configuration (returned by Daily) ​ start_video_off boolean default: false Whether the camera video is turned off by default The class also includes a sip_endpoint property that returns the SIP endpoint URI if available. Copy Ask AI import time from pipecat.transports.services.helpers.daily_rest import ( DailyRoomProperties, DailyRoomSipParams, RecordingsBucketConfig, ) properties = DailyRoomProperties( exp = time.time() + 3600 , # 1 hour from now enable_chat = True , enable_emoji_reactions = True , enable_recording = "cloud" , geo = "us-west" , max_participants = 50 , sip = DailyRoomSipParams( display_name = "conference" ), recordings_bucket = RecordingsBucketConfig( bucket_name = "my-bucket" , bucket_region = "us-west-2" , assume_role_arn = "arn:aws:iam::123456789012:role/DailyRole" ) ) # Access SIP endpoint if available if properties.sip_endpoint: print ( f "SIP endpoint: { properties.sip_endpoint } " ) ​ DailyRoomParams Parameters for creating a new Daily room. ​ name string Room name (if not provided, one will be generated) ​ privacy string default: "public" Room privacy setting (“private” or “public”) ​ properties DailyRoomProperties Room configuration properties Copy Ask AI import time from pipecat.transports.services.helpers.daily_rest import ( DailyRoomParams, DailyRoomProperties, ) params = DailyRoomParams( name = "team-meeting" , privacy = "private" , properties = DailyRoomProperties( enable_chat = True , exp = time.time() + 7200 # 2 hours from now ) ) ​ DailyRoomObject Response object representing a Daily room. ​ id string Unique room identifier ​ name string Room name ​ api_created boolean Whether the room was created via API ​ privacy string Room privacy setting ​ url string Complete room URL ​ created_at string Room creation timestamp in ISO 8601 format ​ config DailyRoomProperties Room configuration Copy Ask AI from pipecat.transports.services.helpers.daily_rest import ( DailyRoomObject, DailyRoomProperties, ) # Example of what a DailyRoomObject looks like when received room = DailyRoomObject( id = "abc123" , name = "team-meeting" , api_created = True , privacy = "private" , url = "https://your-domain.daily.co/team-meeting" , created_at = "2024-01-20T10:00:00.000Z" , config = DailyRoomProperties( enable_chat = True , exp = 1705743600 ) ) ​ DailyMeetingTokenProperties Properties for configuring a Daily meeting token. ​ room_name string The room this token is valid for. If not set, token is valid for all rooms. ​ eject_at_token_exp boolean Whether to eject user when token expires ​ eject_after_elapsed integer Eject user after this many seconds ​ nbf integer “Not before” timestamp - users cannot join before this time ​ exp integer Expiration timestamp - users cannot join after this time ​ is_owner boolean Whether token grants owner privileges ​ user_name string User’s display name in the meeting ​ user_id string Unique identifier for the user (36 char limit) ​ enable_screenshare boolean Whether user can share their screen ​ start_video_off boolean Whether to join with video off ​ start_audio_off boolean Whether to join with audio off ​ enable_recording string Recording settings (“cloud”, “local”, or “raw-tracks”) ​ enable_prejoin_ui boolean Whether to show prejoin UI ​ start_cloud_recording boolean Whether to start cloud recording when user joins ​ permissions dict Initial default permissions for a non-meeting-owner participant ​ DailyMeetingTokenParams Parameters for creating a Daily meeting token. ​ properties DailyMeetingTokenProperties Token configuration properties Copy Ask AI from pipecat.transports.services.helpers.daily_rest import ( DailyMeetingTokenParams, DailyMeetingTokenProperties, ) token_params = DailyMeetingTokenParams( properties = DailyMeetingTokenProperties( user_name = "John Doe" , enable_screenshare = True , start_video_off = True , permissions = { "canSend" : [ "video" , "audio" ]} ) ) ​ Initialize DailyRESTHelper Create a new instance of the Daily REST helper. ​ daily_api_key string required Your Daily API key ​ daily_api_url string default: "https://api.daily.co/v1" The Daily API base URL ​ aiohttp_session aiohttp.ClientSession required An aiohttp client session for making HTTP requests Copy Ask AI helper = DailyRESTHelper( daily_api_key = "your-api-key" , aiohttp_session = session ) ​ Create Room Creates a new Daily room with specified parameters. ​ params DailyRoomParams required Room configuration parameters including name, privacy, and properties Copy Ask AI # Create a room that expires in 1 hour params = DailyRoomParams( name = "my-room" , privacy = "private" , properties = DailyRoomProperties( exp = time.time() + 3600 , enable_chat = True ) ) room = await helper.create_room(params) print ( f "Room URL: { room.url } " ) ​ Get Room From URL Retrieves room information using a Daily room URL. ​ room_url string required The complete Daily room URL Copy Ask AI room = await helper.get_room_from_url( "https://your-domain.daily.co/my-room" ) print ( f "Room name: { room.name } " ) ​ Get Token Generates a meeting token for a specific room. ​ room_url string required The complete Daily room URL ​ expiry_time float default: "3600" Token expiration time in seconds ​ eject_at_token_exp bool default: "False" Whether to eject user when token expires ​ owner bool default: "True" Whether the token should have owner privileges (overrides any setting in params) ​ params DailyMeetingTokenParams Additional token configuration. Note that room_name , exp , eject_at_token_exp , and is_owner will be set based on the other function parameters. Copy Ask AI # Basic token generation token = await helper.get_token( room_url = "https://your-domain.daily.co/my-room" , expiry_time = 1800 , # 30 minutes owner = True , eject_at_token_exp = True ) # Advanced token generation with additional properties token_params = DailyMeetingTokenParams( properties = DailyMeetingTokenProperties( user_name = "John Doe" , start_video_off = True ) ) token = await helper.get_token( room_url = "https://your-domain.daily.co/my-room" , expiry_time = 1800 , owner = False , eject_at_token_exp = True , params = token_params ) ​ Delete Room By URL Deletes a room using its URL. ​ room_url string required The complete Daily room URL Copy Ask AI success = await helper.delete_room_by_url( "https://your-domain.daily.co/my-room" ) if success: print ( "Room deleted successfully" ) ​ Delete Room By Name Deletes a room using its name. ​ room_name string required The name of the Daily room Copy Ask AI success = await helper.delete_room_by_name( "my-room" ) if success: print ( "Room deleted successfully" ) ​ Get Name From URL Extracts the room name from a Daily room URL. ​ room_url string required The complete Daily room URL Copy Ask AI room_name = helper.get_name_from_url( "https://your-domain.daily.co/my-room" ) print ( f "Room name: { room_name } " ) # Outputs: "my-room" Turn Tracking Observer Smart Turn Overview On this page Classes DailyRoomSipParams RecordingsBucketConfig DailyRoomProperties DailyRoomParams DailyRoomObject DailyMeetingTokenProperties DailyMeetingTokenParams Initialize DailyRESTHelper Create Room Get Room From URL Get Token Delete Room By URL Delete Room By Name Get Name From URL Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/daily_rest-helpers_145e485a.txt b/daily_rest-helpers_145e485a.txt
new file mode 100644
index 0000000000000000000000000000000000000000..a2fd4ccd33170fac15b0c7f18117d74933065bda
--- /dev/null
+++ b/daily_rest-helpers_145e485a.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/daily/rest-helpers#dailyroomparams
+Title: Daily REST Helper - Pipecat
+==================================================
+
+Daily REST Helper - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Service Utilities Daily REST Helper Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Daily REST Helper Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline Daily REST API Documentation For complete Daily REST API reference and additional details ​ Classes ​ DailyRoomSipParams Configuration for SIP (Session Initiation Protocol) parameters. ​ display_name string default: "sw-sip-dialin" Display name for the SIP endpoint ​ video boolean default: false Whether video is enabled for SIP ​ sip_mode string default: "dial-in" SIP connection mode ​ num_endpoints integer default: 1 Number of SIP endpoints Copy Ask AI from pipecat.transports.services.helpers.daily_rest import DailyRoomSipParams sip_params = DailyRoomSipParams( display_name = "conference-line" , video = True , num_endpoints = 2 ) ​ RecordingsBucketConfig Configuration for storing Daily recordings in a custom S3 bucket. ​ bucket_name string required Name of the S3 bucket for storing recordings ​ bucket_region string required AWS region where the S3 bucket is located ​ assume_role_arn string required ARN of the IAM role to assume for S3 access ​ allow_api_access boolean default: false Whether to allow API access to the recordings Copy Ask AI from pipecat.transports.services.helpers.daily_rest import RecordingsBucketConfig bucket_config = RecordingsBucketConfig( bucket_name = "my-recordings-bucket" , bucket_region = "us-west-2" , assume_role_arn = "arn:aws:iam::123456789012:role/DailyRecordingsRole" , allow_api_access = True ) ​ DailyRoomProperties Properties that configure a Daily room’s behavior and features. ​ exp float Room expiration time as Unix timestamp (e.g., time.time() + 300 for 5 minutes) ​ enable_chat boolean default: false Whether chat is enabled in the room ​ enable_prejoin_ui boolean default: false Whether the prejoin lobby UI is enabled ​ enable_emoji_reactions boolean default: false Whether emoji reactions are enabled ​ eject_at_room_exp boolean default: false Whether to eject participants when room expires ​ enable_dialout boolean Whether dial-out is enabled ​ enable_recording string Recording settings (“cloud”, “local”, or “raw-tracks”) ​ geo string Geographic region for room ​ max_participants number Maximum number of participants allowed in the room ​ recordings_bucket RecordingsBucketConfig Configuration for custom S3 bucket recordings ​ sip DailyRoomSipParams SIP configuration parameters ​ sip_uri dict SIP URI configuration (returned by Daily) ​ start_video_off boolean default: false Whether the camera video is turned off by default The class also includes a sip_endpoint property that returns the SIP endpoint URI if available. Copy Ask AI import time from pipecat.transports.services.helpers.daily_rest import ( DailyRoomProperties, DailyRoomSipParams, RecordingsBucketConfig, ) properties = DailyRoomProperties( exp = time.time() + 3600 , # 1 hour from now enable_chat = True , enable_emoji_reactions = True , enable_recording = "cloud" , geo = "us-west" , max_participants = 50 , sip = DailyRoomSipParams( display_name = "conference" ), recordings_bucket = RecordingsBucketConfig( bucket_name = "my-bucket" , bucket_region = "us-west-2" , assume_role_arn = "arn:aws:iam::123456789012:role/DailyRole" ) ) # Access SIP endpoint if available if properties.sip_endpoint: print ( f "SIP endpoint: { properties.sip_endpoint } " ) ​ DailyRoomParams Parameters for creating a new Daily room. ​ name string Room name (if not provided, one will be generated) ​ privacy string default: "public" Room privacy setting (“private” or “public”) ​ properties DailyRoomProperties Room configuration properties Copy Ask AI import time from pipecat.transports.services.helpers.daily_rest import ( DailyRoomParams, DailyRoomProperties, ) params = DailyRoomParams( name = "team-meeting" , privacy = "private" , properties = DailyRoomProperties( enable_chat = True , exp = time.time() + 7200 # 2 hours from now ) ) ​ DailyRoomObject Response object representing a Daily room. ​ id string Unique room identifier ​ name string Room name ​ api_created boolean Whether the room was created via API ​ privacy string Room privacy setting ​ url string Complete room URL ​ created_at string Room creation timestamp in ISO 8601 format ​ config DailyRoomProperties Room configuration Copy Ask AI from pipecat.transports.services.helpers.daily_rest import ( DailyRoomObject, DailyRoomProperties, ) # Example of what a DailyRoomObject looks like when received room = DailyRoomObject( id = "abc123" , name = "team-meeting" , api_created = True , privacy = "private" , url = "https://your-domain.daily.co/team-meeting" , created_at = "2024-01-20T10:00:00.000Z" , config = DailyRoomProperties( enable_chat = True , exp = 1705743600 ) ) ​ DailyMeetingTokenProperties Properties for configuring a Daily meeting token. ​ room_name string The room this token is valid for. If not set, token is valid for all rooms. ​ eject_at_token_exp boolean Whether to eject user when token expires ​ eject_after_elapsed integer Eject user after this many seconds ​ nbf integer “Not before” timestamp - users cannot join before this time ​ exp integer Expiration timestamp - users cannot join after this time ​ is_owner boolean Whether token grants owner privileges ​ user_name string User’s display name in the meeting ​ user_id string Unique identifier for the user (36 char limit) ​ enable_screenshare boolean Whether user can share their screen ​ start_video_off boolean Whether to join with video off ​ start_audio_off boolean Whether to join with audio off ​ enable_recording string Recording settings (“cloud”, “local”, or “raw-tracks”) ​ enable_prejoin_ui boolean Whether to show prejoin UI ​ start_cloud_recording boolean Whether to start cloud recording when user joins ​ permissions dict Initial default permissions for a non-meeting-owner participant ​ DailyMeetingTokenParams Parameters for creating a Daily meeting token. ​ properties DailyMeetingTokenProperties Token configuration properties Copy Ask AI from pipecat.transports.services.helpers.daily_rest import ( DailyMeetingTokenParams, DailyMeetingTokenProperties, ) token_params = DailyMeetingTokenParams( properties = DailyMeetingTokenProperties( user_name = "John Doe" , enable_screenshare = True , start_video_off = True , permissions = { "canSend" : [ "video" , "audio" ]} ) ) ​ Initialize DailyRESTHelper Create a new instance of the Daily REST helper. ​ daily_api_key string required Your Daily API key ​ daily_api_url string default: "https://api.daily.co/v1" The Daily API base URL ​ aiohttp_session aiohttp.ClientSession required An aiohttp client session for making HTTP requests Copy Ask AI helper = DailyRESTHelper( daily_api_key = "your-api-key" , aiohttp_session = session ) ​ Create Room Creates a new Daily room with specified parameters. ​ params DailyRoomParams required Room configuration parameters including name, privacy, and properties Copy Ask AI # Create a room that expires in 1 hour params = DailyRoomParams( name = "my-room" , privacy = "private" , properties = DailyRoomProperties( exp = time.time() + 3600 , enable_chat = True ) ) room = await helper.create_room(params) print ( f "Room URL: { room.url } " ) ​ Get Room From URL Retrieves room information using a Daily room URL. ​ room_url string required The complete Daily room URL Copy Ask AI room = await helper.get_room_from_url( "https://your-domain.daily.co/my-room" ) print ( f "Room name: { room.name } " ) ​ Get Token Generates a meeting token for a specific room. ​ room_url string required The complete Daily room URL ​ expiry_time float default: "3600" Token expiration time in seconds ​ eject_at_token_exp bool default: "False" Whether to eject user when token expires ​ owner bool default: "True" Whether the token should have owner privileges (overrides any setting in params) ​ params DailyMeetingTokenParams Additional token configuration. Note that room_name , exp , eject_at_token_exp , and is_owner will be set based on the other function parameters. Copy Ask AI # Basic token generation token = await helper.get_token( room_url = "https://your-domain.daily.co/my-room" , expiry_time = 1800 , # 30 minutes owner = True , eject_at_token_exp = True ) # Advanced token generation with additional properties token_params = DailyMeetingTokenParams( properties = DailyMeetingTokenProperties( user_name = "John Doe" , start_video_off = True ) ) token = await helper.get_token( room_url = "https://your-domain.daily.co/my-room" , expiry_time = 1800 , owner = False , eject_at_token_exp = True , params = token_params ) ​ Delete Room By URL Deletes a room using its URL. ​ room_url string required The complete Daily room URL Copy Ask AI success = await helper.delete_room_by_url( "https://your-domain.daily.co/my-room" ) if success: print ( "Room deleted successfully" ) ​ Delete Room By Name Deletes a room using its name. ​ room_name string required The name of the Daily room Copy Ask AI success = await helper.delete_room_by_name( "my-room" ) if success: print ( "Room deleted successfully" ) ​ Get Name From URL Extracts the room name from a Daily room URL. ​ room_url string required The complete Daily room URL Copy Ask AI room_name = helper.get_name_from_url( "https://your-domain.daily.co/my-room" ) print ( f "Room name: { room_name } " ) # Outputs: "my-room" Turn Tracking Observer Smart Turn Overview On this page Classes DailyRoomSipParams RecordingsBucketConfig DailyRoomProperties DailyRoomParams DailyRoomObject DailyMeetingTokenProperties DailyMeetingTokenParams Initialize DailyRESTHelper Create Room Get Room From URL Get Token Delete Room By URL Delete Room By Name Get Name From URL Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/daily_rest-helpers_2e9d64de.txt b/daily_rest-helpers_2e9d64de.txt
new file mode 100644
index 0000000000000000000000000000000000000000..68aafa5fadb3df662da99b86daa961a3d90baecc
--- /dev/null
+++ b/daily_rest-helpers_2e9d64de.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/daily/rest-helpers#recordingsbucketconfig
+Title: Daily REST Helper - Pipecat
+==================================================
+
+Daily REST Helper - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Service Utilities Daily REST Helper Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Daily REST Helper Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline Daily REST API Documentation For complete Daily REST API reference and additional details ​ Classes ​ DailyRoomSipParams Configuration for SIP (Session Initiation Protocol) parameters. ​ display_name string default: "sw-sip-dialin" Display name for the SIP endpoint ​ video boolean default: false Whether video is enabled for SIP ​ sip_mode string default: "dial-in" SIP connection mode ​ num_endpoints integer default: 1 Number of SIP endpoints Copy Ask AI from pipecat.transports.services.helpers.daily_rest import DailyRoomSipParams sip_params = DailyRoomSipParams( display_name = "conference-line" , video = True , num_endpoints = 2 ) ​ RecordingsBucketConfig Configuration for storing Daily recordings in a custom S3 bucket. ​ bucket_name string required Name of the S3 bucket for storing recordings ​ bucket_region string required AWS region where the S3 bucket is located ​ assume_role_arn string required ARN of the IAM role to assume for S3 access ​ allow_api_access boolean default: false Whether to allow API access to the recordings Copy Ask AI from pipecat.transports.services.helpers.daily_rest import RecordingsBucketConfig bucket_config = RecordingsBucketConfig( bucket_name = "my-recordings-bucket" , bucket_region = "us-west-2" , assume_role_arn = "arn:aws:iam::123456789012:role/DailyRecordingsRole" , allow_api_access = True ) ​ DailyRoomProperties Properties that configure a Daily room’s behavior and features. ​ exp float Room expiration time as Unix timestamp (e.g., time.time() + 300 for 5 minutes) ​ enable_chat boolean default: false Whether chat is enabled in the room ​ enable_prejoin_ui boolean default: false Whether the prejoin lobby UI is enabled ​ enable_emoji_reactions boolean default: false Whether emoji reactions are enabled ​ eject_at_room_exp boolean default: false Whether to eject participants when room expires ​ enable_dialout boolean Whether dial-out is enabled ​ enable_recording string Recording settings (“cloud”, “local”, or “raw-tracks”) ​ geo string Geographic region for room ​ max_participants number Maximum number of participants allowed in the room ​ recordings_bucket RecordingsBucketConfig Configuration for custom S3 bucket recordings ​ sip DailyRoomSipParams SIP configuration parameters ​ sip_uri dict SIP URI configuration (returned by Daily) ​ start_video_off boolean default: false Whether the camera video is turned off by default The class also includes a sip_endpoint property that returns the SIP endpoint URI if available. Copy Ask AI import time from pipecat.transports.services.helpers.daily_rest import ( DailyRoomProperties, DailyRoomSipParams, RecordingsBucketConfig, ) properties = DailyRoomProperties( exp = time.time() + 3600 , # 1 hour from now enable_chat = True , enable_emoji_reactions = True , enable_recording = "cloud" , geo = "us-west" , max_participants = 50 , sip = DailyRoomSipParams( display_name = "conference" ), recordings_bucket = RecordingsBucketConfig( bucket_name = "my-bucket" , bucket_region = "us-west-2" , assume_role_arn = "arn:aws:iam::123456789012:role/DailyRole" ) ) # Access SIP endpoint if available if properties.sip_endpoint: print ( f "SIP endpoint: { properties.sip_endpoint } " ) ​ DailyRoomParams Parameters for creating a new Daily room. ​ name string Room name (if not provided, one will be generated) ​ privacy string default: "public" Room privacy setting (“private” or “public”) ​ properties DailyRoomProperties Room configuration properties Copy Ask AI import time from pipecat.transports.services.helpers.daily_rest import ( DailyRoomParams, DailyRoomProperties, ) params = DailyRoomParams( name = "team-meeting" , privacy = "private" , properties = DailyRoomProperties( enable_chat = True , exp = time.time() + 7200 # 2 hours from now ) ) ​ DailyRoomObject Response object representing a Daily room. ​ id string Unique room identifier ​ name string Room name ​ api_created boolean Whether the room was created via API ​ privacy string Room privacy setting ​ url string Complete room URL ​ created_at string Room creation timestamp in ISO 8601 format ​ config DailyRoomProperties Room configuration Copy Ask AI from pipecat.transports.services.helpers.daily_rest import ( DailyRoomObject, DailyRoomProperties, ) # Example of what a DailyRoomObject looks like when received room = DailyRoomObject( id = "abc123" , name = "team-meeting" , api_created = True , privacy = "private" , url = "https://your-domain.daily.co/team-meeting" , created_at = "2024-01-20T10:00:00.000Z" , config = DailyRoomProperties( enable_chat = True , exp = 1705743600 ) ) ​ DailyMeetingTokenProperties Properties for configuring a Daily meeting token. ​ room_name string The room this token is valid for. If not set, token is valid for all rooms. ​ eject_at_token_exp boolean Whether to eject user when token expires ​ eject_after_elapsed integer Eject user after this many seconds ​ nbf integer “Not before” timestamp - users cannot join before this time ​ exp integer Expiration timestamp - users cannot join after this time ​ is_owner boolean Whether token grants owner privileges ​ user_name string User’s display name in the meeting ​ user_id string Unique identifier for the user (36 char limit) ​ enable_screenshare boolean Whether user can share their screen ​ start_video_off boolean Whether to join with video off ​ start_audio_off boolean Whether to join with audio off ​ enable_recording string Recording settings (“cloud”, “local”, or “raw-tracks”) ​ enable_prejoin_ui boolean Whether to show prejoin UI ​ start_cloud_recording boolean Whether to start cloud recording when user joins ​ permissions dict Initial default permissions for a non-meeting-owner participant ​ DailyMeetingTokenParams Parameters for creating a Daily meeting token. ​ properties DailyMeetingTokenProperties Token configuration properties Copy Ask AI from pipecat.transports.services.helpers.daily_rest import ( DailyMeetingTokenParams, DailyMeetingTokenProperties, ) token_params = DailyMeetingTokenParams( properties = DailyMeetingTokenProperties( user_name = "John Doe" , enable_screenshare = True , start_video_off = True , permissions = { "canSend" : [ "video" , "audio" ]} ) ) ​ Initialize DailyRESTHelper Create a new instance of the Daily REST helper. ​ daily_api_key string required Your Daily API key ​ daily_api_url string default: "https://api.daily.co/v1" The Daily API base URL ​ aiohttp_session aiohttp.ClientSession required An aiohttp client session for making HTTP requests Copy Ask AI helper = DailyRESTHelper( daily_api_key = "your-api-key" , aiohttp_session = session ) ​ Create Room Creates a new Daily room with specified parameters. ​ params DailyRoomParams required Room configuration parameters including name, privacy, and properties Copy Ask AI # Create a room that expires in 1 hour params = DailyRoomParams( name = "my-room" , privacy = "private" , properties = DailyRoomProperties( exp = time.time() + 3600 , enable_chat = True ) ) room = await helper.create_room(params) print ( f "Room URL: { room.url } " ) ​ Get Room From URL Retrieves room information using a Daily room URL. ​ room_url string required The complete Daily room URL Copy Ask AI room = await helper.get_room_from_url( "https://your-domain.daily.co/my-room" ) print ( f "Room name: { room.name } " ) ​ Get Token Generates a meeting token for a specific room. ​ room_url string required The complete Daily room URL ​ expiry_time float default: "3600" Token expiration time in seconds ​ eject_at_token_exp bool default: "False" Whether to eject user when token expires ​ owner bool default: "True" Whether the token should have owner privileges (overrides any setting in params) ​ params DailyMeetingTokenParams Additional token configuration. Note that room_name , exp , eject_at_token_exp , and is_owner will be set based on the other function parameters. Copy Ask AI # Basic token generation token = await helper.get_token( room_url = "https://your-domain.daily.co/my-room" , expiry_time = 1800 , # 30 minutes owner = True , eject_at_token_exp = True ) # Advanced token generation with additional properties token_params = DailyMeetingTokenParams( properties = DailyMeetingTokenProperties( user_name = "John Doe" , start_video_off = True ) ) token = await helper.get_token( room_url = "https://your-domain.daily.co/my-room" , expiry_time = 1800 , owner = False , eject_at_token_exp = True , params = token_params ) ​ Delete Room By URL Deletes a room using its URL. ​ room_url string required The complete Daily room URL Copy Ask AI success = await helper.delete_room_by_url( "https://your-domain.daily.co/my-room" ) if success: print ( "Room deleted successfully" ) ​ Delete Room By Name Deletes a room using its name. ​ room_name string required The name of the Daily room Copy Ask AI success = await helper.delete_room_by_name( "my-room" ) if success: print ( "Room deleted successfully" ) ​ Get Name From URL Extracts the room name from a Daily room URL. ​ room_url string required The complete Daily room URL Copy Ask AI room_name = helper.get_name_from_url( "https://your-domain.daily.co/my-room" ) print ( f "Room name: { room_name } " ) # Outputs: "my-room" Turn Tracking Observer Smart Turn Overview On this page Classes DailyRoomSipParams RecordingsBucketConfig DailyRoomProperties DailyRoomParams DailyRoomObject DailyMeetingTokenProperties DailyMeetingTokenParams Initialize DailyRESTHelper Create Room Get Room From URL Get Token Delete Room By URL Delete Room By Name Get Name From URL Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/daily_rest-helpers_58b79f3d.txt b/daily_rest-helpers_58b79f3d.txt
new file mode 100644
index 0000000000000000000000000000000000000000..b57c255b8083df189f858048904ab92021905648
--- /dev/null
+++ b/daily_rest-helpers_58b79f3d.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/daily/rest-helpers#initialize-dailyresthelper
+Title: Daily REST Helper - Pipecat
+==================================================
+
+Daily REST Helper - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Service Utilities Daily REST Helper Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Daily REST Helper Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline Daily REST API Documentation For complete Daily REST API reference and additional details ​ Classes ​ DailyRoomSipParams Configuration for SIP (Session Initiation Protocol) parameters. ​ display_name string default: "sw-sip-dialin" Display name for the SIP endpoint ​ video boolean default: false Whether video is enabled for SIP ​ sip_mode string default: "dial-in" SIP connection mode ​ num_endpoints integer default: 1 Number of SIP endpoints Copy Ask AI from pipecat.transports.services.helpers.daily_rest import DailyRoomSipParams sip_params = DailyRoomSipParams( display_name = "conference-line" , video = True , num_endpoints = 2 ) ​ RecordingsBucketConfig Configuration for storing Daily recordings in a custom S3 bucket. ​ bucket_name string required Name of the S3 bucket for storing recordings ​ bucket_region string required AWS region where the S3 bucket is located ​ assume_role_arn string required ARN of the IAM role to assume for S3 access ​ allow_api_access boolean default: false Whether to allow API access to the recordings Copy Ask AI from pipecat.transports.services.helpers.daily_rest import RecordingsBucketConfig bucket_config = RecordingsBucketConfig( bucket_name = "my-recordings-bucket" , bucket_region = "us-west-2" , assume_role_arn = "arn:aws:iam::123456789012:role/DailyRecordingsRole" , allow_api_access = True ) ​ DailyRoomProperties Properties that configure a Daily room’s behavior and features. ​ exp float Room expiration time as Unix timestamp (e.g., time.time() + 300 for 5 minutes) ​ enable_chat boolean default: false Whether chat is enabled in the room ​ enable_prejoin_ui boolean default: false Whether the prejoin lobby UI is enabled ​ enable_emoji_reactions boolean default: false Whether emoji reactions are enabled ​ eject_at_room_exp boolean default: false Whether to eject participants when room expires ​ enable_dialout boolean Whether dial-out is enabled ​ enable_recording string Recording settings (“cloud”, “local”, or “raw-tracks”) ​ geo string Geographic region for room ​ max_participants number Maximum number of participants allowed in the room ​ recordings_bucket RecordingsBucketConfig Configuration for custom S3 bucket recordings ​ sip DailyRoomSipParams SIP configuration parameters ​ sip_uri dict SIP URI configuration (returned by Daily) ​ start_video_off boolean default: false Whether the camera video is turned off by default The class also includes a sip_endpoint property that returns the SIP endpoint URI if available. Copy Ask AI import time from pipecat.transports.services.helpers.daily_rest import ( DailyRoomProperties, DailyRoomSipParams, RecordingsBucketConfig, ) properties = DailyRoomProperties( exp = time.time() + 3600 , # 1 hour from now enable_chat = True , enable_emoji_reactions = True , enable_recording = "cloud" , geo = "us-west" , max_participants = 50 , sip = DailyRoomSipParams( display_name = "conference" ), recordings_bucket = RecordingsBucketConfig( bucket_name = "my-bucket" , bucket_region = "us-west-2" , assume_role_arn = "arn:aws:iam::123456789012:role/DailyRole" ) ) # Access SIP endpoint if available if properties.sip_endpoint: print ( f "SIP endpoint: { properties.sip_endpoint } " ) ​ DailyRoomParams Parameters for creating a new Daily room. ​ name string Room name (if not provided, one will be generated) ​ privacy string default: "public" Room privacy setting (“private” or “public”) ​ properties DailyRoomProperties Room configuration properties Copy Ask AI import time from pipecat.transports.services.helpers.daily_rest import ( DailyRoomParams, DailyRoomProperties, ) params = DailyRoomParams( name = "team-meeting" , privacy = "private" , properties = DailyRoomProperties( enable_chat = True , exp = time.time() + 7200 # 2 hours from now ) ) ​ DailyRoomObject Response object representing a Daily room. ​ id string Unique room identifier ​ name string Room name ​ api_created boolean Whether the room was created via API ​ privacy string Room privacy setting ​ url string Complete room URL ​ created_at string Room creation timestamp in ISO 8601 format ​ config DailyRoomProperties Room configuration Copy Ask AI from pipecat.transports.services.helpers.daily_rest import ( DailyRoomObject, DailyRoomProperties, ) # Example of what a DailyRoomObject looks like when received room = DailyRoomObject( id = "abc123" , name = "team-meeting" , api_created = True , privacy = "private" , url = "https://your-domain.daily.co/team-meeting" , created_at = "2024-01-20T10:00:00.000Z" , config = DailyRoomProperties( enable_chat = True , exp = 1705743600 ) ) ​ DailyMeetingTokenProperties Properties for configuring a Daily meeting token. ​ room_name string The room this token is valid for. If not set, token is valid for all rooms. ​ eject_at_token_exp boolean Whether to eject user when token expires ​ eject_after_elapsed integer Eject user after this many seconds ​ nbf integer “Not before” timestamp - users cannot join before this time ​ exp integer Expiration timestamp - users cannot join after this time ​ is_owner boolean Whether token grants owner privileges ​ user_name string User’s display name in the meeting ​ user_id string Unique identifier for the user (36 char limit) ​ enable_screenshare boolean Whether user can share their screen ​ start_video_off boolean Whether to join with video off ​ start_audio_off boolean Whether to join with audio off ​ enable_recording string Recording settings (“cloud”, “local”, or “raw-tracks”) ​ enable_prejoin_ui boolean Whether to show prejoin UI ​ start_cloud_recording boolean Whether to start cloud recording when user joins ​ permissions dict Initial default permissions for a non-meeting-owner participant ​ DailyMeetingTokenParams Parameters for creating a Daily meeting token. ​ properties DailyMeetingTokenProperties Token configuration properties Copy Ask AI from pipecat.transports.services.helpers.daily_rest import ( DailyMeetingTokenParams, DailyMeetingTokenProperties, ) token_params = DailyMeetingTokenParams( properties = DailyMeetingTokenProperties( user_name = "John Doe" , enable_screenshare = True , start_video_off = True , permissions = { "canSend" : [ "video" , "audio" ]} ) ) ​ Initialize DailyRESTHelper Create a new instance of the Daily REST helper. ​ daily_api_key string required Your Daily API key ​ daily_api_url string default: "https://api.daily.co/v1" The Daily API base URL ​ aiohttp_session aiohttp.ClientSession required An aiohttp client session for making HTTP requests Copy Ask AI helper = DailyRESTHelper( daily_api_key = "your-api-key" , aiohttp_session = session ) ​ Create Room Creates a new Daily room with specified parameters. ​ params DailyRoomParams required Room configuration parameters including name, privacy, and properties Copy Ask AI # Create a room that expires in 1 hour params = DailyRoomParams( name = "my-room" , privacy = "private" , properties = DailyRoomProperties( exp = time.time() + 3600 , enable_chat = True ) ) room = await helper.create_room(params) print ( f "Room URL: { room.url } " ) ​ Get Room From URL Retrieves room information using a Daily room URL. ​ room_url string required The complete Daily room URL Copy Ask AI room = await helper.get_room_from_url( "https://your-domain.daily.co/my-room" ) print ( f "Room name: { room.name } " ) ​ Get Token Generates a meeting token for a specific room. ​ room_url string required The complete Daily room URL ​ expiry_time float default: "3600" Token expiration time in seconds ​ eject_at_token_exp bool default: "False" Whether to eject user when token expires ​ owner bool default: "True" Whether the token should have owner privileges (overrides any setting in params) ​ params DailyMeetingTokenParams Additional token configuration. Note that room_name , exp , eject_at_token_exp , and is_owner will be set based on the other function parameters. Copy Ask AI # Basic token generation token = await helper.get_token( room_url = "https://your-domain.daily.co/my-room" , expiry_time = 1800 , # 30 minutes owner = True , eject_at_token_exp = True ) # Advanced token generation with additional properties token_params = DailyMeetingTokenParams( properties = DailyMeetingTokenProperties( user_name = "John Doe" , start_video_off = True ) ) token = await helper.get_token( room_url = "https://your-domain.daily.co/my-room" , expiry_time = 1800 , owner = False , eject_at_token_exp = True , params = token_params ) ​ Delete Room By URL Deletes a room using its URL. ​ room_url string required The complete Daily room URL Copy Ask AI success = await helper.delete_room_by_url( "https://your-domain.daily.co/my-room" ) if success: print ( "Room deleted successfully" ) ​ Delete Room By Name Deletes a room using its name. ​ room_name string required The name of the Daily room Copy Ask AI success = await helper.delete_room_by_name( "my-room" ) if success: print ( "Room deleted successfully" ) ​ Get Name From URL Extracts the room name from a Daily room URL. ​ room_url string required The complete Daily room URL Copy Ask AI room_name = helper.get_name_from_url( "https://your-domain.daily.co/my-room" ) print ( f "Room name: { room_name } " ) # Outputs: "my-room" Turn Tracking Observer Smart Turn Overview On this page Classes DailyRoomSipParams RecordingsBucketConfig DailyRoomProperties DailyRoomParams DailyRoomObject DailyMeetingTokenProperties DailyMeetingTokenParams Initialize DailyRESTHelper Create Room Get Room From URL Get Token Delete Room By URL Delete Room By Name Get Name From URL Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/daily_rest-helpers_5c23af5e.txt b/daily_rest-helpers_5c23af5e.txt
new file mode 100644
index 0000000000000000000000000000000000000000..503a069a601322aed7f5f600b19a2bef0ba57663
--- /dev/null
+++ b/daily_rest-helpers_5c23af5e.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/daily/rest-helpers#param-display-name
+Title: Daily REST Helper - Pipecat
+==================================================
+
+Daily REST Helper - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Service Utilities Daily REST Helper Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Daily REST Helper Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline Daily REST API Documentation For complete Daily REST API reference and additional details ​ Classes ​ DailyRoomSipParams Configuration for SIP (Session Initiation Protocol) parameters. ​ display_name string default: "sw-sip-dialin" Display name for the SIP endpoint ​ video boolean default: false Whether video is enabled for SIP ​ sip_mode string default: "dial-in" SIP connection mode ​ num_endpoints integer default: 1 Number of SIP endpoints Copy Ask AI from pipecat.transports.services.helpers.daily_rest import DailyRoomSipParams sip_params = DailyRoomSipParams( display_name = "conference-line" , video = True , num_endpoints = 2 ) ​ RecordingsBucketConfig Configuration for storing Daily recordings in a custom S3 bucket. ​ bucket_name string required Name of the S3 bucket for storing recordings ​ bucket_region string required AWS region where the S3 bucket is located ​ assume_role_arn string required ARN of the IAM role to assume for S3 access ​ allow_api_access boolean default: false Whether to allow API access to the recordings Copy Ask AI from pipecat.transports.services.helpers.daily_rest import RecordingsBucketConfig bucket_config = RecordingsBucketConfig( bucket_name = "my-recordings-bucket" , bucket_region = "us-west-2" , assume_role_arn = "arn:aws:iam::123456789012:role/DailyRecordingsRole" , allow_api_access = True ) ​ DailyRoomProperties Properties that configure a Daily room’s behavior and features. ​ exp float Room expiration time as Unix timestamp (e.g., time.time() + 300 for 5 minutes) ​ enable_chat boolean default: false Whether chat is enabled in the room ​ enable_prejoin_ui boolean default: false Whether the prejoin lobby UI is enabled ​ enable_emoji_reactions boolean default: false Whether emoji reactions are enabled ​ eject_at_room_exp boolean default: false Whether to eject participants when room expires ​ enable_dialout boolean Whether dial-out is enabled ​ enable_recording string Recording settings (“cloud”, “local”, or “raw-tracks”) ​ geo string Geographic region for room ​ max_participants number Maximum number of participants allowed in the room ​ recordings_bucket RecordingsBucketConfig Configuration for custom S3 bucket recordings ​ sip DailyRoomSipParams SIP configuration parameters ​ sip_uri dict SIP URI configuration (returned by Daily) ​ start_video_off boolean default: false Whether the camera video is turned off by default The class also includes a sip_endpoint property that returns the SIP endpoint URI if available. Copy Ask AI import time from pipecat.transports.services.helpers.daily_rest import ( DailyRoomProperties, DailyRoomSipParams, RecordingsBucketConfig, ) properties = DailyRoomProperties( exp = time.time() + 3600 , # 1 hour from now enable_chat = True , enable_emoji_reactions = True , enable_recording = "cloud" , geo = "us-west" , max_participants = 50 , sip = DailyRoomSipParams( display_name = "conference" ), recordings_bucket = RecordingsBucketConfig( bucket_name = "my-bucket" , bucket_region = "us-west-2" , assume_role_arn = "arn:aws:iam::123456789012:role/DailyRole" ) ) # Access SIP endpoint if available if properties.sip_endpoint: print ( f "SIP endpoint: { properties.sip_endpoint } " ) ​ DailyRoomParams Parameters for creating a new Daily room. ​ name string Room name (if not provided, one will be generated) ​ privacy string default: "public" Room privacy setting (“private” or “public”) ​ properties DailyRoomProperties Room configuration properties Copy Ask AI import time from pipecat.transports.services.helpers.daily_rest import ( DailyRoomParams, DailyRoomProperties, ) params = DailyRoomParams( name = "team-meeting" , privacy = "private" , properties = DailyRoomProperties( enable_chat = True , exp = time.time() + 7200 # 2 hours from now ) ) ​ DailyRoomObject Response object representing a Daily room. ​ id string Unique room identifier ​ name string Room name ​ api_created boolean Whether the room was created via API ​ privacy string Room privacy setting ​ url string Complete room URL ​ created_at string Room creation timestamp in ISO 8601 format ​ config DailyRoomProperties Room configuration Copy Ask AI from pipecat.transports.services.helpers.daily_rest import ( DailyRoomObject, DailyRoomProperties, ) # Example of what a DailyRoomObject looks like when received room = DailyRoomObject( id = "abc123" , name = "team-meeting" , api_created = True , privacy = "private" , url = "https://your-domain.daily.co/team-meeting" , created_at = "2024-01-20T10:00:00.000Z" , config = DailyRoomProperties( enable_chat = True , exp = 1705743600 ) ) ​ DailyMeetingTokenProperties Properties for configuring a Daily meeting token. ​ room_name string The room this token is valid for. If not set, token is valid for all rooms. ​ eject_at_token_exp boolean Whether to eject user when token expires ​ eject_after_elapsed integer Eject user after this many seconds ​ nbf integer “Not before” timestamp - users cannot join before this time ​ exp integer Expiration timestamp - users cannot join after this time ​ is_owner boolean Whether token grants owner privileges ​ user_name string User’s display name in the meeting ​ user_id string Unique identifier for the user (36 char limit) ​ enable_screenshare boolean Whether user can share their screen ​ start_video_off boolean Whether to join with video off ​ start_audio_off boolean Whether to join with audio off ​ enable_recording string Recording settings (“cloud”, “local”, or “raw-tracks”) ​ enable_prejoin_ui boolean Whether to show prejoin UI ​ start_cloud_recording boolean Whether to start cloud recording when user joins ​ permissions dict Initial default permissions for a non-meeting-owner participant ​ DailyMeetingTokenParams Parameters for creating a Daily meeting token. ​ properties DailyMeetingTokenProperties Token configuration properties Copy Ask AI from pipecat.transports.services.helpers.daily_rest import ( DailyMeetingTokenParams, DailyMeetingTokenProperties, ) token_params = DailyMeetingTokenParams( properties = DailyMeetingTokenProperties( user_name = "John Doe" , enable_screenshare = True , start_video_off = True , permissions = { "canSend" : [ "video" , "audio" ]} ) ) ​ Initialize DailyRESTHelper Create a new instance of the Daily REST helper. ​ daily_api_key string required Your Daily API key ​ daily_api_url string default: "https://api.daily.co/v1" The Daily API base URL ​ aiohttp_session aiohttp.ClientSession required An aiohttp client session for making HTTP requests Copy Ask AI helper = DailyRESTHelper( daily_api_key = "your-api-key" , aiohttp_session = session ) ​ Create Room Creates a new Daily room with specified parameters. ​ params DailyRoomParams required Room configuration parameters including name, privacy, and properties Copy Ask AI # Create a room that expires in 1 hour params = DailyRoomParams( name = "my-room" , privacy = "private" , properties = DailyRoomProperties( exp = time.time() + 3600 , enable_chat = True ) ) room = await helper.create_room(params) print ( f "Room URL: { room.url } " ) ​ Get Room From URL Retrieves room information using a Daily room URL. ​ room_url string required The complete Daily room URL Copy Ask AI room = await helper.get_room_from_url( "https://your-domain.daily.co/my-room" ) print ( f "Room name: { room.name } " ) ​ Get Token Generates a meeting token for a specific room. ​ room_url string required The complete Daily room URL ​ expiry_time float default: "3600" Token expiration time in seconds ​ eject_at_token_exp bool default: "False" Whether to eject user when token expires ​ owner bool default: "True" Whether the token should have owner privileges (overrides any setting in params) ​ params DailyMeetingTokenParams Additional token configuration. Note that room_name , exp , eject_at_token_exp , and is_owner will be set based on the other function parameters. Copy Ask AI # Basic token generation token = await helper.get_token( room_url = "https://your-domain.daily.co/my-room" , expiry_time = 1800 , # 30 minutes owner = True , eject_at_token_exp = True ) # Advanced token generation with additional properties token_params = DailyMeetingTokenParams( properties = DailyMeetingTokenProperties( user_name = "John Doe" , start_video_off = True ) ) token = await helper.get_token( room_url = "https://your-domain.daily.co/my-room" , expiry_time = 1800 , owner = False , eject_at_token_exp = True , params = token_params ) ​ Delete Room By URL Deletes a room using its URL. ​ room_url string required The complete Daily room URL Copy Ask AI success = await helper.delete_room_by_url( "https://your-domain.daily.co/my-room" ) if success: print ( "Room deleted successfully" ) ​ Delete Room By Name Deletes a room using its name. ​ room_name string required The name of the Daily room Copy Ask AI success = await helper.delete_room_by_name( "my-room" ) if success: print ( "Room deleted successfully" ) ​ Get Name From URL Extracts the room name from a Daily room URL. ​ room_url string required The complete Daily room URL Copy Ask AI room_name = helper.get_name_from_url( "https://your-domain.daily.co/my-room" ) print ( f "Room name: { room_name } " ) # Outputs: "my-room" Turn Tracking Observer Smart Turn Overview On this page Classes DailyRoomSipParams RecordingsBucketConfig DailyRoomProperties DailyRoomParams DailyRoomObject DailyMeetingTokenProperties DailyMeetingTokenParams Initialize DailyRESTHelper Create Room Get Room From URL Get Token Delete Room By URL Delete Room By Name Get Name From URL Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/daily_rest-helpers_5c76a6ce.txt b/daily_rest-helpers_5c76a6ce.txt
new file mode 100644
index 0000000000000000000000000000000000000000..e2a9e23197060af5baa54dee9ba2eded8def599e
--- /dev/null
+++ b/daily_rest-helpers_5c76a6ce.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/daily/rest-helpers#param-room-url-2
+Title: Daily REST Helper - Pipecat
+==================================================
+
+Daily REST Helper - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Service Utilities Daily REST Helper Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Daily REST Helper Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline Daily REST API Documentation For complete Daily REST API reference and additional details ​ Classes ​ DailyRoomSipParams Configuration for SIP (Session Initiation Protocol) parameters. ​ display_name string default: "sw-sip-dialin" Display name for the SIP endpoint ​ video boolean default: false Whether video is enabled for SIP ​ sip_mode string default: "dial-in" SIP connection mode ​ num_endpoints integer default: 1 Number of SIP endpoints Copy Ask AI from pipecat.transports.services.helpers.daily_rest import DailyRoomSipParams sip_params = DailyRoomSipParams( display_name = "conference-line" , video = True , num_endpoints = 2 ) ​ RecordingsBucketConfig Configuration for storing Daily recordings in a custom S3 bucket. ​ bucket_name string required Name of the S3 bucket for storing recordings ​ bucket_region string required AWS region where the S3 bucket is located ​ assume_role_arn string required ARN of the IAM role to assume for S3 access ​ allow_api_access boolean default: false Whether to allow API access to the recordings Copy Ask AI from pipecat.transports.services.helpers.daily_rest import RecordingsBucketConfig bucket_config = RecordingsBucketConfig( bucket_name = "my-recordings-bucket" , bucket_region = "us-west-2" , assume_role_arn = "arn:aws:iam::123456789012:role/DailyRecordingsRole" , allow_api_access = True ) ​ DailyRoomProperties Properties that configure a Daily room’s behavior and features. ​ exp float Room expiration time as Unix timestamp (e.g., time.time() + 300 for 5 minutes) ​ enable_chat boolean default: false Whether chat is enabled in the room ​ enable_prejoin_ui boolean default: false Whether the prejoin lobby UI is enabled ​ enable_emoji_reactions boolean default: false Whether emoji reactions are enabled ​ eject_at_room_exp boolean default: false Whether to eject participants when room expires ​ enable_dialout boolean Whether dial-out is enabled ​ enable_recording string Recording settings (“cloud”, “local”, or “raw-tracks”) ​ geo string Geographic region for room ​ max_participants number Maximum number of participants allowed in the room ​ recordings_bucket RecordingsBucketConfig Configuration for custom S3 bucket recordings ​ sip DailyRoomSipParams SIP configuration parameters ​ sip_uri dict SIP URI configuration (returned by Daily) ​ start_video_off boolean default: false Whether the camera video is turned off by default The class also includes a sip_endpoint property that returns the SIP endpoint URI if available. Copy Ask AI import time from pipecat.transports.services.helpers.daily_rest import ( DailyRoomProperties, DailyRoomSipParams, RecordingsBucketConfig, ) properties = DailyRoomProperties( exp = time.time() + 3600 , # 1 hour from now enable_chat = True , enable_emoji_reactions = True , enable_recording = "cloud" , geo = "us-west" , max_participants = 50 , sip = DailyRoomSipParams( display_name = "conference" ), recordings_bucket = RecordingsBucketConfig( bucket_name = "my-bucket" , bucket_region = "us-west-2" , assume_role_arn = "arn:aws:iam::123456789012:role/DailyRole" ) ) # Access SIP endpoint if available if properties.sip_endpoint: print ( f "SIP endpoint: { properties.sip_endpoint } " ) ​ DailyRoomParams Parameters for creating a new Daily room. ​ name string Room name (if not provided, one will be generated) ​ privacy string default: "public" Room privacy setting (“private” or “public”) ​ properties DailyRoomProperties Room configuration properties Copy Ask AI import time from pipecat.transports.services.helpers.daily_rest import ( DailyRoomParams, DailyRoomProperties, ) params = DailyRoomParams( name = "team-meeting" , privacy = "private" , properties = DailyRoomProperties( enable_chat = True , exp = time.time() + 7200 # 2 hours from now ) ) ​ DailyRoomObject Response object representing a Daily room. ​ id string Unique room identifier ​ name string Room name ​ api_created boolean Whether the room was created via API ​ privacy string Room privacy setting ​ url string Complete room URL ​ created_at string Room creation timestamp in ISO 8601 format ​ config DailyRoomProperties Room configuration Copy Ask AI from pipecat.transports.services.helpers.daily_rest import ( DailyRoomObject, DailyRoomProperties, ) # Example of what a DailyRoomObject looks like when received room = DailyRoomObject( id = "abc123" , name = "team-meeting" , api_created = True , privacy = "private" , url = "https://your-domain.daily.co/team-meeting" , created_at = "2024-01-20T10:00:00.000Z" , config = DailyRoomProperties( enable_chat = True , exp = 1705743600 ) ) ​ DailyMeetingTokenProperties Properties for configuring a Daily meeting token. ​ room_name string The room this token is valid for. If not set, token is valid for all rooms. ​ eject_at_token_exp boolean Whether to eject user when token expires ​ eject_after_elapsed integer Eject user after this many seconds ​ nbf integer “Not before” timestamp - users cannot join before this time ​ exp integer Expiration timestamp - users cannot join after this time ​ is_owner boolean Whether token grants owner privileges ​ user_name string User’s display name in the meeting ​ user_id string Unique identifier for the user (36 char limit) ​ enable_screenshare boolean Whether user can share their screen ​ start_video_off boolean Whether to join with video off ​ start_audio_off boolean Whether to join with audio off ​ enable_recording string Recording settings (“cloud”, “local”, or “raw-tracks”) ​ enable_prejoin_ui boolean Whether to show prejoin UI ​ start_cloud_recording boolean Whether to start cloud recording when user joins ​ permissions dict Initial default permissions for a non-meeting-owner participant ​ DailyMeetingTokenParams Parameters for creating a Daily meeting token. ​ properties DailyMeetingTokenProperties Token configuration properties Copy Ask AI from pipecat.transports.services.helpers.daily_rest import ( DailyMeetingTokenParams, DailyMeetingTokenProperties, ) token_params = DailyMeetingTokenParams( properties = DailyMeetingTokenProperties( user_name = "John Doe" , enable_screenshare = True , start_video_off = True , permissions = { "canSend" : [ "video" , "audio" ]} ) ) ​ Initialize DailyRESTHelper Create a new instance of the Daily REST helper. ​ daily_api_key string required Your Daily API key ​ daily_api_url string default: "https://api.daily.co/v1" The Daily API base URL ​ aiohttp_session aiohttp.ClientSession required An aiohttp client session for making HTTP requests Copy Ask AI helper = DailyRESTHelper( daily_api_key = "your-api-key" , aiohttp_session = session ) ​ Create Room Creates a new Daily room with specified parameters. ​ params DailyRoomParams required Room configuration parameters including name, privacy, and properties Copy Ask AI # Create a room that expires in 1 hour params = DailyRoomParams( name = "my-room" , privacy = "private" , properties = DailyRoomProperties( exp = time.time() + 3600 , enable_chat = True ) ) room = await helper.create_room(params) print ( f "Room URL: { room.url } " ) ​ Get Room From URL Retrieves room information using a Daily room URL. ​ room_url string required The complete Daily room URL Copy Ask AI room = await helper.get_room_from_url( "https://your-domain.daily.co/my-room" ) print ( f "Room name: { room.name } " ) ​ Get Token Generates a meeting token for a specific room. ​ room_url string required The complete Daily room URL ​ expiry_time float default: "3600" Token expiration time in seconds ​ eject_at_token_exp bool default: "False" Whether to eject user when token expires ​ owner bool default: "True" Whether the token should have owner privileges (overrides any setting in params) ​ params DailyMeetingTokenParams Additional token configuration. Note that room_name , exp , eject_at_token_exp , and is_owner will be set based on the other function parameters. Copy Ask AI # Basic token generation token = await helper.get_token( room_url = "https://your-domain.daily.co/my-room" , expiry_time = 1800 , # 30 minutes owner = True , eject_at_token_exp = True ) # Advanced token generation with additional properties token_params = DailyMeetingTokenParams( properties = DailyMeetingTokenProperties( user_name = "John Doe" , start_video_off = True ) ) token = await helper.get_token( room_url = "https://your-domain.daily.co/my-room" , expiry_time = 1800 , owner = False , eject_at_token_exp = True , params = token_params ) ​ Delete Room By URL Deletes a room using its URL. ​ room_url string required The complete Daily room URL Copy Ask AI success = await helper.delete_room_by_url( "https://your-domain.daily.co/my-room" ) if success: print ( "Room deleted successfully" ) ​ Delete Room By Name Deletes a room using its name. ​ room_name string required The name of the Daily room Copy Ask AI success = await helper.delete_room_by_name( "my-room" ) if success: print ( "Room deleted successfully" ) ​ Get Name From URL Extracts the room name from a Daily room URL. ​ room_url string required The complete Daily room URL Copy Ask AI room_name = helper.get_name_from_url( "https://your-domain.daily.co/my-room" ) print ( f "Room name: { room_name } " ) # Outputs: "my-room" Turn Tracking Observer Smart Turn Overview On this page Classes DailyRoomSipParams RecordingsBucketConfig DailyRoomProperties DailyRoomParams DailyRoomObject DailyMeetingTokenProperties DailyMeetingTokenParams Initialize DailyRESTHelper Create Room Get Room From URL Get Token Delete Room By URL Delete Room By Name Get Name From URL Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/daily_rest-helpers_5d90463c.txt b/daily_rest-helpers_5d90463c.txt
new file mode 100644
index 0000000000000000000000000000000000000000..e14e3853eefc71e6b39e29927bc6bee963b87866
--- /dev/null
+++ b/daily_rest-helpers_5d90463c.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/daily/rest-helpers#param-start-cloud-recording
+Title: Daily REST Helper - Pipecat
+==================================================
+
+Daily REST Helper - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Service Utilities Daily REST Helper Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Daily REST Helper Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline Daily REST API Documentation For complete Daily REST API reference and additional details ​ Classes ​ DailyRoomSipParams Configuration for SIP (Session Initiation Protocol) parameters. ​ display_name string default: "sw-sip-dialin" Display name for the SIP endpoint ​ video boolean default: false Whether video is enabled for SIP ​ sip_mode string default: "dial-in" SIP connection mode ​ num_endpoints integer default: 1 Number of SIP endpoints Copy Ask AI from pipecat.transports.services.helpers.daily_rest import DailyRoomSipParams sip_params = DailyRoomSipParams( display_name = "conference-line" , video = True , num_endpoints = 2 ) ​ RecordingsBucketConfig Configuration for storing Daily recordings in a custom S3 bucket. ​ bucket_name string required Name of the S3 bucket for storing recordings ​ bucket_region string required AWS region where the S3 bucket is located ​ assume_role_arn string required ARN of the IAM role to assume for S3 access ​ allow_api_access boolean default: false Whether to allow API access to the recordings Copy Ask AI from pipecat.transports.services.helpers.daily_rest import RecordingsBucketConfig bucket_config = RecordingsBucketConfig( bucket_name = "my-recordings-bucket" , bucket_region = "us-west-2" , assume_role_arn = "arn:aws:iam::123456789012:role/DailyRecordingsRole" , allow_api_access = True ) ​ DailyRoomProperties Properties that configure a Daily room’s behavior and features. ​ exp float Room expiration time as Unix timestamp (e.g., time.time() + 300 for 5 minutes) ​ enable_chat boolean default: false Whether chat is enabled in the room ​ enable_prejoin_ui boolean default: false Whether the prejoin lobby UI is enabled ​ enable_emoji_reactions boolean default: false Whether emoji reactions are enabled ​ eject_at_room_exp boolean default: false Whether to eject participants when room expires ​ enable_dialout boolean Whether dial-out is enabled ​ enable_recording string Recording settings (“cloud”, “local”, or “raw-tracks”) ​ geo string Geographic region for room ​ max_participants number Maximum number of participants allowed in the room ​ recordings_bucket RecordingsBucketConfig Configuration for custom S3 bucket recordings ​ sip DailyRoomSipParams SIP configuration parameters ​ sip_uri dict SIP URI configuration (returned by Daily) ​ start_video_off boolean default: false Whether the camera video is turned off by default The class also includes a sip_endpoint property that returns the SIP endpoint URI if available. Copy Ask AI import time from pipecat.transports.services.helpers.daily_rest import ( DailyRoomProperties, DailyRoomSipParams, RecordingsBucketConfig, ) properties = DailyRoomProperties( exp = time.time() + 3600 , # 1 hour from now enable_chat = True , enable_emoji_reactions = True , enable_recording = "cloud" , geo = "us-west" , max_participants = 50 , sip = DailyRoomSipParams( display_name = "conference" ), recordings_bucket = RecordingsBucketConfig( bucket_name = "my-bucket" , bucket_region = "us-west-2" , assume_role_arn = "arn:aws:iam::123456789012:role/DailyRole" ) ) # Access SIP endpoint if available if properties.sip_endpoint: print ( f "SIP endpoint: { properties.sip_endpoint } " ) ​ DailyRoomParams Parameters for creating a new Daily room. ​ name string Room name (if not provided, one will be generated) ​ privacy string default: "public" Room privacy setting (“private” or “public”) ​ properties DailyRoomProperties Room configuration properties Copy Ask AI import time from pipecat.transports.services.helpers.daily_rest import ( DailyRoomParams, DailyRoomProperties, ) params = DailyRoomParams( name = "team-meeting" , privacy = "private" , properties = DailyRoomProperties( enable_chat = True , exp = time.time() + 7200 # 2 hours from now ) ) ​ DailyRoomObject Response object representing a Daily room. ​ id string Unique room identifier ​ name string Room name ​ api_created boolean Whether the room was created via API ​ privacy string Room privacy setting ​ url string Complete room URL ​ created_at string Room creation timestamp in ISO 8601 format ​ config DailyRoomProperties Room configuration Copy Ask AI from pipecat.transports.services.helpers.daily_rest import ( DailyRoomObject, DailyRoomProperties, ) # Example of what a DailyRoomObject looks like when received room = DailyRoomObject( id = "abc123" , name = "team-meeting" , api_created = True , privacy = "private" , url = "https://your-domain.daily.co/team-meeting" , created_at = "2024-01-20T10:00:00.000Z" , config = DailyRoomProperties( enable_chat = True , exp = 1705743600 ) ) ​ DailyMeetingTokenProperties Properties for configuring a Daily meeting token. ​ room_name string The room this token is valid for. If not set, token is valid for all rooms. ​ eject_at_token_exp boolean Whether to eject user when token expires ​ eject_after_elapsed integer Eject user after this many seconds ​ nbf integer “Not before” timestamp - users cannot join before this time ​ exp integer Expiration timestamp - users cannot join after this time ​ is_owner boolean Whether token grants owner privileges ​ user_name string User’s display name in the meeting ​ user_id string Unique identifier for the user (36 char limit) ​ enable_screenshare boolean Whether user can share their screen ​ start_video_off boolean Whether to join with video off ​ start_audio_off boolean Whether to join with audio off ​ enable_recording string Recording settings (“cloud”, “local”, or “raw-tracks”) ​ enable_prejoin_ui boolean Whether to show prejoin UI ​ start_cloud_recording boolean Whether to start cloud recording when user joins ​ permissions dict Initial default permissions for a non-meeting-owner participant ​ DailyMeetingTokenParams Parameters for creating a Daily meeting token. ​ properties DailyMeetingTokenProperties Token configuration properties Copy Ask AI from pipecat.transports.services.helpers.daily_rest import ( DailyMeetingTokenParams, DailyMeetingTokenProperties, ) token_params = DailyMeetingTokenParams( properties = DailyMeetingTokenProperties( user_name = "John Doe" , enable_screenshare = True , start_video_off = True , permissions = { "canSend" : [ "video" , "audio" ]} ) ) ​ Initialize DailyRESTHelper Create a new instance of the Daily REST helper. ​ daily_api_key string required Your Daily API key ​ daily_api_url string default: "https://api.daily.co/v1" The Daily API base URL ​ aiohttp_session aiohttp.ClientSession required An aiohttp client session for making HTTP requests Copy Ask AI helper = DailyRESTHelper( daily_api_key = "your-api-key" , aiohttp_session = session ) ​ Create Room Creates a new Daily room with specified parameters. ​ params DailyRoomParams required Room configuration parameters including name, privacy, and properties Copy Ask AI # Create a room that expires in 1 hour params = DailyRoomParams( name = "my-room" , privacy = "private" , properties = DailyRoomProperties( exp = time.time() + 3600 , enable_chat = True ) ) room = await helper.create_room(params) print ( f "Room URL: { room.url } " ) ​ Get Room From URL Retrieves room information using a Daily room URL. ​ room_url string required The complete Daily room URL Copy Ask AI room = await helper.get_room_from_url( "https://your-domain.daily.co/my-room" ) print ( f "Room name: { room.name } " ) ​ Get Token Generates a meeting token for a specific room. ​ room_url string required The complete Daily room URL ​ expiry_time float default: "3600" Token expiration time in seconds ​ eject_at_token_exp bool default: "False" Whether to eject user when token expires ​ owner bool default: "True" Whether the token should have owner privileges (overrides any setting in params) ​ params DailyMeetingTokenParams Additional token configuration. Note that room_name , exp , eject_at_token_exp , and is_owner will be set based on the other function parameters. Copy Ask AI # Basic token generation token = await helper.get_token( room_url = "https://your-domain.daily.co/my-room" , expiry_time = 1800 , # 30 minutes owner = True , eject_at_token_exp = True ) # Advanced token generation with additional properties token_params = DailyMeetingTokenParams( properties = DailyMeetingTokenProperties( user_name = "John Doe" , start_video_off = True ) ) token = await helper.get_token( room_url = "https://your-domain.daily.co/my-room" , expiry_time = 1800 , owner = False , eject_at_token_exp = True , params = token_params ) ​ Delete Room By URL Deletes a room using its URL. ​ room_url string required The complete Daily room URL Copy Ask AI success = await helper.delete_room_by_url( "https://your-domain.daily.co/my-room" ) if success: print ( "Room deleted successfully" ) ​ Delete Room By Name Deletes a room using its name. ​ room_name string required The name of the Daily room Copy Ask AI success = await helper.delete_room_by_name( "my-room" ) if success: print ( "Room deleted successfully" ) ​ Get Name From URL Extracts the room name from a Daily room URL. ​ room_url string required The complete Daily room URL Copy Ask AI room_name = helper.get_name_from_url( "https://your-domain.daily.co/my-room" ) print ( f "Room name: { room_name } " ) # Outputs: "my-room" Turn Tracking Observer Smart Turn Overview On this page Classes DailyRoomSipParams RecordingsBucketConfig DailyRoomProperties DailyRoomParams DailyRoomObject DailyMeetingTokenProperties DailyMeetingTokenParams Initialize DailyRESTHelper Create Room Get Room From URL Get Token Delete Room By URL Delete Room By Name Get Name From URL Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/daily_rest-helpers_63e24261.txt b/daily_rest-helpers_63e24261.txt
new file mode 100644
index 0000000000000000000000000000000000000000..6d134b63ec961a7b0e7637d64f69af1ef8d64cdf
--- /dev/null
+++ b/daily_rest-helpers_63e24261.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/daily/rest-helpers#param-enable-recording-1
+Title: Daily REST Helper - Pipecat
+==================================================
+
+Daily REST Helper - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Service Utilities Daily REST Helper Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Daily REST Helper Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline Daily REST API Documentation For complete Daily REST API reference and additional details ​ Classes ​ DailyRoomSipParams Configuration for SIP (Session Initiation Protocol) parameters. ​ display_name string default: "sw-sip-dialin" Display name for the SIP endpoint ​ video boolean default: false Whether video is enabled for SIP ​ sip_mode string default: "dial-in" SIP connection mode ​ num_endpoints integer default: 1 Number of SIP endpoints Copy Ask AI from pipecat.transports.services.helpers.daily_rest import DailyRoomSipParams sip_params = DailyRoomSipParams( display_name = "conference-line" , video = True , num_endpoints = 2 ) ​ RecordingsBucketConfig Configuration for storing Daily recordings in a custom S3 bucket. ​ bucket_name string required Name of the S3 bucket for storing recordings ​ bucket_region string required AWS region where the S3 bucket is located ​ assume_role_arn string required ARN of the IAM role to assume for S3 access ​ allow_api_access boolean default: false Whether to allow API access to the recordings Copy Ask AI from pipecat.transports.services.helpers.daily_rest import RecordingsBucketConfig bucket_config = RecordingsBucketConfig( bucket_name = "my-recordings-bucket" , bucket_region = "us-west-2" , assume_role_arn = "arn:aws:iam::123456789012:role/DailyRecordingsRole" , allow_api_access = True ) ​ DailyRoomProperties Properties that configure a Daily room’s behavior and features. ​ exp float Room expiration time as Unix timestamp (e.g., time.time() + 300 for 5 minutes) ​ enable_chat boolean default: false Whether chat is enabled in the room ​ enable_prejoin_ui boolean default: false Whether the prejoin lobby UI is enabled ​ enable_emoji_reactions boolean default: false Whether emoji reactions are enabled ​ eject_at_room_exp boolean default: false Whether to eject participants when room expires ​ enable_dialout boolean Whether dial-out is enabled ​ enable_recording string Recording settings (“cloud”, “local”, or “raw-tracks”) ​ geo string Geographic region for room ​ max_participants number Maximum number of participants allowed in the room ​ recordings_bucket RecordingsBucketConfig Configuration for custom S3 bucket recordings ​ sip DailyRoomSipParams SIP configuration parameters ​ sip_uri dict SIP URI configuration (returned by Daily) ​ start_video_off boolean default: false Whether the camera video is turned off by default The class also includes a sip_endpoint property that returns the SIP endpoint URI if available. Copy Ask AI import time from pipecat.transports.services.helpers.daily_rest import ( DailyRoomProperties, DailyRoomSipParams, RecordingsBucketConfig, ) properties = DailyRoomProperties( exp = time.time() + 3600 , # 1 hour from now enable_chat = True , enable_emoji_reactions = True , enable_recording = "cloud" , geo = "us-west" , max_participants = 50 , sip = DailyRoomSipParams( display_name = "conference" ), recordings_bucket = RecordingsBucketConfig( bucket_name = "my-bucket" , bucket_region = "us-west-2" , assume_role_arn = "arn:aws:iam::123456789012:role/DailyRole" ) ) # Access SIP endpoint if available if properties.sip_endpoint: print ( f "SIP endpoint: { properties.sip_endpoint } " ) ​ DailyRoomParams Parameters for creating a new Daily room. ​ name string Room name (if not provided, one will be generated) ​ privacy string default: "public" Room privacy setting (“private” or “public”) ​ properties DailyRoomProperties Room configuration properties Copy Ask AI import time from pipecat.transports.services.helpers.daily_rest import ( DailyRoomParams, DailyRoomProperties, ) params = DailyRoomParams( name = "team-meeting" , privacy = "private" , properties = DailyRoomProperties( enable_chat = True , exp = time.time() + 7200 # 2 hours from now ) ) ​ DailyRoomObject Response object representing a Daily room. ​ id string Unique room identifier ​ name string Room name ​ api_created boolean Whether the room was created via API ​ privacy string Room privacy setting ​ url string Complete room URL ​ created_at string Room creation timestamp in ISO 8601 format ​ config DailyRoomProperties Room configuration Copy Ask AI from pipecat.transports.services.helpers.daily_rest import ( DailyRoomObject, DailyRoomProperties, ) # Example of what a DailyRoomObject looks like when received room = DailyRoomObject( id = "abc123" , name = "team-meeting" , api_created = True , privacy = "private" , url = "https://your-domain.daily.co/team-meeting" , created_at = "2024-01-20T10:00:00.000Z" , config = DailyRoomProperties( enable_chat = True , exp = 1705743600 ) ) ​ DailyMeetingTokenProperties Properties for configuring a Daily meeting token. ​ room_name string The room this token is valid for. If not set, token is valid for all rooms. ​ eject_at_token_exp boolean Whether to eject user when token expires ​ eject_after_elapsed integer Eject user after this many seconds ​ nbf integer “Not before” timestamp - users cannot join before this time ​ exp integer Expiration timestamp - users cannot join after this time ​ is_owner boolean Whether token grants owner privileges ​ user_name string User’s display name in the meeting ​ user_id string Unique identifier for the user (36 char limit) ​ enable_screenshare boolean Whether user can share their screen ​ start_video_off boolean Whether to join with video off ​ start_audio_off boolean Whether to join with audio off ​ enable_recording string Recording settings (“cloud”, “local”, or “raw-tracks”) ​ enable_prejoin_ui boolean Whether to show prejoin UI ​ start_cloud_recording boolean Whether to start cloud recording when user joins ​ permissions dict Initial default permissions for a non-meeting-owner participant ​ DailyMeetingTokenParams Parameters for creating a Daily meeting token. ​ properties DailyMeetingTokenProperties Token configuration properties Copy Ask AI from pipecat.transports.services.helpers.daily_rest import ( DailyMeetingTokenParams, DailyMeetingTokenProperties, ) token_params = DailyMeetingTokenParams( properties = DailyMeetingTokenProperties( user_name = "John Doe" , enable_screenshare = True , start_video_off = True , permissions = { "canSend" : [ "video" , "audio" ]} ) ) ​ Initialize DailyRESTHelper Create a new instance of the Daily REST helper. ​ daily_api_key string required Your Daily API key ​ daily_api_url string default: "https://api.daily.co/v1" The Daily API base URL ​ aiohttp_session aiohttp.ClientSession required An aiohttp client session for making HTTP requests Copy Ask AI helper = DailyRESTHelper( daily_api_key = "your-api-key" , aiohttp_session = session ) ​ Create Room Creates a new Daily room with specified parameters. ​ params DailyRoomParams required Room configuration parameters including name, privacy, and properties Copy Ask AI # Create a room that expires in 1 hour params = DailyRoomParams( name = "my-room" , privacy = "private" , properties = DailyRoomProperties( exp = time.time() + 3600 , enable_chat = True ) ) room = await helper.create_room(params) print ( f "Room URL: { room.url } " ) ​ Get Room From URL Retrieves room information using a Daily room URL. ​ room_url string required The complete Daily room URL Copy Ask AI room = await helper.get_room_from_url( "https://your-domain.daily.co/my-room" ) print ( f "Room name: { room.name } " ) ​ Get Token Generates a meeting token for a specific room. ​ room_url string required The complete Daily room URL ​ expiry_time float default: "3600" Token expiration time in seconds ​ eject_at_token_exp bool default: "False" Whether to eject user when token expires ​ owner bool default: "True" Whether the token should have owner privileges (overrides any setting in params) ​ params DailyMeetingTokenParams Additional token configuration. Note that room_name , exp , eject_at_token_exp , and is_owner will be set based on the other function parameters. Copy Ask AI # Basic token generation token = await helper.get_token( room_url = "https://your-domain.daily.co/my-room" , expiry_time = 1800 , # 30 minutes owner = True , eject_at_token_exp = True ) # Advanced token generation with additional properties token_params = DailyMeetingTokenParams( properties = DailyMeetingTokenProperties( user_name = "John Doe" , start_video_off = True ) ) token = await helper.get_token( room_url = "https://your-domain.daily.co/my-room" , expiry_time = 1800 , owner = False , eject_at_token_exp = True , params = token_params ) ​ Delete Room By URL Deletes a room using its URL. ​ room_url string required The complete Daily room URL Copy Ask AI success = await helper.delete_room_by_url( "https://your-domain.daily.co/my-room" ) if success: print ( "Room deleted successfully" ) ​ Delete Room By Name Deletes a room using its name. ​ room_name string required The name of the Daily room Copy Ask AI success = await helper.delete_room_by_name( "my-room" ) if success: print ( "Room deleted successfully" ) ​ Get Name From URL Extracts the room name from a Daily room URL. ​ room_url string required The complete Daily room URL Copy Ask AI room_name = helper.get_name_from_url( "https://your-domain.daily.co/my-room" ) print ( f "Room name: { room_name } " ) # Outputs: "my-room" Turn Tracking Observer Smart Turn Overview On this page Classes DailyRoomSipParams RecordingsBucketConfig DailyRoomProperties DailyRoomParams DailyRoomObject DailyMeetingTokenProperties DailyMeetingTokenParams Initialize DailyRESTHelper Create Room Get Room From URL Get Token Delete Room By URL Delete Room By Name Get Name From URL Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/daily_rest-helpers_69acdcb3.txt b/daily_rest-helpers_69acdcb3.txt
new file mode 100644
index 0000000000000000000000000000000000000000..0c04420b9a6eabe5d6e7203dff09e6ca1e7bcaa1
--- /dev/null
+++ b/daily_rest-helpers_69acdcb3.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/daily/rest-helpers#param-exp
+Title: Daily REST Helper - Pipecat
+==================================================
+
+Daily REST Helper - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Service Utilities Daily REST Helper Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Daily REST Helper Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline Daily REST API Documentation For complete Daily REST API reference and additional details ​ Classes ​ DailyRoomSipParams Configuration for SIP (Session Initiation Protocol) parameters. ​ display_name string default: "sw-sip-dialin" Display name for the SIP endpoint ​ video boolean default: false Whether video is enabled for SIP ​ sip_mode string default: "dial-in" SIP connection mode ​ num_endpoints integer default: 1 Number of SIP endpoints Copy Ask AI from pipecat.transports.services.helpers.daily_rest import DailyRoomSipParams sip_params = DailyRoomSipParams( display_name = "conference-line" , video = True , num_endpoints = 2 ) ​ RecordingsBucketConfig Configuration for storing Daily recordings in a custom S3 bucket. ​ bucket_name string required Name of the S3 bucket for storing recordings ​ bucket_region string required AWS region where the S3 bucket is located ​ assume_role_arn string required ARN of the IAM role to assume for S3 access ​ allow_api_access boolean default: false Whether to allow API access to the recordings Copy Ask AI from pipecat.transports.services.helpers.daily_rest import RecordingsBucketConfig bucket_config = RecordingsBucketConfig( bucket_name = "my-recordings-bucket" , bucket_region = "us-west-2" , assume_role_arn = "arn:aws:iam::123456789012:role/DailyRecordingsRole" , allow_api_access = True ) ​ DailyRoomProperties Properties that configure a Daily room’s behavior and features. ​ exp float Room expiration time as Unix timestamp (e.g., time.time() + 300 for 5 minutes) ​ enable_chat boolean default: false Whether chat is enabled in the room ​ enable_prejoin_ui boolean default: false Whether the prejoin lobby UI is enabled ​ enable_emoji_reactions boolean default: false Whether emoji reactions are enabled ​ eject_at_room_exp boolean default: false Whether to eject participants when room expires ​ enable_dialout boolean Whether dial-out is enabled ​ enable_recording string Recording settings (“cloud”, “local”, or “raw-tracks”) ​ geo string Geographic region for room ​ max_participants number Maximum number of participants allowed in the room ​ recordings_bucket RecordingsBucketConfig Configuration for custom S3 bucket recordings ​ sip DailyRoomSipParams SIP configuration parameters ​ sip_uri dict SIP URI configuration (returned by Daily) ​ start_video_off boolean default: false Whether the camera video is turned off by default The class also includes a sip_endpoint property that returns the SIP endpoint URI if available. Copy Ask AI import time from pipecat.transports.services.helpers.daily_rest import ( DailyRoomProperties, DailyRoomSipParams, RecordingsBucketConfig, ) properties = DailyRoomProperties( exp = time.time() + 3600 , # 1 hour from now enable_chat = True , enable_emoji_reactions = True , enable_recording = "cloud" , geo = "us-west" , max_participants = 50 , sip = DailyRoomSipParams( display_name = "conference" ), recordings_bucket = RecordingsBucketConfig( bucket_name = "my-bucket" , bucket_region = "us-west-2" , assume_role_arn = "arn:aws:iam::123456789012:role/DailyRole" ) ) # Access SIP endpoint if available if properties.sip_endpoint: print ( f "SIP endpoint: { properties.sip_endpoint } " ) ​ DailyRoomParams Parameters for creating a new Daily room. ​ name string Room name (if not provided, one will be generated) ​ privacy string default: "public" Room privacy setting (“private” or “public”) ​ properties DailyRoomProperties Room configuration properties Copy Ask AI import time from pipecat.transports.services.helpers.daily_rest import ( DailyRoomParams, DailyRoomProperties, ) params = DailyRoomParams( name = "team-meeting" , privacy = "private" , properties = DailyRoomProperties( enable_chat = True , exp = time.time() + 7200 # 2 hours from now ) ) ​ DailyRoomObject Response object representing a Daily room. ​ id string Unique room identifier ​ name string Room name ​ api_created boolean Whether the room was created via API ​ privacy string Room privacy setting ​ url string Complete room URL ​ created_at string Room creation timestamp in ISO 8601 format ​ config DailyRoomProperties Room configuration Copy Ask AI from pipecat.transports.services.helpers.daily_rest import ( DailyRoomObject, DailyRoomProperties, ) # Example of what a DailyRoomObject looks like when received room = DailyRoomObject( id = "abc123" , name = "team-meeting" , api_created = True , privacy = "private" , url = "https://your-domain.daily.co/team-meeting" , created_at = "2024-01-20T10:00:00.000Z" , config = DailyRoomProperties( enable_chat = True , exp = 1705743600 ) ) ​ DailyMeetingTokenProperties Properties for configuring a Daily meeting token. ​ room_name string The room this token is valid for. If not set, token is valid for all rooms. ​ eject_at_token_exp boolean Whether to eject user when token expires ​ eject_after_elapsed integer Eject user after this many seconds ​ nbf integer “Not before” timestamp - users cannot join before this time ​ exp integer Expiration timestamp - users cannot join after this time ​ is_owner boolean Whether token grants owner privileges ​ user_name string User’s display name in the meeting ​ user_id string Unique identifier for the user (36 char limit) ​ enable_screenshare boolean Whether user can share their screen ​ start_video_off boolean Whether to join with video off ​ start_audio_off boolean Whether to join with audio off ​ enable_recording string Recording settings (“cloud”, “local”, or “raw-tracks”) ​ enable_prejoin_ui boolean Whether to show prejoin UI ​ start_cloud_recording boolean Whether to start cloud recording when user joins ​ permissions dict Initial default permissions for a non-meeting-owner participant ​ DailyMeetingTokenParams Parameters for creating a Daily meeting token. ​ properties DailyMeetingTokenProperties Token configuration properties Copy Ask AI from pipecat.transports.services.helpers.daily_rest import ( DailyMeetingTokenParams, DailyMeetingTokenProperties, ) token_params = DailyMeetingTokenParams( properties = DailyMeetingTokenProperties( user_name = "John Doe" , enable_screenshare = True , start_video_off = True , permissions = { "canSend" : [ "video" , "audio" ]} ) ) ​ Initialize DailyRESTHelper Create a new instance of the Daily REST helper. ​ daily_api_key string required Your Daily API key ​ daily_api_url string default: "https://api.daily.co/v1" The Daily API base URL ​ aiohttp_session aiohttp.ClientSession required An aiohttp client session for making HTTP requests Copy Ask AI helper = DailyRESTHelper( daily_api_key = "your-api-key" , aiohttp_session = session ) ​ Create Room Creates a new Daily room with specified parameters. ​ params DailyRoomParams required Room configuration parameters including name, privacy, and properties Copy Ask AI # Create a room that expires in 1 hour params = DailyRoomParams( name = "my-room" , privacy = "private" , properties = DailyRoomProperties( exp = time.time() + 3600 , enable_chat = True ) ) room = await helper.create_room(params) print ( f "Room URL: { room.url } " ) ​ Get Room From URL Retrieves room information using a Daily room URL. ​ room_url string required The complete Daily room URL Copy Ask AI room = await helper.get_room_from_url( "https://your-domain.daily.co/my-room" ) print ( f "Room name: { room.name } " ) ​ Get Token Generates a meeting token for a specific room. ​ room_url string required The complete Daily room URL ​ expiry_time float default: "3600" Token expiration time in seconds ​ eject_at_token_exp bool default: "False" Whether to eject user when token expires ​ owner bool default: "True" Whether the token should have owner privileges (overrides any setting in params) ​ params DailyMeetingTokenParams Additional token configuration. Note that room_name , exp , eject_at_token_exp , and is_owner will be set based on the other function parameters. Copy Ask AI # Basic token generation token = await helper.get_token( room_url = "https://your-domain.daily.co/my-room" , expiry_time = 1800 , # 30 minutes owner = True , eject_at_token_exp = True ) # Advanced token generation with additional properties token_params = DailyMeetingTokenParams( properties = DailyMeetingTokenProperties( user_name = "John Doe" , start_video_off = True ) ) token = await helper.get_token( room_url = "https://your-domain.daily.co/my-room" , expiry_time = 1800 , owner = False , eject_at_token_exp = True , params = token_params ) ​ Delete Room By URL Deletes a room using its URL. ​ room_url string required The complete Daily room URL Copy Ask AI success = await helper.delete_room_by_url( "https://your-domain.daily.co/my-room" ) if success: print ( "Room deleted successfully" ) ​ Delete Room By Name Deletes a room using its name. ​ room_name string required The name of the Daily room Copy Ask AI success = await helper.delete_room_by_name( "my-room" ) if success: print ( "Room deleted successfully" ) ​ Get Name From URL Extracts the room name from a Daily room URL. ​ room_url string required The complete Daily room URL Copy Ask AI room_name = helper.get_name_from_url( "https://your-domain.daily.co/my-room" ) print ( f "Room name: { room_name } " ) # Outputs: "my-room" Turn Tracking Observer Smart Turn Overview On this page Classes DailyRoomSipParams RecordingsBucketConfig DailyRoomProperties DailyRoomParams DailyRoomObject DailyMeetingTokenProperties DailyMeetingTokenParams Initialize DailyRESTHelper Create Room Get Room From URL Get Token Delete Room By URL Delete Room By Name Get Name From URL Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/daily_rest-helpers_6f051d35.txt b/daily_rest-helpers_6f051d35.txt
new file mode 100644
index 0000000000000000000000000000000000000000..6fa0c10a3df6eecf8a3c00c779066796418c8871
--- /dev/null
+++ b/daily_rest-helpers_6f051d35.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/daily/rest-helpers#param-bucket-name
+Title: Daily REST Helper - Pipecat
+==================================================
+
+Daily REST Helper - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Service Utilities Daily REST Helper Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Daily REST Helper Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline Daily REST API Documentation For complete Daily REST API reference and additional details ​ Classes ​ DailyRoomSipParams Configuration for SIP (Session Initiation Protocol) parameters. ​ display_name string default: "sw-sip-dialin" Display name for the SIP endpoint ​ video boolean default: false Whether video is enabled for SIP ​ sip_mode string default: "dial-in" SIP connection mode ​ num_endpoints integer default: 1 Number of SIP endpoints Copy Ask AI from pipecat.transports.services.helpers.daily_rest import DailyRoomSipParams sip_params = DailyRoomSipParams( display_name = "conference-line" , video = True , num_endpoints = 2 ) ​ RecordingsBucketConfig Configuration for storing Daily recordings in a custom S3 bucket. ​ bucket_name string required Name of the S3 bucket for storing recordings ​ bucket_region string required AWS region where the S3 bucket is located ​ assume_role_arn string required ARN of the IAM role to assume for S3 access ​ allow_api_access boolean default: false Whether to allow API access to the recordings Copy Ask AI from pipecat.transports.services.helpers.daily_rest import RecordingsBucketConfig bucket_config = RecordingsBucketConfig( bucket_name = "my-recordings-bucket" , bucket_region = "us-west-2" , assume_role_arn = "arn:aws:iam::123456789012:role/DailyRecordingsRole" , allow_api_access = True ) ​ DailyRoomProperties Properties that configure a Daily room’s behavior and features. ​ exp float Room expiration time as Unix timestamp (e.g., time.time() + 300 for 5 minutes) ​ enable_chat boolean default: false Whether chat is enabled in the room ​ enable_prejoin_ui boolean default: false Whether the prejoin lobby UI is enabled ​ enable_emoji_reactions boolean default: false Whether emoji reactions are enabled ​ eject_at_room_exp boolean default: false Whether to eject participants when room expires ​ enable_dialout boolean Whether dial-out is enabled ​ enable_recording string Recording settings (“cloud”, “local”, or “raw-tracks”) ​ geo string Geographic region for room ​ max_participants number Maximum number of participants allowed in the room ​ recordings_bucket RecordingsBucketConfig Configuration for custom S3 bucket recordings ​ sip DailyRoomSipParams SIP configuration parameters ​ sip_uri dict SIP URI configuration (returned by Daily) ​ start_video_off boolean default: false Whether the camera video is turned off by default The class also includes a sip_endpoint property that returns the SIP endpoint URI if available. Copy Ask AI import time from pipecat.transports.services.helpers.daily_rest import ( DailyRoomProperties, DailyRoomSipParams, RecordingsBucketConfig, ) properties = DailyRoomProperties( exp = time.time() + 3600 , # 1 hour from now enable_chat = True , enable_emoji_reactions = True , enable_recording = "cloud" , geo = "us-west" , max_participants = 50 , sip = DailyRoomSipParams( display_name = "conference" ), recordings_bucket = RecordingsBucketConfig( bucket_name = "my-bucket" , bucket_region = "us-west-2" , assume_role_arn = "arn:aws:iam::123456789012:role/DailyRole" ) ) # Access SIP endpoint if available if properties.sip_endpoint: print ( f "SIP endpoint: { properties.sip_endpoint } " ) ​ DailyRoomParams Parameters for creating a new Daily room. ​ name string Room name (if not provided, one will be generated) ​ privacy string default: "public" Room privacy setting (“private” or “public”) ​ properties DailyRoomProperties Room configuration properties Copy Ask AI import time from pipecat.transports.services.helpers.daily_rest import ( DailyRoomParams, DailyRoomProperties, ) params = DailyRoomParams( name = "team-meeting" , privacy = "private" , properties = DailyRoomProperties( enable_chat = True , exp = time.time() + 7200 # 2 hours from now ) ) ​ DailyRoomObject Response object representing a Daily room. ​ id string Unique room identifier ​ name string Room name ​ api_created boolean Whether the room was created via API ​ privacy string Room privacy setting ​ url string Complete room URL ​ created_at string Room creation timestamp in ISO 8601 format ​ config DailyRoomProperties Room configuration Copy Ask AI from pipecat.transports.services.helpers.daily_rest import ( DailyRoomObject, DailyRoomProperties, ) # Example of what a DailyRoomObject looks like when received room = DailyRoomObject( id = "abc123" , name = "team-meeting" , api_created = True , privacy = "private" , url = "https://your-domain.daily.co/team-meeting" , created_at = "2024-01-20T10:00:00.000Z" , config = DailyRoomProperties( enable_chat = True , exp = 1705743600 ) ) ​ DailyMeetingTokenProperties Properties for configuring a Daily meeting token. ​ room_name string The room this token is valid for. If not set, token is valid for all rooms. ​ eject_at_token_exp boolean Whether to eject user when token expires ​ eject_after_elapsed integer Eject user after this many seconds ​ nbf integer “Not before” timestamp - users cannot join before this time ​ exp integer Expiration timestamp - users cannot join after this time ​ is_owner boolean Whether token grants owner privileges ​ user_name string User’s display name in the meeting ​ user_id string Unique identifier for the user (36 char limit) ​ enable_screenshare boolean Whether user can share their screen ​ start_video_off boolean Whether to join with video off ​ start_audio_off boolean Whether to join with audio off ​ enable_recording string Recording settings (“cloud”, “local”, or “raw-tracks”) ​ enable_prejoin_ui boolean Whether to show prejoin UI ​ start_cloud_recording boolean Whether to start cloud recording when user joins ​ permissions dict Initial default permissions for a non-meeting-owner participant ​ DailyMeetingTokenParams Parameters for creating a Daily meeting token. ​ properties DailyMeetingTokenProperties Token configuration properties Copy Ask AI from pipecat.transports.services.helpers.daily_rest import ( DailyMeetingTokenParams, DailyMeetingTokenProperties, ) token_params = DailyMeetingTokenParams( properties = DailyMeetingTokenProperties( user_name = "John Doe" , enable_screenshare = True , start_video_off = True , permissions = { "canSend" : [ "video" , "audio" ]} ) ) ​ Initialize DailyRESTHelper Create a new instance of the Daily REST helper. ​ daily_api_key string required Your Daily API key ​ daily_api_url string default: "https://api.daily.co/v1" The Daily API base URL ​ aiohttp_session aiohttp.ClientSession required An aiohttp client session for making HTTP requests Copy Ask AI helper = DailyRESTHelper( daily_api_key = "your-api-key" , aiohttp_session = session ) ​ Create Room Creates a new Daily room with specified parameters. ​ params DailyRoomParams required Room configuration parameters including name, privacy, and properties Copy Ask AI # Create a room that expires in 1 hour params = DailyRoomParams( name = "my-room" , privacy = "private" , properties = DailyRoomProperties( exp = time.time() + 3600 , enable_chat = True ) ) room = await helper.create_room(params) print ( f "Room URL: { room.url } " ) ​ Get Room From URL Retrieves room information using a Daily room URL. ​ room_url string required The complete Daily room URL Copy Ask AI room = await helper.get_room_from_url( "https://your-domain.daily.co/my-room" ) print ( f "Room name: { room.name } " ) ​ Get Token Generates a meeting token for a specific room. ​ room_url string required The complete Daily room URL ​ expiry_time float default: "3600" Token expiration time in seconds ​ eject_at_token_exp bool default: "False" Whether to eject user when token expires ​ owner bool default: "True" Whether the token should have owner privileges (overrides any setting in params) ​ params DailyMeetingTokenParams Additional token configuration. Note that room_name , exp , eject_at_token_exp , and is_owner will be set based on the other function parameters. Copy Ask AI # Basic token generation token = await helper.get_token( room_url = "https://your-domain.daily.co/my-room" , expiry_time = 1800 , # 30 minutes owner = True , eject_at_token_exp = True ) # Advanced token generation with additional properties token_params = DailyMeetingTokenParams( properties = DailyMeetingTokenProperties( user_name = "John Doe" , start_video_off = True ) ) token = await helper.get_token( room_url = "https://your-domain.daily.co/my-room" , expiry_time = 1800 , owner = False , eject_at_token_exp = True , params = token_params ) ​ Delete Room By URL Deletes a room using its URL. ​ room_url string required The complete Daily room URL Copy Ask AI success = await helper.delete_room_by_url( "https://your-domain.daily.co/my-room" ) if success: print ( "Room deleted successfully" ) ​ Delete Room By Name Deletes a room using its name. ​ room_name string required The name of the Daily room Copy Ask AI success = await helper.delete_room_by_name( "my-room" ) if success: print ( "Room deleted successfully" ) ​ Get Name From URL Extracts the room name from a Daily room URL. ​ room_url string required The complete Daily room URL Copy Ask AI room_name = helper.get_name_from_url( "https://your-domain.daily.co/my-room" ) print ( f "Room name: { room_name } " ) # Outputs: "my-room" Turn Tracking Observer Smart Turn Overview On this page Classes DailyRoomSipParams RecordingsBucketConfig DailyRoomProperties DailyRoomParams DailyRoomObject DailyMeetingTokenProperties DailyMeetingTokenParams Initialize DailyRESTHelper Create Room Get Room From URL Get Token Delete Room By URL Delete Room By Name Get Name From URL Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/daily_rest-helpers_73593cc5.txt b/daily_rest-helpers_73593cc5.txt
new file mode 100644
index 0000000000000000000000000000000000000000..d6b37965d3805a25871b128950a8af9a7bc3b733
--- /dev/null
+++ b/daily_rest-helpers_73593cc5.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/daily/rest-helpers#dailyroomobject
+Title: Daily REST Helper - Pipecat
+==================================================
+
+Daily REST Helper - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Service Utilities Daily REST Helper Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Daily REST Helper Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline Daily REST API Documentation For complete Daily REST API reference and additional details ​ Classes ​ DailyRoomSipParams Configuration for SIP (Session Initiation Protocol) parameters. ​ display_name string default: "sw-sip-dialin" Display name for the SIP endpoint ​ video boolean default: false Whether video is enabled for SIP ​ sip_mode string default: "dial-in" SIP connection mode ​ num_endpoints integer default: 1 Number of SIP endpoints Copy Ask AI from pipecat.transports.services.helpers.daily_rest import DailyRoomSipParams sip_params = DailyRoomSipParams( display_name = "conference-line" , video = True , num_endpoints = 2 ) ​ RecordingsBucketConfig Configuration for storing Daily recordings in a custom S3 bucket. ​ bucket_name string required Name of the S3 bucket for storing recordings ​ bucket_region string required AWS region where the S3 bucket is located ​ assume_role_arn string required ARN of the IAM role to assume for S3 access ​ allow_api_access boolean default: false Whether to allow API access to the recordings Copy Ask AI from pipecat.transports.services.helpers.daily_rest import RecordingsBucketConfig bucket_config = RecordingsBucketConfig( bucket_name = "my-recordings-bucket" , bucket_region = "us-west-2" , assume_role_arn = "arn:aws:iam::123456789012:role/DailyRecordingsRole" , allow_api_access = True ) ​ DailyRoomProperties Properties that configure a Daily room’s behavior and features. ​ exp float Room expiration time as Unix timestamp (e.g., time.time() + 300 for 5 minutes) ​ enable_chat boolean default: false Whether chat is enabled in the room ​ enable_prejoin_ui boolean default: false Whether the prejoin lobby UI is enabled ​ enable_emoji_reactions boolean default: false Whether emoji reactions are enabled ​ eject_at_room_exp boolean default: false Whether to eject participants when room expires ​ enable_dialout boolean Whether dial-out is enabled ​ enable_recording string Recording settings (“cloud”, “local”, or “raw-tracks”) ​ geo string Geographic region for room ​ max_participants number Maximum number of participants allowed in the room ​ recordings_bucket RecordingsBucketConfig Configuration for custom S3 bucket recordings ​ sip DailyRoomSipParams SIP configuration parameters ​ sip_uri dict SIP URI configuration (returned by Daily) ​ start_video_off boolean default: false Whether the camera video is turned off by default The class also includes a sip_endpoint property that returns the SIP endpoint URI if available. Copy Ask AI import time from pipecat.transports.services.helpers.daily_rest import ( DailyRoomProperties, DailyRoomSipParams, RecordingsBucketConfig, ) properties = DailyRoomProperties( exp = time.time() + 3600 , # 1 hour from now enable_chat = True , enable_emoji_reactions = True , enable_recording = "cloud" , geo = "us-west" , max_participants = 50 , sip = DailyRoomSipParams( display_name = "conference" ), recordings_bucket = RecordingsBucketConfig( bucket_name = "my-bucket" , bucket_region = "us-west-2" , assume_role_arn = "arn:aws:iam::123456789012:role/DailyRole" ) ) # Access SIP endpoint if available if properties.sip_endpoint: print ( f "SIP endpoint: { properties.sip_endpoint } " ) ​ DailyRoomParams Parameters for creating a new Daily room. ​ name string Room name (if not provided, one will be generated) ​ privacy string default: "public" Room privacy setting (“private” or “public”) ​ properties DailyRoomProperties Room configuration properties Copy Ask AI import time from pipecat.transports.services.helpers.daily_rest import ( DailyRoomParams, DailyRoomProperties, ) params = DailyRoomParams( name = "team-meeting" , privacy = "private" , properties = DailyRoomProperties( enable_chat = True , exp = time.time() + 7200 # 2 hours from now ) ) ​ DailyRoomObject Response object representing a Daily room. ​ id string Unique room identifier ​ name string Room name ​ api_created boolean Whether the room was created via API ​ privacy string Room privacy setting ​ url string Complete room URL ​ created_at string Room creation timestamp in ISO 8601 format ​ config DailyRoomProperties Room configuration Copy Ask AI from pipecat.transports.services.helpers.daily_rest import ( DailyRoomObject, DailyRoomProperties, ) # Example of what a DailyRoomObject looks like when received room = DailyRoomObject( id = "abc123" , name = "team-meeting" , api_created = True , privacy = "private" , url = "https://your-domain.daily.co/team-meeting" , created_at = "2024-01-20T10:00:00.000Z" , config = DailyRoomProperties( enable_chat = True , exp = 1705743600 ) ) ​ DailyMeetingTokenProperties Properties for configuring a Daily meeting token. ​ room_name string The room this token is valid for. If not set, token is valid for all rooms. ​ eject_at_token_exp boolean Whether to eject user when token expires ​ eject_after_elapsed integer Eject user after this many seconds ​ nbf integer “Not before” timestamp - users cannot join before this time ​ exp integer Expiration timestamp - users cannot join after this time ​ is_owner boolean Whether token grants owner privileges ​ user_name string User’s display name in the meeting ​ user_id string Unique identifier for the user (36 char limit) ​ enable_screenshare boolean Whether user can share their screen ​ start_video_off boolean Whether to join with video off ​ start_audio_off boolean Whether to join with audio off ​ enable_recording string Recording settings (“cloud”, “local”, or “raw-tracks”) ​ enable_prejoin_ui boolean Whether to show prejoin UI ​ start_cloud_recording boolean Whether to start cloud recording when user joins ​ permissions dict Initial default permissions for a non-meeting-owner participant ​ DailyMeetingTokenParams Parameters for creating a Daily meeting token. ​ properties DailyMeetingTokenProperties Token configuration properties Copy Ask AI from pipecat.transports.services.helpers.daily_rest import ( DailyMeetingTokenParams, DailyMeetingTokenProperties, ) token_params = DailyMeetingTokenParams( properties = DailyMeetingTokenProperties( user_name = "John Doe" , enable_screenshare = True , start_video_off = True , permissions = { "canSend" : [ "video" , "audio" ]} ) ) ​ Initialize DailyRESTHelper Create a new instance of the Daily REST helper. ​ daily_api_key string required Your Daily API key ​ daily_api_url string default: "https://api.daily.co/v1" The Daily API base URL ​ aiohttp_session aiohttp.ClientSession required An aiohttp client session for making HTTP requests Copy Ask AI helper = DailyRESTHelper( daily_api_key = "your-api-key" , aiohttp_session = session ) ​ Create Room Creates a new Daily room with specified parameters. ​ params DailyRoomParams required Room configuration parameters including name, privacy, and properties Copy Ask AI # Create a room that expires in 1 hour params = DailyRoomParams( name = "my-room" , privacy = "private" , properties = DailyRoomProperties( exp = time.time() + 3600 , enable_chat = True ) ) room = await helper.create_room(params) print ( f "Room URL: { room.url } " ) ​ Get Room From URL Retrieves room information using a Daily room URL. ​ room_url string required The complete Daily room URL Copy Ask AI room = await helper.get_room_from_url( "https://your-domain.daily.co/my-room" ) print ( f "Room name: { room.name } " ) ​ Get Token Generates a meeting token for a specific room. ​ room_url string required The complete Daily room URL ​ expiry_time float default: "3600" Token expiration time in seconds ​ eject_at_token_exp bool default: "False" Whether to eject user when token expires ​ owner bool default: "True" Whether the token should have owner privileges (overrides any setting in params) ​ params DailyMeetingTokenParams Additional token configuration. Note that room_name , exp , eject_at_token_exp , and is_owner will be set based on the other function parameters. Copy Ask AI # Basic token generation token = await helper.get_token( room_url = "https://your-domain.daily.co/my-room" , expiry_time = 1800 , # 30 minutes owner = True , eject_at_token_exp = True ) # Advanced token generation with additional properties token_params = DailyMeetingTokenParams( properties = DailyMeetingTokenProperties( user_name = "John Doe" , start_video_off = True ) ) token = await helper.get_token( room_url = "https://your-domain.daily.co/my-room" , expiry_time = 1800 , owner = False , eject_at_token_exp = True , params = token_params ) ​ Delete Room By URL Deletes a room using its URL. ​ room_url string required The complete Daily room URL Copy Ask AI success = await helper.delete_room_by_url( "https://your-domain.daily.co/my-room" ) if success: print ( "Room deleted successfully" ) ​ Delete Room By Name Deletes a room using its name. ​ room_name string required The name of the Daily room Copy Ask AI success = await helper.delete_room_by_name( "my-room" ) if success: print ( "Room deleted successfully" ) ​ Get Name From URL Extracts the room name from a Daily room URL. ​ room_url string required The complete Daily room URL Copy Ask AI room_name = helper.get_name_from_url( "https://your-domain.daily.co/my-room" ) print ( f "Room name: { room_name } " ) # Outputs: "my-room" Turn Tracking Observer Smart Turn Overview On this page Classes DailyRoomSipParams RecordingsBucketConfig DailyRoomProperties DailyRoomParams DailyRoomObject DailyMeetingTokenProperties DailyMeetingTokenParams Initialize DailyRESTHelper Create Room Get Room From URL Get Token Delete Room By URL Delete Room By Name Get Name From URL Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/daily_rest-helpers_75322475.txt b/daily_rest-helpers_75322475.txt
new file mode 100644
index 0000000000000000000000000000000000000000..d8ef4d1a9953262711d5d5426d8e0002259e2f87
--- /dev/null
+++ b/daily_rest-helpers_75322475.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/daily/rest-helpers#param-eject-at-room-exp
+Title: Daily REST Helper - Pipecat
+==================================================
+
+Daily REST Helper - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Service Utilities Daily REST Helper Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Daily REST Helper Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline Daily REST API Documentation For complete Daily REST API reference and additional details ​ Classes ​ DailyRoomSipParams Configuration for SIP (Session Initiation Protocol) parameters. ​ display_name string default: "sw-sip-dialin" Display name for the SIP endpoint ​ video boolean default: false Whether video is enabled for SIP ​ sip_mode string default: "dial-in" SIP connection mode ​ num_endpoints integer default: 1 Number of SIP endpoints Copy Ask AI from pipecat.transports.services.helpers.daily_rest import DailyRoomSipParams sip_params = DailyRoomSipParams( display_name = "conference-line" , video = True , num_endpoints = 2 ) ​ RecordingsBucketConfig Configuration for storing Daily recordings in a custom S3 bucket. ​ bucket_name string required Name of the S3 bucket for storing recordings ​ bucket_region string required AWS region where the S3 bucket is located ​ assume_role_arn string required ARN of the IAM role to assume for S3 access ​ allow_api_access boolean default: false Whether to allow API access to the recordings Copy Ask AI from pipecat.transports.services.helpers.daily_rest import RecordingsBucketConfig bucket_config = RecordingsBucketConfig( bucket_name = "my-recordings-bucket" , bucket_region = "us-west-2" , assume_role_arn = "arn:aws:iam::123456789012:role/DailyRecordingsRole" , allow_api_access = True ) ​ DailyRoomProperties Properties that configure a Daily room’s behavior and features. ​ exp float Room expiration time as Unix timestamp (e.g., time.time() + 300 for 5 minutes) ​ enable_chat boolean default: false Whether chat is enabled in the room ​ enable_prejoin_ui boolean default: false Whether the prejoin lobby UI is enabled ​ enable_emoji_reactions boolean default: false Whether emoji reactions are enabled ​ eject_at_room_exp boolean default: false Whether to eject participants when room expires ​ enable_dialout boolean Whether dial-out is enabled ​ enable_recording string Recording settings (“cloud”, “local”, or “raw-tracks”) ​ geo string Geographic region for room ​ max_participants number Maximum number of participants allowed in the room ​ recordings_bucket RecordingsBucketConfig Configuration for custom S3 bucket recordings ​ sip DailyRoomSipParams SIP configuration parameters ​ sip_uri dict SIP URI configuration (returned by Daily) ​ start_video_off boolean default: false Whether the camera video is turned off by default The class also includes a sip_endpoint property that returns the SIP endpoint URI if available. Copy Ask AI import time from pipecat.transports.services.helpers.daily_rest import ( DailyRoomProperties, DailyRoomSipParams, RecordingsBucketConfig, ) properties = DailyRoomProperties( exp = time.time() + 3600 , # 1 hour from now enable_chat = True , enable_emoji_reactions = True , enable_recording = "cloud" , geo = "us-west" , max_participants = 50 , sip = DailyRoomSipParams( display_name = "conference" ), recordings_bucket = RecordingsBucketConfig( bucket_name = "my-bucket" , bucket_region = "us-west-2" , assume_role_arn = "arn:aws:iam::123456789012:role/DailyRole" ) ) # Access SIP endpoint if available if properties.sip_endpoint: print ( f "SIP endpoint: { properties.sip_endpoint } " ) ​ DailyRoomParams Parameters for creating a new Daily room. ​ name string Room name (if not provided, one will be generated) ​ privacy string default: "public" Room privacy setting (“private” or “public”) ​ properties DailyRoomProperties Room configuration properties Copy Ask AI import time from pipecat.transports.services.helpers.daily_rest import ( DailyRoomParams, DailyRoomProperties, ) params = DailyRoomParams( name = "team-meeting" , privacy = "private" , properties = DailyRoomProperties( enable_chat = True , exp = time.time() + 7200 # 2 hours from now ) ) ​ DailyRoomObject Response object representing a Daily room. ​ id string Unique room identifier ​ name string Room name ​ api_created boolean Whether the room was created via API ​ privacy string Room privacy setting ​ url string Complete room URL ​ created_at string Room creation timestamp in ISO 8601 format ​ config DailyRoomProperties Room configuration Copy Ask AI from pipecat.transports.services.helpers.daily_rest import ( DailyRoomObject, DailyRoomProperties, ) # Example of what a DailyRoomObject looks like when received room = DailyRoomObject( id = "abc123" , name = "team-meeting" , api_created = True , privacy = "private" , url = "https://your-domain.daily.co/team-meeting" , created_at = "2024-01-20T10:00:00.000Z" , config = DailyRoomProperties( enable_chat = True , exp = 1705743600 ) ) ​ DailyMeetingTokenProperties Properties for configuring a Daily meeting token. ​ room_name string The room this token is valid for. If not set, token is valid for all rooms. ​ eject_at_token_exp boolean Whether to eject user when token expires ​ eject_after_elapsed integer Eject user after this many seconds ​ nbf integer “Not before” timestamp - users cannot join before this time ​ exp integer Expiration timestamp - users cannot join after this time ​ is_owner boolean Whether token grants owner privileges ​ user_name string User’s display name in the meeting ​ user_id string Unique identifier for the user (36 char limit) ​ enable_screenshare boolean Whether user can share their screen ​ start_video_off boolean Whether to join with video off ​ start_audio_off boolean Whether to join with audio off ​ enable_recording string Recording settings (“cloud”, “local”, or “raw-tracks”) ​ enable_prejoin_ui boolean Whether to show prejoin UI ​ start_cloud_recording boolean Whether to start cloud recording when user joins ​ permissions dict Initial default permissions for a non-meeting-owner participant ​ DailyMeetingTokenParams Parameters for creating a Daily meeting token. ​ properties DailyMeetingTokenProperties Token configuration properties Copy Ask AI from pipecat.transports.services.helpers.daily_rest import ( DailyMeetingTokenParams, DailyMeetingTokenProperties, ) token_params = DailyMeetingTokenParams( properties = DailyMeetingTokenProperties( user_name = "John Doe" , enable_screenshare = True , start_video_off = True , permissions = { "canSend" : [ "video" , "audio" ]} ) ) ​ Initialize DailyRESTHelper Create a new instance of the Daily REST helper. ​ daily_api_key string required Your Daily API key ​ daily_api_url string default: "https://api.daily.co/v1" The Daily API base URL ​ aiohttp_session aiohttp.ClientSession required An aiohttp client session for making HTTP requests Copy Ask AI helper = DailyRESTHelper( daily_api_key = "your-api-key" , aiohttp_session = session ) ​ Create Room Creates a new Daily room with specified parameters. ​ params DailyRoomParams required Room configuration parameters including name, privacy, and properties Copy Ask AI # Create a room that expires in 1 hour params = DailyRoomParams( name = "my-room" , privacy = "private" , properties = DailyRoomProperties( exp = time.time() + 3600 , enable_chat = True ) ) room = await helper.create_room(params) print ( f "Room URL: { room.url } " ) ​ Get Room From URL Retrieves room information using a Daily room URL. ​ room_url string required The complete Daily room URL Copy Ask AI room = await helper.get_room_from_url( "https://your-domain.daily.co/my-room" ) print ( f "Room name: { room.name } " ) ​ Get Token Generates a meeting token for a specific room. ​ room_url string required The complete Daily room URL ​ expiry_time float default: "3600" Token expiration time in seconds ​ eject_at_token_exp bool default: "False" Whether to eject user when token expires ​ owner bool default: "True" Whether the token should have owner privileges (overrides any setting in params) ​ params DailyMeetingTokenParams Additional token configuration. Note that room_name , exp , eject_at_token_exp , and is_owner will be set based on the other function parameters. Copy Ask AI # Basic token generation token = await helper.get_token( room_url = "https://your-domain.daily.co/my-room" , expiry_time = 1800 , # 30 minutes owner = True , eject_at_token_exp = True ) # Advanced token generation with additional properties token_params = DailyMeetingTokenParams( properties = DailyMeetingTokenProperties( user_name = "John Doe" , start_video_off = True ) ) token = await helper.get_token( room_url = "https://your-domain.daily.co/my-room" , expiry_time = 1800 , owner = False , eject_at_token_exp = True , params = token_params ) ​ Delete Room By URL Deletes a room using its URL. ​ room_url string required The complete Daily room URL Copy Ask AI success = await helper.delete_room_by_url( "https://your-domain.daily.co/my-room" ) if success: print ( "Room deleted successfully" ) ​ Delete Room By Name Deletes a room using its name. ​ room_name string required The name of the Daily room Copy Ask AI success = await helper.delete_room_by_name( "my-room" ) if success: print ( "Room deleted successfully" ) ​ Get Name From URL Extracts the room name from a Daily room URL. ​ room_url string required The complete Daily room URL Copy Ask AI room_name = helper.get_name_from_url( "https://your-domain.daily.co/my-room" ) print ( f "Room name: { room_name } " ) # Outputs: "my-room" Turn Tracking Observer Smart Turn Overview On this page Classes DailyRoomSipParams RecordingsBucketConfig DailyRoomProperties DailyRoomParams DailyRoomObject DailyMeetingTokenProperties DailyMeetingTokenParams Initialize DailyRESTHelper Create Room Get Room From URL Get Token Delete Room By URL Delete Room By Name Get Name From URL Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/daily_rest-helpers_8a51e4a9.txt b/daily_rest-helpers_8a51e4a9.txt
new file mode 100644
index 0000000000000000000000000000000000000000..635438d3a21cd8145cf67487f7d668c2af3d38cd
--- /dev/null
+++ b/daily_rest-helpers_8a51e4a9.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/daily/rest-helpers#param-room-name
+Title: Daily REST Helper - Pipecat
+==================================================
+
+Daily REST Helper - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Service Utilities Daily REST Helper Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Daily REST Helper Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline Daily REST API Documentation For complete Daily REST API reference and additional details ​ Classes ​ DailyRoomSipParams Configuration for SIP (Session Initiation Protocol) parameters. ​ display_name string default: "sw-sip-dialin" Display name for the SIP endpoint ​ video boolean default: false Whether video is enabled for SIP ​ sip_mode string default: "dial-in" SIP connection mode ​ num_endpoints integer default: 1 Number of SIP endpoints Copy Ask AI from pipecat.transports.services.helpers.daily_rest import DailyRoomSipParams sip_params = DailyRoomSipParams( display_name = "conference-line" , video = True , num_endpoints = 2 ) ​ RecordingsBucketConfig Configuration for storing Daily recordings in a custom S3 bucket. ​ bucket_name string required Name of the S3 bucket for storing recordings ​ bucket_region string required AWS region where the S3 bucket is located ​ assume_role_arn string required ARN of the IAM role to assume for S3 access ​ allow_api_access boolean default: false Whether to allow API access to the recordings Copy Ask AI from pipecat.transports.services.helpers.daily_rest import RecordingsBucketConfig bucket_config = RecordingsBucketConfig( bucket_name = "my-recordings-bucket" , bucket_region = "us-west-2" , assume_role_arn = "arn:aws:iam::123456789012:role/DailyRecordingsRole" , allow_api_access = True ) ​ DailyRoomProperties Properties that configure a Daily room’s behavior and features. ​ exp float Room expiration time as Unix timestamp (e.g., time.time() + 300 for 5 minutes) ​ enable_chat boolean default: false Whether chat is enabled in the room ​ enable_prejoin_ui boolean default: false Whether the prejoin lobby UI is enabled ​ enable_emoji_reactions boolean default: false Whether emoji reactions are enabled ​ eject_at_room_exp boolean default: false Whether to eject participants when room expires ​ enable_dialout boolean Whether dial-out is enabled ​ enable_recording string Recording settings (“cloud”, “local”, or “raw-tracks”) ​ geo string Geographic region for room ​ max_participants number Maximum number of participants allowed in the room ​ recordings_bucket RecordingsBucketConfig Configuration for custom S3 bucket recordings ​ sip DailyRoomSipParams SIP configuration parameters ​ sip_uri dict SIP URI configuration (returned by Daily) ​ start_video_off boolean default: false Whether the camera video is turned off by default The class also includes a sip_endpoint property that returns the SIP endpoint URI if available. Copy Ask AI import time from pipecat.transports.services.helpers.daily_rest import ( DailyRoomProperties, DailyRoomSipParams, RecordingsBucketConfig, ) properties = DailyRoomProperties( exp = time.time() + 3600 , # 1 hour from now enable_chat = True , enable_emoji_reactions = True , enable_recording = "cloud" , geo = "us-west" , max_participants = 50 , sip = DailyRoomSipParams( display_name = "conference" ), recordings_bucket = RecordingsBucketConfig( bucket_name = "my-bucket" , bucket_region = "us-west-2" , assume_role_arn = "arn:aws:iam::123456789012:role/DailyRole" ) ) # Access SIP endpoint if available if properties.sip_endpoint: print ( f "SIP endpoint: { properties.sip_endpoint } " ) ​ DailyRoomParams Parameters for creating a new Daily room. ​ name string Room name (if not provided, one will be generated) ​ privacy string default: "public" Room privacy setting (“private” or “public”) ​ properties DailyRoomProperties Room configuration properties Copy Ask AI import time from pipecat.transports.services.helpers.daily_rest import ( DailyRoomParams, DailyRoomProperties, ) params = DailyRoomParams( name = "team-meeting" , privacy = "private" , properties = DailyRoomProperties( enable_chat = True , exp = time.time() + 7200 # 2 hours from now ) ) ​ DailyRoomObject Response object representing a Daily room. ​ id string Unique room identifier ​ name string Room name ​ api_created boolean Whether the room was created via API ​ privacy string Room privacy setting ​ url string Complete room URL ​ created_at string Room creation timestamp in ISO 8601 format ​ config DailyRoomProperties Room configuration Copy Ask AI from pipecat.transports.services.helpers.daily_rest import ( DailyRoomObject, DailyRoomProperties, ) # Example of what a DailyRoomObject looks like when received room = DailyRoomObject( id = "abc123" , name = "team-meeting" , api_created = True , privacy = "private" , url = "https://your-domain.daily.co/team-meeting" , created_at = "2024-01-20T10:00:00.000Z" , config = DailyRoomProperties( enable_chat = True , exp = 1705743600 ) ) ​ DailyMeetingTokenProperties Properties for configuring a Daily meeting token. ​ room_name string The room this token is valid for. If not set, token is valid for all rooms. ​ eject_at_token_exp boolean Whether to eject user when token expires ​ eject_after_elapsed integer Eject user after this many seconds ​ nbf integer “Not before” timestamp - users cannot join before this time ​ exp integer Expiration timestamp - users cannot join after this time ​ is_owner boolean Whether token grants owner privileges ​ user_name string User’s display name in the meeting ​ user_id string Unique identifier for the user (36 char limit) ​ enable_screenshare boolean Whether user can share their screen ​ start_video_off boolean Whether to join with video off ​ start_audio_off boolean Whether to join with audio off ​ enable_recording string Recording settings (“cloud”, “local”, or “raw-tracks”) ​ enable_prejoin_ui boolean Whether to show prejoin UI ​ start_cloud_recording boolean Whether to start cloud recording when user joins ​ permissions dict Initial default permissions for a non-meeting-owner participant ​ DailyMeetingTokenParams Parameters for creating a Daily meeting token. ​ properties DailyMeetingTokenProperties Token configuration properties Copy Ask AI from pipecat.transports.services.helpers.daily_rest import ( DailyMeetingTokenParams, DailyMeetingTokenProperties, ) token_params = DailyMeetingTokenParams( properties = DailyMeetingTokenProperties( user_name = "John Doe" , enable_screenshare = True , start_video_off = True , permissions = { "canSend" : [ "video" , "audio" ]} ) ) ​ Initialize DailyRESTHelper Create a new instance of the Daily REST helper. ​ daily_api_key string required Your Daily API key ​ daily_api_url string default: "https://api.daily.co/v1" The Daily API base URL ​ aiohttp_session aiohttp.ClientSession required An aiohttp client session for making HTTP requests Copy Ask AI helper = DailyRESTHelper( daily_api_key = "your-api-key" , aiohttp_session = session ) ​ Create Room Creates a new Daily room with specified parameters. ​ params DailyRoomParams required Room configuration parameters including name, privacy, and properties Copy Ask AI # Create a room that expires in 1 hour params = DailyRoomParams( name = "my-room" , privacy = "private" , properties = DailyRoomProperties( exp = time.time() + 3600 , enable_chat = True ) ) room = await helper.create_room(params) print ( f "Room URL: { room.url } " ) ​ Get Room From URL Retrieves room information using a Daily room URL. ​ room_url string required The complete Daily room URL Copy Ask AI room = await helper.get_room_from_url( "https://your-domain.daily.co/my-room" ) print ( f "Room name: { room.name } " ) ​ Get Token Generates a meeting token for a specific room. ​ room_url string required The complete Daily room URL ​ expiry_time float default: "3600" Token expiration time in seconds ​ eject_at_token_exp bool default: "False" Whether to eject user when token expires ​ owner bool default: "True" Whether the token should have owner privileges (overrides any setting in params) ​ params DailyMeetingTokenParams Additional token configuration. Note that room_name , exp , eject_at_token_exp , and is_owner will be set based on the other function parameters. Copy Ask AI # Basic token generation token = await helper.get_token( room_url = "https://your-domain.daily.co/my-room" , expiry_time = 1800 , # 30 minutes owner = True , eject_at_token_exp = True ) # Advanced token generation with additional properties token_params = DailyMeetingTokenParams( properties = DailyMeetingTokenProperties( user_name = "John Doe" , start_video_off = True ) ) token = await helper.get_token( room_url = "https://your-domain.daily.co/my-room" , expiry_time = 1800 , owner = False , eject_at_token_exp = True , params = token_params ) ​ Delete Room By URL Deletes a room using its URL. ​ room_url string required The complete Daily room URL Copy Ask AI success = await helper.delete_room_by_url( "https://your-domain.daily.co/my-room" ) if success: print ( "Room deleted successfully" ) ​ Delete Room By Name Deletes a room using its name. ​ room_name string required The name of the Daily room Copy Ask AI success = await helper.delete_room_by_name( "my-room" ) if success: print ( "Room deleted successfully" ) ​ Get Name From URL Extracts the room name from a Daily room URL. ​ room_url string required The complete Daily room URL Copy Ask AI room_name = helper.get_name_from_url( "https://your-domain.daily.co/my-room" ) print ( f "Room name: { room_name } " ) # Outputs: "my-room" Turn Tracking Observer Smart Turn Overview On this page Classes DailyRoomSipParams RecordingsBucketConfig DailyRoomProperties DailyRoomParams DailyRoomObject DailyMeetingTokenProperties DailyMeetingTokenParams Initialize DailyRESTHelper Create Room Get Room From URL Get Token Delete Room By URL Delete Room By Name Get Name From URL Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/daily_rest-helpers_92ae04c7.txt b/daily_rest-helpers_92ae04c7.txt
new file mode 100644
index 0000000000000000000000000000000000000000..d59adae6c7752ded0d169a52a46b2ded21845e0e
--- /dev/null
+++ b/daily_rest-helpers_92ae04c7.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/daily/rest-helpers#param-params
+Title: Daily REST Helper - Pipecat
+==================================================
+
+Daily REST Helper - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Service Utilities Daily REST Helper Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Daily REST Helper Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline Daily REST API Documentation For complete Daily REST API reference and additional details ​ Classes ​ DailyRoomSipParams Configuration for SIP (Session Initiation Protocol) parameters. ​ display_name string default: "sw-sip-dialin" Display name for the SIP endpoint ​ video boolean default: false Whether video is enabled for SIP ​ sip_mode string default: "dial-in" SIP connection mode ​ num_endpoints integer default: 1 Number of SIP endpoints Copy Ask AI from pipecat.transports.services.helpers.daily_rest import DailyRoomSipParams sip_params = DailyRoomSipParams( display_name = "conference-line" , video = True , num_endpoints = 2 ) ​ RecordingsBucketConfig Configuration for storing Daily recordings in a custom S3 bucket. ​ bucket_name string required Name of the S3 bucket for storing recordings ​ bucket_region string required AWS region where the S3 bucket is located ​ assume_role_arn string required ARN of the IAM role to assume for S3 access ​ allow_api_access boolean default: false Whether to allow API access to the recordings Copy Ask AI from pipecat.transports.services.helpers.daily_rest import RecordingsBucketConfig bucket_config = RecordingsBucketConfig( bucket_name = "my-recordings-bucket" , bucket_region = "us-west-2" , assume_role_arn = "arn:aws:iam::123456789012:role/DailyRecordingsRole" , allow_api_access = True ) ​ DailyRoomProperties Properties that configure a Daily room’s behavior and features. ​ exp float Room expiration time as Unix timestamp (e.g., time.time() + 300 for 5 minutes) ​ enable_chat boolean default: false Whether chat is enabled in the room ​ enable_prejoin_ui boolean default: false Whether the prejoin lobby UI is enabled ​ enable_emoji_reactions boolean default: false Whether emoji reactions are enabled ​ eject_at_room_exp boolean default: false Whether to eject participants when room expires ​ enable_dialout boolean Whether dial-out is enabled ​ enable_recording string Recording settings (“cloud”, “local”, or “raw-tracks”) ​ geo string Geographic region for room ​ max_participants number Maximum number of participants allowed in the room ​ recordings_bucket RecordingsBucketConfig Configuration for custom S3 bucket recordings ​ sip DailyRoomSipParams SIP configuration parameters ​ sip_uri dict SIP URI configuration (returned by Daily) ​ start_video_off boolean default: false Whether the camera video is turned off by default The class also includes a sip_endpoint property that returns the SIP endpoint URI if available. Copy Ask AI import time from pipecat.transports.services.helpers.daily_rest import ( DailyRoomProperties, DailyRoomSipParams, RecordingsBucketConfig, ) properties = DailyRoomProperties( exp = time.time() + 3600 , # 1 hour from now enable_chat = True , enable_emoji_reactions = True , enable_recording = "cloud" , geo = "us-west" , max_participants = 50 , sip = DailyRoomSipParams( display_name = "conference" ), recordings_bucket = RecordingsBucketConfig( bucket_name = "my-bucket" , bucket_region = "us-west-2" , assume_role_arn = "arn:aws:iam::123456789012:role/DailyRole" ) ) # Access SIP endpoint if available if properties.sip_endpoint: print ( f "SIP endpoint: { properties.sip_endpoint } " ) ​ DailyRoomParams Parameters for creating a new Daily room. ​ name string Room name (if not provided, one will be generated) ​ privacy string default: "public" Room privacy setting (“private” or “public”) ​ properties DailyRoomProperties Room configuration properties Copy Ask AI import time from pipecat.transports.services.helpers.daily_rest import ( DailyRoomParams, DailyRoomProperties, ) params = DailyRoomParams( name = "team-meeting" , privacy = "private" , properties = DailyRoomProperties( enable_chat = True , exp = time.time() + 7200 # 2 hours from now ) ) ​ DailyRoomObject Response object representing a Daily room. ​ id string Unique room identifier ​ name string Room name ​ api_created boolean Whether the room was created via API ​ privacy string Room privacy setting ​ url string Complete room URL ​ created_at string Room creation timestamp in ISO 8601 format ​ config DailyRoomProperties Room configuration Copy Ask AI from pipecat.transports.services.helpers.daily_rest import ( DailyRoomObject, DailyRoomProperties, ) # Example of what a DailyRoomObject looks like when received room = DailyRoomObject( id = "abc123" , name = "team-meeting" , api_created = True , privacy = "private" , url = "https://your-domain.daily.co/team-meeting" , created_at = "2024-01-20T10:00:00.000Z" , config = DailyRoomProperties( enable_chat = True , exp = 1705743600 ) ) ​ DailyMeetingTokenProperties Properties for configuring a Daily meeting token. ​ room_name string The room this token is valid for. If not set, token is valid for all rooms. ​ eject_at_token_exp boolean Whether to eject user when token expires ​ eject_after_elapsed integer Eject user after this many seconds ​ nbf integer “Not before” timestamp - users cannot join before this time ​ exp integer Expiration timestamp - users cannot join after this time ​ is_owner boolean Whether token grants owner privileges ​ user_name string User’s display name in the meeting ​ user_id string Unique identifier for the user (36 char limit) ​ enable_screenshare boolean Whether user can share their screen ​ start_video_off boolean Whether to join with video off ​ start_audio_off boolean Whether to join with audio off ​ enable_recording string Recording settings (“cloud”, “local”, or “raw-tracks”) ​ enable_prejoin_ui boolean Whether to show prejoin UI ​ start_cloud_recording boolean Whether to start cloud recording when user joins ​ permissions dict Initial default permissions for a non-meeting-owner participant ​ DailyMeetingTokenParams Parameters for creating a Daily meeting token. ​ properties DailyMeetingTokenProperties Token configuration properties Copy Ask AI from pipecat.transports.services.helpers.daily_rest import ( DailyMeetingTokenParams, DailyMeetingTokenProperties, ) token_params = DailyMeetingTokenParams( properties = DailyMeetingTokenProperties( user_name = "John Doe" , enable_screenshare = True , start_video_off = True , permissions = { "canSend" : [ "video" , "audio" ]} ) ) ​ Initialize DailyRESTHelper Create a new instance of the Daily REST helper. ​ daily_api_key string required Your Daily API key ​ daily_api_url string default: "https://api.daily.co/v1" The Daily API base URL ​ aiohttp_session aiohttp.ClientSession required An aiohttp client session for making HTTP requests Copy Ask AI helper = DailyRESTHelper( daily_api_key = "your-api-key" , aiohttp_session = session ) ​ Create Room Creates a new Daily room with specified parameters. ​ params DailyRoomParams required Room configuration parameters including name, privacy, and properties Copy Ask AI # Create a room that expires in 1 hour params = DailyRoomParams( name = "my-room" , privacy = "private" , properties = DailyRoomProperties( exp = time.time() + 3600 , enable_chat = True ) ) room = await helper.create_room(params) print ( f "Room URL: { room.url } " ) ​ Get Room From URL Retrieves room information using a Daily room URL. ​ room_url string required The complete Daily room URL Copy Ask AI room = await helper.get_room_from_url( "https://your-domain.daily.co/my-room" ) print ( f "Room name: { room.name } " ) ​ Get Token Generates a meeting token for a specific room. ​ room_url string required The complete Daily room URL ​ expiry_time float default: "3600" Token expiration time in seconds ​ eject_at_token_exp bool default: "False" Whether to eject user when token expires ​ owner bool default: "True" Whether the token should have owner privileges (overrides any setting in params) ​ params DailyMeetingTokenParams Additional token configuration. Note that room_name , exp , eject_at_token_exp , and is_owner will be set based on the other function parameters. Copy Ask AI # Basic token generation token = await helper.get_token( room_url = "https://your-domain.daily.co/my-room" , expiry_time = 1800 , # 30 minutes owner = True , eject_at_token_exp = True ) # Advanced token generation with additional properties token_params = DailyMeetingTokenParams( properties = DailyMeetingTokenProperties( user_name = "John Doe" , start_video_off = True ) ) token = await helper.get_token( room_url = "https://your-domain.daily.co/my-room" , expiry_time = 1800 , owner = False , eject_at_token_exp = True , params = token_params ) ​ Delete Room By URL Deletes a room using its URL. ​ room_url string required The complete Daily room URL Copy Ask AI success = await helper.delete_room_by_url( "https://your-domain.daily.co/my-room" ) if success: print ( "Room deleted successfully" ) ​ Delete Room By Name Deletes a room using its name. ​ room_name string required The name of the Daily room Copy Ask AI success = await helper.delete_room_by_name( "my-room" ) if success: print ( "Room deleted successfully" ) ​ Get Name From URL Extracts the room name from a Daily room URL. ​ room_url string required The complete Daily room URL Copy Ask AI room_name = helper.get_name_from_url( "https://your-domain.daily.co/my-room" ) print ( f "Room name: { room_name } " ) # Outputs: "my-room" Turn Tracking Observer Smart Turn Overview On this page Classes DailyRoomSipParams RecordingsBucketConfig DailyRoomProperties DailyRoomParams DailyRoomObject DailyMeetingTokenProperties DailyMeetingTokenParams Initialize DailyRESTHelper Create Room Get Room From URL Get Token Delete Room By URL Delete Room By Name Get Name From URL Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/daily_rest-helpers_9dc32dd0.txt b/daily_rest-helpers_9dc32dd0.txt
new file mode 100644
index 0000000000000000000000000000000000000000..7d424491474d25a4b5f9771e233844e6e9795bc0
--- /dev/null
+++ b/daily_rest-helpers_9dc32dd0.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/daily/rest-helpers#param-permissions
+Title: Daily REST Helper - Pipecat
+==================================================
+
+Daily REST Helper - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Service Utilities Daily REST Helper Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Daily REST Helper Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline Daily REST API Documentation For complete Daily REST API reference and additional details ​ Classes ​ DailyRoomSipParams Configuration for SIP (Session Initiation Protocol) parameters. ​ display_name string default: "sw-sip-dialin" Display name for the SIP endpoint ​ video boolean default: false Whether video is enabled for SIP ​ sip_mode string default: "dial-in" SIP connection mode ​ num_endpoints integer default: 1 Number of SIP endpoints Copy Ask AI from pipecat.transports.services.helpers.daily_rest import DailyRoomSipParams sip_params = DailyRoomSipParams( display_name = "conference-line" , video = True , num_endpoints = 2 ) ​ RecordingsBucketConfig Configuration for storing Daily recordings in a custom S3 bucket. ​ bucket_name string required Name of the S3 bucket for storing recordings ​ bucket_region string required AWS region where the S3 bucket is located ​ assume_role_arn string required ARN of the IAM role to assume for S3 access ​ allow_api_access boolean default: false Whether to allow API access to the recordings Copy Ask AI from pipecat.transports.services.helpers.daily_rest import RecordingsBucketConfig bucket_config = RecordingsBucketConfig( bucket_name = "my-recordings-bucket" , bucket_region = "us-west-2" , assume_role_arn = "arn:aws:iam::123456789012:role/DailyRecordingsRole" , allow_api_access = True ) ​ DailyRoomProperties Properties that configure a Daily room’s behavior and features. ​ exp float Room expiration time as Unix timestamp (e.g., time.time() + 300 for 5 minutes) ​ enable_chat boolean default: false Whether chat is enabled in the room ​ enable_prejoin_ui boolean default: false Whether the prejoin lobby UI is enabled ​ enable_emoji_reactions boolean default: false Whether emoji reactions are enabled ​ eject_at_room_exp boolean default: false Whether to eject participants when room expires ​ enable_dialout boolean Whether dial-out is enabled ​ enable_recording string Recording settings (“cloud”, “local”, or “raw-tracks”) ​ geo string Geographic region for room ​ max_participants number Maximum number of participants allowed in the room ​ recordings_bucket RecordingsBucketConfig Configuration for custom S3 bucket recordings ​ sip DailyRoomSipParams SIP configuration parameters ​ sip_uri dict SIP URI configuration (returned by Daily) ​ start_video_off boolean default: false Whether the camera video is turned off by default The class also includes a sip_endpoint property that returns the SIP endpoint URI if available. Copy Ask AI import time from pipecat.transports.services.helpers.daily_rest import ( DailyRoomProperties, DailyRoomSipParams, RecordingsBucketConfig, ) properties = DailyRoomProperties( exp = time.time() + 3600 , # 1 hour from now enable_chat = True , enable_emoji_reactions = True , enable_recording = "cloud" , geo = "us-west" , max_participants = 50 , sip = DailyRoomSipParams( display_name = "conference" ), recordings_bucket = RecordingsBucketConfig( bucket_name = "my-bucket" , bucket_region = "us-west-2" , assume_role_arn = "arn:aws:iam::123456789012:role/DailyRole" ) ) # Access SIP endpoint if available if properties.sip_endpoint: print ( f "SIP endpoint: { properties.sip_endpoint } " ) ​ DailyRoomParams Parameters for creating a new Daily room. ​ name string Room name (if not provided, one will be generated) ​ privacy string default: "public" Room privacy setting (“private” or “public”) ​ properties DailyRoomProperties Room configuration properties Copy Ask AI import time from pipecat.transports.services.helpers.daily_rest import ( DailyRoomParams, DailyRoomProperties, ) params = DailyRoomParams( name = "team-meeting" , privacy = "private" , properties = DailyRoomProperties( enable_chat = True , exp = time.time() + 7200 # 2 hours from now ) ) ​ DailyRoomObject Response object representing a Daily room. ​ id string Unique room identifier ​ name string Room name ​ api_created boolean Whether the room was created via API ​ privacy string Room privacy setting ​ url string Complete room URL ​ created_at string Room creation timestamp in ISO 8601 format ​ config DailyRoomProperties Room configuration Copy Ask AI from pipecat.transports.services.helpers.daily_rest import ( DailyRoomObject, DailyRoomProperties, ) # Example of what a DailyRoomObject looks like when received room = DailyRoomObject( id = "abc123" , name = "team-meeting" , api_created = True , privacy = "private" , url = "https://your-domain.daily.co/team-meeting" , created_at = "2024-01-20T10:00:00.000Z" , config = DailyRoomProperties( enable_chat = True , exp = 1705743600 ) ) ​ DailyMeetingTokenProperties Properties for configuring a Daily meeting token. ​ room_name string The room this token is valid for. If not set, token is valid for all rooms. ​ eject_at_token_exp boolean Whether to eject user when token expires ​ eject_after_elapsed integer Eject user after this many seconds ​ nbf integer “Not before” timestamp - users cannot join before this time ​ exp integer Expiration timestamp - users cannot join after this time ​ is_owner boolean Whether token grants owner privileges ​ user_name string User’s display name in the meeting ​ user_id string Unique identifier for the user (36 char limit) ​ enable_screenshare boolean Whether user can share their screen ​ start_video_off boolean Whether to join with video off ​ start_audio_off boolean Whether to join with audio off ​ enable_recording string Recording settings (“cloud”, “local”, or “raw-tracks”) ​ enable_prejoin_ui boolean Whether to show prejoin UI ​ start_cloud_recording boolean Whether to start cloud recording when user joins ​ permissions dict Initial default permissions for a non-meeting-owner participant ​ DailyMeetingTokenParams Parameters for creating a Daily meeting token. ​ properties DailyMeetingTokenProperties Token configuration properties Copy Ask AI from pipecat.transports.services.helpers.daily_rest import ( DailyMeetingTokenParams, DailyMeetingTokenProperties, ) token_params = DailyMeetingTokenParams( properties = DailyMeetingTokenProperties( user_name = "John Doe" , enable_screenshare = True , start_video_off = True , permissions = { "canSend" : [ "video" , "audio" ]} ) ) ​ Initialize DailyRESTHelper Create a new instance of the Daily REST helper. ​ daily_api_key string required Your Daily API key ​ daily_api_url string default: "https://api.daily.co/v1" The Daily API base URL ​ aiohttp_session aiohttp.ClientSession required An aiohttp client session for making HTTP requests Copy Ask AI helper = DailyRESTHelper( daily_api_key = "your-api-key" , aiohttp_session = session ) ​ Create Room Creates a new Daily room with specified parameters. ​ params DailyRoomParams required Room configuration parameters including name, privacy, and properties Copy Ask AI # Create a room that expires in 1 hour params = DailyRoomParams( name = "my-room" , privacy = "private" , properties = DailyRoomProperties( exp = time.time() + 3600 , enable_chat = True ) ) room = await helper.create_room(params) print ( f "Room URL: { room.url } " ) ​ Get Room From URL Retrieves room information using a Daily room URL. ​ room_url string required The complete Daily room URL Copy Ask AI room = await helper.get_room_from_url( "https://your-domain.daily.co/my-room" ) print ( f "Room name: { room.name } " ) ​ Get Token Generates a meeting token for a specific room. ​ room_url string required The complete Daily room URL ​ expiry_time float default: "3600" Token expiration time in seconds ​ eject_at_token_exp bool default: "False" Whether to eject user when token expires ​ owner bool default: "True" Whether the token should have owner privileges (overrides any setting in params) ​ params DailyMeetingTokenParams Additional token configuration. Note that room_name , exp , eject_at_token_exp , and is_owner will be set based on the other function parameters. Copy Ask AI # Basic token generation token = await helper.get_token( room_url = "https://your-domain.daily.co/my-room" , expiry_time = 1800 , # 30 minutes owner = True , eject_at_token_exp = True ) # Advanced token generation with additional properties token_params = DailyMeetingTokenParams( properties = DailyMeetingTokenProperties( user_name = "John Doe" , start_video_off = True ) ) token = await helper.get_token( room_url = "https://your-domain.daily.co/my-room" , expiry_time = 1800 , owner = False , eject_at_token_exp = True , params = token_params ) ​ Delete Room By URL Deletes a room using its URL. ​ room_url string required The complete Daily room URL Copy Ask AI success = await helper.delete_room_by_url( "https://your-domain.daily.co/my-room" ) if success: print ( "Room deleted successfully" ) ​ Delete Room By Name Deletes a room using its name. ​ room_name string required The name of the Daily room Copy Ask AI success = await helper.delete_room_by_name( "my-room" ) if success: print ( "Room deleted successfully" ) ​ Get Name From URL Extracts the room name from a Daily room URL. ​ room_url string required The complete Daily room URL Copy Ask AI room_name = helper.get_name_from_url( "https://your-domain.daily.co/my-room" ) print ( f "Room name: { room_name } " ) # Outputs: "my-room" Turn Tracking Observer Smart Turn Overview On this page Classes DailyRoomSipParams RecordingsBucketConfig DailyRoomProperties DailyRoomParams DailyRoomObject DailyMeetingTokenProperties DailyMeetingTokenParams Initialize DailyRESTHelper Create Room Get Room From URL Get Token Delete Room By URL Delete Room By Name Get Name From URL Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/daily_rest-helpers_a1065cc5.txt b/daily_rest-helpers_a1065cc5.txt
new file mode 100644
index 0000000000000000000000000000000000000000..0e5d0ce9bee27f2a9a71ae1ddb2b8084816b5415
--- /dev/null
+++ b/daily_rest-helpers_a1065cc5.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/daily/rest-helpers#param-is-owner
+Title: Daily REST Helper - Pipecat
+==================================================
+
+Daily REST Helper - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Service Utilities Daily REST Helper Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Daily REST Helper Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline Daily REST API Documentation For complete Daily REST API reference and additional details ​ Classes ​ DailyRoomSipParams Configuration for SIP (Session Initiation Protocol) parameters. ​ display_name string default: "sw-sip-dialin" Display name for the SIP endpoint ​ video boolean default: false Whether video is enabled for SIP ​ sip_mode string default: "dial-in" SIP connection mode ​ num_endpoints integer default: 1 Number of SIP endpoints Copy Ask AI from pipecat.transports.services.helpers.daily_rest import DailyRoomSipParams sip_params = DailyRoomSipParams( display_name = "conference-line" , video = True , num_endpoints = 2 ) ​ RecordingsBucketConfig Configuration for storing Daily recordings in a custom S3 bucket. ​ bucket_name string required Name of the S3 bucket for storing recordings ​ bucket_region string required AWS region where the S3 bucket is located ​ assume_role_arn string required ARN of the IAM role to assume for S3 access ​ allow_api_access boolean default: false Whether to allow API access to the recordings Copy Ask AI from pipecat.transports.services.helpers.daily_rest import RecordingsBucketConfig bucket_config = RecordingsBucketConfig( bucket_name = "my-recordings-bucket" , bucket_region = "us-west-2" , assume_role_arn = "arn:aws:iam::123456789012:role/DailyRecordingsRole" , allow_api_access = True ) ​ DailyRoomProperties Properties that configure a Daily room’s behavior and features. ​ exp float Room expiration time as Unix timestamp (e.g., time.time() + 300 for 5 minutes) ​ enable_chat boolean default: false Whether chat is enabled in the room ​ enable_prejoin_ui boolean default: false Whether the prejoin lobby UI is enabled ​ enable_emoji_reactions boolean default: false Whether emoji reactions are enabled ​ eject_at_room_exp boolean default: false Whether to eject participants when room expires ​ enable_dialout boolean Whether dial-out is enabled ​ enable_recording string Recording settings (“cloud”, “local”, or “raw-tracks”) ​ geo string Geographic region for room ​ max_participants number Maximum number of participants allowed in the room ​ recordings_bucket RecordingsBucketConfig Configuration for custom S3 bucket recordings ​ sip DailyRoomSipParams SIP configuration parameters ​ sip_uri dict SIP URI configuration (returned by Daily) ​ start_video_off boolean default: false Whether the camera video is turned off by default The class also includes a sip_endpoint property that returns the SIP endpoint URI if available. Copy Ask AI import time from pipecat.transports.services.helpers.daily_rest import ( DailyRoomProperties, DailyRoomSipParams, RecordingsBucketConfig, ) properties = DailyRoomProperties( exp = time.time() + 3600 , # 1 hour from now enable_chat = True , enable_emoji_reactions = True , enable_recording = "cloud" , geo = "us-west" , max_participants = 50 , sip = DailyRoomSipParams( display_name = "conference" ), recordings_bucket = RecordingsBucketConfig( bucket_name = "my-bucket" , bucket_region = "us-west-2" , assume_role_arn = "arn:aws:iam::123456789012:role/DailyRole" ) ) # Access SIP endpoint if available if properties.sip_endpoint: print ( f "SIP endpoint: { properties.sip_endpoint } " ) ​ DailyRoomParams Parameters for creating a new Daily room. ​ name string Room name (if not provided, one will be generated) ​ privacy string default: "public" Room privacy setting (“private” or “public”) ​ properties DailyRoomProperties Room configuration properties Copy Ask AI import time from pipecat.transports.services.helpers.daily_rest import ( DailyRoomParams, DailyRoomProperties, ) params = DailyRoomParams( name = "team-meeting" , privacy = "private" , properties = DailyRoomProperties( enable_chat = True , exp = time.time() + 7200 # 2 hours from now ) ) ​ DailyRoomObject Response object representing a Daily room. ​ id string Unique room identifier ​ name string Room name ​ api_created boolean Whether the room was created via API ​ privacy string Room privacy setting ​ url string Complete room URL ​ created_at string Room creation timestamp in ISO 8601 format ​ config DailyRoomProperties Room configuration Copy Ask AI from pipecat.transports.services.helpers.daily_rest import ( DailyRoomObject, DailyRoomProperties, ) # Example of what a DailyRoomObject looks like when received room = DailyRoomObject( id = "abc123" , name = "team-meeting" , api_created = True , privacy = "private" , url = "https://your-domain.daily.co/team-meeting" , created_at = "2024-01-20T10:00:00.000Z" , config = DailyRoomProperties( enable_chat = True , exp = 1705743600 ) ) ​ DailyMeetingTokenProperties Properties for configuring a Daily meeting token. ​ room_name string The room this token is valid for. If not set, token is valid for all rooms. ​ eject_at_token_exp boolean Whether to eject user when token expires ​ eject_after_elapsed integer Eject user after this many seconds ​ nbf integer “Not before” timestamp - users cannot join before this time ​ exp integer Expiration timestamp - users cannot join after this time ​ is_owner boolean Whether token grants owner privileges ​ user_name string User’s display name in the meeting ​ user_id string Unique identifier for the user (36 char limit) ​ enable_screenshare boolean Whether user can share their screen ​ start_video_off boolean Whether to join with video off ​ start_audio_off boolean Whether to join with audio off ​ enable_recording string Recording settings (“cloud”, “local”, or “raw-tracks”) ​ enable_prejoin_ui boolean Whether to show prejoin UI ​ start_cloud_recording boolean Whether to start cloud recording when user joins ​ permissions dict Initial default permissions for a non-meeting-owner participant ​ DailyMeetingTokenParams Parameters for creating a Daily meeting token. ​ properties DailyMeetingTokenProperties Token configuration properties Copy Ask AI from pipecat.transports.services.helpers.daily_rest import ( DailyMeetingTokenParams, DailyMeetingTokenProperties, ) token_params = DailyMeetingTokenParams( properties = DailyMeetingTokenProperties( user_name = "John Doe" , enable_screenshare = True , start_video_off = True , permissions = { "canSend" : [ "video" , "audio" ]} ) ) ​ Initialize DailyRESTHelper Create a new instance of the Daily REST helper. ​ daily_api_key string required Your Daily API key ​ daily_api_url string default: "https://api.daily.co/v1" The Daily API base URL ​ aiohttp_session aiohttp.ClientSession required An aiohttp client session for making HTTP requests Copy Ask AI helper = DailyRESTHelper( daily_api_key = "your-api-key" , aiohttp_session = session ) ​ Create Room Creates a new Daily room with specified parameters. ​ params DailyRoomParams required Room configuration parameters including name, privacy, and properties Copy Ask AI # Create a room that expires in 1 hour params = DailyRoomParams( name = "my-room" , privacy = "private" , properties = DailyRoomProperties( exp = time.time() + 3600 , enable_chat = True ) ) room = await helper.create_room(params) print ( f "Room URL: { room.url } " ) ​ Get Room From URL Retrieves room information using a Daily room URL. ​ room_url string required The complete Daily room URL Copy Ask AI room = await helper.get_room_from_url( "https://your-domain.daily.co/my-room" ) print ( f "Room name: { room.name } " ) ​ Get Token Generates a meeting token for a specific room. ​ room_url string required The complete Daily room URL ​ expiry_time float default: "3600" Token expiration time in seconds ​ eject_at_token_exp bool default: "False" Whether to eject user when token expires ​ owner bool default: "True" Whether the token should have owner privileges (overrides any setting in params) ​ params DailyMeetingTokenParams Additional token configuration. Note that room_name , exp , eject_at_token_exp , and is_owner will be set based on the other function parameters. Copy Ask AI # Basic token generation token = await helper.get_token( room_url = "https://your-domain.daily.co/my-room" , expiry_time = 1800 , # 30 minutes owner = True , eject_at_token_exp = True ) # Advanced token generation with additional properties token_params = DailyMeetingTokenParams( properties = DailyMeetingTokenProperties( user_name = "John Doe" , start_video_off = True ) ) token = await helper.get_token( room_url = "https://your-domain.daily.co/my-room" , expiry_time = 1800 , owner = False , eject_at_token_exp = True , params = token_params ) ​ Delete Room By URL Deletes a room using its URL. ​ room_url string required The complete Daily room URL Copy Ask AI success = await helper.delete_room_by_url( "https://your-domain.daily.co/my-room" ) if success: print ( "Room deleted successfully" ) ​ Delete Room By Name Deletes a room using its name. ​ room_name string required The name of the Daily room Copy Ask AI success = await helper.delete_room_by_name( "my-room" ) if success: print ( "Room deleted successfully" ) ​ Get Name From URL Extracts the room name from a Daily room URL. ​ room_url string required The complete Daily room URL Copy Ask AI room_name = helper.get_name_from_url( "https://your-domain.daily.co/my-room" ) print ( f "Room name: { room_name } " ) # Outputs: "my-room" Turn Tracking Observer Smart Turn Overview On this page Classes DailyRoomSipParams RecordingsBucketConfig DailyRoomProperties DailyRoomParams DailyRoomObject DailyMeetingTokenProperties DailyMeetingTokenParams Initialize DailyRESTHelper Create Room Get Room From URL Get Token Delete Room By URL Delete Room By Name Get Name From URL Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/daily_rest-helpers_a7beaed6.txt b/daily_rest-helpers_a7beaed6.txt
new file mode 100644
index 0000000000000000000000000000000000000000..c0d00e525fae26cdb382d298e692020939a00a90
--- /dev/null
+++ b/daily_rest-helpers_a7beaed6.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/daily/rest-helpers#param-exp-1
+Title: Daily REST Helper - Pipecat
+==================================================
+
+Daily REST Helper - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Service Utilities Daily REST Helper Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Daily REST Helper Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline Daily REST API Documentation For complete Daily REST API reference and additional details ​ Classes ​ DailyRoomSipParams Configuration for SIP (Session Initiation Protocol) parameters. ​ display_name string default: "sw-sip-dialin" Display name for the SIP endpoint ​ video boolean default: false Whether video is enabled for SIP ​ sip_mode string default: "dial-in" SIP connection mode ​ num_endpoints integer default: 1 Number of SIP endpoints Copy Ask AI from pipecat.transports.services.helpers.daily_rest import DailyRoomSipParams sip_params = DailyRoomSipParams( display_name = "conference-line" , video = True , num_endpoints = 2 ) ​ RecordingsBucketConfig Configuration for storing Daily recordings in a custom S3 bucket. ​ bucket_name string required Name of the S3 bucket for storing recordings ​ bucket_region string required AWS region where the S3 bucket is located ​ assume_role_arn string required ARN of the IAM role to assume for S3 access ​ allow_api_access boolean default: false Whether to allow API access to the recordings Copy Ask AI from pipecat.transports.services.helpers.daily_rest import RecordingsBucketConfig bucket_config = RecordingsBucketConfig( bucket_name = "my-recordings-bucket" , bucket_region = "us-west-2" , assume_role_arn = "arn:aws:iam::123456789012:role/DailyRecordingsRole" , allow_api_access = True ) ​ DailyRoomProperties Properties that configure a Daily room’s behavior and features. ​ exp float Room expiration time as Unix timestamp (e.g., time.time() + 300 for 5 minutes) ​ enable_chat boolean default: false Whether chat is enabled in the room ​ enable_prejoin_ui boolean default: false Whether the prejoin lobby UI is enabled ​ enable_emoji_reactions boolean default: false Whether emoji reactions are enabled ​ eject_at_room_exp boolean default: false Whether to eject participants when room expires ​ enable_dialout boolean Whether dial-out is enabled ​ enable_recording string Recording settings (“cloud”, “local”, or “raw-tracks”) ​ geo string Geographic region for room ​ max_participants number Maximum number of participants allowed in the room ​ recordings_bucket RecordingsBucketConfig Configuration for custom S3 bucket recordings ​ sip DailyRoomSipParams SIP configuration parameters ​ sip_uri dict SIP URI configuration (returned by Daily) ​ start_video_off boolean default: false Whether the camera video is turned off by default The class also includes a sip_endpoint property that returns the SIP endpoint URI if available. Copy Ask AI import time from pipecat.transports.services.helpers.daily_rest import ( DailyRoomProperties, DailyRoomSipParams, RecordingsBucketConfig, ) properties = DailyRoomProperties( exp = time.time() + 3600 , # 1 hour from now enable_chat = True , enable_emoji_reactions = True , enable_recording = "cloud" , geo = "us-west" , max_participants = 50 , sip = DailyRoomSipParams( display_name = "conference" ), recordings_bucket = RecordingsBucketConfig( bucket_name = "my-bucket" , bucket_region = "us-west-2" , assume_role_arn = "arn:aws:iam::123456789012:role/DailyRole" ) ) # Access SIP endpoint if available if properties.sip_endpoint: print ( f "SIP endpoint: { properties.sip_endpoint } " ) ​ DailyRoomParams Parameters for creating a new Daily room. ​ name string Room name (if not provided, one will be generated) ​ privacy string default: "public" Room privacy setting (“private” or “public”) ​ properties DailyRoomProperties Room configuration properties Copy Ask AI import time from pipecat.transports.services.helpers.daily_rest import ( DailyRoomParams, DailyRoomProperties, ) params = DailyRoomParams( name = "team-meeting" , privacy = "private" , properties = DailyRoomProperties( enable_chat = True , exp = time.time() + 7200 # 2 hours from now ) ) ​ DailyRoomObject Response object representing a Daily room. ​ id string Unique room identifier ​ name string Room name ​ api_created boolean Whether the room was created via API ​ privacy string Room privacy setting ​ url string Complete room URL ​ created_at string Room creation timestamp in ISO 8601 format ​ config DailyRoomProperties Room configuration Copy Ask AI from pipecat.transports.services.helpers.daily_rest import ( DailyRoomObject, DailyRoomProperties, ) # Example of what a DailyRoomObject looks like when received room = DailyRoomObject( id = "abc123" , name = "team-meeting" , api_created = True , privacy = "private" , url = "https://your-domain.daily.co/team-meeting" , created_at = "2024-01-20T10:00:00.000Z" , config = DailyRoomProperties( enable_chat = True , exp = 1705743600 ) ) ​ DailyMeetingTokenProperties Properties for configuring a Daily meeting token. ​ room_name string The room this token is valid for. If not set, token is valid for all rooms. ​ eject_at_token_exp boolean Whether to eject user when token expires ​ eject_after_elapsed integer Eject user after this many seconds ​ nbf integer “Not before” timestamp - users cannot join before this time ​ exp integer Expiration timestamp - users cannot join after this time ​ is_owner boolean Whether token grants owner privileges ​ user_name string User’s display name in the meeting ​ user_id string Unique identifier for the user (36 char limit) ​ enable_screenshare boolean Whether user can share their screen ​ start_video_off boolean Whether to join with video off ​ start_audio_off boolean Whether to join with audio off ​ enable_recording string Recording settings (“cloud”, “local”, or “raw-tracks”) ​ enable_prejoin_ui boolean Whether to show prejoin UI ​ start_cloud_recording boolean Whether to start cloud recording when user joins ​ permissions dict Initial default permissions for a non-meeting-owner participant ​ DailyMeetingTokenParams Parameters for creating a Daily meeting token. ​ properties DailyMeetingTokenProperties Token configuration properties Copy Ask AI from pipecat.transports.services.helpers.daily_rest import ( DailyMeetingTokenParams, DailyMeetingTokenProperties, ) token_params = DailyMeetingTokenParams( properties = DailyMeetingTokenProperties( user_name = "John Doe" , enable_screenshare = True , start_video_off = True , permissions = { "canSend" : [ "video" , "audio" ]} ) ) ​ Initialize DailyRESTHelper Create a new instance of the Daily REST helper. ​ daily_api_key string required Your Daily API key ​ daily_api_url string default: "https://api.daily.co/v1" The Daily API base URL ​ aiohttp_session aiohttp.ClientSession required An aiohttp client session for making HTTP requests Copy Ask AI helper = DailyRESTHelper( daily_api_key = "your-api-key" , aiohttp_session = session ) ​ Create Room Creates a new Daily room with specified parameters. ​ params DailyRoomParams required Room configuration parameters including name, privacy, and properties Copy Ask AI # Create a room that expires in 1 hour params = DailyRoomParams( name = "my-room" , privacy = "private" , properties = DailyRoomProperties( exp = time.time() + 3600 , enable_chat = True ) ) room = await helper.create_room(params) print ( f "Room URL: { room.url } " ) ​ Get Room From URL Retrieves room information using a Daily room URL. ​ room_url string required The complete Daily room URL Copy Ask AI room = await helper.get_room_from_url( "https://your-domain.daily.co/my-room" ) print ( f "Room name: { room.name } " ) ​ Get Token Generates a meeting token for a specific room. ​ room_url string required The complete Daily room URL ​ expiry_time float default: "3600" Token expiration time in seconds ​ eject_at_token_exp bool default: "False" Whether to eject user when token expires ​ owner bool default: "True" Whether the token should have owner privileges (overrides any setting in params) ​ params DailyMeetingTokenParams Additional token configuration. Note that room_name , exp , eject_at_token_exp , and is_owner will be set based on the other function parameters. Copy Ask AI # Basic token generation token = await helper.get_token( room_url = "https://your-domain.daily.co/my-room" , expiry_time = 1800 , # 30 minutes owner = True , eject_at_token_exp = True ) # Advanced token generation with additional properties token_params = DailyMeetingTokenParams( properties = DailyMeetingTokenProperties( user_name = "John Doe" , start_video_off = True ) ) token = await helper.get_token( room_url = "https://your-domain.daily.co/my-room" , expiry_time = 1800 , owner = False , eject_at_token_exp = True , params = token_params ) ​ Delete Room By URL Deletes a room using its URL. ​ room_url string required The complete Daily room URL Copy Ask AI success = await helper.delete_room_by_url( "https://your-domain.daily.co/my-room" ) if success: print ( "Room deleted successfully" ) ​ Delete Room By Name Deletes a room using its name. ​ room_name string required The name of the Daily room Copy Ask AI success = await helper.delete_room_by_name( "my-room" ) if success: print ( "Room deleted successfully" ) ​ Get Name From URL Extracts the room name from a Daily room URL. ​ room_url string required The complete Daily room URL Copy Ask AI room_name = helper.get_name_from_url( "https://your-domain.daily.co/my-room" ) print ( f "Room name: { room_name } " ) # Outputs: "my-room" Turn Tracking Observer Smart Turn Overview On this page Classes DailyRoomSipParams RecordingsBucketConfig DailyRoomProperties DailyRoomParams DailyRoomObject DailyMeetingTokenProperties DailyMeetingTokenParams Initialize DailyRESTHelper Create Room Get Room From URL Get Token Delete Room By URL Delete Room By Name Get Name From URL Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/daily_rest-helpers_bbc6ec14.txt b/daily_rest-helpers_bbc6ec14.txt
new file mode 100644
index 0000000000000000000000000000000000000000..67003937ce10e06bed8eb776b60ad16f0092dad3
--- /dev/null
+++ b/daily_rest-helpers_bbc6ec14.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/daily/rest-helpers#param-allow-api-access
+Title: Daily REST Helper - Pipecat
+==================================================
+
+Daily REST Helper - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Service Utilities Daily REST Helper Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Daily REST Helper Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline Daily REST API Documentation For complete Daily REST API reference and additional details ​ Classes ​ DailyRoomSipParams Configuration for SIP (Session Initiation Protocol) parameters. ​ display_name string default: "sw-sip-dialin" Display name for the SIP endpoint ​ video boolean default: false Whether video is enabled for SIP ​ sip_mode string default: "dial-in" SIP connection mode ​ num_endpoints integer default: 1 Number of SIP endpoints Copy Ask AI from pipecat.transports.services.helpers.daily_rest import DailyRoomSipParams sip_params = DailyRoomSipParams( display_name = "conference-line" , video = True , num_endpoints = 2 ) ​ RecordingsBucketConfig Configuration for storing Daily recordings in a custom S3 bucket. ​ bucket_name string required Name of the S3 bucket for storing recordings ​ bucket_region string required AWS region where the S3 bucket is located ​ assume_role_arn string required ARN of the IAM role to assume for S3 access ​ allow_api_access boolean default: false Whether to allow API access to the recordings Copy Ask AI from pipecat.transports.services.helpers.daily_rest import RecordingsBucketConfig bucket_config = RecordingsBucketConfig( bucket_name = "my-recordings-bucket" , bucket_region = "us-west-2" , assume_role_arn = "arn:aws:iam::123456789012:role/DailyRecordingsRole" , allow_api_access = True ) ​ DailyRoomProperties Properties that configure a Daily room’s behavior and features. ​ exp float Room expiration time as Unix timestamp (e.g., time.time() + 300 for 5 minutes) ​ enable_chat boolean default: false Whether chat is enabled in the room ​ enable_prejoin_ui boolean default: false Whether the prejoin lobby UI is enabled ​ enable_emoji_reactions boolean default: false Whether emoji reactions are enabled ​ eject_at_room_exp boolean default: false Whether to eject participants when room expires ​ enable_dialout boolean Whether dial-out is enabled ​ enable_recording string Recording settings (“cloud”, “local”, or “raw-tracks”) ​ geo string Geographic region for room ​ max_participants number Maximum number of participants allowed in the room ​ recordings_bucket RecordingsBucketConfig Configuration for custom S3 bucket recordings ​ sip DailyRoomSipParams SIP configuration parameters ​ sip_uri dict SIP URI configuration (returned by Daily) ​ start_video_off boolean default: false Whether the camera video is turned off by default The class also includes a sip_endpoint property that returns the SIP endpoint URI if available. Copy Ask AI import time from pipecat.transports.services.helpers.daily_rest import ( DailyRoomProperties, DailyRoomSipParams, RecordingsBucketConfig, ) properties = DailyRoomProperties( exp = time.time() + 3600 , # 1 hour from now enable_chat = True , enable_emoji_reactions = True , enable_recording = "cloud" , geo = "us-west" , max_participants = 50 , sip = DailyRoomSipParams( display_name = "conference" ), recordings_bucket = RecordingsBucketConfig( bucket_name = "my-bucket" , bucket_region = "us-west-2" , assume_role_arn = "arn:aws:iam::123456789012:role/DailyRole" ) ) # Access SIP endpoint if available if properties.sip_endpoint: print ( f "SIP endpoint: { properties.sip_endpoint } " ) ​ DailyRoomParams Parameters for creating a new Daily room. ​ name string Room name (if not provided, one will be generated) ​ privacy string default: "public" Room privacy setting (“private” or “public”) ​ properties DailyRoomProperties Room configuration properties Copy Ask AI import time from pipecat.transports.services.helpers.daily_rest import ( DailyRoomParams, DailyRoomProperties, ) params = DailyRoomParams( name = "team-meeting" , privacy = "private" , properties = DailyRoomProperties( enable_chat = True , exp = time.time() + 7200 # 2 hours from now ) ) ​ DailyRoomObject Response object representing a Daily room. ​ id string Unique room identifier ​ name string Room name ​ api_created boolean Whether the room was created via API ​ privacy string Room privacy setting ​ url string Complete room URL ​ created_at string Room creation timestamp in ISO 8601 format ​ config DailyRoomProperties Room configuration Copy Ask AI from pipecat.transports.services.helpers.daily_rest import ( DailyRoomObject, DailyRoomProperties, ) # Example of what a DailyRoomObject looks like when received room = DailyRoomObject( id = "abc123" , name = "team-meeting" , api_created = True , privacy = "private" , url = "https://your-domain.daily.co/team-meeting" , created_at = "2024-01-20T10:00:00.000Z" , config = DailyRoomProperties( enable_chat = True , exp = 1705743600 ) ) ​ DailyMeetingTokenProperties Properties for configuring a Daily meeting token. ​ room_name string The room this token is valid for. If not set, token is valid for all rooms. ​ eject_at_token_exp boolean Whether to eject user when token expires ​ eject_after_elapsed integer Eject user after this many seconds ​ nbf integer “Not before” timestamp - users cannot join before this time ​ exp integer Expiration timestamp - users cannot join after this time ​ is_owner boolean Whether token grants owner privileges ​ user_name string User’s display name in the meeting ​ user_id string Unique identifier for the user (36 char limit) ​ enable_screenshare boolean Whether user can share their screen ​ start_video_off boolean Whether to join with video off ​ start_audio_off boolean Whether to join with audio off ​ enable_recording string Recording settings (“cloud”, “local”, or “raw-tracks”) ​ enable_prejoin_ui boolean Whether to show prejoin UI ​ start_cloud_recording boolean Whether to start cloud recording when user joins ​ permissions dict Initial default permissions for a non-meeting-owner participant ​ DailyMeetingTokenParams Parameters for creating a Daily meeting token. ​ properties DailyMeetingTokenProperties Token configuration properties Copy Ask AI from pipecat.transports.services.helpers.daily_rest import ( DailyMeetingTokenParams, DailyMeetingTokenProperties, ) token_params = DailyMeetingTokenParams( properties = DailyMeetingTokenProperties( user_name = "John Doe" , enable_screenshare = True , start_video_off = True , permissions = { "canSend" : [ "video" , "audio" ]} ) ) ​ Initialize DailyRESTHelper Create a new instance of the Daily REST helper. ​ daily_api_key string required Your Daily API key ​ daily_api_url string default: "https://api.daily.co/v1" The Daily API base URL ​ aiohttp_session aiohttp.ClientSession required An aiohttp client session for making HTTP requests Copy Ask AI helper = DailyRESTHelper( daily_api_key = "your-api-key" , aiohttp_session = session ) ​ Create Room Creates a new Daily room with specified parameters. ​ params DailyRoomParams required Room configuration parameters including name, privacy, and properties Copy Ask AI # Create a room that expires in 1 hour params = DailyRoomParams( name = "my-room" , privacy = "private" , properties = DailyRoomProperties( exp = time.time() + 3600 , enable_chat = True ) ) room = await helper.create_room(params) print ( f "Room URL: { room.url } " ) ​ Get Room From URL Retrieves room information using a Daily room URL. ​ room_url string required The complete Daily room URL Copy Ask AI room = await helper.get_room_from_url( "https://your-domain.daily.co/my-room" ) print ( f "Room name: { room.name } " ) ​ Get Token Generates a meeting token for a specific room. ​ room_url string required The complete Daily room URL ​ expiry_time float default: "3600" Token expiration time in seconds ​ eject_at_token_exp bool default: "False" Whether to eject user when token expires ​ owner bool default: "True" Whether the token should have owner privileges (overrides any setting in params) ​ params DailyMeetingTokenParams Additional token configuration. Note that room_name , exp , eject_at_token_exp , and is_owner will be set based on the other function parameters. Copy Ask AI # Basic token generation token = await helper.get_token( room_url = "https://your-domain.daily.co/my-room" , expiry_time = 1800 , # 30 minutes owner = True , eject_at_token_exp = True ) # Advanced token generation with additional properties token_params = DailyMeetingTokenParams( properties = DailyMeetingTokenProperties( user_name = "John Doe" , start_video_off = True ) ) token = await helper.get_token( room_url = "https://your-domain.daily.co/my-room" , expiry_time = 1800 , owner = False , eject_at_token_exp = True , params = token_params ) ​ Delete Room By URL Deletes a room using its URL. ​ room_url string required The complete Daily room URL Copy Ask AI success = await helper.delete_room_by_url( "https://your-domain.daily.co/my-room" ) if success: print ( "Room deleted successfully" ) ​ Delete Room By Name Deletes a room using its name. ​ room_name string required The name of the Daily room Copy Ask AI success = await helper.delete_room_by_name( "my-room" ) if success: print ( "Room deleted successfully" ) ​ Get Name From URL Extracts the room name from a Daily room URL. ​ room_url string required The complete Daily room URL Copy Ask AI room_name = helper.get_name_from_url( "https://your-domain.daily.co/my-room" ) print ( f "Room name: { room_name } " ) # Outputs: "my-room" Turn Tracking Observer Smart Turn Overview On this page Classes DailyRoomSipParams RecordingsBucketConfig DailyRoomProperties DailyRoomParams DailyRoomObject DailyMeetingTokenProperties DailyMeetingTokenParams Initialize DailyRESTHelper Create Room Get Room From URL Get Token Delete Room By URL Delete Room By Name Get Name From URL Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/daily_rest-helpers_bd54d15a.txt b/daily_rest-helpers_bd54d15a.txt
new file mode 100644
index 0000000000000000000000000000000000000000..be6135ebae0d656bfa281efc887b059ad1e44b9d
--- /dev/null
+++ b/daily_rest-helpers_bd54d15a.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/daily/rest-helpers#param-eject-at-token-exp
+Title: Daily REST Helper - Pipecat
+==================================================
+
+Daily REST Helper - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Service Utilities Daily REST Helper Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Daily REST Helper Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline Daily REST API Documentation For complete Daily REST API reference and additional details ​ Classes ​ DailyRoomSipParams Configuration for SIP (Session Initiation Protocol) parameters. ​ display_name string default: "sw-sip-dialin" Display name for the SIP endpoint ​ video boolean default: false Whether video is enabled for SIP ​ sip_mode string default: "dial-in" SIP connection mode ​ num_endpoints integer default: 1 Number of SIP endpoints Copy Ask AI from pipecat.transports.services.helpers.daily_rest import DailyRoomSipParams sip_params = DailyRoomSipParams( display_name = "conference-line" , video = True , num_endpoints = 2 ) ​ RecordingsBucketConfig Configuration for storing Daily recordings in a custom S3 bucket. ​ bucket_name string required Name of the S3 bucket for storing recordings ​ bucket_region string required AWS region where the S3 bucket is located ​ assume_role_arn string required ARN of the IAM role to assume for S3 access ​ allow_api_access boolean default: false Whether to allow API access to the recordings Copy Ask AI from pipecat.transports.services.helpers.daily_rest import RecordingsBucketConfig bucket_config = RecordingsBucketConfig( bucket_name = "my-recordings-bucket" , bucket_region = "us-west-2" , assume_role_arn = "arn:aws:iam::123456789012:role/DailyRecordingsRole" , allow_api_access = True ) ​ DailyRoomProperties Properties that configure a Daily room’s behavior and features. ​ exp float Room expiration time as Unix timestamp (e.g., time.time() + 300 for 5 minutes) ​ enable_chat boolean default: false Whether chat is enabled in the room ​ enable_prejoin_ui boolean default: false Whether the prejoin lobby UI is enabled ​ enable_emoji_reactions boolean default: false Whether emoji reactions are enabled ​ eject_at_room_exp boolean default: false Whether to eject participants when room expires ​ enable_dialout boolean Whether dial-out is enabled ​ enable_recording string Recording settings (“cloud”, “local”, or “raw-tracks”) ​ geo string Geographic region for room ​ max_participants number Maximum number of participants allowed in the room ​ recordings_bucket RecordingsBucketConfig Configuration for custom S3 bucket recordings ​ sip DailyRoomSipParams SIP configuration parameters ​ sip_uri dict SIP URI configuration (returned by Daily) ​ start_video_off boolean default: false Whether the camera video is turned off by default The class also includes a sip_endpoint property that returns the SIP endpoint URI if available. Copy Ask AI import time from pipecat.transports.services.helpers.daily_rest import ( DailyRoomProperties, DailyRoomSipParams, RecordingsBucketConfig, ) properties = DailyRoomProperties( exp = time.time() + 3600 , # 1 hour from now enable_chat = True , enable_emoji_reactions = True , enable_recording = "cloud" , geo = "us-west" , max_participants = 50 , sip = DailyRoomSipParams( display_name = "conference" ), recordings_bucket = RecordingsBucketConfig( bucket_name = "my-bucket" , bucket_region = "us-west-2" , assume_role_arn = "arn:aws:iam::123456789012:role/DailyRole" ) ) # Access SIP endpoint if available if properties.sip_endpoint: print ( f "SIP endpoint: { properties.sip_endpoint } " ) ​ DailyRoomParams Parameters for creating a new Daily room. ​ name string Room name (if not provided, one will be generated) ​ privacy string default: "public" Room privacy setting (“private” or “public”) ​ properties DailyRoomProperties Room configuration properties Copy Ask AI import time from pipecat.transports.services.helpers.daily_rest import ( DailyRoomParams, DailyRoomProperties, ) params = DailyRoomParams( name = "team-meeting" , privacy = "private" , properties = DailyRoomProperties( enable_chat = True , exp = time.time() + 7200 # 2 hours from now ) ) ​ DailyRoomObject Response object representing a Daily room. ​ id string Unique room identifier ​ name string Room name ​ api_created boolean Whether the room was created via API ​ privacy string Room privacy setting ​ url string Complete room URL ​ created_at string Room creation timestamp in ISO 8601 format ​ config DailyRoomProperties Room configuration Copy Ask AI from pipecat.transports.services.helpers.daily_rest import ( DailyRoomObject, DailyRoomProperties, ) # Example of what a DailyRoomObject looks like when received room = DailyRoomObject( id = "abc123" , name = "team-meeting" , api_created = True , privacy = "private" , url = "https://your-domain.daily.co/team-meeting" , created_at = "2024-01-20T10:00:00.000Z" , config = DailyRoomProperties( enable_chat = True , exp = 1705743600 ) ) ​ DailyMeetingTokenProperties Properties for configuring a Daily meeting token. ​ room_name string The room this token is valid for. If not set, token is valid for all rooms. ​ eject_at_token_exp boolean Whether to eject user when token expires ​ eject_after_elapsed integer Eject user after this many seconds ​ nbf integer “Not before” timestamp - users cannot join before this time ​ exp integer Expiration timestamp - users cannot join after this time ​ is_owner boolean Whether token grants owner privileges ​ user_name string User’s display name in the meeting ​ user_id string Unique identifier for the user (36 char limit) ​ enable_screenshare boolean Whether user can share their screen ​ start_video_off boolean Whether to join with video off ​ start_audio_off boolean Whether to join with audio off ​ enable_recording string Recording settings (“cloud”, “local”, or “raw-tracks”) ​ enable_prejoin_ui boolean Whether to show prejoin UI ​ start_cloud_recording boolean Whether to start cloud recording when user joins ​ permissions dict Initial default permissions for a non-meeting-owner participant ​ DailyMeetingTokenParams Parameters for creating a Daily meeting token. ​ properties DailyMeetingTokenProperties Token configuration properties Copy Ask AI from pipecat.transports.services.helpers.daily_rest import ( DailyMeetingTokenParams, DailyMeetingTokenProperties, ) token_params = DailyMeetingTokenParams( properties = DailyMeetingTokenProperties( user_name = "John Doe" , enable_screenshare = True , start_video_off = True , permissions = { "canSend" : [ "video" , "audio" ]} ) ) ​ Initialize DailyRESTHelper Create a new instance of the Daily REST helper. ​ daily_api_key string required Your Daily API key ​ daily_api_url string default: "https://api.daily.co/v1" The Daily API base URL ​ aiohttp_session aiohttp.ClientSession required An aiohttp client session for making HTTP requests Copy Ask AI helper = DailyRESTHelper( daily_api_key = "your-api-key" , aiohttp_session = session ) ​ Create Room Creates a new Daily room with specified parameters. ​ params DailyRoomParams required Room configuration parameters including name, privacy, and properties Copy Ask AI # Create a room that expires in 1 hour params = DailyRoomParams( name = "my-room" , privacy = "private" , properties = DailyRoomProperties( exp = time.time() + 3600 , enable_chat = True ) ) room = await helper.create_room(params) print ( f "Room URL: { room.url } " ) ​ Get Room From URL Retrieves room information using a Daily room URL. ​ room_url string required The complete Daily room URL Copy Ask AI room = await helper.get_room_from_url( "https://your-domain.daily.co/my-room" ) print ( f "Room name: { room.name } " ) ​ Get Token Generates a meeting token for a specific room. ​ room_url string required The complete Daily room URL ​ expiry_time float default: "3600" Token expiration time in seconds ​ eject_at_token_exp bool default: "False" Whether to eject user when token expires ​ owner bool default: "True" Whether the token should have owner privileges (overrides any setting in params) ​ params DailyMeetingTokenParams Additional token configuration. Note that room_name , exp , eject_at_token_exp , and is_owner will be set based on the other function parameters. Copy Ask AI # Basic token generation token = await helper.get_token( room_url = "https://your-domain.daily.co/my-room" , expiry_time = 1800 , # 30 minutes owner = True , eject_at_token_exp = True ) # Advanced token generation with additional properties token_params = DailyMeetingTokenParams( properties = DailyMeetingTokenProperties( user_name = "John Doe" , start_video_off = True ) ) token = await helper.get_token( room_url = "https://your-domain.daily.co/my-room" , expiry_time = 1800 , owner = False , eject_at_token_exp = True , params = token_params ) ​ Delete Room By URL Deletes a room using its URL. ​ room_url string required The complete Daily room URL Copy Ask AI success = await helper.delete_room_by_url( "https://your-domain.daily.co/my-room" ) if success: print ( "Room deleted successfully" ) ​ Delete Room By Name Deletes a room using its name. ​ room_name string required The name of the Daily room Copy Ask AI success = await helper.delete_room_by_name( "my-room" ) if success: print ( "Room deleted successfully" ) ​ Get Name From URL Extracts the room name from a Daily room URL. ​ room_url string required The complete Daily room URL Copy Ask AI room_name = helper.get_name_from_url( "https://your-domain.daily.co/my-room" ) print ( f "Room name: { room_name } " ) # Outputs: "my-room" Turn Tracking Observer Smart Turn Overview On this page Classes DailyRoomSipParams RecordingsBucketConfig DailyRoomProperties DailyRoomParams DailyRoomObject DailyMeetingTokenProperties DailyMeetingTokenParams Initialize DailyRESTHelper Create Room Get Room From URL Get Token Delete Room By URL Delete Room By Name Get Name From URL Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/daily_rest-helpers_c173069f.txt b/daily_rest-helpers_c173069f.txt
new file mode 100644
index 0000000000000000000000000000000000000000..5bbfd836845ec4a43196451ff1c74ecd1656e90a
--- /dev/null
+++ b/daily_rest-helpers_c173069f.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/daily/rest-helpers#param-start-video-off-1
+Title: Daily REST Helper - Pipecat
+==================================================
+
+Daily REST Helper - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Service Utilities Daily REST Helper Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Daily REST Helper Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline Daily REST API Documentation For complete Daily REST API reference and additional details ​ Classes ​ DailyRoomSipParams Configuration for SIP (Session Initiation Protocol) parameters. ​ display_name string default: "sw-sip-dialin" Display name for the SIP endpoint ​ video boolean default: false Whether video is enabled for SIP ​ sip_mode string default: "dial-in" SIP connection mode ​ num_endpoints integer default: 1 Number of SIP endpoints Copy Ask AI from pipecat.transports.services.helpers.daily_rest import DailyRoomSipParams sip_params = DailyRoomSipParams( display_name = "conference-line" , video = True , num_endpoints = 2 ) ​ RecordingsBucketConfig Configuration for storing Daily recordings in a custom S3 bucket. ​ bucket_name string required Name of the S3 bucket for storing recordings ​ bucket_region string required AWS region where the S3 bucket is located ​ assume_role_arn string required ARN of the IAM role to assume for S3 access ​ allow_api_access boolean default: false Whether to allow API access to the recordings Copy Ask AI from pipecat.transports.services.helpers.daily_rest import RecordingsBucketConfig bucket_config = RecordingsBucketConfig( bucket_name = "my-recordings-bucket" , bucket_region = "us-west-2" , assume_role_arn = "arn:aws:iam::123456789012:role/DailyRecordingsRole" , allow_api_access = True ) ​ DailyRoomProperties Properties that configure a Daily room’s behavior and features. ​ exp float Room expiration time as Unix timestamp (e.g., time.time() + 300 for 5 minutes) ​ enable_chat boolean default: false Whether chat is enabled in the room ​ enable_prejoin_ui boolean default: false Whether the prejoin lobby UI is enabled ​ enable_emoji_reactions boolean default: false Whether emoji reactions are enabled ​ eject_at_room_exp boolean default: false Whether to eject participants when room expires ​ enable_dialout boolean Whether dial-out is enabled ​ enable_recording string Recording settings (“cloud”, “local”, or “raw-tracks”) ​ geo string Geographic region for room ​ max_participants number Maximum number of participants allowed in the room ​ recordings_bucket RecordingsBucketConfig Configuration for custom S3 bucket recordings ​ sip DailyRoomSipParams SIP configuration parameters ​ sip_uri dict SIP URI configuration (returned by Daily) ​ start_video_off boolean default: false Whether the camera video is turned off by default The class also includes a sip_endpoint property that returns the SIP endpoint URI if available. Copy Ask AI import time from pipecat.transports.services.helpers.daily_rest import ( DailyRoomProperties, DailyRoomSipParams, RecordingsBucketConfig, ) properties = DailyRoomProperties( exp = time.time() + 3600 , # 1 hour from now enable_chat = True , enable_emoji_reactions = True , enable_recording = "cloud" , geo = "us-west" , max_participants = 50 , sip = DailyRoomSipParams( display_name = "conference" ), recordings_bucket = RecordingsBucketConfig( bucket_name = "my-bucket" , bucket_region = "us-west-2" , assume_role_arn = "arn:aws:iam::123456789012:role/DailyRole" ) ) # Access SIP endpoint if available if properties.sip_endpoint: print ( f "SIP endpoint: { properties.sip_endpoint } " ) ​ DailyRoomParams Parameters for creating a new Daily room. ​ name string Room name (if not provided, one will be generated) ​ privacy string default: "public" Room privacy setting (“private” or “public”) ​ properties DailyRoomProperties Room configuration properties Copy Ask AI import time from pipecat.transports.services.helpers.daily_rest import ( DailyRoomParams, DailyRoomProperties, ) params = DailyRoomParams( name = "team-meeting" , privacy = "private" , properties = DailyRoomProperties( enable_chat = True , exp = time.time() + 7200 # 2 hours from now ) ) ​ DailyRoomObject Response object representing a Daily room. ​ id string Unique room identifier ​ name string Room name ​ api_created boolean Whether the room was created via API ​ privacy string Room privacy setting ​ url string Complete room URL ​ created_at string Room creation timestamp in ISO 8601 format ​ config DailyRoomProperties Room configuration Copy Ask AI from pipecat.transports.services.helpers.daily_rest import ( DailyRoomObject, DailyRoomProperties, ) # Example of what a DailyRoomObject looks like when received room = DailyRoomObject( id = "abc123" , name = "team-meeting" , api_created = True , privacy = "private" , url = "https://your-domain.daily.co/team-meeting" , created_at = "2024-01-20T10:00:00.000Z" , config = DailyRoomProperties( enable_chat = True , exp = 1705743600 ) ) ​ DailyMeetingTokenProperties Properties for configuring a Daily meeting token. ​ room_name string The room this token is valid for. If not set, token is valid for all rooms. ​ eject_at_token_exp boolean Whether to eject user when token expires ​ eject_after_elapsed integer Eject user after this many seconds ​ nbf integer “Not before” timestamp - users cannot join before this time ​ exp integer Expiration timestamp - users cannot join after this time ​ is_owner boolean Whether token grants owner privileges ​ user_name string User’s display name in the meeting ​ user_id string Unique identifier for the user (36 char limit) ​ enable_screenshare boolean Whether user can share their screen ​ start_video_off boolean Whether to join with video off ​ start_audio_off boolean Whether to join with audio off ​ enable_recording string Recording settings (“cloud”, “local”, or “raw-tracks”) ​ enable_prejoin_ui boolean Whether to show prejoin UI ​ start_cloud_recording boolean Whether to start cloud recording when user joins ​ permissions dict Initial default permissions for a non-meeting-owner participant ​ DailyMeetingTokenParams Parameters for creating a Daily meeting token. ​ properties DailyMeetingTokenProperties Token configuration properties Copy Ask AI from pipecat.transports.services.helpers.daily_rest import ( DailyMeetingTokenParams, DailyMeetingTokenProperties, ) token_params = DailyMeetingTokenParams( properties = DailyMeetingTokenProperties( user_name = "John Doe" , enable_screenshare = True , start_video_off = True , permissions = { "canSend" : [ "video" , "audio" ]} ) ) ​ Initialize DailyRESTHelper Create a new instance of the Daily REST helper. ​ daily_api_key string required Your Daily API key ​ daily_api_url string default: "https://api.daily.co/v1" The Daily API base URL ​ aiohttp_session aiohttp.ClientSession required An aiohttp client session for making HTTP requests Copy Ask AI helper = DailyRESTHelper( daily_api_key = "your-api-key" , aiohttp_session = session ) ​ Create Room Creates a new Daily room with specified parameters. ​ params DailyRoomParams required Room configuration parameters including name, privacy, and properties Copy Ask AI # Create a room that expires in 1 hour params = DailyRoomParams( name = "my-room" , privacy = "private" , properties = DailyRoomProperties( exp = time.time() + 3600 , enable_chat = True ) ) room = await helper.create_room(params) print ( f "Room URL: { room.url } " ) ​ Get Room From URL Retrieves room information using a Daily room URL. ​ room_url string required The complete Daily room URL Copy Ask AI room = await helper.get_room_from_url( "https://your-domain.daily.co/my-room" ) print ( f "Room name: { room.name } " ) ​ Get Token Generates a meeting token for a specific room. ​ room_url string required The complete Daily room URL ​ expiry_time float default: "3600" Token expiration time in seconds ​ eject_at_token_exp bool default: "False" Whether to eject user when token expires ​ owner bool default: "True" Whether the token should have owner privileges (overrides any setting in params) ​ params DailyMeetingTokenParams Additional token configuration. Note that room_name , exp , eject_at_token_exp , and is_owner will be set based on the other function parameters. Copy Ask AI # Basic token generation token = await helper.get_token( room_url = "https://your-domain.daily.co/my-room" , expiry_time = 1800 , # 30 minutes owner = True , eject_at_token_exp = True ) # Advanced token generation with additional properties token_params = DailyMeetingTokenParams( properties = DailyMeetingTokenProperties( user_name = "John Doe" , start_video_off = True ) ) token = await helper.get_token( room_url = "https://your-domain.daily.co/my-room" , expiry_time = 1800 , owner = False , eject_at_token_exp = True , params = token_params ) ​ Delete Room By URL Deletes a room using its URL. ​ room_url string required The complete Daily room URL Copy Ask AI success = await helper.delete_room_by_url( "https://your-domain.daily.co/my-room" ) if success: print ( "Room deleted successfully" ) ​ Delete Room By Name Deletes a room using its name. ​ room_name string required The name of the Daily room Copy Ask AI success = await helper.delete_room_by_name( "my-room" ) if success: print ( "Room deleted successfully" ) ​ Get Name From URL Extracts the room name from a Daily room URL. ​ room_url string required The complete Daily room URL Copy Ask AI room_name = helper.get_name_from_url( "https://your-domain.daily.co/my-room" ) print ( f "Room name: { room_name } " ) # Outputs: "my-room" Turn Tracking Observer Smart Turn Overview On this page Classes DailyRoomSipParams RecordingsBucketConfig DailyRoomProperties DailyRoomParams DailyRoomObject DailyMeetingTokenProperties DailyMeetingTokenParams Initialize DailyRESTHelper Create Room Get Room From URL Get Token Delete Room By URL Delete Room By Name Get Name From URL Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/daily_rest-helpers_c501276a.txt b/daily_rest-helpers_c501276a.txt
new file mode 100644
index 0000000000000000000000000000000000000000..0ab32c889feb90e8fc172bff8826b8a27ca85293
--- /dev/null
+++ b/daily_rest-helpers_c501276a.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/daily/rest-helpers#param-start-video-off
+Title: Daily REST Helper - Pipecat
+==================================================
+
+Daily REST Helper - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Service Utilities Daily REST Helper Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Daily REST Helper Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline Daily REST API Documentation For complete Daily REST API reference and additional details ​ Classes ​ DailyRoomSipParams Configuration for SIP (Session Initiation Protocol) parameters. ​ display_name string default: "sw-sip-dialin" Display name for the SIP endpoint ​ video boolean default: false Whether video is enabled for SIP ​ sip_mode string default: "dial-in" SIP connection mode ​ num_endpoints integer default: 1 Number of SIP endpoints Copy Ask AI from pipecat.transports.services.helpers.daily_rest import DailyRoomSipParams sip_params = DailyRoomSipParams( display_name = "conference-line" , video = True , num_endpoints = 2 ) ​ RecordingsBucketConfig Configuration for storing Daily recordings in a custom S3 bucket. ​ bucket_name string required Name of the S3 bucket for storing recordings ​ bucket_region string required AWS region where the S3 bucket is located ​ assume_role_arn string required ARN of the IAM role to assume for S3 access ​ allow_api_access boolean default: false Whether to allow API access to the recordings Copy Ask AI from pipecat.transports.services.helpers.daily_rest import RecordingsBucketConfig bucket_config = RecordingsBucketConfig( bucket_name = "my-recordings-bucket" , bucket_region = "us-west-2" , assume_role_arn = "arn:aws:iam::123456789012:role/DailyRecordingsRole" , allow_api_access = True ) ​ DailyRoomProperties Properties that configure a Daily room’s behavior and features. ​ exp float Room expiration time as Unix timestamp (e.g., time.time() + 300 for 5 minutes) ​ enable_chat boolean default: false Whether chat is enabled in the room ​ enable_prejoin_ui boolean default: false Whether the prejoin lobby UI is enabled ​ enable_emoji_reactions boolean default: false Whether emoji reactions are enabled ​ eject_at_room_exp boolean default: false Whether to eject participants when room expires ​ enable_dialout boolean Whether dial-out is enabled ​ enable_recording string Recording settings (“cloud”, “local”, or “raw-tracks”) ​ geo string Geographic region for room ​ max_participants number Maximum number of participants allowed in the room ​ recordings_bucket RecordingsBucketConfig Configuration for custom S3 bucket recordings ​ sip DailyRoomSipParams SIP configuration parameters ​ sip_uri dict SIP URI configuration (returned by Daily) ​ start_video_off boolean default: false Whether the camera video is turned off by default The class also includes a sip_endpoint property that returns the SIP endpoint URI if available. Copy Ask AI import time from pipecat.transports.services.helpers.daily_rest import ( DailyRoomProperties, DailyRoomSipParams, RecordingsBucketConfig, ) properties = DailyRoomProperties( exp = time.time() + 3600 , # 1 hour from now enable_chat = True , enable_emoji_reactions = True , enable_recording = "cloud" , geo = "us-west" , max_participants = 50 , sip = DailyRoomSipParams( display_name = "conference" ), recordings_bucket = RecordingsBucketConfig( bucket_name = "my-bucket" , bucket_region = "us-west-2" , assume_role_arn = "arn:aws:iam::123456789012:role/DailyRole" ) ) # Access SIP endpoint if available if properties.sip_endpoint: print ( f "SIP endpoint: { properties.sip_endpoint } " ) ​ DailyRoomParams Parameters for creating a new Daily room. ​ name string Room name (if not provided, one will be generated) ​ privacy string default: "public" Room privacy setting (“private” or “public”) ​ properties DailyRoomProperties Room configuration properties Copy Ask AI import time from pipecat.transports.services.helpers.daily_rest import ( DailyRoomParams, DailyRoomProperties, ) params = DailyRoomParams( name = "team-meeting" , privacy = "private" , properties = DailyRoomProperties( enable_chat = True , exp = time.time() + 7200 # 2 hours from now ) ) ​ DailyRoomObject Response object representing a Daily room. ​ id string Unique room identifier ​ name string Room name ​ api_created boolean Whether the room was created via API ​ privacy string Room privacy setting ​ url string Complete room URL ​ created_at string Room creation timestamp in ISO 8601 format ​ config DailyRoomProperties Room configuration Copy Ask AI from pipecat.transports.services.helpers.daily_rest import ( DailyRoomObject, DailyRoomProperties, ) # Example of what a DailyRoomObject looks like when received room = DailyRoomObject( id = "abc123" , name = "team-meeting" , api_created = True , privacy = "private" , url = "https://your-domain.daily.co/team-meeting" , created_at = "2024-01-20T10:00:00.000Z" , config = DailyRoomProperties( enable_chat = True , exp = 1705743600 ) ) ​ DailyMeetingTokenProperties Properties for configuring a Daily meeting token. ​ room_name string The room this token is valid for. If not set, token is valid for all rooms. ​ eject_at_token_exp boolean Whether to eject user when token expires ​ eject_after_elapsed integer Eject user after this many seconds ​ nbf integer “Not before” timestamp - users cannot join before this time ​ exp integer Expiration timestamp - users cannot join after this time ​ is_owner boolean Whether token grants owner privileges ​ user_name string User’s display name in the meeting ​ user_id string Unique identifier for the user (36 char limit) ​ enable_screenshare boolean Whether user can share their screen ​ start_video_off boolean Whether to join with video off ​ start_audio_off boolean Whether to join with audio off ​ enable_recording string Recording settings (“cloud”, “local”, or “raw-tracks”) ​ enable_prejoin_ui boolean Whether to show prejoin UI ​ start_cloud_recording boolean Whether to start cloud recording when user joins ​ permissions dict Initial default permissions for a non-meeting-owner participant ​ DailyMeetingTokenParams Parameters for creating a Daily meeting token. ​ properties DailyMeetingTokenProperties Token configuration properties Copy Ask AI from pipecat.transports.services.helpers.daily_rest import ( DailyMeetingTokenParams, DailyMeetingTokenProperties, ) token_params = DailyMeetingTokenParams( properties = DailyMeetingTokenProperties( user_name = "John Doe" , enable_screenshare = True , start_video_off = True , permissions = { "canSend" : [ "video" , "audio" ]} ) ) ​ Initialize DailyRESTHelper Create a new instance of the Daily REST helper. ​ daily_api_key string required Your Daily API key ​ daily_api_url string default: "https://api.daily.co/v1" The Daily API base URL ​ aiohttp_session aiohttp.ClientSession required An aiohttp client session for making HTTP requests Copy Ask AI helper = DailyRESTHelper( daily_api_key = "your-api-key" , aiohttp_session = session ) ​ Create Room Creates a new Daily room with specified parameters. ​ params DailyRoomParams required Room configuration parameters including name, privacy, and properties Copy Ask AI # Create a room that expires in 1 hour params = DailyRoomParams( name = "my-room" , privacy = "private" , properties = DailyRoomProperties( exp = time.time() + 3600 , enable_chat = True ) ) room = await helper.create_room(params) print ( f "Room URL: { room.url } " ) ​ Get Room From URL Retrieves room information using a Daily room URL. ​ room_url string required The complete Daily room URL Copy Ask AI room = await helper.get_room_from_url( "https://your-domain.daily.co/my-room" ) print ( f "Room name: { room.name } " ) ​ Get Token Generates a meeting token for a specific room. ​ room_url string required The complete Daily room URL ​ expiry_time float default: "3600" Token expiration time in seconds ​ eject_at_token_exp bool default: "False" Whether to eject user when token expires ​ owner bool default: "True" Whether the token should have owner privileges (overrides any setting in params) ​ params DailyMeetingTokenParams Additional token configuration. Note that room_name , exp , eject_at_token_exp , and is_owner will be set based on the other function parameters. Copy Ask AI # Basic token generation token = await helper.get_token( room_url = "https://your-domain.daily.co/my-room" , expiry_time = 1800 , # 30 minutes owner = True , eject_at_token_exp = True ) # Advanced token generation with additional properties token_params = DailyMeetingTokenParams( properties = DailyMeetingTokenProperties( user_name = "John Doe" , start_video_off = True ) ) token = await helper.get_token( room_url = "https://your-domain.daily.co/my-room" , expiry_time = 1800 , owner = False , eject_at_token_exp = True , params = token_params ) ​ Delete Room By URL Deletes a room using its URL. ​ room_url string required The complete Daily room URL Copy Ask AI success = await helper.delete_room_by_url( "https://your-domain.daily.co/my-room" ) if success: print ( "Room deleted successfully" ) ​ Delete Room By Name Deletes a room using its name. ​ room_name string required The name of the Daily room Copy Ask AI success = await helper.delete_room_by_name( "my-room" ) if success: print ( "Room deleted successfully" ) ​ Get Name From URL Extracts the room name from a Daily room URL. ​ room_url string required The complete Daily room URL Copy Ask AI room_name = helper.get_name_from_url( "https://your-domain.daily.co/my-room" ) print ( f "Room name: { room_name } " ) # Outputs: "my-room" Turn Tracking Observer Smart Turn Overview On this page Classes DailyRoomSipParams RecordingsBucketConfig DailyRoomProperties DailyRoomParams DailyRoomObject DailyMeetingTokenProperties DailyMeetingTokenParams Initialize DailyRESTHelper Create Room Get Room From URL Get Token Delete Room By URL Delete Room By Name Get Name From URL Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/daily_rest-helpers_dbfab31b.txt b/daily_rest-helpers_dbfab31b.txt
new file mode 100644
index 0000000000000000000000000000000000000000..cd71093f22374e79e14646133c398068ade93f0f
--- /dev/null
+++ b/daily_rest-helpers_dbfab31b.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/daily/rest-helpers#param-api-created
+Title: Daily REST Helper - Pipecat
+==================================================
+
+Daily REST Helper - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Service Utilities Daily REST Helper Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Daily REST Helper Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline Daily REST API Documentation For complete Daily REST API reference and additional details ​ Classes ​ DailyRoomSipParams Configuration for SIP (Session Initiation Protocol) parameters. ​ display_name string default: "sw-sip-dialin" Display name for the SIP endpoint ​ video boolean default: false Whether video is enabled for SIP ​ sip_mode string default: "dial-in" SIP connection mode ​ num_endpoints integer default: 1 Number of SIP endpoints Copy Ask AI from pipecat.transports.services.helpers.daily_rest import DailyRoomSipParams sip_params = DailyRoomSipParams( display_name = "conference-line" , video = True , num_endpoints = 2 ) ​ RecordingsBucketConfig Configuration for storing Daily recordings in a custom S3 bucket. ​ bucket_name string required Name of the S3 bucket for storing recordings ​ bucket_region string required AWS region where the S3 bucket is located ​ assume_role_arn string required ARN of the IAM role to assume for S3 access ​ allow_api_access boolean default: false Whether to allow API access to the recordings Copy Ask AI from pipecat.transports.services.helpers.daily_rest import RecordingsBucketConfig bucket_config = RecordingsBucketConfig( bucket_name = "my-recordings-bucket" , bucket_region = "us-west-2" , assume_role_arn = "arn:aws:iam::123456789012:role/DailyRecordingsRole" , allow_api_access = True ) ​ DailyRoomProperties Properties that configure a Daily room’s behavior and features. ​ exp float Room expiration time as Unix timestamp (e.g., time.time() + 300 for 5 minutes) ​ enable_chat boolean default: false Whether chat is enabled in the room ​ enable_prejoin_ui boolean default: false Whether the prejoin lobby UI is enabled ​ enable_emoji_reactions boolean default: false Whether emoji reactions are enabled ​ eject_at_room_exp boolean default: false Whether to eject participants when room expires ​ enable_dialout boolean Whether dial-out is enabled ​ enable_recording string Recording settings (“cloud”, “local”, or “raw-tracks”) ​ geo string Geographic region for room ​ max_participants number Maximum number of participants allowed in the room ​ recordings_bucket RecordingsBucketConfig Configuration for custom S3 bucket recordings ​ sip DailyRoomSipParams SIP configuration parameters ​ sip_uri dict SIP URI configuration (returned by Daily) ​ start_video_off boolean default: false Whether the camera video is turned off by default The class also includes a sip_endpoint property that returns the SIP endpoint URI if available. Copy Ask AI import time from pipecat.transports.services.helpers.daily_rest import ( DailyRoomProperties, DailyRoomSipParams, RecordingsBucketConfig, ) properties = DailyRoomProperties( exp = time.time() + 3600 , # 1 hour from now enable_chat = True , enable_emoji_reactions = True , enable_recording = "cloud" , geo = "us-west" , max_participants = 50 , sip = DailyRoomSipParams( display_name = "conference" ), recordings_bucket = RecordingsBucketConfig( bucket_name = "my-bucket" , bucket_region = "us-west-2" , assume_role_arn = "arn:aws:iam::123456789012:role/DailyRole" ) ) # Access SIP endpoint if available if properties.sip_endpoint: print ( f "SIP endpoint: { properties.sip_endpoint } " ) ​ DailyRoomParams Parameters for creating a new Daily room. ​ name string Room name (if not provided, one will be generated) ​ privacy string default: "public" Room privacy setting (“private” or “public”) ​ properties DailyRoomProperties Room configuration properties Copy Ask AI import time from pipecat.transports.services.helpers.daily_rest import ( DailyRoomParams, DailyRoomProperties, ) params = DailyRoomParams( name = "team-meeting" , privacy = "private" , properties = DailyRoomProperties( enable_chat = True , exp = time.time() + 7200 # 2 hours from now ) ) ​ DailyRoomObject Response object representing a Daily room. ​ id string Unique room identifier ​ name string Room name ​ api_created boolean Whether the room was created via API ​ privacy string Room privacy setting ​ url string Complete room URL ​ created_at string Room creation timestamp in ISO 8601 format ​ config DailyRoomProperties Room configuration Copy Ask AI from pipecat.transports.services.helpers.daily_rest import ( DailyRoomObject, DailyRoomProperties, ) # Example of what a DailyRoomObject looks like when received room = DailyRoomObject( id = "abc123" , name = "team-meeting" , api_created = True , privacy = "private" , url = "https://your-domain.daily.co/team-meeting" , created_at = "2024-01-20T10:00:00.000Z" , config = DailyRoomProperties( enable_chat = True , exp = 1705743600 ) ) ​ DailyMeetingTokenProperties Properties for configuring a Daily meeting token. ​ room_name string The room this token is valid for. If not set, token is valid for all rooms. ​ eject_at_token_exp boolean Whether to eject user when token expires ​ eject_after_elapsed integer Eject user after this many seconds ​ nbf integer “Not before” timestamp - users cannot join before this time ​ exp integer Expiration timestamp - users cannot join after this time ​ is_owner boolean Whether token grants owner privileges ​ user_name string User’s display name in the meeting ​ user_id string Unique identifier for the user (36 char limit) ​ enable_screenshare boolean Whether user can share their screen ​ start_video_off boolean Whether to join with video off ​ start_audio_off boolean Whether to join with audio off ​ enable_recording string Recording settings (“cloud”, “local”, or “raw-tracks”) ​ enable_prejoin_ui boolean Whether to show prejoin UI ​ start_cloud_recording boolean Whether to start cloud recording when user joins ​ permissions dict Initial default permissions for a non-meeting-owner participant ​ DailyMeetingTokenParams Parameters for creating a Daily meeting token. ​ properties DailyMeetingTokenProperties Token configuration properties Copy Ask AI from pipecat.transports.services.helpers.daily_rest import ( DailyMeetingTokenParams, DailyMeetingTokenProperties, ) token_params = DailyMeetingTokenParams( properties = DailyMeetingTokenProperties( user_name = "John Doe" , enable_screenshare = True , start_video_off = True , permissions = { "canSend" : [ "video" , "audio" ]} ) ) ​ Initialize DailyRESTHelper Create a new instance of the Daily REST helper. ​ daily_api_key string required Your Daily API key ​ daily_api_url string default: "https://api.daily.co/v1" The Daily API base URL ​ aiohttp_session aiohttp.ClientSession required An aiohttp client session for making HTTP requests Copy Ask AI helper = DailyRESTHelper( daily_api_key = "your-api-key" , aiohttp_session = session ) ​ Create Room Creates a new Daily room with specified parameters. ​ params DailyRoomParams required Room configuration parameters including name, privacy, and properties Copy Ask AI # Create a room that expires in 1 hour params = DailyRoomParams( name = "my-room" , privacy = "private" , properties = DailyRoomProperties( exp = time.time() + 3600 , enable_chat = True ) ) room = await helper.create_room(params) print ( f "Room URL: { room.url } " ) ​ Get Room From URL Retrieves room information using a Daily room URL. ​ room_url string required The complete Daily room URL Copy Ask AI room = await helper.get_room_from_url( "https://your-domain.daily.co/my-room" ) print ( f "Room name: { room.name } " ) ​ Get Token Generates a meeting token for a specific room. ​ room_url string required The complete Daily room URL ​ expiry_time float default: "3600" Token expiration time in seconds ​ eject_at_token_exp bool default: "False" Whether to eject user when token expires ​ owner bool default: "True" Whether the token should have owner privileges (overrides any setting in params) ​ params DailyMeetingTokenParams Additional token configuration. Note that room_name , exp , eject_at_token_exp , and is_owner will be set based on the other function parameters. Copy Ask AI # Basic token generation token = await helper.get_token( room_url = "https://your-domain.daily.co/my-room" , expiry_time = 1800 , # 30 minutes owner = True , eject_at_token_exp = True ) # Advanced token generation with additional properties token_params = DailyMeetingTokenParams( properties = DailyMeetingTokenProperties( user_name = "John Doe" , start_video_off = True ) ) token = await helper.get_token( room_url = "https://your-domain.daily.co/my-room" , expiry_time = 1800 , owner = False , eject_at_token_exp = True , params = token_params ) ​ Delete Room By URL Deletes a room using its URL. ​ room_url string required The complete Daily room URL Copy Ask AI success = await helper.delete_room_by_url( "https://your-domain.daily.co/my-room" ) if success: print ( "Room deleted successfully" ) ​ Delete Room By Name Deletes a room using its name. ​ room_name string required The name of the Daily room Copy Ask AI success = await helper.delete_room_by_name( "my-room" ) if success: print ( "Room deleted successfully" ) ​ Get Name From URL Extracts the room name from a Daily room URL. ​ room_url string required The complete Daily room URL Copy Ask AI room_name = helper.get_name_from_url( "https://your-domain.daily.co/my-room" ) print ( f "Room name: { room_name } " ) # Outputs: "my-room" Turn Tracking Observer Smart Turn Overview On this page Classes DailyRoomSipParams RecordingsBucketConfig DailyRoomProperties DailyRoomParams DailyRoomObject DailyMeetingTokenProperties DailyMeetingTokenParams Initialize DailyRESTHelper Create Room Get Room From URL Get Token Delete Room By URL Delete Room By Name Get Name From URL Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/daily_rest-helpers_e0baf334.txt b/daily_rest-helpers_e0baf334.txt
new file mode 100644
index 0000000000000000000000000000000000000000..2003fd1ae1339df3c57cb320ff1ccfefe0c1b891
--- /dev/null
+++ b/daily_rest-helpers_e0baf334.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/daily/rest-helpers#param-daily-api-url
+Title: Daily REST Helper - Pipecat
+==================================================
+
+Daily REST Helper - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Service Utilities Daily REST Helper Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Daily REST Helper Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline Daily REST API Documentation For complete Daily REST API reference and additional details ​ Classes ​ DailyRoomSipParams Configuration for SIP (Session Initiation Protocol) parameters. ​ display_name string default: "sw-sip-dialin" Display name for the SIP endpoint ​ video boolean default: false Whether video is enabled for SIP ​ sip_mode string default: "dial-in" SIP connection mode ​ num_endpoints integer default: 1 Number of SIP endpoints Copy Ask AI from pipecat.transports.services.helpers.daily_rest import DailyRoomSipParams sip_params = DailyRoomSipParams( display_name = "conference-line" , video = True , num_endpoints = 2 ) ​ RecordingsBucketConfig Configuration for storing Daily recordings in a custom S3 bucket. ​ bucket_name string required Name of the S3 bucket for storing recordings ​ bucket_region string required AWS region where the S3 bucket is located ​ assume_role_arn string required ARN of the IAM role to assume for S3 access ​ allow_api_access boolean default: false Whether to allow API access to the recordings Copy Ask AI from pipecat.transports.services.helpers.daily_rest import RecordingsBucketConfig bucket_config = RecordingsBucketConfig( bucket_name = "my-recordings-bucket" , bucket_region = "us-west-2" , assume_role_arn = "arn:aws:iam::123456789012:role/DailyRecordingsRole" , allow_api_access = True ) ​ DailyRoomProperties Properties that configure a Daily room’s behavior and features. ​ exp float Room expiration time as Unix timestamp (e.g., time.time() + 300 for 5 minutes) ​ enable_chat boolean default: false Whether chat is enabled in the room ​ enable_prejoin_ui boolean default: false Whether the prejoin lobby UI is enabled ​ enable_emoji_reactions boolean default: false Whether emoji reactions are enabled ​ eject_at_room_exp boolean default: false Whether to eject participants when room expires ​ enable_dialout boolean Whether dial-out is enabled ​ enable_recording string Recording settings (“cloud”, “local”, or “raw-tracks”) ​ geo string Geographic region for room ​ max_participants number Maximum number of participants allowed in the room ​ recordings_bucket RecordingsBucketConfig Configuration for custom S3 bucket recordings ​ sip DailyRoomSipParams SIP configuration parameters ​ sip_uri dict SIP URI configuration (returned by Daily) ​ start_video_off boolean default: false Whether the camera video is turned off by default The class also includes a sip_endpoint property that returns the SIP endpoint URI if available. Copy Ask AI import time from pipecat.transports.services.helpers.daily_rest import ( DailyRoomProperties, DailyRoomSipParams, RecordingsBucketConfig, ) properties = DailyRoomProperties( exp = time.time() + 3600 , # 1 hour from now enable_chat = True , enable_emoji_reactions = True , enable_recording = "cloud" , geo = "us-west" , max_participants = 50 , sip = DailyRoomSipParams( display_name = "conference" ), recordings_bucket = RecordingsBucketConfig( bucket_name = "my-bucket" , bucket_region = "us-west-2" , assume_role_arn = "arn:aws:iam::123456789012:role/DailyRole" ) ) # Access SIP endpoint if available if properties.sip_endpoint: print ( f "SIP endpoint: { properties.sip_endpoint } " ) ​ DailyRoomParams Parameters for creating a new Daily room. ​ name string Room name (if not provided, one will be generated) ​ privacy string default: "public" Room privacy setting (“private” or “public”) ​ properties DailyRoomProperties Room configuration properties Copy Ask AI import time from pipecat.transports.services.helpers.daily_rest import ( DailyRoomParams, DailyRoomProperties, ) params = DailyRoomParams( name = "team-meeting" , privacy = "private" , properties = DailyRoomProperties( enable_chat = True , exp = time.time() + 7200 # 2 hours from now ) ) ​ DailyRoomObject Response object representing a Daily room. ​ id string Unique room identifier ​ name string Room name ​ api_created boolean Whether the room was created via API ​ privacy string Room privacy setting ​ url string Complete room URL ​ created_at string Room creation timestamp in ISO 8601 format ​ config DailyRoomProperties Room configuration Copy Ask AI from pipecat.transports.services.helpers.daily_rest import ( DailyRoomObject, DailyRoomProperties, ) # Example of what a DailyRoomObject looks like when received room = DailyRoomObject( id = "abc123" , name = "team-meeting" , api_created = True , privacy = "private" , url = "https://your-domain.daily.co/team-meeting" , created_at = "2024-01-20T10:00:00.000Z" , config = DailyRoomProperties( enable_chat = True , exp = 1705743600 ) ) ​ DailyMeetingTokenProperties Properties for configuring a Daily meeting token. ​ room_name string The room this token is valid for. If not set, token is valid for all rooms. ​ eject_at_token_exp boolean Whether to eject user when token expires ​ eject_after_elapsed integer Eject user after this many seconds ​ nbf integer “Not before” timestamp - users cannot join before this time ​ exp integer Expiration timestamp - users cannot join after this time ​ is_owner boolean Whether token grants owner privileges ​ user_name string User’s display name in the meeting ​ user_id string Unique identifier for the user (36 char limit) ​ enable_screenshare boolean Whether user can share their screen ​ start_video_off boolean Whether to join with video off ​ start_audio_off boolean Whether to join with audio off ​ enable_recording string Recording settings (“cloud”, “local”, or “raw-tracks”) ​ enable_prejoin_ui boolean Whether to show prejoin UI ​ start_cloud_recording boolean Whether to start cloud recording when user joins ​ permissions dict Initial default permissions for a non-meeting-owner participant ​ DailyMeetingTokenParams Parameters for creating a Daily meeting token. ​ properties DailyMeetingTokenProperties Token configuration properties Copy Ask AI from pipecat.transports.services.helpers.daily_rest import ( DailyMeetingTokenParams, DailyMeetingTokenProperties, ) token_params = DailyMeetingTokenParams( properties = DailyMeetingTokenProperties( user_name = "John Doe" , enable_screenshare = True , start_video_off = True , permissions = { "canSend" : [ "video" , "audio" ]} ) ) ​ Initialize DailyRESTHelper Create a new instance of the Daily REST helper. ​ daily_api_key string required Your Daily API key ​ daily_api_url string default: "https://api.daily.co/v1" The Daily API base URL ​ aiohttp_session aiohttp.ClientSession required An aiohttp client session for making HTTP requests Copy Ask AI helper = DailyRESTHelper( daily_api_key = "your-api-key" , aiohttp_session = session ) ​ Create Room Creates a new Daily room with specified parameters. ​ params DailyRoomParams required Room configuration parameters including name, privacy, and properties Copy Ask AI # Create a room that expires in 1 hour params = DailyRoomParams( name = "my-room" , privacy = "private" , properties = DailyRoomProperties( exp = time.time() + 3600 , enable_chat = True ) ) room = await helper.create_room(params) print ( f "Room URL: { room.url } " ) ​ Get Room From URL Retrieves room information using a Daily room URL. ​ room_url string required The complete Daily room URL Copy Ask AI room = await helper.get_room_from_url( "https://your-domain.daily.co/my-room" ) print ( f "Room name: { room.name } " ) ​ Get Token Generates a meeting token for a specific room. ​ room_url string required The complete Daily room URL ​ expiry_time float default: "3600" Token expiration time in seconds ​ eject_at_token_exp bool default: "False" Whether to eject user when token expires ​ owner bool default: "True" Whether the token should have owner privileges (overrides any setting in params) ​ params DailyMeetingTokenParams Additional token configuration. Note that room_name , exp , eject_at_token_exp , and is_owner will be set based on the other function parameters. Copy Ask AI # Basic token generation token = await helper.get_token( room_url = "https://your-domain.daily.co/my-room" , expiry_time = 1800 , # 30 minutes owner = True , eject_at_token_exp = True ) # Advanced token generation with additional properties token_params = DailyMeetingTokenParams( properties = DailyMeetingTokenProperties( user_name = "John Doe" , start_video_off = True ) ) token = await helper.get_token( room_url = "https://your-domain.daily.co/my-room" , expiry_time = 1800 , owner = False , eject_at_token_exp = True , params = token_params ) ​ Delete Room By URL Deletes a room using its URL. ​ room_url string required The complete Daily room URL Copy Ask AI success = await helper.delete_room_by_url( "https://your-domain.daily.co/my-room" ) if success: print ( "Room deleted successfully" ) ​ Delete Room By Name Deletes a room using its name. ​ room_name string required The name of the Daily room Copy Ask AI success = await helper.delete_room_by_name( "my-room" ) if success: print ( "Room deleted successfully" ) ​ Get Name From URL Extracts the room name from a Daily room URL. ​ room_url string required The complete Daily room URL Copy Ask AI room_name = helper.get_name_from_url( "https://your-domain.daily.co/my-room" ) print ( f "Room name: { room_name } " ) # Outputs: "my-room" Turn Tracking Observer Smart Turn Overview On this page Classes DailyRoomSipParams RecordingsBucketConfig DailyRoomProperties DailyRoomParams DailyRoomObject DailyMeetingTokenProperties DailyMeetingTokenParams Initialize DailyRESTHelper Create Room Get Room From URL Get Token Delete Room By URL Delete Room By Name Get Name From URL Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/daily_rest-helpers_ebdb430e.txt b/daily_rest-helpers_ebdb430e.txt
new file mode 100644
index 0000000000000000000000000000000000000000..8ff77b06c177430ffafcd8b3744167b3a7e1c180
--- /dev/null
+++ b/daily_rest-helpers_ebdb430e.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/daily/rest-helpers#param-params-1
+Title: Daily REST Helper - Pipecat
+==================================================
+
+Daily REST Helper - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Service Utilities Daily REST Helper Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Daily REST Helper Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline Daily REST API Documentation For complete Daily REST API reference and additional details ​ Classes ​ DailyRoomSipParams Configuration for SIP (Session Initiation Protocol) parameters. ​ display_name string default: "sw-sip-dialin" Display name for the SIP endpoint ​ video boolean default: false Whether video is enabled for SIP ​ sip_mode string default: "dial-in" SIP connection mode ​ num_endpoints integer default: 1 Number of SIP endpoints Copy Ask AI from pipecat.transports.services.helpers.daily_rest import DailyRoomSipParams sip_params = DailyRoomSipParams( display_name = "conference-line" , video = True , num_endpoints = 2 ) ​ RecordingsBucketConfig Configuration for storing Daily recordings in a custom S3 bucket. ​ bucket_name string required Name of the S3 bucket for storing recordings ​ bucket_region string required AWS region where the S3 bucket is located ​ assume_role_arn string required ARN of the IAM role to assume for S3 access ​ allow_api_access boolean default: false Whether to allow API access to the recordings Copy Ask AI from pipecat.transports.services.helpers.daily_rest import RecordingsBucketConfig bucket_config = RecordingsBucketConfig( bucket_name = "my-recordings-bucket" , bucket_region = "us-west-2" , assume_role_arn = "arn:aws:iam::123456789012:role/DailyRecordingsRole" , allow_api_access = True ) ​ DailyRoomProperties Properties that configure a Daily room’s behavior and features. ​ exp float Room expiration time as Unix timestamp (e.g., time.time() + 300 for 5 minutes) ​ enable_chat boolean default: false Whether chat is enabled in the room ​ enable_prejoin_ui boolean default: false Whether the prejoin lobby UI is enabled ​ enable_emoji_reactions boolean default: false Whether emoji reactions are enabled ​ eject_at_room_exp boolean default: false Whether to eject participants when room expires ​ enable_dialout boolean Whether dial-out is enabled ​ enable_recording string Recording settings (“cloud”, “local”, or “raw-tracks”) ​ geo string Geographic region for room ​ max_participants number Maximum number of participants allowed in the room ​ recordings_bucket RecordingsBucketConfig Configuration for custom S3 bucket recordings ​ sip DailyRoomSipParams SIP configuration parameters ​ sip_uri dict SIP URI configuration (returned by Daily) ​ start_video_off boolean default: false Whether the camera video is turned off by default The class also includes a sip_endpoint property that returns the SIP endpoint URI if available. Copy Ask AI import time from pipecat.transports.services.helpers.daily_rest import ( DailyRoomProperties, DailyRoomSipParams, RecordingsBucketConfig, ) properties = DailyRoomProperties( exp = time.time() + 3600 , # 1 hour from now enable_chat = True , enable_emoji_reactions = True , enable_recording = "cloud" , geo = "us-west" , max_participants = 50 , sip = DailyRoomSipParams( display_name = "conference" ), recordings_bucket = RecordingsBucketConfig( bucket_name = "my-bucket" , bucket_region = "us-west-2" , assume_role_arn = "arn:aws:iam::123456789012:role/DailyRole" ) ) # Access SIP endpoint if available if properties.sip_endpoint: print ( f "SIP endpoint: { properties.sip_endpoint } " ) ​ DailyRoomParams Parameters for creating a new Daily room. ​ name string Room name (if not provided, one will be generated) ​ privacy string default: "public" Room privacy setting (“private” or “public”) ​ properties DailyRoomProperties Room configuration properties Copy Ask AI import time from pipecat.transports.services.helpers.daily_rest import ( DailyRoomParams, DailyRoomProperties, ) params = DailyRoomParams( name = "team-meeting" , privacy = "private" , properties = DailyRoomProperties( enable_chat = True , exp = time.time() + 7200 # 2 hours from now ) ) ​ DailyRoomObject Response object representing a Daily room. ​ id string Unique room identifier ​ name string Room name ​ api_created boolean Whether the room was created via API ​ privacy string Room privacy setting ​ url string Complete room URL ​ created_at string Room creation timestamp in ISO 8601 format ​ config DailyRoomProperties Room configuration Copy Ask AI from pipecat.transports.services.helpers.daily_rest import ( DailyRoomObject, DailyRoomProperties, ) # Example of what a DailyRoomObject looks like when received room = DailyRoomObject( id = "abc123" , name = "team-meeting" , api_created = True , privacy = "private" , url = "https://your-domain.daily.co/team-meeting" , created_at = "2024-01-20T10:00:00.000Z" , config = DailyRoomProperties( enable_chat = True , exp = 1705743600 ) ) ​ DailyMeetingTokenProperties Properties for configuring a Daily meeting token. ​ room_name string The room this token is valid for. If not set, token is valid for all rooms. ​ eject_at_token_exp boolean Whether to eject user when token expires ​ eject_after_elapsed integer Eject user after this many seconds ​ nbf integer “Not before” timestamp - users cannot join before this time ​ exp integer Expiration timestamp - users cannot join after this time ​ is_owner boolean Whether token grants owner privileges ​ user_name string User’s display name in the meeting ​ user_id string Unique identifier for the user (36 char limit) ​ enable_screenshare boolean Whether user can share their screen ​ start_video_off boolean Whether to join with video off ​ start_audio_off boolean Whether to join with audio off ​ enable_recording string Recording settings (“cloud”, “local”, or “raw-tracks”) ​ enable_prejoin_ui boolean Whether to show prejoin UI ​ start_cloud_recording boolean Whether to start cloud recording when user joins ​ permissions dict Initial default permissions for a non-meeting-owner participant ​ DailyMeetingTokenParams Parameters for creating a Daily meeting token. ​ properties DailyMeetingTokenProperties Token configuration properties Copy Ask AI from pipecat.transports.services.helpers.daily_rest import ( DailyMeetingTokenParams, DailyMeetingTokenProperties, ) token_params = DailyMeetingTokenParams( properties = DailyMeetingTokenProperties( user_name = "John Doe" , enable_screenshare = True , start_video_off = True , permissions = { "canSend" : [ "video" , "audio" ]} ) ) ​ Initialize DailyRESTHelper Create a new instance of the Daily REST helper. ​ daily_api_key string required Your Daily API key ​ daily_api_url string default: "https://api.daily.co/v1" The Daily API base URL ​ aiohttp_session aiohttp.ClientSession required An aiohttp client session for making HTTP requests Copy Ask AI helper = DailyRESTHelper( daily_api_key = "your-api-key" , aiohttp_session = session ) ​ Create Room Creates a new Daily room with specified parameters. ​ params DailyRoomParams required Room configuration parameters including name, privacy, and properties Copy Ask AI # Create a room that expires in 1 hour params = DailyRoomParams( name = "my-room" , privacy = "private" , properties = DailyRoomProperties( exp = time.time() + 3600 , enable_chat = True ) ) room = await helper.create_room(params) print ( f "Room URL: { room.url } " ) ​ Get Room From URL Retrieves room information using a Daily room URL. ​ room_url string required The complete Daily room URL Copy Ask AI room = await helper.get_room_from_url( "https://your-domain.daily.co/my-room" ) print ( f "Room name: { room.name } " ) ​ Get Token Generates a meeting token for a specific room. ​ room_url string required The complete Daily room URL ​ expiry_time float default: "3600" Token expiration time in seconds ​ eject_at_token_exp bool default: "False" Whether to eject user when token expires ​ owner bool default: "True" Whether the token should have owner privileges (overrides any setting in params) ​ params DailyMeetingTokenParams Additional token configuration. Note that room_name , exp , eject_at_token_exp , and is_owner will be set based on the other function parameters. Copy Ask AI # Basic token generation token = await helper.get_token( room_url = "https://your-domain.daily.co/my-room" , expiry_time = 1800 , # 30 minutes owner = True , eject_at_token_exp = True ) # Advanced token generation with additional properties token_params = DailyMeetingTokenParams( properties = DailyMeetingTokenProperties( user_name = "John Doe" , start_video_off = True ) ) token = await helper.get_token( room_url = "https://your-domain.daily.co/my-room" , expiry_time = 1800 , owner = False , eject_at_token_exp = True , params = token_params ) ​ Delete Room By URL Deletes a room using its URL. ​ room_url string required The complete Daily room URL Copy Ask AI success = await helper.delete_room_by_url( "https://your-domain.daily.co/my-room" ) if success: print ( "Room deleted successfully" ) ​ Delete Room By Name Deletes a room using its name. ​ room_name string required The name of the Daily room Copy Ask AI success = await helper.delete_room_by_name( "my-room" ) if success: print ( "Room deleted successfully" ) ​ Get Name From URL Extracts the room name from a Daily room URL. ​ room_url string required The complete Daily room URL Copy Ask AI room_name = helper.get_name_from_url( "https://your-domain.daily.co/my-room" ) print ( f "Room name: { room_name } " ) # Outputs: "my-room" Turn Tracking Observer Smart Turn Overview On this page Classes DailyRoomSipParams RecordingsBucketConfig DailyRoomProperties DailyRoomParams DailyRoomObject DailyMeetingTokenProperties DailyMeetingTokenParams Initialize DailyRESTHelper Create Room Get Room From URL Get Token Delete Room By URL Delete Room By Name Get Name From URL Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/daily_rest-helpers_f29284ae.txt b/daily_rest-helpers_f29284ae.txt
new file mode 100644
index 0000000000000000000000000000000000000000..f76225d382c6958967f3bdd71d35601e84c65a16
--- /dev/null
+++ b/daily_rest-helpers_f29284ae.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/daily/rest-helpers#delete-room-by-name
+Title: Daily REST Helper - Pipecat
+==================================================
+
+Daily REST Helper - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Service Utilities Daily REST Helper Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Daily REST Helper Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline Daily REST API Documentation For complete Daily REST API reference and additional details ​ Classes ​ DailyRoomSipParams Configuration for SIP (Session Initiation Protocol) parameters. ​ display_name string default: "sw-sip-dialin" Display name for the SIP endpoint ​ video boolean default: false Whether video is enabled for SIP ​ sip_mode string default: "dial-in" SIP connection mode ​ num_endpoints integer default: 1 Number of SIP endpoints Copy Ask AI from pipecat.transports.services.helpers.daily_rest import DailyRoomSipParams sip_params = DailyRoomSipParams( display_name = "conference-line" , video = True , num_endpoints = 2 ) ​ RecordingsBucketConfig Configuration for storing Daily recordings in a custom S3 bucket. ​ bucket_name string required Name of the S3 bucket for storing recordings ​ bucket_region string required AWS region where the S3 bucket is located ​ assume_role_arn string required ARN of the IAM role to assume for S3 access ​ allow_api_access boolean default: false Whether to allow API access to the recordings Copy Ask AI from pipecat.transports.services.helpers.daily_rest import RecordingsBucketConfig bucket_config = RecordingsBucketConfig( bucket_name = "my-recordings-bucket" , bucket_region = "us-west-2" , assume_role_arn = "arn:aws:iam::123456789012:role/DailyRecordingsRole" , allow_api_access = True ) ​ DailyRoomProperties Properties that configure a Daily room’s behavior and features. ​ exp float Room expiration time as Unix timestamp (e.g., time.time() + 300 for 5 minutes) ​ enable_chat boolean default: false Whether chat is enabled in the room ​ enable_prejoin_ui boolean default: false Whether the prejoin lobby UI is enabled ​ enable_emoji_reactions boolean default: false Whether emoji reactions are enabled ​ eject_at_room_exp boolean default: false Whether to eject participants when room expires ​ enable_dialout boolean Whether dial-out is enabled ​ enable_recording string Recording settings (“cloud”, “local”, or “raw-tracks”) ​ geo string Geographic region for room ​ max_participants number Maximum number of participants allowed in the room ​ recordings_bucket RecordingsBucketConfig Configuration for custom S3 bucket recordings ​ sip DailyRoomSipParams SIP configuration parameters ​ sip_uri dict SIP URI configuration (returned by Daily) ​ start_video_off boolean default: false Whether the camera video is turned off by default The class also includes a sip_endpoint property that returns the SIP endpoint URI if available. Copy Ask AI import time from pipecat.transports.services.helpers.daily_rest import ( DailyRoomProperties, DailyRoomSipParams, RecordingsBucketConfig, ) properties = DailyRoomProperties( exp = time.time() + 3600 , # 1 hour from now enable_chat = True , enable_emoji_reactions = True , enable_recording = "cloud" , geo = "us-west" , max_participants = 50 , sip = DailyRoomSipParams( display_name = "conference" ), recordings_bucket = RecordingsBucketConfig( bucket_name = "my-bucket" , bucket_region = "us-west-2" , assume_role_arn = "arn:aws:iam::123456789012:role/DailyRole" ) ) # Access SIP endpoint if available if properties.sip_endpoint: print ( f "SIP endpoint: { properties.sip_endpoint } " ) ​ DailyRoomParams Parameters for creating a new Daily room. ​ name string Room name (if not provided, one will be generated) ​ privacy string default: "public" Room privacy setting (“private” or “public”) ​ properties DailyRoomProperties Room configuration properties Copy Ask AI import time from pipecat.transports.services.helpers.daily_rest import ( DailyRoomParams, DailyRoomProperties, ) params = DailyRoomParams( name = "team-meeting" , privacy = "private" , properties = DailyRoomProperties( enable_chat = True , exp = time.time() + 7200 # 2 hours from now ) ) ​ DailyRoomObject Response object representing a Daily room. ​ id string Unique room identifier ​ name string Room name ​ api_created boolean Whether the room was created via API ​ privacy string Room privacy setting ​ url string Complete room URL ​ created_at string Room creation timestamp in ISO 8601 format ​ config DailyRoomProperties Room configuration Copy Ask AI from pipecat.transports.services.helpers.daily_rest import ( DailyRoomObject, DailyRoomProperties, ) # Example of what a DailyRoomObject looks like when received room = DailyRoomObject( id = "abc123" , name = "team-meeting" , api_created = True , privacy = "private" , url = "https://your-domain.daily.co/team-meeting" , created_at = "2024-01-20T10:00:00.000Z" , config = DailyRoomProperties( enable_chat = True , exp = 1705743600 ) ) ​ DailyMeetingTokenProperties Properties for configuring a Daily meeting token. ​ room_name string The room this token is valid for. If not set, token is valid for all rooms. ​ eject_at_token_exp boolean Whether to eject user when token expires ​ eject_after_elapsed integer Eject user after this many seconds ​ nbf integer “Not before” timestamp - users cannot join before this time ​ exp integer Expiration timestamp - users cannot join after this time ​ is_owner boolean Whether token grants owner privileges ​ user_name string User’s display name in the meeting ​ user_id string Unique identifier for the user (36 char limit) ​ enable_screenshare boolean Whether user can share their screen ​ start_video_off boolean Whether to join with video off ​ start_audio_off boolean Whether to join with audio off ​ enable_recording string Recording settings (“cloud”, “local”, or “raw-tracks”) ​ enable_prejoin_ui boolean Whether to show prejoin UI ​ start_cloud_recording boolean Whether to start cloud recording when user joins ​ permissions dict Initial default permissions for a non-meeting-owner participant ​ DailyMeetingTokenParams Parameters for creating a Daily meeting token. ​ properties DailyMeetingTokenProperties Token configuration properties Copy Ask AI from pipecat.transports.services.helpers.daily_rest import ( DailyMeetingTokenParams, DailyMeetingTokenProperties, ) token_params = DailyMeetingTokenParams( properties = DailyMeetingTokenProperties( user_name = "John Doe" , enable_screenshare = True , start_video_off = True , permissions = { "canSend" : [ "video" , "audio" ]} ) ) ​ Initialize DailyRESTHelper Create a new instance of the Daily REST helper. ​ daily_api_key string required Your Daily API key ​ daily_api_url string default: "https://api.daily.co/v1" The Daily API base URL ​ aiohttp_session aiohttp.ClientSession required An aiohttp client session for making HTTP requests Copy Ask AI helper = DailyRESTHelper( daily_api_key = "your-api-key" , aiohttp_session = session ) ​ Create Room Creates a new Daily room with specified parameters. ​ params DailyRoomParams required Room configuration parameters including name, privacy, and properties Copy Ask AI # Create a room that expires in 1 hour params = DailyRoomParams( name = "my-room" , privacy = "private" , properties = DailyRoomProperties( exp = time.time() + 3600 , enable_chat = True ) ) room = await helper.create_room(params) print ( f "Room URL: { room.url } " ) ​ Get Room From URL Retrieves room information using a Daily room URL. ​ room_url string required The complete Daily room URL Copy Ask AI room = await helper.get_room_from_url( "https://your-domain.daily.co/my-room" ) print ( f "Room name: { room.name } " ) ​ Get Token Generates a meeting token for a specific room. ​ room_url string required The complete Daily room URL ​ expiry_time float default: "3600" Token expiration time in seconds ​ eject_at_token_exp bool default: "False" Whether to eject user when token expires ​ owner bool default: "True" Whether the token should have owner privileges (overrides any setting in params) ​ params DailyMeetingTokenParams Additional token configuration. Note that room_name , exp , eject_at_token_exp , and is_owner will be set based on the other function parameters. Copy Ask AI # Basic token generation token = await helper.get_token( room_url = "https://your-domain.daily.co/my-room" , expiry_time = 1800 , # 30 minutes owner = True , eject_at_token_exp = True ) # Advanced token generation with additional properties token_params = DailyMeetingTokenParams( properties = DailyMeetingTokenProperties( user_name = "John Doe" , start_video_off = True ) ) token = await helper.get_token( room_url = "https://your-domain.daily.co/my-room" , expiry_time = 1800 , owner = False , eject_at_token_exp = True , params = token_params ) ​ Delete Room By URL Deletes a room using its URL. ​ room_url string required The complete Daily room URL Copy Ask AI success = await helper.delete_room_by_url( "https://your-domain.daily.co/my-room" ) if success: print ( "Room deleted successfully" ) ​ Delete Room By Name Deletes a room using its name. ​ room_name string required The name of the Daily room Copy Ask AI success = await helper.delete_room_by_name( "my-room" ) if success: print ( "Room deleted successfully" ) ​ Get Name From URL Extracts the room name from a Daily room URL. ​ room_url string required The complete Daily room URL Copy Ask AI room_name = helper.get_name_from_url( "https://your-domain.daily.co/my-room" ) print ( f "Room name: { room_name } " ) # Outputs: "my-room" Turn Tracking Observer Smart Turn Overview On this page Classes DailyRoomSipParams RecordingsBucketConfig DailyRoomProperties DailyRoomParams DailyRoomObject DailyMeetingTokenProperties DailyMeetingTokenParams Initialize DailyRESTHelper Create Room Get Room From URL Get Token Delete Room By URL Delete Room By Name Get Name From URL Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/daily_rest-helpers_fd2f43e9.txt b/daily_rest-helpers_fd2f43e9.txt
new file mode 100644
index 0000000000000000000000000000000000000000..d5d5aba4861192d11160a0b7f76b2dd3b2c796a2
--- /dev/null
+++ b/daily_rest-helpers_fd2f43e9.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/daily/rest-helpers#param-sip
+Title: Daily REST Helper - Pipecat
+==================================================
+
+Daily REST Helper - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Service Utilities Daily REST Helper Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Daily REST Helper Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline Daily REST API Documentation For complete Daily REST API reference and additional details ​ Classes ​ DailyRoomSipParams Configuration for SIP (Session Initiation Protocol) parameters. ​ display_name string default: "sw-sip-dialin" Display name for the SIP endpoint ​ video boolean default: false Whether video is enabled for SIP ​ sip_mode string default: "dial-in" SIP connection mode ​ num_endpoints integer default: 1 Number of SIP endpoints Copy Ask AI from pipecat.transports.services.helpers.daily_rest import DailyRoomSipParams sip_params = DailyRoomSipParams( display_name = "conference-line" , video = True , num_endpoints = 2 ) ​ RecordingsBucketConfig Configuration for storing Daily recordings in a custom S3 bucket. ​ bucket_name string required Name of the S3 bucket for storing recordings ​ bucket_region string required AWS region where the S3 bucket is located ​ assume_role_arn string required ARN of the IAM role to assume for S3 access ​ allow_api_access boolean default: false Whether to allow API access to the recordings Copy Ask AI from pipecat.transports.services.helpers.daily_rest import RecordingsBucketConfig bucket_config = RecordingsBucketConfig( bucket_name = "my-recordings-bucket" , bucket_region = "us-west-2" , assume_role_arn = "arn:aws:iam::123456789012:role/DailyRecordingsRole" , allow_api_access = True ) ​ DailyRoomProperties Properties that configure a Daily room’s behavior and features. ​ exp float Room expiration time as Unix timestamp (e.g., time.time() + 300 for 5 minutes) ​ enable_chat boolean default: false Whether chat is enabled in the room ​ enable_prejoin_ui boolean default: false Whether the prejoin lobby UI is enabled ​ enable_emoji_reactions boolean default: false Whether emoji reactions are enabled ​ eject_at_room_exp boolean default: false Whether to eject participants when room expires ​ enable_dialout boolean Whether dial-out is enabled ​ enable_recording string Recording settings (“cloud”, “local”, or “raw-tracks”) ​ geo string Geographic region for room ​ max_participants number Maximum number of participants allowed in the room ​ recordings_bucket RecordingsBucketConfig Configuration for custom S3 bucket recordings ​ sip DailyRoomSipParams SIP configuration parameters ​ sip_uri dict SIP URI configuration (returned by Daily) ​ start_video_off boolean default: false Whether the camera video is turned off by default The class also includes a sip_endpoint property that returns the SIP endpoint URI if available. Copy Ask AI import time from pipecat.transports.services.helpers.daily_rest import ( DailyRoomProperties, DailyRoomSipParams, RecordingsBucketConfig, ) properties = DailyRoomProperties( exp = time.time() + 3600 , # 1 hour from now enable_chat = True , enable_emoji_reactions = True , enable_recording = "cloud" , geo = "us-west" , max_participants = 50 , sip = DailyRoomSipParams( display_name = "conference" ), recordings_bucket = RecordingsBucketConfig( bucket_name = "my-bucket" , bucket_region = "us-west-2" , assume_role_arn = "arn:aws:iam::123456789012:role/DailyRole" ) ) # Access SIP endpoint if available if properties.sip_endpoint: print ( f "SIP endpoint: { properties.sip_endpoint } " ) ​ DailyRoomParams Parameters for creating a new Daily room. ​ name string Room name (if not provided, one will be generated) ​ privacy string default: "public" Room privacy setting (“private” or “public”) ​ properties DailyRoomProperties Room configuration properties Copy Ask AI import time from pipecat.transports.services.helpers.daily_rest import ( DailyRoomParams, DailyRoomProperties, ) params = DailyRoomParams( name = "team-meeting" , privacy = "private" , properties = DailyRoomProperties( enable_chat = True , exp = time.time() + 7200 # 2 hours from now ) ) ​ DailyRoomObject Response object representing a Daily room. ​ id string Unique room identifier ​ name string Room name ​ api_created boolean Whether the room was created via API ​ privacy string Room privacy setting ​ url string Complete room URL ​ created_at string Room creation timestamp in ISO 8601 format ​ config DailyRoomProperties Room configuration Copy Ask AI from pipecat.transports.services.helpers.daily_rest import ( DailyRoomObject, DailyRoomProperties, ) # Example of what a DailyRoomObject looks like when received room = DailyRoomObject( id = "abc123" , name = "team-meeting" , api_created = True , privacy = "private" , url = "https://your-domain.daily.co/team-meeting" , created_at = "2024-01-20T10:00:00.000Z" , config = DailyRoomProperties( enable_chat = True , exp = 1705743600 ) ) ​ DailyMeetingTokenProperties Properties for configuring a Daily meeting token. ​ room_name string The room this token is valid for. If not set, token is valid for all rooms. ​ eject_at_token_exp boolean Whether to eject user when token expires ​ eject_after_elapsed integer Eject user after this many seconds ​ nbf integer “Not before” timestamp - users cannot join before this time ​ exp integer Expiration timestamp - users cannot join after this time ​ is_owner boolean Whether token grants owner privileges ​ user_name string User’s display name in the meeting ​ user_id string Unique identifier for the user (36 char limit) ​ enable_screenshare boolean Whether user can share their screen ​ start_video_off boolean Whether to join with video off ​ start_audio_off boolean Whether to join with audio off ​ enable_recording string Recording settings (“cloud”, “local”, or “raw-tracks”) ​ enable_prejoin_ui boolean Whether to show prejoin UI ​ start_cloud_recording boolean Whether to start cloud recording when user joins ​ permissions dict Initial default permissions for a non-meeting-owner participant ​ DailyMeetingTokenParams Parameters for creating a Daily meeting token. ​ properties DailyMeetingTokenProperties Token configuration properties Copy Ask AI from pipecat.transports.services.helpers.daily_rest import ( DailyMeetingTokenParams, DailyMeetingTokenProperties, ) token_params = DailyMeetingTokenParams( properties = DailyMeetingTokenProperties( user_name = "John Doe" , enable_screenshare = True , start_video_off = True , permissions = { "canSend" : [ "video" , "audio" ]} ) ) ​ Initialize DailyRESTHelper Create a new instance of the Daily REST helper. ​ daily_api_key string required Your Daily API key ​ daily_api_url string default: "https://api.daily.co/v1" The Daily API base URL ​ aiohttp_session aiohttp.ClientSession required An aiohttp client session for making HTTP requests Copy Ask AI helper = DailyRESTHelper( daily_api_key = "your-api-key" , aiohttp_session = session ) ​ Create Room Creates a new Daily room with specified parameters. ​ params DailyRoomParams required Room configuration parameters including name, privacy, and properties Copy Ask AI # Create a room that expires in 1 hour params = DailyRoomParams( name = "my-room" , privacy = "private" , properties = DailyRoomProperties( exp = time.time() + 3600 , enable_chat = True ) ) room = await helper.create_room(params) print ( f "Room URL: { room.url } " ) ​ Get Room From URL Retrieves room information using a Daily room URL. ​ room_url string required The complete Daily room URL Copy Ask AI room = await helper.get_room_from_url( "https://your-domain.daily.co/my-room" ) print ( f "Room name: { room.name } " ) ​ Get Token Generates a meeting token for a specific room. ​ room_url string required The complete Daily room URL ​ expiry_time float default: "3600" Token expiration time in seconds ​ eject_at_token_exp bool default: "False" Whether to eject user when token expires ​ owner bool default: "True" Whether the token should have owner privileges (overrides any setting in params) ​ params DailyMeetingTokenParams Additional token configuration. Note that room_name , exp , eject_at_token_exp , and is_owner will be set based on the other function parameters. Copy Ask AI # Basic token generation token = await helper.get_token( room_url = "https://your-domain.daily.co/my-room" , expiry_time = 1800 , # 30 minutes owner = True , eject_at_token_exp = True ) # Advanced token generation with additional properties token_params = DailyMeetingTokenParams( properties = DailyMeetingTokenProperties( user_name = "John Doe" , start_video_off = True ) ) token = await helper.get_token( room_url = "https://your-domain.daily.co/my-room" , expiry_time = 1800 , owner = False , eject_at_token_exp = True , params = token_params ) ​ Delete Room By URL Deletes a room using its URL. ​ room_url string required The complete Daily room URL Copy Ask AI success = await helper.delete_room_by_url( "https://your-domain.daily.co/my-room" ) if success: print ( "Room deleted successfully" ) ​ Delete Room By Name Deletes a room using its name. ​ room_name string required The name of the Daily room Copy Ask AI success = await helper.delete_room_by_name( "my-room" ) if success: print ( "Room deleted successfully" ) ​ Get Name From URL Extracts the room name from a Daily room URL. ​ room_url string required The complete Daily room URL Copy Ask AI room_name = helper.get_name_from_url( "https://your-domain.daily.co/my-room" ) print ( f "Room name: { room_name } " ) # Outputs: "my-room" Turn Tracking Observer Smart Turn Overview On this page Classes DailyRoomSipParams RecordingsBucketConfig DailyRoomProperties DailyRoomParams DailyRoomObject DailyMeetingTokenProperties DailyMeetingTokenParams Initialize DailyRESTHelper Create Room Get Room From URL Get Token Delete Room By URL Delete Room By Name Get Name From URL Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/deployment_cerebrium_0f0cbad7.txt b/deployment_cerebrium_0f0cbad7.txt
new file mode 100644
index 0000000000000000000000000000000000000000..1a3e7e72674c2b957459b7cd2dd4e2afd54c7fbc
--- /dev/null
+++ b/deployment_cerebrium_0f0cbad7.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/guides/deployment/cerebrium#agent-setup
+Title: Example: Cerebrium - Pipecat
+==================================================
+
+Example: Cerebrium - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Deploying your bot Example: Cerebrium Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Guides Fundamentals Context Management Custom FrameProcessor Detecting Idle Users Ending a Pipeline Function Calling Muting User Input Recording Audio Recording Transcripts Features Gemini Multimodal Live Metrics Noise cancellation with Krisp OpenAI Audio Models and APIs Pipecat Flows Telephony Overview Dial-in: WebRTC (Daily) Dial-in: WebRTC (Twilio + Daily) Dial-in: Twilio (Media Streams) Dialout: WebRTC (Daily) Deploying your bot Overview Deployment pattern Example: Pipecat Cloud Example: Fly.io Example: Cerebrium Example: Modal Cerebrium is a serverless Infrastructure platform that makes it easy for companies to build, deploy and scale AI applications. Cerebrium offers both CPUs and GPUs (H100s, A100s etc) with extremely low cold start times allowing us to create highly performant applications in the most cost efficient manner. ​ Install the Cerebrium CLI To get started, let us run the following commands: Run pip install cerebrium to install the Python package. Run cerebrium login to authenticate yourself. If you don’t have a Cerebrium account, you can create one and get started with $30 in free credits. ​ Create a Cerebrium project Create a new Cerebrium project: Copy Ask AI cerebrium init pipecat-agent This will create two key files: main.py - Your application entrypoint cerebrium.toml - Configuration for build and environment settings Update your cerebrium.toml with the necessary configuration: Copy Ask AI [ cerebrium . hardware ] region = "us-east-1" provider = "aws" compute = "CPU" cpu = 4 memory = 18.0 [ cerebrium . dependencies . pip ] torch = ">=2.0.0" " pipecat-ai[silero, daily, openai, cartesia] " = "latest" aiohttp = "latest" torchaudio = "latest" In order for our application to work, we need to copy our API keys from the various platforms. Navigate to the Secrets section in your Cerebrium dashboard to store your API keys: OPENAI_API_KEY - We use OpenAI For the LLM. You can get your API key from here DAILY_TOKEN - For WebRTC communication. You can get your token from here CARTERSIA_API_KEY - For text-to-speech services. You can get your API key from here We access these secrets in our code as if they are normal ENV vars. For the above, You can swap in any LLM or TTS service you wish to use. ​ Agent setup We create a basic pipeline setup in our main.py that combines our LLM, TTS and Daily WebRTC transport layer. Copy Ask AI import os import sys import time import aiohttp from loguru import logger from pipecat.frames.frames import EndFrame from pipecat.pipeline.pipeline import Pipeline from pipecat.pipeline.runner import PipelineRunner from pipecat.pipeline.task import PipelineParams, PipelineTask from pipecat.services.openai.llm import OpenAILLMService from pipecat.processors.aggregators.openai_llm_context import ( OpenAILLMContext, ) from pipecat.services.cartesia.tts import CartesiaTTSService from pipecat.transports.services.daily import DailyParams, DailyTransport from pipecat.vad.silero import SileroVADAnalyzer from pipecat.vad.vad_analyzer import VADParams logger.remove( 0 ) logger.add(sys.stderr, level = "DEBUG" ) async def main ( room_url : str , token : str ): async with aiohttp.ClientSession() as session: transport = DailyTransport( room_url, token, "Friendly bot" , DailyParams( audio_in_enabled = True , audio_out_enabled = True , transcription_enabled = True , vad_analyzer = SileroVADAnalyzer( params = VADParams( stop_secs = 0.6 )), ), ) messages = [ { "role" : "system" , "content" : "You are a helpful AI assistant that can switch between two services to showcase the difference in performance and cost: 'openai_realtime' and 'custom'. Respond to user queries and switch services when asked." , }, ] llm = OpenAILLMService( name = "LLM" , api_key = os.environ.get( "OPENAI_API_KEY" ), model = "gpt-4" , ) tts = CartesiaTTSService( api_key = os.getenv( "CARTESIA_API_KEY" ), voice_id = "79a125e8-cd45-4c13-8a67-188112f4dd22" , # British Lady ) custom_context = OpenAILLMContext( messages = messages) context_aggregator_custom = llm.create_context_aggregator(custom_context) pipeline = Pipeline( [ transport.input(), # Transport user input context_aggregator_custom.user(), llm, tts, context_aggregator_custom.assistant(), transport.output(), # Transport bot output ] ) task = PipelineTask( pipeline, params = PipelineParams( allow_interruptions = True , enable_metrics = True , enable_usage_metrics = True , ), ) @transport.event_handler ( "on_first_participant_joined" ) async def on_first_participant_joined ( transport , participant ): transport.capture_participant_transcription(participant[ "id" ]) time.sleep( 1.5 ) messages.append( { "role" : "system" , "content" : "Introduce yourself." , } ) await task.queue_frame(LLMMessagesFrame(messages)) @transport.event_handler ( "on_participant_left" ) async def on_participant_left ( transport , participant , reason ): await task.queue_frame(EndFrame()) @transport.event_handler ( "on_call_state_updated" ) async def on_call_state_updated ( transport , state ): if state == "left" : await task.queue_frame(EndFrame()) runner = PipelineRunner() await runner.run(task) await session.close() First, in our main function, we initialize the daily transport layer to receive/send the audio/video data from the Daily room we will connect to. You can see we pass the room_url we would like to join as well as a token to authenticate us programmatically joining. We also set our VAD stop seconds which is the amount of time we wait for a pause before our bot will respond - in this example, we set it to 600 milliseconds. Next we connect to our LLM (OpenAI) as well as our TTS model (Cartesia). By setting ‘transcription_enabled=true’ we are using the STT from Daily itself. This is where the Pipecat framework helps convert audio data to text and vice versa. We then put this all together as a PipelineTask which is what Pipecat runs all together. The make up of a task is completely customizable and has support for Image and Vision use cases. Lastly, we have some event handlers for when a user joins/leaves the room. ​ Deploy bot Deploy your application to Cerebrium: Copy Ask AI cerebrium deploy You will then see that an endpoint is created for your bot at POST \<BASE_URL\>/main that you can call with your room_url and token. Let us test it. ​ Test it out Copy Ask AI def create_room (): url = "https://api.daily.co/v1/rooms/" headers = { "Content-Type" : "application/json" , "Authorization" : f "Bearer { os.environ.get( 'DAILY_TOKEN' ) } " , } data = { "properties" : { "exp" : int (time.time()) + 60 * 5 , ##5 mins "eject_at_room_exp" : True , } } response = requests.post(url, headers = headers, json = data) if response.status_code == 200 : room_info = response.json() token = create_token(room_info[ "name" ]) if token and "token" in token: room_info[ "token" ] = token[ "token" ] else : logger.error( "Failed to create token" ) return { "message" : "There was an error creating your room" , "status_code" : 500 , } return room_info else : logger.error( f "Failed to create room: { response.status_code } " ) return { "message" : "There was an error creating your room" , "status_code" : 500 } def create_token ( room_name : str ): url = "https://api.daily.co/v1/meeting-tokens" headers = { "Content-Type" : "application/json" , "Authorization" : f "Bearer { os.environ.get( 'DAILY_TOKEN' ) } " , } data = { "properties" : { "room_name" : room_name, "is_owner" : True }} response = requests.post(url, headers = headers, json = data) if response.status_code == 200 : token_info = response.json() return token_info else : logger.error( f "Failed to create token: { response.status_code } " ) return None if __name__ == "__main__" : room_info = create_room() print ( f "Join room: { room_info[ "url" ] } " ) asyncio.run(main(room_info[ "url" ], room_info[ "token" ])) ​ Future Considerations Since Cerebrium supports both CPU and GPU workloads if you would like to lower the latency of your application then the best would be to get model weights from various providers and run them locally. You can do this for: LLM: Run any OpenSource model using a framework such as vLLM TTS: Both PlayHt and Deepgram offer TTS models that can be run locally STT: Deepgram offers a local model that can be run locally If you implement all three models locally, you should have much better performance. We have been able to get ~300ms voice-to-voice responses. ​ Examples Fastest voice agent : Local only implementation RAG voice agent : Create a voice agent that can do RAG using Cerebrium + OpenAI + Pinecone Twilio voice agent : Create a voice agent that can receive phone calls via Twilio OpenAI Realtime API implementation : Create a voice agent that can receive phone calls via OpenAI Realtime API Example: Fly.io Example: Modal On this page Install the Cerebrium CLI Create a Cerebrium project Agent setup Deploy bot Test it out Future Considerations Examples Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/deployment_cerebrium_1a6deaf3.txt b/deployment_cerebrium_1a6deaf3.txt
new file mode 100644
index 0000000000000000000000000000000000000000..5bb84a113a2d62440a6fe7b7f487fc3376e311c3
--- /dev/null
+++ b/deployment_cerebrium_1a6deaf3.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/deployment/cerebrium#what-you-can-build
+Title: Overview - Pipecat
+==================================================
+
+Overview - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Get Started Overview Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Get Started Overview Installation & Setup Quickstart Core Concepts Next Steps & Examples Pipecat is an open source Python framework that handles the complex orchestration of AI services, network transport, audio processing, and multimodal interactions. “Multimodal” means you can use any combination of audio, video, images, and/or text in your interactions. And “real-time” means that things are happening quickly enough that it feels conversational—a “back-and-forth” with a bot, not submitting a query and waiting for results. ​ What You Can Build Voice Assistants Natural, real-time conversations with AI using speech recognition and synthesis Interactive Agents Personal coaches and meeting assistants that can understand context and provide guidance Multimodal Apps Applications that combine voice, video, images, and text for rich interactions Creative Tools Storytelling experiences and social companions that engage users Business Solutions Customer intake flows and support bots for automated business processes Complex Flows Structured conversations using Pipecat Flows for managing complex interactions ​ How It Works The flow of interactions in a Pipecat application is typically straightforward: The bot says something The user says something The bot says something The user says something This continues until the conversation naturally ends. While this flow seems simple, making it feel natural requires sophisticated real-time processing. ​ Real-time Processing Pipecat’s pipeline architecture handles both simple voice interactions and complex multimodal processing. Let’s look at how data flows through the system: Voice app Multimodal app 1 Send Audio Transmit and capture streamed audio from the user 2 Transcribe Speech Convert speech to text as the user is talking 3 Process with LLM Generate responses using a large language model 4 Convert to Speech Transform text responses into natural speech 5 Play Audio Stream the audio response back to the user 1 Send Audio Transmit and capture streamed audio from the user 2 Transcribe Speech Convert speech to text as the user is talking 3 Process with LLM Generate responses using a large language model 4 Convert to Speech Transform text responses into natural speech 5 Play Audio Stream the audio response back to the user 1 Send Audio and Video Transmit and capture audio, video, and image inputs simultaneously 2 Process Streams Handle multiple input streams in parallel 3 Model Processing Send combined inputs to multimodal models (like GPT-4V) 4 Generate Outputs Create various outputs (text, images, audio, etc.) 5 Coordinate Presentation Synchronize and present multiple output types In both cases, Pipecat: Processes responses as they stream in Handles multiple input/output modalities concurrently Manages resource allocation and synchronization Coordinates parallel processing tasks This architecture creates fluid, natural interactions without noticeable delays, whether you’re building a simple voice assistant or a complex multimodal application. Pipecat’s pipeline architecture is particularly valuable for managing the complexity of real-time, multimodal interactions, ensuring smooth data flow and proper synchronization regardless of the input/output types involved. Pipecat handles all this complexity for you, letting you focus on building your application rather than managing the underlying infrastructure. ​ Next Steps Ready to build your first Pipecat application? Installation & Setup Prepare your environment and install required dependencies Quickstart Build and run your first Pipecat application Core Concepts Learn about pipelines, frames, and real-time processing Use Cases Explore example implementations and patterns ​ Join Our Community Discord Community Connect with other developers, share your projects, and get support from the Pipecat team. Installation & Setup On this page What You Can Build How It Works Real-time Processing Next Steps Join Our Community Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/deployment_cerebrium_45f86715.txt b/deployment_cerebrium_45f86715.txt
new file mode 100644
index 0000000000000000000000000000000000000000..10ce00fa5869a7996dea16589446cc5c5ac8d922
--- /dev/null
+++ b/deployment_cerebrium_45f86715.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/guides/deployment/cerebrium#future-considerations
+Title: Example: Cerebrium - Pipecat
+==================================================
+
+Example: Cerebrium - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Deploying your bot Example: Cerebrium Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Guides Fundamentals Context Management Custom FrameProcessor Detecting Idle Users Ending a Pipeline Function Calling Muting User Input Recording Audio Recording Transcripts Features Gemini Multimodal Live Metrics Noise cancellation with Krisp OpenAI Audio Models and APIs Pipecat Flows Telephony Overview Dial-in: WebRTC (Daily) Dial-in: WebRTC (Twilio + Daily) Dial-in: Twilio (Media Streams) Dialout: WebRTC (Daily) Deploying your bot Overview Deployment pattern Example: Pipecat Cloud Example: Fly.io Example: Cerebrium Example: Modal Cerebrium is a serverless Infrastructure platform that makes it easy for companies to build, deploy and scale AI applications. Cerebrium offers both CPUs and GPUs (H100s, A100s etc) with extremely low cold start times allowing us to create highly performant applications in the most cost efficient manner. ​ Install the Cerebrium CLI To get started, let us run the following commands: Run pip install cerebrium to install the Python package. Run cerebrium login to authenticate yourself. If you don’t have a Cerebrium account, you can create one and get started with $30 in free credits. ​ Create a Cerebrium project Create a new Cerebrium project: Copy Ask AI cerebrium init pipecat-agent This will create two key files: main.py - Your application entrypoint cerebrium.toml - Configuration for build and environment settings Update your cerebrium.toml with the necessary configuration: Copy Ask AI [ cerebrium . hardware ] region = "us-east-1" provider = "aws" compute = "CPU" cpu = 4 memory = 18.0 [ cerebrium . dependencies . pip ] torch = ">=2.0.0" " pipecat-ai[silero, daily, openai, cartesia] " = "latest" aiohttp = "latest" torchaudio = "latest" In order for our application to work, we need to copy our API keys from the various platforms. Navigate to the Secrets section in your Cerebrium dashboard to store your API keys: OPENAI_API_KEY - We use OpenAI For the LLM. You can get your API key from here DAILY_TOKEN - For WebRTC communication. You can get your token from here CARTERSIA_API_KEY - For text-to-speech services. You can get your API key from here We access these secrets in our code as if they are normal ENV vars. For the above, You can swap in any LLM or TTS service you wish to use. ​ Agent setup We create a basic pipeline setup in our main.py that combines our LLM, TTS and Daily WebRTC transport layer. Copy Ask AI import os import sys import time import aiohttp from loguru import logger from pipecat.frames.frames import EndFrame from pipecat.pipeline.pipeline import Pipeline from pipecat.pipeline.runner import PipelineRunner from pipecat.pipeline.task import PipelineParams, PipelineTask from pipecat.services.openai.llm import OpenAILLMService from pipecat.processors.aggregators.openai_llm_context import ( OpenAILLMContext, ) from pipecat.services.cartesia.tts import CartesiaTTSService from pipecat.transports.services.daily import DailyParams, DailyTransport from pipecat.vad.silero import SileroVADAnalyzer from pipecat.vad.vad_analyzer import VADParams logger.remove( 0 ) logger.add(sys.stderr, level = "DEBUG" ) async def main ( room_url : str , token : str ): async with aiohttp.ClientSession() as session: transport = DailyTransport( room_url, token, "Friendly bot" , DailyParams( audio_in_enabled = True , audio_out_enabled = True , transcription_enabled = True , vad_analyzer = SileroVADAnalyzer( params = VADParams( stop_secs = 0.6 )), ), ) messages = [ { "role" : "system" , "content" : "You are a helpful AI assistant that can switch between two services to showcase the difference in performance and cost: 'openai_realtime' and 'custom'. Respond to user queries and switch services when asked." , }, ] llm = OpenAILLMService( name = "LLM" , api_key = os.environ.get( "OPENAI_API_KEY" ), model = "gpt-4" , ) tts = CartesiaTTSService( api_key = os.getenv( "CARTESIA_API_KEY" ), voice_id = "79a125e8-cd45-4c13-8a67-188112f4dd22" , # British Lady ) custom_context = OpenAILLMContext( messages = messages) context_aggregator_custom = llm.create_context_aggregator(custom_context) pipeline = Pipeline( [ transport.input(), # Transport user input context_aggregator_custom.user(), llm, tts, context_aggregator_custom.assistant(), transport.output(), # Transport bot output ] ) task = PipelineTask( pipeline, params = PipelineParams( allow_interruptions = True , enable_metrics = True , enable_usage_metrics = True , ), ) @transport.event_handler ( "on_first_participant_joined" ) async def on_first_participant_joined ( transport , participant ): transport.capture_participant_transcription(participant[ "id" ]) time.sleep( 1.5 ) messages.append( { "role" : "system" , "content" : "Introduce yourself." , } ) await task.queue_frame(LLMMessagesFrame(messages)) @transport.event_handler ( "on_participant_left" ) async def on_participant_left ( transport , participant , reason ): await task.queue_frame(EndFrame()) @transport.event_handler ( "on_call_state_updated" ) async def on_call_state_updated ( transport , state ): if state == "left" : await task.queue_frame(EndFrame()) runner = PipelineRunner() await runner.run(task) await session.close() First, in our main function, we initialize the daily transport layer to receive/send the audio/video data from the Daily room we will connect to. You can see we pass the room_url we would like to join as well as a token to authenticate us programmatically joining. We also set our VAD stop seconds which is the amount of time we wait for a pause before our bot will respond - in this example, we set it to 600 milliseconds. Next we connect to our LLM (OpenAI) as well as our TTS model (Cartesia). By setting ‘transcription_enabled=true’ we are using the STT from Daily itself. This is where the Pipecat framework helps convert audio data to text and vice versa. We then put this all together as a PipelineTask which is what Pipecat runs all together. The make up of a task is completely customizable and has support for Image and Vision use cases. Lastly, we have some event handlers for when a user joins/leaves the room. ​ Deploy bot Deploy your application to Cerebrium: Copy Ask AI cerebrium deploy You will then see that an endpoint is created for your bot at POST \<BASE_URL\>/main that you can call with your room_url and token. Let us test it. ​ Test it out Copy Ask AI def create_room (): url = "https://api.daily.co/v1/rooms/" headers = { "Content-Type" : "application/json" , "Authorization" : f "Bearer { os.environ.get( 'DAILY_TOKEN' ) } " , } data = { "properties" : { "exp" : int (time.time()) + 60 * 5 , ##5 mins "eject_at_room_exp" : True , } } response = requests.post(url, headers = headers, json = data) if response.status_code == 200 : room_info = response.json() token = create_token(room_info[ "name" ]) if token and "token" in token: room_info[ "token" ] = token[ "token" ] else : logger.error( "Failed to create token" ) return { "message" : "There was an error creating your room" , "status_code" : 500 , } return room_info else : logger.error( f "Failed to create room: { response.status_code } " ) return { "message" : "There was an error creating your room" , "status_code" : 500 } def create_token ( room_name : str ): url = "https://api.daily.co/v1/meeting-tokens" headers = { "Content-Type" : "application/json" , "Authorization" : f "Bearer { os.environ.get( 'DAILY_TOKEN' ) } " , } data = { "properties" : { "room_name" : room_name, "is_owner" : True }} response = requests.post(url, headers = headers, json = data) if response.status_code == 200 : token_info = response.json() return token_info else : logger.error( f "Failed to create token: { response.status_code } " ) return None if __name__ == "__main__" : room_info = create_room() print ( f "Join room: { room_info[ "url" ] } " ) asyncio.run(main(room_info[ "url" ], room_info[ "token" ])) ​ Future Considerations Since Cerebrium supports both CPU and GPU workloads if you would like to lower the latency of your application then the best would be to get model weights from various providers and run them locally. You can do this for: LLM: Run any OpenSource model using a framework such as vLLM TTS: Both PlayHt and Deepgram offer TTS models that can be run locally STT: Deepgram offers a local model that can be run locally If you implement all three models locally, you should have much better performance. We have been able to get ~300ms voice-to-voice responses. ​ Examples Fastest voice agent : Local only implementation RAG voice agent : Create a voice agent that can do RAG using Cerebrium + OpenAI + Pinecone Twilio voice agent : Create a voice agent that can receive phone calls via Twilio OpenAI Realtime API implementation : Create a voice agent that can receive phone calls via OpenAI Realtime API Example: Fly.io Example: Modal On this page Install the Cerebrium CLI Create a Cerebrium project Agent setup Deploy bot Test it out Future Considerations Examples Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/deployment_cerebrium_5e33c947.txt b/deployment_cerebrium_5e33c947.txt
new file mode 100644
index 0000000000000000000000000000000000000000..157e4967f76eeb5ffbb648dcbd44230ae04b22be
--- /dev/null
+++ b/deployment_cerebrium_5e33c947.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/guides/deployment/cerebrium#examples
+Title: Example: Cerebrium - Pipecat
+==================================================
+
+Example: Cerebrium - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Deploying your bot Example: Cerebrium Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Guides Fundamentals Context Management Custom FrameProcessor Detecting Idle Users Ending a Pipeline Function Calling Muting User Input Recording Audio Recording Transcripts Features Gemini Multimodal Live Metrics Noise cancellation with Krisp OpenAI Audio Models and APIs Pipecat Flows Telephony Overview Dial-in: WebRTC (Daily) Dial-in: WebRTC (Twilio + Daily) Dial-in: Twilio (Media Streams) Dialout: WebRTC (Daily) Deploying your bot Overview Deployment pattern Example: Pipecat Cloud Example: Fly.io Example: Cerebrium Example: Modal Cerebrium is a serverless Infrastructure platform that makes it easy for companies to build, deploy and scale AI applications. Cerebrium offers both CPUs and GPUs (H100s, A100s etc) with extremely low cold start times allowing us to create highly performant applications in the most cost efficient manner. ​ Install the Cerebrium CLI To get started, let us run the following commands: Run pip install cerebrium to install the Python package. Run cerebrium login to authenticate yourself. If you don’t have a Cerebrium account, you can create one and get started with $30 in free credits. ​ Create a Cerebrium project Create a new Cerebrium project: Copy Ask AI cerebrium init pipecat-agent This will create two key files: main.py - Your application entrypoint cerebrium.toml - Configuration for build and environment settings Update your cerebrium.toml with the necessary configuration: Copy Ask AI [ cerebrium . hardware ] region = "us-east-1" provider = "aws" compute = "CPU" cpu = 4 memory = 18.0 [ cerebrium . dependencies . pip ] torch = ">=2.0.0" " pipecat-ai[silero, daily, openai, cartesia] " = "latest" aiohttp = "latest" torchaudio = "latest" In order for our application to work, we need to copy our API keys from the various platforms. Navigate to the Secrets section in your Cerebrium dashboard to store your API keys: OPENAI_API_KEY - We use OpenAI For the LLM. You can get your API key from here DAILY_TOKEN - For WebRTC communication. You can get your token from here CARTERSIA_API_KEY - For text-to-speech services. You can get your API key from here We access these secrets in our code as if they are normal ENV vars. For the above, You can swap in any LLM or TTS service you wish to use. ​ Agent setup We create a basic pipeline setup in our main.py that combines our LLM, TTS and Daily WebRTC transport layer. Copy Ask AI import os import sys import time import aiohttp from loguru import logger from pipecat.frames.frames import EndFrame from pipecat.pipeline.pipeline import Pipeline from pipecat.pipeline.runner import PipelineRunner from pipecat.pipeline.task import PipelineParams, PipelineTask from pipecat.services.openai.llm import OpenAILLMService from pipecat.processors.aggregators.openai_llm_context import ( OpenAILLMContext, ) from pipecat.services.cartesia.tts import CartesiaTTSService from pipecat.transports.services.daily import DailyParams, DailyTransport from pipecat.vad.silero import SileroVADAnalyzer from pipecat.vad.vad_analyzer import VADParams logger.remove( 0 ) logger.add(sys.stderr, level = "DEBUG" ) async def main ( room_url : str , token : str ): async with aiohttp.ClientSession() as session: transport = DailyTransport( room_url, token, "Friendly bot" , DailyParams( audio_in_enabled = True , audio_out_enabled = True , transcription_enabled = True , vad_analyzer = SileroVADAnalyzer( params = VADParams( stop_secs = 0.6 )), ), ) messages = [ { "role" : "system" , "content" : "You are a helpful AI assistant that can switch between two services to showcase the difference in performance and cost: 'openai_realtime' and 'custom'. Respond to user queries and switch services when asked." , }, ] llm = OpenAILLMService( name = "LLM" , api_key = os.environ.get( "OPENAI_API_KEY" ), model = "gpt-4" , ) tts = CartesiaTTSService( api_key = os.getenv( "CARTESIA_API_KEY" ), voice_id = "79a125e8-cd45-4c13-8a67-188112f4dd22" , # British Lady ) custom_context = OpenAILLMContext( messages = messages) context_aggregator_custom = llm.create_context_aggregator(custom_context) pipeline = Pipeline( [ transport.input(), # Transport user input context_aggregator_custom.user(), llm, tts, context_aggregator_custom.assistant(), transport.output(), # Transport bot output ] ) task = PipelineTask( pipeline, params = PipelineParams( allow_interruptions = True , enable_metrics = True , enable_usage_metrics = True , ), ) @transport.event_handler ( "on_first_participant_joined" ) async def on_first_participant_joined ( transport , participant ): transport.capture_participant_transcription(participant[ "id" ]) time.sleep( 1.5 ) messages.append( { "role" : "system" , "content" : "Introduce yourself." , } ) await task.queue_frame(LLMMessagesFrame(messages)) @transport.event_handler ( "on_participant_left" ) async def on_participant_left ( transport , participant , reason ): await task.queue_frame(EndFrame()) @transport.event_handler ( "on_call_state_updated" ) async def on_call_state_updated ( transport , state ): if state == "left" : await task.queue_frame(EndFrame()) runner = PipelineRunner() await runner.run(task) await session.close() First, in our main function, we initialize the daily transport layer to receive/send the audio/video data from the Daily room we will connect to. You can see we pass the room_url we would like to join as well as a token to authenticate us programmatically joining. We also set our VAD stop seconds which is the amount of time we wait for a pause before our bot will respond - in this example, we set it to 600 milliseconds. Next we connect to our LLM (OpenAI) as well as our TTS model (Cartesia). By setting ‘transcription_enabled=true’ we are using the STT from Daily itself. This is where the Pipecat framework helps convert audio data to text and vice versa. We then put this all together as a PipelineTask which is what Pipecat runs all together. The make up of a task is completely customizable and has support for Image and Vision use cases. Lastly, we have some event handlers for when a user joins/leaves the room. ​ Deploy bot Deploy your application to Cerebrium: Copy Ask AI cerebrium deploy You will then see that an endpoint is created for your bot at POST \<BASE_URL\>/main that you can call with your room_url and token. Let us test it. ​ Test it out Copy Ask AI def create_room (): url = "https://api.daily.co/v1/rooms/" headers = { "Content-Type" : "application/json" , "Authorization" : f "Bearer { os.environ.get( 'DAILY_TOKEN' ) } " , } data = { "properties" : { "exp" : int (time.time()) + 60 * 5 , ##5 mins "eject_at_room_exp" : True , } } response = requests.post(url, headers = headers, json = data) if response.status_code == 200 : room_info = response.json() token = create_token(room_info[ "name" ]) if token and "token" in token: room_info[ "token" ] = token[ "token" ] else : logger.error( "Failed to create token" ) return { "message" : "There was an error creating your room" , "status_code" : 500 , } return room_info else : logger.error( f "Failed to create room: { response.status_code } " ) return { "message" : "There was an error creating your room" , "status_code" : 500 } def create_token ( room_name : str ): url = "https://api.daily.co/v1/meeting-tokens" headers = { "Content-Type" : "application/json" , "Authorization" : f "Bearer { os.environ.get( 'DAILY_TOKEN' ) } " , } data = { "properties" : { "room_name" : room_name, "is_owner" : True }} response = requests.post(url, headers = headers, json = data) if response.status_code == 200 : token_info = response.json() return token_info else : logger.error( f "Failed to create token: { response.status_code } " ) return None if __name__ == "__main__" : room_info = create_room() print ( f "Join room: { room_info[ "url" ] } " ) asyncio.run(main(room_info[ "url" ], room_info[ "token" ])) ​ Future Considerations Since Cerebrium supports both CPU and GPU workloads if you would like to lower the latency of your application then the best would be to get model weights from various providers and run them locally. You can do this for: LLM: Run any OpenSource model using a framework such as vLLM TTS: Both PlayHt and Deepgram offer TTS models that can be run locally STT: Deepgram offers a local model that can be run locally If you implement all three models locally, you should have much better performance. We have been able to get ~300ms voice-to-voice responses. ​ Examples Fastest voice agent : Local only implementation RAG voice agent : Create a voice agent that can do RAG using Cerebrium + OpenAI + Pinecone Twilio voice agent : Create a voice agent that can receive phone calls via Twilio OpenAI Realtime API implementation : Create a voice agent that can receive phone calls via OpenAI Realtime API Example: Fly.io Example: Modal On this page Install the Cerebrium CLI Create a Cerebrium project Agent setup Deploy bot Test it out Future Considerations Examples Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/deployment_cerebrium_94b7d04e.txt b/deployment_cerebrium_94b7d04e.txt
new file mode 100644
index 0000000000000000000000000000000000000000..879ce2ba81f842d1ffa2ce64a3e386edab2ebf9f
--- /dev/null
+++ b/deployment_cerebrium_94b7d04e.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/guides/deployment/cerebrium#create-a-cerebrium-project
+Title: Example: Cerebrium - Pipecat
+==================================================
+
+Example: Cerebrium - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Deploying your bot Example: Cerebrium Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Guides Fundamentals Context Management Custom FrameProcessor Detecting Idle Users Ending a Pipeline Function Calling Muting User Input Recording Audio Recording Transcripts Features Gemini Multimodal Live Metrics Noise cancellation with Krisp OpenAI Audio Models and APIs Pipecat Flows Telephony Overview Dial-in: WebRTC (Daily) Dial-in: WebRTC (Twilio + Daily) Dial-in: Twilio (Media Streams) Dialout: WebRTC (Daily) Deploying your bot Overview Deployment pattern Example: Pipecat Cloud Example: Fly.io Example: Cerebrium Example: Modal Cerebrium is a serverless Infrastructure platform that makes it easy for companies to build, deploy and scale AI applications. Cerebrium offers both CPUs and GPUs (H100s, A100s etc) with extremely low cold start times allowing us to create highly performant applications in the most cost efficient manner. ​ Install the Cerebrium CLI To get started, let us run the following commands: Run pip install cerebrium to install the Python package. Run cerebrium login to authenticate yourself. If you don’t have a Cerebrium account, you can create one and get started with $30 in free credits. ​ Create a Cerebrium project Create a new Cerebrium project: Copy Ask AI cerebrium init pipecat-agent This will create two key files: main.py - Your application entrypoint cerebrium.toml - Configuration for build and environment settings Update your cerebrium.toml with the necessary configuration: Copy Ask AI [ cerebrium . hardware ] region = "us-east-1" provider = "aws" compute = "CPU" cpu = 4 memory = 18.0 [ cerebrium . dependencies . pip ] torch = ">=2.0.0" " pipecat-ai[silero, daily, openai, cartesia] " = "latest" aiohttp = "latest" torchaudio = "latest" In order for our application to work, we need to copy our API keys from the various platforms. Navigate to the Secrets section in your Cerebrium dashboard to store your API keys: OPENAI_API_KEY - We use OpenAI For the LLM. You can get your API key from here DAILY_TOKEN - For WebRTC communication. You can get your token from here CARTERSIA_API_KEY - For text-to-speech services. You can get your API key from here We access these secrets in our code as if they are normal ENV vars. For the above, You can swap in any LLM or TTS service you wish to use. ​ Agent setup We create a basic pipeline setup in our main.py that combines our LLM, TTS and Daily WebRTC transport layer. Copy Ask AI import os import sys import time import aiohttp from loguru import logger from pipecat.frames.frames import EndFrame from pipecat.pipeline.pipeline import Pipeline from pipecat.pipeline.runner import PipelineRunner from pipecat.pipeline.task import PipelineParams, PipelineTask from pipecat.services.openai.llm import OpenAILLMService from pipecat.processors.aggregators.openai_llm_context import ( OpenAILLMContext, ) from pipecat.services.cartesia.tts import CartesiaTTSService from pipecat.transports.services.daily import DailyParams, DailyTransport from pipecat.vad.silero import SileroVADAnalyzer from pipecat.vad.vad_analyzer import VADParams logger.remove( 0 ) logger.add(sys.stderr, level = "DEBUG" ) async def main ( room_url : str , token : str ): async with aiohttp.ClientSession() as session: transport = DailyTransport( room_url, token, "Friendly bot" , DailyParams( audio_in_enabled = True , audio_out_enabled = True , transcription_enabled = True , vad_analyzer = SileroVADAnalyzer( params = VADParams( stop_secs = 0.6 )), ), ) messages = [ { "role" : "system" , "content" : "You are a helpful AI assistant that can switch between two services to showcase the difference in performance and cost: 'openai_realtime' and 'custom'. Respond to user queries and switch services when asked." , }, ] llm = OpenAILLMService( name = "LLM" , api_key = os.environ.get( "OPENAI_API_KEY" ), model = "gpt-4" , ) tts = CartesiaTTSService( api_key = os.getenv( "CARTESIA_API_KEY" ), voice_id = "79a125e8-cd45-4c13-8a67-188112f4dd22" , # British Lady ) custom_context = OpenAILLMContext( messages = messages) context_aggregator_custom = llm.create_context_aggregator(custom_context) pipeline = Pipeline( [ transport.input(), # Transport user input context_aggregator_custom.user(), llm, tts, context_aggregator_custom.assistant(), transport.output(), # Transport bot output ] ) task = PipelineTask( pipeline, params = PipelineParams( allow_interruptions = True , enable_metrics = True , enable_usage_metrics = True , ), ) @transport.event_handler ( "on_first_participant_joined" ) async def on_first_participant_joined ( transport , participant ): transport.capture_participant_transcription(participant[ "id" ]) time.sleep( 1.5 ) messages.append( { "role" : "system" , "content" : "Introduce yourself." , } ) await task.queue_frame(LLMMessagesFrame(messages)) @transport.event_handler ( "on_participant_left" ) async def on_participant_left ( transport , participant , reason ): await task.queue_frame(EndFrame()) @transport.event_handler ( "on_call_state_updated" ) async def on_call_state_updated ( transport , state ): if state == "left" : await task.queue_frame(EndFrame()) runner = PipelineRunner() await runner.run(task) await session.close() First, in our main function, we initialize the daily transport layer to receive/send the audio/video data from the Daily room we will connect to. You can see we pass the room_url we would like to join as well as a token to authenticate us programmatically joining. We also set our VAD stop seconds which is the amount of time we wait for a pause before our bot will respond - in this example, we set it to 600 milliseconds. Next we connect to our LLM (OpenAI) as well as our TTS model (Cartesia). By setting ‘transcription_enabled=true’ we are using the STT from Daily itself. This is where the Pipecat framework helps convert audio data to text and vice versa. We then put this all together as a PipelineTask which is what Pipecat runs all together. The make up of a task is completely customizable and has support for Image and Vision use cases. Lastly, we have some event handlers for when a user joins/leaves the room. ​ Deploy bot Deploy your application to Cerebrium: Copy Ask AI cerebrium deploy You will then see that an endpoint is created for your bot at POST \<BASE_URL\>/main that you can call with your room_url and token. Let us test it. ​ Test it out Copy Ask AI def create_room (): url = "https://api.daily.co/v1/rooms/" headers = { "Content-Type" : "application/json" , "Authorization" : f "Bearer { os.environ.get( 'DAILY_TOKEN' ) } " , } data = { "properties" : { "exp" : int (time.time()) + 60 * 5 , ##5 mins "eject_at_room_exp" : True , } } response = requests.post(url, headers = headers, json = data) if response.status_code == 200 : room_info = response.json() token = create_token(room_info[ "name" ]) if token and "token" in token: room_info[ "token" ] = token[ "token" ] else : logger.error( "Failed to create token" ) return { "message" : "There was an error creating your room" , "status_code" : 500 , } return room_info else : logger.error( f "Failed to create room: { response.status_code } " ) return { "message" : "There was an error creating your room" , "status_code" : 500 } def create_token ( room_name : str ): url = "https://api.daily.co/v1/meeting-tokens" headers = { "Content-Type" : "application/json" , "Authorization" : f "Bearer { os.environ.get( 'DAILY_TOKEN' ) } " , } data = { "properties" : { "room_name" : room_name, "is_owner" : True }} response = requests.post(url, headers = headers, json = data) if response.status_code == 200 : token_info = response.json() return token_info else : logger.error( f "Failed to create token: { response.status_code } " ) return None if __name__ == "__main__" : room_info = create_room() print ( f "Join room: { room_info[ "url" ] } " ) asyncio.run(main(room_info[ "url" ], room_info[ "token" ])) ​ Future Considerations Since Cerebrium supports both CPU and GPU workloads if you would like to lower the latency of your application then the best would be to get model weights from various providers and run them locally. You can do this for: LLM: Run any OpenSource model using a framework such as vLLM TTS: Both PlayHt and Deepgram offer TTS models that can be run locally STT: Deepgram offers a local model that can be run locally If you implement all three models locally, you should have much better performance. We have been able to get ~300ms voice-to-voice responses. ​ Examples Fastest voice agent : Local only implementation RAG voice agent : Create a voice agent that can do RAG using Cerebrium + OpenAI + Pinecone Twilio voice agent : Create a voice agent that can receive phone calls via Twilio OpenAI Realtime API implementation : Create a voice agent that can receive phone calls via OpenAI Realtime API Example: Fly.io Example: Modal On this page Install the Cerebrium CLI Create a Cerebrium project Agent setup Deploy bot Test it out Future Considerations Examples Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/deployment_cerebrium_a842c793.txt b/deployment_cerebrium_a842c793.txt
new file mode 100644
index 0000000000000000000000000000000000000000..c9b39e79efe57ede6be3b4b5cb778f900541112a
--- /dev/null
+++ b/deployment_cerebrium_a842c793.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/guides/deployment/cerebrium#test-it-out
+Title: Example: Cerebrium - Pipecat
+==================================================
+
+Example: Cerebrium - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Deploying your bot Example: Cerebrium Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Guides Fundamentals Context Management Custom FrameProcessor Detecting Idle Users Ending a Pipeline Function Calling Muting User Input Recording Audio Recording Transcripts Features Gemini Multimodal Live Metrics Noise cancellation with Krisp OpenAI Audio Models and APIs Pipecat Flows Telephony Overview Dial-in: WebRTC (Daily) Dial-in: WebRTC (Twilio + Daily) Dial-in: Twilio (Media Streams) Dialout: WebRTC (Daily) Deploying your bot Overview Deployment pattern Example: Pipecat Cloud Example: Fly.io Example: Cerebrium Example: Modal Cerebrium is a serverless Infrastructure platform that makes it easy for companies to build, deploy and scale AI applications. Cerebrium offers both CPUs and GPUs (H100s, A100s etc) with extremely low cold start times allowing us to create highly performant applications in the most cost efficient manner. ​ Install the Cerebrium CLI To get started, let us run the following commands: Run pip install cerebrium to install the Python package. Run cerebrium login to authenticate yourself. If you don’t have a Cerebrium account, you can create one and get started with $30 in free credits. ​ Create a Cerebrium project Create a new Cerebrium project: Copy Ask AI cerebrium init pipecat-agent This will create two key files: main.py - Your application entrypoint cerebrium.toml - Configuration for build and environment settings Update your cerebrium.toml with the necessary configuration: Copy Ask AI [ cerebrium . hardware ] region = "us-east-1" provider = "aws" compute = "CPU" cpu = 4 memory = 18.0 [ cerebrium . dependencies . pip ] torch = ">=2.0.0" " pipecat-ai[silero, daily, openai, cartesia] " = "latest" aiohttp = "latest" torchaudio = "latest" In order for our application to work, we need to copy our API keys from the various platforms. Navigate to the Secrets section in your Cerebrium dashboard to store your API keys: OPENAI_API_KEY - We use OpenAI For the LLM. You can get your API key from here DAILY_TOKEN - For WebRTC communication. You can get your token from here CARTERSIA_API_KEY - For text-to-speech services. You can get your API key from here We access these secrets in our code as if they are normal ENV vars. For the above, You can swap in any LLM or TTS service you wish to use. ​ Agent setup We create a basic pipeline setup in our main.py that combines our LLM, TTS and Daily WebRTC transport layer. Copy Ask AI import os import sys import time import aiohttp from loguru import logger from pipecat.frames.frames import EndFrame from pipecat.pipeline.pipeline import Pipeline from pipecat.pipeline.runner import PipelineRunner from pipecat.pipeline.task import PipelineParams, PipelineTask from pipecat.services.openai.llm import OpenAILLMService from pipecat.processors.aggregators.openai_llm_context import ( OpenAILLMContext, ) from pipecat.services.cartesia.tts import CartesiaTTSService from pipecat.transports.services.daily import DailyParams, DailyTransport from pipecat.vad.silero import SileroVADAnalyzer from pipecat.vad.vad_analyzer import VADParams logger.remove( 0 ) logger.add(sys.stderr, level = "DEBUG" ) async def main ( room_url : str , token : str ): async with aiohttp.ClientSession() as session: transport = DailyTransport( room_url, token, "Friendly bot" , DailyParams( audio_in_enabled = True , audio_out_enabled = True , transcription_enabled = True , vad_analyzer = SileroVADAnalyzer( params = VADParams( stop_secs = 0.6 )), ), ) messages = [ { "role" : "system" , "content" : "You are a helpful AI assistant that can switch between two services to showcase the difference in performance and cost: 'openai_realtime' and 'custom'. Respond to user queries and switch services when asked." , }, ] llm = OpenAILLMService( name = "LLM" , api_key = os.environ.get( "OPENAI_API_KEY" ), model = "gpt-4" , ) tts = CartesiaTTSService( api_key = os.getenv( "CARTESIA_API_KEY" ), voice_id = "79a125e8-cd45-4c13-8a67-188112f4dd22" , # British Lady ) custom_context = OpenAILLMContext( messages = messages) context_aggregator_custom = llm.create_context_aggregator(custom_context) pipeline = Pipeline( [ transport.input(), # Transport user input context_aggregator_custom.user(), llm, tts, context_aggregator_custom.assistant(), transport.output(), # Transport bot output ] ) task = PipelineTask( pipeline, params = PipelineParams( allow_interruptions = True , enable_metrics = True , enable_usage_metrics = True , ), ) @transport.event_handler ( "on_first_participant_joined" ) async def on_first_participant_joined ( transport , participant ): transport.capture_participant_transcription(participant[ "id" ]) time.sleep( 1.5 ) messages.append( { "role" : "system" , "content" : "Introduce yourself." , } ) await task.queue_frame(LLMMessagesFrame(messages)) @transport.event_handler ( "on_participant_left" ) async def on_participant_left ( transport , participant , reason ): await task.queue_frame(EndFrame()) @transport.event_handler ( "on_call_state_updated" ) async def on_call_state_updated ( transport , state ): if state == "left" : await task.queue_frame(EndFrame()) runner = PipelineRunner() await runner.run(task) await session.close() First, in our main function, we initialize the daily transport layer to receive/send the audio/video data from the Daily room we will connect to. You can see we pass the room_url we would like to join as well as a token to authenticate us programmatically joining. We also set our VAD stop seconds which is the amount of time we wait for a pause before our bot will respond - in this example, we set it to 600 milliseconds. Next we connect to our LLM (OpenAI) as well as our TTS model (Cartesia). By setting ‘transcription_enabled=true’ we are using the STT from Daily itself. This is where the Pipecat framework helps convert audio data to text and vice versa. We then put this all together as a PipelineTask which is what Pipecat runs all together. The make up of a task is completely customizable and has support for Image and Vision use cases. Lastly, we have some event handlers for when a user joins/leaves the room. ​ Deploy bot Deploy your application to Cerebrium: Copy Ask AI cerebrium deploy You will then see that an endpoint is created for your bot at POST \<BASE_URL\>/main that you can call with your room_url and token. Let us test it. ​ Test it out Copy Ask AI def create_room (): url = "https://api.daily.co/v1/rooms/" headers = { "Content-Type" : "application/json" , "Authorization" : f "Bearer { os.environ.get( 'DAILY_TOKEN' ) } " , } data = { "properties" : { "exp" : int (time.time()) + 60 * 5 , ##5 mins "eject_at_room_exp" : True , } } response = requests.post(url, headers = headers, json = data) if response.status_code == 200 : room_info = response.json() token = create_token(room_info[ "name" ]) if token and "token" in token: room_info[ "token" ] = token[ "token" ] else : logger.error( "Failed to create token" ) return { "message" : "There was an error creating your room" , "status_code" : 500 , } return room_info else : logger.error( f "Failed to create room: { response.status_code } " ) return { "message" : "There was an error creating your room" , "status_code" : 500 } def create_token ( room_name : str ): url = "https://api.daily.co/v1/meeting-tokens" headers = { "Content-Type" : "application/json" , "Authorization" : f "Bearer { os.environ.get( 'DAILY_TOKEN' ) } " , } data = { "properties" : { "room_name" : room_name, "is_owner" : True }} response = requests.post(url, headers = headers, json = data) if response.status_code == 200 : token_info = response.json() return token_info else : logger.error( f "Failed to create token: { response.status_code } " ) return None if __name__ == "__main__" : room_info = create_room() print ( f "Join room: { room_info[ "url" ] } " ) asyncio.run(main(room_info[ "url" ], room_info[ "token" ])) ​ Future Considerations Since Cerebrium supports both CPU and GPU workloads if you would like to lower the latency of your application then the best would be to get model weights from various providers and run them locally. You can do this for: LLM: Run any OpenSource model using a framework such as vLLM TTS: Both PlayHt and Deepgram offer TTS models that can be run locally STT: Deepgram offers a local model that can be run locally If you implement all three models locally, you should have much better performance. We have been able to get ~300ms voice-to-voice responses. ​ Examples Fastest voice agent : Local only implementation RAG voice agent : Create a voice agent that can do RAG using Cerebrium + OpenAI + Pinecone Twilio voice agent : Create a voice agent that can receive phone calls via Twilio OpenAI Realtime API implementation : Create a voice agent that can receive phone calls via OpenAI Realtime API Example: Fly.io Example: Modal On this page Install the Cerebrium CLI Create a Cerebrium project Agent setup Deploy bot Test it out Future Considerations Examples Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/deployment_cerebrium_fa7f78d3.txt b/deployment_cerebrium_fa7f78d3.txt
new file mode 100644
index 0000000000000000000000000000000000000000..bd7ebd740c7ab5e79267f2fb9944a6322ee78929
--- /dev/null
+++ b/deployment_cerebrium_fa7f78d3.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/guides/deployment/cerebrium
+Title: Example: Cerebrium - Pipecat
+==================================================
+
+Example: Cerebrium - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Deploying your bot Example: Cerebrium Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Guides Fundamentals Context Management Custom FrameProcessor Detecting Idle Users Ending a Pipeline Function Calling Muting User Input Recording Audio Recording Transcripts Features Gemini Multimodal Live Metrics Noise cancellation with Krisp OpenAI Audio Models and APIs Pipecat Flows Telephony Overview Dial-in: WebRTC (Daily) Dial-in: WebRTC (Twilio + Daily) Dial-in: Twilio (Media Streams) Dialout: WebRTC (Daily) Deploying your bot Overview Deployment pattern Example: Pipecat Cloud Example: Fly.io Example: Cerebrium Example: Modal Cerebrium is a serverless Infrastructure platform that makes it easy for companies to build, deploy and scale AI applications. Cerebrium offers both CPUs and GPUs (H100s, A100s etc) with extremely low cold start times allowing us to create highly performant applications in the most cost efficient manner. ​ Install the Cerebrium CLI To get started, let us run the following commands: Run pip install cerebrium to install the Python package. Run cerebrium login to authenticate yourself. If you don’t have a Cerebrium account, you can create one and get started with $30 in free credits. ​ Create a Cerebrium project Create a new Cerebrium project: Copy Ask AI cerebrium init pipecat-agent This will create two key files: main.py - Your application entrypoint cerebrium.toml - Configuration for build and environment settings Update your cerebrium.toml with the necessary configuration: Copy Ask AI [ cerebrium . hardware ] region = "us-east-1" provider = "aws" compute = "CPU" cpu = 4 memory = 18.0 [ cerebrium . dependencies . pip ] torch = ">=2.0.0" " pipecat-ai[silero, daily, openai, cartesia] " = "latest" aiohttp = "latest" torchaudio = "latest" In order for our application to work, we need to copy our API keys from the various platforms. Navigate to the Secrets section in your Cerebrium dashboard to store your API keys: OPENAI_API_KEY - We use OpenAI For the LLM. You can get your API key from here DAILY_TOKEN - For WebRTC communication. You can get your token from here CARTERSIA_API_KEY - For text-to-speech services. You can get your API key from here We access these secrets in our code as if they are normal ENV vars. For the above, You can swap in any LLM or TTS service you wish to use. ​ Agent setup We create a basic pipeline setup in our main.py that combines our LLM, TTS and Daily WebRTC transport layer. Copy Ask AI import os import sys import time import aiohttp from loguru import logger from pipecat.frames.frames import EndFrame from pipecat.pipeline.pipeline import Pipeline from pipecat.pipeline.runner import PipelineRunner from pipecat.pipeline.task import PipelineParams, PipelineTask from pipecat.services.openai.llm import OpenAILLMService from pipecat.processors.aggregators.openai_llm_context import ( OpenAILLMContext, ) from pipecat.services.cartesia.tts import CartesiaTTSService from pipecat.transports.services.daily import DailyParams, DailyTransport from pipecat.vad.silero import SileroVADAnalyzer from pipecat.vad.vad_analyzer import VADParams logger.remove( 0 ) logger.add(sys.stderr, level = "DEBUG" ) async def main ( room_url : str , token : str ): async with aiohttp.ClientSession() as session: transport = DailyTransport( room_url, token, "Friendly bot" , DailyParams( audio_in_enabled = True , audio_out_enabled = True , transcription_enabled = True , vad_analyzer = SileroVADAnalyzer( params = VADParams( stop_secs = 0.6 )), ), ) messages = [ { "role" : "system" , "content" : "You are a helpful AI assistant that can switch between two services to showcase the difference in performance and cost: 'openai_realtime' and 'custom'. Respond to user queries and switch services when asked." , }, ] llm = OpenAILLMService( name = "LLM" , api_key = os.environ.get( "OPENAI_API_KEY" ), model = "gpt-4" , ) tts = CartesiaTTSService( api_key = os.getenv( "CARTESIA_API_KEY" ), voice_id = "79a125e8-cd45-4c13-8a67-188112f4dd22" , # British Lady ) custom_context = OpenAILLMContext( messages = messages) context_aggregator_custom = llm.create_context_aggregator(custom_context) pipeline = Pipeline( [ transport.input(), # Transport user input context_aggregator_custom.user(), llm, tts, context_aggregator_custom.assistant(), transport.output(), # Transport bot output ] ) task = PipelineTask( pipeline, params = PipelineParams( allow_interruptions = True , enable_metrics = True , enable_usage_metrics = True , ), ) @transport.event_handler ( "on_first_participant_joined" ) async def on_first_participant_joined ( transport , participant ): transport.capture_participant_transcription(participant[ "id" ]) time.sleep( 1.5 ) messages.append( { "role" : "system" , "content" : "Introduce yourself." , } ) await task.queue_frame(LLMMessagesFrame(messages)) @transport.event_handler ( "on_participant_left" ) async def on_participant_left ( transport , participant , reason ): await task.queue_frame(EndFrame()) @transport.event_handler ( "on_call_state_updated" ) async def on_call_state_updated ( transport , state ): if state == "left" : await task.queue_frame(EndFrame()) runner = PipelineRunner() await runner.run(task) await session.close() First, in our main function, we initialize the daily transport layer to receive/send the audio/video data from the Daily room we will connect to. You can see we pass the room_url we would like to join as well as a token to authenticate us programmatically joining. We also set our VAD stop seconds which is the amount of time we wait for a pause before our bot will respond - in this example, we set it to 600 milliseconds. Next we connect to our LLM (OpenAI) as well as our TTS model (Cartesia). By setting ‘transcription_enabled=true’ we are using the STT from Daily itself. This is where the Pipecat framework helps convert audio data to text and vice versa. We then put this all together as a PipelineTask which is what Pipecat runs all together. The make up of a task is completely customizable and has support for Image and Vision use cases. Lastly, we have some event handlers for when a user joins/leaves the room. ​ Deploy bot Deploy your application to Cerebrium: Copy Ask AI cerebrium deploy You will then see that an endpoint is created for your bot at POST \<BASE_URL\>/main that you can call with your room_url and token. Let us test it. ​ Test it out Copy Ask AI def create_room (): url = "https://api.daily.co/v1/rooms/" headers = { "Content-Type" : "application/json" , "Authorization" : f "Bearer { os.environ.get( 'DAILY_TOKEN' ) } " , } data = { "properties" : { "exp" : int (time.time()) + 60 * 5 , ##5 mins "eject_at_room_exp" : True , } } response = requests.post(url, headers = headers, json = data) if response.status_code == 200 : room_info = response.json() token = create_token(room_info[ "name" ]) if token and "token" in token: room_info[ "token" ] = token[ "token" ] else : logger.error( "Failed to create token" ) return { "message" : "There was an error creating your room" , "status_code" : 500 , } return room_info else : logger.error( f "Failed to create room: { response.status_code } " ) return { "message" : "There was an error creating your room" , "status_code" : 500 } def create_token ( room_name : str ): url = "https://api.daily.co/v1/meeting-tokens" headers = { "Content-Type" : "application/json" , "Authorization" : f "Bearer { os.environ.get( 'DAILY_TOKEN' ) } " , } data = { "properties" : { "room_name" : room_name, "is_owner" : True }} response = requests.post(url, headers = headers, json = data) if response.status_code == 200 : token_info = response.json() return token_info else : logger.error( f "Failed to create token: { response.status_code } " ) return None if __name__ == "__main__" : room_info = create_room() print ( f "Join room: { room_info[ "url" ] } " ) asyncio.run(main(room_info[ "url" ], room_info[ "token" ])) ​ Future Considerations Since Cerebrium supports both CPU and GPU workloads if you would like to lower the latency of your application then the best would be to get model weights from various providers and run them locally. You can do this for: LLM: Run any OpenSource model using a framework such as vLLM TTS: Both PlayHt and Deepgram offer TTS models that can be run locally STT: Deepgram offers a local model that can be run locally If you implement all three models locally, you should have much better performance. We have been able to get ~300ms voice-to-voice responses. ​ Examples Fastest voice agent : Local only implementation RAG voice agent : Create a voice agent that can do RAG using Cerebrium + OpenAI + Pinecone Twilio voice agent : Create a voice agent that can receive phone calls via Twilio OpenAI Realtime API implementation : Create a voice agent that can receive phone calls via OpenAI Realtime API Example: Fly.io Example: Modal On this page Install the Cerebrium CLI Create a Cerebrium project Agent setup Deploy bot Test it out Future Considerations Examples Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/deployment_fly_00717271.txt b/deployment_fly_00717271.txt
new file mode 100644
index 0000000000000000000000000000000000000000..5870954494b60143771974d7bf62689852e4220b
--- /dev/null
+++ b/deployment_fly_00717271.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/guides/deployment/fly#launch-the-fly-project
+Title: Example: Fly.io - Pipecat
+==================================================
+
+Example: Fly.io - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Deploying your bot Example: Fly.io Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Guides Fundamentals Context Management Custom FrameProcessor Detecting Idle Users Ending a Pipeline Function Calling Muting User Input Recording Audio Recording Transcripts Features Gemini Multimodal Live Metrics Noise cancellation with Krisp OpenAI Audio Models and APIs Pipecat Flows Telephony Overview Dial-in: WebRTC (Daily) Dial-in: WebRTC (Twilio + Daily) Dial-in: Twilio (Media Streams) Dialout: WebRTC (Daily) Deploying your bot Overview Deployment pattern Example: Pipecat Cloud Example: Fly.io Example: Cerebrium Example: Modal ​ Project setup Let’s explore how we can use fly.io to make our app scalable for production by spawning our Pipecat bots on virtual machines with their own resources. We mentioned before that you would ideally containerize the bot_runner.py web service and the bot.py separately. To keep this example simple, we’ll use the same container image for both services. ​ Install the Fly CLI You can find instructions for creating and setting up your fly account here . ​ Creating the Pipecat project We have created a template project here which you can clone. Since we’re targeting production use-cases, this example uses Daily (WebRTC) as a transport, but you can configure your bot however you like. ​ Adding a fly.toml Add a fly.toml to the root of your project directory. Here is a basic example: fly.toml Copy Ask AI app = 'some-unique-app-name' primary_region = 'sjc' [ build ] [ env ] FLY_APP_NAME = 'some-unique-app-name' [ http_service ] internal_port = 7860 force_https = true auto_stop_machines = true auto_start_machines = true min_machines_running = 0 processes = [ 'app' ] [[ vm ]] memory = 512 cpu_kind = 'shared' cpus = 1 For apps with lots of users, consider what resources your HTTP service will require to meet load. We’ll define our bot.py resources later, so you can set and scale these as you like ( fly scale ... ) ​ Environment setup Our bot requires some API keys and configuration, so create a .env in your project root: .env Copy Ask AI DAILY_API_KEY = OPENAI_API_KEY = ELEVENLABS_API_KEY = ELEVENLABS_VOICE_ID = FLY_API_KEY = FLY_APP_NAME = Of course, the exact keys you need will depend on which services you are using within your bot.py . Important: your FLY_APP_NAME should match the name of your fly instance, such as that declared in your fly.toml. The .env will allow us to test in local development, but is not included in the deployment. You’ll need to set them as Fly app secrets, which you can do via the Fly dashboard or cli. fly secrets set ... ​ Containerize our app Our Fly deployment will need a container image; let’s create a simple Dockerfile in the root of the project: Dockerfile .dockerignore Copy Ask AI FROM python:3.11-slim-bookworm # Open port 7860 for http service ENV FAST_API_PORT= 7860 EXPOSE 7860 # Install Python dependencies COPY \* .py . COPY ./requirements.txt requirements.txt RUN pip3 install --no-cache-dir --upgrade -r requirements.txt # Install models RUN python3 install_deps.py # Start the FastAPI server CMD python3 bot_runner.py --port ${ FAST_API_PORT } You can use any base image as long as Python is available Our container does the following: Opens port 7860 to serve our bot_runner.py FastAPI service. Downloads the necessary python dependencies. Download / cache the model dependencies the bot.py requires. Runs the bot_runner.py and listens for web requests. ​ What models are we downloading? To support voice activity detection, we’re using Silero VAD. Whilst the filesize is not huge, having each new machine download the Silero model at runtime will impact bootup time. Instead, we include the model as part of the Docker image so it’s cached and available. You could, of course, also attach a network volume to each instance if you plan to include larger files as part of your deployment and don’t want to bloat the size of your image. ​ Launching new machines in bot_runner.py When a user starts a session with our Pipecat bot, we want to launch a new machine on fly.io with it’s own system resources. Let’s grab the bot_runner.py from the example repo here . This runner differs from others in the Pipecat repo; we’ve added a new method that sends a REST request to Fly to provision a new machine for the session. This method is invoked as part of the /start_bot endpoint: bot_runner.py Copy Ask AI FLY_API_HOST = os.getenv( "FLY_API_HOST" , "https://api.machines.dev/v1" ) FLY_APP_NAME = os.getenv( "FLY_APP_NAME" , "your-fly-app-name" ) FLY_API_KEY = os.getenv( "FLY_API_KEY" , "" ) FLY_HEADERS = { 'Authorization' : f "Bearer { FLY_API_KEY } " , 'Content-Type' : 'application/json' } def spawn_fly_machine ( room_url : str , token : str ): # Use the same image as the bot runner res = requests.get( f " { FLY_API_HOST } /apps/ { FLY_APP_NAME } /machines" , headers = FLY_HEADERS ) if res.status_code != 200 : raise Exception ( f "Unable to get machine info from Fly: { res.text } " ) image = res.json()[ 0 ][ 'config' ][ 'image' ] # Machine configuration cmd = f "python3 bot.py -u { room_url } -t { token } " cmd = cmd.split() worker_props = { "config" : { "image" : image, "auto_destroy" : True , "init" : { "cmd" : cmd }, "restart" : { "policy" : "no" }, "guest" : { "cpu_kind" : "shared" , "cpus" : 1 , "memory_mb" : 1024 # Note: 512 is just enough to run VAD, but 1gb is better } }, } # Spawn a new machine instance res = requests.post( f " { FLY_API_HOST } /apps/ { FLY_APP_NAME } /machines" , headers = FLY_HEADERS , json = worker_props) if res.status_code != 200 : raise Exception ( f "Problem starting a bot worker: { res.text } " ) # Wait for the machine to enter the started state vm_id = res.json()[ 'id' ] res = requests.get( f " { FLY_API_HOST } /apps/ { FLY_APP_NAME } /machines/ { vm_id } /wait?state=started" , headers = FLY_HEADERS ) if res.status_code != 200 : raise Exception ( f "Bot was unable to enter started state: { res.text } " ) We want to make sure the machine started ok before returning any data to the user. Fly launches machines pretty fast, but will timeout if things take longer than they should. Depending on your transport method, you may want to optimistically return a response to the user, so they can join the room and poll for the status of their bot. ​ Launch the Fly project Getting your bot on Fly is as simple as: fly launch or fly launch --org orgname if you’re part of a team. This will step you through some configuration, and build and deploy your Docker image. Be sure to configure your app secrets with the necessary environment variables once the deployment has complete. Assuming all goes well, you can update with any changes with fly deploy . ​ Test it out Start a new bot instance by sending a POST request to https://your-fly-url.fly.dev/start_bot . All being well, this will return a room URL and token. A nice feature of Fly is the ability to monitor your machines (with live logs) via their dashboard: https://fly.io/apps/YOUR-APP_NAME/machines This is really helpful for monitoring the status of your spawned machine, and debugging if things do not work as expected. This example is configured to expire after 5 minutes. The bot process is also configured to exit after the user leaves the room. This is a good way to ensure we don’t have any hanging VMs, although you’ll likely need to configure this behaviour this to meet your own needs. You’ll also notice that we set restart policy to no . This prevents the machine attempting to restart after the session has concluded and the process exits. ​ Important considerations This example does little in the way of load balancing or app security. Indeed, a user can spawn a new machine on your account simply by sending a POST request to the bot_runner.py . Be sure to configure a maximum number of instances, or authenticate requests to avoid costs getting out of control. We also deployed our bot.py on a machine with the same image as our bot_runner.py . To optimize container file sizes and increase security, consider individual images that only deploy resources they require. Example: Pipecat Cloud Example: Cerebrium On this page Project setup Install the Fly CLI Creating the Pipecat project Adding a fly.toml Environment setup Containerize our app What models are we downloading? Launching new machines in bot_runner.py Launch the Fly project Test it out Important considerations Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/deployment_fly_43bbd4b9.txt b/deployment_fly_43bbd4b9.txt
new file mode 100644
index 0000000000000000000000000000000000000000..7165dcd1a25a70ecde890ad2df0436d81e95f0e8
--- /dev/null
+++ b/deployment_fly_43bbd4b9.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/guides/deployment/fly#adding-a-fly-toml
+Title: Example: Fly.io - Pipecat
+==================================================
+
+Example: Fly.io - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Deploying your bot Example: Fly.io Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Guides Fundamentals Context Management Custom FrameProcessor Detecting Idle Users Ending a Pipeline Function Calling Muting User Input Recording Audio Recording Transcripts Features Gemini Multimodal Live Metrics Noise cancellation with Krisp OpenAI Audio Models and APIs Pipecat Flows Telephony Overview Dial-in: WebRTC (Daily) Dial-in: WebRTC (Twilio + Daily) Dial-in: Twilio (Media Streams) Dialout: WebRTC (Daily) Deploying your bot Overview Deployment pattern Example: Pipecat Cloud Example: Fly.io Example: Cerebrium Example: Modal ​ Project setup Let’s explore how we can use fly.io to make our app scalable for production by spawning our Pipecat bots on virtual machines with their own resources. We mentioned before that you would ideally containerize the bot_runner.py web service and the bot.py separately. To keep this example simple, we’ll use the same container image for both services. ​ Install the Fly CLI You can find instructions for creating and setting up your fly account here . ​ Creating the Pipecat project We have created a template project here which you can clone. Since we’re targeting production use-cases, this example uses Daily (WebRTC) as a transport, but you can configure your bot however you like. ​ Adding a fly.toml Add a fly.toml to the root of your project directory. Here is a basic example: fly.toml Copy Ask AI app = 'some-unique-app-name' primary_region = 'sjc' [ build ] [ env ] FLY_APP_NAME = 'some-unique-app-name' [ http_service ] internal_port = 7860 force_https = true auto_stop_machines = true auto_start_machines = true min_machines_running = 0 processes = [ 'app' ] [[ vm ]] memory = 512 cpu_kind = 'shared' cpus = 1 For apps with lots of users, consider what resources your HTTP service will require to meet load. We’ll define our bot.py resources later, so you can set and scale these as you like ( fly scale ... ) ​ Environment setup Our bot requires some API keys and configuration, so create a .env in your project root: .env Copy Ask AI DAILY_API_KEY = OPENAI_API_KEY = ELEVENLABS_API_KEY = ELEVENLABS_VOICE_ID = FLY_API_KEY = FLY_APP_NAME = Of course, the exact keys you need will depend on which services you are using within your bot.py . Important: your FLY_APP_NAME should match the name of your fly instance, such as that declared in your fly.toml. The .env will allow us to test in local development, but is not included in the deployment. You’ll need to set them as Fly app secrets, which you can do via the Fly dashboard or cli. fly secrets set ... ​ Containerize our app Our Fly deployment will need a container image; let’s create a simple Dockerfile in the root of the project: Dockerfile .dockerignore Copy Ask AI FROM python:3.11-slim-bookworm # Open port 7860 for http service ENV FAST_API_PORT= 7860 EXPOSE 7860 # Install Python dependencies COPY \* .py . COPY ./requirements.txt requirements.txt RUN pip3 install --no-cache-dir --upgrade -r requirements.txt # Install models RUN python3 install_deps.py # Start the FastAPI server CMD python3 bot_runner.py --port ${ FAST_API_PORT } You can use any base image as long as Python is available Our container does the following: Opens port 7860 to serve our bot_runner.py FastAPI service. Downloads the necessary python dependencies. Download / cache the model dependencies the bot.py requires. Runs the bot_runner.py and listens for web requests. ​ What models are we downloading? To support voice activity detection, we’re using Silero VAD. Whilst the filesize is not huge, having each new machine download the Silero model at runtime will impact bootup time. Instead, we include the model as part of the Docker image so it’s cached and available. You could, of course, also attach a network volume to each instance if you plan to include larger files as part of your deployment and don’t want to bloat the size of your image. ​ Launching new machines in bot_runner.py When a user starts a session with our Pipecat bot, we want to launch a new machine on fly.io with it’s own system resources. Let’s grab the bot_runner.py from the example repo here . This runner differs from others in the Pipecat repo; we’ve added a new method that sends a REST request to Fly to provision a new machine for the session. This method is invoked as part of the /start_bot endpoint: bot_runner.py Copy Ask AI FLY_API_HOST = os.getenv( "FLY_API_HOST" , "https://api.machines.dev/v1" ) FLY_APP_NAME = os.getenv( "FLY_APP_NAME" , "your-fly-app-name" ) FLY_API_KEY = os.getenv( "FLY_API_KEY" , "" ) FLY_HEADERS = { 'Authorization' : f "Bearer { FLY_API_KEY } " , 'Content-Type' : 'application/json' } def spawn_fly_machine ( room_url : str , token : str ): # Use the same image as the bot runner res = requests.get( f " { FLY_API_HOST } /apps/ { FLY_APP_NAME } /machines" , headers = FLY_HEADERS ) if res.status_code != 200 : raise Exception ( f "Unable to get machine info from Fly: { res.text } " ) image = res.json()[ 0 ][ 'config' ][ 'image' ] # Machine configuration cmd = f "python3 bot.py -u { room_url } -t { token } " cmd = cmd.split() worker_props = { "config" : { "image" : image, "auto_destroy" : True , "init" : { "cmd" : cmd }, "restart" : { "policy" : "no" }, "guest" : { "cpu_kind" : "shared" , "cpus" : 1 , "memory_mb" : 1024 # Note: 512 is just enough to run VAD, but 1gb is better } }, } # Spawn a new machine instance res = requests.post( f " { FLY_API_HOST } /apps/ { FLY_APP_NAME } /machines" , headers = FLY_HEADERS , json = worker_props) if res.status_code != 200 : raise Exception ( f "Problem starting a bot worker: { res.text } " ) # Wait for the machine to enter the started state vm_id = res.json()[ 'id' ] res = requests.get( f " { FLY_API_HOST } /apps/ { FLY_APP_NAME } /machines/ { vm_id } /wait?state=started" , headers = FLY_HEADERS ) if res.status_code != 200 : raise Exception ( f "Bot was unable to enter started state: { res.text } " ) We want to make sure the machine started ok before returning any data to the user. Fly launches machines pretty fast, but will timeout if things take longer than they should. Depending on your transport method, you may want to optimistically return a response to the user, so they can join the room and poll for the status of their bot. ​ Launch the Fly project Getting your bot on Fly is as simple as: fly launch or fly launch --org orgname if you’re part of a team. This will step you through some configuration, and build and deploy your Docker image. Be sure to configure your app secrets with the necessary environment variables once the deployment has complete. Assuming all goes well, you can update with any changes with fly deploy . ​ Test it out Start a new bot instance by sending a POST request to https://your-fly-url.fly.dev/start_bot . All being well, this will return a room URL and token. A nice feature of Fly is the ability to monitor your machines (with live logs) via their dashboard: https://fly.io/apps/YOUR-APP_NAME/machines This is really helpful for monitoring the status of your spawned machine, and debugging if things do not work as expected. This example is configured to expire after 5 minutes. The bot process is also configured to exit after the user leaves the room. This is a good way to ensure we don’t have any hanging VMs, although you’ll likely need to configure this behaviour this to meet your own needs. You’ll also notice that we set restart policy to no . This prevents the machine attempting to restart after the session has concluded and the process exits. ​ Important considerations This example does little in the way of load balancing or app security. Indeed, a user can spawn a new machine on your account simply by sending a POST request to the bot_runner.py . Be sure to configure a maximum number of instances, or authenticate requests to avoid costs getting out of control. We also deployed our bot.py on a machine with the same image as our bot_runner.py . To optimize container file sizes and increase security, consider individual images that only deploy resources they require. Example: Pipecat Cloud Example: Cerebrium On this page Project setup Install the Fly CLI Creating the Pipecat project Adding a fly.toml Environment setup Containerize our app What models are we downloading? Launching new machines in bot_runner.py Launch the Fly project Test it out Important considerations Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/deployment_fly_4ddcf692.txt b/deployment_fly_4ddcf692.txt
new file mode 100644
index 0000000000000000000000000000000000000000..5525c95746b3c47cb3be17a52823ad109085a8f8
--- /dev/null
+++ b/deployment_fly_4ddcf692.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/deployment/fly#what-you-can-build
+Title: Overview - Pipecat
+==================================================
+
+Overview - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Get Started Overview Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Get Started Overview Installation & Setup Quickstart Core Concepts Next Steps & Examples Pipecat is an open source Python framework that handles the complex orchestration of AI services, network transport, audio processing, and multimodal interactions. “Multimodal” means you can use any combination of audio, video, images, and/or text in your interactions. And “real-time” means that things are happening quickly enough that it feels conversational—a “back-and-forth” with a bot, not submitting a query and waiting for results. ​ What You Can Build Voice Assistants Natural, real-time conversations with AI using speech recognition and synthesis Interactive Agents Personal coaches and meeting assistants that can understand context and provide guidance Multimodal Apps Applications that combine voice, video, images, and text for rich interactions Creative Tools Storytelling experiences and social companions that engage users Business Solutions Customer intake flows and support bots for automated business processes Complex Flows Structured conversations using Pipecat Flows for managing complex interactions ​ How It Works The flow of interactions in a Pipecat application is typically straightforward: The bot says something The user says something The bot says something The user says something This continues until the conversation naturally ends. While this flow seems simple, making it feel natural requires sophisticated real-time processing. ​ Real-time Processing Pipecat’s pipeline architecture handles both simple voice interactions and complex multimodal processing. Let’s look at how data flows through the system: Voice app Multimodal app 1 Send Audio Transmit and capture streamed audio from the user 2 Transcribe Speech Convert speech to text as the user is talking 3 Process with LLM Generate responses using a large language model 4 Convert to Speech Transform text responses into natural speech 5 Play Audio Stream the audio response back to the user 1 Send Audio Transmit and capture streamed audio from the user 2 Transcribe Speech Convert speech to text as the user is talking 3 Process with LLM Generate responses using a large language model 4 Convert to Speech Transform text responses into natural speech 5 Play Audio Stream the audio response back to the user 1 Send Audio and Video Transmit and capture audio, video, and image inputs simultaneously 2 Process Streams Handle multiple input streams in parallel 3 Model Processing Send combined inputs to multimodal models (like GPT-4V) 4 Generate Outputs Create various outputs (text, images, audio, etc.) 5 Coordinate Presentation Synchronize and present multiple output types In both cases, Pipecat: Processes responses as they stream in Handles multiple input/output modalities concurrently Manages resource allocation and synchronization Coordinates parallel processing tasks This architecture creates fluid, natural interactions without noticeable delays, whether you’re building a simple voice assistant or a complex multimodal application. Pipecat’s pipeline architecture is particularly valuable for managing the complexity of real-time, multimodal interactions, ensuring smooth data flow and proper synchronization regardless of the input/output types involved. Pipecat handles all this complexity for you, letting you focus on building your application rather than managing the underlying infrastructure. ​ Next Steps Ready to build your first Pipecat application? Installation & Setup Prepare your environment and install required dependencies Quickstart Build and run your first Pipecat application Core Concepts Learn about pipelines, frames, and real-time processing Use Cases Explore example implementations and patterns ​ Join Our Community Discord Community Connect with other developers, share your projects, and get support from the Pipecat team. Installation & Setup On this page What You Can Build How It Works Real-time Processing Next Steps Join Our Community Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/deployment_fly_ac56e353.txt b/deployment_fly_ac56e353.txt
new file mode 100644
index 0000000000000000000000000000000000000000..3cc294343ddde0025dcc24a1897181493d43a459
--- /dev/null
+++ b/deployment_fly_ac56e353.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/guides/deployment/fly#test-it-out
+Title: Example: Fly.io - Pipecat
+==================================================
+
+Example: Fly.io - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Deploying your bot Example: Fly.io Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Guides Fundamentals Context Management Custom FrameProcessor Detecting Idle Users Ending a Pipeline Function Calling Muting User Input Recording Audio Recording Transcripts Features Gemini Multimodal Live Metrics Noise cancellation with Krisp OpenAI Audio Models and APIs Pipecat Flows Telephony Overview Dial-in: WebRTC (Daily) Dial-in: WebRTC (Twilio + Daily) Dial-in: Twilio (Media Streams) Dialout: WebRTC (Daily) Deploying your bot Overview Deployment pattern Example: Pipecat Cloud Example: Fly.io Example: Cerebrium Example: Modal ​ Project setup Let’s explore how we can use fly.io to make our app scalable for production by spawning our Pipecat bots on virtual machines with their own resources. We mentioned before that you would ideally containerize the bot_runner.py web service and the bot.py separately. To keep this example simple, we’ll use the same container image for both services. ​ Install the Fly CLI You can find instructions for creating and setting up your fly account here . ​ Creating the Pipecat project We have created a template project here which you can clone. Since we’re targeting production use-cases, this example uses Daily (WebRTC) as a transport, but you can configure your bot however you like. ​ Adding a fly.toml Add a fly.toml to the root of your project directory. Here is a basic example: fly.toml Copy Ask AI app = 'some-unique-app-name' primary_region = 'sjc' [ build ] [ env ] FLY_APP_NAME = 'some-unique-app-name' [ http_service ] internal_port = 7860 force_https = true auto_stop_machines = true auto_start_machines = true min_machines_running = 0 processes = [ 'app' ] [[ vm ]] memory = 512 cpu_kind = 'shared' cpus = 1 For apps with lots of users, consider what resources your HTTP service will require to meet load. We’ll define our bot.py resources later, so you can set and scale these as you like ( fly scale ... ) ​ Environment setup Our bot requires some API keys and configuration, so create a .env in your project root: .env Copy Ask AI DAILY_API_KEY = OPENAI_API_KEY = ELEVENLABS_API_KEY = ELEVENLABS_VOICE_ID = FLY_API_KEY = FLY_APP_NAME = Of course, the exact keys you need will depend on which services you are using within your bot.py . Important: your FLY_APP_NAME should match the name of your fly instance, such as that declared in your fly.toml. The .env will allow us to test in local development, but is not included in the deployment. You’ll need to set them as Fly app secrets, which you can do via the Fly dashboard or cli. fly secrets set ... ​ Containerize our app Our Fly deployment will need a container image; let’s create a simple Dockerfile in the root of the project: Dockerfile .dockerignore Copy Ask AI FROM python:3.11-slim-bookworm # Open port 7860 for http service ENV FAST_API_PORT= 7860 EXPOSE 7860 # Install Python dependencies COPY \* .py . COPY ./requirements.txt requirements.txt RUN pip3 install --no-cache-dir --upgrade -r requirements.txt # Install models RUN python3 install_deps.py # Start the FastAPI server CMD python3 bot_runner.py --port ${ FAST_API_PORT } You can use any base image as long as Python is available Our container does the following: Opens port 7860 to serve our bot_runner.py FastAPI service. Downloads the necessary python dependencies. Download / cache the model dependencies the bot.py requires. Runs the bot_runner.py and listens for web requests. ​ What models are we downloading? To support voice activity detection, we’re using Silero VAD. Whilst the filesize is not huge, having each new machine download the Silero model at runtime will impact bootup time. Instead, we include the model as part of the Docker image so it’s cached and available. You could, of course, also attach a network volume to each instance if you plan to include larger files as part of your deployment and don’t want to bloat the size of your image. ​ Launching new machines in bot_runner.py When a user starts a session with our Pipecat bot, we want to launch a new machine on fly.io with it’s own system resources. Let’s grab the bot_runner.py from the example repo here . This runner differs from others in the Pipecat repo; we’ve added a new method that sends a REST request to Fly to provision a new machine for the session. This method is invoked as part of the /start_bot endpoint: bot_runner.py Copy Ask AI FLY_API_HOST = os.getenv( "FLY_API_HOST" , "https://api.machines.dev/v1" ) FLY_APP_NAME = os.getenv( "FLY_APP_NAME" , "your-fly-app-name" ) FLY_API_KEY = os.getenv( "FLY_API_KEY" , "" ) FLY_HEADERS = { 'Authorization' : f "Bearer { FLY_API_KEY } " , 'Content-Type' : 'application/json' } def spawn_fly_machine ( room_url : str , token : str ): # Use the same image as the bot runner res = requests.get( f " { FLY_API_HOST } /apps/ { FLY_APP_NAME } /machines" , headers = FLY_HEADERS ) if res.status_code != 200 : raise Exception ( f "Unable to get machine info from Fly: { res.text } " ) image = res.json()[ 0 ][ 'config' ][ 'image' ] # Machine configuration cmd = f "python3 bot.py -u { room_url } -t { token } " cmd = cmd.split() worker_props = { "config" : { "image" : image, "auto_destroy" : True , "init" : { "cmd" : cmd }, "restart" : { "policy" : "no" }, "guest" : { "cpu_kind" : "shared" , "cpus" : 1 , "memory_mb" : 1024 # Note: 512 is just enough to run VAD, but 1gb is better } }, } # Spawn a new machine instance res = requests.post( f " { FLY_API_HOST } /apps/ { FLY_APP_NAME } /machines" , headers = FLY_HEADERS , json = worker_props) if res.status_code != 200 : raise Exception ( f "Problem starting a bot worker: { res.text } " ) # Wait for the machine to enter the started state vm_id = res.json()[ 'id' ] res = requests.get( f " { FLY_API_HOST } /apps/ { FLY_APP_NAME } /machines/ { vm_id } /wait?state=started" , headers = FLY_HEADERS ) if res.status_code != 200 : raise Exception ( f "Bot was unable to enter started state: { res.text } " ) We want to make sure the machine started ok before returning any data to the user. Fly launches machines pretty fast, but will timeout if things take longer than they should. Depending on your transport method, you may want to optimistically return a response to the user, so they can join the room and poll for the status of their bot. ​ Launch the Fly project Getting your bot on Fly is as simple as: fly launch or fly launch --org orgname if you’re part of a team. This will step you through some configuration, and build and deploy your Docker image. Be sure to configure your app secrets with the necessary environment variables once the deployment has complete. Assuming all goes well, you can update with any changes with fly deploy . ​ Test it out Start a new bot instance by sending a POST request to https://your-fly-url.fly.dev/start_bot . All being well, this will return a room URL and token. A nice feature of Fly is the ability to monitor your machines (with live logs) via their dashboard: https://fly.io/apps/YOUR-APP_NAME/machines This is really helpful for monitoring the status of your spawned machine, and debugging if things do not work as expected. This example is configured to expire after 5 minutes. The bot process is also configured to exit after the user leaves the room. This is a good way to ensure we don’t have any hanging VMs, although you’ll likely need to configure this behaviour this to meet your own needs. You’ll also notice that we set restart policy to no . This prevents the machine attempting to restart after the session has concluded and the process exits. ​ Important considerations This example does little in the way of load balancing or app security. Indeed, a user can spawn a new machine on your account simply by sending a POST request to the bot_runner.py . Be sure to configure a maximum number of instances, or authenticate requests to avoid costs getting out of control. We also deployed our bot.py on a machine with the same image as our bot_runner.py . To optimize container file sizes and increase security, consider individual images that only deploy resources they require. Example: Pipecat Cloud Example: Cerebrium On this page Project setup Install the Fly CLI Creating the Pipecat project Adding a fly.toml Environment setup Containerize our app What models are we downloading? Launching new machines in bot_runner.py Launch the Fly project Test it out Important considerations Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/deployment_fly_e1938c41.txt b/deployment_fly_e1938c41.txt
new file mode 100644
index 0000000000000000000000000000000000000000..bb549beb488dfbdda5640727c4a5c2235f26b29f
--- /dev/null
+++ b/deployment_fly_e1938c41.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/deployment/fly#join-our-community
+Title: Overview - Pipecat
+==================================================
+
+Overview - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Get Started Overview Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Get Started Overview Installation & Setup Quickstart Core Concepts Next Steps & Examples Pipecat is an open source Python framework that handles the complex orchestration of AI services, network transport, audio processing, and multimodal interactions. “Multimodal” means you can use any combination of audio, video, images, and/or text in your interactions. And “real-time” means that things are happening quickly enough that it feels conversational—a “back-and-forth” with a bot, not submitting a query and waiting for results. ​ What You Can Build Voice Assistants Natural, real-time conversations with AI using speech recognition and synthesis Interactive Agents Personal coaches and meeting assistants that can understand context and provide guidance Multimodal Apps Applications that combine voice, video, images, and text for rich interactions Creative Tools Storytelling experiences and social companions that engage users Business Solutions Customer intake flows and support bots for automated business processes Complex Flows Structured conversations using Pipecat Flows for managing complex interactions ​ How It Works The flow of interactions in a Pipecat application is typically straightforward: The bot says something The user says something The bot says something The user says something This continues until the conversation naturally ends. While this flow seems simple, making it feel natural requires sophisticated real-time processing. ​ Real-time Processing Pipecat’s pipeline architecture handles both simple voice interactions and complex multimodal processing. Let’s look at how data flows through the system: Voice app Multimodal app 1 Send Audio Transmit and capture streamed audio from the user 2 Transcribe Speech Convert speech to text as the user is talking 3 Process with LLM Generate responses using a large language model 4 Convert to Speech Transform text responses into natural speech 5 Play Audio Stream the audio response back to the user 1 Send Audio Transmit and capture streamed audio from the user 2 Transcribe Speech Convert speech to text as the user is talking 3 Process with LLM Generate responses using a large language model 4 Convert to Speech Transform text responses into natural speech 5 Play Audio Stream the audio response back to the user 1 Send Audio and Video Transmit and capture audio, video, and image inputs simultaneously 2 Process Streams Handle multiple input streams in parallel 3 Model Processing Send combined inputs to multimodal models (like GPT-4V) 4 Generate Outputs Create various outputs (text, images, audio, etc.) 5 Coordinate Presentation Synchronize and present multiple output types In both cases, Pipecat: Processes responses as they stream in Handles multiple input/output modalities concurrently Manages resource allocation and synchronization Coordinates parallel processing tasks This architecture creates fluid, natural interactions without noticeable delays, whether you’re building a simple voice assistant or a complex multimodal application. Pipecat’s pipeline architecture is particularly valuable for managing the complexity of real-time, multimodal interactions, ensuring smooth data flow and proper synchronization regardless of the input/output types involved. Pipecat handles all this complexity for you, letting you focus on building your application rather than managing the underlying infrastructure. ​ Next Steps Ready to build your first Pipecat application? Installation & Setup Prepare your environment and install required dependencies Quickstart Build and run your first Pipecat application Core Concepts Learn about pipelines, frames, and real-time processing Use Cases Explore example implementations and patterns ​ Join Our Community Discord Community Connect with other developers, share your projects, and get support from the Pipecat team. Installation & Setup On this page What You Can Build How It Works Real-time Processing Next Steps Join Our Community Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/deployment_modal_78db198e.txt b/deployment_modal_78db198e.txt
new file mode 100644
index 0000000000000000000000000000000000000000..8a7ccdfd2398861f32281e7e6e84292801ff5662
--- /dev/null
+++ b/deployment_modal_78db198e.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/guides/deployment/modal#option-2%3A-connect-via-an-rtvi-client
+Title: Example: Modal - Pipecat
+==================================================
+
+Example: Modal - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Deploying your bot Example: Modal Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Guides Fundamentals Context Management Custom FrameProcessor Detecting Idle Users Ending a Pipeline Function Calling Muting User Input Recording Audio Recording Transcripts Features Gemini Multimodal Live Metrics Noise cancellation with Krisp OpenAI Audio Models and APIs Pipecat Flows Telephony Overview Dial-in: WebRTC (Daily) Dial-in: WebRTC (Twilio + Daily) Dial-in: Twilio (Media Streams) Dialout: WebRTC (Daily) Deploying your bot Overview Deployment pattern Example: Pipecat Cloud Example: Fly.io Example: Cerebrium Example: Modal Modal is well-suited for Pipecat deployments because it handles container orchestration, scaling, and cold starts efficiently. This makes it a good choice for production Pipecat bots that need reliable performance. This guide walks through the Modal example included in the Pipecat repository, which follows the same deployment pattern . Modal example View the complete Modal deployment example in our GitHub repository ​ Install the Modal CLI Set up Modal Follow Modal’s official instructions for creating an account and setting up the CLI ​ Deploy a self-serve LLM Deploy Modal’s OpenAI-compatible LLM service: Copy Ask AI git clone https://github.com/modal-labs/modal-examples cd modal-examples modal deploy 06_gpu_and_ml/llm-serving/vllm_inference.py Refer to Modal’s guide and example for Deploying an OpenAI-compatible LLM service with vLLM for more details. Take note of the endpoint URL from the previous step, which will look like: Copy Ask AI https://{your-workspace}--example-vllm-openai-compatible-serve.modal.run You’ll need this for the bot_vllm.py file in the next section. The default Modal LLM example uses Llama-3.1 and will shut down after 15 minutes of inactivity. Cold starts take 5-10 minutes. To prepare the service, we recommend visiting the /docs endpoint ( https://<Modal workspace>--example-vllm-openai-compatible-serve.modal.run/docs ) for your deployed LLM and wait for it to fully load before connecting your client. ​ Deploy FastAPI App and Pipecat pipeline to Modal Setup environment variables: Copy Ask AI cd server cp env.example .env # Modify .env to provide your service API Keys Alternatively, you can configure your Modal app to use secrets . Update the modal_url in server/src/bot_vllm.py to point to the URL you received from the self-serve LLM deployment in the previous step. From within the server directory, test the app locally: Copy Ask AI modal serve app.py Deploy to production: Copy Ask AI modal deploy app.py Note the endpoint URL produced from this deployment. It will look like: Copy Ask AI https:// {your-workspace} --pipecat-modal-fastapi-app.modal.run You’ll need this URL for the client’s app.js configuration mentioned in its README. ​ Launch your bots on Modal ​ Option 1: Direct Link Simply click on the URL displayed after running the server or deploy step to launch an agent and be redirected to a Daily room to talk with the launched bot. This will use the OpenAI pipeline. ​ Option 2: Connect via an RTVI Client Follow the instructions provided in the client folder’s README for building and running a custom client that connects to your Modal endpoint. The provided client includes a dropdown for choosing which bot pipeline to run. ​ Navigating your LLM, server, and Pipecat logs On your Modal dashboard , you should have two Apps listed under Live Apps: example-vllm-openai-compatible : This App contains the containers and logs used to run your self-hosted LLM. There will be just one App Function listed: serve . Click on this function to view logs for your LLM. pipecat-modal : This App contains the containers and logs used to run your connect endpoints and Pipecat pipelines. It will list two App Functions: fastapi_app : This function is running the endpoints that your client will interact with and initiate starting a new pipeline ( / , /connect , /status ). Click on this function to see logs for each endpoint hit. bot_runner : This function handles launching and running a bot pipeline. Click on this function to get a list of all pipeline runs and access each run’s logs. ​ Modal & Pipecat Tips In most other Pipecat examples, we use Popen to launch the pipeline process from the /connect endpoint. In this example, we use a Modal function instead. This allows us to run the pipelines using a separately defined Modal image as well as run each pipeline in an isolated container. For the FastAPI and most common Pipecat Pipeline containers, a default debian_slim CPU-only should be all that’s required to run. GPU containers are needed for self-hosted services. To minimize cold starts of the pipeline and reduce latency for users, set min_containers=1 on the Modal Function that launches the pipeline to ensure at least one warm instance of your function is always available. ​ Next steps Explore Modal's LLM Examples For next steps on running a self-hosted LLM and reducing latency, check out all of Modal’s LLM examples Example: Cerebrium On this page Install the Modal CLI Deploy a self-serve LLM Deploy FastAPI App and Pipecat pipeline to Modal Launch your bots on Modal Option 1: Direct Link Option 2: Connect via an RTVI Client Navigating your LLM, server, and Pipecat logs Modal & Pipecat Tips Next steps Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/deployment_modal_8c971f91.txt b/deployment_modal_8c971f91.txt
new file mode 100644
index 0000000000000000000000000000000000000000..6e5b6909d97f89dbe74f166b4964164a2ae00a17
--- /dev/null
+++ b/deployment_modal_8c971f91.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/guides/deployment/modal#deploy-fastapi-app-and-pipecat-pipeline-to-modal
+Title: Example: Modal - Pipecat
+==================================================
+
+Example: Modal - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Deploying your bot Example: Modal Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Guides Fundamentals Context Management Custom FrameProcessor Detecting Idle Users Ending a Pipeline Function Calling Muting User Input Recording Audio Recording Transcripts Features Gemini Multimodal Live Metrics Noise cancellation with Krisp OpenAI Audio Models and APIs Pipecat Flows Telephony Overview Dial-in: WebRTC (Daily) Dial-in: WebRTC (Twilio + Daily) Dial-in: Twilio (Media Streams) Dialout: WebRTC (Daily) Deploying your bot Overview Deployment pattern Example: Pipecat Cloud Example: Fly.io Example: Cerebrium Example: Modal Modal is well-suited for Pipecat deployments because it handles container orchestration, scaling, and cold starts efficiently. This makes it a good choice for production Pipecat bots that need reliable performance. This guide walks through the Modal example included in the Pipecat repository, which follows the same deployment pattern . Modal example View the complete Modal deployment example in our GitHub repository ​ Install the Modal CLI Set up Modal Follow Modal’s official instructions for creating an account and setting up the CLI ​ Deploy a self-serve LLM Deploy Modal’s OpenAI-compatible LLM service: Copy Ask AI git clone https://github.com/modal-labs/modal-examples cd modal-examples modal deploy 06_gpu_and_ml/llm-serving/vllm_inference.py Refer to Modal’s guide and example for Deploying an OpenAI-compatible LLM service with vLLM for more details. Take note of the endpoint URL from the previous step, which will look like: Copy Ask AI https://{your-workspace}--example-vllm-openai-compatible-serve.modal.run You’ll need this for the bot_vllm.py file in the next section. The default Modal LLM example uses Llama-3.1 and will shut down after 15 minutes of inactivity. Cold starts take 5-10 minutes. To prepare the service, we recommend visiting the /docs endpoint ( https://<Modal workspace>--example-vllm-openai-compatible-serve.modal.run/docs ) for your deployed LLM and wait for it to fully load before connecting your client. ​ Deploy FastAPI App and Pipecat pipeline to Modal Setup environment variables: Copy Ask AI cd server cp env.example .env # Modify .env to provide your service API Keys Alternatively, you can configure your Modal app to use secrets . Update the modal_url in server/src/bot_vllm.py to point to the URL you received from the self-serve LLM deployment in the previous step. From within the server directory, test the app locally: Copy Ask AI modal serve app.py Deploy to production: Copy Ask AI modal deploy app.py Note the endpoint URL produced from this deployment. It will look like: Copy Ask AI https:// {your-workspace} --pipecat-modal-fastapi-app.modal.run You’ll need this URL for the client’s app.js configuration mentioned in its README. ​ Launch your bots on Modal ​ Option 1: Direct Link Simply click on the URL displayed after running the server or deploy step to launch an agent and be redirected to a Daily room to talk with the launched bot. This will use the OpenAI pipeline. ​ Option 2: Connect via an RTVI Client Follow the instructions provided in the client folder’s README for building and running a custom client that connects to your Modal endpoint. The provided client includes a dropdown for choosing which bot pipeline to run. ​ Navigating your LLM, server, and Pipecat logs On your Modal dashboard , you should have two Apps listed under Live Apps: example-vllm-openai-compatible : This App contains the containers and logs used to run your self-hosted LLM. There will be just one App Function listed: serve . Click on this function to view logs for your LLM. pipecat-modal : This App contains the containers and logs used to run your connect endpoints and Pipecat pipelines. It will list two App Functions: fastapi_app : This function is running the endpoints that your client will interact with and initiate starting a new pipeline ( / , /connect , /status ). Click on this function to see logs for each endpoint hit. bot_runner : This function handles launching and running a bot pipeline. Click on this function to get a list of all pipeline runs and access each run’s logs. ​ Modal & Pipecat Tips In most other Pipecat examples, we use Popen to launch the pipeline process from the /connect endpoint. In this example, we use a Modal function instead. This allows us to run the pipelines using a separately defined Modal image as well as run each pipeline in an isolated container. For the FastAPI and most common Pipecat Pipeline containers, a default debian_slim CPU-only should be all that’s required to run. GPU containers are needed for self-hosted services. To minimize cold starts of the pipeline and reduce latency for users, set min_containers=1 on the Modal Function that launches the pipeline to ensure at least one warm instance of your function is always available. ​ Next steps Explore Modal's LLM Examples For next steps on running a self-hosted LLM and reducing latency, check out all of Modal’s LLM examples Example: Cerebrium On this page Install the Modal CLI Deploy a self-serve LLM Deploy FastAPI App and Pipecat pipeline to Modal Launch your bots on Modal Option 1: Direct Link Option 2: Connect via an RTVI Client Navigating your LLM, server, and Pipecat logs Modal & Pipecat Tips Next steps Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/deployment_modal_99a08319.txt b/deployment_modal_99a08319.txt
new file mode 100644
index 0000000000000000000000000000000000000000..3b2ea5d088ecf3106d6d17b164477081e14c06b0
--- /dev/null
+++ b/deployment_modal_99a08319.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/guides/deployment/modal
+Title: Example: Modal - Pipecat
+==================================================
+
+Example: Modal - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Deploying your bot Example: Modal Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Guides Fundamentals Context Management Custom FrameProcessor Detecting Idle Users Ending a Pipeline Function Calling Muting User Input Recording Audio Recording Transcripts Features Gemini Multimodal Live Metrics Noise cancellation with Krisp OpenAI Audio Models and APIs Pipecat Flows Telephony Overview Dial-in: WebRTC (Daily) Dial-in: WebRTC (Twilio + Daily) Dial-in: Twilio (Media Streams) Dialout: WebRTC (Daily) Deploying your bot Overview Deployment pattern Example: Pipecat Cloud Example: Fly.io Example: Cerebrium Example: Modal Modal is well-suited for Pipecat deployments because it handles container orchestration, scaling, and cold starts efficiently. This makes it a good choice for production Pipecat bots that need reliable performance. This guide walks through the Modal example included in the Pipecat repository, which follows the same deployment pattern . Modal example View the complete Modal deployment example in our GitHub repository ​ Install the Modal CLI Set up Modal Follow Modal’s official instructions for creating an account and setting up the CLI ​ Deploy a self-serve LLM Deploy Modal’s OpenAI-compatible LLM service: Copy Ask AI git clone https://github.com/modal-labs/modal-examples cd modal-examples modal deploy 06_gpu_and_ml/llm-serving/vllm_inference.py Refer to Modal’s guide and example for Deploying an OpenAI-compatible LLM service with vLLM for more details. Take note of the endpoint URL from the previous step, which will look like: Copy Ask AI https://{your-workspace}--example-vllm-openai-compatible-serve.modal.run You’ll need this for the bot_vllm.py file in the next section. The default Modal LLM example uses Llama-3.1 and will shut down after 15 minutes of inactivity. Cold starts take 5-10 minutes. To prepare the service, we recommend visiting the /docs endpoint ( https://<Modal workspace>--example-vllm-openai-compatible-serve.modal.run/docs ) for your deployed LLM and wait for it to fully load before connecting your client. ​ Deploy FastAPI App and Pipecat pipeline to Modal Setup environment variables: Copy Ask AI cd server cp env.example .env # Modify .env to provide your service API Keys Alternatively, you can configure your Modal app to use secrets . Update the modal_url in server/src/bot_vllm.py to point to the URL you received from the self-serve LLM deployment in the previous step. From within the server directory, test the app locally: Copy Ask AI modal serve app.py Deploy to production: Copy Ask AI modal deploy app.py Note the endpoint URL produced from this deployment. It will look like: Copy Ask AI https:// {your-workspace} --pipecat-modal-fastapi-app.modal.run You’ll need this URL for the client’s app.js configuration mentioned in its README. ​ Launch your bots on Modal ​ Option 1: Direct Link Simply click on the URL displayed after running the server or deploy step to launch an agent and be redirected to a Daily room to talk with the launched bot. This will use the OpenAI pipeline. ​ Option 2: Connect via an RTVI Client Follow the instructions provided in the client folder’s README for building and running a custom client that connects to your Modal endpoint. The provided client includes a dropdown for choosing which bot pipeline to run. ​ Navigating your LLM, server, and Pipecat logs On your Modal dashboard , you should have two Apps listed under Live Apps: example-vllm-openai-compatible : This App contains the containers and logs used to run your self-hosted LLM. There will be just one App Function listed: serve . Click on this function to view logs for your LLM. pipecat-modal : This App contains the containers and logs used to run your connect endpoints and Pipecat pipelines. It will list two App Functions: fastapi_app : This function is running the endpoints that your client will interact with and initiate starting a new pipeline ( / , /connect , /status ). Click on this function to see logs for each endpoint hit. bot_runner : This function handles launching and running a bot pipeline. Click on this function to get a list of all pipeline runs and access each run’s logs. ​ Modal & Pipecat Tips In most other Pipecat examples, we use Popen to launch the pipeline process from the /connect endpoint. In this example, we use a Modal function instead. This allows us to run the pipelines using a separately defined Modal image as well as run each pipeline in an isolated container. For the FastAPI and most common Pipecat Pipeline containers, a default debian_slim CPU-only should be all that’s required to run. GPU containers are needed for self-hosted services. To minimize cold starts of the pipeline and reduce latency for users, set min_containers=1 on the Modal Function that launches the pipeline to ensure at least one warm instance of your function is always available. ​ Next steps Explore Modal's LLM Examples For next steps on running a self-hosted LLM and reducing latency, check out all of Modal’s LLM examples Example: Cerebrium On this page Install the Modal CLI Deploy a self-serve LLM Deploy FastAPI App and Pipecat pipeline to Modal Launch your bots on Modal Option 1: Direct Link Option 2: Connect via an RTVI Client Navigating your LLM, server, and Pipecat logs Modal & Pipecat Tips Next steps Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/deployment_modal_b6adc173.txt b/deployment_modal_b6adc173.txt
new file mode 100644
index 0000000000000000000000000000000000000000..eba0cfd3193606f7e0fbad574bf7a69512ca710c
--- /dev/null
+++ b/deployment_modal_b6adc173.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/guides/deployment/modal#launch-your-bots-on-modal
+Title: Example: Modal - Pipecat
+==================================================
+
+Example: Modal - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Deploying your bot Example: Modal Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Guides Fundamentals Context Management Custom FrameProcessor Detecting Idle Users Ending a Pipeline Function Calling Muting User Input Recording Audio Recording Transcripts Features Gemini Multimodal Live Metrics Noise cancellation with Krisp OpenAI Audio Models and APIs Pipecat Flows Telephony Overview Dial-in: WebRTC (Daily) Dial-in: WebRTC (Twilio + Daily) Dial-in: Twilio (Media Streams) Dialout: WebRTC (Daily) Deploying your bot Overview Deployment pattern Example: Pipecat Cloud Example: Fly.io Example: Cerebrium Example: Modal Modal is well-suited for Pipecat deployments because it handles container orchestration, scaling, and cold starts efficiently. This makes it a good choice for production Pipecat bots that need reliable performance. This guide walks through the Modal example included in the Pipecat repository, which follows the same deployment pattern . Modal example View the complete Modal deployment example in our GitHub repository ​ Install the Modal CLI Set up Modal Follow Modal’s official instructions for creating an account and setting up the CLI ​ Deploy a self-serve LLM Deploy Modal’s OpenAI-compatible LLM service: Copy Ask AI git clone https://github.com/modal-labs/modal-examples cd modal-examples modal deploy 06_gpu_and_ml/llm-serving/vllm_inference.py Refer to Modal’s guide and example for Deploying an OpenAI-compatible LLM service with vLLM for more details. Take note of the endpoint URL from the previous step, which will look like: Copy Ask AI https://{your-workspace}--example-vllm-openai-compatible-serve.modal.run You’ll need this for the bot_vllm.py file in the next section. The default Modal LLM example uses Llama-3.1 and will shut down after 15 minutes of inactivity. Cold starts take 5-10 minutes. To prepare the service, we recommend visiting the /docs endpoint ( https://<Modal workspace>--example-vllm-openai-compatible-serve.modal.run/docs ) for your deployed LLM and wait for it to fully load before connecting your client. ​ Deploy FastAPI App and Pipecat pipeline to Modal Setup environment variables: Copy Ask AI cd server cp env.example .env # Modify .env to provide your service API Keys Alternatively, you can configure your Modal app to use secrets . Update the modal_url in server/src/bot_vllm.py to point to the URL you received from the self-serve LLM deployment in the previous step. From within the server directory, test the app locally: Copy Ask AI modal serve app.py Deploy to production: Copy Ask AI modal deploy app.py Note the endpoint URL produced from this deployment. It will look like: Copy Ask AI https:// {your-workspace} --pipecat-modal-fastapi-app.modal.run You’ll need this URL for the client’s app.js configuration mentioned in its README. ​ Launch your bots on Modal ​ Option 1: Direct Link Simply click on the URL displayed after running the server or deploy step to launch an agent and be redirected to a Daily room to talk with the launched bot. This will use the OpenAI pipeline. ​ Option 2: Connect via an RTVI Client Follow the instructions provided in the client folder’s README for building and running a custom client that connects to your Modal endpoint. The provided client includes a dropdown for choosing which bot pipeline to run. ​ Navigating your LLM, server, and Pipecat logs On your Modal dashboard , you should have two Apps listed under Live Apps: example-vllm-openai-compatible : This App contains the containers and logs used to run your self-hosted LLM. There will be just one App Function listed: serve . Click on this function to view logs for your LLM. pipecat-modal : This App contains the containers and logs used to run your connect endpoints and Pipecat pipelines. It will list two App Functions: fastapi_app : This function is running the endpoints that your client will interact with and initiate starting a new pipeline ( / , /connect , /status ). Click on this function to see logs for each endpoint hit. bot_runner : This function handles launching and running a bot pipeline. Click on this function to get a list of all pipeline runs and access each run’s logs. ​ Modal & Pipecat Tips In most other Pipecat examples, we use Popen to launch the pipeline process from the /connect endpoint. In this example, we use a Modal function instead. This allows us to run the pipelines using a separately defined Modal image as well as run each pipeline in an isolated container. For the FastAPI and most common Pipecat Pipeline containers, a default debian_slim CPU-only should be all that’s required to run. GPU containers are needed for self-hosted services. To minimize cold starts of the pipeline and reduce latency for users, set min_containers=1 on the Modal Function that launches the pipeline to ensure at least one warm instance of your function is always available. ​ Next steps Explore Modal's LLM Examples For next steps on running a self-hosted LLM and reducing latency, check out all of Modal’s LLM examples Example: Cerebrium On this page Install the Modal CLI Deploy a self-serve LLM Deploy FastAPI App and Pipecat pipeline to Modal Launch your bots on Modal Option 1: Direct Link Option 2: Connect via an RTVI Client Navigating your LLM, server, and Pipecat logs Modal & Pipecat Tips Next steps Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/deployment_overview_6aafd2fa.txt b/deployment_overview_6aafd2fa.txt
new file mode 100644
index 0000000000000000000000000000000000000000..eeda03e9c120b4058c55fa7ff8e551cd51febbb5
--- /dev/null
+++ b/deployment_overview_6aafd2fa.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/guides/deployment/overview#introduction-to-pipecat-deployment
+Title: Overview - Pipecat
+==================================================
+
+Overview - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Deploying your bot Overview Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Guides Fundamentals Context Management Custom FrameProcessor Detecting Idle Users Ending a Pipeline Function Calling Muting User Input Recording Audio Recording Transcripts Features Gemini Multimodal Live Metrics Noise cancellation with Krisp OpenAI Audio Models and APIs Pipecat Flows Telephony Overview Dial-in: WebRTC (Daily) Dial-in: WebRTC (Twilio + Daily) Dial-in: Twilio (Media Streams) Dialout: WebRTC (Daily) Deploying your bot Overview Deployment pattern Example: Pipecat Cloud Example: Fly.io Example: Cerebrium Example: Modal ​ Introduction to Pipecat deployment You’ve created your Pipecat bot, had a good chat with it locally, and are eager to share it with the world. Let’s explore how to approach deployment. We’re continually adding further deployment example projects to the Pipecat repo, which you can find here . ​ Deployment options You have several options for deploying your Pipecat bot: Pipecat Cloud - A purpose-built managed service designed specifically for Pipecat deployments Self-managed cloud deployment - Deploy to providers like Fly.io, AWS, Google Cloud Run, etc. Custom infrastructure - Run on your own servers or specialized AI infrastructure The best choice depends on your specific requirements, scale, and expertise. ​ Things you’ll need Transport service - Pipecat has existing services for various different media transport modes, such as WebRTC or WebSockets. If you’re not using a third-party service for handling media transport, you’ll want to make sure that infrastructure is hosted and ready to receive connections. Deployment target - You can deploy and run Pipecat bots anywhere that can run Python code - Google Cloud Run, AWS, Fly.io etc. We recommend providers that offer APIs, so you can programmatically spawn new bot agents on-demand. Docker - If you’re targeting cloud architecture / VMs, they will most often expect a containerized app. It’s worth having Docker installed and setup to run builds. We’ll step through creating a Dockerfile in this documentation. ​ Production-ready bots In local development, things often work great as you’re testing on controlled, stable network conditions. In real-world use-cases, however, your users will likely interact with your bot across a variety of different devices and network conditions. WebSockets are fine for server-to-server communication or for initial development. But for production use, you’ll likely want client-server audio that uses a protocol designed for real-time media transport. For an explanation of the difference between WebSockets and WebRTC, see this post . If you’re targeting scalable, client-server interactions, we recommend you use WebRTC for the best results. ​ Supporting models Most chatbots require very little in the way of system resources, but if you are making use of custom models or require GPU-powered infrastructure, it’s important to consider how to pre-cache local resources so that they are not downloaded at runtime. Your bot processes / VMs should aim to launch and connect as quickly as possible, so the user is not left waiting. Designing and operating a pool of workers is out of scope for our documentation, but we’ll highlight best practices in all of our examples. As an example of a supporting model, most Pipecat examples make use of Silero VAD which we recommend including as part of your Docker image (so it’s cached and readily available when your bot runs.) Since the Silero model is quite small, this doesn’t inflate the size of the container too much. You may, however, want to consider making large models availabile via a network volume and ensuring your bot knows where to find it. For Silero specifically, you can read more about how to do download it directly here . Copy Ask AI # Run at buildtime torch.hub.load( repo_or_dir = 'snakers4/silero-vad' , model = 'silero_vad' , force_reload = True ) ​ Getting started Basic deployment pattern Introduction to a model for deploying Pipecat bots ​ Provider guides Once you’ve familiarized yourself with the Pipecat deployment pattern , here are some guides that walk you through the process for various deployment options. Remember, your Pipecat bots are simply Python processes, so you can host them on whichever infrastructure or service best suits your project. Pipecat Cloud Managed service purpose-built for Pipecat deployments Fly.io For service-driven / CPU bots Cerebrium For GPU-accelerated models & specialized AI infrastructure Dialout: WebRTC (Daily) Deployment pattern On this page Introduction to Pipecat deployment Deployment options Things you’ll need Production-ready bots Supporting models Getting started Provider guides Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/deployment_overview_72e99db5.txt b/deployment_overview_72e99db5.txt
new file mode 100644
index 0000000000000000000000000000000000000000..5af9a544f462f21ccc36221be25be017b12fd0fc
--- /dev/null
+++ b/deployment_overview_72e99db5.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/guides/deployment/overview#provider-guides
+Title: Overview - Pipecat
+==================================================
+
+Overview - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Deploying your bot Overview Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Guides Fundamentals Context Management Custom FrameProcessor Detecting Idle Users Ending a Pipeline Function Calling Muting User Input Recording Audio Recording Transcripts Features Gemini Multimodal Live Metrics Noise cancellation with Krisp OpenAI Audio Models and APIs Pipecat Flows Telephony Overview Dial-in: WebRTC (Daily) Dial-in: WebRTC (Twilio + Daily) Dial-in: Twilio (Media Streams) Dialout: WebRTC (Daily) Deploying your bot Overview Deployment pattern Example: Pipecat Cloud Example: Fly.io Example: Cerebrium Example: Modal ​ Introduction to Pipecat deployment You’ve created your Pipecat bot, had a good chat with it locally, and are eager to share it with the world. Let’s explore how to approach deployment. We’re continually adding further deployment example projects to the Pipecat repo, which you can find here . ​ Deployment options You have several options for deploying your Pipecat bot: Pipecat Cloud - A purpose-built managed service designed specifically for Pipecat deployments Self-managed cloud deployment - Deploy to providers like Fly.io, AWS, Google Cloud Run, etc. Custom infrastructure - Run on your own servers or specialized AI infrastructure The best choice depends on your specific requirements, scale, and expertise. ​ Things you’ll need Transport service - Pipecat has existing services for various different media transport modes, such as WebRTC or WebSockets. If you’re not using a third-party service for handling media transport, you’ll want to make sure that infrastructure is hosted and ready to receive connections. Deployment target - You can deploy and run Pipecat bots anywhere that can run Python code - Google Cloud Run, AWS, Fly.io etc. We recommend providers that offer APIs, so you can programmatically spawn new bot agents on-demand. Docker - If you’re targeting cloud architecture / VMs, they will most often expect a containerized app. It’s worth having Docker installed and setup to run builds. We’ll step through creating a Dockerfile in this documentation. ​ Production-ready bots In local development, things often work great as you’re testing on controlled, stable network conditions. In real-world use-cases, however, your users will likely interact with your bot across a variety of different devices and network conditions. WebSockets are fine for server-to-server communication or for initial development. But for production use, you’ll likely want client-server audio that uses a protocol designed for real-time media transport. For an explanation of the difference between WebSockets and WebRTC, see this post . If you’re targeting scalable, client-server interactions, we recommend you use WebRTC for the best results. ​ Supporting models Most chatbots require very little in the way of system resources, but if you are making use of custom models or require GPU-powered infrastructure, it’s important to consider how to pre-cache local resources so that they are not downloaded at runtime. Your bot processes / VMs should aim to launch and connect as quickly as possible, so the user is not left waiting. Designing and operating a pool of workers is out of scope for our documentation, but we’ll highlight best practices in all of our examples. As an example of a supporting model, most Pipecat examples make use of Silero VAD which we recommend including as part of your Docker image (so it’s cached and readily available when your bot runs.) Since the Silero model is quite small, this doesn’t inflate the size of the container too much. You may, however, want to consider making large models availabile via a network volume and ensuring your bot knows where to find it. For Silero specifically, you can read more about how to do download it directly here . Copy Ask AI # Run at buildtime torch.hub.load( repo_or_dir = 'snakers4/silero-vad' , model = 'silero_vad' , force_reload = True ) ​ Getting started Basic deployment pattern Introduction to a model for deploying Pipecat bots ​ Provider guides Once you’ve familiarized yourself with the Pipecat deployment pattern , here are some guides that walk you through the process for various deployment options. Remember, your Pipecat bots are simply Python processes, so you can host them on whichever infrastructure or service best suits your project. Pipecat Cloud Managed service purpose-built for Pipecat deployments Fly.io For service-driven / CPU bots Cerebrium For GPU-accelerated models & specialized AI infrastructure Dialout: WebRTC (Daily) Deployment pattern On this page Introduction to Pipecat deployment Deployment options Things you’ll need Production-ready bots Supporting models Getting started Provider guides Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/deployment_overview_9b81ec12.txt b/deployment_overview_9b81ec12.txt
new file mode 100644
index 0000000000000000000000000000000000000000..5af105b8266d23f87b6ad465cf748075edd6f088
--- /dev/null
+++ b/deployment_overview_9b81ec12.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/guides/deployment/overview#deployment-options
+Title: Overview - Pipecat
+==================================================
+
+Overview - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Deploying your bot Overview Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Guides Fundamentals Context Management Custom FrameProcessor Detecting Idle Users Ending a Pipeline Function Calling Muting User Input Recording Audio Recording Transcripts Features Gemini Multimodal Live Metrics Noise cancellation with Krisp OpenAI Audio Models and APIs Pipecat Flows Telephony Overview Dial-in: WebRTC (Daily) Dial-in: WebRTC (Twilio + Daily) Dial-in: Twilio (Media Streams) Dialout: WebRTC (Daily) Deploying your bot Overview Deployment pattern Example: Pipecat Cloud Example: Fly.io Example: Cerebrium Example: Modal ​ Introduction to Pipecat deployment You’ve created your Pipecat bot, had a good chat with it locally, and are eager to share it with the world. Let’s explore how to approach deployment. We’re continually adding further deployment example projects to the Pipecat repo, which you can find here . ​ Deployment options You have several options for deploying your Pipecat bot: Pipecat Cloud - A purpose-built managed service designed specifically for Pipecat deployments Self-managed cloud deployment - Deploy to providers like Fly.io, AWS, Google Cloud Run, etc. Custom infrastructure - Run on your own servers or specialized AI infrastructure The best choice depends on your specific requirements, scale, and expertise. ​ Things you’ll need Transport service - Pipecat has existing services for various different media transport modes, such as WebRTC or WebSockets. If you’re not using a third-party service for handling media transport, you’ll want to make sure that infrastructure is hosted and ready to receive connections. Deployment target - You can deploy and run Pipecat bots anywhere that can run Python code - Google Cloud Run, AWS, Fly.io etc. We recommend providers that offer APIs, so you can programmatically spawn new bot agents on-demand. Docker - If you’re targeting cloud architecture / VMs, they will most often expect a containerized app. It’s worth having Docker installed and setup to run builds. We’ll step through creating a Dockerfile in this documentation. ​ Production-ready bots In local development, things often work great as you’re testing on controlled, stable network conditions. In real-world use-cases, however, your users will likely interact with your bot across a variety of different devices and network conditions. WebSockets are fine for server-to-server communication or for initial development. But for production use, you’ll likely want client-server audio that uses a protocol designed for real-time media transport. For an explanation of the difference between WebSockets and WebRTC, see this post . If you’re targeting scalable, client-server interactions, we recommend you use WebRTC for the best results. ​ Supporting models Most chatbots require very little in the way of system resources, but if you are making use of custom models or require GPU-powered infrastructure, it’s important to consider how to pre-cache local resources so that they are not downloaded at runtime. Your bot processes / VMs should aim to launch and connect as quickly as possible, so the user is not left waiting. Designing and operating a pool of workers is out of scope for our documentation, but we’ll highlight best practices in all of our examples. As an example of a supporting model, most Pipecat examples make use of Silero VAD which we recommend including as part of your Docker image (so it’s cached and readily available when your bot runs.) Since the Silero model is quite small, this doesn’t inflate the size of the container too much. You may, however, want to consider making large models availabile via a network volume and ensuring your bot knows where to find it. For Silero specifically, you can read more about how to do download it directly here . Copy Ask AI # Run at buildtime torch.hub.load( repo_or_dir = 'snakers4/silero-vad' , model = 'silero_vad' , force_reload = True ) ​ Getting started Basic deployment pattern Introduction to a model for deploying Pipecat bots ​ Provider guides Once you’ve familiarized yourself with the Pipecat deployment pattern , here are some guides that walk you through the process for various deployment options. Remember, your Pipecat bots are simply Python processes, so you can host them on whichever infrastructure or service best suits your project. Pipecat Cloud Managed service purpose-built for Pipecat deployments Fly.io For service-driven / CPU bots Cerebrium For GPU-accelerated models & specialized AI infrastructure Dialout: WebRTC (Daily) Deployment pattern On this page Introduction to Pipecat deployment Deployment options Things you’ll need Production-ready bots Supporting models Getting started Provider guides Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/deployment_overview_bf24934b.txt b/deployment_overview_bf24934b.txt
new file mode 100644
index 0000000000000000000000000000000000000000..1e17b4b4ac828d338936a537d324091d0ea1d7ea
--- /dev/null
+++ b/deployment_overview_bf24934b.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/guides/deployment/overview#things-you%E2%80%99ll-need
+Title: Overview - Pipecat
+==================================================
+
+Overview - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Deploying your bot Overview Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Guides Fundamentals Context Management Custom FrameProcessor Detecting Idle Users Ending a Pipeline Function Calling Muting User Input Recording Audio Recording Transcripts Features Gemini Multimodal Live Metrics Noise cancellation with Krisp OpenAI Audio Models and APIs Pipecat Flows Telephony Overview Dial-in: WebRTC (Daily) Dial-in: WebRTC (Twilio + Daily) Dial-in: Twilio (Media Streams) Dialout: WebRTC (Daily) Deploying your bot Overview Deployment pattern Example: Pipecat Cloud Example: Fly.io Example: Cerebrium Example: Modal ​ Introduction to Pipecat deployment You’ve created your Pipecat bot, had a good chat with it locally, and are eager to share it with the world. Let’s explore how to approach deployment. We’re continually adding further deployment example projects to the Pipecat repo, which you can find here . ​ Deployment options You have several options for deploying your Pipecat bot: Pipecat Cloud - A purpose-built managed service designed specifically for Pipecat deployments Self-managed cloud deployment - Deploy to providers like Fly.io, AWS, Google Cloud Run, etc. Custom infrastructure - Run on your own servers or specialized AI infrastructure The best choice depends on your specific requirements, scale, and expertise. ​ Things you’ll need Transport service - Pipecat has existing services for various different media transport modes, such as WebRTC or WebSockets. If you’re not using a third-party service for handling media transport, you’ll want to make sure that infrastructure is hosted and ready to receive connections. Deployment target - You can deploy and run Pipecat bots anywhere that can run Python code - Google Cloud Run, AWS, Fly.io etc. We recommend providers that offer APIs, so you can programmatically spawn new bot agents on-demand. Docker - If you’re targeting cloud architecture / VMs, they will most often expect a containerized app. It’s worth having Docker installed and setup to run builds. We’ll step through creating a Dockerfile in this documentation. ​ Production-ready bots In local development, things often work great as you’re testing on controlled, stable network conditions. In real-world use-cases, however, your users will likely interact with your bot across a variety of different devices and network conditions. WebSockets are fine for server-to-server communication or for initial development. But for production use, you’ll likely want client-server audio that uses a protocol designed for real-time media transport. For an explanation of the difference between WebSockets and WebRTC, see this post . If you’re targeting scalable, client-server interactions, we recommend you use WebRTC for the best results. ​ Supporting models Most chatbots require very little in the way of system resources, but if you are making use of custom models or require GPU-powered infrastructure, it’s important to consider how to pre-cache local resources so that they are not downloaded at runtime. Your bot processes / VMs should aim to launch and connect as quickly as possible, so the user is not left waiting. Designing and operating a pool of workers is out of scope for our documentation, but we’ll highlight best practices in all of our examples. As an example of a supporting model, most Pipecat examples make use of Silero VAD which we recommend including as part of your Docker image (so it’s cached and readily available when your bot runs.) Since the Silero model is quite small, this doesn’t inflate the size of the container too much. You may, however, want to consider making large models availabile via a network volume and ensuring your bot knows where to find it. For Silero specifically, you can read more about how to do download it directly here . Copy Ask AI # Run at buildtime torch.hub.load( repo_or_dir = 'snakers4/silero-vad' , model = 'silero_vad' , force_reload = True ) ​ Getting started Basic deployment pattern Introduction to a model for deploying Pipecat bots ​ Provider guides Once you’ve familiarized yourself with the Pipecat deployment pattern , here are some guides that walk you through the process for various deployment options. Remember, your Pipecat bots are simply Python processes, so you can host them on whichever infrastructure or service best suits your project. Pipecat Cloud Managed service purpose-built for Pipecat deployments Fly.io For service-driven / CPU bots Cerebrium For GPU-accelerated models & specialized AI infrastructure Dialout: WebRTC (Daily) Deployment pattern On this page Introduction to Pipecat deployment Deployment options Things you’ll need Production-ready bots Supporting models Getting started Provider guides Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/deployment_pattern_2e1ba9aa.txt b/deployment_pattern_2e1ba9aa.txt
new file mode 100644
index 0000000000000000000000000000000000000000..9f54024caf6fcd388d7269fc3cfffece2cc2246e
--- /dev/null
+++ b/deployment_pattern_2e1ba9aa.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/guides/deployment/pattern
+Title: Deployment pattern - Pipecat
+==================================================
+
+Deployment pattern - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Deploying your bot Deployment pattern Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Guides Fundamentals Context Management Custom FrameProcessor Detecting Idle Users Ending a Pipeline Function Calling Muting User Input Recording Audio Recording Transcripts Features Gemini Multimodal Live Metrics Noise cancellation with Krisp OpenAI Audio Models and APIs Pipecat Flows Telephony Overview Dial-in: WebRTC (Daily) Dial-in: WebRTC (Twilio + Daily) Dial-in: Twilio (Media Streams) Dialout: WebRTC (Daily) Deploying your bot Overview Deployment pattern Example: Pipecat Cloud Example: Fly.io Example: Cerebrium Example: Modal ​ Project structure A Pipecat project will often consist of the following: 1. Bot file E.g. bot.py . Your Pipecat bot / agent, containing all the pipelines that you want to run in order to communicate with an end-user. A bot file may take some command line arguments, such as a transport URL and configuration. 2. Bot runner E.g. bot_runner.py . Typically a basic HTTP service that listens for incoming user requests and spawns the relevant bot file in response. You can call these files whatever you like! We use bot.py and bot_runner.py for simplicity. ​ Typical user / bot flow There are many ways to approach connecting users to bots. Pipecat is unopinionated about how exactly you should do this, but it’s helpful to put an idea forward. At a very basic level, it may look something like this: 1 User requests to join session via client / app Client initiates a HTTP request to a hosted bot runner service. 2 Bot runner handles the request Authenticates, configures and instantiates everything necessary for the session to commence (e.g. a new WebSocket channel, or WebRTC room, etc.) 3 Bot runner spawns bot / agent A new bot process / VM is created for the user to connect with (passing across any necessary configuration.) Your project may load just one bot file, contextually swap between multiple, or launch many at once. 4 Bot instantiates and joins session via specified transport credentials Bot initializes, connects to the session (e.g. locally or via WebSockets, WebRTC etc) and runs your bot code. 5 Bot runner returns status to client Once the bot is ready, the runner resolves the HTTP request with details for the client to connect. ​ Bot runner The majority of use-cases require a way to trigger and manage a bot session over the internet. We call these bot runners; a HTTP service that provides a gateway for spawning bots on-demand. The anatomy of a bot runner service is entirery arbitrary, but at very least will have a method that spawns a new bot process, for example: Copy Ask AI import uvicorn from fastapi import FastAPI, Request, HTTPException from fastapi.responses import JSONResponse app = FastAPI() @app.post ( "/start_bot" ) async def start_bot ( request : Request) -> JSONResponse: # ... handle / authenticate the request # ... setup the transport session # Spawn a new bot process try : #... create a new bot instance except Exception as e: raise HTTPException( status_code = 500 , detail = f "Failed to start bot: { e } " ) # Return a URL for the user to join return JSONResponse({ ... }) if __name__ == "__main__" : uvicorn.run( "bot_runner:app" , host = "0.0.0.0" , port = 7860 ) This pseudo code defines a /start_bot/ endpoint which listens for incoming user POST requests or webhooks, then configures the session (such as creating rooms on your transport provider) and instantiates a new bot process. A client will typically require some information regarding the newly spawned bot, such as a web address, so we also return some JSON with the necessary details. ​ Data transport Your transport layer is responsible for sending and receiving audio and video data over the internet. You will have implemented a transport layer as part of your bot.py pipeline. This may be a service that you want to host and include in your deployment, or it may be a third-party service waiting for peers to connect (such as Daily , or a websocket.) For this example, we will make use of Daily’s WebRTC transport. This will mean that our bot_runner.py will need to do some configuration when it spawns a new bot: Create and configure a new Daily room for the session to take place in. Issue both the bot and the user an authentication token to join the session. Whatever you use for your transport layer, you’ll likely need to setup some environmental variables and run some custom code before spawning the agent. ​ Best practice for bot files A good pattern to work to is the assumption that your bot.py is an encapsulated entity and does not have any knowledge of the bot_runner.py . You should provide the bot everything it needs to operate during instantiation. Sticking to this approach helps keep things simple and makes it easier to step through debugging (if the bot launched and something goes wrong, you know to look for errors in your bot file.) ​ Example Let’s assume we have a fully service-driven bot.py that connects to a WebRTC session, passes audio transcription to GPT4 and returns audio text-to-speech with ElevenLabs. We’ll also use Silero voice activity detection, to better know when the user has stopped talking. bot.py Copy Ask AI import asyncio import aiohttp import os import sys import argparse from pipecat.pipeline.pipeline import Pipeline from pipecat.pipeline.runner import PipelineRunner from pipecat.pipeline.task import PipelineParams, PipelineTask from pipecat.processors.aggregators.llm_response import LLMAssistantResponseAggregator, LLMUserResponseAggregator from pipecat.frames.frames import LLMMessagesFrame, EndFrame from pipecat.services.openai.llm import OpenAILLMService from pipecat.services.elevenlabs.tts import ElevenLabsTTSService from pipecat.transports.services.daily import DailyParams, DailyTransport from pipecat.vad.silero import SileroVADAnalyzer from loguru import logger from dotenv import load_dotenv load_dotenv( override = True ) logger.remove( 0 ) logger.add(sys.stderr, level = "DEBUG" ) daily_api_key = os.getenv( "DAILY_API_KEY" , "" ) daily_api_url = os.getenv( "DAILY_API_URL" , "https://api.daily.co/v1" ) async def main ( room_url : str , token : str ): async with aiohttp.ClientSession() as session: transport = DailyTransport( room_url, token, "Chatbot" , DailyParams( api_url = daily_api_url, api_key = daily_api_key, audio_in_enabled = True , audio_out_enabled = True , video_out_enabled = False , vad_analyzer = SileroVADAnalyzer(), transcription_enabled = True , ) ) tts = ElevenLabsTTSService( aiohttp_session = session, api_key = os.getenv( "ELEVENLABS_API_KEY" , "" ), voice_id = os.getenv( "ELEVENLABS_VOICE_ID" , "" ), ) llm = OpenAILLMService( api_key = os.getenv( "OPENAI_API_KEY" ), model = "gpt-4o" ) messages = [ { "role" : "system" , "content" : "You are Chatbot, a friendly, helpful robot. Your output will be converted to audio so don't include special characters other than '!' or '?' in your answers. Respond to what the user said in a creative and helpful way, but keep your responses brief. Start by saying hello." , }, ] tma_in = LLMUserResponseAggregator(messages) tma_out = LLMAssistantResponseAggregator(messages) pipeline = Pipeline([ transport.input(), tma_in, llm, tts, transport.output(), tma_out, ]) task = PipelineTask(pipeline, params = PipelineParams( allow_interruptions = True )) @transport.event_handler ( "on_first_participant_joined" ) async def on_first_participant_joined ( transport , participant ): transport.capture_participant_transcription(participant[ "id" ]) await task.queue_frames([LLMMessagesFrame(messages)]) @transport.event_handler ( "on_participant_left" ) async def on_participant_left ( transport , participant , reason ): await task.queue_frame(EndFrame()) @transport.event_handler ( "on_call_state_updated" ) async def on_call_state_updated ( transport , state ): if state == "left" : await task.queue_frame(EndFrame()) runner = PipelineRunner() await runner.run(task) if __name__ == "__main__" : parser = argparse.ArgumentParser( description = "Pipecat Bot" ) parser.add_argument( "-u" , type = str , help = "Room URL" ) parser.add_argument( "-t" , type = str , help = "Token" ) config = parser.parse_args() asyncio.run(main(config.u, config.t)) ​ HTTP API To launch this bot, let’s create a bot_runner.py that: Creates an API for users to send requests to. Launches a bot as a subprocess. bot_runner.py Copy Ask AI import os import argparse import subprocess from pipecat.transports.services.helpers.daily_rest import DailyRESTHelper, DailyRoomObject, DailyRoomProperties, DailyRoomParams from fastapi import FastAPI, Request, HTTPException from fastapi.middleware.cors import CORSMiddleware from fastapi.responses import JSONResponse # Load API keys from env from dotenv import load_dotenv load_dotenv( override = True ) # ------------ Configuration ------------ # MAX_SESSION_TIME = 5 * 60 # 5 minutes # List of require env vars our bot requires REQUIRED_ENV_VARS = [ 'DAILY_API_KEY' , 'OPENAI_API_KEY' , 'ELEVENLABS_API_KEY' , 'ELEVENLABS_VOICE_ID' ] daily_rest_helper = DailyRESTHelper( os.getenv( "DAILY_API_KEY" , "" ), os.getenv( "DAILY_API_URL" , 'https://api.daily.co/v1' )) # ----------------- API ----------------- # app = FastAPI() app.add_middleware( CORSMiddleware, allow_origins = [ "*" ], allow_credentials = True , allow_methods = [ "*" ], allow_headers = [ "*" ] ) # ----------------- Main ----------------- # @app.post ( "/start_bot" ) async def start_bot ( request : Request) -> JSONResponse: try : # Grab any data included in the post request data = await request.json() except Exception as e: pass # Create a new Daily WebRTC room for the session to take place in try : params = DailyRoomParams( properties = DailyRoomProperties() ) room: DailyRoomObject = daily_rest_helper.create_room( params = params) except Exception as e: raise HTTPException( status_code = 500 , detail = f "Unable to provision room { e } " ) # Give the agent a token to join the session token = daily_rest_helper.get_token(room.url, MAX_SESSION_TIME ) # Return an error if we were unable to create a room or a token if not room or not token: raise HTTPException( status_code = 500 , detail = f "Failed to get token for room: { room_url } " ) try : # Start a new subprocess, passing the room and token to the bot file subprocess.Popen( [ f "python3 -m bot -u { room.url } -t { token } " ], shell = True , bufsize = 1 , cwd = os.path.dirname(os.path.abspath( __file__ ))) except Exception as e: raise HTTPException( status_code = 500 , detail = f "Failed to start subprocess: { e } " ) # Grab a token for the user to join with user_token = daily_rest_helper.get_token(room.url, MAX_SESSION_TIME ) # Return the room url and user token back to the user return JSONResponse({ "room_url" : room.url, "token" : user_token, }) if __name__ == "__main__" : # Check for required environment variables for env_var in REQUIRED_ENV_VARS : if env_var not in os.environ: raise Exception ( f "Missing environment variable: { env_var } ." ) parser = argparse.ArgumentParser( description = "Pipecat Bot Runner" ) parser.add_argument( "--host" , type = str , default = os.getenv( "HOST" , "0.0.0.0" ), help = "Host address" ) parser.add_argument( "--port" , type = int , default = os.getenv( "PORT" , 7860 ), help = "Port number" ) parser.add_argument( "--reload" , action = "store_true" , default = False , help = "Reload code on change" ) config = parser.parse_args() try : import uvicorn uvicorn.run( "bot_runner:app" , host = config.host, port = config.port, reload = config.reload ) except KeyboardInterrupt : print ( "Pipecat runner shutting down..." ) ​ Dockerfile Since our bot is just using Python, our Dockerfile can be quite simple: Dockerfile install_deps.py Copy Ask AI FROM python:3.11-bullseye # Open port 7860 for http service ENV FAST_API_PORT= 7860 EXPOSE 7860 # Install Python dependencies COPY \* .py . COPY ./requirements.txt requirements.txt RUN pip3 install --no-cache-dir --upgrade -r requirements.txt # Install models RUN python3 install_deps.py # Start the FastAPI server CMD python3 bot_runner.py --port ${ FAST_API_PORT } The bot runner and bot requirements.txt : requirements.txt Copy Ask AI pipecat-ai[daily,openai,silero] fastapi uvicorn python-dotenv And finally, let’s create a .env file with our service keys .env Copy Ask AI DAILY_API_KEY = ... OPENAI_API_KEY = ... ELEVENLABS_API_KEY = ... ELEVENLABS_VOICE_ID = ... ​ How it works Right now, this runner is spawning bot.py as a subprocess. When spawning the process, we pass through the transport room and token as system arguments to our bot, so it knows where to connect. Subprocesses serve as a great way to test out your bot in the cloud without too much hassle, but depending on the size of the host machine, it will likely not hold up well under load. Whilst some bots are just simple operators between the transport and third-party AI services (such as OpenAI), others have somewhat CPU-intensive operations, such as running and loading VAD models, so you may find you’re only able to scale this to support up to 5-10 concurrent bots. Scaling your setup would require virtualizing your bot with it’s own set of system resources, the process of which depends on your cloud provider. ​ Best practices In an ideal world, we’d recommend containerizing your bot and bot runner independently so you can deploy each without any unnecessary dependencies or models. Most cloud providers will offer a way to deploy various images programmatically, which we explore in the various provider examples in these docs. For the sake of simplicity defining this pattern, we’re just using one container for everything. ​ Build and run We should now have a project that contains the following files: bot.py bot_runner.py requirements.txt .env Dockerfile You can now docker build ... and deploy your container. Of course, you can still work with your bot in local development too: Copy Ask AI # Install and activate a virtual env python -m venv venv source venv/bin/activate # or OS equivalent pip install -r requirements.txt python bot_runner.py --host localhost --reload Overview Example: Pipecat Cloud On this page Project structure Typical user / bot flow Bot runner Data transport Best practice for bot files Example HTTP API Dockerfile How it works Best practices Build and run Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/deployment_pattern_9efb7c01.txt b/deployment_pattern_9efb7c01.txt
new file mode 100644
index 0000000000000000000000000000000000000000..ee69d5d0ade9418cca74db2273dc9ced4fbc8392
--- /dev/null
+++ b/deployment_pattern_9efb7c01.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/guides/deployment/pattern#best-practices
+Title: Deployment pattern - Pipecat
+==================================================
+
+Deployment pattern - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Deploying your bot Deployment pattern Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Guides Fundamentals Context Management Custom FrameProcessor Detecting Idle Users Ending a Pipeline Function Calling Muting User Input Recording Audio Recording Transcripts Features Gemini Multimodal Live Metrics Noise cancellation with Krisp OpenAI Audio Models and APIs Pipecat Flows Telephony Overview Dial-in: WebRTC (Daily) Dial-in: WebRTC (Twilio + Daily) Dial-in: Twilio (Media Streams) Dialout: WebRTC (Daily) Deploying your bot Overview Deployment pattern Example: Pipecat Cloud Example: Fly.io Example: Cerebrium Example: Modal ​ Project structure A Pipecat project will often consist of the following: 1. Bot file E.g. bot.py . Your Pipecat bot / agent, containing all the pipelines that you want to run in order to communicate with an end-user. A bot file may take some command line arguments, such as a transport URL and configuration. 2. Bot runner E.g. bot_runner.py . Typically a basic HTTP service that listens for incoming user requests and spawns the relevant bot file in response. You can call these files whatever you like! We use bot.py and bot_runner.py for simplicity. ​ Typical user / bot flow There are many ways to approach connecting users to bots. Pipecat is unopinionated about how exactly you should do this, but it’s helpful to put an idea forward. At a very basic level, it may look something like this: 1 User requests to join session via client / app Client initiates a HTTP request to a hosted bot runner service. 2 Bot runner handles the request Authenticates, configures and instantiates everything necessary for the session to commence (e.g. a new WebSocket channel, or WebRTC room, etc.) 3 Bot runner spawns bot / agent A new bot process / VM is created for the user to connect with (passing across any necessary configuration.) Your project may load just one bot file, contextually swap between multiple, or launch many at once. 4 Bot instantiates and joins session via specified transport credentials Bot initializes, connects to the session (e.g. locally or via WebSockets, WebRTC etc) and runs your bot code. 5 Bot runner returns status to client Once the bot is ready, the runner resolves the HTTP request with details for the client to connect. ​ Bot runner The majority of use-cases require a way to trigger and manage a bot session over the internet. We call these bot runners; a HTTP service that provides a gateway for spawning bots on-demand. The anatomy of a bot runner service is entirery arbitrary, but at very least will have a method that spawns a new bot process, for example: Copy Ask AI import uvicorn from fastapi import FastAPI, Request, HTTPException from fastapi.responses import JSONResponse app = FastAPI() @app.post ( "/start_bot" ) async def start_bot ( request : Request) -> JSONResponse: # ... handle / authenticate the request # ... setup the transport session # Spawn a new bot process try : #... create a new bot instance except Exception as e: raise HTTPException( status_code = 500 , detail = f "Failed to start bot: { e } " ) # Return a URL for the user to join return JSONResponse({ ... }) if __name__ == "__main__" : uvicorn.run( "bot_runner:app" , host = "0.0.0.0" , port = 7860 ) This pseudo code defines a /start_bot/ endpoint which listens for incoming user POST requests or webhooks, then configures the session (such as creating rooms on your transport provider) and instantiates a new bot process. A client will typically require some information regarding the newly spawned bot, such as a web address, so we also return some JSON with the necessary details. ​ Data transport Your transport layer is responsible for sending and receiving audio and video data over the internet. You will have implemented a transport layer as part of your bot.py pipeline. This may be a service that you want to host and include in your deployment, or it may be a third-party service waiting for peers to connect (such as Daily , or a websocket.) For this example, we will make use of Daily’s WebRTC transport. This will mean that our bot_runner.py will need to do some configuration when it spawns a new bot: Create and configure a new Daily room for the session to take place in. Issue both the bot and the user an authentication token to join the session. Whatever you use for your transport layer, you’ll likely need to setup some environmental variables and run some custom code before spawning the agent. ​ Best practice for bot files A good pattern to work to is the assumption that your bot.py is an encapsulated entity and does not have any knowledge of the bot_runner.py . You should provide the bot everything it needs to operate during instantiation. Sticking to this approach helps keep things simple and makes it easier to step through debugging (if the bot launched and something goes wrong, you know to look for errors in your bot file.) ​ Example Let’s assume we have a fully service-driven bot.py that connects to a WebRTC session, passes audio transcription to GPT4 and returns audio text-to-speech with ElevenLabs. We’ll also use Silero voice activity detection, to better know when the user has stopped talking. bot.py Copy Ask AI import asyncio import aiohttp import os import sys import argparse from pipecat.pipeline.pipeline import Pipeline from pipecat.pipeline.runner import PipelineRunner from pipecat.pipeline.task import PipelineParams, PipelineTask from pipecat.processors.aggregators.llm_response import LLMAssistantResponseAggregator, LLMUserResponseAggregator from pipecat.frames.frames import LLMMessagesFrame, EndFrame from pipecat.services.openai.llm import OpenAILLMService from pipecat.services.elevenlabs.tts import ElevenLabsTTSService from pipecat.transports.services.daily import DailyParams, DailyTransport from pipecat.vad.silero import SileroVADAnalyzer from loguru import logger from dotenv import load_dotenv load_dotenv( override = True ) logger.remove( 0 ) logger.add(sys.stderr, level = "DEBUG" ) daily_api_key = os.getenv( "DAILY_API_KEY" , "" ) daily_api_url = os.getenv( "DAILY_API_URL" , "https://api.daily.co/v1" ) async def main ( room_url : str , token : str ): async with aiohttp.ClientSession() as session: transport = DailyTransport( room_url, token, "Chatbot" , DailyParams( api_url = daily_api_url, api_key = daily_api_key, audio_in_enabled = True , audio_out_enabled = True , video_out_enabled = False , vad_analyzer = SileroVADAnalyzer(), transcription_enabled = True , ) ) tts = ElevenLabsTTSService( aiohttp_session = session, api_key = os.getenv( "ELEVENLABS_API_KEY" , "" ), voice_id = os.getenv( "ELEVENLABS_VOICE_ID" , "" ), ) llm = OpenAILLMService( api_key = os.getenv( "OPENAI_API_KEY" ), model = "gpt-4o" ) messages = [ { "role" : "system" , "content" : "You are Chatbot, a friendly, helpful robot. Your output will be converted to audio so don't include special characters other than '!' or '?' in your answers. Respond to what the user said in a creative and helpful way, but keep your responses brief. Start by saying hello." , }, ] tma_in = LLMUserResponseAggregator(messages) tma_out = LLMAssistantResponseAggregator(messages) pipeline = Pipeline([ transport.input(), tma_in, llm, tts, transport.output(), tma_out, ]) task = PipelineTask(pipeline, params = PipelineParams( allow_interruptions = True )) @transport.event_handler ( "on_first_participant_joined" ) async def on_first_participant_joined ( transport , participant ): transport.capture_participant_transcription(participant[ "id" ]) await task.queue_frames([LLMMessagesFrame(messages)]) @transport.event_handler ( "on_participant_left" ) async def on_participant_left ( transport , participant , reason ): await task.queue_frame(EndFrame()) @transport.event_handler ( "on_call_state_updated" ) async def on_call_state_updated ( transport , state ): if state == "left" : await task.queue_frame(EndFrame()) runner = PipelineRunner() await runner.run(task) if __name__ == "__main__" : parser = argparse.ArgumentParser( description = "Pipecat Bot" ) parser.add_argument( "-u" , type = str , help = "Room URL" ) parser.add_argument( "-t" , type = str , help = "Token" ) config = parser.parse_args() asyncio.run(main(config.u, config.t)) ​ HTTP API To launch this bot, let’s create a bot_runner.py that: Creates an API for users to send requests to. Launches a bot as a subprocess. bot_runner.py Copy Ask AI import os import argparse import subprocess from pipecat.transports.services.helpers.daily_rest import DailyRESTHelper, DailyRoomObject, DailyRoomProperties, DailyRoomParams from fastapi import FastAPI, Request, HTTPException from fastapi.middleware.cors import CORSMiddleware from fastapi.responses import JSONResponse # Load API keys from env from dotenv import load_dotenv load_dotenv( override = True ) # ------------ Configuration ------------ # MAX_SESSION_TIME = 5 * 60 # 5 minutes # List of require env vars our bot requires REQUIRED_ENV_VARS = [ 'DAILY_API_KEY' , 'OPENAI_API_KEY' , 'ELEVENLABS_API_KEY' , 'ELEVENLABS_VOICE_ID' ] daily_rest_helper = DailyRESTHelper( os.getenv( "DAILY_API_KEY" , "" ), os.getenv( "DAILY_API_URL" , 'https://api.daily.co/v1' )) # ----------------- API ----------------- # app = FastAPI() app.add_middleware( CORSMiddleware, allow_origins = [ "*" ], allow_credentials = True , allow_methods = [ "*" ], allow_headers = [ "*" ] ) # ----------------- Main ----------------- # @app.post ( "/start_bot" ) async def start_bot ( request : Request) -> JSONResponse: try : # Grab any data included in the post request data = await request.json() except Exception as e: pass # Create a new Daily WebRTC room for the session to take place in try : params = DailyRoomParams( properties = DailyRoomProperties() ) room: DailyRoomObject = daily_rest_helper.create_room( params = params) except Exception as e: raise HTTPException( status_code = 500 , detail = f "Unable to provision room { e } " ) # Give the agent a token to join the session token = daily_rest_helper.get_token(room.url, MAX_SESSION_TIME ) # Return an error if we were unable to create a room or a token if not room or not token: raise HTTPException( status_code = 500 , detail = f "Failed to get token for room: { room_url } " ) try : # Start a new subprocess, passing the room and token to the bot file subprocess.Popen( [ f "python3 -m bot -u { room.url } -t { token } " ], shell = True , bufsize = 1 , cwd = os.path.dirname(os.path.abspath( __file__ ))) except Exception as e: raise HTTPException( status_code = 500 , detail = f "Failed to start subprocess: { e } " ) # Grab a token for the user to join with user_token = daily_rest_helper.get_token(room.url, MAX_SESSION_TIME ) # Return the room url and user token back to the user return JSONResponse({ "room_url" : room.url, "token" : user_token, }) if __name__ == "__main__" : # Check for required environment variables for env_var in REQUIRED_ENV_VARS : if env_var not in os.environ: raise Exception ( f "Missing environment variable: { env_var } ." ) parser = argparse.ArgumentParser( description = "Pipecat Bot Runner" ) parser.add_argument( "--host" , type = str , default = os.getenv( "HOST" , "0.0.0.0" ), help = "Host address" ) parser.add_argument( "--port" , type = int , default = os.getenv( "PORT" , 7860 ), help = "Port number" ) parser.add_argument( "--reload" , action = "store_true" , default = False , help = "Reload code on change" ) config = parser.parse_args() try : import uvicorn uvicorn.run( "bot_runner:app" , host = config.host, port = config.port, reload = config.reload ) except KeyboardInterrupt : print ( "Pipecat runner shutting down..." ) ​ Dockerfile Since our bot is just using Python, our Dockerfile can be quite simple: Dockerfile install_deps.py Copy Ask AI FROM python:3.11-bullseye # Open port 7860 for http service ENV FAST_API_PORT= 7860 EXPOSE 7860 # Install Python dependencies COPY \* .py . COPY ./requirements.txt requirements.txt RUN pip3 install --no-cache-dir --upgrade -r requirements.txt # Install models RUN python3 install_deps.py # Start the FastAPI server CMD python3 bot_runner.py --port ${ FAST_API_PORT } The bot runner and bot requirements.txt : requirements.txt Copy Ask AI pipecat-ai[daily,openai,silero] fastapi uvicorn python-dotenv And finally, let’s create a .env file with our service keys .env Copy Ask AI DAILY_API_KEY = ... OPENAI_API_KEY = ... ELEVENLABS_API_KEY = ... ELEVENLABS_VOICE_ID = ... ​ How it works Right now, this runner is spawning bot.py as a subprocess. When spawning the process, we pass through the transport room and token as system arguments to our bot, so it knows where to connect. Subprocesses serve as a great way to test out your bot in the cloud without too much hassle, but depending on the size of the host machine, it will likely not hold up well under load. Whilst some bots are just simple operators between the transport and third-party AI services (such as OpenAI), others have somewhat CPU-intensive operations, such as running and loading VAD models, so you may find you’re only able to scale this to support up to 5-10 concurrent bots. Scaling your setup would require virtualizing your bot with it’s own set of system resources, the process of which depends on your cloud provider. ​ Best practices In an ideal world, we’d recommend containerizing your bot and bot runner independently so you can deploy each without any unnecessary dependencies or models. Most cloud providers will offer a way to deploy various images programmatically, which we explore in the various provider examples in these docs. For the sake of simplicity defining this pattern, we’re just using one container for everything. ​ Build and run We should now have a project that contains the following files: bot.py bot_runner.py requirements.txt .env Dockerfile You can now docker build ... and deploy your container. Of course, you can still work with your bot in local development too: Copy Ask AI # Install and activate a virtual env python -m venv venv source venv/bin/activate # or OS equivalent pip install -r requirements.txt python bot_runner.py --host localhost --reload Overview Example: Pipecat Cloud On this page Project structure Typical user / bot flow Bot runner Data transport Best practice for bot files Example HTTP API Dockerfile How it works Best practices Build and run Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/deployment_pattern_cdd80bfe.txt b/deployment_pattern_cdd80bfe.txt
new file mode 100644
index 0000000000000000000000000000000000000000..3f284b50b15e848fdbd150a392d3b452e8b1b93f
--- /dev/null
+++ b/deployment_pattern_cdd80bfe.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/guides/deployment/pattern#how-it-works
+Title: Deployment pattern - Pipecat
+==================================================
+
+Deployment pattern - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Deploying your bot Deployment pattern Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Guides Fundamentals Context Management Custom FrameProcessor Detecting Idle Users Ending a Pipeline Function Calling Muting User Input Recording Audio Recording Transcripts Features Gemini Multimodal Live Metrics Noise cancellation with Krisp OpenAI Audio Models and APIs Pipecat Flows Telephony Overview Dial-in: WebRTC (Daily) Dial-in: WebRTC (Twilio + Daily) Dial-in: Twilio (Media Streams) Dialout: WebRTC (Daily) Deploying your bot Overview Deployment pattern Example: Pipecat Cloud Example: Fly.io Example: Cerebrium Example: Modal ​ Project structure A Pipecat project will often consist of the following: 1. Bot file E.g. bot.py . Your Pipecat bot / agent, containing all the pipelines that you want to run in order to communicate with an end-user. A bot file may take some command line arguments, such as a transport URL and configuration. 2. Bot runner E.g. bot_runner.py . Typically a basic HTTP service that listens for incoming user requests and spawns the relevant bot file in response. You can call these files whatever you like! We use bot.py and bot_runner.py for simplicity. ​ Typical user / bot flow There are many ways to approach connecting users to bots. Pipecat is unopinionated about how exactly you should do this, but it’s helpful to put an idea forward. At a very basic level, it may look something like this: 1 User requests to join session via client / app Client initiates a HTTP request to a hosted bot runner service. 2 Bot runner handles the request Authenticates, configures and instantiates everything necessary for the session to commence (e.g. a new WebSocket channel, or WebRTC room, etc.) 3 Bot runner spawns bot / agent A new bot process / VM is created for the user to connect with (passing across any necessary configuration.) Your project may load just one bot file, contextually swap between multiple, or launch many at once. 4 Bot instantiates and joins session via specified transport credentials Bot initializes, connects to the session (e.g. locally or via WebSockets, WebRTC etc) and runs your bot code. 5 Bot runner returns status to client Once the bot is ready, the runner resolves the HTTP request with details for the client to connect. ​ Bot runner The majority of use-cases require a way to trigger and manage a bot session over the internet. We call these bot runners; a HTTP service that provides a gateway for spawning bots on-demand. The anatomy of a bot runner service is entirery arbitrary, but at very least will have a method that spawns a new bot process, for example: Copy Ask AI import uvicorn from fastapi import FastAPI, Request, HTTPException from fastapi.responses import JSONResponse app = FastAPI() @app.post ( "/start_bot" ) async def start_bot ( request : Request) -> JSONResponse: # ... handle / authenticate the request # ... setup the transport session # Spawn a new bot process try : #... create a new bot instance except Exception as e: raise HTTPException( status_code = 500 , detail = f "Failed to start bot: { e } " ) # Return a URL for the user to join return JSONResponse({ ... }) if __name__ == "__main__" : uvicorn.run( "bot_runner:app" , host = "0.0.0.0" , port = 7860 ) This pseudo code defines a /start_bot/ endpoint which listens for incoming user POST requests or webhooks, then configures the session (such as creating rooms on your transport provider) and instantiates a new bot process. A client will typically require some information regarding the newly spawned bot, such as a web address, so we also return some JSON with the necessary details. ​ Data transport Your transport layer is responsible for sending and receiving audio and video data over the internet. You will have implemented a transport layer as part of your bot.py pipeline. This may be a service that you want to host and include in your deployment, or it may be a third-party service waiting for peers to connect (such as Daily , or a websocket.) For this example, we will make use of Daily’s WebRTC transport. This will mean that our bot_runner.py will need to do some configuration when it spawns a new bot: Create and configure a new Daily room for the session to take place in. Issue both the bot and the user an authentication token to join the session. Whatever you use for your transport layer, you’ll likely need to setup some environmental variables and run some custom code before spawning the agent. ​ Best practice for bot files A good pattern to work to is the assumption that your bot.py is an encapsulated entity and does not have any knowledge of the bot_runner.py . You should provide the bot everything it needs to operate during instantiation. Sticking to this approach helps keep things simple and makes it easier to step through debugging (if the bot launched and something goes wrong, you know to look for errors in your bot file.) ​ Example Let’s assume we have a fully service-driven bot.py that connects to a WebRTC session, passes audio transcription to GPT4 and returns audio text-to-speech with ElevenLabs. We’ll also use Silero voice activity detection, to better know when the user has stopped talking. bot.py Copy Ask AI import asyncio import aiohttp import os import sys import argparse from pipecat.pipeline.pipeline import Pipeline from pipecat.pipeline.runner import PipelineRunner from pipecat.pipeline.task import PipelineParams, PipelineTask from pipecat.processors.aggregators.llm_response import LLMAssistantResponseAggregator, LLMUserResponseAggregator from pipecat.frames.frames import LLMMessagesFrame, EndFrame from pipecat.services.openai.llm import OpenAILLMService from pipecat.services.elevenlabs.tts import ElevenLabsTTSService from pipecat.transports.services.daily import DailyParams, DailyTransport from pipecat.vad.silero import SileroVADAnalyzer from loguru import logger from dotenv import load_dotenv load_dotenv( override = True ) logger.remove( 0 ) logger.add(sys.stderr, level = "DEBUG" ) daily_api_key = os.getenv( "DAILY_API_KEY" , "" ) daily_api_url = os.getenv( "DAILY_API_URL" , "https://api.daily.co/v1" ) async def main ( room_url : str , token : str ): async with aiohttp.ClientSession() as session: transport = DailyTransport( room_url, token, "Chatbot" , DailyParams( api_url = daily_api_url, api_key = daily_api_key, audio_in_enabled = True , audio_out_enabled = True , video_out_enabled = False , vad_analyzer = SileroVADAnalyzer(), transcription_enabled = True , ) ) tts = ElevenLabsTTSService( aiohttp_session = session, api_key = os.getenv( "ELEVENLABS_API_KEY" , "" ), voice_id = os.getenv( "ELEVENLABS_VOICE_ID" , "" ), ) llm = OpenAILLMService( api_key = os.getenv( "OPENAI_API_KEY" ), model = "gpt-4o" ) messages = [ { "role" : "system" , "content" : "You are Chatbot, a friendly, helpful robot. Your output will be converted to audio so don't include special characters other than '!' or '?' in your answers. Respond to what the user said in a creative and helpful way, but keep your responses brief. Start by saying hello." , }, ] tma_in = LLMUserResponseAggregator(messages) tma_out = LLMAssistantResponseAggregator(messages) pipeline = Pipeline([ transport.input(), tma_in, llm, tts, transport.output(), tma_out, ]) task = PipelineTask(pipeline, params = PipelineParams( allow_interruptions = True )) @transport.event_handler ( "on_first_participant_joined" ) async def on_first_participant_joined ( transport , participant ): transport.capture_participant_transcription(participant[ "id" ]) await task.queue_frames([LLMMessagesFrame(messages)]) @transport.event_handler ( "on_participant_left" ) async def on_participant_left ( transport , participant , reason ): await task.queue_frame(EndFrame()) @transport.event_handler ( "on_call_state_updated" ) async def on_call_state_updated ( transport , state ): if state == "left" : await task.queue_frame(EndFrame()) runner = PipelineRunner() await runner.run(task) if __name__ == "__main__" : parser = argparse.ArgumentParser( description = "Pipecat Bot" ) parser.add_argument( "-u" , type = str , help = "Room URL" ) parser.add_argument( "-t" , type = str , help = "Token" ) config = parser.parse_args() asyncio.run(main(config.u, config.t)) ​ HTTP API To launch this bot, let’s create a bot_runner.py that: Creates an API for users to send requests to. Launches a bot as a subprocess. bot_runner.py Copy Ask AI import os import argparse import subprocess from pipecat.transports.services.helpers.daily_rest import DailyRESTHelper, DailyRoomObject, DailyRoomProperties, DailyRoomParams from fastapi import FastAPI, Request, HTTPException from fastapi.middleware.cors import CORSMiddleware from fastapi.responses import JSONResponse # Load API keys from env from dotenv import load_dotenv load_dotenv( override = True ) # ------------ Configuration ------------ # MAX_SESSION_TIME = 5 * 60 # 5 minutes # List of require env vars our bot requires REQUIRED_ENV_VARS = [ 'DAILY_API_KEY' , 'OPENAI_API_KEY' , 'ELEVENLABS_API_KEY' , 'ELEVENLABS_VOICE_ID' ] daily_rest_helper = DailyRESTHelper( os.getenv( "DAILY_API_KEY" , "" ), os.getenv( "DAILY_API_URL" , 'https://api.daily.co/v1' )) # ----------------- API ----------------- # app = FastAPI() app.add_middleware( CORSMiddleware, allow_origins = [ "*" ], allow_credentials = True , allow_methods = [ "*" ], allow_headers = [ "*" ] ) # ----------------- Main ----------------- # @app.post ( "/start_bot" ) async def start_bot ( request : Request) -> JSONResponse: try : # Grab any data included in the post request data = await request.json() except Exception as e: pass # Create a new Daily WebRTC room for the session to take place in try : params = DailyRoomParams( properties = DailyRoomProperties() ) room: DailyRoomObject = daily_rest_helper.create_room( params = params) except Exception as e: raise HTTPException( status_code = 500 , detail = f "Unable to provision room { e } " ) # Give the agent a token to join the session token = daily_rest_helper.get_token(room.url, MAX_SESSION_TIME ) # Return an error if we were unable to create a room or a token if not room or not token: raise HTTPException( status_code = 500 , detail = f "Failed to get token for room: { room_url } " ) try : # Start a new subprocess, passing the room and token to the bot file subprocess.Popen( [ f "python3 -m bot -u { room.url } -t { token } " ], shell = True , bufsize = 1 , cwd = os.path.dirname(os.path.abspath( __file__ ))) except Exception as e: raise HTTPException( status_code = 500 , detail = f "Failed to start subprocess: { e } " ) # Grab a token for the user to join with user_token = daily_rest_helper.get_token(room.url, MAX_SESSION_TIME ) # Return the room url and user token back to the user return JSONResponse({ "room_url" : room.url, "token" : user_token, }) if __name__ == "__main__" : # Check for required environment variables for env_var in REQUIRED_ENV_VARS : if env_var not in os.environ: raise Exception ( f "Missing environment variable: { env_var } ." ) parser = argparse.ArgumentParser( description = "Pipecat Bot Runner" ) parser.add_argument( "--host" , type = str , default = os.getenv( "HOST" , "0.0.0.0" ), help = "Host address" ) parser.add_argument( "--port" , type = int , default = os.getenv( "PORT" , 7860 ), help = "Port number" ) parser.add_argument( "--reload" , action = "store_true" , default = False , help = "Reload code on change" ) config = parser.parse_args() try : import uvicorn uvicorn.run( "bot_runner:app" , host = config.host, port = config.port, reload = config.reload ) except KeyboardInterrupt : print ( "Pipecat runner shutting down..." ) ​ Dockerfile Since our bot is just using Python, our Dockerfile can be quite simple: Dockerfile install_deps.py Copy Ask AI FROM python:3.11-bullseye # Open port 7860 for http service ENV FAST_API_PORT= 7860 EXPOSE 7860 # Install Python dependencies COPY \* .py . COPY ./requirements.txt requirements.txt RUN pip3 install --no-cache-dir --upgrade -r requirements.txt # Install models RUN python3 install_deps.py # Start the FastAPI server CMD python3 bot_runner.py --port ${ FAST_API_PORT } The bot runner and bot requirements.txt : requirements.txt Copy Ask AI pipecat-ai[daily,openai,silero] fastapi uvicorn python-dotenv And finally, let’s create a .env file with our service keys .env Copy Ask AI DAILY_API_KEY = ... OPENAI_API_KEY = ... ELEVENLABS_API_KEY = ... ELEVENLABS_VOICE_ID = ... ​ How it works Right now, this runner is spawning bot.py as a subprocess. When spawning the process, we pass through the transport room and token as system arguments to our bot, so it knows where to connect. Subprocesses serve as a great way to test out your bot in the cloud without too much hassle, but depending on the size of the host machine, it will likely not hold up well under load. Whilst some bots are just simple operators between the transport and third-party AI services (such as OpenAI), others have somewhat CPU-intensive operations, such as running and loading VAD models, so you may find you’re only able to scale this to support up to 5-10 concurrent bots. Scaling your setup would require virtualizing your bot with it’s own set of system resources, the process of which depends on your cloud provider. ​ Best practices In an ideal world, we’d recommend containerizing your bot and bot runner independently so you can deploy each without any unnecessary dependencies or models. Most cloud providers will offer a way to deploy various images programmatically, which we explore in the various provider examples in these docs. For the sake of simplicity defining this pattern, we’re just using one container for everything. ​ Build and run We should now have a project that contains the following files: bot.py bot_runner.py requirements.txt .env Dockerfile You can now docker build ... and deploy your container. Of course, you can still work with your bot in local development too: Copy Ask AI # Install and activate a virtual env python -m venv venv source venv/bin/activate # or OS equivalent pip install -r requirements.txt python bot_runner.py --host localhost --reload Overview Example: Pipecat Cloud On this page Project structure Typical user / bot flow Bot runner Data transport Best practice for bot files Example HTTP API Dockerfile How it works Best practices Build and run Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/deployment_pattern_fda49ccf.txt b/deployment_pattern_fda49ccf.txt
new file mode 100644
index 0000000000000000000000000000000000000000..9ac44149bef7dea7f4a85bb8514ea594dbddbb83
--- /dev/null
+++ b/deployment_pattern_fda49ccf.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/deployment/pattern#join-our-community
+Title: Overview - Pipecat
+==================================================
+
+Overview - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Get Started Overview Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Get Started Overview Installation & Setup Quickstart Core Concepts Next Steps & Examples Pipecat is an open source Python framework that handles the complex orchestration of AI services, network transport, audio processing, and multimodal interactions. “Multimodal” means you can use any combination of audio, video, images, and/or text in your interactions. And “real-time” means that things are happening quickly enough that it feels conversational—a “back-and-forth” with a bot, not submitting a query and waiting for results. ​ What You Can Build Voice Assistants Natural, real-time conversations with AI using speech recognition and synthesis Interactive Agents Personal coaches and meeting assistants that can understand context and provide guidance Multimodal Apps Applications that combine voice, video, images, and text for rich interactions Creative Tools Storytelling experiences and social companions that engage users Business Solutions Customer intake flows and support bots for automated business processes Complex Flows Structured conversations using Pipecat Flows for managing complex interactions ​ How It Works The flow of interactions in a Pipecat application is typically straightforward: The bot says something The user says something The bot says something The user says something This continues until the conversation naturally ends. While this flow seems simple, making it feel natural requires sophisticated real-time processing. ​ Real-time Processing Pipecat’s pipeline architecture handles both simple voice interactions and complex multimodal processing. Let’s look at how data flows through the system: Voice app Multimodal app 1 Send Audio Transmit and capture streamed audio from the user 2 Transcribe Speech Convert speech to text as the user is talking 3 Process with LLM Generate responses using a large language model 4 Convert to Speech Transform text responses into natural speech 5 Play Audio Stream the audio response back to the user 1 Send Audio Transmit and capture streamed audio from the user 2 Transcribe Speech Convert speech to text as the user is talking 3 Process with LLM Generate responses using a large language model 4 Convert to Speech Transform text responses into natural speech 5 Play Audio Stream the audio response back to the user 1 Send Audio and Video Transmit and capture audio, video, and image inputs simultaneously 2 Process Streams Handle multiple input streams in parallel 3 Model Processing Send combined inputs to multimodal models (like GPT-4V) 4 Generate Outputs Create various outputs (text, images, audio, etc.) 5 Coordinate Presentation Synchronize and present multiple output types In both cases, Pipecat: Processes responses as they stream in Handles multiple input/output modalities concurrently Manages resource allocation and synchronization Coordinates parallel processing tasks This architecture creates fluid, natural interactions without noticeable delays, whether you’re building a simple voice assistant or a complex multimodal application. Pipecat’s pipeline architecture is particularly valuable for managing the complexity of real-time, multimodal interactions, ensuring smooth data flow and proper synchronization regardless of the input/output types involved. Pipecat handles all this complexity for you, letting you focus on building your application rather than managing the underlying infrastructure. ​ Next Steps Ready to build your first Pipecat application? Installation & Setup Prepare your environment and install required dependencies Quickstart Build and run your first Pipecat application Core Concepts Learn about pipelines, frames, and real-time processing Use Cases Explore example implementations and patterns ​ Join Our Community Discord Community Connect with other developers, share your projects, and get support from the Pipecat team. Installation & Setup On this page What You Can Build How It Works Real-time Processing Next Steps Join Our Community Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/deployment_pipecat-cloud_0193ac3e.txt b/deployment_pipecat-cloud_0193ac3e.txt
new file mode 100644
index 0000000000000000000000000000000000000000..6be9fab5a66357ce11709568b736d4012b3fe5f7
--- /dev/null
+++ b/deployment_pipecat-cloud_0193ac3e.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/guides/deployment/pipecat-cloud#checking-deployment-status
+Title: Example: Pipecat Cloud - Pipecat
+==================================================
+
+Example: Pipecat Cloud - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Deploying your bot Example: Pipecat Cloud Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Guides Fundamentals Context Management Custom FrameProcessor Detecting Idle Users Ending a Pipeline Function Calling Muting User Input Recording Audio Recording Transcripts Features Gemini Multimodal Live Metrics Noise cancellation with Krisp OpenAI Audio Models and APIs Pipecat Flows Telephony Overview Dial-in: WebRTC (Daily) Dial-in: WebRTC (Twilio + Daily) Dial-in: Twilio (Media Streams) Dialout: WebRTC (Daily) Deploying your bot Overview Deployment pattern Example: Pipecat Cloud Example: Fly.io Example: Cerebrium Example: Modal Pipecat Cloud is a managed platform for hosting and scaling Pipecat agents in production. ​ Prerequisites Before you begin, you’ll need: A Pipecat Cloud account Docker installed Python 3.10+ The Pipecat Cloud CLI: pip install pipecatcloud Quickstart Guide Follow a step-by-step guided experience to deploy your first agent ​ Choosing a starting point Pipecat Cloud offers several ways to get started: Use a starter template - Pre-built agent configurations for common use cases Build from the base image - Create a custom agent using the official base image Clone the starter project - A bare-bones project template to customize ​ Starter templates Pipecat Cloud provides several ready-made templates for common agent types: Template Description voice Voice conversation agent with STT, LLM and TTS twilio Telephony agent that works with Twilio natural_conversation Agent focused on natural dialogue flow, allowing a user time to think openai_realtime Agent using OpenAI’s Realtime API gemini_multimodal_live Multimodal agent using Google’s Gemini Multimodal Live API vision Computer vision agent that can analyze images These templates include a functioning implementation and Dockerfile. You can use them directly: Copy Ask AI # Clone the repository git clone https://github.com/daily-co/pipecat-cloud-images.git # Navigate to a starter template cd pipecat-cloud-images/pipecat-starters/voice # Customize the agent for your needs ​ Project structure Whether using a starter template or building from scratch, a basic Pipecat Cloud project typically includes: Copy Ask AI my-agent/ ├── bot.py # Your Pipecat pipeline ├── Dockerfile # Container definition ├── requirements.txt # Python dependencies └── pcc-deploy.toml # Deployment config (optional) ​ Agent implementation with bot.py Your agent’s bot.py code must include a specific bot() function that serves as the entry point for Pipecat Cloud. This function has different signatures depending on the transport method: ​ For WebRTC/Daily transports Copy Ask AI async def bot ( args : DailySessionArguments): """Main bot entry point compatible with the FastAPI route handler. Args: config: The configuration object from the request body room_url: The Daily room URL token: The Daily room token session_id: The session ID for logging """ logger.info( f "Bot process initialized { args.room_url } { args.token } " ) try : await main(args.room_url, args.token) logger.info( "Bot process completed" ) except Exception as e: logger.exception( f "Error in bot process: { str (e) } " ) raise ​ For WebSocket transports (e.g., Twilio) Copy Ask AI async def bot ( args : WebSocketSessionArguments): """Main bot entry point for WebSocket connections. Args: ws: The WebSocket connection session_id: The session ID for logging """ logger.info( "WebSocket bot process initialized" ) try : await main(args.websocket) logger.info( "WebSocket bot process completed" ) except Exception as e: logger.exception( f "Error in WebSocket bot process: { str (e) } " ) raise Complete Example: Voice Bot View a complete WebRTC voice bot implementation Complete Example: Twilio Bot View a complete Twilio WebSocket bot implementation ​ Example Dockerfile Pipecat Cloud provides base images that include common dependencies for Pipecat agents: Copy Ask AI FROM dailyco/pipecat-base:latest COPY ./requirements.txt requirements.txt RUN pip install --no-cache-dir --upgrade -r requirements.txt COPY ./bot.py bot.py This Dockerfile: Uses the official Pipecat base image Installs Python dependencies from requirements.txt Copies your bot.py file to the container The base image (dailyco/pipecat-base) includes the HTTP API server, session management, and platform integration required to run on Pipecat Cloud. See the base image source code for details. ​ Building and pushing your Docker image With your project structure in place, build and push your Docker image: Copy Ask AI # Build the image docker build --platform=linux/arm64 -t my-first-agent:latest . # Tag the image for your repository docker tag my-first-agent:latest your-username/my-first-agent:0.1 # Push the tagged image to the repository docker push your-username/my-first-agent:0.1 Pipecat Cloud requires ARM64 images. Make sure to specify the --platform=linux/arm64 flag when building. ​ Managing secrets Your agent likely requires API keys and other credentials. Create a secret set to store them securely: Copy Ask AI # Create an .env file with your credentials touch .env # Create a secret set from the file pcc secrets set my-first-agent-secrets --file .env ​ Deploying your agent Deploy your agent with the CLI: Copy Ask AI pcc deploy my-first-agent your-username/my-first-agent:0.1 --secret-set my-first-agent-secrets For a more maintainable approach, create a pcc-deploy.toml file: Copy Ask AI agent_name = "my-first-agent" image = "your-username/my-first-agent:0.1" secret_set = "my-first-agent-secrets" image_credentials = "my-first-agent-image-credentials" # For private repos [ scaling ] min_instances = 0 Then deploy using: Copy Ask AI pcc deploy ​ Starting a session Once deployed, you can start a session with your agent: Copy Ask AI # Create and set a public access key if needed pcc organizations keys create pcc organizations keys use # Start a session using Daily for WebRTC pcc agent start my-first-agent --use-daily This will open a Daily room where you can interact with your agent. ​ Checking deployment status Monitor your agent deployment: Copy Ask AI # Check deployment status pcc agent status my-first-agent # View deployment logs pcc agent logs my-first-agent ​ Next steps Agent Images Learn about containerizing your agent Secrets Managing sensitive information Scaling Configure scaling for production workloads Active Sessions Understand how sessions work Deployment pattern Example: Fly.io On this page Prerequisites Choosing a starting point Starter templates Project structure Agent implementation with bot.py For WebRTC/Daily transports For WebSocket transports (e.g., Twilio) Example Dockerfile Building and pushing your Docker image Managing secrets Deploying your agent Starting a session Checking deployment status Next steps Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/deployment_pipecat-cloud_2798e7f6.txt b/deployment_pipecat-cloud_2798e7f6.txt
new file mode 100644
index 0000000000000000000000000000000000000000..204ec05877ff14412265d27cace34f59e9076f50
--- /dev/null
+++ b/deployment_pipecat-cloud_2798e7f6.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/guides/deployment/pipecat-cloud#agent-implementation-with-bot-py
+Title: Example: Pipecat Cloud - Pipecat
+==================================================
+
+Example: Pipecat Cloud - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Deploying your bot Example: Pipecat Cloud Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Guides Fundamentals Context Management Custom FrameProcessor Detecting Idle Users Ending a Pipeline Function Calling Muting User Input Recording Audio Recording Transcripts Features Gemini Multimodal Live Metrics Noise cancellation with Krisp OpenAI Audio Models and APIs Pipecat Flows Telephony Overview Dial-in: WebRTC (Daily) Dial-in: WebRTC (Twilio + Daily) Dial-in: Twilio (Media Streams) Dialout: WebRTC (Daily) Deploying your bot Overview Deployment pattern Example: Pipecat Cloud Example: Fly.io Example: Cerebrium Example: Modal Pipecat Cloud is a managed platform for hosting and scaling Pipecat agents in production. ​ Prerequisites Before you begin, you’ll need: A Pipecat Cloud account Docker installed Python 3.10+ The Pipecat Cloud CLI: pip install pipecatcloud Quickstart Guide Follow a step-by-step guided experience to deploy your first agent ​ Choosing a starting point Pipecat Cloud offers several ways to get started: Use a starter template - Pre-built agent configurations for common use cases Build from the base image - Create a custom agent using the official base image Clone the starter project - A bare-bones project template to customize ​ Starter templates Pipecat Cloud provides several ready-made templates for common agent types: Template Description voice Voice conversation agent with STT, LLM and TTS twilio Telephony agent that works with Twilio natural_conversation Agent focused on natural dialogue flow, allowing a user time to think openai_realtime Agent using OpenAI’s Realtime API gemini_multimodal_live Multimodal agent using Google’s Gemini Multimodal Live API vision Computer vision agent that can analyze images These templates include a functioning implementation and Dockerfile. You can use them directly: Copy Ask AI # Clone the repository git clone https://github.com/daily-co/pipecat-cloud-images.git # Navigate to a starter template cd pipecat-cloud-images/pipecat-starters/voice # Customize the agent for your needs ​ Project structure Whether using a starter template or building from scratch, a basic Pipecat Cloud project typically includes: Copy Ask AI my-agent/ ├── bot.py # Your Pipecat pipeline ├── Dockerfile # Container definition ├── requirements.txt # Python dependencies └── pcc-deploy.toml # Deployment config (optional) ​ Agent implementation with bot.py Your agent’s bot.py code must include a specific bot() function that serves as the entry point for Pipecat Cloud. This function has different signatures depending on the transport method: ​ For WebRTC/Daily transports Copy Ask AI async def bot ( args : DailySessionArguments): """Main bot entry point compatible with the FastAPI route handler. Args: config: The configuration object from the request body room_url: The Daily room URL token: The Daily room token session_id: The session ID for logging """ logger.info( f "Bot process initialized { args.room_url } { args.token } " ) try : await main(args.room_url, args.token) logger.info( "Bot process completed" ) except Exception as e: logger.exception( f "Error in bot process: { str (e) } " ) raise ​ For WebSocket transports (e.g., Twilio) Copy Ask AI async def bot ( args : WebSocketSessionArguments): """Main bot entry point for WebSocket connections. Args: ws: The WebSocket connection session_id: The session ID for logging """ logger.info( "WebSocket bot process initialized" ) try : await main(args.websocket) logger.info( "WebSocket bot process completed" ) except Exception as e: logger.exception( f "Error in WebSocket bot process: { str (e) } " ) raise Complete Example: Voice Bot View a complete WebRTC voice bot implementation Complete Example: Twilio Bot View a complete Twilio WebSocket bot implementation ​ Example Dockerfile Pipecat Cloud provides base images that include common dependencies for Pipecat agents: Copy Ask AI FROM dailyco/pipecat-base:latest COPY ./requirements.txt requirements.txt RUN pip install --no-cache-dir --upgrade -r requirements.txt COPY ./bot.py bot.py This Dockerfile: Uses the official Pipecat base image Installs Python dependencies from requirements.txt Copies your bot.py file to the container The base image (dailyco/pipecat-base) includes the HTTP API server, session management, and platform integration required to run on Pipecat Cloud. See the base image source code for details. ​ Building and pushing your Docker image With your project structure in place, build and push your Docker image: Copy Ask AI # Build the image docker build --platform=linux/arm64 -t my-first-agent:latest . # Tag the image for your repository docker tag my-first-agent:latest your-username/my-first-agent:0.1 # Push the tagged image to the repository docker push your-username/my-first-agent:0.1 Pipecat Cloud requires ARM64 images. Make sure to specify the --platform=linux/arm64 flag when building. ​ Managing secrets Your agent likely requires API keys and other credentials. Create a secret set to store them securely: Copy Ask AI # Create an .env file with your credentials touch .env # Create a secret set from the file pcc secrets set my-first-agent-secrets --file .env ​ Deploying your agent Deploy your agent with the CLI: Copy Ask AI pcc deploy my-first-agent your-username/my-first-agent:0.1 --secret-set my-first-agent-secrets For a more maintainable approach, create a pcc-deploy.toml file: Copy Ask AI agent_name = "my-first-agent" image = "your-username/my-first-agent:0.1" secret_set = "my-first-agent-secrets" image_credentials = "my-first-agent-image-credentials" # For private repos [ scaling ] min_instances = 0 Then deploy using: Copy Ask AI pcc deploy ​ Starting a session Once deployed, you can start a session with your agent: Copy Ask AI # Create and set a public access key if needed pcc organizations keys create pcc organizations keys use # Start a session using Daily for WebRTC pcc agent start my-first-agent --use-daily This will open a Daily room where you can interact with your agent. ​ Checking deployment status Monitor your agent deployment: Copy Ask AI # Check deployment status pcc agent status my-first-agent # View deployment logs pcc agent logs my-first-agent ​ Next steps Agent Images Learn about containerizing your agent Secrets Managing sensitive information Scaling Configure scaling for production workloads Active Sessions Understand how sessions work Deployment pattern Example: Fly.io On this page Prerequisites Choosing a starting point Starter templates Project structure Agent implementation with bot.py For WebRTC/Daily transports For WebSocket transports (e.g., Twilio) Example Dockerfile Building and pushing your Docker image Managing secrets Deploying your agent Starting a session Checking deployment status Next steps Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/deployment_pipecat-cloud_6faceaf7.txt b/deployment_pipecat-cloud_6faceaf7.txt
new file mode 100644
index 0000000000000000000000000000000000000000..4b45a44b483911d5a5a14ab7e079c5c0851ef245
--- /dev/null
+++ b/deployment_pipecat-cloud_6faceaf7.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/deployment/pipecat-cloud#join-our-community
+Title: Overview - Pipecat
+==================================================
+
+Overview - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Get Started Overview Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Get Started Overview Installation & Setup Quickstart Core Concepts Next Steps & Examples Pipecat is an open source Python framework that handles the complex orchestration of AI services, network transport, audio processing, and multimodal interactions. “Multimodal” means you can use any combination of audio, video, images, and/or text in your interactions. And “real-time” means that things are happening quickly enough that it feels conversational—a “back-and-forth” with a bot, not submitting a query and waiting for results. ​ What You Can Build Voice Assistants Natural, real-time conversations with AI using speech recognition and synthesis Interactive Agents Personal coaches and meeting assistants that can understand context and provide guidance Multimodal Apps Applications that combine voice, video, images, and text for rich interactions Creative Tools Storytelling experiences and social companions that engage users Business Solutions Customer intake flows and support bots for automated business processes Complex Flows Structured conversations using Pipecat Flows for managing complex interactions ​ How It Works The flow of interactions in a Pipecat application is typically straightforward: The bot says something The user says something The bot says something The user says something This continues until the conversation naturally ends. While this flow seems simple, making it feel natural requires sophisticated real-time processing. ​ Real-time Processing Pipecat’s pipeline architecture handles both simple voice interactions and complex multimodal processing. Let’s look at how data flows through the system: Voice app Multimodal app 1 Send Audio Transmit and capture streamed audio from the user 2 Transcribe Speech Convert speech to text as the user is talking 3 Process with LLM Generate responses using a large language model 4 Convert to Speech Transform text responses into natural speech 5 Play Audio Stream the audio response back to the user 1 Send Audio Transmit and capture streamed audio from the user 2 Transcribe Speech Convert speech to text as the user is talking 3 Process with LLM Generate responses using a large language model 4 Convert to Speech Transform text responses into natural speech 5 Play Audio Stream the audio response back to the user 1 Send Audio and Video Transmit and capture audio, video, and image inputs simultaneously 2 Process Streams Handle multiple input streams in parallel 3 Model Processing Send combined inputs to multimodal models (like GPT-4V) 4 Generate Outputs Create various outputs (text, images, audio, etc.) 5 Coordinate Presentation Synchronize and present multiple output types In both cases, Pipecat: Processes responses as they stream in Handles multiple input/output modalities concurrently Manages resource allocation and synchronization Coordinates parallel processing tasks This architecture creates fluid, natural interactions without noticeable delays, whether you’re building a simple voice assistant or a complex multimodal application. Pipecat’s pipeline architecture is particularly valuable for managing the complexity of real-time, multimodal interactions, ensuring smooth data flow and proper synchronization regardless of the input/output types involved. Pipecat handles all this complexity for you, letting you focus on building your application rather than managing the underlying infrastructure. ​ Next Steps Ready to build your first Pipecat application? Installation & Setup Prepare your environment and install required dependencies Quickstart Build and run your first Pipecat application Core Concepts Learn about pipelines, frames, and real-time processing Use Cases Explore example implementations and patterns ​ Join Our Community Discord Community Connect with other developers, share your projects, and get support from the Pipecat team. Installation & Setup On this page What You Can Build How It Works Real-time Processing Next Steps Join Our Community Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/deployment_pipecat-cloud_885203fe.txt b/deployment_pipecat-cloud_885203fe.txt
new file mode 100644
index 0000000000000000000000000000000000000000..15f81a14d5db73f8f4ea7b653a54a86bc4083de1
--- /dev/null
+++ b/deployment_pipecat-cloud_885203fe.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/deployment/pipecat-cloud#how-it-works
+Title: Overview - Pipecat
+==================================================
+
+Overview - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Get Started Overview Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Get Started Overview Installation & Setup Quickstart Core Concepts Next Steps & Examples Pipecat is an open source Python framework that handles the complex orchestration of AI services, network transport, audio processing, and multimodal interactions. “Multimodal” means you can use any combination of audio, video, images, and/or text in your interactions. And “real-time” means that things are happening quickly enough that it feels conversational—a “back-and-forth” with a bot, not submitting a query and waiting for results. ​ What You Can Build Voice Assistants Natural, real-time conversations with AI using speech recognition and synthesis Interactive Agents Personal coaches and meeting assistants that can understand context and provide guidance Multimodal Apps Applications that combine voice, video, images, and text for rich interactions Creative Tools Storytelling experiences and social companions that engage users Business Solutions Customer intake flows and support bots for automated business processes Complex Flows Structured conversations using Pipecat Flows for managing complex interactions ​ How It Works The flow of interactions in a Pipecat application is typically straightforward: The bot says something The user says something The bot says something The user says something This continues until the conversation naturally ends. While this flow seems simple, making it feel natural requires sophisticated real-time processing. ​ Real-time Processing Pipecat’s pipeline architecture handles both simple voice interactions and complex multimodal processing. Let’s look at how data flows through the system: Voice app Multimodal app 1 Send Audio Transmit and capture streamed audio from the user 2 Transcribe Speech Convert speech to text as the user is talking 3 Process with LLM Generate responses using a large language model 4 Convert to Speech Transform text responses into natural speech 5 Play Audio Stream the audio response back to the user 1 Send Audio Transmit and capture streamed audio from the user 2 Transcribe Speech Convert speech to text as the user is talking 3 Process with LLM Generate responses using a large language model 4 Convert to Speech Transform text responses into natural speech 5 Play Audio Stream the audio response back to the user 1 Send Audio and Video Transmit and capture audio, video, and image inputs simultaneously 2 Process Streams Handle multiple input streams in parallel 3 Model Processing Send combined inputs to multimodal models (like GPT-4V) 4 Generate Outputs Create various outputs (text, images, audio, etc.) 5 Coordinate Presentation Synchronize and present multiple output types In both cases, Pipecat: Processes responses as they stream in Handles multiple input/output modalities concurrently Manages resource allocation and synchronization Coordinates parallel processing tasks This architecture creates fluid, natural interactions without noticeable delays, whether you’re building a simple voice assistant or a complex multimodal application. Pipecat’s pipeline architecture is particularly valuable for managing the complexity of real-time, multimodal interactions, ensuring smooth data flow and proper synchronization regardless of the input/output types involved. Pipecat handles all this complexity for you, letting you focus on building your application rather than managing the underlying infrastructure. ​ Next Steps Ready to build your first Pipecat application? Installation & Setup Prepare your environment and install required dependencies Quickstart Build and run your first Pipecat application Core Concepts Learn about pipelines, frames, and real-time processing Use Cases Explore example implementations and patterns ​ Join Our Community Discord Community Connect with other developers, share your projects, and get support from the Pipecat team. Installation & Setup On this page What You Can Build How It Works Real-time Processing Next Steps Join Our Community Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/deployment_pipecat-cloud_9c4b72f4.txt b/deployment_pipecat-cloud_9c4b72f4.txt
new file mode 100644
index 0000000000000000000000000000000000000000..f748f6a74078b198ae0e1a86138b40853a5bff24
--- /dev/null
+++ b/deployment_pipecat-cloud_9c4b72f4.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/guides/deployment/pipecat-cloud#example-dockerfile
+Title: Example: Pipecat Cloud - Pipecat
+==================================================
+
+Example: Pipecat Cloud - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Deploying your bot Example: Pipecat Cloud Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Guides Fundamentals Context Management Custom FrameProcessor Detecting Idle Users Ending a Pipeline Function Calling Muting User Input Recording Audio Recording Transcripts Features Gemini Multimodal Live Metrics Noise cancellation with Krisp OpenAI Audio Models and APIs Pipecat Flows Telephony Overview Dial-in: WebRTC (Daily) Dial-in: WebRTC (Twilio + Daily) Dial-in: Twilio (Media Streams) Dialout: WebRTC (Daily) Deploying your bot Overview Deployment pattern Example: Pipecat Cloud Example: Fly.io Example: Cerebrium Example: Modal Pipecat Cloud is a managed platform for hosting and scaling Pipecat agents in production. ​ Prerequisites Before you begin, you’ll need: A Pipecat Cloud account Docker installed Python 3.10+ The Pipecat Cloud CLI: pip install pipecatcloud Quickstart Guide Follow a step-by-step guided experience to deploy your first agent ​ Choosing a starting point Pipecat Cloud offers several ways to get started: Use a starter template - Pre-built agent configurations for common use cases Build from the base image - Create a custom agent using the official base image Clone the starter project - A bare-bones project template to customize ​ Starter templates Pipecat Cloud provides several ready-made templates for common agent types: Template Description voice Voice conversation agent with STT, LLM and TTS twilio Telephony agent that works with Twilio natural_conversation Agent focused on natural dialogue flow, allowing a user time to think openai_realtime Agent using OpenAI’s Realtime API gemini_multimodal_live Multimodal agent using Google’s Gemini Multimodal Live API vision Computer vision agent that can analyze images These templates include a functioning implementation and Dockerfile. You can use them directly: Copy Ask AI # Clone the repository git clone https://github.com/daily-co/pipecat-cloud-images.git # Navigate to a starter template cd pipecat-cloud-images/pipecat-starters/voice # Customize the agent for your needs ​ Project structure Whether using a starter template or building from scratch, a basic Pipecat Cloud project typically includes: Copy Ask AI my-agent/ ├── bot.py # Your Pipecat pipeline ├── Dockerfile # Container definition ├── requirements.txt # Python dependencies └── pcc-deploy.toml # Deployment config (optional) ​ Agent implementation with bot.py Your agent’s bot.py code must include a specific bot() function that serves as the entry point for Pipecat Cloud. This function has different signatures depending on the transport method: ​ For WebRTC/Daily transports Copy Ask AI async def bot ( args : DailySessionArguments): """Main bot entry point compatible with the FastAPI route handler. Args: config: The configuration object from the request body room_url: The Daily room URL token: The Daily room token session_id: The session ID for logging """ logger.info( f "Bot process initialized { args.room_url } { args.token } " ) try : await main(args.room_url, args.token) logger.info( "Bot process completed" ) except Exception as e: logger.exception( f "Error in bot process: { str (e) } " ) raise ​ For WebSocket transports (e.g., Twilio) Copy Ask AI async def bot ( args : WebSocketSessionArguments): """Main bot entry point for WebSocket connections. Args: ws: The WebSocket connection session_id: The session ID for logging """ logger.info( "WebSocket bot process initialized" ) try : await main(args.websocket) logger.info( "WebSocket bot process completed" ) except Exception as e: logger.exception( f "Error in WebSocket bot process: { str (e) } " ) raise Complete Example: Voice Bot View a complete WebRTC voice bot implementation Complete Example: Twilio Bot View a complete Twilio WebSocket bot implementation ​ Example Dockerfile Pipecat Cloud provides base images that include common dependencies for Pipecat agents: Copy Ask AI FROM dailyco/pipecat-base:latest COPY ./requirements.txt requirements.txt RUN pip install --no-cache-dir --upgrade -r requirements.txt COPY ./bot.py bot.py This Dockerfile: Uses the official Pipecat base image Installs Python dependencies from requirements.txt Copies your bot.py file to the container The base image (dailyco/pipecat-base) includes the HTTP API server, session management, and platform integration required to run on Pipecat Cloud. See the base image source code for details. ​ Building and pushing your Docker image With your project structure in place, build and push your Docker image: Copy Ask AI # Build the image docker build --platform=linux/arm64 -t my-first-agent:latest . # Tag the image for your repository docker tag my-first-agent:latest your-username/my-first-agent:0.1 # Push the tagged image to the repository docker push your-username/my-first-agent:0.1 Pipecat Cloud requires ARM64 images. Make sure to specify the --platform=linux/arm64 flag when building. ​ Managing secrets Your agent likely requires API keys and other credentials. Create a secret set to store them securely: Copy Ask AI # Create an .env file with your credentials touch .env # Create a secret set from the file pcc secrets set my-first-agent-secrets --file .env ​ Deploying your agent Deploy your agent with the CLI: Copy Ask AI pcc deploy my-first-agent your-username/my-first-agent:0.1 --secret-set my-first-agent-secrets For a more maintainable approach, create a pcc-deploy.toml file: Copy Ask AI agent_name = "my-first-agent" image = "your-username/my-first-agent:0.1" secret_set = "my-first-agent-secrets" image_credentials = "my-first-agent-image-credentials" # For private repos [ scaling ] min_instances = 0 Then deploy using: Copy Ask AI pcc deploy ​ Starting a session Once deployed, you can start a session with your agent: Copy Ask AI # Create and set a public access key if needed pcc organizations keys create pcc organizations keys use # Start a session using Daily for WebRTC pcc agent start my-first-agent --use-daily This will open a Daily room where you can interact with your agent. ​ Checking deployment status Monitor your agent deployment: Copy Ask AI # Check deployment status pcc agent status my-first-agent # View deployment logs pcc agent logs my-first-agent ​ Next steps Agent Images Learn about containerizing your agent Secrets Managing sensitive information Scaling Configure scaling for production workloads Active Sessions Understand how sessions work Deployment pattern Example: Fly.io On this page Prerequisites Choosing a starting point Starter templates Project structure Agent implementation with bot.py For WebRTC/Daily transports For WebSocket transports (e.g., Twilio) Example Dockerfile Building and pushing your Docker image Managing secrets Deploying your agent Starting a session Checking deployment status Next steps Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/deployment_pipecat-cloud_ab1f48ac.txt b/deployment_pipecat-cloud_ab1f48ac.txt
new file mode 100644
index 0000000000000000000000000000000000000000..509538dbe84bc03caa8a2118a39e3b5f8ace0e26
--- /dev/null
+++ b/deployment_pipecat-cloud_ab1f48ac.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/guides/deployment/pipecat-cloud
+Title: Example: Pipecat Cloud - Pipecat
+==================================================
+
+Example: Pipecat Cloud - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Deploying your bot Example: Pipecat Cloud Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Guides Fundamentals Context Management Custom FrameProcessor Detecting Idle Users Ending a Pipeline Function Calling Muting User Input Recording Audio Recording Transcripts Features Gemini Multimodal Live Metrics Noise cancellation with Krisp OpenAI Audio Models and APIs Pipecat Flows Telephony Overview Dial-in: WebRTC (Daily) Dial-in: WebRTC (Twilio + Daily) Dial-in: Twilio (Media Streams) Dialout: WebRTC (Daily) Deploying your bot Overview Deployment pattern Example: Pipecat Cloud Example: Fly.io Example: Cerebrium Example: Modal Pipecat Cloud is a managed platform for hosting and scaling Pipecat agents in production. ​ Prerequisites Before you begin, you’ll need: A Pipecat Cloud account Docker installed Python 3.10+ The Pipecat Cloud CLI: pip install pipecatcloud Quickstart Guide Follow a step-by-step guided experience to deploy your first agent ​ Choosing a starting point Pipecat Cloud offers several ways to get started: Use a starter template - Pre-built agent configurations for common use cases Build from the base image - Create a custom agent using the official base image Clone the starter project - A bare-bones project template to customize ​ Starter templates Pipecat Cloud provides several ready-made templates for common agent types: Template Description voice Voice conversation agent with STT, LLM and TTS twilio Telephony agent that works with Twilio natural_conversation Agent focused on natural dialogue flow, allowing a user time to think openai_realtime Agent using OpenAI’s Realtime API gemini_multimodal_live Multimodal agent using Google’s Gemini Multimodal Live API vision Computer vision agent that can analyze images These templates include a functioning implementation and Dockerfile. You can use them directly: Copy Ask AI # Clone the repository git clone https://github.com/daily-co/pipecat-cloud-images.git # Navigate to a starter template cd pipecat-cloud-images/pipecat-starters/voice # Customize the agent for your needs ​ Project structure Whether using a starter template or building from scratch, a basic Pipecat Cloud project typically includes: Copy Ask AI my-agent/ ├── bot.py # Your Pipecat pipeline ├── Dockerfile # Container definition ├── requirements.txt # Python dependencies └── pcc-deploy.toml # Deployment config (optional) ​ Agent implementation with bot.py Your agent’s bot.py code must include a specific bot() function that serves as the entry point for Pipecat Cloud. This function has different signatures depending on the transport method: ​ For WebRTC/Daily transports Copy Ask AI async def bot ( args : DailySessionArguments): """Main bot entry point compatible with the FastAPI route handler. Args: config: The configuration object from the request body room_url: The Daily room URL token: The Daily room token session_id: The session ID for logging """ logger.info( f "Bot process initialized { args.room_url } { args.token } " ) try : await main(args.room_url, args.token) logger.info( "Bot process completed" ) except Exception as e: logger.exception( f "Error in bot process: { str (e) } " ) raise ​ For WebSocket transports (e.g., Twilio) Copy Ask AI async def bot ( args : WebSocketSessionArguments): """Main bot entry point for WebSocket connections. Args: ws: The WebSocket connection session_id: The session ID for logging """ logger.info( "WebSocket bot process initialized" ) try : await main(args.websocket) logger.info( "WebSocket bot process completed" ) except Exception as e: logger.exception( f "Error in WebSocket bot process: { str (e) } " ) raise Complete Example: Voice Bot View a complete WebRTC voice bot implementation Complete Example: Twilio Bot View a complete Twilio WebSocket bot implementation ​ Example Dockerfile Pipecat Cloud provides base images that include common dependencies for Pipecat agents: Copy Ask AI FROM dailyco/pipecat-base:latest COPY ./requirements.txt requirements.txt RUN pip install --no-cache-dir --upgrade -r requirements.txt COPY ./bot.py bot.py This Dockerfile: Uses the official Pipecat base image Installs Python dependencies from requirements.txt Copies your bot.py file to the container The base image (dailyco/pipecat-base) includes the HTTP API server, session management, and platform integration required to run on Pipecat Cloud. See the base image source code for details. ​ Building and pushing your Docker image With your project structure in place, build and push your Docker image: Copy Ask AI # Build the image docker build --platform=linux/arm64 -t my-first-agent:latest . # Tag the image for your repository docker tag my-first-agent:latest your-username/my-first-agent:0.1 # Push the tagged image to the repository docker push your-username/my-first-agent:0.1 Pipecat Cloud requires ARM64 images. Make sure to specify the --platform=linux/arm64 flag when building. ​ Managing secrets Your agent likely requires API keys and other credentials. Create a secret set to store them securely: Copy Ask AI # Create an .env file with your credentials touch .env # Create a secret set from the file pcc secrets set my-first-agent-secrets --file .env ​ Deploying your agent Deploy your agent with the CLI: Copy Ask AI pcc deploy my-first-agent your-username/my-first-agent:0.1 --secret-set my-first-agent-secrets For a more maintainable approach, create a pcc-deploy.toml file: Copy Ask AI agent_name = "my-first-agent" image = "your-username/my-first-agent:0.1" secret_set = "my-first-agent-secrets" image_credentials = "my-first-agent-image-credentials" # For private repos [ scaling ] min_instances = 0 Then deploy using: Copy Ask AI pcc deploy ​ Starting a session Once deployed, you can start a session with your agent: Copy Ask AI # Create and set a public access key if needed pcc organizations keys create pcc organizations keys use # Start a session using Daily for WebRTC pcc agent start my-first-agent --use-daily This will open a Daily room where you can interact with your agent. ​ Checking deployment status Monitor your agent deployment: Copy Ask AI # Check deployment status pcc agent status my-first-agent # View deployment logs pcc agent logs my-first-agent ​ Next steps Agent Images Learn about containerizing your agent Secrets Managing sensitive information Scaling Configure scaling for production workloads Active Sessions Understand how sessions work Deployment pattern Example: Fly.io On this page Prerequisites Choosing a starting point Starter templates Project structure Agent implementation with bot.py For WebRTC/Daily transports For WebSocket transports (e.g., Twilio) Example Dockerfile Building and pushing your Docker image Managing secrets Deploying your agent Starting a session Checking deployment status Next steps Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/features_gemini-multimodal-live_07b20525.txt b/features_gemini-multimodal-live_07b20525.txt
new file mode 100644
index 0000000000000000000000000000000000000000..f5d5eb80938ddf2831baa4216be7ebb82526f508
--- /dev/null
+++ b/features_gemini-multimodal-live_07b20525.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/guides/features/gemini-multimodal-live#3-testing-the-connection
+Title: Building with Gemini Multimodal Live - Pipecat
+==================================================
+
+Building with Gemini Multimodal Live - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Features Building with Gemini Multimodal Live Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Guides Fundamentals Context Management Custom FrameProcessor Detecting Idle Users Ending a Pipeline Function Calling Muting User Input Recording Audio Recording Transcripts Features Gemini Multimodal Live Metrics Noise cancellation with Krisp OpenAI Audio Models and APIs Pipecat Flows Telephony Overview Dial-in: WebRTC (Daily) Dial-in: WebRTC (Twilio + Daily) Dial-in: Twilio (Media Streams) Dialout: WebRTC (Daily) Deploying your bot Overview Deployment pattern Example: Pipecat Cloud Example: Fly.io Example: Cerebrium Example: Modal This guide will walk you through building a real-time AI chatbot using Gemini Multimodal Live and Pipecat. We’ll create a complete application with a Pipecat server and a Pipecat React client that enables natural conversations with an AI assistant. API Reference Gemini Multimodal Live API documentation Example Code Find the complete client and server code in Github Client SDK Pipecat React SDK documentation ​ What We’ll Build In this guide, you’ll create: A FastAPI server that manages bot instances A Gemini-powered conversational AI bot A React client with real-time audio/video A complete pipeline for speech-to-speech interaction ​ Key Concepts Before we dive into implementation, let’s cover some important concepts that will help you understand how Pipecat and Gemini work together. ​ Understanding Pipelines At the heart of Pipecat is the pipeline system. A pipeline is a sequence of processors that handle different aspects of the conversation flow. Think of it like an assembly line where each station (processor) performs a specific task. For our chatbot, the pipeline looks like this: Copy Ask AI pipeline = Pipeline([ transport.input(), # Receives audio/video from the user via WebRTC rtvi, # Handles client/server messaging and events context_aggregator.user(), # Manages user message history llm, # Processes speech through Gemini talking_animation, # Controls bot's avatar transport.output(), # Sends audio/video back to the user via WebRTC context_aggregator.assistant(), # Manages bot message history ]) ​ Processors Each processor in the pipeline handles a specific task: Transport transport.input() and transport.output() handle media streaming with Daily Context context_aggregator maintains conversation history for natural dialogue Speech Processing rtvi_user_transcription and rtvi_bot_transcription handle speech-to-text Animation talking_animation controls the bot’s visual state based on speaking activity The order of processors matters! Data flows through the pipeline in sequence, so each processor should receive the data it needs from previous processors. Learn more about the Core Concepts to Pipecat server. ​ Gemini Integration The GeminiMultimodalLiveLLMService is a speech-to-speech LLM service that interfaces with the Gemini Multimodal Live API. It provides: Real-time speech-to-speech conversation Context management Voice activity detection Tool use Pipecat manages two types of connections: A WebRTC connection between the Pipecat client and server for reliable audio/video streaming A WebSocket connection between the Pipecat server and Gemini for real-time AI processing This architecture ensures stable media streaming while maintaining responsive AI interactions. ​ Prerequisites Before we begin, you’ll need: Python 3.10 or higher Node.js 16 or higher A Daily API key A Google API key with Gemini Multimodal Live access Clone the Pipecat repo: Copy Ask AI git clone git@github.com:pipecat-ai/pipecat.git ​ Server Implementation Let’s start by setting up the server components. Our server will handle bot management, room creation, and client connections. ​ Environment Setup Navigate to the simple-chatbot’s server directory: Copy Ask AI cd examples/simple-chatbot/server Set up a python virtual environment: Copy Ask AI python -m venv venv source venv/bin/activate # On Windows: venv\Scripts\activate Install requirements: Copy Ask AI pip install -r requirements.txt Copy env.example to .env and make a few changes: Copy Ask AI # Remove the hard-coded example room URL DAILY_SAMPLE_ROOM_URL = # Add your Daily and Gemini API keys DAILY_API_KEY = [your key here] GEMINI_API_KEY = [your key here] # Use Gemini implementation BOT_IMPLEMENTATION = gemini ​ Server Setup (server.py) server.py is a FastAPI server that creates the meeting room where clients and bots interact, manages bot instances, and handles client connections. It’s the orchestrator that brings everything on the server-side together. ​ Creating Meeting Room The server uses Daily’s API via a REST API helper to create rooms where clients and bots can meet. Each room is a secure space for audio/video communication: server/server.py Copy Ask AI async def create_room_and_token (): """Create a Daily room and generate access credentials.""" room = await daily_helpers[ "rest" ].create_room(DailyRoomParams()) token = await daily_helpers[ "rest" ].get_token(room.url) return room.url, token ​ Managing Bot Instances When a client connects, the server starts a new bot instance configured specifically for that room. It keeps track of running bots and ensures there’s only one bot per room: server/server.py Copy Ask AI # Start the bot process for a specific room bot_file = "bot-gemini.py" proc = subprocess.Popen([ f "python3 -m { bot_file } -u { room_url } -t { token } " ]) bot_procs[proc.pid] = (proc, room_url) ​ Connection Endpoints The server provides two ways to connect: Browser Access (/) Creates a room, starts a bot, and redirects the browser to the Daily meeting URL. Perfect for quick testing and development. RTVI Client (/connect) Creates a room, starts a bot, and returns connection credentials. Used by RTVI clients for custom implementations. ​ Bot Implementation (bot-gemini.py) The bot implementation connects all the pieces: Daily transport, Gemini service, conversation context, and processors. Let’s break down each component: ​ Transport Setup First, we configure the Daily transport, which handles WebRTC communication between the client and server. server/bot-gemini.py Copy Ask AI transport = DailyTransport( room_url, token, "Chatbot" , DailyParams( audio_in_enabled = True , # Enable audio input audio_out_enabled = True , # Enable audio output video_out_enabled = True , # Enable video output vad_analyzer = SileroVADAnalyzer( params = VADParams( stop_secs = 0.5 )), ), ) Gemini Multimodal Live audio requirements: Input: 16 kHz sample rate Output: 24 kHz sample rate ​ Gemini Service Configuration Next, we initialize the Gemini service which will provide speech-to-speech inference and communication: server/bot-gemini.py Copy Ask AI llm = GeminiMultimodalLiveLLMService( api_key = os.getenv( "GEMINI_API_KEY" ), voice_id = "Puck" , # Choose your bot's voice params = InputParams( temperature = 0.7 ) # Set model input params ) ​ Conversation Context We give our bot its personality and initial instructions: server/bot-gemini.py Copy Ask AI messages = [{ "role" : "user" , "content" : """You are Chatbot, a friendly, helpful robot. Keep responses brief and avoid special characters since output will be converted to audio.""" }] context = OpenAILLMContext(messages) context_aggregator = llm.create_context_aggregator(context) OpenAILLMContext is used as a common LLM base service for context management. In the future, we may add a specific context manager for Gemini. The context aggregator automatically maintains conversation history, helping the bot remember previous interactions. ​ Processor Setup We initialize two additional processors in our pipeline to handle different aspects of the interaction: RTVI Processors RTVIProcessor : Handles all client communication events including transcriptions, speaking states, and performance metrics Animation TalkingAnimation : Controls the bot’s visual state, switching between static and animated frames based on speaking status Learn more about the RTVI framework and available processors. ​ Pipeline Assembly Finally, we bring everything together in a pipeline: server/bot-gemini.py Copy Ask AI pipeline = Pipeline([ transport.input(), # Receive media rtvi, # Client UI events context_aggregator.user(), # Process user context llm, # Gemini processing ta, # Animation (talking/quiet states) transport.output(), # Send media context_aggregator.assistant() # Process bot context ]) task = PipelineTask( pipeline, params = PipelineParams( allow_interruptions = True , enable_metrics = True , enable_usage_metrics = True , ), observers = [RTVIObserver(rtvi)], ) The order of processors is crucial! For example, the RTVI processor should be early in the pipeline to capture all relevant events. The RTVIObserver monitors the entire pipeline and automatically collects relevant events to send to the client. ​ Client Implementation Our React client uses the Pipecat React SDK to communicate with the bot. Let’s explore how the client connects and interacts with our Pipecat server. ​ Connection Setup The client needs to connect to our bot server using the same transport type (Daily WebRTC) that we configured on the server: examples/react/src/providers/PipecatProvider.tsx Copy Ask AI const client = new PipecatClient ({ transport: new DailyTransport (), enableMic: true , // Enable audio input enableCam: false , // Disable video input enableScreenShare: false , // Disable screen sharing }); client . connect ({ endpoint: "http://localhost:7860/connect" , // Your bot connection endpoint }); The connection configuration must match your server: DailyTransport : Matches the WebRTC transport used in bot-gemini.py connect endpoint: Matches the /connect route in server.py Media settings: Controls which devices are enabled on join ​ Media Handling Pipecat’s React components handle all the complex media stream management for you: Copy Ask AI function App () { return ( < PipecatClientProvider client = { client } > < div className = "app" > < PipecatClientVideo participant = "bot" /> { /* Bot's video feed */ } < PipecatClientAudio /> { /* Audio input/output */ } </ div > </ PipecatClientProvider > ); } The PipecatClientProvider is the root component for providing Pipecat client context to your application. By wrapping your PipecatClientAudio and PipecatClientVideo components in this provider, they can access the client instance and receive and process the streams received from the Pipecat server. ​ Real-time Events The RTVI processors we configured in the pipeline emit events that we can handle in our client: Copy Ask AI // Listen for transcription events useRTVIClientEvent ( RTVIEvent . UserTranscript , ( data : TranscriptData ) => { if ( data . final ) { console . log ( `User said: ${ data . text } ` ); } }); // Listen for bot responses useRTVIClientEvent ( RTVIEvent . BotTranscript , ( data : BotLLMTextData ) => { console . log ( `Bot responded: ${ data . text } ` ); }); Available Events Speaking state changes Transcription updates Bot responses Connection status Performance metrics Event Usage Use these events to: Show speaking indicators Display transcripts Update UI state Monitor performance Optionally, uses callbacks to handle events in your application. Learn more in the Pipecat client docs. ​ Complete Example Here’s a basic client implementation with connection status and transcription display: Copy Ask AI function ChatApp () { return ( < PipecatClientProvider client = { client } > < div className = "app" > { /* Connection UI */ } < StatusDisplay /> < ConnectButton /> { /* Media Components */ } < BotVideo /> < PipecatClientAudio /> { /* Debug/Transcript Display */ } < DebugDisplay /> </ div > </ PipecatClientProvider > ); } Check out the example repository for a complete client implementation with styling and error handling. ​ Running the Application From the simple-chatbot directory, start the server and client to test the chatbot: ​ 1. Start the Server In one terminal: Copy Ask AI python server/server.py ​ 2. Start the Client In another terminal: Copy Ask AI cd examples/react npm install npm run dev ​ 3. Testing the Connection Open http://localhost:5173 in your browser Click “Connect” to join a room Allow microphone access when prompted Start talking with your AI assistant Troubleshooting: Check that all API keys are properly configured in .env Grant your browser access to your microphone, so it can receive your audio input Verify WebRTC ports aren’t blocked by firewalls ​ Next Steps Now that you have a working chatbot, consider these enhancements: Add custom avatar animations Implement function calling for external integrations Add support for multiple languages Enhance error recovery and reconnection logic ​ Examples Foundational Example A basic implementation demonstrating core Gemini Multimodal Live features and transcription capabilities Simple Chatbot A complete client/server implementation showing how to build a Pipecat JS or React client that connects to a Gemini Live Pipecat bot ​ Learn More Gemini Multimodal Live API Reference React Client SDK Documentation Recording Transcripts Metrics On this page What We’ll Build Key Concepts Understanding Pipelines Processors Gemini Integration Prerequisites Server Implementation Environment Setup Server Setup (server.py) Creating Meeting Room Managing Bot Instances Connection Endpoints Bot Implementation (bot-gemini.py) Transport Setup Gemini Service Configuration Conversation Context Processor Setup Pipeline Assembly Client Implementation Connection Setup Media Handling Real-time Events Complete Example Running the Application 1. Start the Server 2. Start the Client 3. Testing the Connection Next Steps Examples Learn More Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/features_gemini-multimodal-live_13e5aad8.txt b/features_gemini-multimodal-live_13e5aad8.txt
new file mode 100644
index 0000000000000000000000000000000000000000..cd56ede5ef5c5ebfc6fd042ef64728840f810aae
--- /dev/null
+++ b/features_gemini-multimodal-live_13e5aad8.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/guides/features/gemini-multimodal-live#bot-implementation-bot-gemini-py
+Title: Building with Gemini Multimodal Live - Pipecat
+==================================================
+
+Building with Gemini Multimodal Live - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Features Building with Gemini Multimodal Live Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Guides Fundamentals Context Management Custom FrameProcessor Detecting Idle Users Ending a Pipeline Function Calling Muting User Input Recording Audio Recording Transcripts Features Gemini Multimodal Live Metrics Noise cancellation with Krisp OpenAI Audio Models and APIs Pipecat Flows Telephony Overview Dial-in: WebRTC (Daily) Dial-in: WebRTC (Twilio + Daily) Dial-in: Twilio (Media Streams) Dialout: WebRTC (Daily) Deploying your bot Overview Deployment pattern Example: Pipecat Cloud Example: Fly.io Example: Cerebrium Example: Modal This guide will walk you through building a real-time AI chatbot using Gemini Multimodal Live and Pipecat. We’ll create a complete application with a Pipecat server and a Pipecat React client that enables natural conversations with an AI assistant. API Reference Gemini Multimodal Live API documentation Example Code Find the complete client and server code in Github Client SDK Pipecat React SDK documentation ​ What We’ll Build In this guide, you’ll create: A FastAPI server that manages bot instances A Gemini-powered conversational AI bot A React client with real-time audio/video A complete pipeline for speech-to-speech interaction ​ Key Concepts Before we dive into implementation, let’s cover some important concepts that will help you understand how Pipecat and Gemini work together. ​ Understanding Pipelines At the heart of Pipecat is the pipeline system. A pipeline is a sequence of processors that handle different aspects of the conversation flow. Think of it like an assembly line where each station (processor) performs a specific task. For our chatbot, the pipeline looks like this: Copy Ask AI pipeline = Pipeline([ transport.input(), # Receives audio/video from the user via WebRTC rtvi, # Handles client/server messaging and events context_aggregator.user(), # Manages user message history llm, # Processes speech through Gemini talking_animation, # Controls bot's avatar transport.output(), # Sends audio/video back to the user via WebRTC context_aggregator.assistant(), # Manages bot message history ]) ​ Processors Each processor in the pipeline handles a specific task: Transport transport.input() and transport.output() handle media streaming with Daily Context context_aggregator maintains conversation history for natural dialogue Speech Processing rtvi_user_transcription and rtvi_bot_transcription handle speech-to-text Animation talking_animation controls the bot’s visual state based on speaking activity The order of processors matters! Data flows through the pipeline in sequence, so each processor should receive the data it needs from previous processors. Learn more about the Core Concepts to Pipecat server. ​ Gemini Integration The GeminiMultimodalLiveLLMService is a speech-to-speech LLM service that interfaces with the Gemini Multimodal Live API. It provides: Real-time speech-to-speech conversation Context management Voice activity detection Tool use Pipecat manages two types of connections: A WebRTC connection between the Pipecat client and server for reliable audio/video streaming A WebSocket connection between the Pipecat server and Gemini for real-time AI processing This architecture ensures stable media streaming while maintaining responsive AI interactions. ​ Prerequisites Before we begin, you’ll need: Python 3.10 or higher Node.js 16 or higher A Daily API key A Google API key with Gemini Multimodal Live access Clone the Pipecat repo: Copy Ask AI git clone git@github.com:pipecat-ai/pipecat.git ​ Server Implementation Let’s start by setting up the server components. Our server will handle bot management, room creation, and client connections. ​ Environment Setup Navigate to the simple-chatbot’s server directory: Copy Ask AI cd examples/simple-chatbot/server Set up a python virtual environment: Copy Ask AI python -m venv venv source venv/bin/activate # On Windows: venv\Scripts\activate Install requirements: Copy Ask AI pip install -r requirements.txt Copy env.example to .env and make a few changes: Copy Ask AI # Remove the hard-coded example room URL DAILY_SAMPLE_ROOM_URL = # Add your Daily and Gemini API keys DAILY_API_KEY = [your key here] GEMINI_API_KEY = [your key here] # Use Gemini implementation BOT_IMPLEMENTATION = gemini ​ Server Setup (server.py) server.py is a FastAPI server that creates the meeting room where clients and bots interact, manages bot instances, and handles client connections. It’s the orchestrator that brings everything on the server-side together. ​ Creating Meeting Room The server uses Daily’s API via a REST API helper to create rooms where clients and bots can meet. Each room is a secure space for audio/video communication: server/server.py Copy Ask AI async def create_room_and_token (): """Create a Daily room and generate access credentials.""" room = await daily_helpers[ "rest" ].create_room(DailyRoomParams()) token = await daily_helpers[ "rest" ].get_token(room.url) return room.url, token ​ Managing Bot Instances When a client connects, the server starts a new bot instance configured specifically for that room. It keeps track of running bots and ensures there’s only one bot per room: server/server.py Copy Ask AI # Start the bot process for a specific room bot_file = "bot-gemini.py" proc = subprocess.Popen([ f "python3 -m { bot_file } -u { room_url } -t { token } " ]) bot_procs[proc.pid] = (proc, room_url) ​ Connection Endpoints The server provides two ways to connect: Browser Access (/) Creates a room, starts a bot, and redirects the browser to the Daily meeting URL. Perfect for quick testing and development. RTVI Client (/connect) Creates a room, starts a bot, and returns connection credentials. Used by RTVI clients for custom implementations. ​ Bot Implementation (bot-gemini.py) The bot implementation connects all the pieces: Daily transport, Gemini service, conversation context, and processors. Let’s break down each component: ​ Transport Setup First, we configure the Daily transport, which handles WebRTC communication between the client and server. server/bot-gemini.py Copy Ask AI transport = DailyTransport( room_url, token, "Chatbot" , DailyParams( audio_in_enabled = True , # Enable audio input audio_out_enabled = True , # Enable audio output video_out_enabled = True , # Enable video output vad_analyzer = SileroVADAnalyzer( params = VADParams( stop_secs = 0.5 )), ), ) Gemini Multimodal Live audio requirements: Input: 16 kHz sample rate Output: 24 kHz sample rate ​ Gemini Service Configuration Next, we initialize the Gemini service which will provide speech-to-speech inference and communication: server/bot-gemini.py Copy Ask AI llm = GeminiMultimodalLiveLLMService( api_key = os.getenv( "GEMINI_API_KEY" ), voice_id = "Puck" , # Choose your bot's voice params = InputParams( temperature = 0.7 ) # Set model input params ) ​ Conversation Context We give our bot its personality and initial instructions: server/bot-gemini.py Copy Ask AI messages = [{ "role" : "user" , "content" : """You are Chatbot, a friendly, helpful robot. Keep responses brief and avoid special characters since output will be converted to audio.""" }] context = OpenAILLMContext(messages) context_aggregator = llm.create_context_aggregator(context) OpenAILLMContext is used as a common LLM base service for context management. In the future, we may add a specific context manager for Gemini. The context aggregator automatically maintains conversation history, helping the bot remember previous interactions. ​ Processor Setup We initialize two additional processors in our pipeline to handle different aspects of the interaction: RTVI Processors RTVIProcessor : Handles all client communication events including transcriptions, speaking states, and performance metrics Animation TalkingAnimation : Controls the bot’s visual state, switching between static and animated frames based on speaking status Learn more about the RTVI framework and available processors. ​ Pipeline Assembly Finally, we bring everything together in a pipeline: server/bot-gemini.py Copy Ask AI pipeline = Pipeline([ transport.input(), # Receive media rtvi, # Client UI events context_aggregator.user(), # Process user context llm, # Gemini processing ta, # Animation (talking/quiet states) transport.output(), # Send media context_aggregator.assistant() # Process bot context ]) task = PipelineTask( pipeline, params = PipelineParams( allow_interruptions = True , enable_metrics = True , enable_usage_metrics = True , ), observers = [RTVIObserver(rtvi)], ) The order of processors is crucial! For example, the RTVI processor should be early in the pipeline to capture all relevant events. The RTVIObserver monitors the entire pipeline and automatically collects relevant events to send to the client. ​ Client Implementation Our React client uses the Pipecat React SDK to communicate with the bot. Let’s explore how the client connects and interacts with our Pipecat server. ​ Connection Setup The client needs to connect to our bot server using the same transport type (Daily WebRTC) that we configured on the server: examples/react/src/providers/PipecatProvider.tsx Copy Ask AI const client = new PipecatClient ({ transport: new DailyTransport (), enableMic: true , // Enable audio input enableCam: false , // Disable video input enableScreenShare: false , // Disable screen sharing }); client . connect ({ endpoint: "http://localhost:7860/connect" , // Your bot connection endpoint }); The connection configuration must match your server: DailyTransport : Matches the WebRTC transport used in bot-gemini.py connect endpoint: Matches the /connect route in server.py Media settings: Controls which devices are enabled on join ​ Media Handling Pipecat’s React components handle all the complex media stream management for you: Copy Ask AI function App () { return ( < PipecatClientProvider client = { client } > < div className = "app" > < PipecatClientVideo participant = "bot" /> { /* Bot's video feed */ } < PipecatClientAudio /> { /* Audio input/output */ } </ div > </ PipecatClientProvider > ); } The PipecatClientProvider is the root component for providing Pipecat client context to your application. By wrapping your PipecatClientAudio and PipecatClientVideo components in this provider, they can access the client instance and receive and process the streams received from the Pipecat server. ​ Real-time Events The RTVI processors we configured in the pipeline emit events that we can handle in our client: Copy Ask AI // Listen for transcription events useRTVIClientEvent ( RTVIEvent . UserTranscript , ( data : TranscriptData ) => { if ( data . final ) { console . log ( `User said: ${ data . text } ` ); } }); // Listen for bot responses useRTVIClientEvent ( RTVIEvent . BotTranscript , ( data : BotLLMTextData ) => { console . log ( `Bot responded: ${ data . text } ` ); }); Available Events Speaking state changes Transcription updates Bot responses Connection status Performance metrics Event Usage Use these events to: Show speaking indicators Display transcripts Update UI state Monitor performance Optionally, uses callbacks to handle events in your application. Learn more in the Pipecat client docs. ​ Complete Example Here’s a basic client implementation with connection status and transcription display: Copy Ask AI function ChatApp () { return ( < PipecatClientProvider client = { client } > < div className = "app" > { /* Connection UI */ } < StatusDisplay /> < ConnectButton /> { /* Media Components */ } < BotVideo /> < PipecatClientAudio /> { /* Debug/Transcript Display */ } < DebugDisplay /> </ div > </ PipecatClientProvider > ); } Check out the example repository for a complete client implementation with styling and error handling. ​ Running the Application From the simple-chatbot directory, start the server and client to test the chatbot: ​ 1. Start the Server In one terminal: Copy Ask AI python server/server.py ​ 2. Start the Client In another terminal: Copy Ask AI cd examples/react npm install npm run dev ​ 3. Testing the Connection Open http://localhost:5173 in your browser Click “Connect” to join a room Allow microphone access when prompted Start talking with your AI assistant Troubleshooting: Check that all API keys are properly configured in .env Grant your browser access to your microphone, so it can receive your audio input Verify WebRTC ports aren’t blocked by firewalls ​ Next Steps Now that you have a working chatbot, consider these enhancements: Add custom avatar animations Implement function calling for external integrations Add support for multiple languages Enhance error recovery and reconnection logic ​ Examples Foundational Example A basic implementation demonstrating core Gemini Multimodal Live features and transcription capabilities Simple Chatbot A complete client/server implementation showing how to build a Pipecat JS or React client that connects to a Gemini Live Pipecat bot ​ Learn More Gemini Multimodal Live API Reference React Client SDK Documentation Recording Transcripts Metrics On this page What We’ll Build Key Concepts Understanding Pipelines Processors Gemini Integration Prerequisites Server Implementation Environment Setup Server Setup (server.py) Creating Meeting Room Managing Bot Instances Connection Endpoints Bot Implementation (bot-gemini.py) Transport Setup Gemini Service Configuration Conversation Context Processor Setup Pipeline Assembly Client Implementation Connection Setup Media Handling Real-time Events Complete Example Running the Application 1. Start the Server 2. Start the Client 3. Testing the Connection Next Steps Examples Learn More Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/features_gemini-multimodal-live_231418c5.txt b/features_gemini-multimodal-live_231418c5.txt
new file mode 100644
index 0000000000000000000000000000000000000000..d05a16b54a621f219c27200cfefbb055d0ddf4d8
--- /dev/null
+++ b/features_gemini-multimodal-live_231418c5.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/guides/features/gemini-multimodal-live#managing-bot-instances
+Title: Building with Gemini Multimodal Live - Pipecat
+==================================================
+
+Building with Gemini Multimodal Live - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Features Building with Gemini Multimodal Live Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Guides Fundamentals Context Management Custom FrameProcessor Detecting Idle Users Ending a Pipeline Function Calling Muting User Input Recording Audio Recording Transcripts Features Gemini Multimodal Live Metrics Noise cancellation with Krisp OpenAI Audio Models and APIs Pipecat Flows Telephony Overview Dial-in: WebRTC (Daily) Dial-in: WebRTC (Twilio + Daily) Dial-in: Twilio (Media Streams) Dialout: WebRTC (Daily) Deploying your bot Overview Deployment pattern Example: Pipecat Cloud Example: Fly.io Example: Cerebrium Example: Modal This guide will walk you through building a real-time AI chatbot using Gemini Multimodal Live and Pipecat. We’ll create a complete application with a Pipecat server and a Pipecat React client that enables natural conversations with an AI assistant. API Reference Gemini Multimodal Live API documentation Example Code Find the complete client and server code in Github Client SDK Pipecat React SDK documentation ​ What We’ll Build In this guide, you’ll create: A FastAPI server that manages bot instances A Gemini-powered conversational AI bot A React client with real-time audio/video A complete pipeline for speech-to-speech interaction ​ Key Concepts Before we dive into implementation, let’s cover some important concepts that will help you understand how Pipecat and Gemini work together. ​ Understanding Pipelines At the heart of Pipecat is the pipeline system. A pipeline is a sequence of processors that handle different aspects of the conversation flow. Think of it like an assembly line where each station (processor) performs a specific task. For our chatbot, the pipeline looks like this: Copy Ask AI pipeline = Pipeline([ transport.input(), # Receives audio/video from the user via WebRTC rtvi, # Handles client/server messaging and events context_aggregator.user(), # Manages user message history llm, # Processes speech through Gemini talking_animation, # Controls bot's avatar transport.output(), # Sends audio/video back to the user via WebRTC context_aggregator.assistant(), # Manages bot message history ]) ​ Processors Each processor in the pipeline handles a specific task: Transport transport.input() and transport.output() handle media streaming with Daily Context context_aggregator maintains conversation history for natural dialogue Speech Processing rtvi_user_transcription and rtvi_bot_transcription handle speech-to-text Animation talking_animation controls the bot’s visual state based on speaking activity The order of processors matters! Data flows through the pipeline in sequence, so each processor should receive the data it needs from previous processors. Learn more about the Core Concepts to Pipecat server. ​ Gemini Integration The GeminiMultimodalLiveLLMService is a speech-to-speech LLM service that interfaces with the Gemini Multimodal Live API. It provides: Real-time speech-to-speech conversation Context management Voice activity detection Tool use Pipecat manages two types of connections: A WebRTC connection between the Pipecat client and server for reliable audio/video streaming A WebSocket connection between the Pipecat server and Gemini for real-time AI processing This architecture ensures stable media streaming while maintaining responsive AI interactions. ​ Prerequisites Before we begin, you’ll need: Python 3.10 or higher Node.js 16 or higher A Daily API key A Google API key with Gemini Multimodal Live access Clone the Pipecat repo: Copy Ask AI git clone git@github.com:pipecat-ai/pipecat.git ​ Server Implementation Let’s start by setting up the server components. Our server will handle bot management, room creation, and client connections. ​ Environment Setup Navigate to the simple-chatbot’s server directory: Copy Ask AI cd examples/simple-chatbot/server Set up a python virtual environment: Copy Ask AI python -m venv venv source venv/bin/activate # On Windows: venv\Scripts\activate Install requirements: Copy Ask AI pip install -r requirements.txt Copy env.example to .env and make a few changes: Copy Ask AI # Remove the hard-coded example room URL DAILY_SAMPLE_ROOM_URL = # Add your Daily and Gemini API keys DAILY_API_KEY = [your key here] GEMINI_API_KEY = [your key here] # Use Gemini implementation BOT_IMPLEMENTATION = gemini ​ Server Setup (server.py) server.py is a FastAPI server that creates the meeting room where clients and bots interact, manages bot instances, and handles client connections. It’s the orchestrator that brings everything on the server-side together. ​ Creating Meeting Room The server uses Daily’s API via a REST API helper to create rooms where clients and bots can meet. Each room is a secure space for audio/video communication: server/server.py Copy Ask AI async def create_room_and_token (): """Create a Daily room and generate access credentials.""" room = await daily_helpers[ "rest" ].create_room(DailyRoomParams()) token = await daily_helpers[ "rest" ].get_token(room.url) return room.url, token ​ Managing Bot Instances When a client connects, the server starts a new bot instance configured specifically for that room. It keeps track of running bots and ensures there’s only one bot per room: server/server.py Copy Ask AI # Start the bot process for a specific room bot_file = "bot-gemini.py" proc = subprocess.Popen([ f "python3 -m { bot_file } -u { room_url } -t { token } " ]) bot_procs[proc.pid] = (proc, room_url) ​ Connection Endpoints The server provides two ways to connect: Browser Access (/) Creates a room, starts a bot, and redirects the browser to the Daily meeting URL. Perfect for quick testing and development. RTVI Client (/connect) Creates a room, starts a bot, and returns connection credentials. Used by RTVI clients for custom implementations. ​ Bot Implementation (bot-gemini.py) The bot implementation connects all the pieces: Daily transport, Gemini service, conversation context, and processors. Let’s break down each component: ​ Transport Setup First, we configure the Daily transport, which handles WebRTC communication between the client and server. server/bot-gemini.py Copy Ask AI transport = DailyTransport( room_url, token, "Chatbot" , DailyParams( audio_in_enabled = True , # Enable audio input audio_out_enabled = True , # Enable audio output video_out_enabled = True , # Enable video output vad_analyzer = SileroVADAnalyzer( params = VADParams( stop_secs = 0.5 )), ), ) Gemini Multimodal Live audio requirements: Input: 16 kHz sample rate Output: 24 kHz sample rate ​ Gemini Service Configuration Next, we initialize the Gemini service which will provide speech-to-speech inference and communication: server/bot-gemini.py Copy Ask AI llm = GeminiMultimodalLiveLLMService( api_key = os.getenv( "GEMINI_API_KEY" ), voice_id = "Puck" , # Choose your bot's voice params = InputParams( temperature = 0.7 ) # Set model input params ) ​ Conversation Context We give our bot its personality and initial instructions: server/bot-gemini.py Copy Ask AI messages = [{ "role" : "user" , "content" : """You are Chatbot, a friendly, helpful robot. Keep responses brief and avoid special characters since output will be converted to audio.""" }] context = OpenAILLMContext(messages) context_aggregator = llm.create_context_aggregator(context) OpenAILLMContext is used as a common LLM base service for context management. In the future, we may add a specific context manager for Gemini. The context aggregator automatically maintains conversation history, helping the bot remember previous interactions. ​ Processor Setup We initialize two additional processors in our pipeline to handle different aspects of the interaction: RTVI Processors RTVIProcessor : Handles all client communication events including transcriptions, speaking states, and performance metrics Animation TalkingAnimation : Controls the bot’s visual state, switching between static and animated frames based on speaking status Learn more about the RTVI framework and available processors. ​ Pipeline Assembly Finally, we bring everything together in a pipeline: server/bot-gemini.py Copy Ask AI pipeline = Pipeline([ transport.input(), # Receive media rtvi, # Client UI events context_aggregator.user(), # Process user context llm, # Gemini processing ta, # Animation (talking/quiet states) transport.output(), # Send media context_aggregator.assistant() # Process bot context ]) task = PipelineTask( pipeline, params = PipelineParams( allow_interruptions = True , enable_metrics = True , enable_usage_metrics = True , ), observers = [RTVIObserver(rtvi)], ) The order of processors is crucial! For example, the RTVI processor should be early in the pipeline to capture all relevant events. The RTVIObserver monitors the entire pipeline and automatically collects relevant events to send to the client. ​ Client Implementation Our React client uses the Pipecat React SDK to communicate with the bot. Let’s explore how the client connects and interacts with our Pipecat server. ​ Connection Setup The client needs to connect to our bot server using the same transport type (Daily WebRTC) that we configured on the server: examples/react/src/providers/PipecatProvider.tsx Copy Ask AI const client = new PipecatClient ({ transport: new DailyTransport (), enableMic: true , // Enable audio input enableCam: false , // Disable video input enableScreenShare: false , // Disable screen sharing }); client . connect ({ endpoint: "http://localhost:7860/connect" , // Your bot connection endpoint }); The connection configuration must match your server: DailyTransport : Matches the WebRTC transport used in bot-gemini.py connect endpoint: Matches the /connect route in server.py Media settings: Controls which devices are enabled on join ​ Media Handling Pipecat’s React components handle all the complex media stream management for you: Copy Ask AI function App () { return ( < PipecatClientProvider client = { client } > < div className = "app" > < PipecatClientVideo participant = "bot" /> { /* Bot's video feed */ } < PipecatClientAudio /> { /* Audio input/output */ } </ div > </ PipecatClientProvider > ); } The PipecatClientProvider is the root component for providing Pipecat client context to your application. By wrapping your PipecatClientAudio and PipecatClientVideo components in this provider, they can access the client instance and receive and process the streams received from the Pipecat server. ​ Real-time Events The RTVI processors we configured in the pipeline emit events that we can handle in our client: Copy Ask AI // Listen for transcription events useRTVIClientEvent ( RTVIEvent . UserTranscript , ( data : TranscriptData ) => { if ( data . final ) { console . log ( `User said: ${ data . text } ` ); } }); // Listen for bot responses useRTVIClientEvent ( RTVIEvent . BotTranscript , ( data : BotLLMTextData ) => { console . log ( `Bot responded: ${ data . text } ` ); }); Available Events Speaking state changes Transcription updates Bot responses Connection status Performance metrics Event Usage Use these events to: Show speaking indicators Display transcripts Update UI state Monitor performance Optionally, uses callbacks to handle events in your application. Learn more in the Pipecat client docs. ​ Complete Example Here’s a basic client implementation with connection status and transcription display: Copy Ask AI function ChatApp () { return ( < PipecatClientProvider client = { client } > < div className = "app" > { /* Connection UI */ } < StatusDisplay /> < ConnectButton /> { /* Media Components */ } < BotVideo /> < PipecatClientAudio /> { /* Debug/Transcript Display */ } < DebugDisplay /> </ div > </ PipecatClientProvider > ); } Check out the example repository for a complete client implementation with styling and error handling. ​ Running the Application From the simple-chatbot directory, start the server and client to test the chatbot: ​ 1. Start the Server In one terminal: Copy Ask AI python server/server.py ​ 2. Start the Client In another terminal: Copy Ask AI cd examples/react npm install npm run dev ​ 3. Testing the Connection Open http://localhost:5173 in your browser Click “Connect” to join a room Allow microphone access when prompted Start talking with your AI assistant Troubleshooting: Check that all API keys are properly configured in .env Grant your browser access to your microphone, so it can receive your audio input Verify WebRTC ports aren’t blocked by firewalls ​ Next Steps Now that you have a working chatbot, consider these enhancements: Add custom avatar animations Implement function calling for external integrations Add support for multiple languages Enhance error recovery and reconnection logic ​ Examples Foundational Example A basic implementation demonstrating core Gemini Multimodal Live features and transcription capabilities Simple Chatbot A complete client/server implementation showing how to build a Pipecat JS or React client that connects to a Gemini Live Pipecat bot ​ Learn More Gemini Multimodal Live API Reference React Client SDK Documentation Recording Transcripts Metrics On this page What We’ll Build Key Concepts Understanding Pipelines Processors Gemini Integration Prerequisites Server Implementation Environment Setup Server Setup (server.py) Creating Meeting Room Managing Bot Instances Connection Endpoints Bot Implementation (bot-gemini.py) Transport Setup Gemini Service Configuration Conversation Context Processor Setup Pipeline Assembly Client Implementation Connection Setup Media Handling Real-time Events Complete Example Running the Application 1. Start the Server 2. Start the Client 3. Testing the Connection Next Steps Examples Learn More Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/features_gemini-multimodal-live_25b9d7ea.txt b/features_gemini-multimodal-live_25b9d7ea.txt
new file mode 100644
index 0000000000000000000000000000000000000000..4c44bbb48f94bf523466b2a41e069581156faf82
--- /dev/null
+++ b/features_gemini-multimodal-live_25b9d7ea.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/guides/features/gemini-multimodal-live#2-start-the-client
+Title: Building with Gemini Multimodal Live - Pipecat
+==================================================
+
+Building with Gemini Multimodal Live - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Features Building with Gemini Multimodal Live Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Guides Fundamentals Context Management Custom FrameProcessor Detecting Idle Users Ending a Pipeline Function Calling Muting User Input Recording Audio Recording Transcripts Features Gemini Multimodal Live Metrics Noise cancellation with Krisp OpenAI Audio Models and APIs Pipecat Flows Telephony Overview Dial-in: WebRTC (Daily) Dial-in: WebRTC (Twilio + Daily) Dial-in: Twilio (Media Streams) Dialout: WebRTC (Daily) Deploying your bot Overview Deployment pattern Example: Pipecat Cloud Example: Fly.io Example: Cerebrium Example: Modal This guide will walk you through building a real-time AI chatbot using Gemini Multimodal Live and Pipecat. We’ll create a complete application with a Pipecat server and a Pipecat React client that enables natural conversations with an AI assistant. API Reference Gemini Multimodal Live API documentation Example Code Find the complete client and server code in Github Client SDK Pipecat React SDK documentation ​ What We’ll Build In this guide, you’ll create: A FastAPI server that manages bot instances A Gemini-powered conversational AI bot A React client with real-time audio/video A complete pipeline for speech-to-speech interaction ​ Key Concepts Before we dive into implementation, let’s cover some important concepts that will help you understand how Pipecat and Gemini work together. ​ Understanding Pipelines At the heart of Pipecat is the pipeline system. A pipeline is a sequence of processors that handle different aspects of the conversation flow. Think of it like an assembly line where each station (processor) performs a specific task. For our chatbot, the pipeline looks like this: Copy Ask AI pipeline = Pipeline([ transport.input(), # Receives audio/video from the user via WebRTC rtvi, # Handles client/server messaging and events context_aggregator.user(), # Manages user message history llm, # Processes speech through Gemini talking_animation, # Controls bot's avatar transport.output(), # Sends audio/video back to the user via WebRTC context_aggregator.assistant(), # Manages bot message history ]) ​ Processors Each processor in the pipeline handles a specific task: Transport transport.input() and transport.output() handle media streaming with Daily Context context_aggregator maintains conversation history for natural dialogue Speech Processing rtvi_user_transcription and rtvi_bot_transcription handle speech-to-text Animation talking_animation controls the bot’s visual state based on speaking activity The order of processors matters! Data flows through the pipeline in sequence, so each processor should receive the data it needs from previous processors. Learn more about the Core Concepts to Pipecat server. ​ Gemini Integration The GeminiMultimodalLiveLLMService is a speech-to-speech LLM service that interfaces with the Gemini Multimodal Live API. It provides: Real-time speech-to-speech conversation Context management Voice activity detection Tool use Pipecat manages two types of connections: A WebRTC connection between the Pipecat client and server for reliable audio/video streaming A WebSocket connection between the Pipecat server and Gemini for real-time AI processing This architecture ensures stable media streaming while maintaining responsive AI interactions. ​ Prerequisites Before we begin, you’ll need: Python 3.10 or higher Node.js 16 or higher A Daily API key A Google API key with Gemini Multimodal Live access Clone the Pipecat repo: Copy Ask AI git clone git@github.com:pipecat-ai/pipecat.git ​ Server Implementation Let’s start by setting up the server components. Our server will handle bot management, room creation, and client connections. ​ Environment Setup Navigate to the simple-chatbot’s server directory: Copy Ask AI cd examples/simple-chatbot/server Set up a python virtual environment: Copy Ask AI python -m venv venv source venv/bin/activate # On Windows: venv\Scripts\activate Install requirements: Copy Ask AI pip install -r requirements.txt Copy env.example to .env and make a few changes: Copy Ask AI # Remove the hard-coded example room URL DAILY_SAMPLE_ROOM_URL = # Add your Daily and Gemini API keys DAILY_API_KEY = [your key here] GEMINI_API_KEY = [your key here] # Use Gemini implementation BOT_IMPLEMENTATION = gemini ​ Server Setup (server.py) server.py is a FastAPI server that creates the meeting room where clients and bots interact, manages bot instances, and handles client connections. It’s the orchestrator that brings everything on the server-side together. ​ Creating Meeting Room The server uses Daily’s API via a REST API helper to create rooms where clients and bots can meet. Each room is a secure space for audio/video communication: server/server.py Copy Ask AI async def create_room_and_token (): """Create a Daily room and generate access credentials.""" room = await daily_helpers[ "rest" ].create_room(DailyRoomParams()) token = await daily_helpers[ "rest" ].get_token(room.url) return room.url, token ​ Managing Bot Instances When a client connects, the server starts a new bot instance configured specifically for that room. It keeps track of running bots and ensures there’s only one bot per room: server/server.py Copy Ask AI # Start the bot process for a specific room bot_file = "bot-gemini.py" proc = subprocess.Popen([ f "python3 -m { bot_file } -u { room_url } -t { token } " ]) bot_procs[proc.pid] = (proc, room_url) ​ Connection Endpoints The server provides two ways to connect: Browser Access (/) Creates a room, starts a bot, and redirects the browser to the Daily meeting URL. Perfect for quick testing and development. RTVI Client (/connect) Creates a room, starts a bot, and returns connection credentials. Used by RTVI clients for custom implementations. ​ Bot Implementation (bot-gemini.py) The bot implementation connects all the pieces: Daily transport, Gemini service, conversation context, and processors. Let’s break down each component: ​ Transport Setup First, we configure the Daily transport, which handles WebRTC communication between the client and server. server/bot-gemini.py Copy Ask AI transport = DailyTransport( room_url, token, "Chatbot" , DailyParams( audio_in_enabled = True , # Enable audio input audio_out_enabled = True , # Enable audio output video_out_enabled = True , # Enable video output vad_analyzer = SileroVADAnalyzer( params = VADParams( stop_secs = 0.5 )), ), ) Gemini Multimodal Live audio requirements: Input: 16 kHz sample rate Output: 24 kHz sample rate ​ Gemini Service Configuration Next, we initialize the Gemini service which will provide speech-to-speech inference and communication: server/bot-gemini.py Copy Ask AI llm = GeminiMultimodalLiveLLMService( api_key = os.getenv( "GEMINI_API_KEY" ), voice_id = "Puck" , # Choose your bot's voice params = InputParams( temperature = 0.7 ) # Set model input params ) ​ Conversation Context We give our bot its personality and initial instructions: server/bot-gemini.py Copy Ask AI messages = [{ "role" : "user" , "content" : """You are Chatbot, a friendly, helpful robot. Keep responses brief and avoid special characters since output will be converted to audio.""" }] context = OpenAILLMContext(messages) context_aggregator = llm.create_context_aggregator(context) OpenAILLMContext is used as a common LLM base service for context management. In the future, we may add a specific context manager for Gemini. The context aggregator automatically maintains conversation history, helping the bot remember previous interactions. ​ Processor Setup We initialize two additional processors in our pipeline to handle different aspects of the interaction: RTVI Processors RTVIProcessor : Handles all client communication events including transcriptions, speaking states, and performance metrics Animation TalkingAnimation : Controls the bot’s visual state, switching between static and animated frames based on speaking status Learn more about the RTVI framework and available processors. ​ Pipeline Assembly Finally, we bring everything together in a pipeline: server/bot-gemini.py Copy Ask AI pipeline = Pipeline([ transport.input(), # Receive media rtvi, # Client UI events context_aggregator.user(), # Process user context llm, # Gemini processing ta, # Animation (talking/quiet states) transport.output(), # Send media context_aggregator.assistant() # Process bot context ]) task = PipelineTask( pipeline, params = PipelineParams( allow_interruptions = True , enable_metrics = True , enable_usage_metrics = True , ), observers = [RTVIObserver(rtvi)], ) The order of processors is crucial! For example, the RTVI processor should be early in the pipeline to capture all relevant events. The RTVIObserver monitors the entire pipeline and automatically collects relevant events to send to the client. ​ Client Implementation Our React client uses the Pipecat React SDK to communicate with the bot. Let’s explore how the client connects and interacts with our Pipecat server. ​ Connection Setup The client needs to connect to our bot server using the same transport type (Daily WebRTC) that we configured on the server: examples/react/src/providers/PipecatProvider.tsx Copy Ask AI const client = new PipecatClient ({ transport: new DailyTransport (), enableMic: true , // Enable audio input enableCam: false , // Disable video input enableScreenShare: false , // Disable screen sharing }); client . connect ({ endpoint: "http://localhost:7860/connect" , // Your bot connection endpoint }); The connection configuration must match your server: DailyTransport : Matches the WebRTC transport used in bot-gemini.py connect endpoint: Matches the /connect route in server.py Media settings: Controls which devices are enabled on join ​ Media Handling Pipecat’s React components handle all the complex media stream management for you: Copy Ask AI function App () { return ( < PipecatClientProvider client = { client } > < div className = "app" > < PipecatClientVideo participant = "bot" /> { /* Bot's video feed */ } < PipecatClientAudio /> { /* Audio input/output */ } </ div > </ PipecatClientProvider > ); } The PipecatClientProvider is the root component for providing Pipecat client context to your application. By wrapping your PipecatClientAudio and PipecatClientVideo components in this provider, they can access the client instance and receive and process the streams received from the Pipecat server. ​ Real-time Events The RTVI processors we configured in the pipeline emit events that we can handle in our client: Copy Ask AI // Listen for transcription events useRTVIClientEvent ( RTVIEvent . UserTranscript , ( data : TranscriptData ) => { if ( data . final ) { console . log ( `User said: ${ data . text } ` ); } }); // Listen for bot responses useRTVIClientEvent ( RTVIEvent . BotTranscript , ( data : BotLLMTextData ) => { console . log ( `Bot responded: ${ data . text } ` ); }); Available Events Speaking state changes Transcription updates Bot responses Connection status Performance metrics Event Usage Use these events to: Show speaking indicators Display transcripts Update UI state Monitor performance Optionally, uses callbacks to handle events in your application. Learn more in the Pipecat client docs. ​ Complete Example Here’s a basic client implementation with connection status and transcription display: Copy Ask AI function ChatApp () { return ( < PipecatClientProvider client = { client } > < div className = "app" > { /* Connection UI */ } < StatusDisplay /> < ConnectButton /> { /* Media Components */ } < BotVideo /> < PipecatClientAudio /> { /* Debug/Transcript Display */ } < DebugDisplay /> </ div > </ PipecatClientProvider > ); } Check out the example repository for a complete client implementation with styling and error handling. ​ Running the Application From the simple-chatbot directory, start the server and client to test the chatbot: ​ 1. Start the Server In one terminal: Copy Ask AI python server/server.py ​ 2. Start the Client In another terminal: Copy Ask AI cd examples/react npm install npm run dev ​ 3. Testing the Connection Open http://localhost:5173 in your browser Click “Connect” to join a room Allow microphone access when prompted Start talking with your AI assistant Troubleshooting: Check that all API keys are properly configured in .env Grant your browser access to your microphone, so it can receive your audio input Verify WebRTC ports aren’t blocked by firewalls ​ Next Steps Now that you have a working chatbot, consider these enhancements: Add custom avatar animations Implement function calling for external integrations Add support for multiple languages Enhance error recovery and reconnection logic ​ Examples Foundational Example A basic implementation demonstrating core Gemini Multimodal Live features and transcription capabilities Simple Chatbot A complete client/server implementation showing how to build a Pipecat JS or React client that connects to a Gemini Live Pipecat bot ​ Learn More Gemini Multimodal Live API Reference React Client SDK Documentation Recording Transcripts Metrics On this page What We’ll Build Key Concepts Understanding Pipelines Processors Gemini Integration Prerequisites Server Implementation Environment Setup Server Setup (server.py) Creating Meeting Room Managing Bot Instances Connection Endpoints Bot Implementation (bot-gemini.py) Transport Setup Gemini Service Configuration Conversation Context Processor Setup Pipeline Assembly Client Implementation Connection Setup Media Handling Real-time Events Complete Example Running the Application 1. Start the Server 2. Start the Client 3. Testing the Connection Next Steps Examples Learn More Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/features_gemini-multimodal-live_36ba8d11.txt b/features_gemini-multimodal-live_36ba8d11.txt
new file mode 100644
index 0000000000000000000000000000000000000000..6f3d86aa1e454bcc9f8615063befaa77f1eb5a14
--- /dev/null
+++ b/features_gemini-multimodal-live_36ba8d11.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/guides/features/gemini-multimodal-live#connection-setup
+Title: Building with Gemini Multimodal Live - Pipecat
+==================================================
+
+Building with Gemini Multimodal Live - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Features Building with Gemini Multimodal Live Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Guides Fundamentals Context Management Custom FrameProcessor Detecting Idle Users Ending a Pipeline Function Calling Muting User Input Recording Audio Recording Transcripts Features Gemini Multimodal Live Metrics Noise cancellation with Krisp OpenAI Audio Models and APIs Pipecat Flows Telephony Overview Dial-in: WebRTC (Daily) Dial-in: WebRTC (Twilio + Daily) Dial-in: Twilio (Media Streams) Dialout: WebRTC (Daily) Deploying your bot Overview Deployment pattern Example: Pipecat Cloud Example: Fly.io Example: Cerebrium Example: Modal This guide will walk you through building a real-time AI chatbot using Gemini Multimodal Live and Pipecat. We’ll create a complete application with a Pipecat server and a Pipecat React client that enables natural conversations with an AI assistant. API Reference Gemini Multimodal Live API documentation Example Code Find the complete client and server code in Github Client SDK Pipecat React SDK documentation ​ What We’ll Build In this guide, you’ll create: A FastAPI server that manages bot instances A Gemini-powered conversational AI bot A React client with real-time audio/video A complete pipeline for speech-to-speech interaction ​ Key Concepts Before we dive into implementation, let’s cover some important concepts that will help you understand how Pipecat and Gemini work together. ​ Understanding Pipelines At the heart of Pipecat is the pipeline system. A pipeline is a sequence of processors that handle different aspects of the conversation flow. Think of it like an assembly line where each station (processor) performs a specific task. For our chatbot, the pipeline looks like this: Copy Ask AI pipeline = Pipeline([ transport.input(), # Receives audio/video from the user via WebRTC rtvi, # Handles client/server messaging and events context_aggregator.user(), # Manages user message history llm, # Processes speech through Gemini talking_animation, # Controls bot's avatar transport.output(), # Sends audio/video back to the user via WebRTC context_aggregator.assistant(), # Manages bot message history ]) ​ Processors Each processor in the pipeline handles a specific task: Transport transport.input() and transport.output() handle media streaming with Daily Context context_aggregator maintains conversation history for natural dialogue Speech Processing rtvi_user_transcription and rtvi_bot_transcription handle speech-to-text Animation talking_animation controls the bot’s visual state based on speaking activity The order of processors matters! Data flows through the pipeline in sequence, so each processor should receive the data it needs from previous processors. Learn more about the Core Concepts to Pipecat server. ​ Gemini Integration The GeminiMultimodalLiveLLMService is a speech-to-speech LLM service that interfaces with the Gemini Multimodal Live API. It provides: Real-time speech-to-speech conversation Context management Voice activity detection Tool use Pipecat manages two types of connections: A WebRTC connection between the Pipecat client and server for reliable audio/video streaming A WebSocket connection between the Pipecat server and Gemini for real-time AI processing This architecture ensures stable media streaming while maintaining responsive AI interactions. ​ Prerequisites Before we begin, you’ll need: Python 3.10 or higher Node.js 16 or higher A Daily API key A Google API key with Gemini Multimodal Live access Clone the Pipecat repo: Copy Ask AI git clone git@github.com:pipecat-ai/pipecat.git ​ Server Implementation Let’s start by setting up the server components. Our server will handle bot management, room creation, and client connections. ​ Environment Setup Navigate to the simple-chatbot’s server directory: Copy Ask AI cd examples/simple-chatbot/server Set up a python virtual environment: Copy Ask AI python -m venv venv source venv/bin/activate # On Windows: venv\Scripts\activate Install requirements: Copy Ask AI pip install -r requirements.txt Copy env.example to .env and make a few changes: Copy Ask AI # Remove the hard-coded example room URL DAILY_SAMPLE_ROOM_URL = # Add your Daily and Gemini API keys DAILY_API_KEY = [your key here] GEMINI_API_KEY = [your key here] # Use Gemini implementation BOT_IMPLEMENTATION = gemini ​ Server Setup (server.py) server.py is a FastAPI server that creates the meeting room where clients and bots interact, manages bot instances, and handles client connections. It’s the orchestrator that brings everything on the server-side together. ​ Creating Meeting Room The server uses Daily’s API via a REST API helper to create rooms where clients and bots can meet. Each room is a secure space for audio/video communication: server/server.py Copy Ask AI async def create_room_and_token (): """Create a Daily room and generate access credentials.""" room = await daily_helpers[ "rest" ].create_room(DailyRoomParams()) token = await daily_helpers[ "rest" ].get_token(room.url) return room.url, token ​ Managing Bot Instances When a client connects, the server starts a new bot instance configured specifically for that room. It keeps track of running bots and ensures there’s only one bot per room: server/server.py Copy Ask AI # Start the bot process for a specific room bot_file = "bot-gemini.py" proc = subprocess.Popen([ f "python3 -m { bot_file } -u { room_url } -t { token } " ]) bot_procs[proc.pid] = (proc, room_url) ​ Connection Endpoints The server provides two ways to connect: Browser Access (/) Creates a room, starts a bot, and redirects the browser to the Daily meeting URL. Perfect for quick testing and development. RTVI Client (/connect) Creates a room, starts a bot, and returns connection credentials. Used by RTVI clients for custom implementations. ​ Bot Implementation (bot-gemini.py) The bot implementation connects all the pieces: Daily transport, Gemini service, conversation context, and processors. Let’s break down each component: ​ Transport Setup First, we configure the Daily transport, which handles WebRTC communication between the client and server. server/bot-gemini.py Copy Ask AI transport = DailyTransport( room_url, token, "Chatbot" , DailyParams( audio_in_enabled = True , # Enable audio input audio_out_enabled = True , # Enable audio output video_out_enabled = True , # Enable video output vad_analyzer = SileroVADAnalyzer( params = VADParams( stop_secs = 0.5 )), ), ) Gemini Multimodal Live audio requirements: Input: 16 kHz sample rate Output: 24 kHz sample rate ​ Gemini Service Configuration Next, we initialize the Gemini service which will provide speech-to-speech inference and communication: server/bot-gemini.py Copy Ask AI llm = GeminiMultimodalLiveLLMService( api_key = os.getenv( "GEMINI_API_KEY" ), voice_id = "Puck" , # Choose your bot's voice params = InputParams( temperature = 0.7 ) # Set model input params ) ​ Conversation Context We give our bot its personality and initial instructions: server/bot-gemini.py Copy Ask AI messages = [{ "role" : "user" , "content" : """You are Chatbot, a friendly, helpful robot. Keep responses brief and avoid special characters since output will be converted to audio.""" }] context = OpenAILLMContext(messages) context_aggregator = llm.create_context_aggregator(context) OpenAILLMContext is used as a common LLM base service for context management. In the future, we may add a specific context manager for Gemini. The context aggregator automatically maintains conversation history, helping the bot remember previous interactions. ​ Processor Setup We initialize two additional processors in our pipeline to handle different aspects of the interaction: RTVI Processors RTVIProcessor : Handles all client communication events including transcriptions, speaking states, and performance metrics Animation TalkingAnimation : Controls the bot’s visual state, switching between static and animated frames based on speaking status Learn more about the RTVI framework and available processors. ​ Pipeline Assembly Finally, we bring everything together in a pipeline: server/bot-gemini.py Copy Ask AI pipeline = Pipeline([ transport.input(), # Receive media rtvi, # Client UI events context_aggregator.user(), # Process user context llm, # Gemini processing ta, # Animation (talking/quiet states) transport.output(), # Send media context_aggregator.assistant() # Process bot context ]) task = PipelineTask( pipeline, params = PipelineParams( allow_interruptions = True , enable_metrics = True , enable_usage_metrics = True , ), observers = [RTVIObserver(rtvi)], ) The order of processors is crucial! For example, the RTVI processor should be early in the pipeline to capture all relevant events. The RTVIObserver monitors the entire pipeline and automatically collects relevant events to send to the client. ​ Client Implementation Our React client uses the Pipecat React SDK to communicate with the bot. Let’s explore how the client connects and interacts with our Pipecat server. ​ Connection Setup The client needs to connect to our bot server using the same transport type (Daily WebRTC) that we configured on the server: examples/react/src/providers/PipecatProvider.tsx Copy Ask AI const client = new PipecatClient ({ transport: new DailyTransport (), enableMic: true , // Enable audio input enableCam: false , // Disable video input enableScreenShare: false , // Disable screen sharing }); client . connect ({ endpoint: "http://localhost:7860/connect" , // Your bot connection endpoint }); The connection configuration must match your server: DailyTransport : Matches the WebRTC transport used in bot-gemini.py connect endpoint: Matches the /connect route in server.py Media settings: Controls which devices are enabled on join ​ Media Handling Pipecat’s React components handle all the complex media stream management for you: Copy Ask AI function App () { return ( < PipecatClientProvider client = { client } > < div className = "app" > < PipecatClientVideo participant = "bot" /> { /* Bot's video feed */ } < PipecatClientAudio /> { /* Audio input/output */ } </ div > </ PipecatClientProvider > ); } The PipecatClientProvider is the root component for providing Pipecat client context to your application. By wrapping your PipecatClientAudio and PipecatClientVideo components in this provider, they can access the client instance and receive and process the streams received from the Pipecat server. ​ Real-time Events The RTVI processors we configured in the pipeline emit events that we can handle in our client: Copy Ask AI // Listen for transcription events useRTVIClientEvent ( RTVIEvent . UserTranscript , ( data : TranscriptData ) => { if ( data . final ) { console . log ( `User said: ${ data . text } ` ); } }); // Listen for bot responses useRTVIClientEvent ( RTVIEvent . BotTranscript , ( data : BotLLMTextData ) => { console . log ( `Bot responded: ${ data . text } ` ); }); Available Events Speaking state changes Transcription updates Bot responses Connection status Performance metrics Event Usage Use these events to: Show speaking indicators Display transcripts Update UI state Monitor performance Optionally, uses callbacks to handle events in your application. Learn more in the Pipecat client docs. ​ Complete Example Here’s a basic client implementation with connection status and transcription display: Copy Ask AI function ChatApp () { return ( < PipecatClientProvider client = { client } > < div className = "app" > { /* Connection UI */ } < StatusDisplay /> < ConnectButton /> { /* Media Components */ } < BotVideo /> < PipecatClientAudio /> { /* Debug/Transcript Display */ } < DebugDisplay /> </ div > </ PipecatClientProvider > ); } Check out the example repository for a complete client implementation with styling and error handling. ​ Running the Application From the simple-chatbot directory, start the server and client to test the chatbot: ​ 1. Start the Server In one terminal: Copy Ask AI python server/server.py ​ 2. Start the Client In another terminal: Copy Ask AI cd examples/react npm install npm run dev ​ 3. Testing the Connection Open http://localhost:5173 in your browser Click “Connect” to join a room Allow microphone access when prompted Start talking with your AI assistant Troubleshooting: Check that all API keys are properly configured in .env Grant your browser access to your microphone, so it can receive your audio input Verify WebRTC ports aren’t blocked by firewalls ​ Next Steps Now that you have a working chatbot, consider these enhancements: Add custom avatar animations Implement function calling for external integrations Add support for multiple languages Enhance error recovery and reconnection logic ​ Examples Foundational Example A basic implementation demonstrating core Gemini Multimodal Live features and transcription capabilities Simple Chatbot A complete client/server implementation showing how to build a Pipecat JS or React client that connects to a Gemini Live Pipecat bot ​ Learn More Gemini Multimodal Live API Reference React Client SDK Documentation Recording Transcripts Metrics On this page What We’ll Build Key Concepts Understanding Pipelines Processors Gemini Integration Prerequisites Server Implementation Environment Setup Server Setup (server.py) Creating Meeting Room Managing Bot Instances Connection Endpoints Bot Implementation (bot-gemini.py) Transport Setup Gemini Service Configuration Conversation Context Processor Setup Pipeline Assembly Client Implementation Connection Setup Media Handling Real-time Events Complete Example Running the Application 1. Start the Server 2. Start the Client 3. Testing the Connection Next Steps Examples Learn More Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/features_gemini-multimodal-live_5ef7ac07.txt b/features_gemini-multimodal-live_5ef7ac07.txt
new file mode 100644
index 0000000000000000000000000000000000000000..3a6dcbcfe7b8f8fd6509dcabcc77f40ac012c0d8
--- /dev/null
+++ b/features_gemini-multimodal-live_5ef7ac07.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/guides/features/gemini-multimodal-live#understanding-pipelines
+Title: Building with Gemini Multimodal Live - Pipecat
+==================================================
+
+Building with Gemini Multimodal Live - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Features Building with Gemini Multimodal Live Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Guides Fundamentals Context Management Custom FrameProcessor Detecting Idle Users Ending a Pipeline Function Calling Muting User Input Recording Audio Recording Transcripts Features Gemini Multimodal Live Metrics Noise cancellation with Krisp OpenAI Audio Models and APIs Pipecat Flows Telephony Overview Dial-in: WebRTC (Daily) Dial-in: WebRTC (Twilio + Daily) Dial-in: Twilio (Media Streams) Dialout: WebRTC (Daily) Deploying your bot Overview Deployment pattern Example: Pipecat Cloud Example: Fly.io Example: Cerebrium Example: Modal This guide will walk you through building a real-time AI chatbot using Gemini Multimodal Live and Pipecat. We’ll create a complete application with a Pipecat server and a Pipecat React client that enables natural conversations with an AI assistant. API Reference Gemini Multimodal Live API documentation Example Code Find the complete client and server code in Github Client SDK Pipecat React SDK documentation ​ What We’ll Build In this guide, you’ll create: A FastAPI server that manages bot instances A Gemini-powered conversational AI bot A React client with real-time audio/video A complete pipeline for speech-to-speech interaction ​ Key Concepts Before we dive into implementation, let’s cover some important concepts that will help you understand how Pipecat and Gemini work together. ​ Understanding Pipelines At the heart of Pipecat is the pipeline system. A pipeline is a sequence of processors that handle different aspects of the conversation flow. Think of it like an assembly line where each station (processor) performs a specific task. For our chatbot, the pipeline looks like this: Copy Ask AI pipeline = Pipeline([ transport.input(), # Receives audio/video from the user via WebRTC rtvi, # Handles client/server messaging and events context_aggregator.user(), # Manages user message history llm, # Processes speech through Gemini talking_animation, # Controls bot's avatar transport.output(), # Sends audio/video back to the user via WebRTC context_aggregator.assistant(), # Manages bot message history ]) ​ Processors Each processor in the pipeline handles a specific task: Transport transport.input() and transport.output() handle media streaming with Daily Context context_aggregator maintains conversation history for natural dialogue Speech Processing rtvi_user_transcription and rtvi_bot_transcription handle speech-to-text Animation talking_animation controls the bot’s visual state based on speaking activity The order of processors matters! Data flows through the pipeline in sequence, so each processor should receive the data it needs from previous processors. Learn more about the Core Concepts to Pipecat server. ​ Gemini Integration The GeminiMultimodalLiveLLMService is a speech-to-speech LLM service that interfaces with the Gemini Multimodal Live API. It provides: Real-time speech-to-speech conversation Context management Voice activity detection Tool use Pipecat manages two types of connections: A WebRTC connection between the Pipecat client and server for reliable audio/video streaming A WebSocket connection between the Pipecat server and Gemini for real-time AI processing This architecture ensures stable media streaming while maintaining responsive AI interactions. ​ Prerequisites Before we begin, you’ll need: Python 3.10 or higher Node.js 16 or higher A Daily API key A Google API key with Gemini Multimodal Live access Clone the Pipecat repo: Copy Ask AI git clone git@github.com:pipecat-ai/pipecat.git ​ Server Implementation Let’s start by setting up the server components. Our server will handle bot management, room creation, and client connections. ​ Environment Setup Navigate to the simple-chatbot’s server directory: Copy Ask AI cd examples/simple-chatbot/server Set up a python virtual environment: Copy Ask AI python -m venv venv source venv/bin/activate # On Windows: venv\Scripts\activate Install requirements: Copy Ask AI pip install -r requirements.txt Copy env.example to .env and make a few changes: Copy Ask AI # Remove the hard-coded example room URL DAILY_SAMPLE_ROOM_URL = # Add your Daily and Gemini API keys DAILY_API_KEY = [your key here] GEMINI_API_KEY = [your key here] # Use Gemini implementation BOT_IMPLEMENTATION = gemini ​ Server Setup (server.py) server.py is a FastAPI server that creates the meeting room where clients and bots interact, manages bot instances, and handles client connections. It’s the orchestrator that brings everything on the server-side together. ​ Creating Meeting Room The server uses Daily’s API via a REST API helper to create rooms where clients and bots can meet. Each room is a secure space for audio/video communication: server/server.py Copy Ask AI async def create_room_and_token (): """Create a Daily room and generate access credentials.""" room = await daily_helpers[ "rest" ].create_room(DailyRoomParams()) token = await daily_helpers[ "rest" ].get_token(room.url) return room.url, token ​ Managing Bot Instances When a client connects, the server starts a new bot instance configured specifically for that room. It keeps track of running bots and ensures there’s only one bot per room: server/server.py Copy Ask AI # Start the bot process for a specific room bot_file = "bot-gemini.py" proc = subprocess.Popen([ f "python3 -m { bot_file } -u { room_url } -t { token } " ]) bot_procs[proc.pid] = (proc, room_url) ​ Connection Endpoints The server provides two ways to connect: Browser Access (/) Creates a room, starts a bot, and redirects the browser to the Daily meeting URL. Perfect for quick testing and development. RTVI Client (/connect) Creates a room, starts a bot, and returns connection credentials. Used by RTVI clients for custom implementations. ​ Bot Implementation (bot-gemini.py) The bot implementation connects all the pieces: Daily transport, Gemini service, conversation context, and processors. Let’s break down each component: ​ Transport Setup First, we configure the Daily transport, which handles WebRTC communication between the client and server. server/bot-gemini.py Copy Ask AI transport = DailyTransport( room_url, token, "Chatbot" , DailyParams( audio_in_enabled = True , # Enable audio input audio_out_enabled = True , # Enable audio output video_out_enabled = True , # Enable video output vad_analyzer = SileroVADAnalyzer( params = VADParams( stop_secs = 0.5 )), ), ) Gemini Multimodal Live audio requirements: Input: 16 kHz sample rate Output: 24 kHz sample rate ​ Gemini Service Configuration Next, we initialize the Gemini service which will provide speech-to-speech inference and communication: server/bot-gemini.py Copy Ask AI llm = GeminiMultimodalLiveLLMService( api_key = os.getenv( "GEMINI_API_KEY" ), voice_id = "Puck" , # Choose your bot's voice params = InputParams( temperature = 0.7 ) # Set model input params ) ​ Conversation Context We give our bot its personality and initial instructions: server/bot-gemini.py Copy Ask AI messages = [{ "role" : "user" , "content" : """You are Chatbot, a friendly, helpful robot. Keep responses brief and avoid special characters since output will be converted to audio.""" }] context = OpenAILLMContext(messages) context_aggregator = llm.create_context_aggregator(context) OpenAILLMContext is used as a common LLM base service for context management. In the future, we may add a specific context manager for Gemini. The context aggregator automatically maintains conversation history, helping the bot remember previous interactions. ​ Processor Setup We initialize two additional processors in our pipeline to handle different aspects of the interaction: RTVI Processors RTVIProcessor : Handles all client communication events including transcriptions, speaking states, and performance metrics Animation TalkingAnimation : Controls the bot’s visual state, switching between static and animated frames based on speaking status Learn more about the RTVI framework and available processors. ​ Pipeline Assembly Finally, we bring everything together in a pipeline: server/bot-gemini.py Copy Ask AI pipeline = Pipeline([ transport.input(), # Receive media rtvi, # Client UI events context_aggregator.user(), # Process user context llm, # Gemini processing ta, # Animation (talking/quiet states) transport.output(), # Send media context_aggregator.assistant() # Process bot context ]) task = PipelineTask( pipeline, params = PipelineParams( allow_interruptions = True , enable_metrics = True , enable_usage_metrics = True , ), observers = [RTVIObserver(rtvi)], ) The order of processors is crucial! For example, the RTVI processor should be early in the pipeline to capture all relevant events. The RTVIObserver monitors the entire pipeline and automatically collects relevant events to send to the client. ​ Client Implementation Our React client uses the Pipecat React SDK to communicate with the bot. Let’s explore how the client connects and interacts with our Pipecat server. ​ Connection Setup The client needs to connect to our bot server using the same transport type (Daily WebRTC) that we configured on the server: examples/react/src/providers/PipecatProvider.tsx Copy Ask AI const client = new PipecatClient ({ transport: new DailyTransport (), enableMic: true , // Enable audio input enableCam: false , // Disable video input enableScreenShare: false , // Disable screen sharing }); client . connect ({ endpoint: "http://localhost:7860/connect" , // Your bot connection endpoint }); The connection configuration must match your server: DailyTransport : Matches the WebRTC transport used in bot-gemini.py connect endpoint: Matches the /connect route in server.py Media settings: Controls which devices are enabled on join ​ Media Handling Pipecat’s React components handle all the complex media stream management for you: Copy Ask AI function App () { return ( < PipecatClientProvider client = { client } > < div className = "app" > < PipecatClientVideo participant = "bot" /> { /* Bot's video feed */ } < PipecatClientAudio /> { /* Audio input/output */ } </ div > </ PipecatClientProvider > ); } The PipecatClientProvider is the root component for providing Pipecat client context to your application. By wrapping your PipecatClientAudio and PipecatClientVideo components in this provider, they can access the client instance and receive and process the streams received from the Pipecat server. ​ Real-time Events The RTVI processors we configured in the pipeline emit events that we can handle in our client: Copy Ask AI // Listen for transcription events useRTVIClientEvent ( RTVIEvent . UserTranscript , ( data : TranscriptData ) => { if ( data . final ) { console . log ( `User said: ${ data . text } ` ); } }); // Listen for bot responses useRTVIClientEvent ( RTVIEvent . BotTranscript , ( data : BotLLMTextData ) => { console . log ( `Bot responded: ${ data . text } ` ); }); Available Events Speaking state changes Transcription updates Bot responses Connection status Performance metrics Event Usage Use these events to: Show speaking indicators Display transcripts Update UI state Monitor performance Optionally, uses callbacks to handle events in your application. Learn more in the Pipecat client docs. ​ Complete Example Here’s a basic client implementation with connection status and transcription display: Copy Ask AI function ChatApp () { return ( < PipecatClientProvider client = { client } > < div className = "app" > { /* Connection UI */ } < StatusDisplay /> < ConnectButton /> { /* Media Components */ } < BotVideo /> < PipecatClientAudio /> { /* Debug/Transcript Display */ } < DebugDisplay /> </ div > </ PipecatClientProvider > ); } Check out the example repository for a complete client implementation with styling and error handling. ​ Running the Application From the simple-chatbot directory, start the server and client to test the chatbot: ​ 1. Start the Server In one terminal: Copy Ask AI python server/server.py ​ 2. Start the Client In another terminal: Copy Ask AI cd examples/react npm install npm run dev ​ 3. Testing the Connection Open http://localhost:5173 in your browser Click “Connect” to join a room Allow microphone access when prompted Start talking with your AI assistant Troubleshooting: Check that all API keys are properly configured in .env Grant your browser access to your microphone, so it can receive your audio input Verify WebRTC ports aren’t blocked by firewalls ​ Next Steps Now that you have a working chatbot, consider these enhancements: Add custom avatar animations Implement function calling for external integrations Add support for multiple languages Enhance error recovery and reconnection logic ​ Examples Foundational Example A basic implementation demonstrating core Gemini Multimodal Live features and transcription capabilities Simple Chatbot A complete client/server implementation showing how to build a Pipecat JS or React client that connects to a Gemini Live Pipecat bot ​ Learn More Gemini Multimodal Live API Reference React Client SDK Documentation Recording Transcripts Metrics On this page What We’ll Build Key Concepts Understanding Pipelines Processors Gemini Integration Prerequisites Server Implementation Environment Setup Server Setup (server.py) Creating Meeting Room Managing Bot Instances Connection Endpoints Bot Implementation (bot-gemini.py) Transport Setup Gemini Service Configuration Conversation Context Processor Setup Pipeline Assembly Client Implementation Connection Setup Media Handling Real-time Events Complete Example Running the Application 1. Start the Server 2. Start the Client 3. Testing the Connection Next Steps Examples Learn More Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/features_gemini-multimodal-live_894a29f1.txt b/features_gemini-multimodal-live_894a29f1.txt
new file mode 100644
index 0000000000000000000000000000000000000000..8c25d752198e15b56fb951a19bdd41709bd93f30
--- /dev/null
+++ b/features_gemini-multimodal-live_894a29f1.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/guides/features/gemini-multimodal-live#transport-setup
+Title: Building with Gemini Multimodal Live - Pipecat
+==================================================
+
+Building with Gemini Multimodal Live - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Features Building with Gemini Multimodal Live Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Guides Fundamentals Context Management Custom FrameProcessor Detecting Idle Users Ending a Pipeline Function Calling Muting User Input Recording Audio Recording Transcripts Features Gemini Multimodal Live Metrics Noise cancellation with Krisp OpenAI Audio Models and APIs Pipecat Flows Telephony Overview Dial-in: WebRTC (Daily) Dial-in: WebRTC (Twilio + Daily) Dial-in: Twilio (Media Streams) Dialout: WebRTC (Daily) Deploying your bot Overview Deployment pattern Example: Pipecat Cloud Example: Fly.io Example: Cerebrium Example: Modal This guide will walk you through building a real-time AI chatbot using Gemini Multimodal Live and Pipecat. We’ll create a complete application with a Pipecat server and a Pipecat React client that enables natural conversations with an AI assistant. API Reference Gemini Multimodal Live API documentation Example Code Find the complete client and server code in Github Client SDK Pipecat React SDK documentation ​ What We’ll Build In this guide, you’ll create: A FastAPI server that manages bot instances A Gemini-powered conversational AI bot A React client with real-time audio/video A complete pipeline for speech-to-speech interaction ​ Key Concepts Before we dive into implementation, let’s cover some important concepts that will help you understand how Pipecat and Gemini work together. ​ Understanding Pipelines At the heart of Pipecat is the pipeline system. A pipeline is a sequence of processors that handle different aspects of the conversation flow. Think of it like an assembly line where each station (processor) performs a specific task. For our chatbot, the pipeline looks like this: Copy Ask AI pipeline = Pipeline([ transport.input(), # Receives audio/video from the user via WebRTC rtvi, # Handles client/server messaging and events context_aggregator.user(), # Manages user message history llm, # Processes speech through Gemini talking_animation, # Controls bot's avatar transport.output(), # Sends audio/video back to the user via WebRTC context_aggregator.assistant(), # Manages bot message history ]) ​ Processors Each processor in the pipeline handles a specific task: Transport transport.input() and transport.output() handle media streaming with Daily Context context_aggregator maintains conversation history for natural dialogue Speech Processing rtvi_user_transcription and rtvi_bot_transcription handle speech-to-text Animation talking_animation controls the bot’s visual state based on speaking activity The order of processors matters! Data flows through the pipeline in sequence, so each processor should receive the data it needs from previous processors. Learn more about the Core Concepts to Pipecat server. ​ Gemini Integration The GeminiMultimodalLiveLLMService is a speech-to-speech LLM service that interfaces with the Gemini Multimodal Live API. It provides: Real-time speech-to-speech conversation Context management Voice activity detection Tool use Pipecat manages two types of connections: A WebRTC connection between the Pipecat client and server for reliable audio/video streaming A WebSocket connection between the Pipecat server and Gemini for real-time AI processing This architecture ensures stable media streaming while maintaining responsive AI interactions. ​ Prerequisites Before we begin, you’ll need: Python 3.10 or higher Node.js 16 or higher A Daily API key A Google API key with Gemini Multimodal Live access Clone the Pipecat repo: Copy Ask AI git clone git@github.com:pipecat-ai/pipecat.git ​ Server Implementation Let’s start by setting up the server components. Our server will handle bot management, room creation, and client connections. ​ Environment Setup Navigate to the simple-chatbot’s server directory: Copy Ask AI cd examples/simple-chatbot/server Set up a python virtual environment: Copy Ask AI python -m venv venv source venv/bin/activate # On Windows: venv\Scripts\activate Install requirements: Copy Ask AI pip install -r requirements.txt Copy env.example to .env and make a few changes: Copy Ask AI # Remove the hard-coded example room URL DAILY_SAMPLE_ROOM_URL = # Add your Daily and Gemini API keys DAILY_API_KEY = [your key here] GEMINI_API_KEY = [your key here] # Use Gemini implementation BOT_IMPLEMENTATION = gemini ​ Server Setup (server.py) server.py is a FastAPI server that creates the meeting room where clients and bots interact, manages bot instances, and handles client connections. It’s the orchestrator that brings everything on the server-side together. ​ Creating Meeting Room The server uses Daily’s API via a REST API helper to create rooms where clients and bots can meet. Each room is a secure space for audio/video communication: server/server.py Copy Ask AI async def create_room_and_token (): """Create a Daily room and generate access credentials.""" room = await daily_helpers[ "rest" ].create_room(DailyRoomParams()) token = await daily_helpers[ "rest" ].get_token(room.url) return room.url, token ​ Managing Bot Instances When a client connects, the server starts a new bot instance configured specifically for that room. It keeps track of running bots and ensures there’s only one bot per room: server/server.py Copy Ask AI # Start the bot process for a specific room bot_file = "bot-gemini.py" proc = subprocess.Popen([ f "python3 -m { bot_file } -u { room_url } -t { token } " ]) bot_procs[proc.pid] = (proc, room_url) ​ Connection Endpoints The server provides two ways to connect: Browser Access (/) Creates a room, starts a bot, and redirects the browser to the Daily meeting URL. Perfect for quick testing and development. RTVI Client (/connect) Creates a room, starts a bot, and returns connection credentials. Used by RTVI clients for custom implementations. ​ Bot Implementation (bot-gemini.py) The bot implementation connects all the pieces: Daily transport, Gemini service, conversation context, and processors. Let’s break down each component: ​ Transport Setup First, we configure the Daily transport, which handles WebRTC communication between the client and server. server/bot-gemini.py Copy Ask AI transport = DailyTransport( room_url, token, "Chatbot" , DailyParams( audio_in_enabled = True , # Enable audio input audio_out_enabled = True , # Enable audio output video_out_enabled = True , # Enable video output vad_analyzer = SileroVADAnalyzer( params = VADParams( stop_secs = 0.5 )), ), ) Gemini Multimodal Live audio requirements: Input: 16 kHz sample rate Output: 24 kHz sample rate ​ Gemini Service Configuration Next, we initialize the Gemini service which will provide speech-to-speech inference and communication: server/bot-gemini.py Copy Ask AI llm = GeminiMultimodalLiveLLMService( api_key = os.getenv( "GEMINI_API_KEY" ), voice_id = "Puck" , # Choose your bot's voice params = InputParams( temperature = 0.7 ) # Set model input params ) ​ Conversation Context We give our bot its personality and initial instructions: server/bot-gemini.py Copy Ask AI messages = [{ "role" : "user" , "content" : """You are Chatbot, a friendly, helpful robot. Keep responses brief and avoid special characters since output will be converted to audio.""" }] context = OpenAILLMContext(messages) context_aggregator = llm.create_context_aggregator(context) OpenAILLMContext is used as a common LLM base service for context management. In the future, we may add a specific context manager for Gemini. The context aggregator automatically maintains conversation history, helping the bot remember previous interactions. ​ Processor Setup We initialize two additional processors in our pipeline to handle different aspects of the interaction: RTVI Processors RTVIProcessor : Handles all client communication events including transcriptions, speaking states, and performance metrics Animation TalkingAnimation : Controls the bot’s visual state, switching between static and animated frames based on speaking status Learn more about the RTVI framework and available processors. ​ Pipeline Assembly Finally, we bring everything together in a pipeline: server/bot-gemini.py Copy Ask AI pipeline = Pipeline([ transport.input(), # Receive media rtvi, # Client UI events context_aggregator.user(), # Process user context llm, # Gemini processing ta, # Animation (talking/quiet states) transport.output(), # Send media context_aggregator.assistant() # Process bot context ]) task = PipelineTask( pipeline, params = PipelineParams( allow_interruptions = True , enable_metrics = True , enable_usage_metrics = True , ), observers = [RTVIObserver(rtvi)], ) The order of processors is crucial! For example, the RTVI processor should be early in the pipeline to capture all relevant events. The RTVIObserver monitors the entire pipeline and automatically collects relevant events to send to the client. ​ Client Implementation Our React client uses the Pipecat React SDK to communicate with the bot. Let’s explore how the client connects and interacts with our Pipecat server. ​ Connection Setup The client needs to connect to our bot server using the same transport type (Daily WebRTC) that we configured on the server: examples/react/src/providers/PipecatProvider.tsx Copy Ask AI const client = new PipecatClient ({ transport: new DailyTransport (), enableMic: true , // Enable audio input enableCam: false , // Disable video input enableScreenShare: false , // Disable screen sharing }); client . connect ({ endpoint: "http://localhost:7860/connect" , // Your bot connection endpoint }); The connection configuration must match your server: DailyTransport : Matches the WebRTC transport used in bot-gemini.py connect endpoint: Matches the /connect route in server.py Media settings: Controls which devices are enabled on join ​ Media Handling Pipecat’s React components handle all the complex media stream management for you: Copy Ask AI function App () { return ( < PipecatClientProvider client = { client } > < div className = "app" > < PipecatClientVideo participant = "bot" /> { /* Bot's video feed */ } < PipecatClientAudio /> { /* Audio input/output */ } </ div > </ PipecatClientProvider > ); } The PipecatClientProvider is the root component for providing Pipecat client context to your application. By wrapping your PipecatClientAudio and PipecatClientVideo components in this provider, they can access the client instance and receive and process the streams received from the Pipecat server. ​ Real-time Events The RTVI processors we configured in the pipeline emit events that we can handle in our client: Copy Ask AI // Listen for transcription events useRTVIClientEvent ( RTVIEvent . UserTranscript , ( data : TranscriptData ) => { if ( data . final ) { console . log ( `User said: ${ data . text } ` ); } }); // Listen for bot responses useRTVIClientEvent ( RTVIEvent . BotTranscript , ( data : BotLLMTextData ) => { console . log ( `Bot responded: ${ data . text } ` ); }); Available Events Speaking state changes Transcription updates Bot responses Connection status Performance metrics Event Usage Use these events to: Show speaking indicators Display transcripts Update UI state Monitor performance Optionally, uses callbacks to handle events in your application. Learn more in the Pipecat client docs. ​ Complete Example Here’s a basic client implementation with connection status and transcription display: Copy Ask AI function ChatApp () { return ( < PipecatClientProvider client = { client } > < div className = "app" > { /* Connection UI */ } < StatusDisplay /> < ConnectButton /> { /* Media Components */ } < BotVideo /> < PipecatClientAudio /> { /* Debug/Transcript Display */ } < DebugDisplay /> </ div > </ PipecatClientProvider > ); } Check out the example repository for a complete client implementation with styling and error handling. ​ Running the Application From the simple-chatbot directory, start the server and client to test the chatbot: ​ 1. Start the Server In one terminal: Copy Ask AI python server/server.py ​ 2. Start the Client In another terminal: Copy Ask AI cd examples/react npm install npm run dev ​ 3. Testing the Connection Open http://localhost:5173 in your browser Click “Connect” to join a room Allow microphone access when prompted Start talking with your AI assistant Troubleshooting: Check that all API keys are properly configured in .env Grant your browser access to your microphone, so it can receive your audio input Verify WebRTC ports aren’t blocked by firewalls ​ Next Steps Now that you have a working chatbot, consider these enhancements: Add custom avatar animations Implement function calling for external integrations Add support for multiple languages Enhance error recovery and reconnection logic ​ Examples Foundational Example A basic implementation demonstrating core Gemini Multimodal Live features and transcription capabilities Simple Chatbot A complete client/server implementation showing how to build a Pipecat JS or React client that connects to a Gemini Live Pipecat bot ​ Learn More Gemini Multimodal Live API Reference React Client SDK Documentation Recording Transcripts Metrics On this page What We’ll Build Key Concepts Understanding Pipelines Processors Gemini Integration Prerequisites Server Implementation Environment Setup Server Setup (server.py) Creating Meeting Room Managing Bot Instances Connection Endpoints Bot Implementation (bot-gemini.py) Transport Setup Gemini Service Configuration Conversation Context Processor Setup Pipeline Assembly Client Implementation Connection Setup Media Handling Real-time Events Complete Example Running the Application 1. Start the Server 2. Start the Client 3. Testing the Connection Next Steps Examples Learn More Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/features_gemini-multimodal-live_8b71475e.txt b/features_gemini-multimodal-live_8b71475e.txt
new file mode 100644
index 0000000000000000000000000000000000000000..5418296d97e297919ff2f61659efb279dd0b9e4b
--- /dev/null
+++ b/features_gemini-multimodal-live_8b71475e.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/guides/features/gemini-multimodal-live#media-handling
+Title: Building with Gemini Multimodal Live - Pipecat
+==================================================
+
+Building with Gemini Multimodal Live - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Features Building with Gemini Multimodal Live Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Guides Fundamentals Context Management Custom FrameProcessor Detecting Idle Users Ending a Pipeline Function Calling Muting User Input Recording Audio Recording Transcripts Features Gemini Multimodal Live Metrics Noise cancellation with Krisp OpenAI Audio Models and APIs Pipecat Flows Telephony Overview Dial-in: WebRTC (Daily) Dial-in: WebRTC (Twilio + Daily) Dial-in: Twilio (Media Streams) Dialout: WebRTC (Daily) Deploying your bot Overview Deployment pattern Example: Pipecat Cloud Example: Fly.io Example: Cerebrium Example: Modal This guide will walk you through building a real-time AI chatbot using Gemini Multimodal Live and Pipecat. We’ll create a complete application with a Pipecat server and a Pipecat React client that enables natural conversations with an AI assistant. API Reference Gemini Multimodal Live API documentation Example Code Find the complete client and server code in Github Client SDK Pipecat React SDK documentation ​ What We’ll Build In this guide, you’ll create: A FastAPI server that manages bot instances A Gemini-powered conversational AI bot A React client with real-time audio/video A complete pipeline for speech-to-speech interaction ​ Key Concepts Before we dive into implementation, let’s cover some important concepts that will help you understand how Pipecat and Gemini work together. ​ Understanding Pipelines At the heart of Pipecat is the pipeline system. A pipeline is a sequence of processors that handle different aspects of the conversation flow. Think of it like an assembly line where each station (processor) performs a specific task. For our chatbot, the pipeline looks like this: Copy Ask AI pipeline = Pipeline([ transport.input(), # Receives audio/video from the user via WebRTC rtvi, # Handles client/server messaging and events context_aggregator.user(), # Manages user message history llm, # Processes speech through Gemini talking_animation, # Controls bot's avatar transport.output(), # Sends audio/video back to the user via WebRTC context_aggregator.assistant(), # Manages bot message history ]) ​ Processors Each processor in the pipeline handles a specific task: Transport transport.input() and transport.output() handle media streaming with Daily Context context_aggregator maintains conversation history for natural dialogue Speech Processing rtvi_user_transcription and rtvi_bot_transcription handle speech-to-text Animation talking_animation controls the bot’s visual state based on speaking activity The order of processors matters! Data flows through the pipeline in sequence, so each processor should receive the data it needs from previous processors. Learn more about the Core Concepts to Pipecat server. ​ Gemini Integration The GeminiMultimodalLiveLLMService is a speech-to-speech LLM service that interfaces with the Gemini Multimodal Live API. It provides: Real-time speech-to-speech conversation Context management Voice activity detection Tool use Pipecat manages two types of connections: A WebRTC connection between the Pipecat client and server for reliable audio/video streaming A WebSocket connection between the Pipecat server and Gemini for real-time AI processing This architecture ensures stable media streaming while maintaining responsive AI interactions. ​ Prerequisites Before we begin, you’ll need: Python 3.10 or higher Node.js 16 or higher A Daily API key A Google API key with Gemini Multimodal Live access Clone the Pipecat repo: Copy Ask AI git clone git@github.com:pipecat-ai/pipecat.git ​ Server Implementation Let’s start by setting up the server components. Our server will handle bot management, room creation, and client connections. ​ Environment Setup Navigate to the simple-chatbot’s server directory: Copy Ask AI cd examples/simple-chatbot/server Set up a python virtual environment: Copy Ask AI python -m venv venv source venv/bin/activate # On Windows: venv\Scripts\activate Install requirements: Copy Ask AI pip install -r requirements.txt Copy env.example to .env and make a few changes: Copy Ask AI # Remove the hard-coded example room URL DAILY_SAMPLE_ROOM_URL = # Add your Daily and Gemini API keys DAILY_API_KEY = [your key here] GEMINI_API_KEY = [your key here] # Use Gemini implementation BOT_IMPLEMENTATION = gemini ​ Server Setup (server.py) server.py is a FastAPI server that creates the meeting room where clients and bots interact, manages bot instances, and handles client connections. It’s the orchestrator that brings everything on the server-side together. ​ Creating Meeting Room The server uses Daily’s API via a REST API helper to create rooms where clients and bots can meet. Each room is a secure space for audio/video communication: server/server.py Copy Ask AI async def create_room_and_token (): """Create a Daily room and generate access credentials.""" room = await daily_helpers[ "rest" ].create_room(DailyRoomParams()) token = await daily_helpers[ "rest" ].get_token(room.url) return room.url, token ​ Managing Bot Instances When a client connects, the server starts a new bot instance configured specifically for that room. It keeps track of running bots and ensures there’s only one bot per room: server/server.py Copy Ask AI # Start the bot process for a specific room bot_file = "bot-gemini.py" proc = subprocess.Popen([ f "python3 -m { bot_file } -u { room_url } -t { token } " ]) bot_procs[proc.pid] = (proc, room_url) ​ Connection Endpoints The server provides two ways to connect: Browser Access (/) Creates a room, starts a bot, and redirects the browser to the Daily meeting URL. Perfect for quick testing and development. RTVI Client (/connect) Creates a room, starts a bot, and returns connection credentials. Used by RTVI clients for custom implementations. ​ Bot Implementation (bot-gemini.py) The bot implementation connects all the pieces: Daily transport, Gemini service, conversation context, and processors. Let’s break down each component: ​ Transport Setup First, we configure the Daily transport, which handles WebRTC communication between the client and server. server/bot-gemini.py Copy Ask AI transport = DailyTransport( room_url, token, "Chatbot" , DailyParams( audio_in_enabled = True , # Enable audio input audio_out_enabled = True , # Enable audio output video_out_enabled = True , # Enable video output vad_analyzer = SileroVADAnalyzer( params = VADParams( stop_secs = 0.5 )), ), ) Gemini Multimodal Live audio requirements: Input: 16 kHz sample rate Output: 24 kHz sample rate ​ Gemini Service Configuration Next, we initialize the Gemini service which will provide speech-to-speech inference and communication: server/bot-gemini.py Copy Ask AI llm = GeminiMultimodalLiveLLMService( api_key = os.getenv( "GEMINI_API_KEY" ), voice_id = "Puck" , # Choose your bot's voice params = InputParams( temperature = 0.7 ) # Set model input params ) ​ Conversation Context We give our bot its personality and initial instructions: server/bot-gemini.py Copy Ask AI messages = [{ "role" : "user" , "content" : """You are Chatbot, a friendly, helpful robot. Keep responses brief and avoid special characters since output will be converted to audio.""" }] context = OpenAILLMContext(messages) context_aggregator = llm.create_context_aggregator(context) OpenAILLMContext is used as a common LLM base service for context management. In the future, we may add a specific context manager for Gemini. The context aggregator automatically maintains conversation history, helping the bot remember previous interactions. ​ Processor Setup We initialize two additional processors in our pipeline to handle different aspects of the interaction: RTVI Processors RTVIProcessor : Handles all client communication events including transcriptions, speaking states, and performance metrics Animation TalkingAnimation : Controls the bot’s visual state, switching between static and animated frames based on speaking status Learn more about the RTVI framework and available processors. ​ Pipeline Assembly Finally, we bring everything together in a pipeline: server/bot-gemini.py Copy Ask AI pipeline = Pipeline([ transport.input(), # Receive media rtvi, # Client UI events context_aggregator.user(), # Process user context llm, # Gemini processing ta, # Animation (talking/quiet states) transport.output(), # Send media context_aggregator.assistant() # Process bot context ]) task = PipelineTask( pipeline, params = PipelineParams( allow_interruptions = True , enable_metrics = True , enable_usage_metrics = True , ), observers = [RTVIObserver(rtvi)], ) The order of processors is crucial! For example, the RTVI processor should be early in the pipeline to capture all relevant events. The RTVIObserver monitors the entire pipeline and automatically collects relevant events to send to the client. ​ Client Implementation Our React client uses the Pipecat React SDK to communicate with the bot. Let’s explore how the client connects and interacts with our Pipecat server. ​ Connection Setup The client needs to connect to our bot server using the same transport type (Daily WebRTC) that we configured on the server: examples/react/src/providers/PipecatProvider.tsx Copy Ask AI const client = new PipecatClient ({ transport: new DailyTransport (), enableMic: true , // Enable audio input enableCam: false , // Disable video input enableScreenShare: false , // Disable screen sharing }); client . connect ({ endpoint: "http://localhost:7860/connect" , // Your bot connection endpoint }); The connection configuration must match your server: DailyTransport : Matches the WebRTC transport used in bot-gemini.py connect endpoint: Matches the /connect route in server.py Media settings: Controls which devices are enabled on join ​ Media Handling Pipecat’s React components handle all the complex media stream management for you: Copy Ask AI function App () { return ( < PipecatClientProvider client = { client } > < div className = "app" > < PipecatClientVideo participant = "bot" /> { /* Bot's video feed */ } < PipecatClientAudio /> { /* Audio input/output */ } </ div > </ PipecatClientProvider > ); } The PipecatClientProvider is the root component for providing Pipecat client context to your application. By wrapping your PipecatClientAudio and PipecatClientVideo components in this provider, they can access the client instance and receive and process the streams received from the Pipecat server. ​ Real-time Events The RTVI processors we configured in the pipeline emit events that we can handle in our client: Copy Ask AI // Listen for transcription events useRTVIClientEvent ( RTVIEvent . UserTranscript , ( data : TranscriptData ) => { if ( data . final ) { console . log ( `User said: ${ data . text } ` ); } }); // Listen for bot responses useRTVIClientEvent ( RTVIEvent . BotTranscript , ( data : BotLLMTextData ) => { console . log ( `Bot responded: ${ data . text } ` ); }); Available Events Speaking state changes Transcription updates Bot responses Connection status Performance metrics Event Usage Use these events to: Show speaking indicators Display transcripts Update UI state Monitor performance Optionally, uses callbacks to handle events in your application. Learn more in the Pipecat client docs. ​ Complete Example Here’s a basic client implementation with connection status and transcription display: Copy Ask AI function ChatApp () { return ( < PipecatClientProvider client = { client } > < div className = "app" > { /* Connection UI */ } < StatusDisplay /> < ConnectButton /> { /* Media Components */ } < BotVideo /> < PipecatClientAudio /> { /* Debug/Transcript Display */ } < DebugDisplay /> </ div > </ PipecatClientProvider > ); } Check out the example repository for a complete client implementation with styling and error handling. ​ Running the Application From the simple-chatbot directory, start the server and client to test the chatbot: ​ 1. Start the Server In one terminal: Copy Ask AI python server/server.py ​ 2. Start the Client In another terminal: Copy Ask AI cd examples/react npm install npm run dev ​ 3. Testing the Connection Open http://localhost:5173 in your browser Click “Connect” to join a room Allow microphone access when prompted Start talking with your AI assistant Troubleshooting: Check that all API keys are properly configured in .env Grant your browser access to your microphone, so it can receive your audio input Verify WebRTC ports aren’t blocked by firewalls ​ Next Steps Now that you have a working chatbot, consider these enhancements: Add custom avatar animations Implement function calling for external integrations Add support for multiple languages Enhance error recovery and reconnection logic ​ Examples Foundational Example A basic implementation demonstrating core Gemini Multimodal Live features and transcription capabilities Simple Chatbot A complete client/server implementation showing how to build a Pipecat JS or React client that connects to a Gemini Live Pipecat bot ​ Learn More Gemini Multimodal Live API Reference React Client SDK Documentation Recording Transcripts Metrics On this page What We’ll Build Key Concepts Understanding Pipelines Processors Gemini Integration Prerequisites Server Implementation Environment Setup Server Setup (server.py) Creating Meeting Room Managing Bot Instances Connection Endpoints Bot Implementation (bot-gemini.py) Transport Setup Gemini Service Configuration Conversation Context Processor Setup Pipeline Assembly Client Implementation Connection Setup Media Handling Real-time Events Complete Example Running the Application 1. Start the Server 2. Start the Client 3. Testing the Connection Next Steps Examples Learn More Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/features_gemini-multimodal-live_98893317.txt b/features_gemini-multimodal-live_98893317.txt
new file mode 100644
index 0000000000000000000000000000000000000000..94bdcf9065c6e228d9eb0c5d3ea64412922a0716
--- /dev/null
+++ b/features_gemini-multimodal-live_98893317.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/guides/features/gemini-multimodal-live#conversation-context
+Title: Building with Gemini Multimodal Live - Pipecat
+==================================================
+
+Building with Gemini Multimodal Live - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Features Building with Gemini Multimodal Live Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Guides Fundamentals Context Management Custom FrameProcessor Detecting Idle Users Ending a Pipeline Function Calling Muting User Input Recording Audio Recording Transcripts Features Gemini Multimodal Live Metrics Noise cancellation with Krisp OpenAI Audio Models and APIs Pipecat Flows Telephony Overview Dial-in: WebRTC (Daily) Dial-in: WebRTC (Twilio + Daily) Dial-in: Twilio (Media Streams) Dialout: WebRTC (Daily) Deploying your bot Overview Deployment pattern Example: Pipecat Cloud Example: Fly.io Example: Cerebrium Example: Modal This guide will walk you through building a real-time AI chatbot using Gemini Multimodal Live and Pipecat. We’ll create a complete application with a Pipecat server and a Pipecat React client that enables natural conversations with an AI assistant. API Reference Gemini Multimodal Live API documentation Example Code Find the complete client and server code in Github Client SDK Pipecat React SDK documentation ​ What We’ll Build In this guide, you’ll create: A FastAPI server that manages bot instances A Gemini-powered conversational AI bot A React client with real-time audio/video A complete pipeline for speech-to-speech interaction ​ Key Concepts Before we dive into implementation, let’s cover some important concepts that will help you understand how Pipecat and Gemini work together. ​ Understanding Pipelines At the heart of Pipecat is the pipeline system. A pipeline is a sequence of processors that handle different aspects of the conversation flow. Think of it like an assembly line where each station (processor) performs a specific task. For our chatbot, the pipeline looks like this: Copy Ask AI pipeline = Pipeline([ transport.input(), # Receives audio/video from the user via WebRTC rtvi, # Handles client/server messaging and events context_aggregator.user(), # Manages user message history llm, # Processes speech through Gemini talking_animation, # Controls bot's avatar transport.output(), # Sends audio/video back to the user via WebRTC context_aggregator.assistant(), # Manages bot message history ]) ​ Processors Each processor in the pipeline handles a specific task: Transport transport.input() and transport.output() handle media streaming with Daily Context context_aggregator maintains conversation history for natural dialogue Speech Processing rtvi_user_transcription and rtvi_bot_transcription handle speech-to-text Animation talking_animation controls the bot’s visual state based on speaking activity The order of processors matters! Data flows through the pipeline in sequence, so each processor should receive the data it needs from previous processors. Learn more about the Core Concepts to Pipecat server. ​ Gemini Integration The GeminiMultimodalLiveLLMService is a speech-to-speech LLM service that interfaces with the Gemini Multimodal Live API. It provides: Real-time speech-to-speech conversation Context management Voice activity detection Tool use Pipecat manages two types of connections: A WebRTC connection between the Pipecat client and server for reliable audio/video streaming A WebSocket connection between the Pipecat server and Gemini for real-time AI processing This architecture ensures stable media streaming while maintaining responsive AI interactions. ​ Prerequisites Before we begin, you’ll need: Python 3.10 or higher Node.js 16 or higher A Daily API key A Google API key with Gemini Multimodal Live access Clone the Pipecat repo: Copy Ask AI git clone git@github.com:pipecat-ai/pipecat.git ​ Server Implementation Let’s start by setting up the server components. Our server will handle bot management, room creation, and client connections. ​ Environment Setup Navigate to the simple-chatbot’s server directory: Copy Ask AI cd examples/simple-chatbot/server Set up a python virtual environment: Copy Ask AI python -m venv venv source venv/bin/activate # On Windows: venv\Scripts\activate Install requirements: Copy Ask AI pip install -r requirements.txt Copy env.example to .env and make a few changes: Copy Ask AI # Remove the hard-coded example room URL DAILY_SAMPLE_ROOM_URL = # Add your Daily and Gemini API keys DAILY_API_KEY = [your key here] GEMINI_API_KEY = [your key here] # Use Gemini implementation BOT_IMPLEMENTATION = gemini ​ Server Setup (server.py) server.py is a FastAPI server that creates the meeting room where clients and bots interact, manages bot instances, and handles client connections. It’s the orchestrator that brings everything on the server-side together. ​ Creating Meeting Room The server uses Daily’s API via a REST API helper to create rooms where clients and bots can meet. Each room is a secure space for audio/video communication: server/server.py Copy Ask AI async def create_room_and_token (): """Create a Daily room and generate access credentials.""" room = await daily_helpers[ "rest" ].create_room(DailyRoomParams()) token = await daily_helpers[ "rest" ].get_token(room.url) return room.url, token ​ Managing Bot Instances When a client connects, the server starts a new bot instance configured specifically for that room. It keeps track of running bots and ensures there’s only one bot per room: server/server.py Copy Ask AI # Start the bot process for a specific room bot_file = "bot-gemini.py" proc = subprocess.Popen([ f "python3 -m { bot_file } -u { room_url } -t { token } " ]) bot_procs[proc.pid] = (proc, room_url) ​ Connection Endpoints The server provides two ways to connect: Browser Access (/) Creates a room, starts a bot, and redirects the browser to the Daily meeting URL. Perfect for quick testing and development. RTVI Client (/connect) Creates a room, starts a bot, and returns connection credentials. Used by RTVI clients for custom implementations. ​ Bot Implementation (bot-gemini.py) The bot implementation connects all the pieces: Daily transport, Gemini service, conversation context, and processors. Let’s break down each component: ​ Transport Setup First, we configure the Daily transport, which handles WebRTC communication between the client and server. server/bot-gemini.py Copy Ask AI transport = DailyTransport( room_url, token, "Chatbot" , DailyParams( audio_in_enabled = True , # Enable audio input audio_out_enabled = True , # Enable audio output video_out_enabled = True , # Enable video output vad_analyzer = SileroVADAnalyzer( params = VADParams( stop_secs = 0.5 )), ), ) Gemini Multimodal Live audio requirements: Input: 16 kHz sample rate Output: 24 kHz sample rate ​ Gemini Service Configuration Next, we initialize the Gemini service which will provide speech-to-speech inference and communication: server/bot-gemini.py Copy Ask AI llm = GeminiMultimodalLiveLLMService( api_key = os.getenv( "GEMINI_API_KEY" ), voice_id = "Puck" , # Choose your bot's voice params = InputParams( temperature = 0.7 ) # Set model input params ) ​ Conversation Context We give our bot its personality and initial instructions: server/bot-gemini.py Copy Ask AI messages = [{ "role" : "user" , "content" : """You are Chatbot, a friendly, helpful robot. Keep responses brief and avoid special characters since output will be converted to audio.""" }] context = OpenAILLMContext(messages) context_aggregator = llm.create_context_aggregator(context) OpenAILLMContext is used as a common LLM base service for context management. In the future, we may add a specific context manager for Gemini. The context aggregator automatically maintains conversation history, helping the bot remember previous interactions. ​ Processor Setup We initialize two additional processors in our pipeline to handle different aspects of the interaction: RTVI Processors RTVIProcessor : Handles all client communication events including transcriptions, speaking states, and performance metrics Animation TalkingAnimation : Controls the bot’s visual state, switching between static and animated frames based on speaking status Learn more about the RTVI framework and available processors. ​ Pipeline Assembly Finally, we bring everything together in a pipeline: server/bot-gemini.py Copy Ask AI pipeline = Pipeline([ transport.input(), # Receive media rtvi, # Client UI events context_aggregator.user(), # Process user context llm, # Gemini processing ta, # Animation (talking/quiet states) transport.output(), # Send media context_aggregator.assistant() # Process bot context ]) task = PipelineTask( pipeline, params = PipelineParams( allow_interruptions = True , enable_metrics = True , enable_usage_metrics = True , ), observers = [RTVIObserver(rtvi)], ) The order of processors is crucial! For example, the RTVI processor should be early in the pipeline to capture all relevant events. The RTVIObserver monitors the entire pipeline and automatically collects relevant events to send to the client. ​ Client Implementation Our React client uses the Pipecat React SDK to communicate with the bot. Let’s explore how the client connects and interacts with our Pipecat server. ​ Connection Setup The client needs to connect to our bot server using the same transport type (Daily WebRTC) that we configured on the server: examples/react/src/providers/PipecatProvider.tsx Copy Ask AI const client = new PipecatClient ({ transport: new DailyTransport (), enableMic: true , // Enable audio input enableCam: false , // Disable video input enableScreenShare: false , // Disable screen sharing }); client . connect ({ endpoint: "http://localhost:7860/connect" , // Your bot connection endpoint }); The connection configuration must match your server: DailyTransport : Matches the WebRTC transport used in bot-gemini.py connect endpoint: Matches the /connect route in server.py Media settings: Controls which devices are enabled on join ​ Media Handling Pipecat’s React components handle all the complex media stream management for you: Copy Ask AI function App () { return ( < PipecatClientProvider client = { client } > < div className = "app" > < PipecatClientVideo participant = "bot" /> { /* Bot's video feed */ } < PipecatClientAudio /> { /* Audio input/output */ } </ div > </ PipecatClientProvider > ); } The PipecatClientProvider is the root component for providing Pipecat client context to your application. By wrapping your PipecatClientAudio and PipecatClientVideo components in this provider, they can access the client instance and receive and process the streams received from the Pipecat server. ​ Real-time Events The RTVI processors we configured in the pipeline emit events that we can handle in our client: Copy Ask AI // Listen for transcription events useRTVIClientEvent ( RTVIEvent . UserTranscript , ( data : TranscriptData ) => { if ( data . final ) { console . log ( `User said: ${ data . text } ` ); } }); // Listen for bot responses useRTVIClientEvent ( RTVIEvent . BotTranscript , ( data : BotLLMTextData ) => { console . log ( `Bot responded: ${ data . text } ` ); }); Available Events Speaking state changes Transcription updates Bot responses Connection status Performance metrics Event Usage Use these events to: Show speaking indicators Display transcripts Update UI state Monitor performance Optionally, uses callbacks to handle events in your application. Learn more in the Pipecat client docs. ​ Complete Example Here’s a basic client implementation with connection status and transcription display: Copy Ask AI function ChatApp () { return ( < PipecatClientProvider client = { client } > < div className = "app" > { /* Connection UI */ } < StatusDisplay /> < ConnectButton /> { /* Media Components */ } < BotVideo /> < PipecatClientAudio /> { /* Debug/Transcript Display */ } < DebugDisplay /> </ div > </ PipecatClientProvider > ); } Check out the example repository for a complete client implementation with styling and error handling. ​ Running the Application From the simple-chatbot directory, start the server and client to test the chatbot: ​ 1. Start the Server In one terminal: Copy Ask AI python server/server.py ​ 2. Start the Client In another terminal: Copy Ask AI cd examples/react npm install npm run dev ​ 3. Testing the Connection Open http://localhost:5173 in your browser Click “Connect” to join a room Allow microphone access when prompted Start talking with your AI assistant Troubleshooting: Check that all API keys are properly configured in .env Grant your browser access to your microphone, so it can receive your audio input Verify WebRTC ports aren’t blocked by firewalls ​ Next Steps Now that you have a working chatbot, consider these enhancements: Add custom avatar animations Implement function calling for external integrations Add support for multiple languages Enhance error recovery and reconnection logic ​ Examples Foundational Example A basic implementation demonstrating core Gemini Multimodal Live features and transcription capabilities Simple Chatbot A complete client/server implementation showing how to build a Pipecat JS or React client that connects to a Gemini Live Pipecat bot ​ Learn More Gemini Multimodal Live API Reference React Client SDK Documentation Recording Transcripts Metrics On this page What We’ll Build Key Concepts Understanding Pipelines Processors Gemini Integration Prerequisites Server Implementation Environment Setup Server Setup (server.py) Creating Meeting Room Managing Bot Instances Connection Endpoints Bot Implementation (bot-gemini.py) Transport Setup Gemini Service Configuration Conversation Context Processor Setup Pipeline Assembly Client Implementation Connection Setup Media Handling Real-time Events Complete Example Running the Application 1. Start the Server 2. Start the Client 3. Testing the Connection Next Steps Examples Learn More Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/features_gemini-multimodal-live_c0a4f183.txt b/features_gemini-multimodal-live_c0a4f183.txt
new file mode 100644
index 0000000000000000000000000000000000000000..8d57b709268abb3ed8e3d7a322a7bc126263b342
--- /dev/null
+++ b/features_gemini-multimodal-live_c0a4f183.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/guides/features/gemini-multimodal-live#learn-more
+Title: Building with Gemini Multimodal Live - Pipecat
+==================================================
+
+Building with Gemini Multimodal Live - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Features Building with Gemini Multimodal Live Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Guides Fundamentals Context Management Custom FrameProcessor Detecting Idle Users Ending a Pipeline Function Calling Muting User Input Recording Audio Recording Transcripts Features Gemini Multimodal Live Metrics Noise cancellation with Krisp OpenAI Audio Models and APIs Pipecat Flows Telephony Overview Dial-in: WebRTC (Daily) Dial-in: WebRTC (Twilio + Daily) Dial-in: Twilio (Media Streams) Dialout: WebRTC (Daily) Deploying your bot Overview Deployment pattern Example: Pipecat Cloud Example: Fly.io Example: Cerebrium Example: Modal This guide will walk you through building a real-time AI chatbot using Gemini Multimodal Live and Pipecat. We’ll create a complete application with a Pipecat server and a Pipecat React client that enables natural conversations with an AI assistant. API Reference Gemini Multimodal Live API documentation Example Code Find the complete client and server code in Github Client SDK Pipecat React SDK documentation ​ What We’ll Build In this guide, you’ll create: A FastAPI server that manages bot instances A Gemini-powered conversational AI bot A React client with real-time audio/video A complete pipeline for speech-to-speech interaction ​ Key Concepts Before we dive into implementation, let’s cover some important concepts that will help you understand how Pipecat and Gemini work together. ​ Understanding Pipelines At the heart of Pipecat is the pipeline system. A pipeline is a sequence of processors that handle different aspects of the conversation flow. Think of it like an assembly line where each station (processor) performs a specific task. For our chatbot, the pipeline looks like this: Copy Ask AI pipeline = Pipeline([ transport.input(), # Receives audio/video from the user via WebRTC rtvi, # Handles client/server messaging and events context_aggregator.user(), # Manages user message history llm, # Processes speech through Gemini talking_animation, # Controls bot's avatar transport.output(), # Sends audio/video back to the user via WebRTC context_aggregator.assistant(), # Manages bot message history ]) ​ Processors Each processor in the pipeline handles a specific task: Transport transport.input() and transport.output() handle media streaming with Daily Context context_aggregator maintains conversation history for natural dialogue Speech Processing rtvi_user_transcription and rtvi_bot_transcription handle speech-to-text Animation talking_animation controls the bot’s visual state based on speaking activity The order of processors matters! Data flows through the pipeline in sequence, so each processor should receive the data it needs from previous processors. Learn more about the Core Concepts to Pipecat server. ​ Gemini Integration The GeminiMultimodalLiveLLMService is a speech-to-speech LLM service that interfaces with the Gemini Multimodal Live API. It provides: Real-time speech-to-speech conversation Context management Voice activity detection Tool use Pipecat manages two types of connections: A WebRTC connection between the Pipecat client and server for reliable audio/video streaming A WebSocket connection between the Pipecat server and Gemini for real-time AI processing This architecture ensures stable media streaming while maintaining responsive AI interactions. ​ Prerequisites Before we begin, you’ll need: Python 3.10 or higher Node.js 16 or higher A Daily API key A Google API key with Gemini Multimodal Live access Clone the Pipecat repo: Copy Ask AI git clone git@github.com:pipecat-ai/pipecat.git ​ Server Implementation Let’s start by setting up the server components. Our server will handle bot management, room creation, and client connections. ​ Environment Setup Navigate to the simple-chatbot’s server directory: Copy Ask AI cd examples/simple-chatbot/server Set up a python virtual environment: Copy Ask AI python -m venv venv source venv/bin/activate # On Windows: venv\Scripts\activate Install requirements: Copy Ask AI pip install -r requirements.txt Copy env.example to .env and make a few changes: Copy Ask AI # Remove the hard-coded example room URL DAILY_SAMPLE_ROOM_URL = # Add your Daily and Gemini API keys DAILY_API_KEY = [your key here] GEMINI_API_KEY = [your key here] # Use Gemini implementation BOT_IMPLEMENTATION = gemini ​ Server Setup (server.py) server.py is a FastAPI server that creates the meeting room where clients and bots interact, manages bot instances, and handles client connections. It’s the orchestrator that brings everything on the server-side together. ​ Creating Meeting Room The server uses Daily’s API via a REST API helper to create rooms where clients and bots can meet. Each room is a secure space for audio/video communication: server/server.py Copy Ask AI async def create_room_and_token (): """Create a Daily room and generate access credentials.""" room = await daily_helpers[ "rest" ].create_room(DailyRoomParams()) token = await daily_helpers[ "rest" ].get_token(room.url) return room.url, token ​ Managing Bot Instances When a client connects, the server starts a new bot instance configured specifically for that room. It keeps track of running bots and ensures there’s only one bot per room: server/server.py Copy Ask AI # Start the bot process for a specific room bot_file = "bot-gemini.py" proc = subprocess.Popen([ f "python3 -m { bot_file } -u { room_url } -t { token } " ]) bot_procs[proc.pid] = (proc, room_url) ​ Connection Endpoints The server provides two ways to connect: Browser Access (/) Creates a room, starts a bot, and redirects the browser to the Daily meeting URL. Perfect for quick testing and development. RTVI Client (/connect) Creates a room, starts a bot, and returns connection credentials. Used by RTVI clients for custom implementations. ​ Bot Implementation (bot-gemini.py) The bot implementation connects all the pieces: Daily transport, Gemini service, conversation context, and processors. Let’s break down each component: ​ Transport Setup First, we configure the Daily transport, which handles WebRTC communication between the client and server. server/bot-gemini.py Copy Ask AI transport = DailyTransport( room_url, token, "Chatbot" , DailyParams( audio_in_enabled = True , # Enable audio input audio_out_enabled = True , # Enable audio output video_out_enabled = True , # Enable video output vad_analyzer = SileroVADAnalyzer( params = VADParams( stop_secs = 0.5 )), ), ) Gemini Multimodal Live audio requirements: Input: 16 kHz sample rate Output: 24 kHz sample rate ​ Gemini Service Configuration Next, we initialize the Gemini service which will provide speech-to-speech inference and communication: server/bot-gemini.py Copy Ask AI llm = GeminiMultimodalLiveLLMService( api_key = os.getenv( "GEMINI_API_KEY" ), voice_id = "Puck" , # Choose your bot's voice params = InputParams( temperature = 0.7 ) # Set model input params ) ​ Conversation Context We give our bot its personality and initial instructions: server/bot-gemini.py Copy Ask AI messages = [{ "role" : "user" , "content" : """You are Chatbot, a friendly, helpful robot. Keep responses brief and avoid special characters since output will be converted to audio.""" }] context = OpenAILLMContext(messages) context_aggregator = llm.create_context_aggregator(context) OpenAILLMContext is used as a common LLM base service for context management. In the future, we may add a specific context manager for Gemini. The context aggregator automatically maintains conversation history, helping the bot remember previous interactions. ​ Processor Setup We initialize two additional processors in our pipeline to handle different aspects of the interaction: RTVI Processors RTVIProcessor : Handles all client communication events including transcriptions, speaking states, and performance metrics Animation TalkingAnimation : Controls the bot’s visual state, switching between static and animated frames based on speaking status Learn more about the RTVI framework and available processors. ​ Pipeline Assembly Finally, we bring everything together in a pipeline: server/bot-gemini.py Copy Ask AI pipeline = Pipeline([ transport.input(), # Receive media rtvi, # Client UI events context_aggregator.user(), # Process user context llm, # Gemini processing ta, # Animation (talking/quiet states) transport.output(), # Send media context_aggregator.assistant() # Process bot context ]) task = PipelineTask( pipeline, params = PipelineParams( allow_interruptions = True , enable_metrics = True , enable_usage_metrics = True , ), observers = [RTVIObserver(rtvi)], ) The order of processors is crucial! For example, the RTVI processor should be early in the pipeline to capture all relevant events. The RTVIObserver monitors the entire pipeline and automatically collects relevant events to send to the client. ​ Client Implementation Our React client uses the Pipecat React SDK to communicate with the bot. Let’s explore how the client connects and interacts with our Pipecat server. ​ Connection Setup The client needs to connect to our bot server using the same transport type (Daily WebRTC) that we configured on the server: examples/react/src/providers/PipecatProvider.tsx Copy Ask AI const client = new PipecatClient ({ transport: new DailyTransport (), enableMic: true , // Enable audio input enableCam: false , // Disable video input enableScreenShare: false , // Disable screen sharing }); client . connect ({ endpoint: "http://localhost:7860/connect" , // Your bot connection endpoint }); The connection configuration must match your server: DailyTransport : Matches the WebRTC transport used in bot-gemini.py connect endpoint: Matches the /connect route in server.py Media settings: Controls which devices are enabled on join ​ Media Handling Pipecat’s React components handle all the complex media stream management for you: Copy Ask AI function App () { return ( < PipecatClientProvider client = { client } > < div className = "app" > < PipecatClientVideo participant = "bot" /> { /* Bot's video feed */ } < PipecatClientAudio /> { /* Audio input/output */ } </ div > </ PipecatClientProvider > ); } The PipecatClientProvider is the root component for providing Pipecat client context to your application. By wrapping your PipecatClientAudio and PipecatClientVideo components in this provider, they can access the client instance and receive and process the streams received from the Pipecat server. ​ Real-time Events The RTVI processors we configured in the pipeline emit events that we can handle in our client: Copy Ask AI // Listen for transcription events useRTVIClientEvent ( RTVIEvent . UserTranscript , ( data : TranscriptData ) => { if ( data . final ) { console . log ( `User said: ${ data . text } ` ); } }); // Listen for bot responses useRTVIClientEvent ( RTVIEvent . BotTranscript , ( data : BotLLMTextData ) => { console . log ( `Bot responded: ${ data . text } ` ); }); Available Events Speaking state changes Transcription updates Bot responses Connection status Performance metrics Event Usage Use these events to: Show speaking indicators Display transcripts Update UI state Monitor performance Optionally, uses callbacks to handle events in your application. Learn more in the Pipecat client docs. ​ Complete Example Here’s a basic client implementation with connection status and transcription display: Copy Ask AI function ChatApp () { return ( < PipecatClientProvider client = { client } > < div className = "app" > { /* Connection UI */ } < StatusDisplay /> < ConnectButton /> { /* Media Components */ } < BotVideo /> < PipecatClientAudio /> { /* Debug/Transcript Display */ } < DebugDisplay /> </ div > </ PipecatClientProvider > ); } Check out the example repository for a complete client implementation with styling and error handling. ​ Running the Application From the simple-chatbot directory, start the server and client to test the chatbot: ​ 1. Start the Server In one terminal: Copy Ask AI python server/server.py ​ 2. Start the Client In another terminal: Copy Ask AI cd examples/react npm install npm run dev ​ 3. Testing the Connection Open http://localhost:5173 in your browser Click “Connect” to join a room Allow microphone access when prompted Start talking with your AI assistant Troubleshooting: Check that all API keys are properly configured in .env Grant your browser access to your microphone, so it can receive your audio input Verify WebRTC ports aren’t blocked by firewalls ​ Next Steps Now that you have a working chatbot, consider these enhancements: Add custom avatar animations Implement function calling for external integrations Add support for multiple languages Enhance error recovery and reconnection logic ​ Examples Foundational Example A basic implementation demonstrating core Gemini Multimodal Live features and transcription capabilities Simple Chatbot A complete client/server implementation showing how to build a Pipecat JS or React client that connects to a Gemini Live Pipecat bot ​ Learn More Gemini Multimodal Live API Reference React Client SDK Documentation Recording Transcripts Metrics On this page What We’ll Build Key Concepts Understanding Pipelines Processors Gemini Integration Prerequisites Server Implementation Environment Setup Server Setup (server.py) Creating Meeting Room Managing Bot Instances Connection Endpoints Bot Implementation (bot-gemini.py) Transport Setup Gemini Service Configuration Conversation Context Processor Setup Pipeline Assembly Client Implementation Connection Setup Media Handling Real-time Events Complete Example Running the Application 1. Start the Server 2. Start the Client 3. Testing the Connection Next Steps Examples Learn More Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/features_gemini-multimodal-live_dc39dc27.txt b/features_gemini-multimodal-live_dc39dc27.txt
new file mode 100644
index 0000000000000000000000000000000000000000..689a11458a3f4ca5b44accf196370ede55b83e0a
--- /dev/null
+++ b/features_gemini-multimodal-live_dc39dc27.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/guides/features/gemini-multimodal-live#client-implementation
+Title: Building with Gemini Multimodal Live - Pipecat
+==================================================
+
+Building with Gemini Multimodal Live - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Features Building with Gemini Multimodal Live Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Guides Fundamentals Context Management Custom FrameProcessor Detecting Idle Users Ending a Pipeline Function Calling Muting User Input Recording Audio Recording Transcripts Features Gemini Multimodal Live Metrics Noise cancellation with Krisp OpenAI Audio Models and APIs Pipecat Flows Telephony Overview Dial-in: WebRTC (Daily) Dial-in: WebRTC (Twilio + Daily) Dial-in: Twilio (Media Streams) Dialout: WebRTC (Daily) Deploying your bot Overview Deployment pattern Example: Pipecat Cloud Example: Fly.io Example: Cerebrium Example: Modal This guide will walk you through building a real-time AI chatbot using Gemini Multimodal Live and Pipecat. We’ll create a complete application with a Pipecat server and a Pipecat React client that enables natural conversations with an AI assistant. API Reference Gemini Multimodal Live API documentation Example Code Find the complete client and server code in Github Client SDK Pipecat React SDK documentation ​ What We’ll Build In this guide, you’ll create: A FastAPI server that manages bot instances A Gemini-powered conversational AI bot A React client with real-time audio/video A complete pipeline for speech-to-speech interaction ​ Key Concepts Before we dive into implementation, let’s cover some important concepts that will help you understand how Pipecat and Gemini work together. ​ Understanding Pipelines At the heart of Pipecat is the pipeline system. A pipeline is a sequence of processors that handle different aspects of the conversation flow. Think of it like an assembly line where each station (processor) performs a specific task. For our chatbot, the pipeline looks like this: Copy Ask AI pipeline = Pipeline([ transport.input(), # Receives audio/video from the user via WebRTC rtvi, # Handles client/server messaging and events context_aggregator.user(), # Manages user message history llm, # Processes speech through Gemini talking_animation, # Controls bot's avatar transport.output(), # Sends audio/video back to the user via WebRTC context_aggregator.assistant(), # Manages bot message history ]) ​ Processors Each processor in the pipeline handles a specific task: Transport transport.input() and transport.output() handle media streaming with Daily Context context_aggregator maintains conversation history for natural dialogue Speech Processing rtvi_user_transcription and rtvi_bot_transcription handle speech-to-text Animation talking_animation controls the bot’s visual state based on speaking activity The order of processors matters! Data flows through the pipeline in sequence, so each processor should receive the data it needs from previous processors. Learn more about the Core Concepts to Pipecat server. ​ Gemini Integration The GeminiMultimodalLiveLLMService is a speech-to-speech LLM service that interfaces with the Gemini Multimodal Live API. It provides: Real-time speech-to-speech conversation Context management Voice activity detection Tool use Pipecat manages two types of connections: A WebRTC connection between the Pipecat client and server for reliable audio/video streaming A WebSocket connection between the Pipecat server and Gemini for real-time AI processing This architecture ensures stable media streaming while maintaining responsive AI interactions. ​ Prerequisites Before we begin, you’ll need: Python 3.10 or higher Node.js 16 or higher A Daily API key A Google API key with Gemini Multimodal Live access Clone the Pipecat repo: Copy Ask AI git clone git@github.com:pipecat-ai/pipecat.git ​ Server Implementation Let’s start by setting up the server components. Our server will handle bot management, room creation, and client connections. ​ Environment Setup Navigate to the simple-chatbot’s server directory: Copy Ask AI cd examples/simple-chatbot/server Set up a python virtual environment: Copy Ask AI python -m venv venv source venv/bin/activate # On Windows: venv\Scripts\activate Install requirements: Copy Ask AI pip install -r requirements.txt Copy env.example to .env and make a few changes: Copy Ask AI # Remove the hard-coded example room URL DAILY_SAMPLE_ROOM_URL = # Add your Daily and Gemini API keys DAILY_API_KEY = [your key here] GEMINI_API_KEY = [your key here] # Use Gemini implementation BOT_IMPLEMENTATION = gemini ​ Server Setup (server.py) server.py is a FastAPI server that creates the meeting room where clients and bots interact, manages bot instances, and handles client connections. It’s the orchestrator that brings everything on the server-side together. ​ Creating Meeting Room The server uses Daily’s API via a REST API helper to create rooms where clients and bots can meet. Each room is a secure space for audio/video communication: server/server.py Copy Ask AI async def create_room_and_token (): """Create a Daily room and generate access credentials.""" room = await daily_helpers[ "rest" ].create_room(DailyRoomParams()) token = await daily_helpers[ "rest" ].get_token(room.url) return room.url, token ​ Managing Bot Instances When a client connects, the server starts a new bot instance configured specifically for that room. It keeps track of running bots and ensures there’s only one bot per room: server/server.py Copy Ask AI # Start the bot process for a specific room bot_file = "bot-gemini.py" proc = subprocess.Popen([ f "python3 -m { bot_file } -u { room_url } -t { token } " ]) bot_procs[proc.pid] = (proc, room_url) ​ Connection Endpoints The server provides two ways to connect: Browser Access (/) Creates a room, starts a bot, and redirects the browser to the Daily meeting URL. Perfect for quick testing and development. RTVI Client (/connect) Creates a room, starts a bot, and returns connection credentials. Used by RTVI clients for custom implementations. ​ Bot Implementation (bot-gemini.py) The bot implementation connects all the pieces: Daily transport, Gemini service, conversation context, and processors. Let’s break down each component: ​ Transport Setup First, we configure the Daily transport, which handles WebRTC communication between the client and server. server/bot-gemini.py Copy Ask AI transport = DailyTransport( room_url, token, "Chatbot" , DailyParams( audio_in_enabled = True , # Enable audio input audio_out_enabled = True , # Enable audio output video_out_enabled = True , # Enable video output vad_analyzer = SileroVADAnalyzer( params = VADParams( stop_secs = 0.5 )), ), ) Gemini Multimodal Live audio requirements: Input: 16 kHz sample rate Output: 24 kHz sample rate ​ Gemini Service Configuration Next, we initialize the Gemini service which will provide speech-to-speech inference and communication: server/bot-gemini.py Copy Ask AI llm = GeminiMultimodalLiveLLMService( api_key = os.getenv( "GEMINI_API_KEY" ), voice_id = "Puck" , # Choose your bot's voice params = InputParams( temperature = 0.7 ) # Set model input params ) ​ Conversation Context We give our bot its personality and initial instructions: server/bot-gemini.py Copy Ask AI messages = [{ "role" : "user" , "content" : """You are Chatbot, a friendly, helpful robot. Keep responses brief and avoid special characters since output will be converted to audio.""" }] context = OpenAILLMContext(messages) context_aggregator = llm.create_context_aggregator(context) OpenAILLMContext is used as a common LLM base service for context management. In the future, we may add a specific context manager for Gemini. The context aggregator automatically maintains conversation history, helping the bot remember previous interactions. ​ Processor Setup We initialize two additional processors in our pipeline to handle different aspects of the interaction: RTVI Processors RTVIProcessor : Handles all client communication events including transcriptions, speaking states, and performance metrics Animation TalkingAnimation : Controls the bot’s visual state, switching between static and animated frames based on speaking status Learn more about the RTVI framework and available processors. ​ Pipeline Assembly Finally, we bring everything together in a pipeline: server/bot-gemini.py Copy Ask AI pipeline = Pipeline([ transport.input(), # Receive media rtvi, # Client UI events context_aggregator.user(), # Process user context llm, # Gemini processing ta, # Animation (talking/quiet states) transport.output(), # Send media context_aggregator.assistant() # Process bot context ]) task = PipelineTask( pipeline, params = PipelineParams( allow_interruptions = True , enable_metrics = True , enable_usage_metrics = True , ), observers = [RTVIObserver(rtvi)], ) The order of processors is crucial! For example, the RTVI processor should be early in the pipeline to capture all relevant events. The RTVIObserver monitors the entire pipeline and automatically collects relevant events to send to the client. ​ Client Implementation Our React client uses the Pipecat React SDK to communicate with the bot. Let’s explore how the client connects and interacts with our Pipecat server. ​ Connection Setup The client needs to connect to our bot server using the same transport type (Daily WebRTC) that we configured on the server: examples/react/src/providers/PipecatProvider.tsx Copy Ask AI const client = new PipecatClient ({ transport: new DailyTransport (), enableMic: true , // Enable audio input enableCam: false , // Disable video input enableScreenShare: false , // Disable screen sharing }); client . connect ({ endpoint: "http://localhost:7860/connect" , // Your bot connection endpoint }); The connection configuration must match your server: DailyTransport : Matches the WebRTC transport used in bot-gemini.py connect endpoint: Matches the /connect route in server.py Media settings: Controls which devices are enabled on join ​ Media Handling Pipecat’s React components handle all the complex media stream management for you: Copy Ask AI function App () { return ( < PipecatClientProvider client = { client } > < div className = "app" > < PipecatClientVideo participant = "bot" /> { /* Bot's video feed */ } < PipecatClientAudio /> { /* Audio input/output */ } </ div > </ PipecatClientProvider > ); } The PipecatClientProvider is the root component for providing Pipecat client context to your application. By wrapping your PipecatClientAudio and PipecatClientVideo components in this provider, they can access the client instance and receive and process the streams received from the Pipecat server. ​ Real-time Events The RTVI processors we configured in the pipeline emit events that we can handle in our client: Copy Ask AI // Listen for transcription events useRTVIClientEvent ( RTVIEvent . UserTranscript , ( data : TranscriptData ) => { if ( data . final ) { console . log ( `User said: ${ data . text } ` ); } }); // Listen for bot responses useRTVIClientEvent ( RTVIEvent . BotTranscript , ( data : BotLLMTextData ) => { console . log ( `Bot responded: ${ data . text } ` ); }); Available Events Speaking state changes Transcription updates Bot responses Connection status Performance metrics Event Usage Use these events to: Show speaking indicators Display transcripts Update UI state Monitor performance Optionally, uses callbacks to handle events in your application. Learn more in the Pipecat client docs. ​ Complete Example Here’s a basic client implementation with connection status and transcription display: Copy Ask AI function ChatApp () { return ( < PipecatClientProvider client = { client } > < div className = "app" > { /* Connection UI */ } < StatusDisplay /> < ConnectButton /> { /* Media Components */ } < BotVideo /> < PipecatClientAudio /> { /* Debug/Transcript Display */ } < DebugDisplay /> </ div > </ PipecatClientProvider > ); } Check out the example repository for a complete client implementation with styling and error handling. ​ Running the Application From the simple-chatbot directory, start the server and client to test the chatbot: ​ 1. Start the Server In one terminal: Copy Ask AI python server/server.py ​ 2. Start the Client In another terminal: Copy Ask AI cd examples/react npm install npm run dev ​ 3. Testing the Connection Open http://localhost:5173 in your browser Click “Connect” to join a room Allow microphone access when prompted Start talking with your AI assistant Troubleshooting: Check that all API keys are properly configured in .env Grant your browser access to your microphone, so it can receive your audio input Verify WebRTC ports aren’t blocked by firewalls ​ Next Steps Now that you have a working chatbot, consider these enhancements: Add custom avatar animations Implement function calling for external integrations Add support for multiple languages Enhance error recovery and reconnection logic ​ Examples Foundational Example A basic implementation demonstrating core Gemini Multimodal Live features and transcription capabilities Simple Chatbot A complete client/server implementation showing how to build a Pipecat JS or React client that connects to a Gemini Live Pipecat bot ​ Learn More Gemini Multimodal Live API Reference React Client SDK Documentation Recording Transcripts Metrics On this page What We’ll Build Key Concepts Understanding Pipelines Processors Gemini Integration Prerequisites Server Implementation Environment Setup Server Setup (server.py) Creating Meeting Room Managing Bot Instances Connection Endpoints Bot Implementation (bot-gemini.py) Transport Setup Gemini Service Configuration Conversation Context Processor Setup Pipeline Assembly Client Implementation Connection Setup Media Handling Real-time Events Complete Example Running the Application 1. Start the Server 2. Start the Client 3. Testing the Connection Next Steps Examples Learn More Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/features_gemini-multimodal-live_f55a8299.txt b/features_gemini-multimodal-live_f55a8299.txt
new file mode 100644
index 0000000000000000000000000000000000000000..9c63349578a66cbacff02d68bf674661497cdd5b
--- /dev/null
+++ b/features_gemini-multimodal-live_f55a8299.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/guides/features/gemini-multimodal-live#examples
+Title: Building with Gemini Multimodal Live - Pipecat
+==================================================
+
+Building with Gemini Multimodal Live - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Features Building with Gemini Multimodal Live Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Guides Fundamentals Context Management Custom FrameProcessor Detecting Idle Users Ending a Pipeline Function Calling Muting User Input Recording Audio Recording Transcripts Features Gemini Multimodal Live Metrics Noise cancellation with Krisp OpenAI Audio Models and APIs Pipecat Flows Telephony Overview Dial-in: WebRTC (Daily) Dial-in: WebRTC (Twilio + Daily) Dial-in: Twilio (Media Streams) Dialout: WebRTC (Daily) Deploying your bot Overview Deployment pattern Example: Pipecat Cloud Example: Fly.io Example: Cerebrium Example: Modal This guide will walk you through building a real-time AI chatbot using Gemini Multimodal Live and Pipecat. We’ll create a complete application with a Pipecat server and a Pipecat React client that enables natural conversations with an AI assistant. API Reference Gemini Multimodal Live API documentation Example Code Find the complete client and server code in Github Client SDK Pipecat React SDK documentation ​ What We’ll Build In this guide, you’ll create: A FastAPI server that manages bot instances A Gemini-powered conversational AI bot A React client with real-time audio/video A complete pipeline for speech-to-speech interaction ​ Key Concepts Before we dive into implementation, let’s cover some important concepts that will help you understand how Pipecat and Gemini work together. ​ Understanding Pipelines At the heart of Pipecat is the pipeline system. A pipeline is a sequence of processors that handle different aspects of the conversation flow. Think of it like an assembly line where each station (processor) performs a specific task. For our chatbot, the pipeline looks like this: Copy Ask AI pipeline = Pipeline([ transport.input(), # Receives audio/video from the user via WebRTC rtvi, # Handles client/server messaging and events context_aggregator.user(), # Manages user message history llm, # Processes speech through Gemini talking_animation, # Controls bot's avatar transport.output(), # Sends audio/video back to the user via WebRTC context_aggregator.assistant(), # Manages bot message history ]) ​ Processors Each processor in the pipeline handles a specific task: Transport transport.input() and transport.output() handle media streaming with Daily Context context_aggregator maintains conversation history for natural dialogue Speech Processing rtvi_user_transcription and rtvi_bot_transcription handle speech-to-text Animation talking_animation controls the bot’s visual state based on speaking activity The order of processors matters! Data flows through the pipeline in sequence, so each processor should receive the data it needs from previous processors. Learn more about the Core Concepts to Pipecat server. ​ Gemini Integration The GeminiMultimodalLiveLLMService is a speech-to-speech LLM service that interfaces with the Gemini Multimodal Live API. It provides: Real-time speech-to-speech conversation Context management Voice activity detection Tool use Pipecat manages two types of connections: A WebRTC connection between the Pipecat client and server for reliable audio/video streaming A WebSocket connection between the Pipecat server and Gemini for real-time AI processing This architecture ensures stable media streaming while maintaining responsive AI interactions. ​ Prerequisites Before we begin, you’ll need: Python 3.10 or higher Node.js 16 or higher A Daily API key A Google API key with Gemini Multimodal Live access Clone the Pipecat repo: Copy Ask AI git clone git@github.com:pipecat-ai/pipecat.git ​ Server Implementation Let’s start by setting up the server components. Our server will handle bot management, room creation, and client connections. ​ Environment Setup Navigate to the simple-chatbot’s server directory: Copy Ask AI cd examples/simple-chatbot/server Set up a python virtual environment: Copy Ask AI python -m venv venv source venv/bin/activate # On Windows: venv\Scripts\activate Install requirements: Copy Ask AI pip install -r requirements.txt Copy env.example to .env and make a few changes: Copy Ask AI # Remove the hard-coded example room URL DAILY_SAMPLE_ROOM_URL = # Add your Daily and Gemini API keys DAILY_API_KEY = [your key here] GEMINI_API_KEY = [your key here] # Use Gemini implementation BOT_IMPLEMENTATION = gemini ​ Server Setup (server.py) server.py is a FastAPI server that creates the meeting room where clients and bots interact, manages bot instances, and handles client connections. It’s the orchestrator that brings everything on the server-side together. ​ Creating Meeting Room The server uses Daily’s API via a REST API helper to create rooms where clients and bots can meet. Each room is a secure space for audio/video communication: server/server.py Copy Ask AI async def create_room_and_token (): """Create a Daily room and generate access credentials.""" room = await daily_helpers[ "rest" ].create_room(DailyRoomParams()) token = await daily_helpers[ "rest" ].get_token(room.url) return room.url, token ​ Managing Bot Instances When a client connects, the server starts a new bot instance configured specifically for that room. It keeps track of running bots and ensures there’s only one bot per room: server/server.py Copy Ask AI # Start the bot process for a specific room bot_file = "bot-gemini.py" proc = subprocess.Popen([ f "python3 -m { bot_file } -u { room_url } -t { token } " ]) bot_procs[proc.pid] = (proc, room_url) ​ Connection Endpoints The server provides two ways to connect: Browser Access (/) Creates a room, starts a bot, and redirects the browser to the Daily meeting URL. Perfect for quick testing and development. RTVI Client (/connect) Creates a room, starts a bot, and returns connection credentials. Used by RTVI clients for custom implementations. ​ Bot Implementation (bot-gemini.py) The bot implementation connects all the pieces: Daily transport, Gemini service, conversation context, and processors. Let’s break down each component: ​ Transport Setup First, we configure the Daily transport, which handles WebRTC communication between the client and server. server/bot-gemini.py Copy Ask AI transport = DailyTransport( room_url, token, "Chatbot" , DailyParams( audio_in_enabled = True , # Enable audio input audio_out_enabled = True , # Enable audio output video_out_enabled = True , # Enable video output vad_analyzer = SileroVADAnalyzer( params = VADParams( stop_secs = 0.5 )), ), ) Gemini Multimodal Live audio requirements: Input: 16 kHz sample rate Output: 24 kHz sample rate ​ Gemini Service Configuration Next, we initialize the Gemini service which will provide speech-to-speech inference and communication: server/bot-gemini.py Copy Ask AI llm = GeminiMultimodalLiveLLMService( api_key = os.getenv( "GEMINI_API_KEY" ), voice_id = "Puck" , # Choose your bot's voice params = InputParams( temperature = 0.7 ) # Set model input params ) ​ Conversation Context We give our bot its personality and initial instructions: server/bot-gemini.py Copy Ask AI messages = [{ "role" : "user" , "content" : """You are Chatbot, a friendly, helpful robot. Keep responses brief and avoid special characters since output will be converted to audio.""" }] context = OpenAILLMContext(messages) context_aggregator = llm.create_context_aggregator(context) OpenAILLMContext is used as a common LLM base service for context management. In the future, we may add a specific context manager for Gemini. The context aggregator automatically maintains conversation history, helping the bot remember previous interactions. ​ Processor Setup We initialize two additional processors in our pipeline to handle different aspects of the interaction: RTVI Processors RTVIProcessor : Handles all client communication events including transcriptions, speaking states, and performance metrics Animation TalkingAnimation : Controls the bot’s visual state, switching between static and animated frames based on speaking status Learn more about the RTVI framework and available processors. ​ Pipeline Assembly Finally, we bring everything together in a pipeline: server/bot-gemini.py Copy Ask AI pipeline = Pipeline([ transport.input(), # Receive media rtvi, # Client UI events context_aggregator.user(), # Process user context llm, # Gemini processing ta, # Animation (talking/quiet states) transport.output(), # Send media context_aggregator.assistant() # Process bot context ]) task = PipelineTask( pipeline, params = PipelineParams( allow_interruptions = True , enable_metrics = True , enable_usage_metrics = True , ), observers = [RTVIObserver(rtvi)], ) The order of processors is crucial! For example, the RTVI processor should be early in the pipeline to capture all relevant events. The RTVIObserver monitors the entire pipeline and automatically collects relevant events to send to the client. ​ Client Implementation Our React client uses the Pipecat React SDK to communicate with the bot. Let’s explore how the client connects and interacts with our Pipecat server. ​ Connection Setup The client needs to connect to our bot server using the same transport type (Daily WebRTC) that we configured on the server: examples/react/src/providers/PipecatProvider.tsx Copy Ask AI const client = new PipecatClient ({ transport: new DailyTransport (), enableMic: true , // Enable audio input enableCam: false , // Disable video input enableScreenShare: false , // Disable screen sharing }); client . connect ({ endpoint: "http://localhost:7860/connect" , // Your bot connection endpoint }); The connection configuration must match your server: DailyTransport : Matches the WebRTC transport used in bot-gemini.py connect endpoint: Matches the /connect route in server.py Media settings: Controls which devices are enabled on join ​ Media Handling Pipecat’s React components handle all the complex media stream management for you: Copy Ask AI function App () { return ( < PipecatClientProvider client = { client } > < div className = "app" > < PipecatClientVideo participant = "bot" /> { /* Bot's video feed */ } < PipecatClientAudio /> { /* Audio input/output */ } </ div > </ PipecatClientProvider > ); } The PipecatClientProvider is the root component for providing Pipecat client context to your application. By wrapping your PipecatClientAudio and PipecatClientVideo components in this provider, they can access the client instance and receive and process the streams received from the Pipecat server. ​ Real-time Events The RTVI processors we configured in the pipeline emit events that we can handle in our client: Copy Ask AI // Listen for transcription events useRTVIClientEvent ( RTVIEvent . UserTranscript , ( data : TranscriptData ) => { if ( data . final ) { console . log ( `User said: ${ data . text } ` ); } }); // Listen for bot responses useRTVIClientEvent ( RTVIEvent . BotTranscript , ( data : BotLLMTextData ) => { console . log ( `Bot responded: ${ data . text } ` ); }); Available Events Speaking state changes Transcription updates Bot responses Connection status Performance metrics Event Usage Use these events to: Show speaking indicators Display transcripts Update UI state Monitor performance Optionally, uses callbacks to handle events in your application. Learn more in the Pipecat client docs. ​ Complete Example Here’s a basic client implementation with connection status and transcription display: Copy Ask AI function ChatApp () { return ( < PipecatClientProvider client = { client } > < div className = "app" > { /* Connection UI */ } < StatusDisplay /> < ConnectButton /> { /* Media Components */ } < BotVideo /> < PipecatClientAudio /> { /* Debug/Transcript Display */ } < DebugDisplay /> </ div > </ PipecatClientProvider > ); } Check out the example repository for a complete client implementation with styling and error handling. ​ Running the Application From the simple-chatbot directory, start the server and client to test the chatbot: ​ 1. Start the Server In one terminal: Copy Ask AI python server/server.py ​ 2. Start the Client In another terminal: Copy Ask AI cd examples/react npm install npm run dev ​ 3. Testing the Connection Open http://localhost:5173 in your browser Click “Connect” to join a room Allow microphone access when prompted Start talking with your AI assistant Troubleshooting: Check that all API keys are properly configured in .env Grant your browser access to your microphone, so it can receive your audio input Verify WebRTC ports aren’t blocked by firewalls ​ Next Steps Now that you have a working chatbot, consider these enhancements: Add custom avatar animations Implement function calling for external integrations Add support for multiple languages Enhance error recovery and reconnection logic ​ Examples Foundational Example A basic implementation demonstrating core Gemini Multimodal Live features and transcription capabilities Simple Chatbot A complete client/server implementation showing how to build a Pipecat JS or React client that connects to a Gemini Live Pipecat bot ​ Learn More Gemini Multimodal Live API Reference React Client SDK Documentation Recording Transcripts Metrics On this page What We’ll Build Key Concepts Understanding Pipelines Processors Gemini Integration Prerequisites Server Implementation Environment Setup Server Setup (server.py) Creating Meeting Room Managing Bot Instances Connection Endpoints Bot Implementation (bot-gemini.py) Transport Setup Gemini Service Configuration Conversation Context Processor Setup Pipeline Assembly Client Implementation Connection Setup Media Handling Real-time Events Complete Example Running the Application 1. Start the Server 2. Start the Client 3. Testing the Connection Next Steps Examples Learn More Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/features_krisp_7f02e06e.txt b/features_krisp_7f02e06e.txt
new file mode 100644
index 0000000000000000000000000000000000000000..8877b54291fa528184b6694043574364dfd011c9
--- /dev/null
+++ b/features_krisp_7f02e06e.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/guides/features/krisp#get-access-to-krisp-sdk-and-models
+Title: Noise cancellation with Krisp - Pipecat
+==================================================
+
+Noise cancellation with Krisp - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Features Noise cancellation with Krisp Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Guides Fundamentals Context Management Custom FrameProcessor Detecting Idle Users Ending a Pipeline Function Calling Muting User Input Recording Audio Recording Transcripts Features Gemini Multimodal Live Metrics Noise cancellation with Krisp OpenAI Audio Models and APIs Pipecat Flows Telephony Overview Dial-in: WebRTC (Daily) Dial-in: WebRTC (Twilio + Daily) Dial-in: Twilio (Media Streams) Dialout: WebRTC (Daily) Deploying your bot Overview Deployment pattern Example: Pipecat Cloud Example: Fly.io Example: Cerebrium Example: Modal ​ Overview This guide will walk you through setting up and using Krisp’s noise reduction capabilities in your Pipecat application. Krisp provides professional-grade noise cancellation that can significantly improve audio quality in real-time communications. To use Krisp’s noise cancellation, you’ll need to obtain their SDK and models through a Krisp developer account. Our Pipecat Krisp module simplifies the integration process, which we’ll cover in this guide. ​ Walkthrough ​ Get Access to Krisp SDK and Models Create a Krisp developers account at krisp.ai/developers Download the Krisp Desktop SDK (v7) that matches your platform: Linux: Desktop SDK v7.0.2: Linux macOS (ARM): Desktop SDK v7.0.1: Mac ARM macOS (Intel): Desktop SDK v7.0.1: Mac Intel Windows(ARM): Desktop SDK v7.0.2: Windows ARM Windows (x64): Desktop SDK v7.0.2: Windows Download the corresponding models. We recommend trying the Background Voice Cancellation model. Recommended model for each platform: Linux (ARM): hs.c6.f.s.de56df.kw Mac (ARM): hs.c6.f.s.de56df.kw Linux (x86_64): hs.c6.f.s.de56df.kw Mac (x86_64): hs.c6.f.s.de56df.kw Windows (x86_64): hs.c6.f.s.de56df.kw ​ Install build dependencies The pipecat-ai-krisp module is a python wrapper around Krisp’s C++ SDK. To build the module, you’ll need to install the following dependencies: macOS Ubuntu/Debian Windows Copy Ask AI # Using Homebrew brew install cmake pybind11 Copy Ask AI # Using Homebrew brew install cmake pybind11 Copy Ask AI # Using apt sudo apt-get update sudo apt-get install cmake python3-dev pybind11-dev g++ Copy Ask AI # Using Chocolatey choco install cmake # pybind11 will be installed via pip during the build process # OR using Visual Studio # 1. Install Visual Studio with C++ development tools # 2. Install CMake from https://cmake.org/download/ # pybind11 will be installed via pip during the build process For Windows users: Make sure you have Visual Studio installed with C++ development tools, or alternatively, have the Visual C++ Build Tools installed. ​ Install the pipecat-ai-krisp module In your Pipecat repo, activate your virtual environment: macOS/Linux Windows Copy Ask AI python3 -m venv venv source venv/bin/activate Copy Ask AI python3 -m venv venv source venv/bin/activate Copy Ask AI # Using PowerShell python - m venv venv .\venv\Scripts\Activate.ps1 # OR using Command Prompt python - m venv venv .\venv\Scripts\ activate.bat # OR using Git Bash python - m venv venv source venv / Scripts / activate Export the path to your Krisp SDK and model files: macOS/Linux Windows Copy Ask AI # Add to your .env file or shell configuration export KRISP_SDK_PATH = / PATH / TO / KRISP / SDK export KRISP_MODEL_PATH = / PATH / TO / KRISP / MODEL . kef Copy Ask AI # Add to your .env file or shell configuration export KRISP_SDK_PATH = / PATH / TO / KRISP / SDK export KRISP_MODEL_PATH = / PATH / TO / KRISP / MODEL . kef Copy Ask AI # Using PowerShell $ env: KRISP_SDK_PATH = "C:\PATH\TO\KRISP\SDK" $ env: KRISP_MODEL_PATH = "C:\PATH\TO\KRISP\MODEL.kef" # OR using Command Prompt set KRISP_SDK_PATH = C:\PATH\TO\KRISP\SDK set KRISP_MODEL_PATH = C:\PATH\TO\KRISP\MODEL.kef When selecting a KRISP_MODEL_PATH , ensure that you’re selecting the actual model file, not just the directory. The path should look something like this: ARM Linux Mac ARM Windows Copy Ask AI export KRISP_MODEL_PATH = ./ krisp / hs . c6 . f . s . de56df . kw Copy Ask AI export KRISP_MODEL_PATH = ./ krisp / hs . c6 . f . s . de56df . kw export Copy Ask AI KRISP_MODEL_PATH = ./krisp/outbound-bvc-models-fp16/hs.c6.f.s.de56df.bucharest.kef Copy Ask AI $ env: KRISP_MODEL_PATH = "C:\krisp\outbound-bvc-models-fp32\hs.c6.f.s.de56df.bucharest.kef" Next, install the pipecat-ai[krisp] module, which will automatically build the pipecat-ai-krisp python wrapper module: Copy Ask AI pip install "pipecat-ai[krisp]" ​ Test the integration You can now test the Krisp integration. The easiest way to do this is to run the foundational example: 07p-interruptible-krisp.py . You can run a foundational example by running the following command: Copy Ask AI python examples/foundational/07p-interruptible-krisp.py -u YOUR_DAILY_ROOM_URL ​ Important for macOS users If you’re running on macOS you may receive a security warning about running the script. This is expected. You can allow access by going to System Settings > Privacy & Security then click Allow Anyway to permit the example to run. After allowing and re-running, you may get a pop-up asking for permission. Select Open Anyway to run the script. ​ Usage Example Here’s a basic example of how to add Krisp noise reduction to your Pipecat pipeline: Copy Ask AI from pipecat.audio.filters.krisp_filter import KrispFilter from pipecat.transports.services.daily import DailyParams, DailyTransport # Add to transport configuration transport = DailyTransport( room_url, token, "Audio Bot" , DailyParams( audio_in_filter = KrispFilter(), # Enable Krisp noise reduction audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer(), ), ) ​ Troubleshooting Common issues and solutions: Missing Dependencies Copy Ask AI Error: Missing module: pipecat_ai_krisp Solution: Ensure you’ve installed with the krisp extra: pip install "pipecat-ai[krisp]" Model Path Not Found Copy Ask AI Error: Model path for KrispAudioProcessor must be provided Solution: Set the KRISP_MODEL_PATH environment variable or provide it in the constructor SDK Path Issues Copy Ask AI Error: Cannot find Krisp SDK Solution: Verify KRISP_SDK_PATH points to a valid Krisp SDK installation ​ Additional Resources KrispFilter Reference Documentation Example Application Metrics OpenAI Audio Models and APIs On this page Overview Walkthrough Get Access to Krisp SDK and Models Install build dependencies Install the pipecat-ai-krisp module Test the integration Important for macOS users Usage Example Troubleshooting Additional Resources Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/features_krisp_cbdaba06.txt b/features_krisp_cbdaba06.txt
new file mode 100644
index 0000000000000000000000000000000000000000..8120e0fcb1a9c36922d784b7d56b37113eb50cd4
--- /dev/null
+++ b/features_krisp_cbdaba06.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/guides/features/krisp#important-for-macos-users
+Title: Noise cancellation with Krisp - Pipecat
+==================================================
+
+Noise cancellation with Krisp - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Features Noise cancellation with Krisp Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Guides Fundamentals Context Management Custom FrameProcessor Detecting Idle Users Ending a Pipeline Function Calling Muting User Input Recording Audio Recording Transcripts Features Gemini Multimodal Live Metrics Noise cancellation with Krisp OpenAI Audio Models and APIs Pipecat Flows Telephony Overview Dial-in: WebRTC (Daily) Dial-in: WebRTC (Twilio + Daily) Dial-in: Twilio (Media Streams) Dialout: WebRTC (Daily) Deploying your bot Overview Deployment pattern Example: Pipecat Cloud Example: Fly.io Example: Cerebrium Example: Modal ​ Overview This guide will walk you through setting up and using Krisp’s noise reduction capabilities in your Pipecat application. Krisp provides professional-grade noise cancellation that can significantly improve audio quality in real-time communications. To use Krisp’s noise cancellation, you’ll need to obtain their SDK and models through a Krisp developer account. Our Pipecat Krisp module simplifies the integration process, which we’ll cover in this guide. ​ Walkthrough ​ Get Access to Krisp SDK and Models Create a Krisp developers account at krisp.ai/developers Download the Krisp Desktop SDK (v7) that matches your platform: Linux: Desktop SDK v7.0.2: Linux macOS (ARM): Desktop SDK v7.0.1: Mac ARM macOS (Intel): Desktop SDK v7.0.1: Mac Intel Windows(ARM): Desktop SDK v7.0.2: Windows ARM Windows (x64): Desktop SDK v7.0.2: Windows Download the corresponding models. We recommend trying the Background Voice Cancellation model. Recommended model for each platform: Linux (ARM): hs.c6.f.s.de56df.kw Mac (ARM): hs.c6.f.s.de56df.kw Linux (x86_64): hs.c6.f.s.de56df.kw Mac (x86_64): hs.c6.f.s.de56df.kw Windows (x86_64): hs.c6.f.s.de56df.kw ​ Install build dependencies The pipecat-ai-krisp module is a python wrapper around Krisp’s C++ SDK. To build the module, you’ll need to install the following dependencies: macOS Ubuntu/Debian Windows Copy Ask AI # Using Homebrew brew install cmake pybind11 Copy Ask AI # Using Homebrew brew install cmake pybind11 Copy Ask AI # Using apt sudo apt-get update sudo apt-get install cmake python3-dev pybind11-dev g++ Copy Ask AI # Using Chocolatey choco install cmake # pybind11 will be installed via pip during the build process # OR using Visual Studio # 1. Install Visual Studio with C++ development tools # 2. Install CMake from https://cmake.org/download/ # pybind11 will be installed via pip during the build process For Windows users: Make sure you have Visual Studio installed with C++ development tools, or alternatively, have the Visual C++ Build Tools installed. ​ Install the pipecat-ai-krisp module In your Pipecat repo, activate your virtual environment: macOS/Linux Windows Copy Ask AI python3 -m venv venv source venv/bin/activate Copy Ask AI python3 -m venv venv source venv/bin/activate Copy Ask AI # Using PowerShell python - m venv venv .\venv\Scripts\Activate.ps1 # OR using Command Prompt python - m venv venv .\venv\Scripts\ activate.bat # OR using Git Bash python - m venv venv source venv / Scripts / activate Export the path to your Krisp SDK and model files: macOS/Linux Windows Copy Ask AI # Add to your .env file or shell configuration export KRISP_SDK_PATH = / PATH / TO / KRISP / SDK export KRISP_MODEL_PATH = / PATH / TO / KRISP / MODEL . kef Copy Ask AI # Add to your .env file or shell configuration export KRISP_SDK_PATH = / PATH / TO / KRISP / SDK export KRISP_MODEL_PATH = / PATH / TO / KRISP / MODEL . kef Copy Ask AI # Using PowerShell $ env: KRISP_SDK_PATH = "C:\PATH\TO\KRISP\SDK" $ env: KRISP_MODEL_PATH = "C:\PATH\TO\KRISP\MODEL.kef" # OR using Command Prompt set KRISP_SDK_PATH = C:\PATH\TO\KRISP\SDK set KRISP_MODEL_PATH = C:\PATH\TO\KRISP\MODEL.kef When selecting a KRISP_MODEL_PATH , ensure that you’re selecting the actual model file, not just the directory. The path should look something like this: ARM Linux Mac ARM Windows Copy Ask AI export KRISP_MODEL_PATH = ./ krisp / hs . c6 . f . s . de56df . kw Copy Ask AI export KRISP_MODEL_PATH = ./ krisp / hs . c6 . f . s . de56df . kw export Copy Ask AI KRISP_MODEL_PATH = ./krisp/outbound-bvc-models-fp16/hs.c6.f.s.de56df.bucharest.kef Copy Ask AI $ env: KRISP_MODEL_PATH = "C:\krisp\outbound-bvc-models-fp32\hs.c6.f.s.de56df.bucharest.kef" Next, install the pipecat-ai[krisp] module, which will automatically build the pipecat-ai-krisp python wrapper module: Copy Ask AI pip install "pipecat-ai[krisp]" ​ Test the integration You can now test the Krisp integration. The easiest way to do this is to run the foundational example: 07p-interruptible-krisp.py . You can run a foundational example by running the following command: Copy Ask AI python examples/foundational/07p-interruptible-krisp.py -u YOUR_DAILY_ROOM_URL ​ Important for macOS users If you’re running on macOS you may receive a security warning about running the script. This is expected. You can allow access by going to System Settings > Privacy & Security then click Allow Anyway to permit the example to run. After allowing and re-running, you may get a pop-up asking for permission. Select Open Anyway to run the script. ​ Usage Example Here’s a basic example of how to add Krisp noise reduction to your Pipecat pipeline: Copy Ask AI from pipecat.audio.filters.krisp_filter import KrispFilter from pipecat.transports.services.daily import DailyParams, DailyTransport # Add to transport configuration transport = DailyTransport( room_url, token, "Audio Bot" , DailyParams( audio_in_filter = KrispFilter(), # Enable Krisp noise reduction audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer(), ), ) ​ Troubleshooting Common issues and solutions: Missing Dependencies Copy Ask AI Error: Missing module: pipecat_ai_krisp Solution: Ensure you’ve installed with the krisp extra: pip install "pipecat-ai[krisp]" Model Path Not Found Copy Ask AI Error: Model path for KrispAudioProcessor must be provided Solution: Set the KRISP_MODEL_PATH environment variable or provide it in the constructor SDK Path Issues Copy Ask AI Error: Cannot find Krisp SDK Solution: Verify KRISP_SDK_PATH points to a valid Krisp SDK installation ​ Additional Resources KrispFilter Reference Documentation Example Application Metrics OpenAI Audio Models and APIs On this page Overview Walkthrough Get Access to Krisp SDK and Models Install build dependencies Install the pipecat-ai-krisp module Test the integration Important for macOS users Usage Example Troubleshooting Additional Resources Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/features_krisp_e3244d5d.txt b/features_krisp_e3244d5d.txt
new file mode 100644
index 0000000000000000000000000000000000000000..7cc817b71c5fff78a14173646285148f5534b920
--- /dev/null
+++ b/features_krisp_e3244d5d.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/guides/features/krisp#install-the-pipecat-ai-krisp-module
+Title: Noise cancellation with Krisp - Pipecat
+==================================================
+
+Noise cancellation with Krisp - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Features Noise cancellation with Krisp Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Guides Fundamentals Context Management Custom FrameProcessor Detecting Idle Users Ending a Pipeline Function Calling Muting User Input Recording Audio Recording Transcripts Features Gemini Multimodal Live Metrics Noise cancellation with Krisp OpenAI Audio Models and APIs Pipecat Flows Telephony Overview Dial-in: WebRTC (Daily) Dial-in: WebRTC (Twilio + Daily) Dial-in: Twilio (Media Streams) Dialout: WebRTC (Daily) Deploying your bot Overview Deployment pattern Example: Pipecat Cloud Example: Fly.io Example: Cerebrium Example: Modal ​ Overview This guide will walk you through setting up and using Krisp’s noise reduction capabilities in your Pipecat application. Krisp provides professional-grade noise cancellation that can significantly improve audio quality in real-time communications. To use Krisp’s noise cancellation, you’ll need to obtain their SDK and models through a Krisp developer account. Our Pipecat Krisp module simplifies the integration process, which we’ll cover in this guide. ​ Walkthrough ​ Get Access to Krisp SDK and Models Create a Krisp developers account at krisp.ai/developers Download the Krisp Desktop SDK (v7) that matches your platform: Linux: Desktop SDK v7.0.2: Linux macOS (ARM): Desktop SDK v7.0.1: Mac ARM macOS (Intel): Desktop SDK v7.0.1: Mac Intel Windows(ARM): Desktop SDK v7.0.2: Windows ARM Windows (x64): Desktop SDK v7.0.2: Windows Download the corresponding models. We recommend trying the Background Voice Cancellation model. Recommended model for each platform: Linux (ARM): hs.c6.f.s.de56df.kw Mac (ARM): hs.c6.f.s.de56df.kw Linux (x86_64): hs.c6.f.s.de56df.kw Mac (x86_64): hs.c6.f.s.de56df.kw Windows (x86_64): hs.c6.f.s.de56df.kw ​ Install build dependencies The pipecat-ai-krisp module is a python wrapper around Krisp’s C++ SDK. To build the module, you’ll need to install the following dependencies: macOS Ubuntu/Debian Windows Copy Ask AI # Using Homebrew brew install cmake pybind11 Copy Ask AI # Using Homebrew brew install cmake pybind11 Copy Ask AI # Using apt sudo apt-get update sudo apt-get install cmake python3-dev pybind11-dev g++ Copy Ask AI # Using Chocolatey choco install cmake # pybind11 will be installed via pip during the build process # OR using Visual Studio # 1. Install Visual Studio with C++ development tools # 2. Install CMake from https://cmake.org/download/ # pybind11 will be installed via pip during the build process For Windows users: Make sure you have Visual Studio installed with C++ development tools, or alternatively, have the Visual C++ Build Tools installed. ​ Install the pipecat-ai-krisp module In your Pipecat repo, activate your virtual environment: macOS/Linux Windows Copy Ask AI python3 -m venv venv source venv/bin/activate Copy Ask AI python3 -m venv venv source venv/bin/activate Copy Ask AI # Using PowerShell python - m venv venv .\venv\Scripts\Activate.ps1 # OR using Command Prompt python - m venv venv .\venv\Scripts\ activate.bat # OR using Git Bash python - m venv venv source venv / Scripts / activate Export the path to your Krisp SDK and model files: macOS/Linux Windows Copy Ask AI # Add to your .env file or shell configuration export KRISP_SDK_PATH = / PATH / TO / KRISP / SDK export KRISP_MODEL_PATH = / PATH / TO / KRISP / MODEL . kef Copy Ask AI # Add to your .env file or shell configuration export KRISP_SDK_PATH = / PATH / TO / KRISP / SDK export KRISP_MODEL_PATH = / PATH / TO / KRISP / MODEL . kef Copy Ask AI # Using PowerShell $ env: KRISP_SDK_PATH = "C:\PATH\TO\KRISP\SDK" $ env: KRISP_MODEL_PATH = "C:\PATH\TO\KRISP\MODEL.kef" # OR using Command Prompt set KRISP_SDK_PATH = C:\PATH\TO\KRISP\SDK set KRISP_MODEL_PATH = C:\PATH\TO\KRISP\MODEL.kef When selecting a KRISP_MODEL_PATH , ensure that you’re selecting the actual model file, not just the directory. The path should look something like this: ARM Linux Mac ARM Windows Copy Ask AI export KRISP_MODEL_PATH = ./ krisp / hs . c6 . f . s . de56df . kw Copy Ask AI export KRISP_MODEL_PATH = ./ krisp / hs . c6 . f . s . de56df . kw export Copy Ask AI KRISP_MODEL_PATH = ./krisp/outbound-bvc-models-fp16/hs.c6.f.s.de56df.bucharest.kef Copy Ask AI $ env: KRISP_MODEL_PATH = "C:\krisp\outbound-bvc-models-fp32\hs.c6.f.s.de56df.bucharest.kef" Next, install the pipecat-ai[krisp] module, which will automatically build the pipecat-ai-krisp python wrapper module: Copy Ask AI pip install "pipecat-ai[krisp]" ​ Test the integration You can now test the Krisp integration. The easiest way to do this is to run the foundational example: 07p-interruptible-krisp.py . You can run a foundational example by running the following command: Copy Ask AI python examples/foundational/07p-interruptible-krisp.py -u YOUR_DAILY_ROOM_URL ​ Important for macOS users If you’re running on macOS you may receive a security warning about running the script. This is expected. You can allow access by going to System Settings > Privacy & Security then click Allow Anyway to permit the example to run. After allowing and re-running, you may get a pop-up asking for permission. Select Open Anyway to run the script. ​ Usage Example Here’s a basic example of how to add Krisp noise reduction to your Pipecat pipeline: Copy Ask AI from pipecat.audio.filters.krisp_filter import KrispFilter from pipecat.transports.services.daily import DailyParams, DailyTransport # Add to transport configuration transport = DailyTransport( room_url, token, "Audio Bot" , DailyParams( audio_in_filter = KrispFilter(), # Enable Krisp noise reduction audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer(), ), ) ​ Troubleshooting Common issues and solutions: Missing Dependencies Copy Ask AI Error: Missing module: pipecat_ai_krisp Solution: Ensure you’ve installed with the krisp extra: pip install "pipecat-ai[krisp]" Model Path Not Found Copy Ask AI Error: Model path for KrispAudioProcessor must be provided Solution: Set the KRISP_MODEL_PATH environment variable or provide it in the constructor SDK Path Issues Copy Ask AI Error: Cannot find Krisp SDK Solution: Verify KRISP_SDK_PATH points to a valid Krisp SDK installation ​ Additional Resources KrispFilter Reference Documentation Example Application Metrics OpenAI Audio Models and APIs On this page Overview Walkthrough Get Access to Krisp SDK and Models Install build dependencies Install the pipecat-ai-krisp module Test the integration Important for macOS users Usage Example Troubleshooting Additional Resources Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/features_krisp_fa663f68.txt b/features_krisp_fa663f68.txt
new file mode 100644
index 0000000000000000000000000000000000000000..3b2d21abf288844d54d9ead683c6e189d9e178f4
--- /dev/null
+++ b/features_krisp_fa663f68.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/guides/features/krisp#walkthrough
+Title: Noise cancellation with Krisp - Pipecat
+==================================================
+
+Noise cancellation with Krisp - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Features Noise cancellation with Krisp Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Guides Fundamentals Context Management Custom FrameProcessor Detecting Idle Users Ending a Pipeline Function Calling Muting User Input Recording Audio Recording Transcripts Features Gemini Multimodal Live Metrics Noise cancellation with Krisp OpenAI Audio Models and APIs Pipecat Flows Telephony Overview Dial-in: WebRTC (Daily) Dial-in: WebRTC (Twilio + Daily) Dial-in: Twilio (Media Streams) Dialout: WebRTC (Daily) Deploying your bot Overview Deployment pattern Example: Pipecat Cloud Example: Fly.io Example: Cerebrium Example: Modal ​ Overview This guide will walk you through setting up and using Krisp’s noise reduction capabilities in your Pipecat application. Krisp provides professional-grade noise cancellation that can significantly improve audio quality in real-time communications. To use Krisp’s noise cancellation, you’ll need to obtain their SDK and models through a Krisp developer account. Our Pipecat Krisp module simplifies the integration process, which we’ll cover in this guide. ​ Walkthrough ​ Get Access to Krisp SDK and Models Create a Krisp developers account at krisp.ai/developers Download the Krisp Desktop SDK (v7) that matches your platform: Linux: Desktop SDK v7.0.2: Linux macOS (ARM): Desktop SDK v7.0.1: Mac ARM macOS (Intel): Desktop SDK v7.0.1: Mac Intel Windows(ARM): Desktop SDK v7.0.2: Windows ARM Windows (x64): Desktop SDK v7.0.2: Windows Download the corresponding models. We recommend trying the Background Voice Cancellation model. Recommended model for each platform: Linux (ARM): hs.c6.f.s.de56df.kw Mac (ARM): hs.c6.f.s.de56df.kw Linux (x86_64): hs.c6.f.s.de56df.kw Mac (x86_64): hs.c6.f.s.de56df.kw Windows (x86_64): hs.c6.f.s.de56df.kw ​ Install build dependencies The pipecat-ai-krisp module is a python wrapper around Krisp’s C++ SDK. To build the module, you’ll need to install the following dependencies: macOS Ubuntu/Debian Windows Copy Ask AI # Using Homebrew brew install cmake pybind11 Copy Ask AI # Using Homebrew brew install cmake pybind11 Copy Ask AI # Using apt sudo apt-get update sudo apt-get install cmake python3-dev pybind11-dev g++ Copy Ask AI # Using Chocolatey choco install cmake # pybind11 will be installed via pip during the build process # OR using Visual Studio # 1. Install Visual Studio with C++ development tools # 2. Install CMake from https://cmake.org/download/ # pybind11 will be installed via pip during the build process For Windows users: Make sure you have Visual Studio installed with C++ development tools, or alternatively, have the Visual C++ Build Tools installed. ​ Install the pipecat-ai-krisp module In your Pipecat repo, activate your virtual environment: macOS/Linux Windows Copy Ask AI python3 -m venv venv source venv/bin/activate Copy Ask AI python3 -m venv venv source venv/bin/activate Copy Ask AI # Using PowerShell python - m venv venv .\venv\Scripts\Activate.ps1 # OR using Command Prompt python - m venv venv .\venv\Scripts\ activate.bat # OR using Git Bash python - m venv venv source venv / Scripts / activate Export the path to your Krisp SDK and model files: macOS/Linux Windows Copy Ask AI # Add to your .env file or shell configuration export KRISP_SDK_PATH = / PATH / TO / KRISP / SDK export KRISP_MODEL_PATH = / PATH / TO / KRISP / MODEL . kef Copy Ask AI # Add to your .env file or shell configuration export KRISP_SDK_PATH = / PATH / TO / KRISP / SDK export KRISP_MODEL_PATH = / PATH / TO / KRISP / MODEL . kef Copy Ask AI # Using PowerShell $ env: KRISP_SDK_PATH = "C:\PATH\TO\KRISP\SDK" $ env: KRISP_MODEL_PATH = "C:\PATH\TO\KRISP\MODEL.kef" # OR using Command Prompt set KRISP_SDK_PATH = C:\PATH\TO\KRISP\SDK set KRISP_MODEL_PATH = C:\PATH\TO\KRISP\MODEL.kef When selecting a KRISP_MODEL_PATH , ensure that you’re selecting the actual model file, not just the directory. The path should look something like this: ARM Linux Mac ARM Windows Copy Ask AI export KRISP_MODEL_PATH = ./ krisp / hs . c6 . f . s . de56df . kw Copy Ask AI export KRISP_MODEL_PATH = ./ krisp / hs . c6 . f . s . de56df . kw export Copy Ask AI KRISP_MODEL_PATH = ./krisp/outbound-bvc-models-fp16/hs.c6.f.s.de56df.bucharest.kef Copy Ask AI $ env: KRISP_MODEL_PATH = "C:\krisp\outbound-bvc-models-fp32\hs.c6.f.s.de56df.bucharest.kef" Next, install the pipecat-ai[krisp] module, which will automatically build the pipecat-ai-krisp python wrapper module: Copy Ask AI pip install "pipecat-ai[krisp]" ​ Test the integration You can now test the Krisp integration. The easiest way to do this is to run the foundational example: 07p-interruptible-krisp.py . You can run a foundational example by running the following command: Copy Ask AI python examples/foundational/07p-interruptible-krisp.py -u YOUR_DAILY_ROOM_URL ​ Important for macOS users If you’re running on macOS you may receive a security warning about running the script. This is expected. You can allow access by going to System Settings > Privacy & Security then click Allow Anyway to permit the example to run. After allowing and re-running, you may get a pop-up asking for permission. Select Open Anyway to run the script. ​ Usage Example Here’s a basic example of how to add Krisp noise reduction to your Pipecat pipeline: Copy Ask AI from pipecat.audio.filters.krisp_filter import KrispFilter from pipecat.transports.services.daily import DailyParams, DailyTransport # Add to transport configuration transport = DailyTransport( room_url, token, "Audio Bot" , DailyParams( audio_in_filter = KrispFilter(), # Enable Krisp noise reduction audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer(), ), ) ​ Troubleshooting Common issues and solutions: Missing Dependencies Copy Ask AI Error: Missing module: pipecat_ai_krisp Solution: Ensure you’ve installed with the krisp extra: pip install "pipecat-ai[krisp]" Model Path Not Found Copy Ask AI Error: Model path for KrispAudioProcessor must be provided Solution: Set the KRISP_MODEL_PATH environment variable or provide it in the constructor SDK Path Issues Copy Ask AI Error: Cannot find Krisp SDK Solution: Verify KRISP_SDK_PATH points to a valid Krisp SDK installation ​ Additional Resources KrispFilter Reference Documentation Example Application Metrics OpenAI Audio Models and APIs On this page Overview Walkthrough Get Access to Krisp SDK and Models Install build dependencies Install the pipecat-ai-krisp module Test the integration Important for macOS users Usage Example Troubleshooting Additional Resources Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/features_openai-audio-models-and-apis_3a5b3150.txt b/features_openai-audio-models-and-apis_3a5b3150.txt
new file mode 100644
index 0000000000000000000000000000000000000000..202c6a39ae5e7147ac1c2e5c2c0b322f4bd8cddf
--- /dev/null
+++ b/features_openai-audio-models-and-apis_3a5b3150.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/guides/features/openai-audio-models-and-apis#transcription-api
+Title: Building With OpenAI Audio Models and APIs - Pipecat
+==================================================
+
+Building With OpenAI Audio Models and APIs - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Features Building With OpenAI Audio Models and APIs Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Guides Fundamentals Context Management Custom FrameProcessor Detecting Idle Users Ending a Pipeline Function Calling Muting User Input Recording Audio Recording Transcripts Features Gemini Multimodal Live Metrics Noise cancellation with Krisp OpenAI Audio Models and APIs Pipecat Flows Telephony Overview Dial-in: WebRTC (Daily) Dial-in: WebRTC (Twilio + Daily) Dial-in: Twilio (Media Streams) Dialout: WebRTC (Daily) Deploying your bot Overview Deployment pattern Example: Pipecat Cloud Example: Fly.io Example: Cerebrium Example: Modal This guide provides an overview of the audio capabilities OpenAI offers via their APIs. We’ll also link to Pipecat sample code. ​ Two Ways To Build Voice-to-voice You can build voice-to-voice applications in two ways: The cascaded models approach, using separate models for transcription, the LLM, and voice generation. A cascaded pipeline looks like this, in Pipecat code. Here’s a single-file example that uses a cascaded pipeline . (See below for an overview of Pipecat core concepts.) Copy Ask AI pipeline = Pipeline( [ transport.input(), speech_to_text, context_aggregator.user(), llm, text_to_speech, context_aggregator.assistant(), transport.output(), ] ) Using a single, speech-to-speech model. This is conceptually much simpler. Though note that most applications also need to implement things like function calling, retrieval-augmented search, context management, and integration with existing systems. So the core pipeline is only part of an app’s complexity. Here’s a speech-to-speech pipeline in Pipecat code. And here’s a single-file example that uses the OpenAI Realtime API . Copy Ask AI pipeline = Pipeline( [ transport.input(), context_aggregator.user(), speech_to_speech_llm, context_aggregator.assistant(), transport.output(), ] ) Which approach should you choose? The cascaded models approach is preferable if you are implementing a complex workflow and need the best possible instruction following performance and function calling reliability. The gpt-4o model operating in text-to-text mode has the strongest instruction following and function calling performance. The speech-to-speech approach offers better audio understanding and human-like voice output. If your application is primarily free-form, open-ended conversation, these attributes might be more important than instruction following and function calling performance. Note also that gpt-4o-audio-preview and the OpenAI Realtime API are currently beta products. ​ OpenAI Audio Models and APIs ​ Transcription API Models: gpt-4o-transcribe , gpt-4o-mini-transcribe Pipecat service: OpenAISTTService ( reference docs ) OpenAI endpoint: /v1/audio/transcriptions ( docs ) ​ Chat Completions API Models: gpt-4o , gpt-4o-mini , gpt-4o-audio-preview Pipecat service: OpenAILLMService ( reference docs ) OpenAI endpoint: /v1/chat/completions ( docs ) ​ Realtime API Models: gpt-4o-realtime-preview , gpt-4o-mini-realtime-preview Pipecat service: OpenAIRealtimeBetaLLMService ( reference docs ) OpenAI docs ( overview ) ​ Speech API Models: gpt-4o-mini-tts Pipecat service: OpenAITTSService ( reference docs ) OpenAI endpoint: /v1/audio/speech ( docs ) ​ Sample code and starter kits If you have a code example or starter kit you would like this doc to link to, please let us know. We can add examples that help people get started with the OpenAI audio models and APIs. ​ Single-file examples OpenAI STT → LLM → TTS A complete implementation demonstrating the cascaded approach with OpenAI services OpenAI Realtime API A speech-to-speech implementation using OpenAI’s Realtime API ​ OpenAI + Twilio + Pipecat Cloud This starter kit is a complete telephone voice agent that can talk about the NCAA March Madness basketball tournaments and look up realtime game information using function calls. The starter kit includes two bot configurations: cascaded model and speech-to-speech. The code can be packaged for deployment to Pipecat Cloud, a commercial platform for Pipecat agent hosting. Noise cancellation with Krisp Pipecat Flows On this page Two Ways To Build Voice-to-voice OpenAI Audio Models and APIs Transcription API Chat Completions API Realtime API Speech API Sample code and starter kits Single-file examples OpenAI + Twilio + Pipecat Cloud Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/features_openai-audio-models-and-apis_59563923.txt b/features_openai-audio-models-and-apis_59563923.txt
new file mode 100644
index 0000000000000000000000000000000000000000..d9116b7333e24d097765b52962ca579c67259e9e
--- /dev/null
+++ b/features_openai-audio-models-and-apis_59563923.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/guides/features/openai-audio-models-and-apis#sample-code-and-starter-kits
+Title: Building With OpenAI Audio Models and APIs - Pipecat
+==================================================
+
+Building With OpenAI Audio Models and APIs - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Features Building With OpenAI Audio Models and APIs Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Guides Fundamentals Context Management Custom FrameProcessor Detecting Idle Users Ending a Pipeline Function Calling Muting User Input Recording Audio Recording Transcripts Features Gemini Multimodal Live Metrics Noise cancellation with Krisp OpenAI Audio Models and APIs Pipecat Flows Telephony Overview Dial-in: WebRTC (Daily) Dial-in: WebRTC (Twilio + Daily) Dial-in: Twilio (Media Streams) Dialout: WebRTC (Daily) Deploying your bot Overview Deployment pattern Example: Pipecat Cloud Example: Fly.io Example: Cerebrium Example: Modal This guide provides an overview of the audio capabilities OpenAI offers via their APIs. We’ll also link to Pipecat sample code. ​ Two Ways To Build Voice-to-voice You can build voice-to-voice applications in two ways: The cascaded models approach, using separate models for transcription, the LLM, and voice generation. A cascaded pipeline looks like this, in Pipecat code. Here’s a single-file example that uses a cascaded pipeline . (See below for an overview of Pipecat core concepts.) Copy Ask AI pipeline = Pipeline( [ transport.input(), speech_to_text, context_aggregator.user(), llm, text_to_speech, context_aggregator.assistant(), transport.output(), ] ) Using a single, speech-to-speech model. This is conceptually much simpler. Though note that most applications also need to implement things like function calling, retrieval-augmented search, context management, and integration with existing systems. So the core pipeline is only part of an app’s complexity. Here’s a speech-to-speech pipeline in Pipecat code. And here’s a single-file example that uses the OpenAI Realtime API . Copy Ask AI pipeline = Pipeline( [ transport.input(), context_aggregator.user(), speech_to_speech_llm, context_aggregator.assistant(), transport.output(), ] ) Which approach should you choose? The cascaded models approach is preferable if you are implementing a complex workflow and need the best possible instruction following performance and function calling reliability. The gpt-4o model operating in text-to-text mode has the strongest instruction following and function calling performance. The speech-to-speech approach offers better audio understanding and human-like voice output. If your application is primarily free-form, open-ended conversation, these attributes might be more important than instruction following and function calling performance. Note also that gpt-4o-audio-preview and the OpenAI Realtime API are currently beta products. ​ OpenAI Audio Models and APIs ​ Transcription API Models: gpt-4o-transcribe , gpt-4o-mini-transcribe Pipecat service: OpenAISTTService ( reference docs ) OpenAI endpoint: /v1/audio/transcriptions ( docs ) ​ Chat Completions API Models: gpt-4o , gpt-4o-mini , gpt-4o-audio-preview Pipecat service: OpenAILLMService ( reference docs ) OpenAI endpoint: /v1/chat/completions ( docs ) ​ Realtime API Models: gpt-4o-realtime-preview , gpt-4o-mini-realtime-preview Pipecat service: OpenAIRealtimeBetaLLMService ( reference docs ) OpenAI docs ( overview ) ​ Speech API Models: gpt-4o-mini-tts Pipecat service: OpenAITTSService ( reference docs ) OpenAI endpoint: /v1/audio/speech ( docs ) ​ Sample code and starter kits If you have a code example or starter kit you would like this doc to link to, please let us know. We can add examples that help people get started with the OpenAI audio models and APIs. ​ Single-file examples OpenAI STT → LLM → TTS A complete implementation demonstrating the cascaded approach with OpenAI services OpenAI Realtime API A speech-to-speech implementation using OpenAI’s Realtime API ​ OpenAI + Twilio + Pipecat Cloud This starter kit is a complete telephone voice agent that can talk about the NCAA March Madness basketball tournaments and look up realtime game information using function calls. The starter kit includes two bot configurations: cascaded model and speech-to-speech. The code can be packaged for deployment to Pipecat Cloud, a commercial platform for Pipecat agent hosting. Noise cancellation with Krisp Pipecat Flows On this page Two Ways To Build Voice-to-voice OpenAI Audio Models and APIs Transcription API Chat Completions API Realtime API Speech API Sample code and starter kits Single-file examples OpenAI + Twilio + Pipecat Cloud Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/features_pipecat-flows_01de153f.txt b/features_pipecat-flows_01de153f.txt
new file mode 100644
index 0000000000000000000000000000000000000000..93fba3f7fe67a18cb2c3becd60d1ab4bda6a2e1d
--- /dev/null
+++ b/features_pipecat-flows_01de153f.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/guides/features/pipecat-flows#google-gemini-format
+Title: Pipecat Flows - Pipecat
+==================================================
+
+Pipecat Flows - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Features Pipecat Flows Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Guides Fundamentals Context Management Custom FrameProcessor Detecting Idle Users Ending a Pipeline Function Calling Muting User Input Recording Audio Recording Transcripts Features Gemini Multimodal Live Metrics Noise cancellation with Krisp OpenAI Audio Models and APIs Pipecat Flows Telephony Overview Dial-in: WebRTC (Daily) Dial-in: WebRTC (Twilio + Daily) Dial-in: Twilio (Media Streams) Dialout: WebRTC (Daily) Deploying your bot Overview Deployment pattern Example: Pipecat Cloud Example: Fly.io Example: Cerebrium Example: Modal Pipecat Flows provides a framework for building structured conversations in your AI applications. It enables you to create both predefined conversation paths and dynamically generated flows while handling the complexities of state management and LLM interactions. The framework consists of: A Python module for building conversation flows with Pipecat A visual editor for designing and exporting flow configurations ​ Key Concepts Nodes : Represent conversation states with specific messages and available functions Messages : Set the role and tasks for each node Functions : Define actions and transitions (Node functions for operations, Edge functions for transitions) Actions : Execute operations during state transitions (pre/post actions) State Management : Handle conversation state and data persistence ​ Example Flows Movie Explorer (Static) A static flow demonstrating movie exploration using OpenAI. Shows real API integration with TMDB, structured data collection, and state management. Insurance Policy (Dynamic) A dynamic flow using Google Gemini that adapts policy recommendations based on user responses. Demonstrates runtime node creation and conditional paths. These examples are fully functional and can be run locally. Make sure you have the required dependencies installed and API keys configured. ​ When to Use Static vs Dynamic Flows Static Flows are ideal when: Conversation structure is known upfront Paths follow predefined patterns Flow can be fully configured in advance Example: Customer service scripts, intake forms Dynamic Flows are better when: Paths depend on external data Flow structure needs runtime modification Complex decision trees are involved Example: Personalized recommendations, adaptive workflows ​ Installation If you’re already using Pipecat: Copy Ask AI pip install pipecat-ai-flows If you’re starting fresh: Copy Ask AI # Basic installation pip install pipecat-ai-flows # Install Pipecat with specific LLM provider options: pip install "pipecat-ai[daily,openai,deepgram]" # For OpenAI pip install "pipecat-ai[daily,anthropic,deepgram]" # For Anthropic pip install "pipecat-ai[daily,google,deepgram]" # For Google 💡 Want to design your flows visually? Try the online Flow Editor ​ Core Concepts ​ Designing Conversation Flows Functions in Pipecat Flows serve two key purposes: Processing data (likely by interfacing with external systems and APIs) Advancing the conversation to the next node Each function can do one or both. LLMs decide when to run each function, via their function calling (or tool calling) mechanism. ​ Defining a Function A function is expected to return a (result, next_node) tuple. More precisely, it’s expected to return: Copy Ask AI # (result, next_node) Tuple[Optional[FlowResult], Optional[Union[NodeConfig, str ]]] If the function processes data, it should return a non- None value for the first element of the tuple. This value should be a FlowResult or subclass. If the function advances the conversation to the next node, it should return a non- None value for the second element of the tuple. This value can be either: A NodeConfig defining the next node (for dynamic flows) A string identifying the next node (for static flows) ​ Example Function Copy Ask AI from pipecat_flows import FlowArgs, FlowManager, FlowResult, NodeConfig async def check_availability ( args : FlowArgs, flow_manager : FlowManager) -> tuple[FlowResult, NodeConfig]: # Read arguments date = args[ "date" ] time = args[ "time" ] # Read previously-stored data party_size = flow_manager.state.get( "party_size" ) # Use flow_manager for immediate user feedback await flow_manager.task.queue_frame(TTSSpeakFrame( "Checking our reservation system..." )) # Store data in flow state for later use flow_manager.state[ "requested_date" ] = date # Interface with reservation system is_available = await reservation_system.check_availability(date, time, party_size) # Assemble result result = { "status" : "success" , "available" : available } # Decide which node to go to next if is_available: next_node = create_confirmation_node() else : next_node = create_no_availability_node() # Return both result and next node return result, next_node ​ Node Structure Each node in your flow represents a conversation state and consists of three main components: ​ Messages Nodes use two types of messages to control the conversation: Role Messages : Define the bot’s personality or role (optional) Copy Ask AI "role_messages" : [ { "role" : "system" , "content" : "You are a friendly pizza ordering assistant. Keep responses casual and upbeat." } ] Task Messages : Define what the bot should do in the current node Copy Ask AI "task_messages" : [ { "role" : "system" , "content" : "Ask the customer which pizza size they'd like: small, medium, or large." } ] Role messages are typically defined in your initial node and inherited by subsequent nodes, while task messages are specific to each node’s purpose. ​ Functions Functions in Pipecat Flows can: Process data Specify node transitions Do both This leads to two conceptual types of functions: Node functions , which only process data. Edge functions , which also (or only) transition to the next node. The function itself ( which you can read more about here ) is usually wrapped in a function configuration, which also contains some metadata about the function. ​ Function Configuration Pipecat Flows supports three ways of specifying function configuration: Provider-specific dictionary format Copy Ask AI # Dictionary format { "type" : "function" , "function" : { "name" : "select_size" , "handler" : select_size, "description" : "Select pizza size" , "parameters" : { "type" : "object" , "properties" : { "size" : { "type" : "string" , "enum" : [ "small" , "medium" , "large" ]} } }, } } FlowsFunctionSchema Copy Ask AI # Using FlowsFunctionSchema from pipecat_flows import FlowsFunctionSchema size_function = FlowsFunctionSchema( name = "select_size" , description = "Select pizza size" , properties = { "size" : { "type" : "string" , "enum" : [ "small" , "medium" , "large" ]} }, required = [ "size" ], handler = select_size ) # Use in node configuration node_config = { "task_messages" : [ ... ], "functions" : [size_function] } The FlowsFunctionSchema approach provides some advantages over the provider-specific dictionary format: Consistent structure across LLM providers Simplified parameter definition Cleaner, more readable code Both dictionary and FlowsFunctionSchema approaches are fully supported. FlowsFunctionSchema is recommended for new projects as it provides better type checking and a provider-independent format. Direct function usage (auto-configuration) This approach lets you bypass specifying a standalone function configuration. Instead, relevant function metadata is automatically extracted from the function’s signature and docstring: name description properties (including individual property description s) required Note that the function signature is a bit different when using direct functions. The first parameter is the FlowManager , followed by any others necessary for the function. Copy Ask AI from pipecat_flows import FlowManager, FlowResult async def select_pizza_order ( flow_manager : FlowManager, size : str , pizza_type : str , additional_toppings : list[ str ] = [], ) -> tuple[FlowResult, str ]: """ Record the pizza order details. Args: size (str): Size of the pizza. Must be one of "small", "medium", or "large". pizza_type (str): Type of pizza. Must be one of "pepperoni", "cheese", "supreme", or "vegetarian". additional_toppings (list[str]): List of additional toppings. Defaults to empty list. """ ... # Use in node configuration node_config = { "task_messages" : [ ... ], "functions" : [select_pizza_order] } ​ Node Functions Functions that process data within a single conversational state, without switching nodes. When called, they: Execute their handler to do the data processing (typically by interfacing with an external system or API) Trigger an immediate LLM completion with the result Copy Ask AI from pipecat_flows import FlowArgs, FlowManager, FlowResult async def select_size ( args : FlowArgs, flow_manager : FlowManager) -> tuple[FlowResult, None ]: """Process pizza size selection.""" size = args[ "size" ] await ordering_system.record_size_selection(size) return { "status" : "success" , "size" : size }, None # Function configuration { "type" : "function" , "function" : { "name" : "select_size" , "handler" : select_size, "description" : "Select pizza size" , "parameters" : { "type" : "object" , "properties" : { "size" : { "type" : "string" , "enum" : [ "small" , "medium" , "large" ]} } }, } } ​ Edge Functions Functions that specify a transition between nodes (optionally processing data first). When called, they: Execute their handler to do any data processing (optional) and determine the next node Add the function result to the LLM context Trigger LLM completion after both the function result and the next node’s messages are in the context Copy Ask AI from pipecat_flows import FlowArgs, FlowManager, FlowResult async def select_size ( args : FlowArgs, flow_manager : FlowManager) -> tuple[FlowResult, NodeConfig]: """Process pizza size selection.""" size = args[ "size" ] await ordering_system.record_size_selection(size) result = { "status" : "success" , "size" : size } next_node = create_confirmation_node() return result, next_node # Function configuration { "type" : "function" , "function" : { "name" : "select_size" , "handler" : select_size, "description" : "Select pizza size" , "parameters" : { "type" : "object" , "properties" : { "size" : { "type" : "string" , "enum" : [ "small" , "medium" , "large" ]} } }, } } ​ Actions Actions are operations that execute as part of the lifecycle of a node, with two distinct timing options: Pre-actions: execute when entering the node, before the LLM completion Post-actions: execute after the LLM completion ​ Pre-Actions Execute when entering the node, before LLM inference. Useful for: Providing immediate feedback while waiting for LLM responses Bridging gaps during longer function calls Setting up state or context Copy Ask AI "pre_actions" : [ { "type" : "tts_say" , "text" : "Hold on a moment..." # Immediate feedback during processing } ], Note that when the node is configured with respond_immediately: False , the pre_actions still run when entering the node, which may be well before LLM inference, depending on how long the user takes to speak first. Avoid mixing tts_say actions with chat completions as this may result in a conversation flow that feels unnatural. tts_say are best used as filler words when the LLM will take time to generate an completion. ​ Post-Actions Execute after LLM inference completes. Useful for: Cleanup operations State finalization Ensuring proper sequence of operations Copy Ask AI "post_actions" : [ { "type" : "end_conversation" # Ensures TTS completes before ending } ] Note that when the node is configured with respond_immediately: False , the post_actions still only run after the first LLM inference, which may be a while depending on how long the user takes to speak first. ​ Timing Considerations Pre-actions : Execute immediately, before any LLM processing begins LLM Inference : Processes the node’s messages and functions Post-actions : Execute after LLM processing and TTS completion For example, when using end_conversation as a post-action, the sequence is: LLM generates response TTS speaks the response End conversation action executes This ordering ensures proper completion of all operations. ​ Action Types Flows comes equipped with pre-canned actions and you can also define your own action behavior. See the reference docs for more information. ​ Deciding Who Speaks First For each node in the conversation, you can decide whether the LLM should respond immediately upon entering the node (the default behavior) or whether the LLM should wait for the user to speak first before responding. You do this using the respond_immediately field. respond_immediately=False may be particularly useful in the very first node, especially in outbound-calling cases where the user has to first answer the phone to trigger the conversation. Copy Ask AI NodeConfig( task_messages = [ { "role" : "system" , "content" : "Warmly greet the customer and ask how many people are in their party. This is your only job for now; if the customer asks for something else, politely remind them you can't do it." , } ], respond_immediately = False , # ... other fields ) Keep in mind that if you specify respond_immediately=False , the user may not be aware of the conversational task at hand when entering the node (the bot hasn’t told them yet). While it’s always important to have guardrails in your node messages to keep the conversation on topic, letting the user speak first makes it even more so. ​ Context Management Pipecat Flows provides three strategies for managing conversation context during node transitions: ​ Context Strategies APPEND (default): Adds new messages to the existing context, maintaining the full conversation history RESET : Clears the context and starts fresh with the new node’s messages RESET_WITH_SUMMARY : Resets the context but includes an AI-generated summary of the previous conversation ​ Configuration Context strategies can be configured globally or per-node: Copy Ask AI from pipecat_flows import ContextStrategy, ContextStrategyConfig # Global strategy configuration flow_manager = FlowManager( task = task, llm = llm, context_aggregator = context_aggregator, context_strategy = ContextStrategyConfig( strategy = ContextStrategy. RESET_WITH_SUMMARY , summary_prompt = "Summarize the key points discussed so far, focusing on decisions made and important information collected." ) ) # Per-node strategy configuration node_config = { "task_messages" : [ ... ], "functions" : [ ... ], "context_strategy" : ContextStrategyConfig( strategy = ContextStrategy. RESET_WITH_SUMMARY , summary_prompt = "Provide a concise summary of the customer's order details and preferences." ) } ​ Strategy Selection Choose your strategy based on your conversation needs: Use APPEND when full conversation history is important Use RESET when previous context might confuse the current node’s purpose Use RESET_WITH_SUMMARY for long conversations where key points need to be preserved When using RESET_WITH_SUMMARY, if summary generation fails or times out, the system automatically falls back to RESET strategy for resilience. ​ State Management The state variable in FlowManager is a shared dictionary that persists throughout the conversation. Think of it as a conversation memory that lets you: Store user information Track conversation progress Share data between nodes Inform decision-making Here’s a practical example of a pizza ordering flow: Copy Ask AI # Store user choices as they're made async def select_size ( args : FlowArgs) -> tuple[FlowResult, str ]: """Handle pizza size selection.""" size = args[ "size" ] # Initialize order in state if it doesn't exist if "order" not in flow_manager.state: flow_manager.state[ "order" ] = {} # Store the selection flow_manager.state[ "order" ][ "size" ] = size return { "status" : "success" , "size" : size}, "toppings" async def select_toppings ( args : FlowArgs) -> tuple[FlowResult, str ]: """Handle topping selection.""" topping = args[ "topping" ] # Get existing order and toppings order = flow_manager.state.get( "order" , {}) toppings = order.get( "toppings" , []) # Add new topping toppings.append(topping) order[ "toppings" ] = toppings flow_manager.state[ "order" ] = order return { "status" : "success" , "toppings" : toppings}, "finalize" async def finalize_order ( args : FlowArgs) -> tuple[FlowResult, str ]: """Process the complete order.""" order = flow_manager.state.get( "order" , {}) # Validate order has required information if "size" not in order: return { "status" : "error" , "error" : "No size selected" } # Calculate price based on stored selections size = order[ "size" ] toppings = order.get( "toppings" , []) price = calculate_price(size, len (toppings)) return { "status" : "success" , "summary" : f "Ordered: { size } pizza with { ', ' .join(toppings) } " , "price" : price }, "end" In this example: select_size initializes the order and stores the size select_toppings builds a list of toppings finalize_order uses the stored information to process the complete order The state variable makes it easy to: Build up information across multiple interactions Access previous choices when needed Validate the complete order Calculate final results This is particularly useful when information needs to be collected across multiple conversation turns or when later decisions depend on earlier choices. ​ LLM Provider Support Pipecat Flows automatically handles format differences between LLM providers: ​ OpenAI Format Copy Ask AI "functions" : [{ "type" : "function" , "function" : { "name" : "function_name" , "handler" : select_size, "description" : "description" , "parameters" : { ... } } }] ​ Anthropic Format Copy Ask AI "functions" : [{ "name" : "function_name" , "handler" : select_size, "description" : "description" , "input_schema" : { ... } }] ​ Google (Gemini) Format Copy Ask AI "functions" : [{ "function_declarations" : [{ "name" : "function_name" , "handler" : select_size, "description" : "description" , "parameters" : { ... } }] }] You don’t need to handle these differences manually - Pipecat Flows adapts your configuration to the correct format based on your LLM provider. ​ Implementation Approaches ​ Static Flows Static flows use a configuration-driven approach where the entire conversation structure is defined upfront. ​ Basic Setup Copy Ask AI from pipecat_flows import FlowManager # Define flow configuration flow_config = { "initial_node" : "greeting" , "nodes" : { "greeting" : { "role_messages" : [ ... ], "task_messages" : [ ... ], "functions" : [ ... ] } } } # Initialize flow manager with static configuration flow_manager = FlowManager( task = task, llm = llm, context_aggregator = context_aggregator, flow_config = flow_config ) @transport.event_handler ( "on_first_participant_joined" ) async def on_first_participant_joined ( transport , participant ): await transport.capture_participant_transcription(participant[ "id" ]) await flow_manager.initialize() ​ Example FlowConfig Copy Ask AI flow_config = { "initial_node" : "start" , "nodes" : { "start" : { "role_messages" : [ { "role" : "system" , "content" : "You are an order-taking assistant. You must ALWAYS use the available functions to progress the conversation. This is a phone conversation and your responses will be converted to audio. Keep the conversation friendly, casual, and polite. Avoid outputting special characters and emojis." , } ], "task_messages" : [ { "role" : "system" , "content" : "You are an order-taking assistant. Ask if they want pizza or sushi." } ], "functions" : [ { "type" : "function" , "function" : { "name" : "choose_pizza" , "handler" : choose_pizza, # Returns [None, "pizza_order"] "description" : "User wants pizza" , "parameters" : { "type" : "object" , "properties" : {}} } } ] }, "pizza_order" : { "task_messages" : [ ... ], "functions" : [ { "type" : "function" , "function" : { "name" : "select_size" , "handler" : select_size, # Returns [FlowResult, "toppings"] "description" : "Select pizza size" , "parameters" : { "type" : "object" , "properties" : { "size" : { "type" : "string" , "enum" : [ "small" , "medium" , "large" ]} } } } } ] } } } ​ Dynamic Flows Dynamic flows create and modify conversation paths at runtime based on data or business logic. ​ Example Implementation Here’s a complete example of a dynamic insurance quote flow: Copy Ask AI from pipecat_flows import FlowManager, FlowArgs, FlowResult # Define handlers and transitions async def collect_age ( args : FlowArgs, flow_manager : FlowManager) -> tuple[AgeResult, NodeConfig]: """Process age collection.""" age = args[ "age" ] # Assemble result result = AgeResult( status = "success" , age = age) # Decide which node to go to next if age < 25 : await flow_manager.set_node_from_config(create_young_adult_node()) else : await flow_manager.set_node_from_config(create_standard_node()) return result, age # Node creation functions def create_initial_node () -> NodeConfig: """Create initial age collection node.""" return { "name" : "initial" , "role_messages" : [ { "role" : "system" , "content" : "You are an insurance quote assistant." } ], "task_messages" : [ { "role" : "system" , "content" : "Ask for the customer's age." } ], "functions" : [ { "type" : "function" , "function" : { "name" : "collect_age" , "handler" : collect_age, "description" : "Collect customer age" , "parameters" : { "type" : "object" , "properties" : { "age" : { "type" : "integer" } } } } } ] } def create_young_adult_node () -> Dict[ str , Any]: """Create node for young adult quotes.""" return { "name" : "young_adult" , "task_messages" : [ { "role" : "system" , "content" : "Explain our special young adult coverage options." } ], "functions" : [ ... ] # Additional quote-specific functions } def create_standard_node () -> Dict[ str , Any]: """Create node for standard quotes.""" return { "name" : "standard" , "task_messages" : [ { "role" : "system" , "content" : "Present our standard coverage options." } ], "functions" : [ ... ] # Additional quote-specific functions } # Initialize flow manager flow_manager = FlowManager( task = task, llm = llm, context_aggregator = context_aggregator, ) @transport.event_handler ( "on_first_participant_joined" ) async def on_first_participant_joined ( transport , participant ): await transport.capture_participant_transcription(participant[ "id" ]) await flow_manager.initialize(create_initial_node()) ​ Best Practices Store shared data in flow_manager.state Create separate functions for node creation ​ Flow Editor The Pipecat Flow Editor provides a visual interface for creating and managing conversation flows. It offers a node-based interface that makes it easier to design, visualize, and modify your flows. ​ Visual Design ​ Node Types Start Node (Green): Entry point of your flow Copy Ask AI "greeting" : { "role_messages" : [ ... ], "task_messages" : [ ... ], "functions" : [ ... ] } Flow Nodes (Blue): Intermediate states Copy Ask AI "collect_info" : { "task_messages" : [ ... ], "functions" : [ ... ], "pre_actions" : [ ... ] } End Node (Red): Final state Copy Ask AI "end" : { "task_messages" : [ ... ], "functions" : [], "post_actions" : [{ "type" : "end_conversation" }] } Function Nodes : Edge Functions (Purple): Create transitions Copy Ask AI { "name" : "next_node" , "description" : "Transition to next state" } Node Functions (Orange): Perform operations Copy Ask AI { "name" : "process_data" , "handler" : process_data_handler, "description" : "Process user data" } ​ Naming Conventions Start Node : Use descriptive names (e.g., “greeting”, “welcome”) Flow Nodes : Name based on purpose (e.g., “collect_info”, “verify_data”) End Node : Conventionally named “end” Functions : Use clear, action-oriented names ​ Function Configuration Copy Ask AI { "type" : "function" , "function" : { "name" : "process_data" , "handler" : process_handler, "description" : "Process user data" , "parameters" : { ... } } } When using the Flow Editor, function handlers can be specified using the __function__: token: Copy Ask AI { "type" : "function" , "function" : { "name" : "process_data" , "handler" : "__function__:process_data" , # References function in main script "description" : "Process user data" , "parameters" : { ... } } } The handler will be looked up in your main script when the flow is executed. When function handlers are specified in the flow editor, they will be exported with the __function__: token. ​ Using the Editor ​ Creating a New Flow Start with a descriptively named Start Node Add Flow Nodes for each conversation state Connect nodes using Edge Functions Add Node Functions for operations Include an End Node ​ Import/Export Copy Ask AI # Export format { "initial_node" : "greeting" , "nodes" : { "greeting" : { "role_messages" : [ ... ], "task_messages" : [ ... ], "functions" : [ ... ], "pre_actions" : [ ... ] }, "process" : { "task_messages" : [ ... ], "functions" : [ ... ], }, "end" : { "task_messages" : [ ... ], "functions" : [], "post_actions" : [ ... ] } } } ​ Tips Use the visual preview to verify flow logic Test exported configurations Document node purposes and transitions Keep flows modular and maintainable Try the editor at flows.pipecat.ai OpenAI Audio Models and APIs Overview On this page Key Concepts Example Flows When to Use Static vs Dynamic Flows Installation Core Concepts Designing Conversation Flows Defining a Function Example Function Node Structure Messages Functions Function Configuration Node Functions Edge Functions Actions Pre-Actions Post-Actions Timing Considerations Action Types Deciding Who Speaks First Context Management Context Strategies Configuration Strategy Selection State Management LLM Provider Support OpenAI Format Anthropic Format Google (Gemini) Format Implementation Approaches Static Flows Basic Setup Example FlowConfig Dynamic Flows Example Implementation Best Practices Flow Editor Visual Design Node Types Naming Conventions Function Configuration Using the Editor Creating a New Flow Import/Export Tips Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/features_pipecat-flows_11f6dd1d.txt b/features_pipecat-flows_11f6dd1d.txt
new file mode 100644
index 0000000000000000000000000000000000000000..2c923e57942b27610562b95198e7da18b03d6f44
--- /dev/null
+++ b/features_pipecat-flows_11f6dd1d.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/_sites/docs.pipecat.ai/guides/features/pipecat-flows#real-time-processing
+Title: Overview - Pipecat
+==================================================
+
+Overview - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Get Started Overview Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Get Started Overview Installation & Setup Quickstart Core Concepts Next Steps & Examples Pipecat is an open source Python framework that handles the complex orchestration of AI services, network transport, audio processing, and multimodal interactions. “Multimodal” means you can use any combination of audio, video, images, and/or text in your interactions. And “real-time” means that things are happening quickly enough that it feels conversational—a “back-and-forth” with a bot, not submitting a query and waiting for results. ​ What You Can Build Voice Assistants Natural, real-time conversations with AI using speech recognition and synthesis Interactive Agents Personal coaches and meeting assistants that can understand context and provide guidance Multimodal Apps Applications that combine voice, video, images, and text for rich interactions Creative Tools Storytelling experiences and social companions that engage users Business Solutions Customer intake flows and support bots for automated business processes Complex Flows Structured conversations using Pipecat Flows for managing complex interactions ​ How It Works The flow of interactions in a Pipecat application is typically straightforward: The bot says something The user says something The bot says something The user says something This continues until the conversation naturally ends. While this flow seems simple, making it feel natural requires sophisticated real-time processing. ​ Real-time Processing Pipecat’s pipeline architecture handles both simple voice interactions and complex multimodal processing. Let’s look at how data flows through the system: Voice app Multimodal app 1 Send Audio Transmit and capture streamed audio from the user 2 Transcribe Speech Convert speech to text as the user is talking 3 Process with LLM Generate responses using a large language model 4 Convert to Speech Transform text responses into natural speech 5 Play Audio Stream the audio response back to the user 1 Send Audio Transmit and capture streamed audio from the user 2 Transcribe Speech Convert speech to text as the user is talking 3 Process with LLM Generate responses using a large language model 4 Convert to Speech Transform text responses into natural speech 5 Play Audio Stream the audio response back to the user 1 Send Audio and Video Transmit and capture audio, video, and image inputs simultaneously 2 Process Streams Handle multiple input streams in parallel 3 Model Processing Send combined inputs to multimodal models (like GPT-4V) 4 Generate Outputs Create various outputs (text, images, audio, etc.) 5 Coordinate Presentation Synchronize and present multiple output types In both cases, Pipecat: Processes responses as they stream in Handles multiple input/output modalities concurrently Manages resource allocation and synchronization Coordinates parallel processing tasks This architecture creates fluid, natural interactions without noticeable delays, whether you’re building a simple voice assistant or a complex multimodal application. Pipecat’s pipeline architecture is particularly valuable for managing the complexity of real-time, multimodal interactions, ensuring smooth data flow and proper synchronization regardless of the input/output types involved. Pipecat handles all this complexity for you, letting you focus on building your application rather than managing the underlying infrastructure. ​ Next Steps Ready to build your first Pipecat application? Installation & Setup Prepare your environment and install required dependencies Quickstart Build and run your first Pipecat application Core Concepts Learn about pipelines, frames, and real-time processing Use Cases Explore example implementations and patterns ​ Join Our Community Discord Community Connect with other developers, share your projects, and get support from the Pipecat team. Installation & Setup On this page What You Can Build How It Works Real-time Processing Next Steps Join Our Community Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/features_pipecat-flows_19496fa3.txt b/features_pipecat-flows_19496fa3.txt
new file mode 100644
index 0000000000000000000000000000000000000000..18ce57c10b69a20f2aebcb14247f5346d02d01ee
--- /dev/null
+++ b/features_pipecat-flows_19496fa3.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/guides/features/pipecat-flows#static-flows
+Title: Pipecat Flows - Pipecat
+==================================================
+
+Pipecat Flows - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Features Pipecat Flows Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Guides Fundamentals Context Management Custom FrameProcessor Detecting Idle Users Ending a Pipeline Function Calling Muting User Input Recording Audio Recording Transcripts Features Gemini Multimodal Live Metrics Noise cancellation with Krisp OpenAI Audio Models and APIs Pipecat Flows Telephony Overview Dial-in: WebRTC (Daily) Dial-in: WebRTC (Twilio + Daily) Dial-in: Twilio (Media Streams) Dialout: WebRTC (Daily) Deploying your bot Overview Deployment pattern Example: Pipecat Cloud Example: Fly.io Example: Cerebrium Example: Modal Pipecat Flows provides a framework for building structured conversations in your AI applications. It enables you to create both predefined conversation paths and dynamically generated flows while handling the complexities of state management and LLM interactions. The framework consists of: A Python module for building conversation flows with Pipecat A visual editor for designing and exporting flow configurations ​ Key Concepts Nodes : Represent conversation states with specific messages and available functions Messages : Set the role and tasks for each node Functions : Define actions and transitions (Node functions for operations, Edge functions for transitions) Actions : Execute operations during state transitions (pre/post actions) State Management : Handle conversation state and data persistence ​ Example Flows Movie Explorer (Static) A static flow demonstrating movie exploration using OpenAI. Shows real API integration with TMDB, structured data collection, and state management. Insurance Policy (Dynamic) A dynamic flow using Google Gemini that adapts policy recommendations based on user responses. Demonstrates runtime node creation and conditional paths. These examples are fully functional and can be run locally. Make sure you have the required dependencies installed and API keys configured. ​ When to Use Static vs Dynamic Flows Static Flows are ideal when: Conversation structure is known upfront Paths follow predefined patterns Flow can be fully configured in advance Example: Customer service scripts, intake forms Dynamic Flows are better when: Paths depend on external data Flow structure needs runtime modification Complex decision trees are involved Example: Personalized recommendations, adaptive workflows ​ Installation If you’re already using Pipecat: Copy Ask AI pip install pipecat-ai-flows If you’re starting fresh: Copy Ask AI # Basic installation pip install pipecat-ai-flows # Install Pipecat with specific LLM provider options: pip install "pipecat-ai[daily,openai,deepgram]" # For OpenAI pip install "pipecat-ai[daily,anthropic,deepgram]" # For Anthropic pip install "pipecat-ai[daily,google,deepgram]" # For Google 💡 Want to design your flows visually? Try the online Flow Editor ​ Core Concepts ​ Designing Conversation Flows Functions in Pipecat Flows serve two key purposes: Processing data (likely by interfacing with external systems and APIs) Advancing the conversation to the next node Each function can do one or both. LLMs decide when to run each function, via their function calling (or tool calling) mechanism. ​ Defining a Function A function is expected to return a (result, next_node) tuple. More precisely, it’s expected to return: Copy Ask AI # (result, next_node) Tuple[Optional[FlowResult], Optional[Union[NodeConfig, str ]]] If the function processes data, it should return a non- None value for the first element of the tuple. This value should be a FlowResult or subclass. If the function advances the conversation to the next node, it should return a non- None value for the second element of the tuple. This value can be either: A NodeConfig defining the next node (for dynamic flows) A string identifying the next node (for static flows) ​ Example Function Copy Ask AI from pipecat_flows import FlowArgs, FlowManager, FlowResult, NodeConfig async def check_availability ( args : FlowArgs, flow_manager : FlowManager) -> tuple[FlowResult, NodeConfig]: # Read arguments date = args[ "date" ] time = args[ "time" ] # Read previously-stored data party_size = flow_manager.state.get( "party_size" ) # Use flow_manager for immediate user feedback await flow_manager.task.queue_frame(TTSSpeakFrame( "Checking our reservation system..." )) # Store data in flow state for later use flow_manager.state[ "requested_date" ] = date # Interface with reservation system is_available = await reservation_system.check_availability(date, time, party_size) # Assemble result result = { "status" : "success" , "available" : available } # Decide which node to go to next if is_available: next_node = create_confirmation_node() else : next_node = create_no_availability_node() # Return both result and next node return result, next_node ​ Node Structure Each node in your flow represents a conversation state and consists of three main components: ​ Messages Nodes use two types of messages to control the conversation: Role Messages : Define the bot’s personality or role (optional) Copy Ask AI "role_messages" : [ { "role" : "system" , "content" : "You are a friendly pizza ordering assistant. Keep responses casual and upbeat." } ] Task Messages : Define what the bot should do in the current node Copy Ask AI "task_messages" : [ { "role" : "system" , "content" : "Ask the customer which pizza size they'd like: small, medium, or large." } ] Role messages are typically defined in your initial node and inherited by subsequent nodes, while task messages are specific to each node’s purpose. ​ Functions Functions in Pipecat Flows can: Process data Specify node transitions Do both This leads to two conceptual types of functions: Node functions , which only process data. Edge functions , which also (or only) transition to the next node. The function itself ( which you can read more about here ) is usually wrapped in a function configuration, which also contains some metadata about the function. ​ Function Configuration Pipecat Flows supports three ways of specifying function configuration: Provider-specific dictionary format Copy Ask AI # Dictionary format { "type" : "function" , "function" : { "name" : "select_size" , "handler" : select_size, "description" : "Select pizza size" , "parameters" : { "type" : "object" , "properties" : { "size" : { "type" : "string" , "enum" : [ "small" , "medium" , "large" ]} } }, } } FlowsFunctionSchema Copy Ask AI # Using FlowsFunctionSchema from pipecat_flows import FlowsFunctionSchema size_function = FlowsFunctionSchema( name = "select_size" , description = "Select pizza size" , properties = { "size" : { "type" : "string" , "enum" : [ "small" , "medium" , "large" ]} }, required = [ "size" ], handler = select_size ) # Use in node configuration node_config = { "task_messages" : [ ... ], "functions" : [size_function] } The FlowsFunctionSchema approach provides some advantages over the provider-specific dictionary format: Consistent structure across LLM providers Simplified parameter definition Cleaner, more readable code Both dictionary and FlowsFunctionSchema approaches are fully supported. FlowsFunctionSchema is recommended for new projects as it provides better type checking and a provider-independent format. Direct function usage (auto-configuration) This approach lets you bypass specifying a standalone function configuration. Instead, relevant function metadata is automatically extracted from the function’s signature and docstring: name description properties (including individual property description s) required Note that the function signature is a bit different when using direct functions. The first parameter is the FlowManager , followed by any others necessary for the function. Copy Ask AI from pipecat_flows import FlowManager, FlowResult async def select_pizza_order ( flow_manager : FlowManager, size : str , pizza_type : str , additional_toppings : list[ str ] = [], ) -> tuple[FlowResult, str ]: """ Record the pizza order details. Args: size (str): Size of the pizza. Must be one of "small", "medium", or "large". pizza_type (str): Type of pizza. Must be one of "pepperoni", "cheese", "supreme", or "vegetarian". additional_toppings (list[str]): List of additional toppings. Defaults to empty list. """ ... # Use in node configuration node_config = { "task_messages" : [ ... ], "functions" : [select_pizza_order] } ​ Node Functions Functions that process data within a single conversational state, without switching nodes. When called, they: Execute their handler to do the data processing (typically by interfacing with an external system or API) Trigger an immediate LLM completion with the result Copy Ask AI from pipecat_flows import FlowArgs, FlowManager, FlowResult async def select_size ( args : FlowArgs, flow_manager : FlowManager) -> tuple[FlowResult, None ]: """Process pizza size selection.""" size = args[ "size" ] await ordering_system.record_size_selection(size) return { "status" : "success" , "size" : size }, None # Function configuration { "type" : "function" , "function" : { "name" : "select_size" , "handler" : select_size, "description" : "Select pizza size" , "parameters" : { "type" : "object" , "properties" : { "size" : { "type" : "string" , "enum" : [ "small" , "medium" , "large" ]} } }, } } ​ Edge Functions Functions that specify a transition between nodes (optionally processing data first). When called, they: Execute their handler to do any data processing (optional) and determine the next node Add the function result to the LLM context Trigger LLM completion after both the function result and the next node’s messages are in the context Copy Ask AI from pipecat_flows import FlowArgs, FlowManager, FlowResult async def select_size ( args : FlowArgs, flow_manager : FlowManager) -> tuple[FlowResult, NodeConfig]: """Process pizza size selection.""" size = args[ "size" ] await ordering_system.record_size_selection(size) result = { "status" : "success" , "size" : size } next_node = create_confirmation_node() return result, next_node # Function configuration { "type" : "function" , "function" : { "name" : "select_size" , "handler" : select_size, "description" : "Select pizza size" , "parameters" : { "type" : "object" , "properties" : { "size" : { "type" : "string" , "enum" : [ "small" , "medium" , "large" ]} } }, } } ​ Actions Actions are operations that execute as part of the lifecycle of a node, with two distinct timing options: Pre-actions: execute when entering the node, before the LLM completion Post-actions: execute after the LLM completion ​ Pre-Actions Execute when entering the node, before LLM inference. Useful for: Providing immediate feedback while waiting for LLM responses Bridging gaps during longer function calls Setting up state or context Copy Ask AI "pre_actions" : [ { "type" : "tts_say" , "text" : "Hold on a moment..." # Immediate feedback during processing } ], Note that when the node is configured with respond_immediately: False , the pre_actions still run when entering the node, which may be well before LLM inference, depending on how long the user takes to speak first. Avoid mixing tts_say actions with chat completions as this may result in a conversation flow that feels unnatural. tts_say are best used as filler words when the LLM will take time to generate an completion. ​ Post-Actions Execute after LLM inference completes. Useful for: Cleanup operations State finalization Ensuring proper sequence of operations Copy Ask AI "post_actions" : [ { "type" : "end_conversation" # Ensures TTS completes before ending } ] Note that when the node is configured with respond_immediately: False , the post_actions still only run after the first LLM inference, which may be a while depending on how long the user takes to speak first. ​ Timing Considerations Pre-actions : Execute immediately, before any LLM processing begins LLM Inference : Processes the node’s messages and functions Post-actions : Execute after LLM processing and TTS completion For example, when using end_conversation as a post-action, the sequence is: LLM generates response TTS speaks the response End conversation action executes This ordering ensures proper completion of all operations. ​ Action Types Flows comes equipped with pre-canned actions and you can also define your own action behavior. See the reference docs for more information. ​ Deciding Who Speaks First For each node in the conversation, you can decide whether the LLM should respond immediately upon entering the node (the default behavior) or whether the LLM should wait for the user to speak first before responding. You do this using the respond_immediately field. respond_immediately=False may be particularly useful in the very first node, especially in outbound-calling cases where the user has to first answer the phone to trigger the conversation. Copy Ask AI NodeConfig( task_messages = [ { "role" : "system" , "content" : "Warmly greet the customer and ask how many people are in their party. This is your only job for now; if the customer asks for something else, politely remind them you can't do it." , } ], respond_immediately = False , # ... other fields ) Keep in mind that if you specify respond_immediately=False , the user may not be aware of the conversational task at hand when entering the node (the bot hasn’t told them yet). While it’s always important to have guardrails in your node messages to keep the conversation on topic, letting the user speak first makes it even more so. ​ Context Management Pipecat Flows provides three strategies for managing conversation context during node transitions: ​ Context Strategies APPEND (default): Adds new messages to the existing context, maintaining the full conversation history RESET : Clears the context and starts fresh with the new node’s messages RESET_WITH_SUMMARY : Resets the context but includes an AI-generated summary of the previous conversation ​ Configuration Context strategies can be configured globally or per-node: Copy Ask AI from pipecat_flows import ContextStrategy, ContextStrategyConfig # Global strategy configuration flow_manager = FlowManager( task = task, llm = llm, context_aggregator = context_aggregator, context_strategy = ContextStrategyConfig( strategy = ContextStrategy. RESET_WITH_SUMMARY , summary_prompt = "Summarize the key points discussed so far, focusing on decisions made and important information collected." ) ) # Per-node strategy configuration node_config = { "task_messages" : [ ... ], "functions" : [ ... ], "context_strategy" : ContextStrategyConfig( strategy = ContextStrategy. RESET_WITH_SUMMARY , summary_prompt = "Provide a concise summary of the customer's order details and preferences." ) } ​ Strategy Selection Choose your strategy based on your conversation needs: Use APPEND when full conversation history is important Use RESET when previous context might confuse the current node’s purpose Use RESET_WITH_SUMMARY for long conversations where key points need to be preserved When using RESET_WITH_SUMMARY, if summary generation fails or times out, the system automatically falls back to RESET strategy for resilience. ​ State Management The state variable in FlowManager is a shared dictionary that persists throughout the conversation. Think of it as a conversation memory that lets you: Store user information Track conversation progress Share data between nodes Inform decision-making Here’s a practical example of a pizza ordering flow: Copy Ask AI # Store user choices as they're made async def select_size ( args : FlowArgs) -> tuple[FlowResult, str ]: """Handle pizza size selection.""" size = args[ "size" ] # Initialize order in state if it doesn't exist if "order" not in flow_manager.state: flow_manager.state[ "order" ] = {} # Store the selection flow_manager.state[ "order" ][ "size" ] = size return { "status" : "success" , "size" : size}, "toppings" async def select_toppings ( args : FlowArgs) -> tuple[FlowResult, str ]: """Handle topping selection.""" topping = args[ "topping" ] # Get existing order and toppings order = flow_manager.state.get( "order" , {}) toppings = order.get( "toppings" , []) # Add new topping toppings.append(topping) order[ "toppings" ] = toppings flow_manager.state[ "order" ] = order return { "status" : "success" , "toppings" : toppings}, "finalize" async def finalize_order ( args : FlowArgs) -> tuple[FlowResult, str ]: """Process the complete order.""" order = flow_manager.state.get( "order" , {}) # Validate order has required information if "size" not in order: return { "status" : "error" , "error" : "No size selected" } # Calculate price based on stored selections size = order[ "size" ] toppings = order.get( "toppings" , []) price = calculate_price(size, len (toppings)) return { "status" : "success" , "summary" : f "Ordered: { size } pizza with { ', ' .join(toppings) } " , "price" : price }, "end" In this example: select_size initializes the order and stores the size select_toppings builds a list of toppings finalize_order uses the stored information to process the complete order The state variable makes it easy to: Build up information across multiple interactions Access previous choices when needed Validate the complete order Calculate final results This is particularly useful when information needs to be collected across multiple conversation turns or when later decisions depend on earlier choices. ​ LLM Provider Support Pipecat Flows automatically handles format differences between LLM providers: ​ OpenAI Format Copy Ask AI "functions" : [{ "type" : "function" , "function" : { "name" : "function_name" , "handler" : select_size, "description" : "description" , "parameters" : { ... } } }] ​ Anthropic Format Copy Ask AI "functions" : [{ "name" : "function_name" , "handler" : select_size, "description" : "description" , "input_schema" : { ... } }] ​ Google (Gemini) Format Copy Ask AI "functions" : [{ "function_declarations" : [{ "name" : "function_name" , "handler" : select_size, "description" : "description" , "parameters" : { ... } }] }] You don’t need to handle these differences manually - Pipecat Flows adapts your configuration to the correct format based on your LLM provider. ​ Implementation Approaches ​ Static Flows Static flows use a configuration-driven approach where the entire conversation structure is defined upfront. ​ Basic Setup Copy Ask AI from pipecat_flows import FlowManager # Define flow configuration flow_config = { "initial_node" : "greeting" , "nodes" : { "greeting" : { "role_messages" : [ ... ], "task_messages" : [ ... ], "functions" : [ ... ] } } } # Initialize flow manager with static configuration flow_manager = FlowManager( task = task, llm = llm, context_aggregator = context_aggregator, flow_config = flow_config ) @transport.event_handler ( "on_first_participant_joined" ) async def on_first_participant_joined ( transport , participant ): await transport.capture_participant_transcription(participant[ "id" ]) await flow_manager.initialize() ​ Example FlowConfig Copy Ask AI flow_config = { "initial_node" : "start" , "nodes" : { "start" : { "role_messages" : [ { "role" : "system" , "content" : "You are an order-taking assistant. You must ALWAYS use the available functions to progress the conversation. This is a phone conversation and your responses will be converted to audio. Keep the conversation friendly, casual, and polite. Avoid outputting special characters and emojis." , } ], "task_messages" : [ { "role" : "system" , "content" : "You are an order-taking assistant. Ask if they want pizza or sushi." } ], "functions" : [ { "type" : "function" , "function" : { "name" : "choose_pizza" , "handler" : choose_pizza, # Returns [None, "pizza_order"] "description" : "User wants pizza" , "parameters" : { "type" : "object" , "properties" : {}} } } ] }, "pizza_order" : { "task_messages" : [ ... ], "functions" : [ { "type" : "function" , "function" : { "name" : "select_size" , "handler" : select_size, # Returns [FlowResult, "toppings"] "description" : "Select pizza size" , "parameters" : { "type" : "object" , "properties" : { "size" : { "type" : "string" , "enum" : [ "small" , "medium" , "large" ]} } } } } ] } } } ​ Dynamic Flows Dynamic flows create and modify conversation paths at runtime based on data or business logic. ​ Example Implementation Here’s a complete example of a dynamic insurance quote flow: Copy Ask AI from pipecat_flows import FlowManager, FlowArgs, FlowResult # Define handlers and transitions async def collect_age ( args : FlowArgs, flow_manager : FlowManager) -> tuple[AgeResult, NodeConfig]: """Process age collection.""" age = args[ "age" ] # Assemble result result = AgeResult( status = "success" , age = age) # Decide which node to go to next if age < 25 : await flow_manager.set_node_from_config(create_young_adult_node()) else : await flow_manager.set_node_from_config(create_standard_node()) return result, age # Node creation functions def create_initial_node () -> NodeConfig: """Create initial age collection node.""" return { "name" : "initial" , "role_messages" : [ { "role" : "system" , "content" : "You are an insurance quote assistant." } ], "task_messages" : [ { "role" : "system" , "content" : "Ask for the customer's age." } ], "functions" : [ { "type" : "function" , "function" : { "name" : "collect_age" , "handler" : collect_age, "description" : "Collect customer age" , "parameters" : { "type" : "object" , "properties" : { "age" : { "type" : "integer" } } } } } ] } def create_young_adult_node () -> Dict[ str , Any]: """Create node for young adult quotes.""" return { "name" : "young_adult" , "task_messages" : [ { "role" : "system" , "content" : "Explain our special young adult coverage options." } ], "functions" : [ ... ] # Additional quote-specific functions } def create_standard_node () -> Dict[ str , Any]: """Create node for standard quotes.""" return { "name" : "standard" , "task_messages" : [ { "role" : "system" , "content" : "Present our standard coverage options." } ], "functions" : [ ... ] # Additional quote-specific functions } # Initialize flow manager flow_manager = FlowManager( task = task, llm = llm, context_aggregator = context_aggregator, ) @transport.event_handler ( "on_first_participant_joined" ) async def on_first_participant_joined ( transport , participant ): await transport.capture_participant_transcription(participant[ "id" ]) await flow_manager.initialize(create_initial_node()) ​ Best Practices Store shared data in flow_manager.state Create separate functions for node creation ​ Flow Editor The Pipecat Flow Editor provides a visual interface for creating and managing conversation flows. It offers a node-based interface that makes it easier to design, visualize, and modify your flows. ​ Visual Design ​ Node Types Start Node (Green): Entry point of your flow Copy Ask AI "greeting" : { "role_messages" : [ ... ], "task_messages" : [ ... ], "functions" : [ ... ] } Flow Nodes (Blue): Intermediate states Copy Ask AI "collect_info" : { "task_messages" : [ ... ], "functions" : [ ... ], "pre_actions" : [ ... ] } End Node (Red): Final state Copy Ask AI "end" : { "task_messages" : [ ... ], "functions" : [], "post_actions" : [{ "type" : "end_conversation" }] } Function Nodes : Edge Functions (Purple): Create transitions Copy Ask AI { "name" : "next_node" , "description" : "Transition to next state" } Node Functions (Orange): Perform operations Copy Ask AI { "name" : "process_data" , "handler" : process_data_handler, "description" : "Process user data" } ​ Naming Conventions Start Node : Use descriptive names (e.g., “greeting”, “welcome”) Flow Nodes : Name based on purpose (e.g., “collect_info”, “verify_data”) End Node : Conventionally named “end” Functions : Use clear, action-oriented names ​ Function Configuration Copy Ask AI { "type" : "function" , "function" : { "name" : "process_data" , "handler" : process_handler, "description" : "Process user data" , "parameters" : { ... } } } When using the Flow Editor, function handlers can be specified using the __function__: token: Copy Ask AI { "type" : "function" , "function" : { "name" : "process_data" , "handler" : "__function__:process_data" , # References function in main script "description" : "Process user data" , "parameters" : { ... } } } The handler will be looked up in your main script when the flow is executed. When function handlers are specified in the flow editor, they will be exported with the __function__: token. ​ Using the Editor ​ Creating a New Flow Start with a descriptively named Start Node Add Flow Nodes for each conversation state Connect nodes using Edge Functions Add Node Functions for operations Include an End Node ​ Import/Export Copy Ask AI # Export format { "initial_node" : "greeting" , "nodes" : { "greeting" : { "role_messages" : [ ... ], "task_messages" : [ ... ], "functions" : [ ... ], "pre_actions" : [ ... ] }, "process" : { "task_messages" : [ ... ], "functions" : [ ... ], }, "end" : { "task_messages" : [ ... ], "functions" : [], "post_actions" : [ ... ] } } } ​ Tips Use the visual preview to verify flow logic Test exported configurations Document node purposes and transitions Keep flows modular and maintainable Try the editor at flows.pipecat.ai OpenAI Audio Models and APIs Overview On this page Key Concepts Example Flows When to Use Static vs Dynamic Flows Installation Core Concepts Designing Conversation Flows Defining a Function Example Function Node Structure Messages Functions Function Configuration Node Functions Edge Functions Actions Pre-Actions Post-Actions Timing Considerations Action Types Deciding Who Speaks First Context Management Context Strategies Configuration Strategy Selection State Management LLM Provider Support OpenAI Format Anthropic Format Google (Gemini) Format Implementation Approaches Static Flows Basic Setup Example FlowConfig Dynamic Flows Example Implementation Best Practices Flow Editor Visual Design Node Types Naming Conventions Function Configuration Using the Editor Creating a New Flow Import/Export Tips Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/features_pipecat-flows_19a8d02a.txt b/features_pipecat-flows_19a8d02a.txt
new file mode 100644
index 0000000000000000000000000000000000000000..79e047d1211c685abccabfd2f82cc917f5f1ca7e
--- /dev/null
+++ b/features_pipecat-flows_19a8d02a.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/guides/features/pipecat-flows#state-management
+Title: Pipecat Flows - Pipecat
+==================================================
+
+Pipecat Flows - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Features Pipecat Flows Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Guides Fundamentals Context Management Custom FrameProcessor Detecting Idle Users Ending a Pipeline Function Calling Muting User Input Recording Audio Recording Transcripts Features Gemini Multimodal Live Metrics Noise cancellation with Krisp OpenAI Audio Models and APIs Pipecat Flows Telephony Overview Dial-in: WebRTC (Daily) Dial-in: WebRTC (Twilio + Daily) Dial-in: Twilio (Media Streams) Dialout: WebRTC (Daily) Deploying your bot Overview Deployment pattern Example: Pipecat Cloud Example: Fly.io Example: Cerebrium Example: Modal Pipecat Flows provides a framework for building structured conversations in your AI applications. It enables you to create both predefined conversation paths and dynamically generated flows while handling the complexities of state management and LLM interactions. The framework consists of: A Python module for building conversation flows with Pipecat A visual editor for designing and exporting flow configurations ​ Key Concepts Nodes : Represent conversation states with specific messages and available functions Messages : Set the role and tasks for each node Functions : Define actions and transitions (Node functions for operations, Edge functions for transitions) Actions : Execute operations during state transitions (pre/post actions) State Management : Handle conversation state and data persistence ​ Example Flows Movie Explorer (Static) A static flow demonstrating movie exploration using OpenAI. Shows real API integration with TMDB, structured data collection, and state management. Insurance Policy (Dynamic) A dynamic flow using Google Gemini that adapts policy recommendations based on user responses. Demonstrates runtime node creation and conditional paths. These examples are fully functional and can be run locally. Make sure you have the required dependencies installed and API keys configured. ​ When to Use Static vs Dynamic Flows Static Flows are ideal when: Conversation structure is known upfront Paths follow predefined patterns Flow can be fully configured in advance Example: Customer service scripts, intake forms Dynamic Flows are better when: Paths depend on external data Flow structure needs runtime modification Complex decision trees are involved Example: Personalized recommendations, adaptive workflows ​ Installation If you’re already using Pipecat: Copy Ask AI pip install pipecat-ai-flows If you’re starting fresh: Copy Ask AI # Basic installation pip install pipecat-ai-flows # Install Pipecat with specific LLM provider options: pip install "pipecat-ai[daily,openai,deepgram]" # For OpenAI pip install "pipecat-ai[daily,anthropic,deepgram]" # For Anthropic pip install "pipecat-ai[daily,google,deepgram]" # For Google 💡 Want to design your flows visually? Try the online Flow Editor ​ Core Concepts ​ Designing Conversation Flows Functions in Pipecat Flows serve two key purposes: Processing data (likely by interfacing with external systems and APIs) Advancing the conversation to the next node Each function can do one or both. LLMs decide when to run each function, via their function calling (or tool calling) mechanism. ​ Defining a Function A function is expected to return a (result, next_node) tuple. More precisely, it’s expected to return: Copy Ask AI # (result, next_node) Tuple[Optional[FlowResult], Optional[Union[NodeConfig, str ]]] If the function processes data, it should return a non- None value for the first element of the tuple. This value should be a FlowResult or subclass. If the function advances the conversation to the next node, it should return a non- None value for the second element of the tuple. This value can be either: A NodeConfig defining the next node (for dynamic flows) A string identifying the next node (for static flows) ​ Example Function Copy Ask AI from pipecat_flows import FlowArgs, FlowManager, FlowResult, NodeConfig async def check_availability ( args : FlowArgs, flow_manager : FlowManager) -> tuple[FlowResult, NodeConfig]: # Read arguments date = args[ "date" ] time = args[ "time" ] # Read previously-stored data party_size = flow_manager.state.get( "party_size" ) # Use flow_manager for immediate user feedback await flow_manager.task.queue_frame(TTSSpeakFrame( "Checking our reservation system..." )) # Store data in flow state for later use flow_manager.state[ "requested_date" ] = date # Interface with reservation system is_available = await reservation_system.check_availability(date, time, party_size) # Assemble result result = { "status" : "success" , "available" : available } # Decide which node to go to next if is_available: next_node = create_confirmation_node() else : next_node = create_no_availability_node() # Return both result and next node return result, next_node ​ Node Structure Each node in your flow represents a conversation state and consists of three main components: ​ Messages Nodes use two types of messages to control the conversation: Role Messages : Define the bot’s personality or role (optional) Copy Ask AI "role_messages" : [ { "role" : "system" , "content" : "You are a friendly pizza ordering assistant. Keep responses casual and upbeat." } ] Task Messages : Define what the bot should do in the current node Copy Ask AI "task_messages" : [ { "role" : "system" , "content" : "Ask the customer which pizza size they'd like: small, medium, or large." } ] Role messages are typically defined in your initial node and inherited by subsequent nodes, while task messages are specific to each node’s purpose. ​ Functions Functions in Pipecat Flows can: Process data Specify node transitions Do both This leads to two conceptual types of functions: Node functions , which only process data. Edge functions , which also (or only) transition to the next node. The function itself ( which you can read more about here ) is usually wrapped in a function configuration, which also contains some metadata about the function. ​ Function Configuration Pipecat Flows supports three ways of specifying function configuration: Provider-specific dictionary format Copy Ask AI # Dictionary format { "type" : "function" , "function" : { "name" : "select_size" , "handler" : select_size, "description" : "Select pizza size" , "parameters" : { "type" : "object" , "properties" : { "size" : { "type" : "string" , "enum" : [ "small" , "medium" , "large" ]} } }, } } FlowsFunctionSchema Copy Ask AI # Using FlowsFunctionSchema from pipecat_flows import FlowsFunctionSchema size_function = FlowsFunctionSchema( name = "select_size" , description = "Select pizza size" , properties = { "size" : { "type" : "string" , "enum" : [ "small" , "medium" , "large" ]} }, required = [ "size" ], handler = select_size ) # Use in node configuration node_config = { "task_messages" : [ ... ], "functions" : [size_function] } The FlowsFunctionSchema approach provides some advantages over the provider-specific dictionary format: Consistent structure across LLM providers Simplified parameter definition Cleaner, more readable code Both dictionary and FlowsFunctionSchema approaches are fully supported. FlowsFunctionSchema is recommended for new projects as it provides better type checking and a provider-independent format. Direct function usage (auto-configuration) This approach lets you bypass specifying a standalone function configuration. Instead, relevant function metadata is automatically extracted from the function’s signature and docstring: name description properties (including individual property description s) required Note that the function signature is a bit different when using direct functions. The first parameter is the FlowManager , followed by any others necessary for the function. Copy Ask AI from pipecat_flows import FlowManager, FlowResult async def select_pizza_order ( flow_manager : FlowManager, size : str , pizza_type : str , additional_toppings : list[ str ] = [], ) -> tuple[FlowResult, str ]: """ Record the pizza order details. Args: size (str): Size of the pizza. Must be one of "small", "medium", or "large". pizza_type (str): Type of pizza. Must be one of "pepperoni", "cheese", "supreme", or "vegetarian". additional_toppings (list[str]): List of additional toppings. Defaults to empty list. """ ... # Use in node configuration node_config = { "task_messages" : [ ... ], "functions" : [select_pizza_order] } ​ Node Functions Functions that process data within a single conversational state, without switching nodes. When called, they: Execute their handler to do the data processing (typically by interfacing with an external system or API) Trigger an immediate LLM completion with the result Copy Ask AI from pipecat_flows import FlowArgs, FlowManager, FlowResult async def select_size ( args : FlowArgs, flow_manager : FlowManager) -> tuple[FlowResult, None ]: """Process pizza size selection.""" size = args[ "size" ] await ordering_system.record_size_selection(size) return { "status" : "success" , "size" : size }, None # Function configuration { "type" : "function" , "function" : { "name" : "select_size" , "handler" : select_size, "description" : "Select pizza size" , "parameters" : { "type" : "object" , "properties" : { "size" : { "type" : "string" , "enum" : [ "small" , "medium" , "large" ]} } }, } } ​ Edge Functions Functions that specify a transition between nodes (optionally processing data first). When called, they: Execute their handler to do any data processing (optional) and determine the next node Add the function result to the LLM context Trigger LLM completion after both the function result and the next node’s messages are in the context Copy Ask AI from pipecat_flows import FlowArgs, FlowManager, FlowResult async def select_size ( args : FlowArgs, flow_manager : FlowManager) -> tuple[FlowResult, NodeConfig]: """Process pizza size selection.""" size = args[ "size" ] await ordering_system.record_size_selection(size) result = { "status" : "success" , "size" : size } next_node = create_confirmation_node() return result, next_node # Function configuration { "type" : "function" , "function" : { "name" : "select_size" , "handler" : select_size, "description" : "Select pizza size" , "parameters" : { "type" : "object" , "properties" : { "size" : { "type" : "string" , "enum" : [ "small" , "medium" , "large" ]} } }, } } ​ Actions Actions are operations that execute as part of the lifecycle of a node, with two distinct timing options: Pre-actions: execute when entering the node, before the LLM completion Post-actions: execute after the LLM completion ​ Pre-Actions Execute when entering the node, before LLM inference. Useful for: Providing immediate feedback while waiting for LLM responses Bridging gaps during longer function calls Setting up state or context Copy Ask AI "pre_actions" : [ { "type" : "tts_say" , "text" : "Hold on a moment..." # Immediate feedback during processing } ], Note that when the node is configured with respond_immediately: False , the pre_actions still run when entering the node, which may be well before LLM inference, depending on how long the user takes to speak first. Avoid mixing tts_say actions with chat completions as this may result in a conversation flow that feels unnatural. tts_say are best used as filler words when the LLM will take time to generate an completion. ​ Post-Actions Execute after LLM inference completes. Useful for: Cleanup operations State finalization Ensuring proper sequence of operations Copy Ask AI "post_actions" : [ { "type" : "end_conversation" # Ensures TTS completes before ending } ] Note that when the node is configured with respond_immediately: False , the post_actions still only run after the first LLM inference, which may be a while depending on how long the user takes to speak first. ​ Timing Considerations Pre-actions : Execute immediately, before any LLM processing begins LLM Inference : Processes the node’s messages and functions Post-actions : Execute after LLM processing and TTS completion For example, when using end_conversation as a post-action, the sequence is: LLM generates response TTS speaks the response End conversation action executes This ordering ensures proper completion of all operations. ​ Action Types Flows comes equipped with pre-canned actions and you can also define your own action behavior. See the reference docs for more information. ​ Deciding Who Speaks First For each node in the conversation, you can decide whether the LLM should respond immediately upon entering the node (the default behavior) or whether the LLM should wait for the user to speak first before responding. You do this using the respond_immediately field. respond_immediately=False may be particularly useful in the very first node, especially in outbound-calling cases where the user has to first answer the phone to trigger the conversation. Copy Ask AI NodeConfig( task_messages = [ { "role" : "system" , "content" : "Warmly greet the customer and ask how many people are in their party. This is your only job for now; if the customer asks for something else, politely remind them you can't do it." , } ], respond_immediately = False , # ... other fields ) Keep in mind that if you specify respond_immediately=False , the user may not be aware of the conversational task at hand when entering the node (the bot hasn’t told them yet). While it’s always important to have guardrails in your node messages to keep the conversation on topic, letting the user speak first makes it even more so. ​ Context Management Pipecat Flows provides three strategies for managing conversation context during node transitions: ​ Context Strategies APPEND (default): Adds new messages to the existing context, maintaining the full conversation history RESET : Clears the context and starts fresh with the new node’s messages RESET_WITH_SUMMARY : Resets the context but includes an AI-generated summary of the previous conversation ​ Configuration Context strategies can be configured globally or per-node: Copy Ask AI from pipecat_flows import ContextStrategy, ContextStrategyConfig # Global strategy configuration flow_manager = FlowManager( task = task, llm = llm, context_aggregator = context_aggregator, context_strategy = ContextStrategyConfig( strategy = ContextStrategy. RESET_WITH_SUMMARY , summary_prompt = "Summarize the key points discussed so far, focusing on decisions made and important information collected." ) ) # Per-node strategy configuration node_config = { "task_messages" : [ ... ], "functions" : [ ... ], "context_strategy" : ContextStrategyConfig( strategy = ContextStrategy. RESET_WITH_SUMMARY , summary_prompt = "Provide a concise summary of the customer's order details and preferences." ) } ​ Strategy Selection Choose your strategy based on your conversation needs: Use APPEND when full conversation history is important Use RESET when previous context might confuse the current node’s purpose Use RESET_WITH_SUMMARY for long conversations where key points need to be preserved When using RESET_WITH_SUMMARY, if summary generation fails or times out, the system automatically falls back to RESET strategy for resilience. ​ State Management The state variable in FlowManager is a shared dictionary that persists throughout the conversation. Think of it as a conversation memory that lets you: Store user information Track conversation progress Share data between nodes Inform decision-making Here’s a practical example of a pizza ordering flow: Copy Ask AI # Store user choices as they're made async def select_size ( args : FlowArgs) -> tuple[FlowResult, str ]: """Handle pizza size selection.""" size = args[ "size" ] # Initialize order in state if it doesn't exist if "order" not in flow_manager.state: flow_manager.state[ "order" ] = {} # Store the selection flow_manager.state[ "order" ][ "size" ] = size return { "status" : "success" , "size" : size}, "toppings" async def select_toppings ( args : FlowArgs) -> tuple[FlowResult, str ]: """Handle topping selection.""" topping = args[ "topping" ] # Get existing order and toppings order = flow_manager.state.get( "order" , {}) toppings = order.get( "toppings" , []) # Add new topping toppings.append(topping) order[ "toppings" ] = toppings flow_manager.state[ "order" ] = order return { "status" : "success" , "toppings" : toppings}, "finalize" async def finalize_order ( args : FlowArgs) -> tuple[FlowResult, str ]: """Process the complete order.""" order = flow_manager.state.get( "order" , {}) # Validate order has required information if "size" not in order: return { "status" : "error" , "error" : "No size selected" } # Calculate price based on stored selections size = order[ "size" ] toppings = order.get( "toppings" , []) price = calculate_price(size, len (toppings)) return { "status" : "success" , "summary" : f "Ordered: { size } pizza with { ', ' .join(toppings) } " , "price" : price }, "end" In this example: select_size initializes the order and stores the size select_toppings builds a list of toppings finalize_order uses the stored information to process the complete order The state variable makes it easy to: Build up information across multiple interactions Access previous choices when needed Validate the complete order Calculate final results This is particularly useful when information needs to be collected across multiple conversation turns or when later decisions depend on earlier choices. ​ LLM Provider Support Pipecat Flows automatically handles format differences between LLM providers: ​ OpenAI Format Copy Ask AI "functions" : [{ "type" : "function" , "function" : { "name" : "function_name" , "handler" : select_size, "description" : "description" , "parameters" : { ... } } }] ​ Anthropic Format Copy Ask AI "functions" : [{ "name" : "function_name" , "handler" : select_size, "description" : "description" , "input_schema" : { ... } }] ​ Google (Gemini) Format Copy Ask AI "functions" : [{ "function_declarations" : [{ "name" : "function_name" , "handler" : select_size, "description" : "description" , "parameters" : { ... } }] }] You don’t need to handle these differences manually - Pipecat Flows adapts your configuration to the correct format based on your LLM provider. ​ Implementation Approaches ​ Static Flows Static flows use a configuration-driven approach where the entire conversation structure is defined upfront. ​ Basic Setup Copy Ask AI from pipecat_flows import FlowManager # Define flow configuration flow_config = { "initial_node" : "greeting" , "nodes" : { "greeting" : { "role_messages" : [ ... ], "task_messages" : [ ... ], "functions" : [ ... ] } } } # Initialize flow manager with static configuration flow_manager = FlowManager( task = task, llm = llm, context_aggregator = context_aggregator, flow_config = flow_config ) @transport.event_handler ( "on_first_participant_joined" ) async def on_first_participant_joined ( transport , participant ): await transport.capture_participant_transcription(participant[ "id" ]) await flow_manager.initialize() ​ Example FlowConfig Copy Ask AI flow_config = { "initial_node" : "start" , "nodes" : { "start" : { "role_messages" : [ { "role" : "system" , "content" : "You are an order-taking assistant. You must ALWAYS use the available functions to progress the conversation. This is a phone conversation and your responses will be converted to audio. Keep the conversation friendly, casual, and polite. Avoid outputting special characters and emojis." , } ], "task_messages" : [ { "role" : "system" , "content" : "You are an order-taking assistant. Ask if they want pizza or sushi." } ], "functions" : [ { "type" : "function" , "function" : { "name" : "choose_pizza" , "handler" : choose_pizza, # Returns [None, "pizza_order"] "description" : "User wants pizza" , "parameters" : { "type" : "object" , "properties" : {}} } } ] }, "pizza_order" : { "task_messages" : [ ... ], "functions" : [ { "type" : "function" , "function" : { "name" : "select_size" , "handler" : select_size, # Returns [FlowResult, "toppings"] "description" : "Select pizza size" , "parameters" : { "type" : "object" , "properties" : { "size" : { "type" : "string" , "enum" : [ "small" , "medium" , "large" ]} } } } } ] } } } ​ Dynamic Flows Dynamic flows create and modify conversation paths at runtime based on data or business logic. ​ Example Implementation Here’s a complete example of a dynamic insurance quote flow: Copy Ask AI from pipecat_flows import FlowManager, FlowArgs, FlowResult # Define handlers and transitions async def collect_age ( args : FlowArgs, flow_manager : FlowManager) -> tuple[AgeResult, NodeConfig]: """Process age collection.""" age = args[ "age" ] # Assemble result result = AgeResult( status = "success" , age = age) # Decide which node to go to next if age < 25 : await flow_manager.set_node_from_config(create_young_adult_node()) else : await flow_manager.set_node_from_config(create_standard_node()) return result, age # Node creation functions def create_initial_node () -> NodeConfig: """Create initial age collection node.""" return { "name" : "initial" , "role_messages" : [ { "role" : "system" , "content" : "You are an insurance quote assistant." } ], "task_messages" : [ { "role" : "system" , "content" : "Ask for the customer's age." } ], "functions" : [ { "type" : "function" , "function" : { "name" : "collect_age" , "handler" : collect_age, "description" : "Collect customer age" , "parameters" : { "type" : "object" , "properties" : { "age" : { "type" : "integer" } } } } } ] } def create_young_adult_node () -> Dict[ str , Any]: """Create node for young adult quotes.""" return { "name" : "young_adult" , "task_messages" : [ { "role" : "system" , "content" : "Explain our special young adult coverage options." } ], "functions" : [ ... ] # Additional quote-specific functions } def create_standard_node () -> Dict[ str , Any]: """Create node for standard quotes.""" return { "name" : "standard" , "task_messages" : [ { "role" : "system" , "content" : "Present our standard coverage options." } ], "functions" : [ ... ] # Additional quote-specific functions } # Initialize flow manager flow_manager = FlowManager( task = task, llm = llm, context_aggregator = context_aggregator, ) @transport.event_handler ( "on_first_participant_joined" ) async def on_first_participant_joined ( transport , participant ): await transport.capture_participant_transcription(participant[ "id" ]) await flow_manager.initialize(create_initial_node()) ​ Best Practices Store shared data in flow_manager.state Create separate functions for node creation ​ Flow Editor The Pipecat Flow Editor provides a visual interface for creating and managing conversation flows. It offers a node-based interface that makes it easier to design, visualize, and modify your flows. ​ Visual Design ​ Node Types Start Node (Green): Entry point of your flow Copy Ask AI "greeting" : { "role_messages" : [ ... ], "task_messages" : [ ... ], "functions" : [ ... ] } Flow Nodes (Blue): Intermediate states Copy Ask AI "collect_info" : { "task_messages" : [ ... ], "functions" : [ ... ], "pre_actions" : [ ... ] } End Node (Red): Final state Copy Ask AI "end" : { "task_messages" : [ ... ], "functions" : [], "post_actions" : [{ "type" : "end_conversation" }] } Function Nodes : Edge Functions (Purple): Create transitions Copy Ask AI { "name" : "next_node" , "description" : "Transition to next state" } Node Functions (Orange): Perform operations Copy Ask AI { "name" : "process_data" , "handler" : process_data_handler, "description" : "Process user data" } ​ Naming Conventions Start Node : Use descriptive names (e.g., “greeting”, “welcome”) Flow Nodes : Name based on purpose (e.g., “collect_info”, “verify_data”) End Node : Conventionally named “end” Functions : Use clear, action-oriented names ​ Function Configuration Copy Ask AI { "type" : "function" , "function" : { "name" : "process_data" , "handler" : process_handler, "description" : "Process user data" , "parameters" : { ... } } } When using the Flow Editor, function handlers can be specified using the __function__: token: Copy Ask AI { "type" : "function" , "function" : { "name" : "process_data" , "handler" : "__function__:process_data" , # References function in main script "description" : "Process user data" , "parameters" : { ... } } } The handler will be looked up in your main script when the flow is executed. When function handlers are specified in the flow editor, they will be exported with the __function__: token. ​ Using the Editor ​ Creating a New Flow Start with a descriptively named Start Node Add Flow Nodes for each conversation state Connect nodes using Edge Functions Add Node Functions for operations Include an End Node ​ Import/Export Copy Ask AI # Export format { "initial_node" : "greeting" , "nodes" : { "greeting" : { "role_messages" : [ ... ], "task_messages" : [ ... ], "functions" : [ ... ], "pre_actions" : [ ... ] }, "process" : { "task_messages" : [ ... ], "functions" : [ ... ], }, "end" : { "task_messages" : [ ... ], "functions" : [], "post_actions" : [ ... ] } } } ​ Tips Use the visual preview to verify flow logic Test exported configurations Document node purposes and transitions Keep flows modular and maintainable Try the editor at flows.pipecat.ai OpenAI Audio Models and APIs Overview On this page Key Concepts Example Flows When to Use Static vs Dynamic Flows Installation Core Concepts Designing Conversation Flows Defining a Function Example Function Node Structure Messages Functions Function Configuration Node Functions Edge Functions Actions Pre-Actions Post-Actions Timing Considerations Action Types Deciding Who Speaks First Context Management Context Strategies Configuration Strategy Selection State Management LLM Provider Support OpenAI Format Anthropic Format Google (Gemini) Format Implementation Approaches Static Flows Basic Setup Example FlowConfig Dynamic Flows Example Implementation Best Practices Flow Editor Visual Design Node Types Naming Conventions Function Configuration Using the Editor Creating a New Flow Import/Export Tips Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/features_pipecat-flows_1d10f654.txt b/features_pipecat-flows_1d10f654.txt
new file mode 100644
index 0000000000000000000000000000000000000000..dad3a8283fd993111a2f3adf6f494f20c3d2feaa
--- /dev/null
+++ b/features_pipecat-flows_1d10f654.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/guides/features/pipecat-flows#edge-functions
+Title: Pipecat Flows - Pipecat
+==================================================
+
+Pipecat Flows - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Features Pipecat Flows Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Guides Fundamentals Context Management Custom FrameProcessor Detecting Idle Users Ending a Pipeline Function Calling Muting User Input Recording Audio Recording Transcripts Features Gemini Multimodal Live Metrics Noise cancellation with Krisp OpenAI Audio Models and APIs Pipecat Flows Telephony Overview Dial-in: WebRTC (Daily) Dial-in: WebRTC (Twilio + Daily) Dial-in: Twilio (Media Streams) Dialout: WebRTC (Daily) Deploying your bot Overview Deployment pattern Example: Pipecat Cloud Example: Fly.io Example: Cerebrium Example: Modal Pipecat Flows provides a framework for building structured conversations in your AI applications. It enables you to create both predefined conversation paths and dynamically generated flows while handling the complexities of state management and LLM interactions. The framework consists of: A Python module for building conversation flows with Pipecat A visual editor for designing and exporting flow configurations ​ Key Concepts Nodes : Represent conversation states with specific messages and available functions Messages : Set the role and tasks for each node Functions : Define actions and transitions (Node functions for operations, Edge functions for transitions) Actions : Execute operations during state transitions (pre/post actions) State Management : Handle conversation state and data persistence ​ Example Flows Movie Explorer (Static) A static flow demonstrating movie exploration using OpenAI. Shows real API integration with TMDB, structured data collection, and state management. Insurance Policy (Dynamic) A dynamic flow using Google Gemini that adapts policy recommendations based on user responses. Demonstrates runtime node creation and conditional paths. These examples are fully functional and can be run locally. Make sure you have the required dependencies installed and API keys configured. ​ When to Use Static vs Dynamic Flows Static Flows are ideal when: Conversation structure is known upfront Paths follow predefined patterns Flow can be fully configured in advance Example: Customer service scripts, intake forms Dynamic Flows are better when: Paths depend on external data Flow structure needs runtime modification Complex decision trees are involved Example: Personalized recommendations, adaptive workflows ​ Installation If you’re already using Pipecat: Copy Ask AI pip install pipecat-ai-flows If you’re starting fresh: Copy Ask AI # Basic installation pip install pipecat-ai-flows # Install Pipecat with specific LLM provider options: pip install "pipecat-ai[daily,openai,deepgram]" # For OpenAI pip install "pipecat-ai[daily,anthropic,deepgram]" # For Anthropic pip install "pipecat-ai[daily,google,deepgram]" # For Google 💡 Want to design your flows visually? Try the online Flow Editor ​ Core Concepts ​ Designing Conversation Flows Functions in Pipecat Flows serve two key purposes: Processing data (likely by interfacing with external systems and APIs) Advancing the conversation to the next node Each function can do one or both. LLMs decide when to run each function, via their function calling (or tool calling) mechanism. ​ Defining a Function A function is expected to return a (result, next_node) tuple. More precisely, it’s expected to return: Copy Ask AI # (result, next_node) Tuple[Optional[FlowResult], Optional[Union[NodeConfig, str ]]] If the function processes data, it should return a non- None value for the first element of the tuple. This value should be a FlowResult or subclass. If the function advances the conversation to the next node, it should return a non- None value for the second element of the tuple. This value can be either: A NodeConfig defining the next node (for dynamic flows) A string identifying the next node (for static flows) ​ Example Function Copy Ask AI from pipecat_flows import FlowArgs, FlowManager, FlowResult, NodeConfig async def check_availability ( args : FlowArgs, flow_manager : FlowManager) -> tuple[FlowResult, NodeConfig]: # Read arguments date = args[ "date" ] time = args[ "time" ] # Read previously-stored data party_size = flow_manager.state.get( "party_size" ) # Use flow_manager for immediate user feedback await flow_manager.task.queue_frame(TTSSpeakFrame( "Checking our reservation system..." )) # Store data in flow state for later use flow_manager.state[ "requested_date" ] = date # Interface with reservation system is_available = await reservation_system.check_availability(date, time, party_size) # Assemble result result = { "status" : "success" , "available" : available } # Decide which node to go to next if is_available: next_node = create_confirmation_node() else : next_node = create_no_availability_node() # Return both result and next node return result, next_node ​ Node Structure Each node in your flow represents a conversation state and consists of three main components: ​ Messages Nodes use two types of messages to control the conversation: Role Messages : Define the bot’s personality or role (optional) Copy Ask AI "role_messages" : [ { "role" : "system" , "content" : "You are a friendly pizza ordering assistant. Keep responses casual and upbeat." } ] Task Messages : Define what the bot should do in the current node Copy Ask AI "task_messages" : [ { "role" : "system" , "content" : "Ask the customer which pizza size they'd like: small, medium, or large." } ] Role messages are typically defined in your initial node and inherited by subsequent nodes, while task messages are specific to each node’s purpose. ​ Functions Functions in Pipecat Flows can: Process data Specify node transitions Do both This leads to two conceptual types of functions: Node functions , which only process data. Edge functions , which also (or only) transition to the next node. The function itself ( which you can read more about here ) is usually wrapped in a function configuration, which also contains some metadata about the function. ​ Function Configuration Pipecat Flows supports three ways of specifying function configuration: Provider-specific dictionary format Copy Ask AI # Dictionary format { "type" : "function" , "function" : { "name" : "select_size" , "handler" : select_size, "description" : "Select pizza size" , "parameters" : { "type" : "object" , "properties" : { "size" : { "type" : "string" , "enum" : [ "small" , "medium" , "large" ]} } }, } } FlowsFunctionSchema Copy Ask AI # Using FlowsFunctionSchema from pipecat_flows import FlowsFunctionSchema size_function = FlowsFunctionSchema( name = "select_size" , description = "Select pizza size" , properties = { "size" : { "type" : "string" , "enum" : [ "small" , "medium" , "large" ]} }, required = [ "size" ], handler = select_size ) # Use in node configuration node_config = { "task_messages" : [ ... ], "functions" : [size_function] } The FlowsFunctionSchema approach provides some advantages over the provider-specific dictionary format: Consistent structure across LLM providers Simplified parameter definition Cleaner, more readable code Both dictionary and FlowsFunctionSchema approaches are fully supported. FlowsFunctionSchema is recommended for new projects as it provides better type checking and a provider-independent format. Direct function usage (auto-configuration) This approach lets you bypass specifying a standalone function configuration. Instead, relevant function metadata is automatically extracted from the function’s signature and docstring: name description properties (including individual property description s) required Note that the function signature is a bit different when using direct functions. The first parameter is the FlowManager , followed by any others necessary for the function. Copy Ask AI from pipecat_flows import FlowManager, FlowResult async def select_pizza_order ( flow_manager : FlowManager, size : str , pizza_type : str , additional_toppings : list[ str ] = [], ) -> tuple[FlowResult, str ]: """ Record the pizza order details. Args: size (str): Size of the pizza. Must be one of "small", "medium", or "large". pizza_type (str): Type of pizza. Must be one of "pepperoni", "cheese", "supreme", or "vegetarian". additional_toppings (list[str]): List of additional toppings. Defaults to empty list. """ ... # Use in node configuration node_config = { "task_messages" : [ ... ], "functions" : [select_pizza_order] } ​ Node Functions Functions that process data within a single conversational state, without switching nodes. When called, they: Execute their handler to do the data processing (typically by interfacing with an external system or API) Trigger an immediate LLM completion with the result Copy Ask AI from pipecat_flows import FlowArgs, FlowManager, FlowResult async def select_size ( args : FlowArgs, flow_manager : FlowManager) -> tuple[FlowResult, None ]: """Process pizza size selection.""" size = args[ "size" ] await ordering_system.record_size_selection(size) return { "status" : "success" , "size" : size }, None # Function configuration { "type" : "function" , "function" : { "name" : "select_size" , "handler" : select_size, "description" : "Select pizza size" , "parameters" : { "type" : "object" , "properties" : { "size" : { "type" : "string" , "enum" : [ "small" , "medium" , "large" ]} } }, } } ​ Edge Functions Functions that specify a transition between nodes (optionally processing data first). When called, they: Execute their handler to do any data processing (optional) and determine the next node Add the function result to the LLM context Trigger LLM completion after both the function result and the next node’s messages are in the context Copy Ask AI from pipecat_flows import FlowArgs, FlowManager, FlowResult async def select_size ( args : FlowArgs, flow_manager : FlowManager) -> tuple[FlowResult, NodeConfig]: """Process pizza size selection.""" size = args[ "size" ] await ordering_system.record_size_selection(size) result = { "status" : "success" , "size" : size } next_node = create_confirmation_node() return result, next_node # Function configuration { "type" : "function" , "function" : { "name" : "select_size" , "handler" : select_size, "description" : "Select pizza size" , "parameters" : { "type" : "object" , "properties" : { "size" : { "type" : "string" , "enum" : [ "small" , "medium" , "large" ]} } }, } } ​ Actions Actions are operations that execute as part of the lifecycle of a node, with two distinct timing options: Pre-actions: execute when entering the node, before the LLM completion Post-actions: execute after the LLM completion ​ Pre-Actions Execute when entering the node, before LLM inference. Useful for: Providing immediate feedback while waiting for LLM responses Bridging gaps during longer function calls Setting up state or context Copy Ask AI "pre_actions" : [ { "type" : "tts_say" , "text" : "Hold on a moment..." # Immediate feedback during processing } ], Note that when the node is configured with respond_immediately: False , the pre_actions still run when entering the node, which may be well before LLM inference, depending on how long the user takes to speak first. Avoid mixing tts_say actions with chat completions as this may result in a conversation flow that feels unnatural. tts_say are best used as filler words when the LLM will take time to generate an completion. ​ Post-Actions Execute after LLM inference completes. Useful for: Cleanup operations State finalization Ensuring proper sequence of operations Copy Ask AI "post_actions" : [ { "type" : "end_conversation" # Ensures TTS completes before ending } ] Note that when the node is configured with respond_immediately: False , the post_actions still only run after the first LLM inference, which may be a while depending on how long the user takes to speak first. ​ Timing Considerations Pre-actions : Execute immediately, before any LLM processing begins LLM Inference : Processes the node’s messages and functions Post-actions : Execute after LLM processing and TTS completion For example, when using end_conversation as a post-action, the sequence is: LLM generates response TTS speaks the response End conversation action executes This ordering ensures proper completion of all operations. ​ Action Types Flows comes equipped with pre-canned actions and you can also define your own action behavior. See the reference docs for more information. ​ Deciding Who Speaks First For each node in the conversation, you can decide whether the LLM should respond immediately upon entering the node (the default behavior) or whether the LLM should wait for the user to speak first before responding. You do this using the respond_immediately field. respond_immediately=False may be particularly useful in the very first node, especially in outbound-calling cases where the user has to first answer the phone to trigger the conversation. Copy Ask AI NodeConfig( task_messages = [ { "role" : "system" , "content" : "Warmly greet the customer and ask how many people are in their party. This is your only job for now; if the customer asks for something else, politely remind them you can't do it." , } ], respond_immediately = False , # ... other fields ) Keep in mind that if you specify respond_immediately=False , the user may not be aware of the conversational task at hand when entering the node (the bot hasn’t told them yet). While it’s always important to have guardrails in your node messages to keep the conversation on topic, letting the user speak first makes it even more so. ​ Context Management Pipecat Flows provides three strategies for managing conversation context during node transitions: ​ Context Strategies APPEND (default): Adds new messages to the existing context, maintaining the full conversation history RESET : Clears the context and starts fresh with the new node’s messages RESET_WITH_SUMMARY : Resets the context but includes an AI-generated summary of the previous conversation ​ Configuration Context strategies can be configured globally or per-node: Copy Ask AI from pipecat_flows import ContextStrategy, ContextStrategyConfig # Global strategy configuration flow_manager = FlowManager( task = task, llm = llm, context_aggregator = context_aggregator, context_strategy = ContextStrategyConfig( strategy = ContextStrategy. RESET_WITH_SUMMARY , summary_prompt = "Summarize the key points discussed so far, focusing on decisions made and important information collected." ) ) # Per-node strategy configuration node_config = { "task_messages" : [ ... ], "functions" : [ ... ], "context_strategy" : ContextStrategyConfig( strategy = ContextStrategy. RESET_WITH_SUMMARY , summary_prompt = "Provide a concise summary of the customer's order details and preferences." ) } ​ Strategy Selection Choose your strategy based on your conversation needs: Use APPEND when full conversation history is important Use RESET when previous context might confuse the current node’s purpose Use RESET_WITH_SUMMARY for long conversations where key points need to be preserved When using RESET_WITH_SUMMARY, if summary generation fails or times out, the system automatically falls back to RESET strategy for resilience. ​ State Management The state variable in FlowManager is a shared dictionary that persists throughout the conversation. Think of it as a conversation memory that lets you: Store user information Track conversation progress Share data between nodes Inform decision-making Here’s a practical example of a pizza ordering flow: Copy Ask AI # Store user choices as they're made async def select_size ( args : FlowArgs) -> tuple[FlowResult, str ]: """Handle pizza size selection.""" size = args[ "size" ] # Initialize order in state if it doesn't exist if "order" not in flow_manager.state: flow_manager.state[ "order" ] = {} # Store the selection flow_manager.state[ "order" ][ "size" ] = size return { "status" : "success" , "size" : size}, "toppings" async def select_toppings ( args : FlowArgs) -> tuple[FlowResult, str ]: """Handle topping selection.""" topping = args[ "topping" ] # Get existing order and toppings order = flow_manager.state.get( "order" , {}) toppings = order.get( "toppings" , []) # Add new topping toppings.append(topping) order[ "toppings" ] = toppings flow_manager.state[ "order" ] = order return { "status" : "success" , "toppings" : toppings}, "finalize" async def finalize_order ( args : FlowArgs) -> tuple[FlowResult, str ]: """Process the complete order.""" order = flow_manager.state.get( "order" , {}) # Validate order has required information if "size" not in order: return { "status" : "error" , "error" : "No size selected" } # Calculate price based on stored selections size = order[ "size" ] toppings = order.get( "toppings" , []) price = calculate_price(size, len (toppings)) return { "status" : "success" , "summary" : f "Ordered: { size } pizza with { ', ' .join(toppings) } " , "price" : price }, "end" In this example: select_size initializes the order and stores the size select_toppings builds a list of toppings finalize_order uses the stored information to process the complete order The state variable makes it easy to: Build up information across multiple interactions Access previous choices when needed Validate the complete order Calculate final results This is particularly useful when information needs to be collected across multiple conversation turns or when later decisions depend on earlier choices. ​ LLM Provider Support Pipecat Flows automatically handles format differences between LLM providers: ​ OpenAI Format Copy Ask AI "functions" : [{ "type" : "function" , "function" : { "name" : "function_name" , "handler" : select_size, "description" : "description" , "parameters" : { ... } } }] ​ Anthropic Format Copy Ask AI "functions" : [{ "name" : "function_name" , "handler" : select_size, "description" : "description" , "input_schema" : { ... } }] ​ Google (Gemini) Format Copy Ask AI "functions" : [{ "function_declarations" : [{ "name" : "function_name" , "handler" : select_size, "description" : "description" , "parameters" : { ... } }] }] You don’t need to handle these differences manually - Pipecat Flows adapts your configuration to the correct format based on your LLM provider. ​ Implementation Approaches ​ Static Flows Static flows use a configuration-driven approach where the entire conversation structure is defined upfront. ​ Basic Setup Copy Ask AI from pipecat_flows import FlowManager # Define flow configuration flow_config = { "initial_node" : "greeting" , "nodes" : { "greeting" : { "role_messages" : [ ... ], "task_messages" : [ ... ], "functions" : [ ... ] } } } # Initialize flow manager with static configuration flow_manager = FlowManager( task = task, llm = llm, context_aggregator = context_aggregator, flow_config = flow_config ) @transport.event_handler ( "on_first_participant_joined" ) async def on_first_participant_joined ( transport , participant ): await transport.capture_participant_transcription(participant[ "id" ]) await flow_manager.initialize() ​ Example FlowConfig Copy Ask AI flow_config = { "initial_node" : "start" , "nodes" : { "start" : { "role_messages" : [ { "role" : "system" , "content" : "You are an order-taking assistant. You must ALWAYS use the available functions to progress the conversation. This is a phone conversation and your responses will be converted to audio. Keep the conversation friendly, casual, and polite. Avoid outputting special characters and emojis." , } ], "task_messages" : [ { "role" : "system" , "content" : "You are an order-taking assistant. Ask if they want pizza or sushi." } ], "functions" : [ { "type" : "function" , "function" : { "name" : "choose_pizza" , "handler" : choose_pizza, # Returns [None, "pizza_order"] "description" : "User wants pizza" , "parameters" : { "type" : "object" , "properties" : {}} } } ] }, "pizza_order" : { "task_messages" : [ ... ], "functions" : [ { "type" : "function" , "function" : { "name" : "select_size" , "handler" : select_size, # Returns [FlowResult, "toppings"] "description" : "Select pizza size" , "parameters" : { "type" : "object" , "properties" : { "size" : { "type" : "string" , "enum" : [ "small" , "medium" , "large" ]} } } } } ] } } } ​ Dynamic Flows Dynamic flows create and modify conversation paths at runtime based on data or business logic. ​ Example Implementation Here’s a complete example of a dynamic insurance quote flow: Copy Ask AI from pipecat_flows import FlowManager, FlowArgs, FlowResult # Define handlers and transitions async def collect_age ( args : FlowArgs, flow_manager : FlowManager) -> tuple[AgeResult, NodeConfig]: """Process age collection.""" age = args[ "age" ] # Assemble result result = AgeResult( status = "success" , age = age) # Decide which node to go to next if age < 25 : await flow_manager.set_node_from_config(create_young_adult_node()) else : await flow_manager.set_node_from_config(create_standard_node()) return result, age # Node creation functions def create_initial_node () -> NodeConfig: """Create initial age collection node.""" return { "name" : "initial" , "role_messages" : [ { "role" : "system" , "content" : "You are an insurance quote assistant." } ], "task_messages" : [ { "role" : "system" , "content" : "Ask for the customer's age." } ], "functions" : [ { "type" : "function" , "function" : { "name" : "collect_age" , "handler" : collect_age, "description" : "Collect customer age" , "parameters" : { "type" : "object" , "properties" : { "age" : { "type" : "integer" } } } } } ] } def create_young_adult_node () -> Dict[ str , Any]: """Create node for young adult quotes.""" return { "name" : "young_adult" , "task_messages" : [ { "role" : "system" , "content" : "Explain our special young adult coverage options." } ], "functions" : [ ... ] # Additional quote-specific functions } def create_standard_node () -> Dict[ str , Any]: """Create node for standard quotes.""" return { "name" : "standard" , "task_messages" : [ { "role" : "system" , "content" : "Present our standard coverage options." } ], "functions" : [ ... ] # Additional quote-specific functions } # Initialize flow manager flow_manager = FlowManager( task = task, llm = llm, context_aggregator = context_aggregator, ) @transport.event_handler ( "on_first_participant_joined" ) async def on_first_participant_joined ( transport , participant ): await transport.capture_participant_transcription(participant[ "id" ]) await flow_manager.initialize(create_initial_node()) ​ Best Practices Store shared data in flow_manager.state Create separate functions for node creation ​ Flow Editor The Pipecat Flow Editor provides a visual interface for creating and managing conversation flows. It offers a node-based interface that makes it easier to design, visualize, and modify your flows. ​ Visual Design ​ Node Types Start Node (Green): Entry point of your flow Copy Ask AI "greeting" : { "role_messages" : [ ... ], "task_messages" : [ ... ], "functions" : [ ... ] } Flow Nodes (Blue): Intermediate states Copy Ask AI "collect_info" : { "task_messages" : [ ... ], "functions" : [ ... ], "pre_actions" : [ ... ] } End Node (Red): Final state Copy Ask AI "end" : { "task_messages" : [ ... ], "functions" : [], "post_actions" : [{ "type" : "end_conversation" }] } Function Nodes : Edge Functions (Purple): Create transitions Copy Ask AI { "name" : "next_node" , "description" : "Transition to next state" } Node Functions (Orange): Perform operations Copy Ask AI { "name" : "process_data" , "handler" : process_data_handler, "description" : "Process user data" } ​ Naming Conventions Start Node : Use descriptive names (e.g., “greeting”, “welcome”) Flow Nodes : Name based on purpose (e.g., “collect_info”, “verify_data”) End Node : Conventionally named “end” Functions : Use clear, action-oriented names ​ Function Configuration Copy Ask AI { "type" : "function" , "function" : { "name" : "process_data" , "handler" : process_handler, "description" : "Process user data" , "parameters" : { ... } } } When using the Flow Editor, function handlers can be specified using the __function__: token: Copy Ask AI { "type" : "function" , "function" : { "name" : "process_data" , "handler" : "__function__:process_data" , # References function in main script "description" : "Process user data" , "parameters" : { ... } } } The handler will be looked up in your main script when the flow is executed. When function handlers are specified in the flow editor, they will be exported with the __function__: token. ​ Using the Editor ​ Creating a New Flow Start with a descriptively named Start Node Add Flow Nodes for each conversation state Connect nodes using Edge Functions Add Node Functions for operations Include an End Node ​ Import/Export Copy Ask AI # Export format { "initial_node" : "greeting" , "nodes" : { "greeting" : { "role_messages" : [ ... ], "task_messages" : [ ... ], "functions" : [ ... ], "pre_actions" : [ ... ] }, "process" : { "task_messages" : [ ... ], "functions" : [ ... ], }, "end" : { "task_messages" : [ ... ], "functions" : [], "post_actions" : [ ... ] } } } ​ Tips Use the visual preview to verify flow logic Test exported configurations Document node purposes and transitions Keep flows modular and maintainable Try the editor at flows.pipecat.ai OpenAI Audio Models and APIs Overview On this page Key Concepts Example Flows When to Use Static vs Dynamic Flows Installation Core Concepts Designing Conversation Flows Defining a Function Example Function Node Structure Messages Functions Function Configuration Node Functions Edge Functions Actions Pre-Actions Post-Actions Timing Considerations Action Types Deciding Who Speaks First Context Management Context Strategies Configuration Strategy Selection State Management LLM Provider Support OpenAI Format Anthropic Format Google (Gemini) Format Implementation Approaches Static Flows Basic Setup Example FlowConfig Dynamic Flows Example Implementation Best Practices Flow Editor Visual Design Node Types Naming Conventions Function Configuration Using the Editor Creating a New Flow Import/Export Tips Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/features_pipecat-flows_24e75092.txt b/features_pipecat-flows_24e75092.txt
new file mode 100644
index 0000000000000000000000000000000000000000..f6e72ec59909b543a71931cbc3af5088dc6d5450
--- /dev/null
+++ b/features_pipecat-flows_24e75092.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/guides/features/pipecat-flows#functions
+Title: Pipecat Flows - Pipecat
+==================================================
+
+Pipecat Flows - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Features Pipecat Flows Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Guides Fundamentals Context Management Custom FrameProcessor Detecting Idle Users Ending a Pipeline Function Calling Muting User Input Recording Audio Recording Transcripts Features Gemini Multimodal Live Metrics Noise cancellation with Krisp OpenAI Audio Models and APIs Pipecat Flows Telephony Overview Dial-in: WebRTC (Daily) Dial-in: WebRTC (Twilio + Daily) Dial-in: Twilio (Media Streams) Dialout: WebRTC (Daily) Deploying your bot Overview Deployment pattern Example: Pipecat Cloud Example: Fly.io Example: Cerebrium Example: Modal Pipecat Flows provides a framework for building structured conversations in your AI applications. It enables you to create both predefined conversation paths and dynamically generated flows while handling the complexities of state management and LLM interactions. The framework consists of: A Python module for building conversation flows with Pipecat A visual editor for designing and exporting flow configurations ​ Key Concepts Nodes : Represent conversation states with specific messages and available functions Messages : Set the role and tasks for each node Functions : Define actions and transitions (Node functions for operations, Edge functions for transitions) Actions : Execute operations during state transitions (pre/post actions) State Management : Handle conversation state and data persistence ​ Example Flows Movie Explorer (Static) A static flow demonstrating movie exploration using OpenAI. Shows real API integration with TMDB, structured data collection, and state management. Insurance Policy (Dynamic) A dynamic flow using Google Gemini that adapts policy recommendations based on user responses. Demonstrates runtime node creation and conditional paths. These examples are fully functional and can be run locally. Make sure you have the required dependencies installed and API keys configured. ​ When to Use Static vs Dynamic Flows Static Flows are ideal when: Conversation structure is known upfront Paths follow predefined patterns Flow can be fully configured in advance Example: Customer service scripts, intake forms Dynamic Flows are better when: Paths depend on external data Flow structure needs runtime modification Complex decision trees are involved Example: Personalized recommendations, adaptive workflows ​ Installation If you’re already using Pipecat: Copy Ask AI pip install pipecat-ai-flows If you’re starting fresh: Copy Ask AI # Basic installation pip install pipecat-ai-flows # Install Pipecat with specific LLM provider options: pip install "pipecat-ai[daily,openai,deepgram]" # For OpenAI pip install "pipecat-ai[daily,anthropic,deepgram]" # For Anthropic pip install "pipecat-ai[daily,google,deepgram]" # For Google 💡 Want to design your flows visually? Try the online Flow Editor ​ Core Concepts ​ Designing Conversation Flows Functions in Pipecat Flows serve two key purposes: Processing data (likely by interfacing with external systems and APIs) Advancing the conversation to the next node Each function can do one or both. LLMs decide when to run each function, via their function calling (or tool calling) mechanism. ​ Defining a Function A function is expected to return a (result, next_node) tuple. More precisely, it’s expected to return: Copy Ask AI # (result, next_node) Tuple[Optional[FlowResult], Optional[Union[NodeConfig, str ]]] If the function processes data, it should return a non- None value for the first element of the tuple. This value should be a FlowResult or subclass. If the function advances the conversation to the next node, it should return a non- None value for the second element of the tuple. This value can be either: A NodeConfig defining the next node (for dynamic flows) A string identifying the next node (for static flows) ​ Example Function Copy Ask AI from pipecat_flows import FlowArgs, FlowManager, FlowResult, NodeConfig async def check_availability ( args : FlowArgs, flow_manager : FlowManager) -> tuple[FlowResult, NodeConfig]: # Read arguments date = args[ "date" ] time = args[ "time" ] # Read previously-stored data party_size = flow_manager.state.get( "party_size" ) # Use flow_manager for immediate user feedback await flow_manager.task.queue_frame(TTSSpeakFrame( "Checking our reservation system..." )) # Store data in flow state for later use flow_manager.state[ "requested_date" ] = date # Interface with reservation system is_available = await reservation_system.check_availability(date, time, party_size) # Assemble result result = { "status" : "success" , "available" : available } # Decide which node to go to next if is_available: next_node = create_confirmation_node() else : next_node = create_no_availability_node() # Return both result and next node return result, next_node ​ Node Structure Each node in your flow represents a conversation state and consists of three main components: ​ Messages Nodes use two types of messages to control the conversation: Role Messages : Define the bot’s personality or role (optional) Copy Ask AI "role_messages" : [ { "role" : "system" , "content" : "You are a friendly pizza ordering assistant. Keep responses casual and upbeat." } ] Task Messages : Define what the bot should do in the current node Copy Ask AI "task_messages" : [ { "role" : "system" , "content" : "Ask the customer which pizza size they'd like: small, medium, or large." } ] Role messages are typically defined in your initial node and inherited by subsequent nodes, while task messages are specific to each node’s purpose. ​ Functions Functions in Pipecat Flows can: Process data Specify node transitions Do both This leads to two conceptual types of functions: Node functions , which only process data. Edge functions , which also (or only) transition to the next node. The function itself ( which you can read more about here ) is usually wrapped in a function configuration, which also contains some metadata about the function. ​ Function Configuration Pipecat Flows supports three ways of specifying function configuration: Provider-specific dictionary format Copy Ask AI # Dictionary format { "type" : "function" , "function" : { "name" : "select_size" , "handler" : select_size, "description" : "Select pizza size" , "parameters" : { "type" : "object" , "properties" : { "size" : { "type" : "string" , "enum" : [ "small" , "medium" , "large" ]} } }, } } FlowsFunctionSchema Copy Ask AI # Using FlowsFunctionSchema from pipecat_flows import FlowsFunctionSchema size_function = FlowsFunctionSchema( name = "select_size" , description = "Select pizza size" , properties = { "size" : { "type" : "string" , "enum" : [ "small" , "medium" , "large" ]} }, required = [ "size" ], handler = select_size ) # Use in node configuration node_config = { "task_messages" : [ ... ], "functions" : [size_function] } The FlowsFunctionSchema approach provides some advantages over the provider-specific dictionary format: Consistent structure across LLM providers Simplified parameter definition Cleaner, more readable code Both dictionary and FlowsFunctionSchema approaches are fully supported. FlowsFunctionSchema is recommended for new projects as it provides better type checking and a provider-independent format. Direct function usage (auto-configuration) This approach lets you bypass specifying a standalone function configuration. Instead, relevant function metadata is automatically extracted from the function’s signature and docstring: name description properties (including individual property description s) required Note that the function signature is a bit different when using direct functions. The first parameter is the FlowManager , followed by any others necessary for the function. Copy Ask AI from pipecat_flows import FlowManager, FlowResult async def select_pizza_order ( flow_manager : FlowManager, size : str , pizza_type : str , additional_toppings : list[ str ] = [], ) -> tuple[FlowResult, str ]: """ Record the pizza order details. Args: size (str): Size of the pizza. Must be one of "small", "medium", or "large". pizza_type (str): Type of pizza. Must be one of "pepperoni", "cheese", "supreme", or "vegetarian". additional_toppings (list[str]): List of additional toppings. Defaults to empty list. """ ... # Use in node configuration node_config = { "task_messages" : [ ... ], "functions" : [select_pizza_order] } ​ Node Functions Functions that process data within a single conversational state, without switching nodes. When called, they: Execute their handler to do the data processing (typically by interfacing with an external system or API) Trigger an immediate LLM completion with the result Copy Ask AI from pipecat_flows import FlowArgs, FlowManager, FlowResult async def select_size ( args : FlowArgs, flow_manager : FlowManager) -> tuple[FlowResult, None ]: """Process pizza size selection.""" size = args[ "size" ] await ordering_system.record_size_selection(size) return { "status" : "success" , "size" : size }, None # Function configuration { "type" : "function" , "function" : { "name" : "select_size" , "handler" : select_size, "description" : "Select pizza size" , "parameters" : { "type" : "object" , "properties" : { "size" : { "type" : "string" , "enum" : [ "small" , "medium" , "large" ]} } }, } } ​ Edge Functions Functions that specify a transition between nodes (optionally processing data first). When called, they: Execute their handler to do any data processing (optional) and determine the next node Add the function result to the LLM context Trigger LLM completion after both the function result and the next node’s messages are in the context Copy Ask AI from pipecat_flows import FlowArgs, FlowManager, FlowResult async def select_size ( args : FlowArgs, flow_manager : FlowManager) -> tuple[FlowResult, NodeConfig]: """Process pizza size selection.""" size = args[ "size" ] await ordering_system.record_size_selection(size) result = { "status" : "success" , "size" : size } next_node = create_confirmation_node() return result, next_node # Function configuration { "type" : "function" , "function" : { "name" : "select_size" , "handler" : select_size, "description" : "Select pizza size" , "parameters" : { "type" : "object" , "properties" : { "size" : { "type" : "string" , "enum" : [ "small" , "medium" , "large" ]} } }, } } ​ Actions Actions are operations that execute as part of the lifecycle of a node, with two distinct timing options: Pre-actions: execute when entering the node, before the LLM completion Post-actions: execute after the LLM completion ​ Pre-Actions Execute when entering the node, before LLM inference. Useful for: Providing immediate feedback while waiting for LLM responses Bridging gaps during longer function calls Setting up state or context Copy Ask AI "pre_actions" : [ { "type" : "tts_say" , "text" : "Hold on a moment..." # Immediate feedback during processing } ], Note that when the node is configured with respond_immediately: False , the pre_actions still run when entering the node, which may be well before LLM inference, depending on how long the user takes to speak first. Avoid mixing tts_say actions with chat completions as this may result in a conversation flow that feels unnatural. tts_say are best used as filler words when the LLM will take time to generate an completion. ​ Post-Actions Execute after LLM inference completes. Useful for: Cleanup operations State finalization Ensuring proper sequence of operations Copy Ask AI "post_actions" : [ { "type" : "end_conversation" # Ensures TTS completes before ending } ] Note that when the node is configured with respond_immediately: False , the post_actions still only run after the first LLM inference, which may be a while depending on how long the user takes to speak first. ​ Timing Considerations Pre-actions : Execute immediately, before any LLM processing begins LLM Inference : Processes the node’s messages and functions Post-actions : Execute after LLM processing and TTS completion For example, when using end_conversation as a post-action, the sequence is: LLM generates response TTS speaks the response End conversation action executes This ordering ensures proper completion of all operations. ​ Action Types Flows comes equipped with pre-canned actions and you can also define your own action behavior. See the reference docs for more information. ​ Deciding Who Speaks First For each node in the conversation, you can decide whether the LLM should respond immediately upon entering the node (the default behavior) or whether the LLM should wait for the user to speak first before responding. You do this using the respond_immediately field. respond_immediately=False may be particularly useful in the very first node, especially in outbound-calling cases where the user has to first answer the phone to trigger the conversation. Copy Ask AI NodeConfig( task_messages = [ { "role" : "system" , "content" : "Warmly greet the customer and ask how many people are in their party. This is your only job for now; if the customer asks for something else, politely remind them you can't do it." , } ], respond_immediately = False , # ... other fields ) Keep in mind that if you specify respond_immediately=False , the user may not be aware of the conversational task at hand when entering the node (the bot hasn’t told them yet). While it’s always important to have guardrails in your node messages to keep the conversation on topic, letting the user speak first makes it even more so. ​ Context Management Pipecat Flows provides three strategies for managing conversation context during node transitions: ​ Context Strategies APPEND (default): Adds new messages to the existing context, maintaining the full conversation history RESET : Clears the context and starts fresh with the new node’s messages RESET_WITH_SUMMARY : Resets the context but includes an AI-generated summary of the previous conversation ​ Configuration Context strategies can be configured globally or per-node: Copy Ask AI from pipecat_flows import ContextStrategy, ContextStrategyConfig # Global strategy configuration flow_manager = FlowManager( task = task, llm = llm, context_aggregator = context_aggregator, context_strategy = ContextStrategyConfig( strategy = ContextStrategy. RESET_WITH_SUMMARY , summary_prompt = "Summarize the key points discussed so far, focusing on decisions made and important information collected." ) ) # Per-node strategy configuration node_config = { "task_messages" : [ ... ], "functions" : [ ... ], "context_strategy" : ContextStrategyConfig( strategy = ContextStrategy. RESET_WITH_SUMMARY , summary_prompt = "Provide a concise summary of the customer's order details and preferences." ) } ​ Strategy Selection Choose your strategy based on your conversation needs: Use APPEND when full conversation history is important Use RESET when previous context might confuse the current node’s purpose Use RESET_WITH_SUMMARY for long conversations where key points need to be preserved When using RESET_WITH_SUMMARY, if summary generation fails or times out, the system automatically falls back to RESET strategy for resilience. ​ State Management The state variable in FlowManager is a shared dictionary that persists throughout the conversation. Think of it as a conversation memory that lets you: Store user information Track conversation progress Share data between nodes Inform decision-making Here’s a practical example of a pizza ordering flow: Copy Ask AI # Store user choices as they're made async def select_size ( args : FlowArgs) -> tuple[FlowResult, str ]: """Handle pizza size selection.""" size = args[ "size" ] # Initialize order in state if it doesn't exist if "order" not in flow_manager.state: flow_manager.state[ "order" ] = {} # Store the selection flow_manager.state[ "order" ][ "size" ] = size return { "status" : "success" , "size" : size}, "toppings" async def select_toppings ( args : FlowArgs) -> tuple[FlowResult, str ]: """Handle topping selection.""" topping = args[ "topping" ] # Get existing order and toppings order = flow_manager.state.get( "order" , {}) toppings = order.get( "toppings" , []) # Add new topping toppings.append(topping) order[ "toppings" ] = toppings flow_manager.state[ "order" ] = order return { "status" : "success" , "toppings" : toppings}, "finalize" async def finalize_order ( args : FlowArgs) -> tuple[FlowResult, str ]: """Process the complete order.""" order = flow_manager.state.get( "order" , {}) # Validate order has required information if "size" not in order: return { "status" : "error" , "error" : "No size selected" } # Calculate price based on stored selections size = order[ "size" ] toppings = order.get( "toppings" , []) price = calculate_price(size, len (toppings)) return { "status" : "success" , "summary" : f "Ordered: { size } pizza with { ', ' .join(toppings) } " , "price" : price }, "end" In this example: select_size initializes the order and stores the size select_toppings builds a list of toppings finalize_order uses the stored information to process the complete order The state variable makes it easy to: Build up information across multiple interactions Access previous choices when needed Validate the complete order Calculate final results This is particularly useful when information needs to be collected across multiple conversation turns or when later decisions depend on earlier choices. ​ LLM Provider Support Pipecat Flows automatically handles format differences between LLM providers: ​ OpenAI Format Copy Ask AI "functions" : [{ "type" : "function" , "function" : { "name" : "function_name" , "handler" : select_size, "description" : "description" , "parameters" : { ... } } }] ​ Anthropic Format Copy Ask AI "functions" : [{ "name" : "function_name" , "handler" : select_size, "description" : "description" , "input_schema" : { ... } }] ​ Google (Gemini) Format Copy Ask AI "functions" : [{ "function_declarations" : [{ "name" : "function_name" , "handler" : select_size, "description" : "description" , "parameters" : { ... } }] }] You don’t need to handle these differences manually - Pipecat Flows adapts your configuration to the correct format based on your LLM provider. ​ Implementation Approaches ​ Static Flows Static flows use a configuration-driven approach where the entire conversation structure is defined upfront. ​ Basic Setup Copy Ask AI from pipecat_flows import FlowManager # Define flow configuration flow_config = { "initial_node" : "greeting" , "nodes" : { "greeting" : { "role_messages" : [ ... ], "task_messages" : [ ... ], "functions" : [ ... ] } } } # Initialize flow manager with static configuration flow_manager = FlowManager( task = task, llm = llm, context_aggregator = context_aggregator, flow_config = flow_config ) @transport.event_handler ( "on_first_participant_joined" ) async def on_first_participant_joined ( transport , participant ): await transport.capture_participant_transcription(participant[ "id" ]) await flow_manager.initialize() ​ Example FlowConfig Copy Ask AI flow_config = { "initial_node" : "start" , "nodes" : { "start" : { "role_messages" : [ { "role" : "system" , "content" : "You are an order-taking assistant. You must ALWAYS use the available functions to progress the conversation. This is a phone conversation and your responses will be converted to audio. Keep the conversation friendly, casual, and polite. Avoid outputting special characters and emojis." , } ], "task_messages" : [ { "role" : "system" , "content" : "You are an order-taking assistant. Ask if they want pizza or sushi." } ], "functions" : [ { "type" : "function" , "function" : { "name" : "choose_pizza" , "handler" : choose_pizza, # Returns [None, "pizza_order"] "description" : "User wants pizza" , "parameters" : { "type" : "object" , "properties" : {}} } } ] }, "pizza_order" : { "task_messages" : [ ... ], "functions" : [ { "type" : "function" , "function" : { "name" : "select_size" , "handler" : select_size, # Returns [FlowResult, "toppings"] "description" : "Select pizza size" , "parameters" : { "type" : "object" , "properties" : { "size" : { "type" : "string" , "enum" : [ "small" , "medium" , "large" ]} } } } } ] } } } ​ Dynamic Flows Dynamic flows create and modify conversation paths at runtime based on data or business logic. ​ Example Implementation Here’s a complete example of a dynamic insurance quote flow: Copy Ask AI from pipecat_flows import FlowManager, FlowArgs, FlowResult # Define handlers and transitions async def collect_age ( args : FlowArgs, flow_manager : FlowManager) -> tuple[AgeResult, NodeConfig]: """Process age collection.""" age = args[ "age" ] # Assemble result result = AgeResult( status = "success" , age = age) # Decide which node to go to next if age < 25 : await flow_manager.set_node_from_config(create_young_adult_node()) else : await flow_manager.set_node_from_config(create_standard_node()) return result, age # Node creation functions def create_initial_node () -> NodeConfig: """Create initial age collection node.""" return { "name" : "initial" , "role_messages" : [ { "role" : "system" , "content" : "You are an insurance quote assistant." } ], "task_messages" : [ { "role" : "system" , "content" : "Ask for the customer's age." } ], "functions" : [ { "type" : "function" , "function" : { "name" : "collect_age" , "handler" : collect_age, "description" : "Collect customer age" , "parameters" : { "type" : "object" , "properties" : { "age" : { "type" : "integer" } } } } } ] } def create_young_adult_node () -> Dict[ str , Any]: """Create node for young adult quotes.""" return { "name" : "young_adult" , "task_messages" : [ { "role" : "system" , "content" : "Explain our special young adult coverage options." } ], "functions" : [ ... ] # Additional quote-specific functions } def create_standard_node () -> Dict[ str , Any]: """Create node for standard quotes.""" return { "name" : "standard" , "task_messages" : [ { "role" : "system" , "content" : "Present our standard coverage options." } ], "functions" : [ ... ] # Additional quote-specific functions } # Initialize flow manager flow_manager = FlowManager( task = task, llm = llm, context_aggregator = context_aggregator, ) @transport.event_handler ( "on_first_participant_joined" ) async def on_first_participant_joined ( transport , participant ): await transport.capture_participant_transcription(participant[ "id" ]) await flow_manager.initialize(create_initial_node()) ​ Best Practices Store shared data in flow_manager.state Create separate functions for node creation ​ Flow Editor The Pipecat Flow Editor provides a visual interface for creating and managing conversation flows. It offers a node-based interface that makes it easier to design, visualize, and modify your flows. ​ Visual Design ​ Node Types Start Node (Green): Entry point of your flow Copy Ask AI "greeting" : { "role_messages" : [ ... ], "task_messages" : [ ... ], "functions" : [ ... ] } Flow Nodes (Blue): Intermediate states Copy Ask AI "collect_info" : { "task_messages" : [ ... ], "functions" : [ ... ], "pre_actions" : [ ... ] } End Node (Red): Final state Copy Ask AI "end" : { "task_messages" : [ ... ], "functions" : [], "post_actions" : [{ "type" : "end_conversation" }] } Function Nodes : Edge Functions (Purple): Create transitions Copy Ask AI { "name" : "next_node" , "description" : "Transition to next state" } Node Functions (Orange): Perform operations Copy Ask AI { "name" : "process_data" , "handler" : process_data_handler, "description" : "Process user data" } ​ Naming Conventions Start Node : Use descriptive names (e.g., “greeting”, “welcome”) Flow Nodes : Name based on purpose (e.g., “collect_info”, “verify_data”) End Node : Conventionally named “end” Functions : Use clear, action-oriented names ​ Function Configuration Copy Ask AI { "type" : "function" , "function" : { "name" : "process_data" , "handler" : process_handler, "description" : "Process user data" , "parameters" : { ... } } } When using the Flow Editor, function handlers can be specified using the __function__: token: Copy Ask AI { "type" : "function" , "function" : { "name" : "process_data" , "handler" : "__function__:process_data" , # References function in main script "description" : "Process user data" , "parameters" : { ... } } } The handler will be looked up in your main script when the flow is executed. When function handlers are specified in the flow editor, they will be exported with the __function__: token. ​ Using the Editor ​ Creating a New Flow Start with a descriptively named Start Node Add Flow Nodes for each conversation state Connect nodes using Edge Functions Add Node Functions for operations Include an End Node ​ Import/Export Copy Ask AI # Export format { "initial_node" : "greeting" , "nodes" : { "greeting" : { "role_messages" : [ ... ], "task_messages" : [ ... ], "functions" : [ ... ], "pre_actions" : [ ... ] }, "process" : { "task_messages" : [ ... ], "functions" : [ ... ], }, "end" : { "task_messages" : [ ... ], "functions" : [], "post_actions" : [ ... ] } } } ​ Tips Use the visual preview to verify flow logic Test exported configurations Document node purposes and transitions Keep flows modular and maintainable Try the editor at flows.pipecat.ai OpenAI Audio Models and APIs Overview On this page Key Concepts Example Flows When to Use Static vs Dynamic Flows Installation Core Concepts Designing Conversation Flows Defining a Function Example Function Node Structure Messages Functions Function Configuration Node Functions Edge Functions Actions Pre-Actions Post-Actions Timing Considerations Action Types Deciding Who Speaks First Context Management Context Strategies Configuration Strategy Selection State Management LLM Provider Support OpenAI Format Anthropic Format Google (Gemini) Format Implementation Approaches Static Flows Basic Setup Example FlowConfig Dynamic Flows Example Implementation Best Practices Flow Editor Visual Design Node Types Naming Conventions Function Configuration Using the Editor Creating a New Flow Import/Export Tips Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/features_pipecat-flows_80e54a27.txt b/features_pipecat-flows_80e54a27.txt
new file mode 100644
index 0000000000000000000000000000000000000000..569d4470125155ff3d6ca9aa06e404c6882d0aae
--- /dev/null
+++ b/features_pipecat-flows_80e54a27.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/guides/features/pipecat-flows#visual-design
+Title: Pipecat Flows - Pipecat
+==================================================
+
+Pipecat Flows - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Features Pipecat Flows Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Guides Fundamentals Context Management Custom FrameProcessor Detecting Idle Users Ending a Pipeline Function Calling Muting User Input Recording Audio Recording Transcripts Features Gemini Multimodal Live Metrics Noise cancellation with Krisp OpenAI Audio Models and APIs Pipecat Flows Telephony Overview Dial-in: WebRTC (Daily) Dial-in: WebRTC (Twilio + Daily) Dial-in: Twilio (Media Streams) Dialout: WebRTC (Daily) Deploying your bot Overview Deployment pattern Example: Pipecat Cloud Example: Fly.io Example: Cerebrium Example: Modal Pipecat Flows provides a framework for building structured conversations in your AI applications. It enables you to create both predefined conversation paths and dynamically generated flows while handling the complexities of state management and LLM interactions. The framework consists of: A Python module for building conversation flows with Pipecat A visual editor for designing and exporting flow configurations ​ Key Concepts Nodes : Represent conversation states with specific messages and available functions Messages : Set the role and tasks for each node Functions : Define actions and transitions (Node functions for operations, Edge functions for transitions) Actions : Execute operations during state transitions (pre/post actions) State Management : Handle conversation state and data persistence ​ Example Flows Movie Explorer (Static) A static flow demonstrating movie exploration using OpenAI. Shows real API integration with TMDB, structured data collection, and state management. Insurance Policy (Dynamic) A dynamic flow using Google Gemini that adapts policy recommendations based on user responses. Demonstrates runtime node creation and conditional paths. These examples are fully functional and can be run locally. Make sure you have the required dependencies installed and API keys configured. ​ When to Use Static vs Dynamic Flows Static Flows are ideal when: Conversation structure is known upfront Paths follow predefined patterns Flow can be fully configured in advance Example: Customer service scripts, intake forms Dynamic Flows are better when: Paths depend on external data Flow structure needs runtime modification Complex decision trees are involved Example: Personalized recommendations, adaptive workflows ​ Installation If you’re already using Pipecat: Copy Ask AI pip install pipecat-ai-flows If you’re starting fresh: Copy Ask AI # Basic installation pip install pipecat-ai-flows # Install Pipecat with specific LLM provider options: pip install "pipecat-ai[daily,openai,deepgram]" # For OpenAI pip install "pipecat-ai[daily,anthropic,deepgram]" # For Anthropic pip install "pipecat-ai[daily,google,deepgram]" # For Google 💡 Want to design your flows visually? Try the online Flow Editor ​ Core Concepts ​ Designing Conversation Flows Functions in Pipecat Flows serve two key purposes: Processing data (likely by interfacing with external systems and APIs) Advancing the conversation to the next node Each function can do one or both. LLMs decide when to run each function, via their function calling (or tool calling) mechanism. ​ Defining a Function A function is expected to return a (result, next_node) tuple. More precisely, it’s expected to return: Copy Ask AI # (result, next_node) Tuple[Optional[FlowResult], Optional[Union[NodeConfig, str ]]] If the function processes data, it should return a non- None value for the first element of the tuple. This value should be a FlowResult or subclass. If the function advances the conversation to the next node, it should return a non- None value for the second element of the tuple. This value can be either: A NodeConfig defining the next node (for dynamic flows) A string identifying the next node (for static flows) ​ Example Function Copy Ask AI from pipecat_flows import FlowArgs, FlowManager, FlowResult, NodeConfig async def check_availability ( args : FlowArgs, flow_manager : FlowManager) -> tuple[FlowResult, NodeConfig]: # Read arguments date = args[ "date" ] time = args[ "time" ] # Read previously-stored data party_size = flow_manager.state.get( "party_size" ) # Use flow_manager for immediate user feedback await flow_manager.task.queue_frame(TTSSpeakFrame( "Checking our reservation system..." )) # Store data in flow state for later use flow_manager.state[ "requested_date" ] = date # Interface with reservation system is_available = await reservation_system.check_availability(date, time, party_size) # Assemble result result = { "status" : "success" , "available" : available } # Decide which node to go to next if is_available: next_node = create_confirmation_node() else : next_node = create_no_availability_node() # Return both result and next node return result, next_node ​ Node Structure Each node in your flow represents a conversation state and consists of three main components: ​ Messages Nodes use two types of messages to control the conversation: Role Messages : Define the bot’s personality or role (optional) Copy Ask AI "role_messages" : [ { "role" : "system" , "content" : "You are a friendly pizza ordering assistant. Keep responses casual and upbeat." } ] Task Messages : Define what the bot should do in the current node Copy Ask AI "task_messages" : [ { "role" : "system" , "content" : "Ask the customer which pizza size they'd like: small, medium, or large." } ] Role messages are typically defined in your initial node and inherited by subsequent nodes, while task messages are specific to each node’s purpose. ​ Functions Functions in Pipecat Flows can: Process data Specify node transitions Do both This leads to two conceptual types of functions: Node functions , which only process data. Edge functions , which also (or only) transition to the next node. The function itself ( which you can read more about here ) is usually wrapped in a function configuration, which also contains some metadata about the function. ​ Function Configuration Pipecat Flows supports three ways of specifying function configuration: Provider-specific dictionary format Copy Ask AI # Dictionary format { "type" : "function" , "function" : { "name" : "select_size" , "handler" : select_size, "description" : "Select pizza size" , "parameters" : { "type" : "object" , "properties" : { "size" : { "type" : "string" , "enum" : [ "small" , "medium" , "large" ]} } }, } } FlowsFunctionSchema Copy Ask AI # Using FlowsFunctionSchema from pipecat_flows import FlowsFunctionSchema size_function = FlowsFunctionSchema( name = "select_size" , description = "Select pizza size" , properties = { "size" : { "type" : "string" , "enum" : [ "small" , "medium" , "large" ]} }, required = [ "size" ], handler = select_size ) # Use in node configuration node_config = { "task_messages" : [ ... ], "functions" : [size_function] } The FlowsFunctionSchema approach provides some advantages over the provider-specific dictionary format: Consistent structure across LLM providers Simplified parameter definition Cleaner, more readable code Both dictionary and FlowsFunctionSchema approaches are fully supported. FlowsFunctionSchema is recommended for new projects as it provides better type checking and a provider-independent format. Direct function usage (auto-configuration) This approach lets you bypass specifying a standalone function configuration. Instead, relevant function metadata is automatically extracted from the function’s signature and docstring: name description properties (including individual property description s) required Note that the function signature is a bit different when using direct functions. The first parameter is the FlowManager , followed by any others necessary for the function. Copy Ask AI from pipecat_flows import FlowManager, FlowResult async def select_pizza_order ( flow_manager : FlowManager, size : str , pizza_type : str , additional_toppings : list[ str ] = [], ) -> tuple[FlowResult, str ]: """ Record the pizza order details. Args: size (str): Size of the pizza. Must be one of "small", "medium", or "large". pizza_type (str): Type of pizza. Must be one of "pepperoni", "cheese", "supreme", or "vegetarian". additional_toppings (list[str]): List of additional toppings. Defaults to empty list. """ ... # Use in node configuration node_config = { "task_messages" : [ ... ], "functions" : [select_pizza_order] } ​ Node Functions Functions that process data within a single conversational state, without switching nodes. When called, they: Execute their handler to do the data processing (typically by interfacing with an external system or API) Trigger an immediate LLM completion with the result Copy Ask AI from pipecat_flows import FlowArgs, FlowManager, FlowResult async def select_size ( args : FlowArgs, flow_manager : FlowManager) -> tuple[FlowResult, None ]: """Process pizza size selection.""" size = args[ "size" ] await ordering_system.record_size_selection(size) return { "status" : "success" , "size" : size }, None # Function configuration { "type" : "function" , "function" : { "name" : "select_size" , "handler" : select_size, "description" : "Select pizza size" , "parameters" : { "type" : "object" , "properties" : { "size" : { "type" : "string" , "enum" : [ "small" , "medium" , "large" ]} } }, } } ​ Edge Functions Functions that specify a transition between nodes (optionally processing data first). When called, they: Execute their handler to do any data processing (optional) and determine the next node Add the function result to the LLM context Trigger LLM completion after both the function result and the next node’s messages are in the context Copy Ask AI from pipecat_flows import FlowArgs, FlowManager, FlowResult async def select_size ( args : FlowArgs, flow_manager : FlowManager) -> tuple[FlowResult, NodeConfig]: """Process pizza size selection.""" size = args[ "size" ] await ordering_system.record_size_selection(size) result = { "status" : "success" , "size" : size } next_node = create_confirmation_node() return result, next_node # Function configuration { "type" : "function" , "function" : { "name" : "select_size" , "handler" : select_size, "description" : "Select pizza size" , "parameters" : { "type" : "object" , "properties" : { "size" : { "type" : "string" , "enum" : [ "small" , "medium" , "large" ]} } }, } } ​ Actions Actions are operations that execute as part of the lifecycle of a node, with two distinct timing options: Pre-actions: execute when entering the node, before the LLM completion Post-actions: execute after the LLM completion ​ Pre-Actions Execute when entering the node, before LLM inference. Useful for: Providing immediate feedback while waiting for LLM responses Bridging gaps during longer function calls Setting up state or context Copy Ask AI "pre_actions" : [ { "type" : "tts_say" , "text" : "Hold on a moment..." # Immediate feedback during processing } ], Note that when the node is configured with respond_immediately: False , the pre_actions still run when entering the node, which may be well before LLM inference, depending on how long the user takes to speak first. Avoid mixing tts_say actions with chat completions as this may result in a conversation flow that feels unnatural. tts_say are best used as filler words when the LLM will take time to generate an completion. ​ Post-Actions Execute after LLM inference completes. Useful for: Cleanup operations State finalization Ensuring proper sequence of operations Copy Ask AI "post_actions" : [ { "type" : "end_conversation" # Ensures TTS completes before ending } ] Note that when the node is configured with respond_immediately: False , the post_actions still only run after the first LLM inference, which may be a while depending on how long the user takes to speak first. ​ Timing Considerations Pre-actions : Execute immediately, before any LLM processing begins LLM Inference : Processes the node’s messages and functions Post-actions : Execute after LLM processing and TTS completion For example, when using end_conversation as a post-action, the sequence is: LLM generates response TTS speaks the response End conversation action executes This ordering ensures proper completion of all operations. ​ Action Types Flows comes equipped with pre-canned actions and you can also define your own action behavior. See the reference docs for more information. ​ Deciding Who Speaks First For each node in the conversation, you can decide whether the LLM should respond immediately upon entering the node (the default behavior) or whether the LLM should wait for the user to speak first before responding. You do this using the respond_immediately field. respond_immediately=False may be particularly useful in the very first node, especially in outbound-calling cases where the user has to first answer the phone to trigger the conversation. Copy Ask AI NodeConfig( task_messages = [ { "role" : "system" , "content" : "Warmly greet the customer and ask how many people are in their party. This is your only job for now; if the customer asks for something else, politely remind them you can't do it." , } ], respond_immediately = False , # ... other fields ) Keep in mind that if you specify respond_immediately=False , the user may not be aware of the conversational task at hand when entering the node (the bot hasn’t told them yet). While it’s always important to have guardrails in your node messages to keep the conversation on topic, letting the user speak first makes it even more so. ​ Context Management Pipecat Flows provides three strategies for managing conversation context during node transitions: ​ Context Strategies APPEND (default): Adds new messages to the existing context, maintaining the full conversation history RESET : Clears the context and starts fresh with the new node’s messages RESET_WITH_SUMMARY : Resets the context but includes an AI-generated summary of the previous conversation ​ Configuration Context strategies can be configured globally or per-node: Copy Ask AI from pipecat_flows import ContextStrategy, ContextStrategyConfig # Global strategy configuration flow_manager = FlowManager( task = task, llm = llm, context_aggregator = context_aggregator, context_strategy = ContextStrategyConfig( strategy = ContextStrategy. RESET_WITH_SUMMARY , summary_prompt = "Summarize the key points discussed so far, focusing on decisions made and important information collected." ) ) # Per-node strategy configuration node_config = { "task_messages" : [ ... ], "functions" : [ ... ], "context_strategy" : ContextStrategyConfig( strategy = ContextStrategy. RESET_WITH_SUMMARY , summary_prompt = "Provide a concise summary of the customer's order details and preferences." ) } ​ Strategy Selection Choose your strategy based on your conversation needs: Use APPEND when full conversation history is important Use RESET when previous context might confuse the current node’s purpose Use RESET_WITH_SUMMARY for long conversations where key points need to be preserved When using RESET_WITH_SUMMARY, if summary generation fails or times out, the system automatically falls back to RESET strategy for resilience. ​ State Management The state variable in FlowManager is a shared dictionary that persists throughout the conversation. Think of it as a conversation memory that lets you: Store user information Track conversation progress Share data between nodes Inform decision-making Here’s a practical example of a pizza ordering flow: Copy Ask AI # Store user choices as they're made async def select_size ( args : FlowArgs) -> tuple[FlowResult, str ]: """Handle pizza size selection.""" size = args[ "size" ] # Initialize order in state if it doesn't exist if "order" not in flow_manager.state: flow_manager.state[ "order" ] = {} # Store the selection flow_manager.state[ "order" ][ "size" ] = size return { "status" : "success" , "size" : size}, "toppings" async def select_toppings ( args : FlowArgs) -> tuple[FlowResult, str ]: """Handle topping selection.""" topping = args[ "topping" ] # Get existing order and toppings order = flow_manager.state.get( "order" , {}) toppings = order.get( "toppings" , []) # Add new topping toppings.append(topping) order[ "toppings" ] = toppings flow_manager.state[ "order" ] = order return { "status" : "success" , "toppings" : toppings}, "finalize" async def finalize_order ( args : FlowArgs) -> tuple[FlowResult, str ]: """Process the complete order.""" order = flow_manager.state.get( "order" , {}) # Validate order has required information if "size" not in order: return { "status" : "error" , "error" : "No size selected" } # Calculate price based on stored selections size = order[ "size" ] toppings = order.get( "toppings" , []) price = calculate_price(size, len (toppings)) return { "status" : "success" , "summary" : f "Ordered: { size } pizza with { ', ' .join(toppings) } " , "price" : price }, "end" In this example: select_size initializes the order and stores the size select_toppings builds a list of toppings finalize_order uses the stored information to process the complete order The state variable makes it easy to: Build up information across multiple interactions Access previous choices when needed Validate the complete order Calculate final results This is particularly useful when information needs to be collected across multiple conversation turns or when later decisions depend on earlier choices. ​ LLM Provider Support Pipecat Flows automatically handles format differences between LLM providers: ​ OpenAI Format Copy Ask AI "functions" : [{ "type" : "function" , "function" : { "name" : "function_name" , "handler" : select_size, "description" : "description" , "parameters" : { ... } } }] ​ Anthropic Format Copy Ask AI "functions" : [{ "name" : "function_name" , "handler" : select_size, "description" : "description" , "input_schema" : { ... } }] ​ Google (Gemini) Format Copy Ask AI "functions" : [{ "function_declarations" : [{ "name" : "function_name" , "handler" : select_size, "description" : "description" , "parameters" : { ... } }] }] You don’t need to handle these differences manually - Pipecat Flows adapts your configuration to the correct format based on your LLM provider. ​ Implementation Approaches ​ Static Flows Static flows use a configuration-driven approach where the entire conversation structure is defined upfront. ​ Basic Setup Copy Ask AI from pipecat_flows import FlowManager # Define flow configuration flow_config = { "initial_node" : "greeting" , "nodes" : { "greeting" : { "role_messages" : [ ... ], "task_messages" : [ ... ], "functions" : [ ... ] } } } # Initialize flow manager with static configuration flow_manager = FlowManager( task = task, llm = llm, context_aggregator = context_aggregator, flow_config = flow_config ) @transport.event_handler ( "on_first_participant_joined" ) async def on_first_participant_joined ( transport , participant ): await transport.capture_participant_transcription(participant[ "id" ]) await flow_manager.initialize() ​ Example FlowConfig Copy Ask AI flow_config = { "initial_node" : "start" , "nodes" : { "start" : { "role_messages" : [ { "role" : "system" , "content" : "You are an order-taking assistant. You must ALWAYS use the available functions to progress the conversation. This is a phone conversation and your responses will be converted to audio. Keep the conversation friendly, casual, and polite. Avoid outputting special characters and emojis." , } ], "task_messages" : [ { "role" : "system" , "content" : "You are an order-taking assistant. Ask if they want pizza or sushi." } ], "functions" : [ { "type" : "function" , "function" : { "name" : "choose_pizza" , "handler" : choose_pizza, # Returns [None, "pizza_order"] "description" : "User wants pizza" , "parameters" : { "type" : "object" , "properties" : {}} } } ] }, "pizza_order" : { "task_messages" : [ ... ], "functions" : [ { "type" : "function" , "function" : { "name" : "select_size" , "handler" : select_size, # Returns [FlowResult, "toppings"] "description" : "Select pizza size" , "parameters" : { "type" : "object" , "properties" : { "size" : { "type" : "string" , "enum" : [ "small" , "medium" , "large" ]} } } } } ] } } } ​ Dynamic Flows Dynamic flows create and modify conversation paths at runtime based on data or business logic. ​ Example Implementation Here’s a complete example of a dynamic insurance quote flow: Copy Ask AI from pipecat_flows import FlowManager, FlowArgs, FlowResult # Define handlers and transitions async def collect_age ( args : FlowArgs, flow_manager : FlowManager) -> tuple[AgeResult, NodeConfig]: """Process age collection.""" age = args[ "age" ] # Assemble result result = AgeResult( status = "success" , age = age) # Decide which node to go to next if age < 25 : await flow_manager.set_node_from_config(create_young_adult_node()) else : await flow_manager.set_node_from_config(create_standard_node()) return result, age # Node creation functions def create_initial_node () -> NodeConfig: """Create initial age collection node.""" return { "name" : "initial" , "role_messages" : [ { "role" : "system" , "content" : "You are an insurance quote assistant." } ], "task_messages" : [ { "role" : "system" , "content" : "Ask for the customer's age." } ], "functions" : [ { "type" : "function" , "function" : { "name" : "collect_age" , "handler" : collect_age, "description" : "Collect customer age" , "parameters" : { "type" : "object" , "properties" : { "age" : { "type" : "integer" } } } } } ] } def create_young_adult_node () -> Dict[ str , Any]: """Create node for young adult quotes.""" return { "name" : "young_adult" , "task_messages" : [ { "role" : "system" , "content" : "Explain our special young adult coverage options." } ], "functions" : [ ... ] # Additional quote-specific functions } def create_standard_node () -> Dict[ str , Any]: """Create node for standard quotes.""" return { "name" : "standard" , "task_messages" : [ { "role" : "system" , "content" : "Present our standard coverage options." } ], "functions" : [ ... ] # Additional quote-specific functions } # Initialize flow manager flow_manager = FlowManager( task = task, llm = llm, context_aggregator = context_aggregator, ) @transport.event_handler ( "on_first_participant_joined" ) async def on_first_participant_joined ( transport , participant ): await transport.capture_participant_transcription(participant[ "id" ]) await flow_manager.initialize(create_initial_node()) ​ Best Practices Store shared data in flow_manager.state Create separate functions for node creation ​ Flow Editor The Pipecat Flow Editor provides a visual interface for creating and managing conversation flows. It offers a node-based interface that makes it easier to design, visualize, and modify your flows. ​ Visual Design ​ Node Types Start Node (Green): Entry point of your flow Copy Ask AI "greeting" : { "role_messages" : [ ... ], "task_messages" : [ ... ], "functions" : [ ... ] } Flow Nodes (Blue): Intermediate states Copy Ask AI "collect_info" : { "task_messages" : [ ... ], "functions" : [ ... ], "pre_actions" : [ ... ] } End Node (Red): Final state Copy Ask AI "end" : { "task_messages" : [ ... ], "functions" : [], "post_actions" : [{ "type" : "end_conversation" }] } Function Nodes : Edge Functions (Purple): Create transitions Copy Ask AI { "name" : "next_node" , "description" : "Transition to next state" } Node Functions (Orange): Perform operations Copy Ask AI { "name" : "process_data" , "handler" : process_data_handler, "description" : "Process user data" } ​ Naming Conventions Start Node : Use descriptive names (e.g., “greeting”, “welcome”) Flow Nodes : Name based on purpose (e.g., “collect_info”, “verify_data”) End Node : Conventionally named “end” Functions : Use clear, action-oriented names ​ Function Configuration Copy Ask AI { "type" : "function" , "function" : { "name" : "process_data" , "handler" : process_handler, "description" : "Process user data" , "parameters" : { ... } } } When using the Flow Editor, function handlers can be specified using the __function__: token: Copy Ask AI { "type" : "function" , "function" : { "name" : "process_data" , "handler" : "__function__:process_data" , # References function in main script "description" : "Process user data" , "parameters" : { ... } } } The handler will be looked up in your main script when the flow is executed. When function handlers are specified in the flow editor, they will be exported with the __function__: token. ​ Using the Editor ​ Creating a New Flow Start with a descriptively named Start Node Add Flow Nodes for each conversation state Connect nodes using Edge Functions Add Node Functions for operations Include an End Node ​ Import/Export Copy Ask AI # Export format { "initial_node" : "greeting" , "nodes" : { "greeting" : { "role_messages" : [ ... ], "task_messages" : [ ... ], "functions" : [ ... ], "pre_actions" : [ ... ] }, "process" : { "task_messages" : [ ... ], "functions" : [ ... ], }, "end" : { "task_messages" : [ ... ], "functions" : [], "post_actions" : [ ... ] } } } ​ Tips Use the visual preview to verify flow logic Test exported configurations Document node purposes and transitions Keep flows modular and maintainable Try the editor at flows.pipecat.ai OpenAI Audio Models and APIs Overview On this page Key Concepts Example Flows When to Use Static vs Dynamic Flows Installation Core Concepts Designing Conversation Flows Defining a Function Example Function Node Structure Messages Functions Function Configuration Node Functions Edge Functions Actions Pre-Actions Post-Actions Timing Considerations Action Types Deciding Who Speaks First Context Management Context Strategies Configuration Strategy Selection State Management LLM Provider Support OpenAI Format Anthropic Format Google (Gemini) Format Implementation Approaches Static Flows Basic Setup Example FlowConfig Dynamic Flows Example Implementation Best Practices Flow Editor Visual Design Node Types Naming Conventions Function Configuration Using the Editor Creating a New Flow Import/Export Tips Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/features_pipecat-flows_854aecd8.txt b/features_pipecat-flows_854aecd8.txt
new file mode 100644
index 0000000000000000000000000000000000000000..e4a24e63cb5d8cdcc77bf3661f4b4cbd8816520c
--- /dev/null
+++ b/features_pipecat-flows_854aecd8.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/guides/features/pipecat-flows
+Title: Pipecat Flows - Pipecat
+==================================================
+
+Pipecat Flows - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Features Pipecat Flows Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Guides Fundamentals Context Management Custom FrameProcessor Detecting Idle Users Ending a Pipeline Function Calling Muting User Input Recording Audio Recording Transcripts Features Gemini Multimodal Live Metrics Noise cancellation with Krisp OpenAI Audio Models and APIs Pipecat Flows Telephony Overview Dial-in: WebRTC (Daily) Dial-in: WebRTC (Twilio + Daily) Dial-in: Twilio (Media Streams) Dialout: WebRTC (Daily) Deploying your bot Overview Deployment pattern Example: Pipecat Cloud Example: Fly.io Example: Cerebrium Example: Modal Pipecat Flows provides a framework for building structured conversations in your AI applications. It enables you to create both predefined conversation paths and dynamically generated flows while handling the complexities of state management and LLM interactions. The framework consists of: A Python module for building conversation flows with Pipecat A visual editor for designing and exporting flow configurations ​ Key Concepts Nodes : Represent conversation states with specific messages and available functions Messages : Set the role and tasks for each node Functions : Define actions and transitions (Node functions for operations, Edge functions for transitions) Actions : Execute operations during state transitions (pre/post actions) State Management : Handle conversation state and data persistence ​ Example Flows Movie Explorer (Static) A static flow demonstrating movie exploration using OpenAI. Shows real API integration with TMDB, structured data collection, and state management. Insurance Policy (Dynamic) A dynamic flow using Google Gemini that adapts policy recommendations based on user responses. Demonstrates runtime node creation and conditional paths. These examples are fully functional and can be run locally. Make sure you have the required dependencies installed and API keys configured. ​ When to Use Static vs Dynamic Flows Static Flows are ideal when: Conversation structure is known upfront Paths follow predefined patterns Flow can be fully configured in advance Example: Customer service scripts, intake forms Dynamic Flows are better when: Paths depend on external data Flow structure needs runtime modification Complex decision trees are involved Example: Personalized recommendations, adaptive workflows ​ Installation If you’re already using Pipecat: Copy Ask AI pip install pipecat-ai-flows If you’re starting fresh: Copy Ask AI # Basic installation pip install pipecat-ai-flows # Install Pipecat with specific LLM provider options: pip install "pipecat-ai[daily,openai,deepgram]" # For OpenAI pip install "pipecat-ai[daily,anthropic,deepgram]" # For Anthropic pip install "pipecat-ai[daily,google,deepgram]" # For Google 💡 Want to design your flows visually? Try the online Flow Editor ​ Core Concepts ​ Designing Conversation Flows Functions in Pipecat Flows serve two key purposes: Processing data (likely by interfacing with external systems and APIs) Advancing the conversation to the next node Each function can do one or both. LLMs decide when to run each function, via their function calling (or tool calling) mechanism. ​ Defining a Function A function is expected to return a (result, next_node) tuple. More precisely, it’s expected to return: Copy Ask AI # (result, next_node) Tuple[Optional[FlowResult], Optional[Union[NodeConfig, str ]]] If the function processes data, it should return a non- None value for the first element of the tuple. This value should be a FlowResult or subclass. If the function advances the conversation to the next node, it should return a non- None value for the second element of the tuple. This value can be either: A NodeConfig defining the next node (for dynamic flows) A string identifying the next node (for static flows) ​ Example Function Copy Ask AI from pipecat_flows import FlowArgs, FlowManager, FlowResult, NodeConfig async def check_availability ( args : FlowArgs, flow_manager : FlowManager) -> tuple[FlowResult, NodeConfig]: # Read arguments date = args[ "date" ] time = args[ "time" ] # Read previously-stored data party_size = flow_manager.state.get( "party_size" ) # Use flow_manager for immediate user feedback await flow_manager.task.queue_frame(TTSSpeakFrame( "Checking our reservation system..." )) # Store data in flow state for later use flow_manager.state[ "requested_date" ] = date # Interface with reservation system is_available = await reservation_system.check_availability(date, time, party_size) # Assemble result result = { "status" : "success" , "available" : available } # Decide which node to go to next if is_available: next_node = create_confirmation_node() else : next_node = create_no_availability_node() # Return both result and next node return result, next_node ​ Node Structure Each node in your flow represents a conversation state and consists of three main components: ​ Messages Nodes use two types of messages to control the conversation: Role Messages : Define the bot’s personality or role (optional) Copy Ask AI "role_messages" : [ { "role" : "system" , "content" : "You are a friendly pizza ordering assistant. Keep responses casual and upbeat." } ] Task Messages : Define what the bot should do in the current node Copy Ask AI "task_messages" : [ { "role" : "system" , "content" : "Ask the customer which pizza size they'd like: small, medium, or large." } ] Role messages are typically defined in your initial node and inherited by subsequent nodes, while task messages are specific to each node’s purpose. ​ Functions Functions in Pipecat Flows can: Process data Specify node transitions Do both This leads to two conceptual types of functions: Node functions , which only process data. Edge functions , which also (or only) transition to the next node. The function itself ( which you can read more about here ) is usually wrapped in a function configuration, which also contains some metadata about the function. ​ Function Configuration Pipecat Flows supports three ways of specifying function configuration: Provider-specific dictionary format Copy Ask AI # Dictionary format { "type" : "function" , "function" : { "name" : "select_size" , "handler" : select_size, "description" : "Select pizza size" , "parameters" : { "type" : "object" , "properties" : { "size" : { "type" : "string" , "enum" : [ "small" , "medium" , "large" ]} } }, } } FlowsFunctionSchema Copy Ask AI # Using FlowsFunctionSchema from pipecat_flows import FlowsFunctionSchema size_function = FlowsFunctionSchema( name = "select_size" , description = "Select pizza size" , properties = { "size" : { "type" : "string" , "enum" : [ "small" , "medium" , "large" ]} }, required = [ "size" ], handler = select_size ) # Use in node configuration node_config = { "task_messages" : [ ... ], "functions" : [size_function] } The FlowsFunctionSchema approach provides some advantages over the provider-specific dictionary format: Consistent structure across LLM providers Simplified parameter definition Cleaner, more readable code Both dictionary and FlowsFunctionSchema approaches are fully supported. FlowsFunctionSchema is recommended for new projects as it provides better type checking and a provider-independent format. Direct function usage (auto-configuration) This approach lets you bypass specifying a standalone function configuration. Instead, relevant function metadata is automatically extracted from the function’s signature and docstring: name description properties (including individual property description s) required Note that the function signature is a bit different when using direct functions. The first parameter is the FlowManager , followed by any others necessary for the function. Copy Ask AI from pipecat_flows import FlowManager, FlowResult async def select_pizza_order ( flow_manager : FlowManager, size : str , pizza_type : str , additional_toppings : list[ str ] = [], ) -> tuple[FlowResult, str ]: """ Record the pizza order details. Args: size (str): Size of the pizza. Must be one of "small", "medium", or "large". pizza_type (str): Type of pizza. Must be one of "pepperoni", "cheese", "supreme", or "vegetarian". additional_toppings (list[str]): List of additional toppings. Defaults to empty list. """ ... # Use in node configuration node_config = { "task_messages" : [ ... ], "functions" : [select_pizza_order] } ​ Node Functions Functions that process data within a single conversational state, without switching nodes. When called, they: Execute their handler to do the data processing (typically by interfacing with an external system or API) Trigger an immediate LLM completion with the result Copy Ask AI from pipecat_flows import FlowArgs, FlowManager, FlowResult async def select_size ( args : FlowArgs, flow_manager : FlowManager) -> tuple[FlowResult, None ]: """Process pizza size selection.""" size = args[ "size" ] await ordering_system.record_size_selection(size) return { "status" : "success" , "size" : size }, None # Function configuration { "type" : "function" , "function" : { "name" : "select_size" , "handler" : select_size, "description" : "Select pizza size" , "parameters" : { "type" : "object" , "properties" : { "size" : { "type" : "string" , "enum" : [ "small" , "medium" , "large" ]} } }, } } ​ Edge Functions Functions that specify a transition between nodes (optionally processing data first). When called, they: Execute their handler to do any data processing (optional) and determine the next node Add the function result to the LLM context Trigger LLM completion after both the function result and the next node’s messages are in the context Copy Ask AI from pipecat_flows import FlowArgs, FlowManager, FlowResult async def select_size ( args : FlowArgs, flow_manager : FlowManager) -> tuple[FlowResult, NodeConfig]: """Process pizza size selection.""" size = args[ "size" ] await ordering_system.record_size_selection(size) result = { "status" : "success" , "size" : size } next_node = create_confirmation_node() return result, next_node # Function configuration { "type" : "function" , "function" : { "name" : "select_size" , "handler" : select_size, "description" : "Select pizza size" , "parameters" : { "type" : "object" , "properties" : { "size" : { "type" : "string" , "enum" : [ "small" , "medium" , "large" ]} } }, } } ​ Actions Actions are operations that execute as part of the lifecycle of a node, with two distinct timing options: Pre-actions: execute when entering the node, before the LLM completion Post-actions: execute after the LLM completion ​ Pre-Actions Execute when entering the node, before LLM inference. Useful for: Providing immediate feedback while waiting for LLM responses Bridging gaps during longer function calls Setting up state or context Copy Ask AI "pre_actions" : [ { "type" : "tts_say" , "text" : "Hold on a moment..." # Immediate feedback during processing } ], Note that when the node is configured with respond_immediately: False , the pre_actions still run when entering the node, which may be well before LLM inference, depending on how long the user takes to speak first. Avoid mixing tts_say actions with chat completions as this may result in a conversation flow that feels unnatural. tts_say are best used as filler words when the LLM will take time to generate an completion. ​ Post-Actions Execute after LLM inference completes. Useful for: Cleanup operations State finalization Ensuring proper sequence of operations Copy Ask AI "post_actions" : [ { "type" : "end_conversation" # Ensures TTS completes before ending } ] Note that when the node is configured with respond_immediately: False , the post_actions still only run after the first LLM inference, which may be a while depending on how long the user takes to speak first. ​ Timing Considerations Pre-actions : Execute immediately, before any LLM processing begins LLM Inference : Processes the node’s messages and functions Post-actions : Execute after LLM processing and TTS completion For example, when using end_conversation as a post-action, the sequence is: LLM generates response TTS speaks the response End conversation action executes This ordering ensures proper completion of all operations. ​ Action Types Flows comes equipped with pre-canned actions and you can also define your own action behavior. See the reference docs for more information. ​ Deciding Who Speaks First For each node in the conversation, you can decide whether the LLM should respond immediately upon entering the node (the default behavior) or whether the LLM should wait for the user to speak first before responding. You do this using the respond_immediately field. respond_immediately=False may be particularly useful in the very first node, especially in outbound-calling cases where the user has to first answer the phone to trigger the conversation. Copy Ask AI NodeConfig( task_messages = [ { "role" : "system" , "content" : "Warmly greet the customer and ask how many people are in their party. This is your only job for now; if the customer asks for something else, politely remind them you can't do it." , } ], respond_immediately = False , # ... other fields ) Keep in mind that if you specify respond_immediately=False , the user may not be aware of the conversational task at hand when entering the node (the bot hasn’t told them yet). While it’s always important to have guardrails in your node messages to keep the conversation on topic, letting the user speak first makes it even more so. ​ Context Management Pipecat Flows provides three strategies for managing conversation context during node transitions: ​ Context Strategies APPEND (default): Adds new messages to the existing context, maintaining the full conversation history RESET : Clears the context and starts fresh with the new node’s messages RESET_WITH_SUMMARY : Resets the context but includes an AI-generated summary of the previous conversation ​ Configuration Context strategies can be configured globally or per-node: Copy Ask AI from pipecat_flows import ContextStrategy, ContextStrategyConfig # Global strategy configuration flow_manager = FlowManager( task = task, llm = llm, context_aggregator = context_aggregator, context_strategy = ContextStrategyConfig( strategy = ContextStrategy. RESET_WITH_SUMMARY , summary_prompt = "Summarize the key points discussed so far, focusing on decisions made and important information collected." ) ) # Per-node strategy configuration node_config = { "task_messages" : [ ... ], "functions" : [ ... ], "context_strategy" : ContextStrategyConfig( strategy = ContextStrategy. RESET_WITH_SUMMARY , summary_prompt = "Provide a concise summary of the customer's order details and preferences." ) } ​ Strategy Selection Choose your strategy based on your conversation needs: Use APPEND when full conversation history is important Use RESET when previous context might confuse the current node’s purpose Use RESET_WITH_SUMMARY for long conversations where key points need to be preserved When using RESET_WITH_SUMMARY, if summary generation fails or times out, the system automatically falls back to RESET strategy for resilience. ​ State Management The state variable in FlowManager is a shared dictionary that persists throughout the conversation. Think of it as a conversation memory that lets you: Store user information Track conversation progress Share data between nodes Inform decision-making Here’s a practical example of a pizza ordering flow: Copy Ask AI # Store user choices as they're made async def select_size ( args : FlowArgs) -> tuple[FlowResult, str ]: """Handle pizza size selection.""" size = args[ "size" ] # Initialize order in state if it doesn't exist if "order" not in flow_manager.state: flow_manager.state[ "order" ] = {} # Store the selection flow_manager.state[ "order" ][ "size" ] = size return { "status" : "success" , "size" : size}, "toppings" async def select_toppings ( args : FlowArgs) -> tuple[FlowResult, str ]: """Handle topping selection.""" topping = args[ "topping" ] # Get existing order and toppings order = flow_manager.state.get( "order" , {}) toppings = order.get( "toppings" , []) # Add new topping toppings.append(topping) order[ "toppings" ] = toppings flow_manager.state[ "order" ] = order return { "status" : "success" , "toppings" : toppings}, "finalize" async def finalize_order ( args : FlowArgs) -> tuple[FlowResult, str ]: """Process the complete order.""" order = flow_manager.state.get( "order" , {}) # Validate order has required information if "size" not in order: return { "status" : "error" , "error" : "No size selected" } # Calculate price based on stored selections size = order[ "size" ] toppings = order.get( "toppings" , []) price = calculate_price(size, len (toppings)) return { "status" : "success" , "summary" : f "Ordered: { size } pizza with { ', ' .join(toppings) } " , "price" : price }, "end" In this example: select_size initializes the order and stores the size select_toppings builds a list of toppings finalize_order uses the stored information to process the complete order The state variable makes it easy to: Build up information across multiple interactions Access previous choices when needed Validate the complete order Calculate final results This is particularly useful when information needs to be collected across multiple conversation turns or when later decisions depend on earlier choices. ​ LLM Provider Support Pipecat Flows automatically handles format differences between LLM providers: ​ OpenAI Format Copy Ask AI "functions" : [{ "type" : "function" , "function" : { "name" : "function_name" , "handler" : select_size, "description" : "description" , "parameters" : { ... } } }] ​ Anthropic Format Copy Ask AI "functions" : [{ "name" : "function_name" , "handler" : select_size, "description" : "description" , "input_schema" : { ... } }] ​ Google (Gemini) Format Copy Ask AI "functions" : [{ "function_declarations" : [{ "name" : "function_name" , "handler" : select_size, "description" : "description" , "parameters" : { ... } }] }] You don’t need to handle these differences manually - Pipecat Flows adapts your configuration to the correct format based on your LLM provider. ​ Implementation Approaches ​ Static Flows Static flows use a configuration-driven approach where the entire conversation structure is defined upfront. ​ Basic Setup Copy Ask AI from pipecat_flows import FlowManager # Define flow configuration flow_config = { "initial_node" : "greeting" , "nodes" : { "greeting" : { "role_messages" : [ ... ], "task_messages" : [ ... ], "functions" : [ ... ] } } } # Initialize flow manager with static configuration flow_manager = FlowManager( task = task, llm = llm, context_aggregator = context_aggregator, flow_config = flow_config ) @transport.event_handler ( "on_first_participant_joined" ) async def on_first_participant_joined ( transport , participant ): await transport.capture_participant_transcription(participant[ "id" ]) await flow_manager.initialize() ​ Example FlowConfig Copy Ask AI flow_config = { "initial_node" : "start" , "nodes" : { "start" : { "role_messages" : [ { "role" : "system" , "content" : "You are an order-taking assistant. You must ALWAYS use the available functions to progress the conversation. This is a phone conversation and your responses will be converted to audio. Keep the conversation friendly, casual, and polite. Avoid outputting special characters and emojis." , } ], "task_messages" : [ { "role" : "system" , "content" : "You are an order-taking assistant. Ask if they want pizza or sushi." } ], "functions" : [ { "type" : "function" , "function" : { "name" : "choose_pizza" , "handler" : choose_pizza, # Returns [None, "pizza_order"] "description" : "User wants pizza" , "parameters" : { "type" : "object" , "properties" : {}} } } ] }, "pizza_order" : { "task_messages" : [ ... ], "functions" : [ { "type" : "function" , "function" : { "name" : "select_size" , "handler" : select_size, # Returns [FlowResult, "toppings"] "description" : "Select pizza size" , "parameters" : { "type" : "object" , "properties" : { "size" : { "type" : "string" , "enum" : [ "small" , "medium" , "large" ]} } } } } ] } } } ​ Dynamic Flows Dynamic flows create and modify conversation paths at runtime based on data or business logic. ​ Example Implementation Here’s a complete example of a dynamic insurance quote flow: Copy Ask AI from pipecat_flows import FlowManager, FlowArgs, FlowResult # Define handlers and transitions async def collect_age ( args : FlowArgs, flow_manager : FlowManager) -> tuple[AgeResult, NodeConfig]: """Process age collection.""" age = args[ "age" ] # Assemble result result = AgeResult( status = "success" , age = age) # Decide which node to go to next if age < 25 : await flow_manager.set_node_from_config(create_young_adult_node()) else : await flow_manager.set_node_from_config(create_standard_node()) return result, age # Node creation functions def create_initial_node () -> NodeConfig: """Create initial age collection node.""" return { "name" : "initial" , "role_messages" : [ { "role" : "system" , "content" : "You are an insurance quote assistant." } ], "task_messages" : [ { "role" : "system" , "content" : "Ask for the customer's age." } ], "functions" : [ { "type" : "function" , "function" : { "name" : "collect_age" , "handler" : collect_age, "description" : "Collect customer age" , "parameters" : { "type" : "object" , "properties" : { "age" : { "type" : "integer" } } } } } ] } def create_young_adult_node () -> Dict[ str , Any]: """Create node for young adult quotes.""" return { "name" : "young_adult" , "task_messages" : [ { "role" : "system" , "content" : "Explain our special young adult coverage options." } ], "functions" : [ ... ] # Additional quote-specific functions } def create_standard_node () -> Dict[ str , Any]: """Create node for standard quotes.""" return { "name" : "standard" , "task_messages" : [ { "role" : "system" , "content" : "Present our standard coverage options." } ], "functions" : [ ... ] # Additional quote-specific functions } # Initialize flow manager flow_manager = FlowManager( task = task, llm = llm, context_aggregator = context_aggregator, ) @transport.event_handler ( "on_first_participant_joined" ) async def on_first_participant_joined ( transport , participant ): await transport.capture_participant_transcription(participant[ "id" ]) await flow_manager.initialize(create_initial_node()) ​ Best Practices Store shared data in flow_manager.state Create separate functions for node creation ​ Flow Editor The Pipecat Flow Editor provides a visual interface for creating and managing conversation flows. It offers a node-based interface that makes it easier to design, visualize, and modify your flows. ​ Visual Design ​ Node Types Start Node (Green): Entry point of your flow Copy Ask AI "greeting" : { "role_messages" : [ ... ], "task_messages" : [ ... ], "functions" : [ ... ] } Flow Nodes (Blue): Intermediate states Copy Ask AI "collect_info" : { "task_messages" : [ ... ], "functions" : [ ... ], "pre_actions" : [ ... ] } End Node (Red): Final state Copy Ask AI "end" : { "task_messages" : [ ... ], "functions" : [], "post_actions" : [{ "type" : "end_conversation" }] } Function Nodes : Edge Functions (Purple): Create transitions Copy Ask AI { "name" : "next_node" , "description" : "Transition to next state" } Node Functions (Orange): Perform operations Copy Ask AI { "name" : "process_data" , "handler" : process_data_handler, "description" : "Process user data" } ​ Naming Conventions Start Node : Use descriptive names (e.g., “greeting”, “welcome”) Flow Nodes : Name based on purpose (e.g., “collect_info”, “verify_data”) End Node : Conventionally named “end” Functions : Use clear, action-oriented names ​ Function Configuration Copy Ask AI { "type" : "function" , "function" : { "name" : "process_data" , "handler" : process_handler, "description" : "Process user data" , "parameters" : { ... } } } When using the Flow Editor, function handlers can be specified using the __function__: token: Copy Ask AI { "type" : "function" , "function" : { "name" : "process_data" , "handler" : "__function__:process_data" , # References function in main script "description" : "Process user data" , "parameters" : { ... } } } The handler will be looked up in your main script when the flow is executed. When function handlers are specified in the flow editor, they will be exported with the __function__: token. ​ Using the Editor ​ Creating a New Flow Start with a descriptively named Start Node Add Flow Nodes for each conversation state Connect nodes using Edge Functions Add Node Functions for operations Include an End Node ​ Import/Export Copy Ask AI # Export format { "initial_node" : "greeting" , "nodes" : { "greeting" : { "role_messages" : [ ... ], "task_messages" : [ ... ], "functions" : [ ... ], "pre_actions" : [ ... ] }, "process" : { "task_messages" : [ ... ], "functions" : [ ... ], }, "end" : { "task_messages" : [ ... ], "functions" : [], "post_actions" : [ ... ] } } } ​ Tips Use the visual preview to verify flow logic Test exported configurations Document node purposes and transitions Keep flows modular and maintainable Try the editor at flows.pipecat.ai OpenAI Audio Models and APIs Overview On this page Key Concepts Example Flows When to Use Static vs Dynamic Flows Installation Core Concepts Designing Conversation Flows Defining a Function Example Function Node Structure Messages Functions Function Configuration Node Functions Edge Functions Actions Pre-Actions Post-Actions Timing Considerations Action Types Deciding Who Speaks First Context Management Context Strategies Configuration Strategy Selection State Management LLM Provider Support OpenAI Format Anthropic Format Google (Gemini) Format Implementation Approaches Static Flows Basic Setup Example FlowConfig Dynamic Flows Example Implementation Best Practices Flow Editor Visual Design Node Types Naming Conventions Function Configuration Using the Editor Creating a New Flow Import/Export Tips Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/features_pipecat-flows_8f93443a.txt b/features_pipecat-flows_8f93443a.txt
new file mode 100644
index 0000000000000000000000000000000000000000..afa9e1d535b64996f6bab6202723e6c575f90f5d
--- /dev/null
+++ b/features_pipecat-flows_8f93443a.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/guides/features/pipecat-flows#node-structure
+Title: Pipecat Flows - Pipecat
+==================================================
+
+Pipecat Flows - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Features Pipecat Flows Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Guides Fundamentals Context Management Custom FrameProcessor Detecting Idle Users Ending a Pipeline Function Calling Muting User Input Recording Audio Recording Transcripts Features Gemini Multimodal Live Metrics Noise cancellation with Krisp OpenAI Audio Models and APIs Pipecat Flows Telephony Overview Dial-in: WebRTC (Daily) Dial-in: WebRTC (Twilio + Daily) Dial-in: Twilio (Media Streams) Dialout: WebRTC (Daily) Deploying your bot Overview Deployment pattern Example: Pipecat Cloud Example: Fly.io Example: Cerebrium Example: Modal Pipecat Flows provides a framework for building structured conversations in your AI applications. It enables you to create both predefined conversation paths and dynamically generated flows while handling the complexities of state management and LLM interactions. The framework consists of: A Python module for building conversation flows with Pipecat A visual editor for designing and exporting flow configurations ​ Key Concepts Nodes : Represent conversation states with specific messages and available functions Messages : Set the role and tasks for each node Functions : Define actions and transitions (Node functions for operations, Edge functions for transitions) Actions : Execute operations during state transitions (pre/post actions) State Management : Handle conversation state and data persistence ​ Example Flows Movie Explorer (Static) A static flow demonstrating movie exploration using OpenAI. Shows real API integration with TMDB, structured data collection, and state management. Insurance Policy (Dynamic) A dynamic flow using Google Gemini that adapts policy recommendations based on user responses. Demonstrates runtime node creation and conditional paths. These examples are fully functional and can be run locally. Make sure you have the required dependencies installed and API keys configured. ​ When to Use Static vs Dynamic Flows Static Flows are ideal when: Conversation structure is known upfront Paths follow predefined patterns Flow can be fully configured in advance Example: Customer service scripts, intake forms Dynamic Flows are better when: Paths depend on external data Flow structure needs runtime modification Complex decision trees are involved Example: Personalized recommendations, adaptive workflows ​ Installation If you’re already using Pipecat: Copy Ask AI pip install pipecat-ai-flows If you’re starting fresh: Copy Ask AI # Basic installation pip install pipecat-ai-flows # Install Pipecat with specific LLM provider options: pip install "pipecat-ai[daily,openai,deepgram]" # For OpenAI pip install "pipecat-ai[daily,anthropic,deepgram]" # For Anthropic pip install "pipecat-ai[daily,google,deepgram]" # For Google 💡 Want to design your flows visually? Try the online Flow Editor ​ Core Concepts ​ Designing Conversation Flows Functions in Pipecat Flows serve two key purposes: Processing data (likely by interfacing with external systems and APIs) Advancing the conversation to the next node Each function can do one or both. LLMs decide when to run each function, via their function calling (or tool calling) mechanism. ​ Defining a Function A function is expected to return a (result, next_node) tuple. More precisely, it’s expected to return: Copy Ask AI # (result, next_node) Tuple[Optional[FlowResult], Optional[Union[NodeConfig, str ]]] If the function processes data, it should return a non- None value for the first element of the tuple. This value should be a FlowResult or subclass. If the function advances the conversation to the next node, it should return a non- None value for the second element of the tuple. This value can be either: A NodeConfig defining the next node (for dynamic flows) A string identifying the next node (for static flows) ​ Example Function Copy Ask AI from pipecat_flows import FlowArgs, FlowManager, FlowResult, NodeConfig async def check_availability ( args : FlowArgs, flow_manager : FlowManager) -> tuple[FlowResult, NodeConfig]: # Read arguments date = args[ "date" ] time = args[ "time" ] # Read previously-stored data party_size = flow_manager.state.get( "party_size" ) # Use flow_manager for immediate user feedback await flow_manager.task.queue_frame(TTSSpeakFrame( "Checking our reservation system..." )) # Store data in flow state for later use flow_manager.state[ "requested_date" ] = date # Interface with reservation system is_available = await reservation_system.check_availability(date, time, party_size) # Assemble result result = { "status" : "success" , "available" : available } # Decide which node to go to next if is_available: next_node = create_confirmation_node() else : next_node = create_no_availability_node() # Return both result and next node return result, next_node ​ Node Structure Each node in your flow represents a conversation state and consists of three main components: ​ Messages Nodes use two types of messages to control the conversation: Role Messages : Define the bot’s personality or role (optional) Copy Ask AI "role_messages" : [ { "role" : "system" , "content" : "You are a friendly pizza ordering assistant. Keep responses casual and upbeat." } ] Task Messages : Define what the bot should do in the current node Copy Ask AI "task_messages" : [ { "role" : "system" , "content" : "Ask the customer which pizza size they'd like: small, medium, or large." } ] Role messages are typically defined in your initial node and inherited by subsequent nodes, while task messages are specific to each node’s purpose. ​ Functions Functions in Pipecat Flows can: Process data Specify node transitions Do both This leads to two conceptual types of functions: Node functions , which only process data. Edge functions , which also (or only) transition to the next node. The function itself ( which you can read more about here ) is usually wrapped in a function configuration, which also contains some metadata about the function. ​ Function Configuration Pipecat Flows supports three ways of specifying function configuration: Provider-specific dictionary format Copy Ask AI # Dictionary format { "type" : "function" , "function" : { "name" : "select_size" , "handler" : select_size, "description" : "Select pizza size" , "parameters" : { "type" : "object" , "properties" : { "size" : { "type" : "string" , "enum" : [ "small" , "medium" , "large" ]} } }, } } FlowsFunctionSchema Copy Ask AI # Using FlowsFunctionSchema from pipecat_flows import FlowsFunctionSchema size_function = FlowsFunctionSchema( name = "select_size" , description = "Select pizza size" , properties = { "size" : { "type" : "string" , "enum" : [ "small" , "medium" , "large" ]} }, required = [ "size" ], handler = select_size ) # Use in node configuration node_config = { "task_messages" : [ ... ], "functions" : [size_function] } The FlowsFunctionSchema approach provides some advantages over the provider-specific dictionary format: Consistent structure across LLM providers Simplified parameter definition Cleaner, more readable code Both dictionary and FlowsFunctionSchema approaches are fully supported. FlowsFunctionSchema is recommended for new projects as it provides better type checking and a provider-independent format. Direct function usage (auto-configuration) This approach lets you bypass specifying a standalone function configuration. Instead, relevant function metadata is automatically extracted from the function’s signature and docstring: name description properties (including individual property description s) required Note that the function signature is a bit different when using direct functions. The first parameter is the FlowManager , followed by any others necessary for the function. Copy Ask AI from pipecat_flows import FlowManager, FlowResult async def select_pizza_order ( flow_manager : FlowManager, size : str , pizza_type : str , additional_toppings : list[ str ] = [], ) -> tuple[FlowResult, str ]: """ Record the pizza order details. Args: size (str): Size of the pizza. Must be one of "small", "medium", or "large". pizza_type (str): Type of pizza. Must be one of "pepperoni", "cheese", "supreme", or "vegetarian". additional_toppings (list[str]): List of additional toppings. Defaults to empty list. """ ... # Use in node configuration node_config = { "task_messages" : [ ... ], "functions" : [select_pizza_order] } ​ Node Functions Functions that process data within a single conversational state, without switching nodes. When called, they: Execute their handler to do the data processing (typically by interfacing with an external system or API) Trigger an immediate LLM completion with the result Copy Ask AI from pipecat_flows import FlowArgs, FlowManager, FlowResult async def select_size ( args : FlowArgs, flow_manager : FlowManager) -> tuple[FlowResult, None ]: """Process pizza size selection.""" size = args[ "size" ] await ordering_system.record_size_selection(size) return { "status" : "success" , "size" : size }, None # Function configuration { "type" : "function" , "function" : { "name" : "select_size" , "handler" : select_size, "description" : "Select pizza size" , "parameters" : { "type" : "object" , "properties" : { "size" : { "type" : "string" , "enum" : [ "small" , "medium" , "large" ]} } }, } } ​ Edge Functions Functions that specify a transition between nodes (optionally processing data first). When called, they: Execute their handler to do any data processing (optional) and determine the next node Add the function result to the LLM context Trigger LLM completion after both the function result and the next node’s messages are in the context Copy Ask AI from pipecat_flows import FlowArgs, FlowManager, FlowResult async def select_size ( args : FlowArgs, flow_manager : FlowManager) -> tuple[FlowResult, NodeConfig]: """Process pizza size selection.""" size = args[ "size" ] await ordering_system.record_size_selection(size) result = { "status" : "success" , "size" : size } next_node = create_confirmation_node() return result, next_node # Function configuration { "type" : "function" , "function" : { "name" : "select_size" , "handler" : select_size, "description" : "Select pizza size" , "parameters" : { "type" : "object" , "properties" : { "size" : { "type" : "string" , "enum" : [ "small" , "medium" , "large" ]} } }, } } ​ Actions Actions are operations that execute as part of the lifecycle of a node, with two distinct timing options: Pre-actions: execute when entering the node, before the LLM completion Post-actions: execute after the LLM completion ​ Pre-Actions Execute when entering the node, before LLM inference. Useful for: Providing immediate feedback while waiting for LLM responses Bridging gaps during longer function calls Setting up state or context Copy Ask AI "pre_actions" : [ { "type" : "tts_say" , "text" : "Hold on a moment..." # Immediate feedback during processing } ], Note that when the node is configured with respond_immediately: False , the pre_actions still run when entering the node, which may be well before LLM inference, depending on how long the user takes to speak first. Avoid mixing tts_say actions with chat completions as this may result in a conversation flow that feels unnatural. tts_say are best used as filler words when the LLM will take time to generate an completion. ​ Post-Actions Execute after LLM inference completes. Useful for: Cleanup operations State finalization Ensuring proper sequence of operations Copy Ask AI "post_actions" : [ { "type" : "end_conversation" # Ensures TTS completes before ending } ] Note that when the node is configured with respond_immediately: False , the post_actions still only run after the first LLM inference, which may be a while depending on how long the user takes to speak first. ​ Timing Considerations Pre-actions : Execute immediately, before any LLM processing begins LLM Inference : Processes the node’s messages and functions Post-actions : Execute after LLM processing and TTS completion For example, when using end_conversation as a post-action, the sequence is: LLM generates response TTS speaks the response End conversation action executes This ordering ensures proper completion of all operations. ​ Action Types Flows comes equipped with pre-canned actions and you can also define your own action behavior. See the reference docs for more information. ​ Deciding Who Speaks First For each node in the conversation, you can decide whether the LLM should respond immediately upon entering the node (the default behavior) or whether the LLM should wait for the user to speak first before responding. You do this using the respond_immediately field. respond_immediately=False may be particularly useful in the very first node, especially in outbound-calling cases where the user has to first answer the phone to trigger the conversation. Copy Ask AI NodeConfig( task_messages = [ { "role" : "system" , "content" : "Warmly greet the customer and ask how many people are in their party. This is your only job for now; if the customer asks for something else, politely remind them you can't do it." , } ], respond_immediately = False , # ... other fields ) Keep in mind that if you specify respond_immediately=False , the user may not be aware of the conversational task at hand when entering the node (the bot hasn’t told them yet). While it’s always important to have guardrails in your node messages to keep the conversation on topic, letting the user speak first makes it even more so. ​ Context Management Pipecat Flows provides three strategies for managing conversation context during node transitions: ​ Context Strategies APPEND (default): Adds new messages to the existing context, maintaining the full conversation history RESET : Clears the context and starts fresh with the new node’s messages RESET_WITH_SUMMARY : Resets the context but includes an AI-generated summary of the previous conversation ​ Configuration Context strategies can be configured globally or per-node: Copy Ask AI from pipecat_flows import ContextStrategy, ContextStrategyConfig # Global strategy configuration flow_manager = FlowManager( task = task, llm = llm, context_aggregator = context_aggregator, context_strategy = ContextStrategyConfig( strategy = ContextStrategy. RESET_WITH_SUMMARY , summary_prompt = "Summarize the key points discussed so far, focusing on decisions made and important information collected." ) ) # Per-node strategy configuration node_config = { "task_messages" : [ ... ], "functions" : [ ... ], "context_strategy" : ContextStrategyConfig( strategy = ContextStrategy. RESET_WITH_SUMMARY , summary_prompt = "Provide a concise summary of the customer's order details and preferences." ) } ​ Strategy Selection Choose your strategy based on your conversation needs: Use APPEND when full conversation history is important Use RESET when previous context might confuse the current node’s purpose Use RESET_WITH_SUMMARY for long conversations where key points need to be preserved When using RESET_WITH_SUMMARY, if summary generation fails or times out, the system automatically falls back to RESET strategy for resilience. ​ State Management The state variable in FlowManager is a shared dictionary that persists throughout the conversation. Think of it as a conversation memory that lets you: Store user information Track conversation progress Share data between nodes Inform decision-making Here’s a practical example of a pizza ordering flow: Copy Ask AI # Store user choices as they're made async def select_size ( args : FlowArgs) -> tuple[FlowResult, str ]: """Handle pizza size selection.""" size = args[ "size" ] # Initialize order in state if it doesn't exist if "order" not in flow_manager.state: flow_manager.state[ "order" ] = {} # Store the selection flow_manager.state[ "order" ][ "size" ] = size return { "status" : "success" , "size" : size}, "toppings" async def select_toppings ( args : FlowArgs) -> tuple[FlowResult, str ]: """Handle topping selection.""" topping = args[ "topping" ] # Get existing order and toppings order = flow_manager.state.get( "order" , {}) toppings = order.get( "toppings" , []) # Add new topping toppings.append(topping) order[ "toppings" ] = toppings flow_manager.state[ "order" ] = order return { "status" : "success" , "toppings" : toppings}, "finalize" async def finalize_order ( args : FlowArgs) -> tuple[FlowResult, str ]: """Process the complete order.""" order = flow_manager.state.get( "order" , {}) # Validate order has required information if "size" not in order: return { "status" : "error" , "error" : "No size selected" } # Calculate price based on stored selections size = order[ "size" ] toppings = order.get( "toppings" , []) price = calculate_price(size, len (toppings)) return { "status" : "success" , "summary" : f "Ordered: { size } pizza with { ', ' .join(toppings) } " , "price" : price }, "end" In this example: select_size initializes the order and stores the size select_toppings builds a list of toppings finalize_order uses the stored information to process the complete order The state variable makes it easy to: Build up information across multiple interactions Access previous choices when needed Validate the complete order Calculate final results This is particularly useful when information needs to be collected across multiple conversation turns or when later decisions depend on earlier choices. ​ LLM Provider Support Pipecat Flows automatically handles format differences between LLM providers: ​ OpenAI Format Copy Ask AI "functions" : [{ "type" : "function" , "function" : { "name" : "function_name" , "handler" : select_size, "description" : "description" , "parameters" : { ... } } }] ​ Anthropic Format Copy Ask AI "functions" : [{ "name" : "function_name" , "handler" : select_size, "description" : "description" , "input_schema" : { ... } }] ​ Google (Gemini) Format Copy Ask AI "functions" : [{ "function_declarations" : [{ "name" : "function_name" , "handler" : select_size, "description" : "description" , "parameters" : { ... } }] }] You don’t need to handle these differences manually - Pipecat Flows adapts your configuration to the correct format based on your LLM provider. ​ Implementation Approaches ​ Static Flows Static flows use a configuration-driven approach where the entire conversation structure is defined upfront. ​ Basic Setup Copy Ask AI from pipecat_flows import FlowManager # Define flow configuration flow_config = { "initial_node" : "greeting" , "nodes" : { "greeting" : { "role_messages" : [ ... ], "task_messages" : [ ... ], "functions" : [ ... ] } } } # Initialize flow manager with static configuration flow_manager = FlowManager( task = task, llm = llm, context_aggregator = context_aggregator, flow_config = flow_config ) @transport.event_handler ( "on_first_participant_joined" ) async def on_first_participant_joined ( transport , participant ): await transport.capture_participant_transcription(participant[ "id" ]) await flow_manager.initialize() ​ Example FlowConfig Copy Ask AI flow_config = { "initial_node" : "start" , "nodes" : { "start" : { "role_messages" : [ { "role" : "system" , "content" : "You are an order-taking assistant. You must ALWAYS use the available functions to progress the conversation. This is a phone conversation and your responses will be converted to audio. Keep the conversation friendly, casual, and polite. Avoid outputting special characters and emojis." , } ], "task_messages" : [ { "role" : "system" , "content" : "You are an order-taking assistant. Ask if they want pizza or sushi." } ], "functions" : [ { "type" : "function" , "function" : { "name" : "choose_pizza" , "handler" : choose_pizza, # Returns [None, "pizza_order"] "description" : "User wants pizza" , "parameters" : { "type" : "object" , "properties" : {}} } } ] }, "pizza_order" : { "task_messages" : [ ... ], "functions" : [ { "type" : "function" , "function" : { "name" : "select_size" , "handler" : select_size, # Returns [FlowResult, "toppings"] "description" : "Select pizza size" , "parameters" : { "type" : "object" , "properties" : { "size" : { "type" : "string" , "enum" : [ "small" , "medium" , "large" ]} } } } } ] } } } ​ Dynamic Flows Dynamic flows create and modify conversation paths at runtime based on data or business logic. ​ Example Implementation Here’s a complete example of a dynamic insurance quote flow: Copy Ask AI from pipecat_flows import FlowManager, FlowArgs, FlowResult # Define handlers and transitions async def collect_age ( args : FlowArgs, flow_manager : FlowManager) -> tuple[AgeResult, NodeConfig]: """Process age collection.""" age = args[ "age" ] # Assemble result result = AgeResult( status = "success" , age = age) # Decide which node to go to next if age < 25 : await flow_manager.set_node_from_config(create_young_adult_node()) else : await flow_manager.set_node_from_config(create_standard_node()) return result, age # Node creation functions def create_initial_node () -> NodeConfig: """Create initial age collection node.""" return { "name" : "initial" , "role_messages" : [ { "role" : "system" , "content" : "You are an insurance quote assistant." } ], "task_messages" : [ { "role" : "system" , "content" : "Ask for the customer's age." } ], "functions" : [ { "type" : "function" , "function" : { "name" : "collect_age" , "handler" : collect_age, "description" : "Collect customer age" , "parameters" : { "type" : "object" , "properties" : { "age" : { "type" : "integer" } } } } } ] } def create_young_adult_node () -> Dict[ str , Any]: """Create node for young adult quotes.""" return { "name" : "young_adult" , "task_messages" : [ { "role" : "system" , "content" : "Explain our special young adult coverage options." } ], "functions" : [ ... ] # Additional quote-specific functions } def create_standard_node () -> Dict[ str , Any]: """Create node for standard quotes.""" return { "name" : "standard" , "task_messages" : [ { "role" : "system" , "content" : "Present our standard coverage options." } ], "functions" : [ ... ] # Additional quote-specific functions } # Initialize flow manager flow_manager = FlowManager( task = task, llm = llm, context_aggregator = context_aggregator, ) @transport.event_handler ( "on_first_participant_joined" ) async def on_first_participant_joined ( transport , participant ): await transport.capture_participant_transcription(participant[ "id" ]) await flow_manager.initialize(create_initial_node()) ​ Best Practices Store shared data in flow_manager.state Create separate functions for node creation ​ Flow Editor The Pipecat Flow Editor provides a visual interface for creating and managing conversation flows. It offers a node-based interface that makes it easier to design, visualize, and modify your flows. ​ Visual Design ​ Node Types Start Node (Green): Entry point of your flow Copy Ask AI "greeting" : { "role_messages" : [ ... ], "task_messages" : [ ... ], "functions" : [ ... ] } Flow Nodes (Blue): Intermediate states Copy Ask AI "collect_info" : { "task_messages" : [ ... ], "functions" : [ ... ], "pre_actions" : [ ... ] } End Node (Red): Final state Copy Ask AI "end" : { "task_messages" : [ ... ], "functions" : [], "post_actions" : [{ "type" : "end_conversation" }] } Function Nodes : Edge Functions (Purple): Create transitions Copy Ask AI { "name" : "next_node" , "description" : "Transition to next state" } Node Functions (Orange): Perform operations Copy Ask AI { "name" : "process_data" , "handler" : process_data_handler, "description" : "Process user data" } ​ Naming Conventions Start Node : Use descriptive names (e.g., “greeting”, “welcome”) Flow Nodes : Name based on purpose (e.g., “collect_info”, “verify_data”) End Node : Conventionally named “end” Functions : Use clear, action-oriented names ​ Function Configuration Copy Ask AI { "type" : "function" , "function" : { "name" : "process_data" , "handler" : process_handler, "description" : "Process user data" , "parameters" : { ... } } } When using the Flow Editor, function handlers can be specified using the __function__: token: Copy Ask AI { "type" : "function" , "function" : { "name" : "process_data" , "handler" : "__function__:process_data" , # References function in main script "description" : "Process user data" , "parameters" : { ... } } } The handler will be looked up in your main script when the flow is executed. When function handlers are specified in the flow editor, they will be exported with the __function__: token. ​ Using the Editor ​ Creating a New Flow Start with a descriptively named Start Node Add Flow Nodes for each conversation state Connect nodes using Edge Functions Add Node Functions for operations Include an End Node ​ Import/Export Copy Ask AI # Export format { "initial_node" : "greeting" , "nodes" : { "greeting" : { "role_messages" : [ ... ], "task_messages" : [ ... ], "functions" : [ ... ], "pre_actions" : [ ... ] }, "process" : { "task_messages" : [ ... ], "functions" : [ ... ], }, "end" : { "task_messages" : [ ... ], "functions" : [], "post_actions" : [ ... ] } } } ​ Tips Use the visual preview to verify flow logic Test exported configurations Document node purposes and transitions Keep flows modular and maintainable Try the editor at flows.pipecat.ai OpenAI Audio Models and APIs Overview On this page Key Concepts Example Flows When to Use Static vs Dynamic Flows Installation Core Concepts Designing Conversation Flows Defining a Function Example Function Node Structure Messages Functions Function Configuration Node Functions Edge Functions Actions Pre-Actions Post-Actions Timing Considerations Action Types Deciding Who Speaks First Context Management Context Strategies Configuration Strategy Selection State Management LLM Provider Support OpenAI Format Anthropic Format Google (Gemini) Format Implementation Approaches Static Flows Basic Setup Example FlowConfig Dynamic Flows Example Implementation Best Practices Flow Editor Visual Design Node Types Naming Conventions Function Configuration Using the Editor Creating a New Flow Import/Export Tips Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/features_pipecat-flows_c3b1bdd4.txt b/features_pipecat-flows_c3b1bdd4.txt
new file mode 100644
index 0000000000000000000000000000000000000000..57fec53222fc613db345c1b9444e594306f06309
--- /dev/null
+++ b/features_pipecat-flows_c3b1bdd4.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/guides/features/pipecat-flows#example-flowconfig
+Title: Pipecat Flows - Pipecat
+==================================================
+
+Pipecat Flows - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Features Pipecat Flows Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Guides Fundamentals Context Management Custom FrameProcessor Detecting Idle Users Ending a Pipeline Function Calling Muting User Input Recording Audio Recording Transcripts Features Gemini Multimodal Live Metrics Noise cancellation with Krisp OpenAI Audio Models and APIs Pipecat Flows Telephony Overview Dial-in: WebRTC (Daily) Dial-in: WebRTC (Twilio + Daily) Dial-in: Twilio (Media Streams) Dialout: WebRTC (Daily) Deploying your bot Overview Deployment pattern Example: Pipecat Cloud Example: Fly.io Example: Cerebrium Example: Modal Pipecat Flows provides a framework for building structured conversations in your AI applications. It enables you to create both predefined conversation paths and dynamically generated flows while handling the complexities of state management and LLM interactions. The framework consists of: A Python module for building conversation flows with Pipecat A visual editor for designing and exporting flow configurations ​ Key Concepts Nodes : Represent conversation states with specific messages and available functions Messages : Set the role and tasks for each node Functions : Define actions and transitions (Node functions for operations, Edge functions for transitions) Actions : Execute operations during state transitions (pre/post actions) State Management : Handle conversation state and data persistence ​ Example Flows Movie Explorer (Static) A static flow demonstrating movie exploration using OpenAI. Shows real API integration with TMDB, structured data collection, and state management. Insurance Policy (Dynamic) A dynamic flow using Google Gemini that adapts policy recommendations based on user responses. Demonstrates runtime node creation and conditional paths. These examples are fully functional and can be run locally. Make sure you have the required dependencies installed and API keys configured. ​ When to Use Static vs Dynamic Flows Static Flows are ideal when: Conversation structure is known upfront Paths follow predefined patterns Flow can be fully configured in advance Example: Customer service scripts, intake forms Dynamic Flows are better when: Paths depend on external data Flow structure needs runtime modification Complex decision trees are involved Example: Personalized recommendations, adaptive workflows ​ Installation If you’re already using Pipecat: Copy Ask AI pip install pipecat-ai-flows If you’re starting fresh: Copy Ask AI # Basic installation pip install pipecat-ai-flows # Install Pipecat with specific LLM provider options: pip install "pipecat-ai[daily,openai,deepgram]" # For OpenAI pip install "pipecat-ai[daily,anthropic,deepgram]" # For Anthropic pip install "pipecat-ai[daily,google,deepgram]" # For Google 💡 Want to design your flows visually? Try the online Flow Editor ​ Core Concepts ​ Designing Conversation Flows Functions in Pipecat Flows serve two key purposes: Processing data (likely by interfacing with external systems and APIs) Advancing the conversation to the next node Each function can do one or both. LLMs decide when to run each function, via their function calling (or tool calling) mechanism. ​ Defining a Function A function is expected to return a (result, next_node) tuple. More precisely, it’s expected to return: Copy Ask AI # (result, next_node) Tuple[Optional[FlowResult], Optional[Union[NodeConfig, str ]]] If the function processes data, it should return a non- None value for the first element of the tuple. This value should be a FlowResult or subclass. If the function advances the conversation to the next node, it should return a non- None value for the second element of the tuple. This value can be either: A NodeConfig defining the next node (for dynamic flows) A string identifying the next node (for static flows) ​ Example Function Copy Ask AI from pipecat_flows import FlowArgs, FlowManager, FlowResult, NodeConfig async def check_availability ( args : FlowArgs, flow_manager : FlowManager) -> tuple[FlowResult, NodeConfig]: # Read arguments date = args[ "date" ] time = args[ "time" ] # Read previously-stored data party_size = flow_manager.state.get( "party_size" ) # Use flow_manager for immediate user feedback await flow_manager.task.queue_frame(TTSSpeakFrame( "Checking our reservation system..." )) # Store data in flow state for later use flow_manager.state[ "requested_date" ] = date # Interface with reservation system is_available = await reservation_system.check_availability(date, time, party_size) # Assemble result result = { "status" : "success" , "available" : available } # Decide which node to go to next if is_available: next_node = create_confirmation_node() else : next_node = create_no_availability_node() # Return both result and next node return result, next_node ​ Node Structure Each node in your flow represents a conversation state and consists of three main components: ​ Messages Nodes use two types of messages to control the conversation: Role Messages : Define the bot’s personality or role (optional) Copy Ask AI "role_messages" : [ { "role" : "system" , "content" : "You are a friendly pizza ordering assistant. Keep responses casual and upbeat." } ] Task Messages : Define what the bot should do in the current node Copy Ask AI "task_messages" : [ { "role" : "system" , "content" : "Ask the customer which pizza size they'd like: small, medium, or large." } ] Role messages are typically defined in your initial node and inherited by subsequent nodes, while task messages are specific to each node’s purpose. ​ Functions Functions in Pipecat Flows can: Process data Specify node transitions Do both This leads to two conceptual types of functions: Node functions , which only process data. Edge functions , which also (or only) transition to the next node. The function itself ( which you can read more about here ) is usually wrapped in a function configuration, which also contains some metadata about the function. ​ Function Configuration Pipecat Flows supports three ways of specifying function configuration: Provider-specific dictionary format Copy Ask AI # Dictionary format { "type" : "function" , "function" : { "name" : "select_size" , "handler" : select_size, "description" : "Select pizza size" , "parameters" : { "type" : "object" , "properties" : { "size" : { "type" : "string" , "enum" : [ "small" , "medium" , "large" ]} } }, } } FlowsFunctionSchema Copy Ask AI # Using FlowsFunctionSchema from pipecat_flows import FlowsFunctionSchema size_function = FlowsFunctionSchema( name = "select_size" , description = "Select pizza size" , properties = { "size" : { "type" : "string" , "enum" : [ "small" , "medium" , "large" ]} }, required = [ "size" ], handler = select_size ) # Use in node configuration node_config = { "task_messages" : [ ... ], "functions" : [size_function] } The FlowsFunctionSchema approach provides some advantages over the provider-specific dictionary format: Consistent structure across LLM providers Simplified parameter definition Cleaner, more readable code Both dictionary and FlowsFunctionSchema approaches are fully supported. FlowsFunctionSchema is recommended for new projects as it provides better type checking and a provider-independent format. Direct function usage (auto-configuration) This approach lets you bypass specifying a standalone function configuration. Instead, relevant function metadata is automatically extracted from the function’s signature and docstring: name description properties (including individual property description s) required Note that the function signature is a bit different when using direct functions. The first parameter is the FlowManager , followed by any others necessary for the function. Copy Ask AI from pipecat_flows import FlowManager, FlowResult async def select_pizza_order ( flow_manager : FlowManager, size : str , pizza_type : str , additional_toppings : list[ str ] = [], ) -> tuple[FlowResult, str ]: """ Record the pizza order details. Args: size (str): Size of the pizza. Must be one of "small", "medium", or "large". pizza_type (str): Type of pizza. Must be one of "pepperoni", "cheese", "supreme", or "vegetarian". additional_toppings (list[str]): List of additional toppings. Defaults to empty list. """ ... # Use in node configuration node_config = { "task_messages" : [ ... ], "functions" : [select_pizza_order] } ​ Node Functions Functions that process data within a single conversational state, without switching nodes. When called, they: Execute their handler to do the data processing (typically by interfacing with an external system or API) Trigger an immediate LLM completion with the result Copy Ask AI from pipecat_flows import FlowArgs, FlowManager, FlowResult async def select_size ( args : FlowArgs, flow_manager : FlowManager) -> tuple[FlowResult, None ]: """Process pizza size selection.""" size = args[ "size" ] await ordering_system.record_size_selection(size) return { "status" : "success" , "size" : size }, None # Function configuration { "type" : "function" , "function" : { "name" : "select_size" , "handler" : select_size, "description" : "Select pizza size" , "parameters" : { "type" : "object" , "properties" : { "size" : { "type" : "string" , "enum" : [ "small" , "medium" , "large" ]} } }, } } ​ Edge Functions Functions that specify a transition between nodes (optionally processing data first). When called, they: Execute their handler to do any data processing (optional) and determine the next node Add the function result to the LLM context Trigger LLM completion after both the function result and the next node’s messages are in the context Copy Ask AI from pipecat_flows import FlowArgs, FlowManager, FlowResult async def select_size ( args : FlowArgs, flow_manager : FlowManager) -> tuple[FlowResult, NodeConfig]: """Process pizza size selection.""" size = args[ "size" ] await ordering_system.record_size_selection(size) result = { "status" : "success" , "size" : size } next_node = create_confirmation_node() return result, next_node # Function configuration { "type" : "function" , "function" : { "name" : "select_size" , "handler" : select_size, "description" : "Select pizza size" , "parameters" : { "type" : "object" , "properties" : { "size" : { "type" : "string" , "enum" : [ "small" , "medium" , "large" ]} } }, } } ​ Actions Actions are operations that execute as part of the lifecycle of a node, with two distinct timing options: Pre-actions: execute when entering the node, before the LLM completion Post-actions: execute after the LLM completion ​ Pre-Actions Execute when entering the node, before LLM inference. Useful for: Providing immediate feedback while waiting for LLM responses Bridging gaps during longer function calls Setting up state or context Copy Ask AI "pre_actions" : [ { "type" : "tts_say" , "text" : "Hold on a moment..." # Immediate feedback during processing } ], Note that when the node is configured with respond_immediately: False , the pre_actions still run when entering the node, which may be well before LLM inference, depending on how long the user takes to speak first. Avoid mixing tts_say actions with chat completions as this may result in a conversation flow that feels unnatural. tts_say are best used as filler words when the LLM will take time to generate an completion. ​ Post-Actions Execute after LLM inference completes. Useful for: Cleanup operations State finalization Ensuring proper sequence of operations Copy Ask AI "post_actions" : [ { "type" : "end_conversation" # Ensures TTS completes before ending } ] Note that when the node is configured with respond_immediately: False , the post_actions still only run after the first LLM inference, which may be a while depending on how long the user takes to speak first. ​ Timing Considerations Pre-actions : Execute immediately, before any LLM processing begins LLM Inference : Processes the node’s messages and functions Post-actions : Execute after LLM processing and TTS completion For example, when using end_conversation as a post-action, the sequence is: LLM generates response TTS speaks the response End conversation action executes This ordering ensures proper completion of all operations. ​ Action Types Flows comes equipped with pre-canned actions and you can also define your own action behavior. See the reference docs for more information. ​ Deciding Who Speaks First For each node in the conversation, you can decide whether the LLM should respond immediately upon entering the node (the default behavior) or whether the LLM should wait for the user to speak first before responding. You do this using the respond_immediately field. respond_immediately=False may be particularly useful in the very first node, especially in outbound-calling cases where the user has to first answer the phone to trigger the conversation. Copy Ask AI NodeConfig( task_messages = [ { "role" : "system" , "content" : "Warmly greet the customer and ask how many people are in their party. This is your only job for now; if the customer asks for something else, politely remind them you can't do it." , } ], respond_immediately = False , # ... other fields ) Keep in mind that if you specify respond_immediately=False , the user may not be aware of the conversational task at hand when entering the node (the bot hasn’t told them yet). While it’s always important to have guardrails in your node messages to keep the conversation on topic, letting the user speak first makes it even more so. ​ Context Management Pipecat Flows provides three strategies for managing conversation context during node transitions: ​ Context Strategies APPEND (default): Adds new messages to the existing context, maintaining the full conversation history RESET : Clears the context and starts fresh with the new node’s messages RESET_WITH_SUMMARY : Resets the context but includes an AI-generated summary of the previous conversation ​ Configuration Context strategies can be configured globally or per-node: Copy Ask AI from pipecat_flows import ContextStrategy, ContextStrategyConfig # Global strategy configuration flow_manager = FlowManager( task = task, llm = llm, context_aggregator = context_aggregator, context_strategy = ContextStrategyConfig( strategy = ContextStrategy. RESET_WITH_SUMMARY , summary_prompt = "Summarize the key points discussed so far, focusing on decisions made and important information collected." ) ) # Per-node strategy configuration node_config = { "task_messages" : [ ... ], "functions" : [ ... ], "context_strategy" : ContextStrategyConfig( strategy = ContextStrategy. RESET_WITH_SUMMARY , summary_prompt = "Provide a concise summary of the customer's order details and preferences." ) } ​ Strategy Selection Choose your strategy based on your conversation needs: Use APPEND when full conversation history is important Use RESET when previous context might confuse the current node’s purpose Use RESET_WITH_SUMMARY for long conversations where key points need to be preserved When using RESET_WITH_SUMMARY, if summary generation fails or times out, the system automatically falls back to RESET strategy for resilience. ​ State Management The state variable in FlowManager is a shared dictionary that persists throughout the conversation. Think of it as a conversation memory that lets you: Store user information Track conversation progress Share data between nodes Inform decision-making Here’s a practical example of a pizza ordering flow: Copy Ask AI # Store user choices as they're made async def select_size ( args : FlowArgs) -> tuple[FlowResult, str ]: """Handle pizza size selection.""" size = args[ "size" ] # Initialize order in state if it doesn't exist if "order" not in flow_manager.state: flow_manager.state[ "order" ] = {} # Store the selection flow_manager.state[ "order" ][ "size" ] = size return { "status" : "success" , "size" : size}, "toppings" async def select_toppings ( args : FlowArgs) -> tuple[FlowResult, str ]: """Handle topping selection.""" topping = args[ "topping" ] # Get existing order and toppings order = flow_manager.state.get( "order" , {}) toppings = order.get( "toppings" , []) # Add new topping toppings.append(topping) order[ "toppings" ] = toppings flow_manager.state[ "order" ] = order return { "status" : "success" , "toppings" : toppings}, "finalize" async def finalize_order ( args : FlowArgs) -> tuple[FlowResult, str ]: """Process the complete order.""" order = flow_manager.state.get( "order" , {}) # Validate order has required information if "size" not in order: return { "status" : "error" , "error" : "No size selected" } # Calculate price based on stored selections size = order[ "size" ] toppings = order.get( "toppings" , []) price = calculate_price(size, len (toppings)) return { "status" : "success" , "summary" : f "Ordered: { size } pizza with { ', ' .join(toppings) } " , "price" : price }, "end" In this example: select_size initializes the order and stores the size select_toppings builds a list of toppings finalize_order uses the stored information to process the complete order The state variable makes it easy to: Build up information across multiple interactions Access previous choices when needed Validate the complete order Calculate final results This is particularly useful when information needs to be collected across multiple conversation turns or when later decisions depend on earlier choices. ​ LLM Provider Support Pipecat Flows automatically handles format differences between LLM providers: ​ OpenAI Format Copy Ask AI "functions" : [{ "type" : "function" , "function" : { "name" : "function_name" , "handler" : select_size, "description" : "description" , "parameters" : { ... } } }] ​ Anthropic Format Copy Ask AI "functions" : [{ "name" : "function_name" , "handler" : select_size, "description" : "description" , "input_schema" : { ... } }] ​ Google (Gemini) Format Copy Ask AI "functions" : [{ "function_declarations" : [{ "name" : "function_name" , "handler" : select_size, "description" : "description" , "parameters" : { ... } }] }] You don’t need to handle these differences manually - Pipecat Flows adapts your configuration to the correct format based on your LLM provider. ​ Implementation Approaches ​ Static Flows Static flows use a configuration-driven approach where the entire conversation structure is defined upfront. ​ Basic Setup Copy Ask AI from pipecat_flows import FlowManager # Define flow configuration flow_config = { "initial_node" : "greeting" , "nodes" : { "greeting" : { "role_messages" : [ ... ], "task_messages" : [ ... ], "functions" : [ ... ] } } } # Initialize flow manager with static configuration flow_manager = FlowManager( task = task, llm = llm, context_aggregator = context_aggregator, flow_config = flow_config ) @transport.event_handler ( "on_first_participant_joined" ) async def on_first_participant_joined ( transport , participant ): await transport.capture_participant_transcription(participant[ "id" ]) await flow_manager.initialize() ​ Example FlowConfig Copy Ask AI flow_config = { "initial_node" : "start" , "nodes" : { "start" : { "role_messages" : [ { "role" : "system" , "content" : "You are an order-taking assistant. You must ALWAYS use the available functions to progress the conversation. This is a phone conversation and your responses will be converted to audio. Keep the conversation friendly, casual, and polite. Avoid outputting special characters and emojis." , } ], "task_messages" : [ { "role" : "system" , "content" : "You are an order-taking assistant. Ask if they want pizza or sushi." } ], "functions" : [ { "type" : "function" , "function" : { "name" : "choose_pizza" , "handler" : choose_pizza, # Returns [None, "pizza_order"] "description" : "User wants pizza" , "parameters" : { "type" : "object" , "properties" : {}} } } ] }, "pizza_order" : { "task_messages" : [ ... ], "functions" : [ { "type" : "function" , "function" : { "name" : "select_size" , "handler" : select_size, # Returns [FlowResult, "toppings"] "description" : "Select pizza size" , "parameters" : { "type" : "object" , "properties" : { "size" : { "type" : "string" , "enum" : [ "small" , "medium" , "large" ]} } } } } ] } } } ​ Dynamic Flows Dynamic flows create and modify conversation paths at runtime based on data or business logic. ​ Example Implementation Here’s a complete example of a dynamic insurance quote flow: Copy Ask AI from pipecat_flows import FlowManager, FlowArgs, FlowResult # Define handlers and transitions async def collect_age ( args : FlowArgs, flow_manager : FlowManager) -> tuple[AgeResult, NodeConfig]: """Process age collection.""" age = args[ "age" ] # Assemble result result = AgeResult( status = "success" , age = age) # Decide which node to go to next if age < 25 : await flow_manager.set_node_from_config(create_young_adult_node()) else : await flow_manager.set_node_from_config(create_standard_node()) return result, age # Node creation functions def create_initial_node () -> NodeConfig: """Create initial age collection node.""" return { "name" : "initial" , "role_messages" : [ { "role" : "system" , "content" : "You are an insurance quote assistant." } ], "task_messages" : [ { "role" : "system" , "content" : "Ask for the customer's age." } ], "functions" : [ { "type" : "function" , "function" : { "name" : "collect_age" , "handler" : collect_age, "description" : "Collect customer age" , "parameters" : { "type" : "object" , "properties" : { "age" : { "type" : "integer" } } } } } ] } def create_young_adult_node () -> Dict[ str , Any]: """Create node for young adult quotes.""" return { "name" : "young_adult" , "task_messages" : [ { "role" : "system" , "content" : "Explain our special young adult coverage options." } ], "functions" : [ ... ] # Additional quote-specific functions } def create_standard_node () -> Dict[ str , Any]: """Create node for standard quotes.""" return { "name" : "standard" , "task_messages" : [ { "role" : "system" , "content" : "Present our standard coverage options." } ], "functions" : [ ... ] # Additional quote-specific functions } # Initialize flow manager flow_manager = FlowManager( task = task, llm = llm, context_aggregator = context_aggregator, ) @transport.event_handler ( "on_first_participant_joined" ) async def on_first_participant_joined ( transport , participant ): await transport.capture_participant_transcription(participant[ "id" ]) await flow_manager.initialize(create_initial_node()) ​ Best Practices Store shared data in flow_manager.state Create separate functions for node creation ​ Flow Editor The Pipecat Flow Editor provides a visual interface for creating and managing conversation flows. It offers a node-based interface that makes it easier to design, visualize, and modify your flows. ​ Visual Design ​ Node Types Start Node (Green): Entry point of your flow Copy Ask AI "greeting" : { "role_messages" : [ ... ], "task_messages" : [ ... ], "functions" : [ ... ] } Flow Nodes (Blue): Intermediate states Copy Ask AI "collect_info" : { "task_messages" : [ ... ], "functions" : [ ... ], "pre_actions" : [ ... ] } End Node (Red): Final state Copy Ask AI "end" : { "task_messages" : [ ... ], "functions" : [], "post_actions" : [{ "type" : "end_conversation" }] } Function Nodes : Edge Functions (Purple): Create transitions Copy Ask AI { "name" : "next_node" , "description" : "Transition to next state" } Node Functions (Orange): Perform operations Copy Ask AI { "name" : "process_data" , "handler" : process_data_handler, "description" : "Process user data" } ​ Naming Conventions Start Node : Use descriptive names (e.g., “greeting”, “welcome”) Flow Nodes : Name based on purpose (e.g., “collect_info”, “verify_data”) End Node : Conventionally named “end” Functions : Use clear, action-oriented names ​ Function Configuration Copy Ask AI { "type" : "function" , "function" : { "name" : "process_data" , "handler" : process_handler, "description" : "Process user data" , "parameters" : { ... } } } When using the Flow Editor, function handlers can be specified using the __function__: token: Copy Ask AI { "type" : "function" , "function" : { "name" : "process_data" , "handler" : "__function__:process_data" , # References function in main script "description" : "Process user data" , "parameters" : { ... } } } The handler will be looked up in your main script when the flow is executed. When function handlers are specified in the flow editor, they will be exported with the __function__: token. ​ Using the Editor ​ Creating a New Flow Start with a descriptively named Start Node Add Flow Nodes for each conversation state Connect nodes using Edge Functions Add Node Functions for operations Include an End Node ​ Import/Export Copy Ask AI # Export format { "initial_node" : "greeting" , "nodes" : { "greeting" : { "role_messages" : [ ... ], "task_messages" : [ ... ], "functions" : [ ... ], "pre_actions" : [ ... ] }, "process" : { "task_messages" : [ ... ], "functions" : [ ... ], }, "end" : { "task_messages" : [ ... ], "functions" : [], "post_actions" : [ ... ] } } } ​ Tips Use the visual preview to verify flow logic Test exported configurations Document node purposes and transitions Keep flows modular and maintainable Try the editor at flows.pipecat.ai OpenAI Audio Models and APIs Overview On this page Key Concepts Example Flows When to Use Static vs Dynamic Flows Installation Core Concepts Designing Conversation Flows Defining a Function Example Function Node Structure Messages Functions Function Configuration Node Functions Edge Functions Actions Pre-Actions Post-Actions Timing Considerations Action Types Deciding Who Speaks First Context Management Context Strategies Configuration Strategy Selection State Management LLM Provider Support OpenAI Format Anthropic Format Google (Gemini) Format Implementation Approaches Static Flows Basic Setup Example FlowConfig Dynamic Flows Example Implementation Best Practices Flow Editor Visual Design Node Types Naming Conventions Function Configuration Using the Editor Creating a New Flow Import/Export Tips Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/features_pipecat-flows_fbd39ed8.txt b/features_pipecat-flows_fbd39ed8.txt
new file mode 100644
index 0000000000000000000000000000000000000000..20d5d867eb65a698342aed80d0ec527233295943
--- /dev/null
+++ b/features_pipecat-flows_fbd39ed8.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/_sites/docs.pipecat.ai/guides/features/pipecat-flows#what-you-can-build
+Title: Overview - Pipecat
+==================================================
+
+Overview - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Get Started Overview Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Get Started Overview Installation & Setup Quickstart Core Concepts Next Steps & Examples Pipecat is an open source Python framework that handles the complex orchestration of AI services, network transport, audio processing, and multimodal interactions. “Multimodal” means you can use any combination of audio, video, images, and/or text in your interactions. And “real-time” means that things are happening quickly enough that it feels conversational—a “back-and-forth” with a bot, not submitting a query and waiting for results. ​ What You Can Build Voice Assistants Natural, real-time conversations with AI using speech recognition and synthesis Interactive Agents Personal coaches and meeting assistants that can understand context and provide guidance Multimodal Apps Applications that combine voice, video, images, and text for rich interactions Creative Tools Storytelling experiences and social companions that engage users Business Solutions Customer intake flows and support bots for automated business processes Complex Flows Structured conversations using Pipecat Flows for managing complex interactions ​ How It Works The flow of interactions in a Pipecat application is typically straightforward: The bot says something The user says something The bot says something The user says something This continues until the conversation naturally ends. While this flow seems simple, making it feel natural requires sophisticated real-time processing. ​ Real-time Processing Pipecat’s pipeline architecture handles both simple voice interactions and complex multimodal processing. Let’s look at how data flows through the system: Voice app Multimodal app 1 Send Audio Transmit and capture streamed audio from the user 2 Transcribe Speech Convert speech to text as the user is talking 3 Process with LLM Generate responses using a large language model 4 Convert to Speech Transform text responses into natural speech 5 Play Audio Stream the audio response back to the user 1 Send Audio Transmit and capture streamed audio from the user 2 Transcribe Speech Convert speech to text as the user is talking 3 Process with LLM Generate responses using a large language model 4 Convert to Speech Transform text responses into natural speech 5 Play Audio Stream the audio response back to the user 1 Send Audio and Video Transmit and capture audio, video, and image inputs simultaneously 2 Process Streams Handle multiple input streams in parallel 3 Model Processing Send combined inputs to multimodal models (like GPT-4V) 4 Generate Outputs Create various outputs (text, images, audio, etc.) 5 Coordinate Presentation Synchronize and present multiple output types In both cases, Pipecat: Processes responses as they stream in Handles multiple input/output modalities concurrently Manages resource allocation and synchronization Coordinates parallel processing tasks This architecture creates fluid, natural interactions without noticeable delays, whether you’re building a simple voice assistant or a complex multimodal application. Pipecat’s pipeline architecture is particularly valuable for managing the complexity of real-time, multimodal interactions, ensuring smooth data flow and proper synchronization regardless of the input/output types involved. Pipecat handles all this complexity for you, letting you focus on building your application rather than managing the underlying infrastructure. ​ Next Steps Ready to build your first Pipecat application? Installation & Setup Prepare your environment and install required dependencies Quickstart Build and run your first Pipecat application Core Concepts Learn about pipelines, frames, and real-time processing Use Cases Explore example implementations and patterns ​ Join Our Community Discord Community Connect with other developers, share your projects, and get support from the Pipecat team. Installation & Setup On this page What You Can Build How It Works Real-time Processing Next Steps Join Our Community Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/filters_frame-filter_47fd3949.txt b/filters_frame-filter_47fd3949.txt
new file mode 100644
index 0000000000000000000000000000000000000000..5a5489dfac6667e32463c1a517359d8224944549
--- /dev/null
+++ b/filters_frame-filter_47fd3949.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/filters/frame-filter#param-types
+Title: FrameFilter - Pipecat
+==================================================
+
+FrameFilter - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Frame Filters FrameFilter Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters FrameFilter FunctionFilter IdentityFilter NullFilter STTMuteFilter WakeCheckFilter WakeNotifierFilter Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview FrameFilter is a processor that filters frames based on their types, only passing through frames that match specified types (plus some system frames like EndFrame and SystemFrame ). ​ Constructor Parameters ​ types Tuple[Type[Frame], ...] required Tuple of frame types that should be passed through the filter ​ Functionality When a frame passes through the filter, it is checked against the provided types. Only frames that match one of the specified types (or are system frames) will be passed downstream. All other frames are dropped. ​ Output Frames The processor always passes through: Frames matching any of the specified types EndFrame and SystemFrame instances (always allowed, so as to not block the pipeline) ​ Usage Example Copy Ask AI from pipecat.frames.frames import TextFrame, AudioRawFrame, Frame from pipecat.processors.filters import FrameFilter from typing import Tuple, Type # Create a filter that only passes TextFrames and AudioRawFrames text_and_audio_filter = FrameFilter( types = (TextFrame, AudioRawFrame) ) # Add to pipeline pipeline = Pipeline([ source, text_and_audio_filter, # Filters out all other frame types destination ]) ​ Frame Flow ​ Notes Simple but powerful way to restrict which frame types flow through parts of your pipeline Always allows system frames to pass through for proper pipeline operation Can be used to isolate specific parts of your pipeline from certain frame types Efficient implementation with minimal overhead SoundfileMixer FunctionFilter On this page Overview Constructor Parameters Functionality Output Frames Usage Example Frame Flow Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/filters_frame-filter_6f249026.txt b/filters_frame-filter_6f249026.txt
new file mode 100644
index 0000000000000000000000000000000000000000..7dbe3dd06fc4c262c15bd51bb32f91b32094e51f
--- /dev/null
+++ b/filters_frame-filter_6f249026.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/filters/frame-filter#functionality
+Title: FrameFilter - Pipecat
+==================================================
+
+FrameFilter - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Frame Filters FrameFilter Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters FrameFilter FunctionFilter IdentityFilter NullFilter STTMuteFilter WakeCheckFilter WakeNotifierFilter Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview FrameFilter is a processor that filters frames based on their types, only passing through frames that match specified types (plus some system frames like EndFrame and SystemFrame ). ​ Constructor Parameters ​ types Tuple[Type[Frame], ...] required Tuple of frame types that should be passed through the filter ​ Functionality When a frame passes through the filter, it is checked against the provided types. Only frames that match one of the specified types (or are system frames) will be passed downstream. All other frames are dropped. ​ Output Frames The processor always passes through: Frames matching any of the specified types EndFrame and SystemFrame instances (always allowed, so as to not block the pipeline) ​ Usage Example Copy Ask AI from pipecat.frames.frames import TextFrame, AudioRawFrame, Frame from pipecat.processors.filters import FrameFilter from typing import Tuple, Type # Create a filter that only passes TextFrames and AudioRawFrames text_and_audio_filter = FrameFilter( types = (TextFrame, AudioRawFrame) ) # Add to pipeline pipeline = Pipeline([ source, text_and_audio_filter, # Filters out all other frame types destination ]) ​ Frame Flow ​ Notes Simple but powerful way to restrict which frame types flow through parts of your pipeline Always allows system frames to pass through for proper pipeline operation Can be used to isolate specific parts of your pipeline from certain frame types Efficient implementation with minimal overhead SoundfileMixer FunctionFilter On this page Overview Constructor Parameters Functionality Output Frames Usage Example Frame Flow Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/filters_function-filter_00f806c7.txt b/filters_function-filter_00f806c7.txt
new file mode 100644
index 0000000000000000000000000000000000000000..8cfbfefe21a0e12a22e85e978059896e2cf5cd2f
--- /dev/null
+++ b/filters_function-filter_00f806c7.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/filters/function-filter#param-direction
+Title: FunctionFilter - Pipecat
+==================================================
+
+FunctionFilter - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Frame Filters FunctionFilter Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters FrameFilter FunctionFilter IdentityFilter NullFilter STTMuteFilter WakeCheckFilter WakeNotifierFilter Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview FunctionFilter is a flexible processor that uses a custom async function to determine which frames to pass through. This allows for complex, dynamic filtering logic beyond simple type checking. ​ Constructor Parameters ​ filter Callable[[Frame], Awaitable[bool]] required Async function that examines each frame and returns True to allow it or False to filter it out ​ direction FrameDirection default: "FrameDirection.DOWNSTREAM" Which direction of frames to filter (DOWNSTREAM or UPSTREAM) ​ Functionality When a frame passes through the processor: System frames and end frames are always passed through Frames moving in a different direction than specified are always passed through Other frames are passed to the filter function If the filter function returns True, the frame is passed through ​ Output Frames The processor conditionally passes through frames based on: Frame type (system frames and end frames always pass) Frame direction (only filters in the specified direction) Result of the custom filter function ​ Usage Example Copy Ask AI from pipecat.frames.frames import TextFrame, Frame from pipecat.processors.filters import FunctionFilter from pipecat.processors.frame_processor import FrameDirection # Create filter that only allows TextFrames with more than 10 characters async def long_text_filter ( frame : Frame) -> bool : if isinstance (frame, TextFrame): return len (frame.text) > 10 return False # Apply filter to downstream frames only text_length_filter = FunctionFilter( filter = long_text_filter, direction = FrameDirection. DOWNSTREAM ) # Add to pipeline pipeline = Pipeline([ source, text_length_filter, # Filters out short text frames destination ]) ​ Frame Flow ​ Notes Provides maximum flexibility for complex filtering logic Can incorporate dynamic conditions that change at runtime Only filters frames moving in the specified direction Always passes through system frames for proper pipeline operation Can be used to create sophisticated content-based filters Supports async filter functions for complex processing FrameFilter IdentityFilter On this page Overview Constructor Parameters Functionality Output Frames Usage Example Frame Flow Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/filters_function-filter_46b78849.txt b/filters_function-filter_46b78849.txt
new file mode 100644
index 0000000000000000000000000000000000000000..f5f17db9d47cd040c351d8a2fd78e00812c6245a
--- /dev/null
+++ b/filters_function-filter_46b78849.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/filters/function-filter#frame-flow
+Title: FunctionFilter - Pipecat
+==================================================
+
+FunctionFilter - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Frame Filters FunctionFilter Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters FrameFilter FunctionFilter IdentityFilter NullFilter STTMuteFilter WakeCheckFilter WakeNotifierFilter Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview FunctionFilter is a flexible processor that uses a custom async function to determine which frames to pass through. This allows for complex, dynamic filtering logic beyond simple type checking. ​ Constructor Parameters ​ filter Callable[[Frame], Awaitable[bool]] required Async function that examines each frame and returns True to allow it or False to filter it out ​ direction FrameDirection default: "FrameDirection.DOWNSTREAM" Which direction of frames to filter (DOWNSTREAM or UPSTREAM) ​ Functionality When a frame passes through the processor: System frames and end frames are always passed through Frames moving in a different direction than specified are always passed through Other frames are passed to the filter function If the filter function returns True, the frame is passed through ​ Output Frames The processor conditionally passes through frames based on: Frame type (system frames and end frames always pass) Frame direction (only filters in the specified direction) Result of the custom filter function ​ Usage Example Copy Ask AI from pipecat.frames.frames import TextFrame, Frame from pipecat.processors.filters import FunctionFilter from pipecat.processors.frame_processor import FrameDirection # Create filter that only allows TextFrames with more than 10 characters async def long_text_filter ( frame : Frame) -> bool : if isinstance (frame, TextFrame): return len (frame.text) > 10 return False # Apply filter to downstream frames only text_length_filter = FunctionFilter( filter = long_text_filter, direction = FrameDirection. DOWNSTREAM ) # Add to pipeline pipeline = Pipeline([ source, text_length_filter, # Filters out short text frames destination ]) ​ Frame Flow ​ Notes Provides maximum flexibility for complex filtering logic Can incorporate dynamic conditions that change at runtime Only filters frames moving in the specified direction Always passes through system frames for proper pipeline operation Can be used to create sophisticated content-based filters Supports async filter functions for complex processing FrameFilter IdentityFilter On this page Overview Constructor Parameters Functionality Output Frames Usage Example Frame Flow Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/filters_function-filter_7f1def06.txt b/filters_function-filter_7f1def06.txt
new file mode 100644
index 0000000000000000000000000000000000000000..54bc0e984949b2d6967f978f445b899c4c910c9b
--- /dev/null
+++ b/filters_function-filter_7f1def06.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/filters/function-filter
+Title: FunctionFilter - Pipecat
+==================================================
+
+FunctionFilter - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Frame Filters FunctionFilter Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters FrameFilter FunctionFilter IdentityFilter NullFilter STTMuteFilter WakeCheckFilter WakeNotifierFilter Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview FunctionFilter is a flexible processor that uses a custom async function to determine which frames to pass through. This allows for complex, dynamic filtering logic beyond simple type checking. ​ Constructor Parameters ​ filter Callable[[Frame], Awaitable[bool]] required Async function that examines each frame and returns True to allow it or False to filter it out ​ direction FrameDirection default: "FrameDirection.DOWNSTREAM" Which direction of frames to filter (DOWNSTREAM or UPSTREAM) ​ Functionality When a frame passes through the processor: System frames and end frames are always passed through Frames moving in a different direction than specified are always passed through Other frames are passed to the filter function If the filter function returns True, the frame is passed through ​ Output Frames The processor conditionally passes through frames based on: Frame type (system frames and end frames always pass) Frame direction (only filters in the specified direction) Result of the custom filter function ​ Usage Example Copy Ask AI from pipecat.frames.frames import TextFrame, Frame from pipecat.processors.filters import FunctionFilter from pipecat.processors.frame_processor import FrameDirection # Create filter that only allows TextFrames with more than 10 characters async def long_text_filter ( frame : Frame) -> bool : if isinstance (frame, TextFrame): return len (frame.text) > 10 return False # Apply filter to downstream frames only text_length_filter = FunctionFilter( filter = long_text_filter, direction = FrameDirection. DOWNSTREAM ) # Add to pipeline pipeline = Pipeline([ source, text_length_filter, # Filters out short text frames destination ]) ​ Frame Flow ​ Notes Provides maximum flexibility for complex filtering logic Can incorporate dynamic conditions that change at runtime Only filters frames moving in the specified direction Always passes through system frames for proper pipeline operation Can be used to create sophisticated content-based filters Supports async filter functions for complex processing FrameFilter IdentityFilter On this page Overview Constructor Parameters Functionality Output Frames Usage Example Frame Flow Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/filters_identify-filter_55985311.txt b/filters_identify-filter_55985311.txt
new file mode 100644
index 0000000000000000000000000000000000000000..62477bca6ebbd8ee6f5f22838527a5b651e36deb
--- /dev/null
+++ b/filters_identify-filter_55985311.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/filters/identify-filter#frame-flow
+Title: IdentityFilter - Pipecat
+==================================================
+
+IdentityFilter - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Frame Filters IdentityFilter Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters FrameFilter FunctionFilter IdentityFilter NullFilter STTMuteFilter WakeCheckFilter WakeNotifierFilter Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview IdentityFilter is a simple pass-through processor that forwards all frames without any modification or filtering. It acts as a transparent layer in your pipeline, allowing all frames to flow through unchanged. Check out Observers for an option that delivers similar functionality but doesn’t require a processor to reside in the Pipeline. ​ Constructor Parameters The IdentityFilter constructor accepts no specific parameters beyond those inherited from FrameProcessor . ​ Functionality When a frame passes through the processor, it is immediately forwarded in the same direction with no changes. This applies to all frame types and both directions (upstream and downstream). ​ Use Cases While functionally equivalent to having no filter at all, IdentityFilter can be useful in several scenarios: Testing ParallelPipeline configurations to ensure frames aren’t duplicated Acting as a placeholder where a more complex filter might be added later Monitoring frame flow in pipelines by adding logging in subclasses Creating a base class for more complex conditional filters ​ Usage Example Copy Ask AI from pipecat.processors.filters import IdentityFilter # Create an identity filter pass_through = IdentityFilter() # Add to pipeline pipeline = Pipeline([ source, pass_through, # All frames pass through unchanged destination ]) ​ Frame Flow ​ Notes Simplest possible filter implementation Passes all frames through without modification Useful in testing parallel pipelines Can serve as a placeholder or base class Zero overhead in normal operation FunctionFilter NullFilter On this page Overview Constructor Parameters Functionality Use Cases Usage Example Frame Flow Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/filters_identify-filter_57443118.txt b/filters_identify-filter_57443118.txt
new file mode 100644
index 0000000000000000000000000000000000000000..4bc452be5893b60ce56c5dadeeee9c6cf2de24a2
--- /dev/null
+++ b/filters_identify-filter_57443118.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/filters/identify-filter#use-cases
+Title: IdentityFilter - Pipecat
+==================================================
+
+IdentityFilter - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Frame Filters IdentityFilter Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters FrameFilter FunctionFilter IdentityFilter NullFilter STTMuteFilter WakeCheckFilter WakeNotifierFilter Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview IdentityFilter is a simple pass-through processor that forwards all frames without any modification or filtering. It acts as a transparent layer in your pipeline, allowing all frames to flow through unchanged. Check out Observers for an option that delivers similar functionality but doesn’t require a processor to reside in the Pipeline. ​ Constructor Parameters The IdentityFilter constructor accepts no specific parameters beyond those inherited from FrameProcessor . ​ Functionality When a frame passes through the processor, it is immediately forwarded in the same direction with no changes. This applies to all frame types and both directions (upstream and downstream). ​ Use Cases While functionally equivalent to having no filter at all, IdentityFilter can be useful in several scenarios: Testing ParallelPipeline configurations to ensure frames aren’t duplicated Acting as a placeholder where a more complex filter might be added later Monitoring frame flow in pipelines by adding logging in subclasses Creating a base class for more complex conditional filters ​ Usage Example Copy Ask AI from pipecat.processors.filters import IdentityFilter # Create an identity filter pass_through = IdentityFilter() # Add to pipeline pipeline = Pipeline([ source, pass_through, # All frames pass through unchanged destination ]) ​ Frame Flow ​ Notes Simplest possible filter implementation Passes all frames through without modification Useful in testing parallel pipelines Can serve as a placeholder or base class Zero overhead in normal operation FunctionFilter NullFilter On this page Overview Constructor Parameters Functionality Use Cases Usage Example Frame Flow Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/filters_null-filter_4bd40459.txt b/filters_null-filter_4bd40459.txt
new file mode 100644
index 0000000000000000000000000000000000000000..b79bbf67df7b73d2de5775480d6f797ad239630d
--- /dev/null
+++ b/filters_null-filter_4bd40459.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/filters/null-filter#usage-example
+Title: NullFilter - Pipecat
+==================================================
+
+NullFilter - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Frame Filters NullFilter Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters FrameFilter FunctionFilter IdentityFilter NullFilter STTMuteFilter WakeCheckFilter WakeNotifierFilter Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview NullFilter is a filtering processor that blocks all frames from passing through, with the exception of system frames and end frames which are required for proper pipeline operation. ​ Constructor Parameters The NullFilter constructor accepts no specific parameters beyond those inherited from FrameProcessor . ​ Functionality When a frame passes through the processor: If the frame is a SystemFrame or EndFrame , it is passed through All other frame types are blocked and do not continue through the pipeline This filter effectively acts as a barrier that allows only the essential system frames required for pipeline initialization, shutdown, and management. ​ Use Cases NullFilter is useful in several scenarios: Temporarily disabling parts of a pipeline without removing components Creating dead-end branches in parallel pipelines Testing pipeline behavior with blocked communication Implementing conditional pipelines where certain paths should be blocked ​ Usage Example Copy Ask AI from pipecat.processors.filters import NullFilter # Create a null filter that blocks all non-system frames blocker = NullFilter() # Add to pipeline pipeline = Pipeline([ source, blocker, # Blocks all regular frames destination # Will only receive system frames ]) ​ Frame Flow ​ Notes Blocks all regular frames in both directions Only allows system frames and end frames to pass through Useful for testing, debugging, and creating conditional pipelines Minimal overhead as it performs simple type checking Can be used to temporarily disable parts of a pipeline IdentityFilter STTMuteFilter On this page Overview Constructor Parameters Functionality Use Cases Usage Example Frame Flow Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/filters_null-filter_5b9b3c2f.txt b/filters_null-filter_5b9b3c2f.txt
new file mode 100644
index 0000000000000000000000000000000000000000..7f51ec6f21d0b2aacaa1eed05536eb8e5774c183
--- /dev/null
+++ b/filters_null-filter_5b9b3c2f.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/filters/null-filter#overview
+Title: NullFilter - Pipecat
+==================================================
+
+NullFilter - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Frame Filters NullFilter Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters FrameFilter FunctionFilter IdentityFilter NullFilter STTMuteFilter WakeCheckFilter WakeNotifierFilter Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview NullFilter is a filtering processor that blocks all frames from passing through, with the exception of system frames and end frames which are required for proper pipeline operation. ​ Constructor Parameters The NullFilter constructor accepts no specific parameters beyond those inherited from FrameProcessor . ​ Functionality When a frame passes through the processor: If the frame is a SystemFrame or EndFrame , it is passed through All other frame types are blocked and do not continue through the pipeline This filter effectively acts as a barrier that allows only the essential system frames required for pipeline initialization, shutdown, and management. ​ Use Cases NullFilter is useful in several scenarios: Temporarily disabling parts of a pipeline without removing components Creating dead-end branches in parallel pipelines Testing pipeline behavior with blocked communication Implementing conditional pipelines where certain paths should be blocked ​ Usage Example Copy Ask AI from pipecat.processors.filters import NullFilter # Create a null filter that blocks all non-system frames blocker = NullFilter() # Add to pipeline pipeline = Pipeline([ source, blocker, # Blocks all regular frames destination # Will only receive system frames ]) ​ Frame Flow ​ Notes Blocks all regular frames in both directions Only allows system frames and end frames to pass through Useful for testing, debugging, and creating conditional pipelines Minimal overhead as it performs simple type checking Can be used to temporarily disable parts of a pipeline IdentityFilter STTMuteFilter On this page Overview Constructor Parameters Functionality Use Cases Usage Example Frame Flow Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/filters_null-filter_9df1306a.txt b/filters_null-filter_9df1306a.txt
new file mode 100644
index 0000000000000000000000000000000000000000..653e5a135bc44598b748671af1a4144dc7773b57
--- /dev/null
+++ b/filters_null-filter_9df1306a.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/filters/null-filter
+Title: NullFilter - Pipecat
+==================================================
+
+NullFilter - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Frame Filters NullFilter Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters FrameFilter FunctionFilter IdentityFilter NullFilter STTMuteFilter WakeCheckFilter WakeNotifierFilter Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview NullFilter is a filtering processor that blocks all frames from passing through, with the exception of system frames and end frames which are required for proper pipeline operation. ​ Constructor Parameters The NullFilter constructor accepts no specific parameters beyond those inherited from FrameProcessor . ​ Functionality When a frame passes through the processor: If the frame is a SystemFrame or EndFrame , it is passed through All other frame types are blocked and do not continue through the pipeline This filter effectively acts as a barrier that allows only the essential system frames required for pipeline initialization, shutdown, and management. ​ Use Cases NullFilter is useful in several scenarios: Temporarily disabling parts of a pipeline without removing components Creating dead-end branches in parallel pipelines Testing pipeline behavior with blocked communication Implementing conditional pipelines where certain paths should be blocked ​ Usage Example Copy Ask AI from pipecat.processors.filters import NullFilter # Create a null filter that blocks all non-system frames blocker = NullFilter() # Add to pipeline pipeline = Pipeline([ source, blocker, # Blocks all regular frames destination # Will only receive system frames ]) ​ Frame Flow ​ Notes Blocks all regular frames in both directions Only allows system frames and end frames to pass through Useful for testing, debugging, and creating conditional pipelines Minimal overhead as it performs simple type checking Can be used to temporarily disable parts of a pipeline IdentityFilter STTMuteFilter On this page Overview Constructor Parameters Functionality Use Cases Usage Example Frame Flow Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/filters_null-filter_d8d6bb3f.txt b/filters_null-filter_d8d6bb3f.txt
new file mode 100644
index 0000000000000000000000000000000000000000..2ef754f161e5252d725b923b96535ea550d87452
--- /dev/null
+++ b/filters_null-filter_d8d6bb3f.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/filters/null-filter#functionality
+Title: NullFilter - Pipecat
+==================================================
+
+NullFilter - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Frame Filters NullFilter Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters FrameFilter FunctionFilter IdentityFilter NullFilter STTMuteFilter WakeCheckFilter WakeNotifierFilter Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview NullFilter is a filtering processor that blocks all frames from passing through, with the exception of system frames and end frames which are required for proper pipeline operation. ​ Constructor Parameters The NullFilter constructor accepts no specific parameters beyond those inherited from FrameProcessor . ​ Functionality When a frame passes through the processor: If the frame is a SystemFrame or EndFrame , it is passed through All other frame types are blocked and do not continue through the pipeline This filter effectively acts as a barrier that allows only the essential system frames required for pipeline initialization, shutdown, and management. ​ Use Cases NullFilter is useful in several scenarios: Temporarily disabling parts of a pipeline without removing components Creating dead-end branches in parallel pipelines Testing pipeline behavior with blocked communication Implementing conditional pipelines where certain paths should be blocked ​ Usage Example Copy Ask AI from pipecat.processors.filters import NullFilter # Create a null filter that blocks all non-system frames blocker = NullFilter() # Add to pipeline pipeline = Pipeline([ source, blocker, # Blocks all regular frames destination # Will only receive system frames ]) ​ Frame Flow ​ Notes Blocks all regular frames in both directions Only allows system frames and end frames to pass through Useful for testing, debugging, and creating conditional pipelines Minimal overhead as it performs simple type checking Can be used to temporarily disable parts of a pipeline IdentityFilter STTMuteFilter On this page Overview Constructor Parameters Functionality Use Cases Usage Example Frame Flow Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/filters_stt-mute_2298cc78.txt b/filters_stt-mute_2298cc78.txt
new file mode 100644
index 0000000000000000000000000000000000000000..ef0dc21300f4415524d81a9ff3017aac3b9a2394
--- /dev/null
+++ b/filters_stt-mute_2298cc78.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/filters/stt-mute#param-always
+Title: STTMuteFilter - Pipecat
+==================================================
+
+STTMuteFilter - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Frame Filters STTMuteFilter Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters FrameFilter FunctionFilter IdentityFilter NullFilter STTMuteFilter WakeCheckFilter WakeNotifierFilter Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview STTMuteFilter is a general-purpose processor that combines STT muting and interruption control. When active, it prevents both transcription and interruptions during specified conditions (e.g., bot speech, function calls), providing a cleaner conversation flow. The processor supports multiple simultaneous strategies for when to mute the STT service, making it flexible for different use cases. Want to try it out? Check out the STTMuteFilter foundational demo ​ Constructor Parameters ​ config STTMuteConfig required Configuration object that defines the muting strategies and optional custom logic ​ stt_service Optional[STTService] required The STT service to control (deprecated, will be removed in a future version) ​ Configuration The processor is configured using STTMuteConfig , which determines when and how the STT service should be muted: ​ strategies set[STTMuteStrategy] Set of muting strategies to apply ​ should_mute_callback Callable[[STTMuteFilter], Awaitable[bool]] default: "None" Optional callback for custom muting logic (required when strategy is CUSTOM ) ​ Muting Strategies STTMuteConfig accepts a set of these STTMuteStrategy values: ​ FIRST_SPEECH STTMuteStrategy Mute only during the bot’s first speech (typically during introduction) ​ MUTE_UNTIL_FIRST_BOT_COMPLETE STTMuteStrategy Start muted and remain muted until first bot speech completes. Useful when bot speaks first and you want to ensure its first response cannot be interrupted. ​ FUNCTION_CALL STTMuteStrategy Mute during LLM function calls (e.g., API requests, external service calls) ​ ALWAYS STTMuteStrategy Mute during all bot speech ​ CUSTOM STTMuteStrategy Use custom logic provided via callback to determine when to mute. The callback is invoked when the bot is speaking and can use application state to decide whether to mute. When the bot stops speaking, unmuting occurs automatically if no other strategy requires muting. MUTE_UNTIL_FIRST_BOT_COMPLETE and FIRST_SPEECH strategies should not be used together as they handle the first bot speech differently. ​ Input Frames ​ BotStartedSpeakingFrame Frame Indicates bot has started speaking ​ BotStoppedSpeakingFrame Frame Indicates bot has stopped speaking ​ FunctionCallInProgressFrame Frame Indicates a function call has started ​ FunctionCallResultFrame Frame Indicates a function call has completed ​ StartInterruptionFrame Frame User interruption start event (suppressed when muted) ​ StopInterruptionFrame Frame User interruption stop event (suppressed when muted) ​ UserStartedSpeakingFrame Frame Indicates user has started speaking (suppressed when muted) ​ UserStoppedSpeakingFrame Frame Indicates user has stopped speaking (suppressed when muted) ​ Output Frames ​ STTMuteFrame Frame Control frame to mute/unmute the STT service All input frames are passed through except VAD-related frames (interruptions and user speaking events) when muted. ​ Usage Examples ​ Basic Usage (Mute During Bot’s First Speech) Copy Ask AI stt = DeepgramSTTService( api_key = os.getenv( "DEEPGRAM_API_KEY" )) stt_mute_filter = STTMuteFilter( config = STTMuteConfig( strategies = { STTMuteStrategy. FIRST_SPEECH }) ) pipeline = Pipeline([ transport.input(), stt_mute_filter, # Add before STT service stt, # ... rest of pipeline ]) ​ Mute Until First Bot Response Completes Copy Ask AI stt_mute_filter = STTMuteFilter( config = STTMuteConfig( strategies = {STTMuteStrategy. MUTE_UNTIL_FIRST_BOT_COMPLETE }) ) This ensures no user speech is processed until after the bot’s first complete response. ​ Always Mute During Bot Speech Copy Ask AI stt_mute_filter = STTMuteFilter( config = STTMuteConfig( strategies = {STTMuteStrategy. ALWAYS }) ) ​ Custom Muting Logic The CUSTOM strategy allows you to control muting based on application state when the bot is speaking. The callback will be invoked whenever the bot is speaking, and your logic decides whether to mute: Copy Ask AI # Create a state manager class SessionState : def __init__ ( self ): self .session_ending = False session_state = SessionState() # Callback function that determines whether to mute async def session_state_mute_logic ( stt_filter : STTMuteFilter) -> bool : # Return True to mute, False otherwise # This is called when the bot is speaking return session_state.session_ending # Configure filter with CUSTOM strategy stt_mute_filter = STTMuteFilter( config = STTMuteConfig( strategies = {STTMuteStrategy. CUSTOM }, should_mute_callback = session_state_mute_logic ) ) # Later, when you want to trigger muting (e.g., during session timeout): async def handle_session_timeout (): # Update state that will be checked by the callback session_state.session_ending = True # Send goodbye message goodbye_message = "Thank you for using our service. This session is now ending." await pipeline.push_frame(TTSSpeakFrame( text = goodbye_message)) # The system will automatically mute during this message because: # 1. Bot starts speaking, triggering the callback # 2. Callback returns True (session_ending is True) # 3. When bot stops speaking, unmuting happens automatically ​ Combining Multiple Strategies Copy Ask AI async def custom_mute_logic ( processor : STTMuteFilter) -> bool : # Example: Mute during business hours only current_hour = datetime.now().hour return 9 <= current_hour < 17 stt_mute_filter = STTMuteFilter( config = STTMuteConfig( strategies = { STTMuteStrategy. FUNCTION_CALL , # Mute during function calls STTMuteStrategy. CUSTOM , # And during business hours STTMuteStrategy. MUTE_UNTIL_FIRST_BOT_COMPLETE # And until first bot speech completes }, should_mute_callback = custom_mute_logic ) ) ​ Frame Flow ​ Notes Combines STT muting and interruption control into a single concept Muting prevents both transcription and interruptions Multiple strategies can be active simultaneously CUSTOM strategy callback is only invoked when the bot is speaking Unmuting happens automatically when bot speech ends (if no other strategy requires muting) Placed before STT service in pipeline Maintains conversation flow during bot speech and function calls Efficient state tracking for minimal overhead NullFilter WakeCheckFilter On this page Overview Constructor Parameters Configuration Muting Strategies Input Frames Output Frames Usage Examples Basic Usage (Mute During Bot’s First Speech) Mute Until First Bot Response Completes Always Mute During Bot Speech Custom Muting Logic Combining Multiple Strategies Frame Flow Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/filters_stt-mute_3fc0eb1b.txt b/filters_stt-mute_3fc0eb1b.txt
new file mode 100644
index 0000000000000000000000000000000000000000..3a372d6969bcc576abb9b7566461295643a0ee8d
--- /dev/null
+++ b/filters_stt-mute_3fc0eb1b.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/filters/stt-mute#muting-strategies
+Title: STTMuteFilter - Pipecat
+==================================================
+
+STTMuteFilter - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Frame Filters STTMuteFilter Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters FrameFilter FunctionFilter IdentityFilter NullFilter STTMuteFilter WakeCheckFilter WakeNotifierFilter Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview STTMuteFilter is a general-purpose processor that combines STT muting and interruption control. When active, it prevents both transcription and interruptions during specified conditions (e.g., bot speech, function calls), providing a cleaner conversation flow. The processor supports multiple simultaneous strategies for when to mute the STT service, making it flexible for different use cases. Want to try it out? Check out the STTMuteFilter foundational demo ​ Constructor Parameters ​ config STTMuteConfig required Configuration object that defines the muting strategies and optional custom logic ​ stt_service Optional[STTService] required The STT service to control (deprecated, will be removed in a future version) ​ Configuration The processor is configured using STTMuteConfig , which determines when and how the STT service should be muted: ​ strategies set[STTMuteStrategy] Set of muting strategies to apply ​ should_mute_callback Callable[[STTMuteFilter], Awaitable[bool]] default: "None" Optional callback for custom muting logic (required when strategy is CUSTOM ) ​ Muting Strategies STTMuteConfig accepts a set of these STTMuteStrategy values: ​ FIRST_SPEECH STTMuteStrategy Mute only during the bot’s first speech (typically during introduction) ​ MUTE_UNTIL_FIRST_BOT_COMPLETE STTMuteStrategy Start muted and remain muted until first bot speech completes. Useful when bot speaks first and you want to ensure its first response cannot be interrupted. ​ FUNCTION_CALL STTMuteStrategy Mute during LLM function calls (e.g., API requests, external service calls) ​ ALWAYS STTMuteStrategy Mute during all bot speech ​ CUSTOM STTMuteStrategy Use custom logic provided via callback to determine when to mute. The callback is invoked when the bot is speaking and can use application state to decide whether to mute. When the bot stops speaking, unmuting occurs automatically if no other strategy requires muting. MUTE_UNTIL_FIRST_BOT_COMPLETE and FIRST_SPEECH strategies should not be used together as they handle the first bot speech differently. ​ Input Frames ​ BotStartedSpeakingFrame Frame Indicates bot has started speaking ​ BotStoppedSpeakingFrame Frame Indicates bot has stopped speaking ​ FunctionCallInProgressFrame Frame Indicates a function call has started ​ FunctionCallResultFrame Frame Indicates a function call has completed ​ StartInterruptionFrame Frame User interruption start event (suppressed when muted) ​ StopInterruptionFrame Frame User interruption stop event (suppressed when muted) ​ UserStartedSpeakingFrame Frame Indicates user has started speaking (suppressed when muted) ​ UserStoppedSpeakingFrame Frame Indicates user has stopped speaking (suppressed when muted) ​ Output Frames ​ STTMuteFrame Frame Control frame to mute/unmute the STT service All input frames are passed through except VAD-related frames (interruptions and user speaking events) when muted. ​ Usage Examples ​ Basic Usage (Mute During Bot’s First Speech) Copy Ask AI stt = DeepgramSTTService( api_key = os.getenv( "DEEPGRAM_API_KEY" )) stt_mute_filter = STTMuteFilter( config = STTMuteConfig( strategies = { STTMuteStrategy. FIRST_SPEECH }) ) pipeline = Pipeline([ transport.input(), stt_mute_filter, # Add before STT service stt, # ... rest of pipeline ]) ​ Mute Until First Bot Response Completes Copy Ask AI stt_mute_filter = STTMuteFilter( config = STTMuteConfig( strategies = {STTMuteStrategy. MUTE_UNTIL_FIRST_BOT_COMPLETE }) ) This ensures no user speech is processed until after the bot’s first complete response. ​ Always Mute During Bot Speech Copy Ask AI stt_mute_filter = STTMuteFilter( config = STTMuteConfig( strategies = {STTMuteStrategy. ALWAYS }) ) ​ Custom Muting Logic The CUSTOM strategy allows you to control muting based on application state when the bot is speaking. The callback will be invoked whenever the bot is speaking, and your logic decides whether to mute: Copy Ask AI # Create a state manager class SessionState : def __init__ ( self ): self .session_ending = False session_state = SessionState() # Callback function that determines whether to mute async def session_state_mute_logic ( stt_filter : STTMuteFilter) -> bool : # Return True to mute, False otherwise # This is called when the bot is speaking return session_state.session_ending # Configure filter with CUSTOM strategy stt_mute_filter = STTMuteFilter( config = STTMuteConfig( strategies = {STTMuteStrategy. CUSTOM }, should_mute_callback = session_state_mute_logic ) ) # Later, when you want to trigger muting (e.g., during session timeout): async def handle_session_timeout (): # Update state that will be checked by the callback session_state.session_ending = True # Send goodbye message goodbye_message = "Thank you for using our service. This session is now ending." await pipeline.push_frame(TTSSpeakFrame( text = goodbye_message)) # The system will automatically mute during this message because: # 1. Bot starts speaking, triggering the callback # 2. Callback returns True (session_ending is True) # 3. When bot stops speaking, unmuting happens automatically ​ Combining Multiple Strategies Copy Ask AI async def custom_mute_logic ( processor : STTMuteFilter) -> bool : # Example: Mute during business hours only current_hour = datetime.now().hour return 9 <= current_hour < 17 stt_mute_filter = STTMuteFilter( config = STTMuteConfig( strategies = { STTMuteStrategy. FUNCTION_CALL , # Mute during function calls STTMuteStrategy. CUSTOM , # And during business hours STTMuteStrategy. MUTE_UNTIL_FIRST_BOT_COMPLETE # And until first bot speech completes }, should_mute_callback = custom_mute_logic ) ) ​ Frame Flow ​ Notes Combines STT muting and interruption control into a single concept Muting prevents both transcription and interruptions Multiple strategies can be active simultaneously CUSTOM strategy callback is only invoked when the bot is speaking Unmuting happens automatically when bot speech ends (if no other strategy requires muting) Placed before STT service in pipeline Maintains conversation flow during bot speech and function calls Efficient state tracking for minimal overhead NullFilter WakeCheckFilter On this page Overview Constructor Parameters Configuration Muting Strategies Input Frames Output Frames Usage Examples Basic Usage (Mute During Bot’s First Speech) Mute Until First Bot Response Completes Always Mute During Bot Speech Custom Muting Logic Combining Multiple Strategies Frame Flow Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/filters_stt-mute_90a8242a.txt b/filters_stt-mute_90a8242a.txt
new file mode 100644
index 0000000000000000000000000000000000000000..f8e0cb85b291ab4df49a6259586599a6b3302e14
--- /dev/null
+++ b/filters_stt-mute_90a8242a.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/filters/stt-mute#param-bot-started-speaking-frame
+Title: STTMuteFilter - Pipecat
+==================================================
+
+STTMuteFilter - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Frame Filters STTMuteFilter Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters FrameFilter FunctionFilter IdentityFilter NullFilter STTMuteFilter WakeCheckFilter WakeNotifierFilter Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview STTMuteFilter is a general-purpose processor that combines STT muting and interruption control. When active, it prevents both transcription and interruptions during specified conditions (e.g., bot speech, function calls), providing a cleaner conversation flow. The processor supports multiple simultaneous strategies for when to mute the STT service, making it flexible for different use cases. Want to try it out? Check out the STTMuteFilter foundational demo ​ Constructor Parameters ​ config STTMuteConfig required Configuration object that defines the muting strategies and optional custom logic ​ stt_service Optional[STTService] required The STT service to control (deprecated, will be removed in a future version) ​ Configuration The processor is configured using STTMuteConfig , which determines when and how the STT service should be muted: ​ strategies set[STTMuteStrategy] Set of muting strategies to apply ​ should_mute_callback Callable[[STTMuteFilter], Awaitable[bool]] default: "None" Optional callback for custom muting logic (required when strategy is CUSTOM ) ​ Muting Strategies STTMuteConfig accepts a set of these STTMuteStrategy values: ​ FIRST_SPEECH STTMuteStrategy Mute only during the bot’s first speech (typically during introduction) ​ MUTE_UNTIL_FIRST_BOT_COMPLETE STTMuteStrategy Start muted and remain muted until first bot speech completes. Useful when bot speaks first and you want to ensure its first response cannot be interrupted. ​ FUNCTION_CALL STTMuteStrategy Mute during LLM function calls (e.g., API requests, external service calls) ​ ALWAYS STTMuteStrategy Mute during all bot speech ​ CUSTOM STTMuteStrategy Use custom logic provided via callback to determine when to mute. The callback is invoked when the bot is speaking and can use application state to decide whether to mute. When the bot stops speaking, unmuting occurs automatically if no other strategy requires muting. MUTE_UNTIL_FIRST_BOT_COMPLETE and FIRST_SPEECH strategies should not be used together as they handle the first bot speech differently. ​ Input Frames ​ BotStartedSpeakingFrame Frame Indicates bot has started speaking ​ BotStoppedSpeakingFrame Frame Indicates bot has stopped speaking ​ FunctionCallInProgressFrame Frame Indicates a function call has started ​ FunctionCallResultFrame Frame Indicates a function call has completed ​ StartInterruptionFrame Frame User interruption start event (suppressed when muted) ​ StopInterruptionFrame Frame User interruption stop event (suppressed when muted) ​ UserStartedSpeakingFrame Frame Indicates user has started speaking (suppressed when muted) ​ UserStoppedSpeakingFrame Frame Indicates user has stopped speaking (suppressed when muted) ​ Output Frames ​ STTMuteFrame Frame Control frame to mute/unmute the STT service All input frames are passed through except VAD-related frames (interruptions and user speaking events) when muted. ​ Usage Examples ​ Basic Usage (Mute During Bot’s First Speech) Copy Ask AI stt = DeepgramSTTService( api_key = os.getenv( "DEEPGRAM_API_KEY" )) stt_mute_filter = STTMuteFilter( config = STTMuteConfig( strategies = { STTMuteStrategy. FIRST_SPEECH }) ) pipeline = Pipeline([ transport.input(), stt_mute_filter, # Add before STT service stt, # ... rest of pipeline ]) ​ Mute Until First Bot Response Completes Copy Ask AI stt_mute_filter = STTMuteFilter( config = STTMuteConfig( strategies = {STTMuteStrategy. MUTE_UNTIL_FIRST_BOT_COMPLETE }) ) This ensures no user speech is processed until after the bot’s first complete response. ​ Always Mute During Bot Speech Copy Ask AI stt_mute_filter = STTMuteFilter( config = STTMuteConfig( strategies = {STTMuteStrategy. ALWAYS }) ) ​ Custom Muting Logic The CUSTOM strategy allows you to control muting based on application state when the bot is speaking. The callback will be invoked whenever the bot is speaking, and your logic decides whether to mute: Copy Ask AI # Create a state manager class SessionState : def __init__ ( self ): self .session_ending = False session_state = SessionState() # Callback function that determines whether to mute async def session_state_mute_logic ( stt_filter : STTMuteFilter) -> bool : # Return True to mute, False otherwise # This is called when the bot is speaking return session_state.session_ending # Configure filter with CUSTOM strategy stt_mute_filter = STTMuteFilter( config = STTMuteConfig( strategies = {STTMuteStrategy. CUSTOM }, should_mute_callback = session_state_mute_logic ) ) # Later, when you want to trigger muting (e.g., during session timeout): async def handle_session_timeout (): # Update state that will be checked by the callback session_state.session_ending = True # Send goodbye message goodbye_message = "Thank you for using our service. This session is now ending." await pipeline.push_frame(TTSSpeakFrame( text = goodbye_message)) # The system will automatically mute during this message because: # 1. Bot starts speaking, triggering the callback # 2. Callback returns True (session_ending is True) # 3. When bot stops speaking, unmuting happens automatically ​ Combining Multiple Strategies Copy Ask AI async def custom_mute_logic ( processor : STTMuteFilter) -> bool : # Example: Mute during business hours only current_hour = datetime.now().hour return 9 <= current_hour < 17 stt_mute_filter = STTMuteFilter( config = STTMuteConfig( strategies = { STTMuteStrategy. FUNCTION_CALL , # Mute during function calls STTMuteStrategy. CUSTOM , # And during business hours STTMuteStrategy. MUTE_UNTIL_FIRST_BOT_COMPLETE # And until first bot speech completes }, should_mute_callback = custom_mute_logic ) ) ​ Frame Flow ​ Notes Combines STT muting and interruption control into a single concept Muting prevents both transcription and interruptions Multiple strategies can be active simultaneously CUSTOM strategy callback is only invoked when the bot is speaking Unmuting happens automatically when bot speech ends (if no other strategy requires muting) Placed before STT service in pipeline Maintains conversation flow during bot speech and function calls Efficient state tracking for minimal overhead NullFilter WakeCheckFilter On this page Overview Constructor Parameters Configuration Muting Strategies Input Frames Output Frames Usage Examples Basic Usage (Mute During Bot’s First Speech) Mute Until First Bot Response Completes Always Mute During Bot Speech Custom Muting Logic Combining Multiple Strategies Frame Flow Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/filters_stt-mute_c775a245.txt b/filters_stt-mute_c775a245.txt
new file mode 100644
index 0000000000000000000000000000000000000000..8aa1d70527a43a6cd2c2425219cc1d450b1ca5ca
--- /dev/null
+++ b/filters_stt-mute_c775a245.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/filters/stt-mute#param-should-mute-callback
+Title: STTMuteFilter - Pipecat
+==================================================
+
+STTMuteFilter - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Frame Filters STTMuteFilter Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters FrameFilter FunctionFilter IdentityFilter NullFilter STTMuteFilter WakeCheckFilter WakeNotifierFilter Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview STTMuteFilter is a general-purpose processor that combines STT muting and interruption control. When active, it prevents both transcription and interruptions during specified conditions (e.g., bot speech, function calls), providing a cleaner conversation flow. The processor supports multiple simultaneous strategies for when to mute the STT service, making it flexible for different use cases. Want to try it out? Check out the STTMuteFilter foundational demo ​ Constructor Parameters ​ config STTMuteConfig required Configuration object that defines the muting strategies and optional custom logic ​ stt_service Optional[STTService] required The STT service to control (deprecated, will be removed in a future version) ​ Configuration The processor is configured using STTMuteConfig , which determines when and how the STT service should be muted: ​ strategies set[STTMuteStrategy] Set of muting strategies to apply ​ should_mute_callback Callable[[STTMuteFilter], Awaitable[bool]] default: "None" Optional callback for custom muting logic (required when strategy is CUSTOM ) ​ Muting Strategies STTMuteConfig accepts a set of these STTMuteStrategy values: ​ FIRST_SPEECH STTMuteStrategy Mute only during the bot’s first speech (typically during introduction) ​ MUTE_UNTIL_FIRST_BOT_COMPLETE STTMuteStrategy Start muted and remain muted until first bot speech completes. Useful when bot speaks first and you want to ensure its first response cannot be interrupted. ​ FUNCTION_CALL STTMuteStrategy Mute during LLM function calls (e.g., API requests, external service calls) ​ ALWAYS STTMuteStrategy Mute during all bot speech ​ CUSTOM STTMuteStrategy Use custom logic provided via callback to determine when to mute. The callback is invoked when the bot is speaking and can use application state to decide whether to mute. When the bot stops speaking, unmuting occurs automatically if no other strategy requires muting. MUTE_UNTIL_FIRST_BOT_COMPLETE and FIRST_SPEECH strategies should not be used together as they handle the first bot speech differently. ​ Input Frames ​ BotStartedSpeakingFrame Frame Indicates bot has started speaking ​ BotStoppedSpeakingFrame Frame Indicates bot has stopped speaking ​ FunctionCallInProgressFrame Frame Indicates a function call has started ​ FunctionCallResultFrame Frame Indicates a function call has completed ​ StartInterruptionFrame Frame User interruption start event (suppressed when muted) ​ StopInterruptionFrame Frame User interruption stop event (suppressed when muted) ​ UserStartedSpeakingFrame Frame Indicates user has started speaking (suppressed when muted) ​ UserStoppedSpeakingFrame Frame Indicates user has stopped speaking (suppressed when muted) ​ Output Frames ​ STTMuteFrame Frame Control frame to mute/unmute the STT service All input frames are passed through except VAD-related frames (interruptions and user speaking events) when muted. ​ Usage Examples ​ Basic Usage (Mute During Bot’s First Speech) Copy Ask AI stt = DeepgramSTTService( api_key = os.getenv( "DEEPGRAM_API_KEY" )) stt_mute_filter = STTMuteFilter( config = STTMuteConfig( strategies = { STTMuteStrategy. FIRST_SPEECH }) ) pipeline = Pipeline([ transport.input(), stt_mute_filter, # Add before STT service stt, # ... rest of pipeline ]) ​ Mute Until First Bot Response Completes Copy Ask AI stt_mute_filter = STTMuteFilter( config = STTMuteConfig( strategies = {STTMuteStrategy. MUTE_UNTIL_FIRST_BOT_COMPLETE }) ) This ensures no user speech is processed until after the bot’s first complete response. ​ Always Mute During Bot Speech Copy Ask AI stt_mute_filter = STTMuteFilter( config = STTMuteConfig( strategies = {STTMuteStrategy. ALWAYS }) ) ​ Custom Muting Logic The CUSTOM strategy allows you to control muting based on application state when the bot is speaking. The callback will be invoked whenever the bot is speaking, and your logic decides whether to mute: Copy Ask AI # Create a state manager class SessionState : def __init__ ( self ): self .session_ending = False session_state = SessionState() # Callback function that determines whether to mute async def session_state_mute_logic ( stt_filter : STTMuteFilter) -> bool : # Return True to mute, False otherwise # This is called when the bot is speaking return session_state.session_ending # Configure filter with CUSTOM strategy stt_mute_filter = STTMuteFilter( config = STTMuteConfig( strategies = {STTMuteStrategy. CUSTOM }, should_mute_callback = session_state_mute_logic ) ) # Later, when you want to trigger muting (e.g., during session timeout): async def handle_session_timeout (): # Update state that will be checked by the callback session_state.session_ending = True # Send goodbye message goodbye_message = "Thank you for using our service. This session is now ending." await pipeline.push_frame(TTSSpeakFrame( text = goodbye_message)) # The system will automatically mute during this message because: # 1. Bot starts speaking, triggering the callback # 2. Callback returns True (session_ending is True) # 3. When bot stops speaking, unmuting happens automatically ​ Combining Multiple Strategies Copy Ask AI async def custom_mute_logic ( processor : STTMuteFilter) -> bool : # Example: Mute during business hours only current_hour = datetime.now().hour return 9 <= current_hour < 17 stt_mute_filter = STTMuteFilter( config = STTMuteConfig( strategies = { STTMuteStrategy. FUNCTION_CALL , # Mute during function calls STTMuteStrategy. CUSTOM , # And during business hours STTMuteStrategy. MUTE_UNTIL_FIRST_BOT_COMPLETE # And until first bot speech completes }, should_mute_callback = custom_mute_logic ) ) ​ Frame Flow ​ Notes Combines STT muting and interruption control into a single concept Muting prevents both transcription and interruptions Multiple strategies can be active simultaneously CUSTOM strategy callback is only invoked when the bot is speaking Unmuting happens automatically when bot speech ends (if no other strategy requires muting) Placed before STT service in pipeline Maintains conversation flow during bot speech and function calls Efficient state tracking for minimal overhead NullFilter WakeCheckFilter On this page Overview Constructor Parameters Configuration Muting Strategies Input Frames Output Frames Usage Examples Basic Usage (Mute During Bot’s First Speech) Mute Until First Bot Response Completes Always Mute During Bot Speech Custom Muting Logic Combining Multiple Strategies Frame Flow Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/filters_stt-mute_fe447232.txt b/filters_stt-mute_fe447232.txt
new file mode 100644
index 0000000000000000000000000000000000000000..653a126df181485621ff018aa02697c7fa40d622
--- /dev/null
+++ b/filters_stt-mute_fe447232.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/filters/stt-mute#usage-examples
+Title: STTMuteFilter - Pipecat
+==================================================
+
+STTMuteFilter - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Frame Filters STTMuteFilter Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters FrameFilter FunctionFilter IdentityFilter NullFilter STTMuteFilter WakeCheckFilter WakeNotifierFilter Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview STTMuteFilter is a general-purpose processor that combines STT muting and interruption control. When active, it prevents both transcription and interruptions during specified conditions (e.g., bot speech, function calls), providing a cleaner conversation flow. The processor supports multiple simultaneous strategies for when to mute the STT service, making it flexible for different use cases. Want to try it out? Check out the STTMuteFilter foundational demo ​ Constructor Parameters ​ config STTMuteConfig required Configuration object that defines the muting strategies and optional custom logic ​ stt_service Optional[STTService] required The STT service to control (deprecated, will be removed in a future version) ​ Configuration The processor is configured using STTMuteConfig , which determines when and how the STT service should be muted: ​ strategies set[STTMuteStrategy] Set of muting strategies to apply ​ should_mute_callback Callable[[STTMuteFilter], Awaitable[bool]] default: "None" Optional callback for custom muting logic (required when strategy is CUSTOM ) ​ Muting Strategies STTMuteConfig accepts a set of these STTMuteStrategy values: ​ FIRST_SPEECH STTMuteStrategy Mute only during the bot’s first speech (typically during introduction) ​ MUTE_UNTIL_FIRST_BOT_COMPLETE STTMuteStrategy Start muted and remain muted until first bot speech completes. Useful when bot speaks first and you want to ensure its first response cannot be interrupted. ​ FUNCTION_CALL STTMuteStrategy Mute during LLM function calls (e.g., API requests, external service calls) ​ ALWAYS STTMuteStrategy Mute during all bot speech ​ CUSTOM STTMuteStrategy Use custom logic provided via callback to determine when to mute. The callback is invoked when the bot is speaking and can use application state to decide whether to mute. When the bot stops speaking, unmuting occurs automatically if no other strategy requires muting. MUTE_UNTIL_FIRST_BOT_COMPLETE and FIRST_SPEECH strategies should not be used together as they handle the first bot speech differently. ​ Input Frames ​ BotStartedSpeakingFrame Frame Indicates bot has started speaking ​ BotStoppedSpeakingFrame Frame Indicates bot has stopped speaking ​ FunctionCallInProgressFrame Frame Indicates a function call has started ​ FunctionCallResultFrame Frame Indicates a function call has completed ​ StartInterruptionFrame Frame User interruption start event (suppressed when muted) ​ StopInterruptionFrame Frame User interruption stop event (suppressed when muted) ​ UserStartedSpeakingFrame Frame Indicates user has started speaking (suppressed when muted) ​ UserStoppedSpeakingFrame Frame Indicates user has stopped speaking (suppressed when muted) ​ Output Frames ​ STTMuteFrame Frame Control frame to mute/unmute the STT service All input frames are passed through except VAD-related frames (interruptions and user speaking events) when muted. ​ Usage Examples ​ Basic Usage (Mute During Bot’s First Speech) Copy Ask AI stt = DeepgramSTTService( api_key = os.getenv( "DEEPGRAM_API_KEY" )) stt_mute_filter = STTMuteFilter( config = STTMuteConfig( strategies = { STTMuteStrategy. FIRST_SPEECH }) ) pipeline = Pipeline([ transport.input(), stt_mute_filter, # Add before STT service stt, # ... rest of pipeline ]) ​ Mute Until First Bot Response Completes Copy Ask AI stt_mute_filter = STTMuteFilter( config = STTMuteConfig( strategies = {STTMuteStrategy. MUTE_UNTIL_FIRST_BOT_COMPLETE }) ) This ensures no user speech is processed until after the bot’s first complete response. ​ Always Mute During Bot Speech Copy Ask AI stt_mute_filter = STTMuteFilter( config = STTMuteConfig( strategies = {STTMuteStrategy. ALWAYS }) ) ​ Custom Muting Logic The CUSTOM strategy allows you to control muting based on application state when the bot is speaking. The callback will be invoked whenever the bot is speaking, and your logic decides whether to mute: Copy Ask AI # Create a state manager class SessionState : def __init__ ( self ): self .session_ending = False session_state = SessionState() # Callback function that determines whether to mute async def session_state_mute_logic ( stt_filter : STTMuteFilter) -> bool : # Return True to mute, False otherwise # This is called when the bot is speaking return session_state.session_ending # Configure filter with CUSTOM strategy stt_mute_filter = STTMuteFilter( config = STTMuteConfig( strategies = {STTMuteStrategy. CUSTOM }, should_mute_callback = session_state_mute_logic ) ) # Later, when you want to trigger muting (e.g., during session timeout): async def handle_session_timeout (): # Update state that will be checked by the callback session_state.session_ending = True # Send goodbye message goodbye_message = "Thank you for using our service. This session is now ending." await pipeline.push_frame(TTSSpeakFrame( text = goodbye_message)) # The system will automatically mute during this message because: # 1. Bot starts speaking, triggering the callback # 2. Callback returns True (session_ending is True) # 3. When bot stops speaking, unmuting happens automatically ​ Combining Multiple Strategies Copy Ask AI async def custom_mute_logic ( processor : STTMuteFilter) -> bool : # Example: Mute during business hours only current_hour = datetime.now().hour return 9 <= current_hour < 17 stt_mute_filter = STTMuteFilter( config = STTMuteConfig( strategies = { STTMuteStrategy. FUNCTION_CALL , # Mute during function calls STTMuteStrategy. CUSTOM , # And during business hours STTMuteStrategy. MUTE_UNTIL_FIRST_BOT_COMPLETE # And until first bot speech completes }, should_mute_callback = custom_mute_logic ) ) ​ Frame Flow ​ Notes Combines STT muting and interruption control into a single concept Muting prevents both transcription and interruptions Multiple strategies can be active simultaneously CUSTOM strategy callback is only invoked when the bot is speaking Unmuting happens automatically when bot speech ends (if no other strategy requires muting) Placed before STT service in pipeline Maintains conversation flow during bot speech and function calls Efficient state tracking for minimal overhead NullFilter WakeCheckFilter On this page Overview Constructor Parameters Configuration Muting Strategies Input Frames Output Frames Usage Examples Basic Usage (Mute During Bot’s First Speech) Mute Until First Bot Response Completes Always Mute During Bot Speech Custom Muting Logic Combining Multiple Strategies Frame Flow Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/filters_wake-check-filter_266300d0.txt b/filters_wake-check-filter_266300d0.txt
new file mode 100644
index 0000000000000000000000000000000000000000..1d2648750b3417f7d35be117216d2f473079ad81
--- /dev/null
+++ b/filters_wake-check-filter_266300d0.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/filters/wake-check-filter#param-awake
+Title: WakeCheckFilter - Pipecat
+==================================================
+
+WakeCheckFilter - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Frame Filters WakeCheckFilter Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters FrameFilter FunctionFilter IdentityFilter NullFilter STTMuteFilter WakeCheckFilter WakeNotifierFilter Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview WakeCheckFilter monitors TranscriptionFrame s for specified wake phrases and only allows frames to pass through after a wake phrase has been detected. It includes a keepalive timeout to maintain the awake state for a period after detection, allowing continuous conversation without requiring repeated wake phrases. ​ Constructor Parameters ​ wake_phrases list[str] required List of wake phrases to detect in transcriptions ​ keepalive_timeout float default: "3" Number of seconds to remain in the awake state after each transcription ​ Functionality The filter maintains state for each participant and processes frames as follows: TranscriptionFrame objects are checked for wake phrases If a wake phrase is detected, the filter enters the “AWAKE” state While in the “AWAKE” state, all transcription frames pass through After no activity for the keepalive timeout period, the filter returns to “IDLE” All non-transcription frames pass through normally Wake phrases are detected using regular expressions that match whole words with flexible spacing, making detection resilient to minor transcription variations. ​ States ​ IDLE WakeState Default state - only non-transcription frames pass through ​ AWAKE WakeState Active state after wake phrase detection - all frames pass through ​ Output Frames All non-transcription frames pass through unchanged After wake phrase detection, transcription frames pass through When awake, transcription frames reset the keepalive timer ​ Usage Example Copy Ask AI from pipecat.processors.filters import WakeCheckFilter # Create filter with wake phrases wake_filter = WakeCheckFilter( wake_phrases = [ "hey assistant" , "ok computer" , "listen up" ], keepalive_timeout = 5.0 # Stay awake for 5 seconds after each transcription ) # Add to pipeline pipeline = Pipeline([ transport.input(), stt_service, wake_filter, # Only passes transcriptions after wake phrases llm_service, tts_service, transport.output() ]) ​ Frame Flow ​ Notes Maintains separate state for each participant ID Uses regex pattern matching for resilient wake phrase detection Accumulates transcription text to detect phrases across multiple frames Trims accumulated text when wake phrase is detected Supports multiple wake phrases Passes all non-transcription frames through unchanged Error handling produces ErrorFrames for robust operation Case-insensitive matching for natural language use STTMuteFilter WakeNotifierFilter On this page Overview Constructor Parameters Functionality States Output Frames Usage Example Frame Flow Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/filters_wake-check-filter_2e14840a.txt b/filters_wake-check-filter_2e14840a.txt
new file mode 100644
index 0000000000000000000000000000000000000000..3738b37f40e070ee40bf79c72872463dac5503ee
--- /dev/null
+++ b/filters_wake-check-filter_2e14840a.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/filters/wake-check-filter#usage-example
+Title: WakeCheckFilter - Pipecat
+==================================================
+
+WakeCheckFilter - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Frame Filters WakeCheckFilter Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters FrameFilter FunctionFilter IdentityFilter NullFilter STTMuteFilter WakeCheckFilter WakeNotifierFilter Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview WakeCheckFilter monitors TranscriptionFrame s for specified wake phrases and only allows frames to pass through after a wake phrase has been detected. It includes a keepalive timeout to maintain the awake state for a period after detection, allowing continuous conversation without requiring repeated wake phrases. ​ Constructor Parameters ​ wake_phrases list[str] required List of wake phrases to detect in transcriptions ​ keepalive_timeout float default: "3" Number of seconds to remain in the awake state after each transcription ​ Functionality The filter maintains state for each participant and processes frames as follows: TranscriptionFrame objects are checked for wake phrases If a wake phrase is detected, the filter enters the “AWAKE” state While in the “AWAKE” state, all transcription frames pass through After no activity for the keepalive timeout period, the filter returns to “IDLE” All non-transcription frames pass through normally Wake phrases are detected using regular expressions that match whole words with flexible spacing, making detection resilient to minor transcription variations. ​ States ​ IDLE WakeState Default state - only non-transcription frames pass through ​ AWAKE WakeState Active state after wake phrase detection - all frames pass through ​ Output Frames All non-transcription frames pass through unchanged After wake phrase detection, transcription frames pass through When awake, transcription frames reset the keepalive timer ​ Usage Example Copy Ask AI from pipecat.processors.filters import WakeCheckFilter # Create filter with wake phrases wake_filter = WakeCheckFilter( wake_phrases = [ "hey assistant" , "ok computer" , "listen up" ], keepalive_timeout = 5.0 # Stay awake for 5 seconds after each transcription ) # Add to pipeline pipeline = Pipeline([ transport.input(), stt_service, wake_filter, # Only passes transcriptions after wake phrases llm_service, tts_service, transport.output() ]) ​ Frame Flow ​ Notes Maintains separate state for each participant ID Uses regex pattern matching for resilient wake phrase detection Accumulates transcription text to detect phrases across multiple frames Trims accumulated text when wake phrase is detected Supports multiple wake phrases Passes all non-transcription frames through unchanged Error handling produces ErrorFrames for robust operation Case-insensitive matching for natural language use STTMuteFilter WakeNotifierFilter On this page Overview Constructor Parameters Functionality States Output Frames Usage Example Frame Flow Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/filters_wake-check-filter_5b1dba15.txt b/filters_wake-check-filter_5b1dba15.txt
new file mode 100644
index 0000000000000000000000000000000000000000..039797f3c7f864f5405a9a1451efdd9e861b5670
--- /dev/null
+++ b/filters_wake-check-filter_5b1dba15.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/filters/wake-check-filter#overview
+Title: WakeCheckFilter - Pipecat
+==================================================
+
+WakeCheckFilter - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Frame Filters WakeCheckFilter Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters FrameFilter FunctionFilter IdentityFilter NullFilter STTMuteFilter WakeCheckFilter WakeNotifierFilter Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview WakeCheckFilter monitors TranscriptionFrame s for specified wake phrases and only allows frames to pass through after a wake phrase has been detected. It includes a keepalive timeout to maintain the awake state for a period after detection, allowing continuous conversation without requiring repeated wake phrases. ​ Constructor Parameters ​ wake_phrases list[str] required List of wake phrases to detect in transcriptions ​ keepalive_timeout float default: "3" Number of seconds to remain in the awake state after each transcription ​ Functionality The filter maintains state for each participant and processes frames as follows: TranscriptionFrame objects are checked for wake phrases If a wake phrase is detected, the filter enters the “AWAKE” state While in the “AWAKE” state, all transcription frames pass through After no activity for the keepalive timeout period, the filter returns to “IDLE” All non-transcription frames pass through normally Wake phrases are detected using regular expressions that match whole words with flexible spacing, making detection resilient to minor transcription variations. ​ States ​ IDLE WakeState Default state - only non-transcription frames pass through ​ AWAKE WakeState Active state after wake phrase detection - all frames pass through ​ Output Frames All non-transcription frames pass through unchanged After wake phrase detection, transcription frames pass through When awake, transcription frames reset the keepalive timer ​ Usage Example Copy Ask AI from pipecat.processors.filters import WakeCheckFilter # Create filter with wake phrases wake_filter = WakeCheckFilter( wake_phrases = [ "hey assistant" , "ok computer" , "listen up" ], keepalive_timeout = 5.0 # Stay awake for 5 seconds after each transcription ) # Add to pipeline pipeline = Pipeline([ transport.input(), stt_service, wake_filter, # Only passes transcriptions after wake phrases llm_service, tts_service, transport.output() ]) ​ Frame Flow ​ Notes Maintains separate state for each participant ID Uses regex pattern matching for resilient wake phrase detection Accumulates transcription text to detect phrases across multiple frames Trims accumulated text when wake phrase is detected Supports multiple wake phrases Passes all non-transcription frames through unchanged Error handling produces ErrorFrames for robust operation Case-insensitive matching for natural language use STTMuteFilter WakeNotifierFilter On this page Overview Constructor Parameters Functionality States Output Frames Usage Example Frame Flow Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/filters_wake-check-filter_77836018.txt b/filters_wake-check-filter_77836018.txt
new file mode 100644
index 0000000000000000000000000000000000000000..500da51233d28e7398533ca0807ab9ad4bf46a97
--- /dev/null
+++ b/filters_wake-check-filter_77836018.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/filters/wake-check-filter#output-frames
+Title: WakeCheckFilter - Pipecat
+==================================================
+
+WakeCheckFilter - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Frame Filters WakeCheckFilter Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters FrameFilter FunctionFilter IdentityFilter NullFilter STTMuteFilter WakeCheckFilter WakeNotifierFilter Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview WakeCheckFilter monitors TranscriptionFrame s for specified wake phrases and only allows frames to pass through after a wake phrase has been detected. It includes a keepalive timeout to maintain the awake state for a period after detection, allowing continuous conversation without requiring repeated wake phrases. ​ Constructor Parameters ​ wake_phrases list[str] required List of wake phrases to detect in transcriptions ​ keepalive_timeout float default: "3" Number of seconds to remain in the awake state after each transcription ​ Functionality The filter maintains state for each participant and processes frames as follows: TranscriptionFrame objects are checked for wake phrases If a wake phrase is detected, the filter enters the “AWAKE” state While in the “AWAKE” state, all transcription frames pass through After no activity for the keepalive timeout period, the filter returns to “IDLE” All non-transcription frames pass through normally Wake phrases are detected using regular expressions that match whole words with flexible spacing, making detection resilient to minor transcription variations. ​ States ​ IDLE WakeState Default state - only non-transcription frames pass through ​ AWAKE WakeState Active state after wake phrase detection - all frames pass through ​ Output Frames All non-transcription frames pass through unchanged After wake phrase detection, transcription frames pass through When awake, transcription frames reset the keepalive timer ​ Usage Example Copy Ask AI from pipecat.processors.filters import WakeCheckFilter # Create filter with wake phrases wake_filter = WakeCheckFilter( wake_phrases = [ "hey assistant" , "ok computer" , "listen up" ], keepalive_timeout = 5.0 # Stay awake for 5 seconds after each transcription ) # Add to pipeline pipeline = Pipeline([ transport.input(), stt_service, wake_filter, # Only passes transcriptions after wake phrases llm_service, tts_service, transport.output() ]) ​ Frame Flow ​ Notes Maintains separate state for each participant ID Uses regex pattern matching for resilient wake phrase detection Accumulates transcription text to detect phrases across multiple frames Trims accumulated text when wake phrase is detected Supports multiple wake phrases Passes all non-transcription frames through unchanged Error handling produces ErrorFrames for robust operation Case-insensitive matching for natural language use STTMuteFilter WakeNotifierFilter On this page Overview Constructor Parameters Functionality States Output Frames Usage Example Frame Flow Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/filters_wake-check-filter_88106ed9.txt b/filters_wake-check-filter_88106ed9.txt
new file mode 100644
index 0000000000000000000000000000000000000000..6ac5947cd4d15a0c879271b9de3a7a3774950d7c
--- /dev/null
+++ b/filters_wake-check-filter_88106ed9.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/filters/wake-check-filter#frame-flow
+Title: WakeCheckFilter - Pipecat
+==================================================
+
+WakeCheckFilter - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Frame Filters WakeCheckFilter Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters FrameFilter FunctionFilter IdentityFilter NullFilter STTMuteFilter WakeCheckFilter WakeNotifierFilter Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview WakeCheckFilter monitors TranscriptionFrame s for specified wake phrases and only allows frames to pass through after a wake phrase has been detected. It includes a keepalive timeout to maintain the awake state for a period after detection, allowing continuous conversation without requiring repeated wake phrases. ​ Constructor Parameters ​ wake_phrases list[str] required List of wake phrases to detect in transcriptions ​ keepalive_timeout float default: "3" Number of seconds to remain in the awake state after each transcription ​ Functionality The filter maintains state for each participant and processes frames as follows: TranscriptionFrame objects are checked for wake phrases If a wake phrase is detected, the filter enters the “AWAKE” state While in the “AWAKE” state, all transcription frames pass through After no activity for the keepalive timeout period, the filter returns to “IDLE” All non-transcription frames pass through normally Wake phrases are detected using regular expressions that match whole words with flexible spacing, making detection resilient to minor transcription variations. ​ States ​ IDLE WakeState Default state - only non-transcription frames pass through ​ AWAKE WakeState Active state after wake phrase detection - all frames pass through ​ Output Frames All non-transcription frames pass through unchanged After wake phrase detection, transcription frames pass through When awake, transcription frames reset the keepalive timer ​ Usage Example Copy Ask AI from pipecat.processors.filters import WakeCheckFilter # Create filter with wake phrases wake_filter = WakeCheckFilter( wake_phrases = [ "hey assistant" , "ok computer" , "listen up" ], keepalive_timeout = 5.0 # Stay awake for 5 seconds after each transcription ) # Add to pipeline pipeline = Pipeline([ transport.input(), stt_service, wake_filter, # Only passes transcriptions after wake phrases llm_service, tts_service, transport.output() ]) ​ Frame Flow ​ Notes Maintains separate state for each participant ID Uses regex pattern matching for resilient wake phrase detection Accumulates transcription text to detect phrases across multiple frames Trims accumulated text when wake phrase is detected Supports multiple wake phrases Passes all non-transcription frames through unchanged Error handling produces ErrorFrames for robust operation Case-insensitive matching for natural language use STTMuteFilter WakeNotifierFilter On this page Overview Constructor Parameters Functionality States Output Frames Usage Example Frame Flow Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/filters_wake-check-filter_b1d28436.txt b/filters_wake-check-filter_b1d28436.txt
new file mode 100644
index 0000000000000000000000000000000000000000..cd3a97d26aaf3d58b63199826573f2bea11a4f4c
--- /dev/null
+++ b/filters_wake-check-filter_b1d28436.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/filters/wake-check-filter#functionality
+Title: WakeCheckFilter - Pipecat
+==================================================
+
+WakeCheckFilter - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Frame Filters WakeCheckFilter Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters FrameFilter FunctionFilter IdentityFilter NullFilter STTMuteFilter WakeCheckFilter WakeNotifierFilter Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview WakeCheckFilter monitors TranscriptionFrame s for specified wake phrases and only allows frames to pass through after a wake phrase has been detected. It includes a keepalive timeout to maintain the awake state for a period after detection, allowing continuous conversation without requiring repeated wake phrases. ​ Constructor Parameters ​ wake_phrases list[str] required List of wake phrases to detect in transcriptions ​ keepalive_timeout float default: "3" Number of seconds to remain in the awake state after each transcription ​ Functionality The filter maintains state for each participant and processes frames as follows: TranscriptionFrame objects are checked for wake phrases If a wake phrase is detected, the filter enters the “AWAKE” state While in the “AWAKE” state, all transcription frames pass through After no activity for the keepalive timeout period, the filter returns to “IDLE” All non-transcription frames pass through normally Wake phrases are detected using regular expressions that match whole words with flexible spacing, making detection resilient to minor transcription variations. ​ States ​ IDLE WakeState Default state - only non-transcription frames pass through ​ AWAKE WakeState Active state after wake phrase detection - all frames pass through ​ Output Frames All non-transcription frames pass through unchanged After wake phrase detection, transcription frames pass through When awake, transcription frames reset the keepalive timer ​ Usage Example Copy Ask AI from pipecat.processors.filters import WakeCheckFilter # Create filter with wake phrases wake_filter = WakeCheckFilter( wake_phrases = [ "hey assistant" , "ok computer" , "listen up" ], keepalive_timeout = 5.0 # Stay awake for 5 seconds after each transcription ) # Add to pipeline pipeline = Pipeline([ transport.input(), stt_service, wake_filter, # Only passes transcriptions after wake phrases llm_service, tts_service, transport.output() ]) ​ Frame Flow ​ Notes Maintains separate state for each participant ID Uses regex pattern matching for resilient wake phrase detection Accumulates transcription text to detect phrases across multiple frames Trims accumulated text when wake phrase is detected Supports multiple wake phrases Passes all non-transcription frames through unchanged Error handling produces ErrorFrames for robust operation Case-insensitive matching for natural language use STTMuteFilter WakeNotifierFilter On this page Overview Constructor Parameters Functionality States Output Frames Usage Example Frame Flow Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/filters_wake-notifier-filter_40cb2d80.txt b/filters_wake-notifier-filter_40cb2d80.txt
new file mode 100644
index 0000000000000000000000000000000000000000..8dbc8cf7c2f9da2b3f8794e2802dd552751931e4
--- /dev/null
+++ b/filters_wake-notifier-filter_40cb2d80.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/filters/wake-notifier-filter#usage-example
+Title: WakeNotifierFilter - Pipecat
+==================================================
+
+WakeNotifierFilter - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Frame Filters WakeNotifierFilter Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters FrameFilter FunctionFilter IdentityFilter NullFilter STTMuteFilter WakeCheckFilter WakeNotifierFilter Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview WakeNotifierFilter monitors the pipeline for specific frame types and triggers a notification when those frames pass a custom filter condition. It passes all frames through unchanged while performing this notification side-effect. ​ Constructor Parameters ​ notifier BaseNotifier required The notifier object to trigger when conditions are met ​ types Tuple[Type[Frame]] required Tuple of frame types to monitor ​ filter Callable[[Frame], Awaitable[bool]] required Async function that examines each matching frame and returns True to trigger notification ​ Functionality The processor operates as follows: Checks if the incoming frame matches any of the specified types If it’s a matching type, calls the filter function with the frame If the filter returns True, triggers the notifier Passes all frames through unchanged, regardless of the filtering result This allows for notification side-effects without modifying the pipeline’s data flow. ​ Output Frames All frames pass through unchanged in their original direction No frames are modified or filtered out ​ Usage Example Copy Ask AI from pipecat.frames.frames import TranscriptionFrame, UserStartedSpeakingFrame from pipecat.processors.filters import WakeNotifierFilter from pipecat.sync.event_notifier import EventNotifier # Create an event notifier wake_event = EventNotifier() # Create filter that notifies when certain wake phrases are detected async def wake_phrase_filter ( frame ): if isinstance (frame, TranscriptionFrame): return "hey assistant" in frame.text.lower() return False # Add to pipeline wake_notifier = WakeNotifierFilter( notifier = wake_event, types = (TranscriptionFrame, UserStartedSpeakingFrame), filter = wake_phrase_filter ) # In another component, wait for the notification async def handle_wake_event (): await wake_event.wait() print ( "Wake phrase detected!" ) ​ Frame Flow ​ Notes Acts as a transparent pass-through for all frames Can trigger external events without modifying pipeline flow Useful for signaling between pipeline components Can monitor for multiple frame types simultaneously Uses async filter function for complex conditions Functions as a “listener” that doesn’t affect the data stream Can be used for logging, analytics, or coordinating external systems WakeCheckFilter OpenTelemetry On this page Overview Constructor Parameters Functionality Output Frames Usage Example Frame Flow Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/filters_wake-notifier-filter_471e5698.txt b/filters_wake-notifier-filter_471e5698.txt
new file mode 100644
index 0000000000000000000000000000000000000000..5c5014e14b8f345637188798ac7f598bea22db1d
--- /dev/null
+++ b/filters_wake-notifier-filter_471e5698.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/filters/wake-notifier-filter#param-notifier
+Title: WakeNotifierFilter - Pipecat
+==================================================
+
+WakeNotifierFilter - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Frame Filters WakeNotifierFilter Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters FrameFilter FunctionFilter IdentityFilter NullFilter STTMuteFilter WakeCheckFilter WakeNotifierFilter Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview WakeNotifierFilter monitors the pipeline for specific frame types and triggers a notification when those frames pass a custom filter condition. It passes all frames through unchanged while performing this notification side-effect. ​ Constructor Parameters ​ notifier BaseNotifier required The notifier object to trigger when conditions are met ​ types Tuple[Type[Frame]] required Tuple of frame types to monitor ​ filter Callable[[Frame], Awaitable[bool]] required Async function that examines each matching frame and returns True to trigger notification ​ Functionality The processor operates as follows: Checks if the incoming frame matches any of the specified types If it’s a matching type, calls the filter function with the frame If the filter returns True, triggers the notifier Passes all frames through unchanged, regardless of the filtering result This allows for notification side-effects without modifying the pipeline’s data flow. ​ Output Frames All frames pass through unchanged in their original direction No frames are modified or filtered out ​ Usage Example Copy Ask AI from pipecat.frames.frames import TranscriptionFrame, UserStartedSpeakingFrame from pipecat.processors.filters import WakeNotifierFilter from pipecat.sync.event_notifier import EventNotifier # Create an event notifier wake_event = EventNotifier() # Create filter that notifies when certain wake phrases are detected async def wake_phrase_filter ( frame ): if isinstance (frame, TranscriptionFrame): return "hey assistant" in frame.text.lower() return False # Add to pipeline wake_notifier = WakeNotifierFilter( notifier = wake_event, types = (TranscriptionFrame, UserStartedSpeakingFrame), filter = wake_phrase_filter ) # In another component, wait for the notification async def handle_wake_event (): await wake_event.wait() print ( "Wake phrase detected!" ) ​ Frame Flow ​ Notes Acts as a transparent pass-through for all frames Can trigger external events without modifying pipeline flow Useful for signaling between pipeline components Can monitor for multiple frame types simultaneously Uses async filter function for complex conditions Functions as a “listener” that doesn’t affect the data stream Can be used for logging, analytics, or coordinating external systems WakeCheckFilter OpenTelemetry On this page Overview Constructor Parameters Functionality Output Frames Usage Example Frame Flow Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/filters_wake-notifier-filter_700fea10.txt b/filters_wake-notifier-filter_700fea10.txt
new file mode 100644
index 0000000000000000000000000000000000000000..99ce3ab96afdff7221bf8415370e9d7e5bb836ba
--- /dev/null
+++ b/filters_wake-notifier-filter_700fea10.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/filters/wake-notifier-filter#frame-flow
+Title: WakeNotifierFilter - Pipecat
+==================================================
+
+WakeNotifierFilter - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Frame Filters WakeNotifierFilter Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters FrameFilter FunctionFilter IdentityFilter NullFilter STTMuteFilter WakeCheckFilter WakeNotifierFilter Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview WakeNotifierFilter monitors the pipeline for specific frame types and triggers a notification when those frames pass a custom filter condition. It passes all frames through unchanged while performing this notification side-effect. ​ Constructor Parameters ​ notifier BaseNotifier required The notifier object to trigger when conditions are met ​ types Tuple[Type[Frame]] required Tuple of frame types to monitor ​ filter Callable[[Frame], Awaitable[bool]] required Async function that examines each matching frame and returns True to trigger notification ​ Functionality The processor operates as follows: Checks if the incoming frame matches any of the specified types If it’s a matching type, calls the filter function with the frame If the filter returns True, triggers the notifier Passes all frames through unchanged, regardless of the filtering result This allows for notification side-effects without modifying the pipeline’s data flow. ​ Output Frames All frames pass through unchanged in their original direction No frames are modified or filtered out ​ Usage Example Copy Ask AI from pipecat.frames.frames import TranscriptionFrame, UserStartedSpeakingFrame from pipecat.processors.filters import WakeNotifierFilter from pipecat.sync.event_notifier import EventNotifier # Create an event notifier wake_event = EventNotifier() # Create filter that notifies when certain wake phrases are detected async def wake_phrase_filter ( frame ): if isinstance (frame, TranscriptionFrame): return "hey assistant" in frame.text.lower() return False # Add to pipeline wake_notifier = WakeNotifierFilter( notifier = wake_event, types = (TranscriptionFrame, UserStartedSpeakingFrame), filter = wake_phrase_filter ) # In another component, wait for the notification async def handle_wake_event (): await wake_event.wait() print ( "Wake phrase detected!" ) ​ Frame Flow ​ Notes Acts as a transparent pass-through for all frames Can trigger external events without modifying pipeline flow Useful for signaling between pipeline components Can monitor for multiple frame types simultaneously Uses async filter function for complex conditions Functions as a “listener” that doesn’t affect the data stream Can be used for logging, analytics, or coordinating external systems WakeCheckFilter OpenTelemetry On this page Overview Constructor Parameters Functionality Output Frames Usage Example Frame Flow Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/filters_wake-notifier-filter_ea00cb63.txt b/filters_wake-notifier-filter_ea00cb63.txt
new file mode 100644
index 0000000000000000000000000000000000000000..39cfe0a708d93f7a8f1434b6d3c72d4949bcf455
--- /dev/null
+++ b/filters_wake-notifier-filter_ea00cb63.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/filters/wake-notifier-filter#overview
+Title: WakeNotifierFilter - Pipecat
+==================================================
+
+WakeNotifierFilter - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Frame Filters WakeNotifierFilter Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters FrameFilter FunctionFilter IdentityFilter NullFilter STTMuteFilter WakeCheckFilter WakeNotifierFilter Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview WakeNotifierFilter monitors the pipeline for specific frame types and triggers a notification when those frames pass a custom filter condition. It passes all frames through unchanged while performing this notification side-effect. ​ Constructor Parameters ​ notifier BaseNotifier required The notifier object to trigger when conditions are met ​ types Tuple[Type[Frame]] required Tuple of frame types to monitor ​ filter Callable[[Frame], Awaitable[bool]] required Async function that examines each matching frame and returns True to trigger notification ​ Functionality The processor operates as follows: Checks if the incoming frame matches any of the specified types If it’s a matching type, calls the filter function with the frame If the filter returns True, triggers the notifier Passes all frames through unchanged, regardless of the filtering result This allows for notification side-effects without modifying the pipeline’s data flow. ​ Output Frames All frames pass through unchanged in their original direction No frames are modified or filtered out ​ Usage Example Copy Ask AI from pipecat.frames.frames import TranscriptionFrame, UserStartedSpeakingFrame from pipecat.processors.filters import WakeNotifierFilter from pipecat.sync.event_notifier import EventNotifier # Create an event notifier wake_event = EventNotifier() # Create filter that notifies when certain wake phrases are detected async def wake_phrase_filter ( frame ): if isinstance (frame, TranscriptionFrame): return "hey assistant" in frame.text.lower() return False # Add to pipeline wake_notifier = WakeNotifierFilter( notifier = wake_event, types = (TranscriptionFrame, UserStartedSpeakingFrame), filter = wake_phrase_filter ) # In another component, wait for the notification async def handle_wake_event (): await wake_event.wait() print ( "Wake phrase detected!" ) ​ Frame Flow ​ Notes Acts as a transparent pass-through for all frames Can trigger external events without modifying pipeline flow Useful for signaling between pipeline components Can monitor for multiple frame types simultaneously Uses async filter function for complex conditions Functions as a “listener” that doesn’t affect the data stream Can be used for logging, analytics, or coordinating external systems WakeCheckFilter OpenTelemetry On this page Overview Constructor Parameters Functionality Output Frames Usage Example Frame Flow Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/flows_pipecat-flows_060d8749.txt b/flows_pipecat-flows_060d8749.txt
new file mode 100644
index 0000000000000000000000000000000000000000..f431aab9aa38208d87b1acb47ac72e8021bcf0b8
--- /dev/null
+++ b/flows_pipecat-flows_060d8749.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/frameworks/flows/pipecat-flows#param-flow-initialization-error-1
+Title: Pipecat Flows - Pipecat
+==================================================
+
+Pipecat Flows - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Frameworks Pipecat Flows Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline New to building conversational flows? Check out our Pipecat Flows guide first. ​ Installation Existing Pipecat installation Fresh Pipecat installation Copy Ask AI pip install pipecat-ai-flows ​ Core Types ​ FlowArgs ​ FlowArgs Dict[str, Any] Type alias for function handler arguments. ​ FlowResult ​ FlowResult TypedDict Base type for function handler results. Additional fields can be included as needed. Show Fields ​ status str Optional status field ​ error str Optional error message ​ FlowConfig ​ FlowConfig TypedDict Configuration for the entire conversation flow. Show Fields ​ initial_node str required Starting node identifier ​ nodes Dict[str, NodeConfig] required Map of node names to configurations ​ NodeConfig ​ NodeConfig TypedDict Configuration for a single node in the flow. Show Fields ​ name str The name of the node, used in debug logging in dynamic flows. If no name is specified, an automatically-generated UUID is used. Copy Ask AI # Example name "name" : "greeting" ​ role_messages List[dict] Defines the role or persona of the LLM. Required for the initial node and optional for subsequent nodes. Copy Ask AI # Example role messages "role_messages" : [ { "role" : "system" , "content" : "You are a helpful assistant..." } ], ​ task_messages List[dict] required Defines the task for a given node. Required for all nodes. Copy Ask AI # Example task messages "task_messages" : [ { "role" : "system" , # May be `user` depending on the LLM "content" : "Ask the user for their name..." } ], ​ context_strategy ContextStrategyConfig Strategy for managing context during transitions to this node. Copy Ask AI # Example context strategy configuration "context_strategy" : ContextStrategyConfig( strategy = ContextStrategy. RESET_WITH_SUMMARY , summary_prompt = "Summarize the key points discussed so far." ) ​ functions List[Union[dict, FlowsFunctionSchema]] required LLM function / tool call configurations, defined in one of the supported formats . Copy Ask AI # Using provider-specific dictionary format "functions" : [ { "type" : "function" , "function" : { "name" : "get_current_movies" , "handler" : get_movies, "description" : "Fetch movies currently playing" , "parameters" : { ... } }, } ] # Using FlowsFunctionSchema "functions" : [ FlowsFunctionSchema( name = "get_current_movies" , description = "Fetch movies currently playing" , properties = { ... }, required = [ ... ], handler = get_movies ) ] # Using direct functions (auto-configuration) "functions" : [get_movies] ​ pre_actions List[dict] Actions that execute before the LLM inference. For example, you can send a message to the TTS to speak a phrase (e.g. “Hold on a moment…”), which may be effective if an LLM function call takes time to execute. Copy Ask AI # Example pre_actions "pre_actions" : [ { "type" : "tts_say" , "text" : "Hold on a moment..." } ], ​ post_actions List[dict] Actions that execute after the LLM inference. For example, you can end the conversation. Copy Ask AI # Example post_actions "post_actions" : [ { "type" : "end_conversation" } ] ​ respond_immediately bool If set to False , the LLM will not respond immediately when the node is set, but will instead wait for the user to speak first before responding. Defaults to True . Copy Ask AI # Example usage "respond_immediately" : False ​ Function Handler Types ​ LegacyFunctionHandler Callable[[FlowArgs], Awaitable[FlowResult | ConsolidatedFunctionResult]] Legacy function handler that only receives arguments. Returns either: A FlowResult (⚠️ deprecated) A “consolidated” result tuple (result, next node) where: result is an optional FlowResult next node is an optional NodeConfig (for dynamic flows) or string (for static flows) ​ FlowFunctionHandler Callable[[FlowArgs, FlowManager], Awaitable[FlowResult | ConsolidatedFunctionResult]] Modern function handler that receives both arguments and FlowManager . Returns either: A FlowResult (⚠️ deprecated) A “consolidated” result tuple (result, next node) where: result is an optional FlowResult next node is an optional NodeConfig (for dynamic flows) or string (for static flows) ​ DirectFunction DirectFunction Function that is meant to be passed directly into a NodeConfig rather than into the handler field of a function configuration. It must be an async function with flow_manager: FlowManager as its first parameter. It must return a ConsolidatedFunctionResult , which is a tuple (result, next node) where: result is an optional FlowResult next node is an optional NodeConfig (for dynamic flows) or string (for static flows) ​ ContextStrategy ​ ContextStrategy Enum Strategy for managing conversation context during node transitions. Show Values ​ APPEND str Default strategy. Adds new messages to existing context. ​ RESET str Clears context and starts fresh with new messages. ​ RESET_WITH_SUMMARY str Resets context but includes an AI-generated summary. ​ ContextStrategyConfig ​ ContextStrategyConfig dataclass Configuration for context management strategy. Show Fields ​ strategy ContextStrategy required The strategy to use for context management ​ summary_prompt Optional[str] Required when using RESET_WITH_SUMMARY. Prompt text for generating the conversation summary. Copy Ask AI # Example usage config = ContextStrategyConfig( strategy = ContextStrategy. RESET_WITH_SUMMARY , summary_prompt = "Summarize the key points discussed so far." ) ​ FlowsFunctionSchema ​ FlowsFunctionSchema class A standardized schema for defining functions in Pipecat Flows with flow-specific properties. Show Constructor Parameters ​ name str required Name of the function ​ description str required Description of the function’s purpose ​ properties Dict[str, Any] required Dictionary defining properties types and descriptions ​ required List[str] required List of required parameter names ​ handler Optional[FunctionHandler] Function handler to process the function call ​ transition_to Optional[str] deprecated Target node to transition to after function execution Deprecated: instead of transition_to , use a “consolidated” handler that returns a tuple (result, next node). ​ transition_callback Optional[Callable] deprecated Callback function for dynamic transitions Deprecated: instead of transition_callback , use a “consolidated” handler that returns a tuple (result, next node). You cannot specify both transition_to and transition_callback in the same function schema. Example usage: Copy Ask AI from pipecat_flows import FlowsFunctionSchema # Define a function schema collect_name_schema = FlowsFunctionSchema( name = "collect_name" , description = "Record the user's name" , properties = { "name" : { "type" : "string" , "description" : "The user's name" } }, required = [ "name" ], handler = collect_name_handler ) # Use in node configuration node_config = { "name" : "greeting" , "task_messages" : [ { "role" : "system" , "content" : "Ask the user for their name." } ], "functions" : [collect_name_schema] } # Pass to flow manager await flow_manager.set_node_from_config(node_config) ​ FlowManager ​ FlowManager class Main class for managing conversation flows, supporting both static (configuration-driven) and dynamic (runtime-determined) flows. Show Constructor Parameters ​ task PipelineTask required Pipeline task for frame queueing ​ llm LLMService required LLM service instance (OpenAI, Anthropic, or Google). Must be initialized with the corresponding pipecat-ai provider dependency installed. ​ context_aggregator Any required Context aggregator used for pushing messages to the LLM service ​ tts Optional[Any] deprecated Optional TTS service for voice actions. Deprecated: No need to explicitly pass tts to FlowManager in order to use tts_say actions. ​ flow_config Optional[FlowConfig] Optional static flow configuration ​ context_strategy Optional[ContextStrategyConfig] Optional configuration for how context should be managed during transitions. Defaults to APPEND strategy if not specified. ​ Methods ​ initialize method Initialize the flow with starting messages. Show Parameters ​ initial_node NodeConfig The initial conversation node (needed for dynamic flows only). If not specified, you’ll need to call set_node_from_config() to kick off the conversation. Show Raises ​ FlowInitializationError If initialization fails ​ set_node method deprecated Set up a new conversation node programmatically (dynamic flows only). In dynamic flows, the application advances the conversation using set_node to set up each next node. In static flows, set_node is triggered under the hood when a node contains a transition_to field. Deprecated: use the following patterns instead of set_node : Prefer “consolidated” function handlers that return a tuple (result, next node), which implicitly sets up the next node Prefer passing your initial node to FlowManager.initialize() If you really need to set a node explicitly, use set_node_from_config() (note: its name will be read from its NodeConfig ) Show Parameters ​ node_id str required Identifier for the new node ​ node_config NodeConfig required Node configuration including messages, functions, and actions Show Raises ​ FlowError If node setup fails ​ set_node_from_config method Set up a new conversation node programmatically (dynamic flows only). Note that this method should only be used in rare circumstances. Most often, you should: Prefer “consolidated” function handlers that return a tuple (result, next node), which implicitly sets up the next node Prefer passing your initial node to FlowManager.initialize() Show Parameters ​ node_config NodeConfig required Node configuration including messages, functions, and actions Show Raises ​ FlowError If node setup fails ​ register_action method Register a handler for a custom action type. Show Parameters ​ action_type str required String identifier for the action ​ handler Callable required Async or sync function that handles the action ​ get_current_context method Get the current conversation context. Returns a list of messages in the current context, including system messages, user messages, and assistant responses. Show Returns ​ messages List[dict] List of messages in the current context Show Raises ​ FlowError If context aggregator is not available Example usage: Copy Ask AI # Access current conversation context context = flow_manager.get_current_context() # Use in handlers async def process_response ( args : FlowArgs) -> tuple[FlowResult, str ]: context = flow_manager.get_current_context() # Process conversation history return { "status" : "success" }, "next" ​ State Management The FlowManager provides a state dictionary for storing conversation data: Access state Access in transitions Copy Ask AI flow_manager.state: Dict[ str , Any] # Store data flow_manager.state[ "user_age" ] = 25 ​ Usage Examples Static Flow Dynamic Flow Copy Ask AI flow_config: FlowConfig = { "initial_node" : "greeting" , "nodes" : { "greeting" : { "role_messages" : [ { "role" : "system" , "content" : "You are a helpful assistant. Your responses will be converted to audio." } ], "task_messages" : [ { "role" : "system" , "content" : "Start by greeting the user and asking for their name." } ], "functions" : [{ "type" : "function" , "function" : { "name" : "collect_name" , "handler" : collect_name_handler, "description" : "Record user's name" , "parameters" : { ... } } }] } } } # Create and initialize the FlowManager flow_manager = FlowManager( task = task, llm = llm, context_aggregator = context_aggregator, flow_config = flow_config ) # Initialize the flow_manager to start the conversation await flow_manager.initialize() ​ Node Functions concept Functions that execute operations within a single conversational state, without switching nodes. Copy Ask AI Copy Ask AI from pipecat_flows import FlowArgs, FlowResult async def process_data ( args : FlowArgs) -> tuple[FlowResult, None ]: """Handle data processing within a node.""" data = args[ "data" ] result = await process(data) return { "status" : "success" , "processed_data" : result }, None # Function configuration { "type" : "function" , "function" : { "name" : "process_data" , "handler" : process_data, "description" : "Process user data" , "parameters" : { "type" : "object" , "properties" : { "data" : { "type" : "string" } } } } } ​ Edge Functions concept Functions that specify a transition between nodes (optionally processing data first). Copy Ask AI Copy Ask AI from pipecat_flows import FlowArgs, FlowResult async def next_step ( args : FlowArgs) -> tuple[ None , str ]: """Specify the next node to transition to.""" return None , "target_node" # Return NodeConfig instead of str for dynamic flows # Function configuration { "type" : "function" , "function" : { "name" : "next_step" , "handler" : next_step, "description" : "Transition to next node" , "parameters" : { "type" : "object" , "properties" : {}} } } ​ Function Properties ​ handler Optional[Callable] Async function that processes data within a node and/or specifies the next node ( more details here ). Can be specified as: Direct function reference Either a Callable function or a string with __function__: prefix (e.g., "__function__:process_data" ) to reference a function in the main script Direct Reference Function Token Copy Ask AI { "type" : "function" , "function" : { "name" : "process_data" , "handler" : process_data, # Callable function "parameters" : { ... } } } ​ transition_callback Optional[Callable] deprecated Handler for dynamic flow transitions. Deprecated: instead of transition_callback , use a “consolidated” handler that returns a tuple (result, next node). Must be an async function with one of these signatures: Copy Ask AI # New style (recommended) async def handle_transition ( args : Dict[ str , Any], result : FlowResult, flow_manager : FlowManager ) -> None : """Handle transition to next node.""" if result.available: # Type-safe access to result await flow_manager.set_node_from_config(create_confirmation_node()) else : await flow_manager.set_node_from_config( create_no_availability_node(result.alternative_times) ) # Legacy style (supported for backwards compatibility) async def handle_transition ( args : Dict[ str , Any], flow_manager : FlowManager ) -> None : """Handle transition to next node.""" await flow_manager.set_node_from_config(create_next_node()) The callback receives: args : Arguments from the function call result : Typed result from the function handler (new style only) flow_manager : Reference to the FlowManager instance Example usage: Copy Ask AI async def handle_availability_check ( args : Dict, result : TimeResult, # Typed result flow_manager : FlowManager ): """Handle availability check and transition based on result.""" if result.available: await flow_manager.set_node_from_config(create_confirmation_node()) else : await flow_manager.set_node_from_config( create_no_availability_node(result.alternative_times) ) # Use in function configuration { "type" : "function" , "function" : { "name" : "check_availability" , "handler" : check_availability, "parameters" : { ... }, "transition_callback" : handle_availability_check } } Note: A function cannot have both transition_to and transition_callback . ​ Handler Signatures Function handlers passed as a handler in a function configuration can be defined with three different signatures: Modern (Args + FlowManager) Legacy (Args Only) No Arguments Copy Ask AI async def handler_with_flow_manager ( args : FlowArgs, flow_manager : FlowManager) -> tuple[FlowResult, NodeConfig]: """Modern handler that receives both arguments and FlowManager access.""" # Access state previous_data = flow_manager.state.get( "stored_data" ) # Access pipeline resources await flow_manager.task.queue_frame(TTSSpeakFrame( "Processing your request..." )) # Store data in state for later flow_manager.state[ "new_data" ] = args[ "input" ] return { "status" : "success" , "result" : "Processed with flow access" }, create_next_node() The framework automatically detects which signature your handler is using and calls it appropriately. If you’re passing your function directly into your NodeConfig rather than as a handler in a function configuration, you’d use a somewhat different signature: Direct Copy Ask AI async def do_something ( flow_manager : FlowManager, foo : int , bar : str = "" ) -> tuple[FlowResult, NodeConfig]: """ Do something interesting. Args: foo (int): The foo to do something interesting with. bar (string): The bar to do something interesting with. Defaults to empty string. """ result = await fetch_data(foo, bar) next_node = create_end_node() return result, next_node ​ Return Types Success Response Error Response Copy Ask AI { "status" : "success" , "data" : "some data" # Optional additional data } ​ Provider-Specific Formats You don’t need to handle these format differences manually - use the standard format and the FlowManager will adapt it for your chosen provider. OpenAI Anthropic Google (Gemini) Copy Ask AI { "type" : "function" , "function" : { "name" : "function_name" , "handler" : handler, "description" : "Description" , "parameters" : { ... } } } ​ Actions pre_actions and post_actions are used to manage conversation flow. They are included in the NodeConfig and executed before and after the LLM completion, respectively. Three kinds of actions are available: Pre-canned actions: These actions perform common tasks and require little configuration. Function actions: These actions run developer-defined functions at the appropriate time. Custom actions: These are fully developer-defined actions, providing flexibility at the expense of complexity. ​ Pre-canned Actions Common actions shipped with Flows for managing conversation flow. To use them, just add them to your NodeConfig . ​ tts_say action Speaks text immediately using the TTS service. Copy Ask AI Copy Ask AI "pre_actions" : [ { "type" : "tts_say" , "text" : "Processing your request..." # Required } ] ​ end_conversation action Ends the conversation and closes the connection. Copy Ask AI Copy Ask AI "post_actions" : [ { "type" : "end_conversation" , "text" : "Goodbye!" # Optional farewell message } ] ​ Function Actions Actions that run developer-defined functions at the appropriate time. For example, if used in post_actions , they’ll run after the bot has finished talking and after any previous post_actions have finished. ​ function action Calls the developer-defined function at the appropriate time. Copy Ask AI Copy Ask AI "post_actions" : [ { "type" : "function" , "handler" : bot_turn_ended # Required } ] ​ Custom Actions Fully developer-defined actions, providing flexibility at the expense of complexity. Here’s the complexity: because these actions aren’t queued in the Pipecat pipeline, they may execute seemingly early if used in post_actions ; they’ll run immediately after the LLM completion is triggered but won’t wait around for the bot to finish talking. Why would you want this behavior? You might be writing an action that: Itself just queues another Frame into the Pipecat pipeline (meaning there would no benefit to waiting around for sequencing purposes) Does work that can be done a bit sooner, like logging that the LLM was updated Custom actions are composed of at least: ​ type str required String identifier for the action ​ handler Callable required Async or sync function that handles the action Example: Copy Ask AI Copy Ask AI # Define custom action handler async def custom_notification ( action : dict , flow_manager : FlowManager): """Custom action handler.""" message = action.get( "message" , "" ) await notify_user(message) # Use in node configuration "pre_actions" : [ { "type" : "notify" , "handler" : send_notification, "message" : "Attention!" , } ] ​ Exceptions ​ FlowError exception Base exception for all flow-related errors. Copy Ask AI Copy Ask AI from pipecat_flows import FlowError try : await flow_manager.set_node_from_config(config) except FlowError as e: print ( f "Flow error: { e } " ) ​ FlowInitializationError exception Raised when flow initialization fails. Copy Ask AI Copy Ask AI from pipecat_flows import FlowInitializationError try : await flow_manager.initialize() except FlowInitializationError as e: print ( f "Initialization failed: { e } " ) ​ FlowTransitionError exception Raised when a state transition fails. Copy Ask AI Copy Ask AI from pipecat_flows import FlowTransitionError try : await flow_manager.set_node_from_config(node_config) except FlowTransitionError as e: print ( f "Transition failed: { e } " ) ​ InvalidFunctionError exception Raised when an invalid or unavailable function is specified. Copy Ask AI Copy Ask AI from pipecat_flows import InvalidFunctionError try : await flow_manager.set_node_from_config({ "functions" : [{ "type" : "function" , "function" : { "name" : "invalid_function" } }] }) except InvalidFunctionError as e: print ( f "Invalid function: { e } " ) RTVI Observer PipelineParams On this page Installation Core Types FlowArgs FlowResult FlowConfig NodeConfig Function Handler Types ContextStrategy ContextStrategyConfig FlowsFunctionSchema FlowManager Methods State Management Usage Examples Function Properties Handler Signatures Return Types Provider-Specific Formats Actions Pre-canned Actions Function Actions Custom Actions Exceptions Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/flows_pipecat-flows_2ba5b4ee.txt b/flows_pipecat-flows_2ba5b4ee.txt
new file mode 100644
index 0000000000000000000000000000000000000000..1b90a6764f7de5544d101e46cd3dee8ad0cdbda0
--- /dev/null
+++ b/flows_pipecat-flows_2ba5b4ee.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/frameworks/flows/pipecat-flows#param-respond-immediately
+Title: Pipecat Flows - Pipecat
+==================================================
+
+Pipecat Flows - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Frameworks Pipecat Flows Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline New to building conversational flows? Check out our Pipecat Flows guide first. ​ Installation Existing Pipecat installation Fresh Pipecat installation Copy Ask AI pip install pipecat-ai-flows ​ Core Types ​ FlowArgs ​ FlowArgs Dict[str, Any] Type alias for function handler arguments. ​ FlowResult ​ FlowResult TypedDict Base type for function handler results. Additional fields can be included as needed. Show Fields ​ status str Optional status field ​ error str Optional error message ​ FlowConfig ​ FlowConfig TypedDict Configuration for the entire conversation flow. Show Fields ​ initial_node str required Starting node identifier ​ nodes Dict[str, NodeConfig] required Map of node names to configurations ​ NodeConfig ​ NodeConfig TypedDict Configuration for a single node in the flow. Show Fields ​ name str The name of the node, used in debug logging in dynamic flows. If no name is specified, an automatically-generated UUID is used. Copy Ask AI # Example name "name" : "greeting" ​ role_messages List[dict] Defines the role or persona of the LLM. Required for the initial node and optional for subsequent nodes. Copy Ask AI # Example role messages "role_messages" : [ { "role" : "system" , "content" : "You are a helpful assistant..." } ], ​ task_messages List[dict] required Defines the task for a given node. Required for all nodes. Copy Ask AI # Example task messages "task_messages" : [ { "role" : "system" , # May be `user` depending on the LLM "content" : "Ask the user for their name..." } ], ​ context_strategy ContextStrategyConfig Strategy for managing context during transitions to this node. Copy Ask AI # Example context strategy configuration "context_strategy" : ContextStrategyConfig( strategy = ContextStrategy. RESET_WITH_SUMMARY , summary_prompt = "Summarize the key points discussed so far." ) ​ functions List[Union[dict, FlowsFunctionSchema]] required LLM function / tool call configurations, defined in one of the supported formats . Copy Ask AI # Using provider-specific dictionary format "functions" : [ { "type" : "function" , "function" : { "name" : "get_current_movies" , "handler" : get_movies, "description" : "Fetch movies currently playing" , "parameters" : { ... } }, } ] # Using FlowsFunctionSchema "functions" : [ FlowsFunctionSchema( name = "get_current_movies" , description = "Fetch movies currently playing" , properties = { ... }, required = [ ... ], handler = get_movies ) ] # Using direct functions (auto-configuration) "functions" : [get_movies] ​ pre_actions List[dict] Actions that execute before the LLM inference. For example, you can send a message to the TTS to speak a phrase (e.g. “Hold on a moment…”), which may be effective if an LLM function call takes time to execute. Copy Ask AI # Example pre_actions "pre_actions" : [ { "type" : "tts_say" , "text" : "Hold on a moment..." } ], ​ post_actions List[dict] Actions that execute after the LLM inference. For example, you can end the conversation. Copy Ask AI # Example post_actions "post_actions" : [ { "type" : "end_conversation" } ] ​ respond_immediately bool If set to False , the LLM will not respond immediately when the node is set, but will instead wait for the user to speak first before responding. Defaults to True . Copy Ask AI # Example usage "respond_immediately" : False ​ Function Handler Types ​ LegacyFunctionHandler Callable[[FlowArgs], Awaitable[FlowResult | ConsolidatedFunctionResult]] Legacy function handler that only receives arguments. Returns either: A FlowResult (⚠️ deprecated) A “consolidated” result tuple (result, next node) where: result is an optional FlowResult next node is an optional NodeConfig (for dynamic flows) or string (for static flows) ​ FlowFunctionHandler Callable[[FlowArgs, FlowManager], Awaitable[FlowResult | ConsolidatedFunctionResult]] Modern function handler that receives both arguments and FlowManager . Returns either: A FlowResult (⚠️ deprecated) A “consolidated” result tuple (result, next node) where: result is an optional FlowResult next node is an optional NodeConfig (for dynamic flows) or string (for static flows) ​ DirectFunction DirectFunction Function that is meant to be passed directly into a NodeConfig rather than into the handler field of a function configuration. It must be an async function with flow_manager: FlowManager as its first parameter. It must return a ConsolidatedFunctionResult , which is a tuple (result, next node) where: result is an optional FlowResult next node is an optional NodeConfig (for dynamic flows) or string (for static flows) ​ ContextStrategy ​ ContextStrategy Enum Strategy for managing conversation context during node transitions. Show Values ​ APPEND str Default strategy. Adds new messages to existing context. ​ RESET str Clears context and starts fresh with new messages. ​ RESET_WITH_SUMMARY str Resets context but includes an AI-generated summary. ​ ContextStrategyConfig ​ ContextStrategyConfig dataclass Configuration for context management strategy. Show Fields ​ strategy ContextStrategy required The strategy to use for context management ​ summary_prompt Optional[str] Required when using RESET_WITH_SUMMARY. Prompt text for generating the conversation summary. Copy Ask AI # Example usage config = ContextStrategyConfig( strategy = ContextStrategy. RESET_WITH_SUMMARY , summary_prompt = "Summarize the key points discussed so far." ) ​ FlowsFunctionSchema ​ FlowsFunctionSchema class A standardized schema for defining functions in Pipecat Flows with flow-specific properties. Show Constructor Parameters ​ name str required Name of the function ​ description str required Description of the function’s purpose ​ properties Dict[str, Any] required Dictionary defining properties types and descriptions ​ required List[str] required List of required parameter names ​ handler Optional[FunctionHandler] Function handler to process the function call ​ transition_to Optional[str] deprecated Target node to transition to after function execution Deprecated: instead of transition_to , use a “consolidated” handler that returns a tuple (result, next node). ​ transition_callback Optional[Callable] deprecated Callback function for dynamic transitions Deprecated: instead of transition_callback , use a “consolidated” handler that returns a tuple (result, next node). You cannot specify both transition_to and transition_callback in the same function schema. Example usage: Copy Ask AI from pipecat_flows import FlowsFunctionSchema # Define a function schema collect_name_schema = FlowsFunctionSchema( name = "collect_name" , description = "Record the user's name" , properties = { "name" : { "type" : "string" , "description" : "The user's name" } }, required = [ "name" ], handler = collect_name_handler ) # Use in node configuration node_config = { "name" : "greeting" , "task_messages" : [ { "role" : "system" , "content" : "Ask the user for their name." } ], "functions" : [collect_name_schema] } # Pass to flow manager await flow_manager.set_node_from_config(node_config) ​ FlowManager ​ FlowManager class Main class for managing conversation flows, supporting both static (configuration-driven) and dynamic (runtime-determined) flows. Show Constructor Parameters ​ task PipelineTask required Pipeline task for frame queueing ​ llm LLMService required LLM service instance (OpenAI, Anthropic, or Google). Must be initialized with the corresponding pipecat-ai provider dependency installed. ​ context_aggregator Any required Context aggregator used for pushing messages to the LLM service ​ tts Optional[Any] deprecated Optional TTS service for voice actions. Deprecated: No need to explicitly pass tts to FlowManager in order to use tts_say actions. ​ flow_config Optional[FlowConfig] Optional static flow configuration ​ context_strategy Optional[ContextStrategyConfig] Optional configuration for how context should be managed during transitions. Defaults to APPEND strategy if not specified. ​ Methods ​ initialize method Initialize the flow with starting messages. Show Parameters ​ initial_node NodeConfig The initial conversation node (needed for dynamic flows only). If not specified, you’ll need to call set_node_from_config() to kick off the conversation. Show Raises ​ FlowInitializationError If initialization fails ​ set_node method deprecated Set up a new conversation node programmatically (dynamic flows only). In dynamic flows, the application advances the conversation using set_node to set up each next node. In static flows, set_node is triggered under the hood when a node contains a transition_to field. Deprecated: use the following patterns instead of set_node : Prefer “consolidated” function handlers that return a tuple (result, next node), which implicitly sets up the next node Prefer passing your initial node to FlowManager.initialize() If you really need to set a node explicitly, use set_node_from_config() (note: its name will be read from its NodeConfig ) Show Parameters ​ node_id str required Identifier for the new node ​ node_config NodeConfig required Node configuration including messages, functions, and actions Show Raises ​ FlowError If node setup fails ​ set_node_from_config method Set up a new conversation node programmatically (dynamic flows only). Note that this method should only be used in rare circumstances. Most often, you should: Prefer “consolidated” function handlers that return a tuple (result, next node), which implicitly sets up the next node Prefer passing your initial node to FlowManager.initialize() Show Parameters ​ node_config NodeConfig required Node configuration including messages, functions, and actions Show Raises ​ FlowError If node setup fails ​ register_action method Register a handler for a custom action type. Show Parameters ​ action_type str required String identifier for the action ​ handler Callable required Async or sync function that handles the action ​ get_current_context method Get the current conversation context. Returns a list of messages in the current context, including system messages, user messages, and assistant responses. Show Returns ​ messages List[dict] List of messages in the current context Show Raises ​ FlowError If context aggregator is not available Example usage: Copy Ask AI # Access current conversation context context = flow_manager.get_current_context() # Use in handlers async def process_response ( args : FlowArgs) -> tuple[FlowResult, str ]: context = flow_manager.get_current_context() # Process conversation history return { "status" : "success" }, "next" ​ State Management The FlowManager provides a state dictionary for storing conversation data: Access state Access in transitions Copy Ask AI flow_manager.state: Dict[ str , Any] # Store data flow_manager.state[ "user_age" ] = 25 ​ Usage Examples Static Flow Dynamic Flow Copy Ask AI flow_config: FlowConfig = { "initial_node" : "greeting" , "nodes" : { "greeting" : { "role_messages" : [ { "role" : "system" , "content" : "You are a helpful assistant. Your responses will be converted to audio." } ], "task_messages" : [ { "role" : "system" , "content" : "Start by greeting the user and asking for their name." } ], "functions" : [{ "type" : "function" , "function" : { "name" : "collect_name" , "handler" : collect_name_handler, "description" : "Record user's name" , "parameters" : { ... } } }] } } } # Create and initialize the FlowManager flow_manager = FlowManager( task = task, llm = llm, context_aggregator = context_aggregator, flow_config = flow_config ) # Initialize the flow_manager to start the conversation await flow_manager.initialize() ​ Node Functions concept Functions that execute operations within a single conversational state, without switching nodes. Copy Ask AI Copy Ask AI from pipecat_flows import FlowArgs, FlowResult async def process_data ( args : FlowArgs) -> tuple[FlowResult, None ]: """Handle data processing within a node.""" data = args[ "data" ] result = await process(data) return { "status" : "success" , "processed_data" : result }, None # Function configuration { "type" : "function" , "function" : { "name" : "process_data" , "handler" : process_data, "description" : "Process user data" , "parameters" : { "type" : "object" , "properties" : { "data" : { "type" : "string" } } } } } ​ Edge Functions concept Functions that specify a transition between nodes (optionally processing data first). Copy Ask AI Copy Ask AI from pipecat_flows import FlowArgs, FlowResult async def next_step ( args : FlowArgs) -> tuple[ None , str ]: """Specify the next node to transition to.""" return None , "target_node" # Return NodeConfig instead of str for dynamic flows # Function configuration { "type" : "function" , "function" : { "name" : "next_step" , "handler" : next_step, "description" : "Transition to next node" , "parameters" : { "type" : "object" , "properties" : {}} } } ​ Function Properties ​ handler Optional[Callable] Async function that processes data within a node and/or specifies the next node ( more details here ). Can be specified as: Direct function reference Either a Callable function or a string with __function__: prefix (e.g., "__function__:process_data" ) to reference a function in the main script Direct Reference Function Token Copy Ask AI { "type" : "function" , "function" : { "name" : "process_data" , "handler" : process_data, # Callable function "parameters" : { ... } } } ​ transition_callback Optional[Callable] deprecated Handler for dynamic flow transitions. Deprecated: instead of transition_callback , use a “consolidated” handler that returns a tuple (result, next node). Must be an async function with one of these signatures: Copy Ask AI # New style (recommended) async def handle_transition ( args : Dict[ str , Any], result : FlowResult, flow_manager : FlowManager ) -> None : """Handle transition to next node.""" if result.available: # Type-safe access to result await flow_manager.set_node_from_config(create_confirmation_node()) else : await flow_manager.set_node_from_config( create_no_availability_node(result.alternative_times) ) # Legacy style (supported for backwards compatibility) async def handle_transition ( args : Dict[ str , Any], flow_manager : FlowManager ) -> None : """Handle transition to next node.""" await flow_manager.set_node_from_config(create_next_node()) The callback receives: args : Arguments from the function call result : Typed result from the function handler (new style only) flow_manager : Reference to the FlowManager instance Example usage: Copy Ask AI async def handle_availability_check ( args : Dict, result : TimeResult, # Typed result flow_manager : FlowManager ): """Handle availability check and transition based on result.""" if result.available: await flow_manager.set_node_from_config(create_confirmation_node()) else : await flow_manager.set_node_from_config( create_no_availability_node(result.alternative_times) ) # Use in function configuration { "type" : "function" , "function" : { "name" : "check_availability" , "handler" : check_availability, "parameters" : { ... }, "transition_callback" : handle_availability_check } } Note: A function cannot have both transition_to and transition_callback . ​ Handler Signatures Function handlers passed as a handler in a function configuration can be defined with three different signatures: Modern (Args + FlowManager) Legacy (Args Only) No Arguments Copy Ask AI async def handler_with_flow_manager ( args : FlowArgs, flow_manager : FlowManager) -> tuple[FlowResult, NodeConfig]: """Modern handler that receives both arguments and FlowManager access.""" # Access state previous_data = flow_manager.state.get( "stored_data" ) # Access pipeline resources await flow_manager.task.queue_frame(TTSSpeakFrame( "Processing your request..." )) # Store data in state for later flow_manager.state[ "new_data" ] = args[ "input" ] return { "status" : "success" , "result" : "Processed with flow access" }, create_next_node() The framework automatically detects which signature your handler is using and calls it appropriately. If you’re passing your function directly into your NodeConfig rather than as a handler in a function configuration, you’d use a somewhat different signature: Direct Copy Ask AI async def do_something ( flow_manager : FlowManager, foo : int , bar : str = "" ) -> tuple[FlowResult, NodeConfig]: """ Do something interesting. Args: foo (int): The foo to do something interesting with. bar (string): The bar to do something interesting with. Defaults to empty string. """ result = await fetch_data(foo, bar) next_node = create_end_node() return result, next_node ​ Return Types Success Response Error Response Copy Ask AI { "status" : "success" , "data" : "some data" # Optional additional data } ​ Provider-Specific Formats You don’t need to handle these format differences manually - use the standard format and the FlowManager will adapt it for your chosen provider. OpenAI Anthropic Google (Gemini) Copy Ask AI { "type" : "function" , "function" : { "name" : "function_name" , "handler" : handler, "description" : "Description" , "parameters" : { ... } } } ​ Actions pre_actions and post_actions are used to manage conversation flow. They are included in the NodeConfig and executed before and after the LLM completion, respectively. Three kinds of actions are available: Pre-canned actions: These actions perform common tasks and require little configuration. Function actions: These actions run developer-defined functions at the appropriate time. Custom actions: These are fully developer-defined actions, providing flexibility at the expense of complexity. ​ Pre-canned Actions Common actions shipped with Flows for managing conversation flow. To use them, just add them to your NodeConfig . ​ tts_say action Speaks text immediately using the TTS service. Copy Ask AI Copy Ask AI "pre_actions" : [ { "type" : "tts_say" , "text" : "Processing your request..." # Required } ] ​ end_conversation action Ends the conversation and closes the connection. Copy Ask AI Copy Ask AI "post_actions" : [ { "type" : "end_conversation" , "text" : "Goodbye!" # Optional farewell message } ] ​ Function Actions Actions that run developer-defined functions at the appropriate time. For example, if used in post_actions , they’ll run after the bot has finished talking and after any previous post_actions have finished. ​ function action Calls the developer-defined function at the appropriate time. Copy Ask AI Copy Ask AI "post_actions" : [ { "type" : "function" , "handler" : bot_turn_ended # Required } ] ​ Custom Actions Fully developer-defined actions, providing flexibility at the expense of complexity. Here’s the complexity: because these actions aren’t queued in the Pipecat pipeline, they may execute seemingly early if used in post_actions ; they’ll run immediately after the LLM completion is triggered but won’t wait around for the bot to finish talking. Why would you want this behavior? You might be writing an action that: Itself just queues another Frame into the Pipecat pipeline (meaning there would no benefit to waiting around for sequencing purposes) Does work that can be done a bit sooner, like logging that the LLM was updated Custom actions are composed of at least: ​ type str required String identifier for the action ​ handler Callable required Async or sync function that handles the action Example: Copy Ask AI Copy Ask AI # Define custom action handler async def custom_notification ( action : dict , flow_manager : FlowManager): """Custom action handler.""" message = action.get( "message" , "" ) await notify_user(message) # Use in node configuration "pre_actions" : [ { "type" : "notify" , "handler" : send_notification, "message" : "Attention!" , } ] ​ Exceptions ​ FlowError exception Base exception for all flow-related errors. Copy Ask AI Copy Ask AI from pipecat_flows import FlowError try : await flow_manager.set_node_from_config(config) except FlowError as e: print ( f "Flow error: { e } " ) ​ FlowInitializationError exception Raised when flow initialization fails. Copy Ask AI Copy Ask AI from pipecat_flows import FlowInitializationError try : await flow_manager.initialize() except FlowInitializationError as e: print ( f "Initialization failed: { e } " ) ​ FlowTransitionError exception Raised when a state transition fails. Copy Ask AI Copy Ask AI from pipecat_flows import FlowTransitionError try : await flow_manager.set_node_from_config(node_config) except FlowTransitionError as e: print ( f "Transition failed: { e } " ) ​ InvalidFunctionError exception Raised when an invalid or unavailable function is specified. Copy Ask AI Copy Ask AI from pipecat_flows import InvalidFunctionError try : await flow_manager.set_node_from_config({ "functions" : [{ "type" : "function" , "function" : { "name" : "invalid_function" } }] }) except InvalidFunctionError as e: print ( f "Invalid function: { e } " ) RTVI Observer PipelineParams On this page Installation Core Types FlowArgs FlowResult FlowConfig NodeConfig Function Handler Types ContextStrategy ContextStrategyConfig FlowsFunctionSchema FlowManager Methods State Management Usage Examples Function Properties Handler Signatures Return Types Provider-Specific Formats Actions Pre-canned Actions Function Actions Custom Actions Exceptions Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/flows_pipecat-flows_36f50953.txt b/flows_pipecat-flows_36f50953.txt
new file mode 100644
index 0000000000000000000000000000000000000000..eec6404535f2328494018a110c5e63637d4df560
--- /dev/null
+++ b/flows_pipecat-flows_36f50953.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/frameworks/flows/pipecat-flows#param-required
+Title: Pipecat Flows - Pipecat
+==================================================
+
+Pipecat Flows - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Frameworks Pipecat Flows Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline New to building conversational flows? Check out our Pipecat Flows guide first. ​ Installation Existing Pipecat installation Fresh Pipecat installation Copy Ask AI pip install pipecat-ai-flows ​ Core Types ​ FlowArgs ​ FlowArgs Dict[str, Any] Type alias for function handler arguments. ​ FlowResult ​ FlowResult TypedDict Base type for function handler results. Additional fields can be included as needed. Show Fields ​ status str Optional status field ​ error str Optional error message ​ FlowConfig ​ FlowConfig TypedDict Configuration for the entire conversation flow. Show Fields ​ initial_node str required Starting node identifier ​ nodes Dict[str, NodeConfig] required Map of node names to configurations ​ NodeConfig ​ NodeConfig TypedDict Configuration for a single node in the flow. Show Fields ​ name str The name of the node, used in debug logging in dynamic flows. If no name is specified, an automatically-generated UUID is used. Copy Ask AI # Example name "name" : "greeting" ​ role_messages List[dict] Defines the role or persona of the LLM. Required for the initial node and optional for subsequent nodes. Copy Ask AI # Example role messages "role_messages" : [ { "role" : "system" , "content" : "You are a helpful assistant..." } ], ​ task_messages List[dict] required Defines the task for a given node. Required for all nodes. Copy Ask AI # Example task messages "task_messages" : [ { "role" : "system" , # May be `user` depending on the LLM "content" : "Ask the user for their name..." } ], ​ context_strategy ContextStrategyConfig Strategy for managing context during transitions to this node. Copy Ask AI # Example context strategy configuration "context_strategy" : ContextStrategyConfig( strategy = ContextStrategy. RESET_WITH_SUMMARY , summary_prompt = "Summarize the key points discussed so far." ) ​ functions List[Union[dict, FlowsFunctionSchema]] required LLM function / tool call configurations, defined in one of the supported formats . Copy Ask AI # Using provider-specific dictionary format "functions" : [ { "type" : "function" , "function" : { "name" : "get_current_movies" , "handler" : get_movies, "description" : "Fetch movies currently playing" , "parameters" : { ... } }, } ] # Using FlowsFunctionSchema "functions" : [ FlowsFunctionSchema( name = "get_current_movies" , description = "Fetch movies currently playing" , properties = { ... }, required = [ ... ], handler = get_movies ) ] # Using direct functions (auto-configuration) "functions" : [get_movies] ​ pre_actions List[dict] Actions that execute before the LLM inference. For example, you can send a message to the TTS to speak a phrase (e.g. “Hold on a moment…”), which may be effective if an LLM function call takes time to execute. Copy Ask AI # Example pre_actions "pre_actions" : [ { "type" : "tts_say" , "text" : "Hold on a moment..." } ], ​ post_actions List[dict] Actions that execute after the LLM inference. For example, you can end the conversation. Copy Ask AI # Example post_actions "post_actions" : [ { "type" : "end_conversation" } ] ​ respond_immediately bool If set to False , the LLM will not respond immediately when the node is set, but will instead wait for the user to speak first before responding. Defaults to True . Copy Ask AI # Example usage "respond_immediately" : False ​ Function Handler Types ​ LegacyFunctionHandler Callable[[FlowArgs], Awaitable[FlowResult | ConsolidatedFunctionResult]] Legacy function handler that only receives arguments. Returns either: A FlowResult (⚠️ deprecated) A “consolidated” result tuple (result, next node) where: result is an optional FlowResult next node is an optional NodeConfig (for dynamic flows) or string (for static flows) ​ FlowFunctionHandler Callable[[FlowArgs, FlowManager], Awaitable[FlowResult | ConsolidatedFunctionResult]] Modern function handler that receives both arguments and FlowManager . Returns either: A FlowResult (⚠️ deprecated) A “consolidated” result tuple (result, next node) where: result is an optional FlowResult next node is an optional NodeConfig (for dynamic flows) or string (for static flows) ​ DirectFunction DirectFunction Function that is meant to be passed directly into a NodeConfig rather than into the handler field of a function configuration. It must be an async function with flow_manager: FlowManager as its first parameter. It must return a ConsolidatedFunctionResult , which is a tuple (result, next node) where: result is an optional FlowResult next node is an optional NodeConfig (for dynamic flows) or string (for static flows) ​ ContextStrategy ​ ContextStrategy Enum Strategy for managing conversation context during node transitions. Show Values ​ APPEND str Default strategy. Adds new messages to existing context. ​ RESET str Clears context and starts fresh with new messages. ​ RESET_WITH_SUMMARY str Resets context but includes an AI-generated summary. ​ ContextStrategyConfig ​ ContextStrategyConfig dataclass Configuration for context management strategy. Show Fields ​ strategy ContextStrategy required The strategy to use for context management ​ summary_prompt Optional[str] Required when using RESET_WITH_SUMMARY. Prompt text for generating the conversation summary. Copy Ask AI # Example usage config = ContextStrategyConfig( strategy = ContextStrategy. RESET_WITH_SUMMARY , summary_prompt = "Summarize the key points discussed so far." ) ​ FlowsFunctionSchema ​ FlowsFunctionSchema class A standardized schema for defining functions in Pipecat Flows with flow-specific properties. Show Constructor Parameters ​ name str required Name of the function ​ description str required Description of the function’s purpose ​ properties Dict[str, Any] required Dictionary defining properties types and descriptions ​ required List[str] required List of required parameter names ​ handler Optional[FunctionHandler] Function handler to process the function call ​ transition_to Optional[str] deprecated Target node to transition to after function execution Deprecated: instead of transition_to , use a “consolidated” handler that returns a tuple (result, next node). ​ transition_callback Optional[Callable] deprecated Callback function for dynamic transitions Deprecated: instead of transition_callback , use a “consolidated” handler that returns a tuple (result, next node). You cannot specify both transition_to and transition_callback in the same function schema. Example usage: Copy Ask AI from pipecat_flows import FlowsFunctionSchema # Define a function schema collect_name_schema = FlowsFunctionSchema( name = "collect_name" , description = "Record the user's name" , properties = { "name" : { "type" : "string" , "description" : "The user's name" } }, required = [ "name" ], handler = collect_name_handler ) # Use in node configuration node_config = { "name" : "greeting" , "task_messages" : [ { "role" : "system" , "content" : "Ask the user for their name." } ], "functions" : [collect_name_schema] } # Pass to flow manager await flow_manager.set_node_from_config(node_config) ​ FlowManager ​ FlowManager class Main class for managing conversation flows, supporting both static (configuration-driven) and dynamic (runtime-determined) flows. Show Constructor Parameters ​ task PipelineTask required Pipeline task for frame queueing ​ llm LLMService required LLM service instance (OpenAI, Anthropic, or Google). Must be initialized with the corresponding pipecat-ai provider dependency installed. ​ context_aggregator Any required Context aggregator used for pushing messages to the LLM service ​ tts Optional[Any] deprecated Optional TTS service for voice actions. Deprecated: No need to explicitly pass tts to FlowManager in order to use tts_say actions. ​ flow_config Optional[FlowConfig] Optional static flow configuration ​ context_strategy Optional[ContextStrategyConfig] Optional configuration for how context should be managed during transitions. Defaults to APPEND strategy if not specified. ​ Methods ​ initialize method Initialize the flow with starting messages. Show Parameters ​ initial_node NodeConfig The initial conversation node (needed for dynamic flows only). If not specified, you’ll need to call set_node_from_config() to kick off the conversation. Show Raises ​ FlowInitializationError If initialization fails ​ set_node method deprecated Set up a new conversation node programmatically (dynamic flows only). In dynamic flows, the application advances the conversation using set_node to set up each next node. In static flows, set_node is triggered under the hood when a node contains a transition_to field. Deprecated: use the following patterns instead of set_node : Prefer “consolidated” function handlers that return a tuple (result, next node), which implicitly sets up the next node Prefer passing your initial node to FlowManager.initialize() If you really need to set a node explicitly, use set_node_from_config() (note: its name will be read from its NodeConfig ) Show Parameters ​ node_id str required Identifier for the new node ​ node_config NodeConfig required Node configuration including messages, functions, and actions Show Raises ​ FlowError If node setup fails ​ set_node_from_config method Set up a new conversation node programmatically (dynamic flows only). Note that this method should only be used in rare circumstances. Most often, you should: Prefer “consolidated” function handlers that return a tuple (result, next node), which implicitly sets up the next node Prefer passing your initial node to FlowManager.initialize() Show Parameters ​ node_config NodeConfig required Node configuration including messages, functions, and actions Show Raises ​ FlowError If node setup fails ​ register_action method Register a handler for a custom action type. Show Parameters ​ action_type str required String identifier for the action ​ handler Callable required Async or sync function that handles the action ​ get_current_context method Get the current conversation context. Returns a list of messages in the current context, including system messages, user messages, and assistant responses. Show Returns ​ messages List[dict] List of messages in the current context Show Raises ​ FlowError If context aggregator is not available Example usage: Copy Ask AI # Access current conversation context context = flow_manager.get_current_context() # Use in handlers async def process_response ( args : FlowArgs) -> tuple[FlowResult, str ]: context = flow_manager.get_current_context() # Process conversation history return { "status" : "success" }, "next" ​ State Management The FlowManager provides a state dictionary for storing conversation data: Access state Access in transitions Copy Ask AI flow_manager.state: Dict[ str , Any] # Store data flow_manager.state[ "user_age" ] = 25 ​ Usage Examples Static Flow Dynamic Flow Copy Ask AI flow_config: FlowConfig = { "initial_node" : "greeting" , "nodes" : { "greeting" : { "role_messages" : [ { "role" : "system" , "content" : "You are a helpful assistant. Your responses will be converted to audio." } ], "task_messages" : [ { "role" : "system" , "content" : "Start by greeting the user and asking for their name." } ], "functions" : [{ "type" : "function" , "function" : { "name" : "collect_name" , "handler" : collect_name_handler, "description" : "Record user's name" , "parameters" : { ... } } }] } } } # Create and initialize the FlowManager flow_manager = FlowManager( task = task, llm = llm, context_aggregator = context_aggregator, flow_config = flow_config ) # Initialize the flow_manager to start the conversation await flow_manager.initialize() ​ Node Functions concept Functions that execute operations within a single conversational state, without switching nodes. Copy Ask AI Copy Ask AI from pipecat_flows import FlowArgs, FlowResult async def process_data ( args : FlowArgs) -> tuple[FlowResult, None ]: """Handle data processing within a node.""" data = args[ "data" ] result = await process(data) return { "status" : "success" , "processed_data" : result }, None # Function configuration { "type" : "function" , "function" : { "name" : "process_data" , "handler" : process_data, "description" : "Process user data" , "parameters" : { "type" : "object" , "properties" : { "data" : { "type" : "string" } } } } } ​ Edge Functions concept Functions that specify a transition between nodes (optionally processing data first). Copy Ask AI Copy Ask AI from pipecat_flows import FlowArgs, FlowResult async def next_step ( args : FlowArgs) -> tuple[ None , str ]: """Specify the next node to transition to.""" return None , "target_node" # Return NodeConfig instead of str for dynamic flows # Function configuration { "type" : "function" , "function" : { "name" : "next_step" , "handler" : next_step, "description" : "Transition to next node" , "parameters" : { "type" : "object" , "properties" : {}} } } ​ Function Properties ​ handler Optional[Callable] Async function that processes data within a node and/or specifies the next node ( more details here ). Can be specified as: Direct function reference Either a Callable function or a string with __function__: prefix (e.g., "__function__:process_data" ) to reference a function in the main script Direct Reference Function Token Copy Ask AI { "type" : "function" , "function" : { "name" : "process_data" , "handler" : process_data, # Callable function "parameters" : { ... } } } ​ transition_callback Optional[Callable] deprecated Handler for dynamic flow transitions. Deprecated: instead of transition_callback , use a “consolidated” handler that returns a tuple (result, next node). Must be an async function with one of these signatures: Copy Ask AI # New style (recommended) async def handle_transition ( args : Dict[ str , Any], result : FlowResult, flow_manager : FlowManager ) -> None : """Handle transition to next node.""" if result.available: # Type-safe access to result await flow_manager.set_node_from_config(create_confirmation_node()) else : await flow_manager.set_node_from_config( create_no_availability_node(result.alternative_times) ) # Legacy style (supported for backwards compatibility) async def handle_transition ( args : Dict[ str , Any], flow_manager : FlowManager ) -> None : """Handle transition to next node.""" await flow_manager.set_node_from_config(create_next_node()) The callback receives: args : Arguments from the function call result : Typed result from the function handler (new style only) flow_manager : Reference to the FlowManager instance Example usage: Copy Ask AI async def handle_availability_check ( args : Dict, result : TimeResult, # Typed result flow_manager : FlowManager ): """Handle availability check and transition based on result.""" if result.available: await flow_manager.set_node_from_config(create_confirmation_node()) else : await flow_manager.set_node_from_config( create_no_availability_node(result.alternative_times) ) # Use in function configuration { "type" : "function" , "function" : { "name" : "check_availability" , "handler" : check_availability, "parameters" : { ... }, "transition_callback" : handle_availability_check } } Note: A function cannot have both transition_to and transition_callback . ​ Handler Signatures Function handlers passed as a handler in a function configuration can be defined with three different signatures: Modern (Args + FlowManager) Legacy (Args Only) No Arguments Copy Ask AI async def handler_with_flow_manager ( args : FlowArgs, flow_manager : FlowManager) -> tuple[FlowResult, NodeConfig]: """Modern handler that receives both arguments and FlowManager access.""" # Access state previous_data = flow_manager.state.get( "stored_data" ) # Access pipeline resources await flow_manager.task.queue_frame(TTSSpeakFrame( "Processing your request..." )) # Store data in state for later flow_manager.state[ "new_data" ] = args[ "input" ] return { "status" : "success" , "result" : "Processed with flow access" }, create_next_node() The framework automatically detects which signature your handler is using and calls it appropriately. If you’re passing your function directly into your NodeConfig rather than as a handler in a function configuration, you’d use a somewhat different signature: Direct Copy Ask AI async def do_something ( flow_manager : FlowManager, foo : int , bar : str = "" ) -> tuple[FlowResult, NodeConfig]: """ Do something interesting. Args: foo (int): The foo to do something interesting with. bar (string): The bar to do something interesting with. Defaults to empty string. """ result = await fetch_data(foo, bar) next_node = create_end_node() return result, next_node ​ Return Types Success Response Error Response Copy Ask AI { "status" : "success" , "data" : "some data" # Optional additional data } ​ Provider-Specific Formats You don’t need to handle these format differences manually - use the standard format and the FlowManager will adapt it for your chosen provider. OpenAI Anthropic Google (Gemini) Copy Ask AI { "type" : "function" , "function" : { "name" : "function_name" , "handler" : handler, "description" : "Description" , "parameters" : { ... } } } ​ Actions pre_actions and post_actions are used to manage conversation flow. They are included in the NodeConfig and executed before and after the LLM completion, respectively. Three kinds of actions are available: Pre-canned actions: These actions perform common tasks and require little configuration. Function actions: These actions run developer-defined functions at the appropriate time. Custom actions: These are fully developer-defined actions, providing flexibility at the expense of complexity. ​ Pre-canned Actions Common actions shipped with Flows for managing conversation flow. To use them, just add them to your NodeConfig . ​ tts_say action Speaks text immediately using the TTS service. Copy Ask AI Copy Ask AI "pre_actions" : [ { "type" : "tts_say" , "text" : "Processing your request..." # Required } ] ​ end_conversation action Ends the conversation and closes the connection. Copy Ask AI Copy Ask AI "post_actions" : [ { "type" : "end_conversation" , "text" : "Goodbye!" # Optional farewell message } ] ​ Function Actions Actions that run developer-defined functions at the appropriate time. For example, if used in post_actions , they’ll run after the bot has finished talking and after any previous post_actions have finished. ​ function action Calls the developer-defined function at the appropriate time. Copy Ask AI Copy Ask AI "post_actions" : [ { "type" : "function" , "handler" : bot_turn_ended # Required } ] ​ Custom Actions Fully developer-defined actions, providing flexibility at the expense of complexity. Here’s the complexity: because these actions aren’t queued in the Pipecat pipeline, they may execute seemingly early if used in post_actions ; they’ll run immediately after the LLM completion is triggered but won’t wait around for the bot to finish talking. Why would you want this behavior? You might be writing an action that: Itself just queues another Frame into the Pipecat pipeline (meaning there would no benefit to waiting around for sequencing purposes) Does work that can be done a bit sooner, like logging that the LLM was updated Custom actions are composed of at least: ​ type str required String identifier for the action ​ handler Callable required Async or sync function that handles the action Example: Copy Ask AI Copy Ask AI # Define custom action handler async def custom_notification ( action : dict , flow_manager : FlowManager): """Custom action handler.""" message = action.get( "message" , "" ) await notify_user(message) # Use in node configuration "pre_actions" : [ { "type" : "notify" , "handler" : send_notification, "message" : "Attention!" , } ] ​ Exceptions ​ FlowError exception Base exception for all flow-related errors. Copy Ask AI Copy Ask AI from pipecat_flows import FlowError try : await flow_manager.set_node_from_config(config) except FlowError as e: print ( f "Flow error: { e } " ) ​ FlowInitializationError exception Raised when flow initialization fails. Copy Ask AI Copy Ask AI from pipecat_flows import FlowInitializationError try : await flow_manager.initialize() except FlowInitializationError as e: print ( f "Initialization failed: { e } " ) ​ FlowTransitionError exception Raised when a state transition fails. Copy Ask AI Copy Ask AI from pipecat_flows import FlowTransitionError try : await flow_manager.set_node_from_config(node_config) except FlowTransitionError as e: print ( f "Transition failed: { e } " ) ​ InvalidFunctionError exception Raised when an invalid or unavailable function is specified. Copy Ask AI Copy Ask AI from pipecat_flows import InvalidFunctionError try : await flow_manager.set_node_from_config({ "functions" : [{ "type" : "function" , "function" : { "name" : "invalid_function" } }] }) except InvalidFunctionError as e: print ( f "Invalid function: { e } " ) RTVI Observer PipelineParams On this page Installation Core Types FlowArgs FlowResult FlowConfig NodeConfig Function Handler Types ContextStrategy ContextStrategyConfig FlowsFunctionSchema FlowManager Methods State Management Usage Examples Function Properties Handler Signatures Return Types Provider-Specific Formats Actions Pre-canned Actions Function Actions Custom Actions Exceptions Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/flows_pipecat-flows_45a9952c.txt b/flows_pipecat-flows_45a9952c.txt
new file mode 100644
index 0000000000000000000000000000000000000000..68de0864751a044651e50f0e4ade512303b092f3
--- /dev/null
+++ b/flows_pipecat-flows_45a9952c.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/frameworks/flows/pipecat-flows#param-reset
+Title: Pipecat Flows - Pipecat
+==================================================
+
+Pipecat Flows - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Frameworks Pipecat Flows Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline New to building conversational flows? Check out our Pipecat Flows guide first. ​ Installation Existing Pipecat installation Fresh Pipecat installation Copy Ask AI pip install pipecat-ai-flows ​ Core Types ​ FlowArgs ​ FlowArgs Dict[str, Any] Type alias for function handler arguments. ​ FlowResult ​ FlowResult TypedDict Base type for function handler results. Additional fields can be included as needed. Show Fields ​ status str Optional status field ​ error str Optional error message ​ FlowConfig ​ FlowConfig TypedDict Configuration for the entire conversation flow. Show Fields ​ initial_node str required Starting node identifier ​ nodes Dict[str, NodeConfig] required Map of node names to configurations ​ NodeConfig ​ NodeConfig TypedDict Configuration for a single node in the flow. Show Fields ​ name str The name of the node, used in debug logging in dynamic flows. If no name is specified, an automatically-generated UUID is used. Copy Ask AI # Example name "name" : "greeting" ​ role_messages List[dict] Defines the role or persona of the LLM. Required for the initial node and optional for subsequent nodes. Copy Ask AI # Example role messages "role_messages" : [ { "role" : "system" , "content" : "You are a helpful assistant..." } ], ​ task_messages List[dict] required Defines the task for a given node. Required for all nodes. Copy Ask AI # Example task messages "task_messages" : [ { "role" : "system" , # May be `user` depending on the LLM "content" : "Ask the user for their name..." } ], ​ context_strategy ContextStrategyConfig Strategy for managing context during transitions to this node. Copy Ask AI # Example context strategy configuration "context_strategy" : ContextStrategyConfig( strategy = ContextStrategy. RESET_WITH_SUMMARY , summary_prompt = "Summarize the key points discussed so far." ) ​ functions List[Union[dict, FlowsFunctionSchema]] required LLM function / tool call configurations, defined in one of the supported formats . Copy Ask AI # Using provider-specific dictionary format "functions" : [ { "type" : "function" , "function" : { "name" : "get_current_movies" , "handler" : get_movies, "description" : "Fetch movies currently playing" , "parameters" : { ... } }, } ] # Using FlowsFunctionSchema "functions" : [ FlowsFunctionSchema( name = "get_current_movies" , description = "Fetch movies currently playing" , properties = { ... }, required = [ ... ], handler = get_movies ) ] # Using direct functions (auto-configuration) "functions" : [get_movies] ​ pre_actions List[dict] Actions that execute before the LLM inference. For example, you can send a message to the TTS to speak a phrase (e.g. “Hold on a moment…”), which may be effective if an LLM function call takes time to execute. Copy Ask AI # Example pre_actions "pre_actions" : [ { "type" : "tts_say" , "text" : "Hold on a moment..." } ], ​ post_actions List[dict] Actions that execute after the LLM inference. For example, you can end the conversation. Copy Ask AI # Example post_actions "post_actions" : [ { "type" : "end_conversation" } ] ​ respond_immediately bool If set to False , the LLM will not respond immediately when the node is set, but will instead wait for the user to speak first before responding. Defaults to True . Copy Ask AI # Example usage "respond_immediately" : False ​ Function Handler Types ​ LegacyFunctionHandler Callable[[FlowArgs], Awaitable[FlowResult | ConsolidatedFunctionResult]] Legacy function handler that only receives arguments. Returns either: A FlowResult (⚠️ deprecated) A “consolidated” result tuple (result, next node) where: result is an optional FlowResult next node is an optional NodeConfig (for dynamic flows) or string (for static flows) ​ FlowFunctionHandler Callable[[FlowArgs, FlowManager], Awaitable[FlowResult | ConsolidatedFunctionResult]] Modern function handler that receives both arguments and FlowManager . Returns either: A FlowResult (⚠️ deprecated) A “consolidated” result tuple (result, next node) where: result is an optional FlowResult next node is an optional NodeConfig (for dynamic flows) or string (for static flows) ​ DirectFunction DirectFunction Function that is meant to be passed directly into a NodeConfig rather than into the handler field of a function configuration. It must be an async function with flow_manager: FlowManager as its first parameter. It must return a ConsolidatedFunctionResult , which is a tuple (result, next node) where: result is an optional FlowResult next node is an optional NodeConfig (for dynamic flows) or string (for static flows) ​ ContextStrategy ​ ContextStrategy Enum Strategy for managing conversation context during node transitions. Show Values ​ APPEND str Default strategy. Adds new messages to existing context. ​ RESET str Clears context and starts fresh with new messages. ​ RESET_WITH_SUMMARY str Resets context but includes an AI-generated summary. ​ ContextStrategyConfig ​ ContextStrategyConfig dataclass Configuration for context management strategy. Show Fields ​ strategy ContextStrategy required The strategy to use for context management ​ summary_prompt Optional[str] Required when using RESET_WITH_SUMMARY. Prompt text for generating the conversation summary. Copy Ask AI # Example usage config = ContextStrategyConfig( strategy = ContextStrategy. RESET_WITH_SUMMARY , summary_prompt = "Summarize the key points discussed so far." ) ​ FlowsFunctionSchema ​ FlowsFunctionSchema class A standardized schema for defining functions in Pipecat Flows with flow-specific properties. Show Constructor Parameters ​ name str required Name of the function ​ description str required Description of the function’s purpose ​ properties Dict[str, Any] required Dictionary defining properties types and descriptions ​ required List[str] required List of required parameter names ​ handler Optional[FunctionHandler] Function handler to process the function call ​ transition_to Optional[str] deprecated Target node to transition to after function execution Deprecated: instead of transition_to , use a “consolidated” handler that returns a tuple (result, next node). ​ transition_callback Optional[Callable] deprecated Callback function for dynamic transitions Deprecated: instead of transition_callback , use a “consolidated” handler that returns a tuple (result, next node). You cannot specify both transition_to and transition_callback in the same function schema. Example usage: Copy Ask AI from pipecat_flows import FlowsFunctionSchema # Define a function schema collect_name_schema = FlowsFunctionSchema( name = "collect_name" , description = "Record the user's name" , properties = { "name" : { "type" : "string" , "description" : "The user's name" } }, required = [ "name" ], handler = collect_name_handler ) # Use in node configuration node_config = { "name" : "greeting" , "task_messages" : [ { "role" : "system" , "content" : "Ask the user for their name." } ], "functions" : [collect_name_schema] } # Pass to flow manager await flow_manager.set_node_from_config(node_config) ​ FlowManager ​ FlowManager class Main class for managing conversation flows, supporting both static (configuration-driven) and dynamic (runtime-determined) flows. Show Constructor Parameters ​ task PipelineTask required Pipeline task for frame queueing ​ llm LLMService required LLM service instance (OpenAI, Anthropic, or Google). Must be initialized with the corresponding pipecat-ai provider dependency installed. ​ context_aggregator Any required Context aggregator used for pushing messages to the LLM service ​ tts Optional[Any] deprecated Optional TTS service for voice actions. Deprecated: No need to explicitly pass tts to FlowManager in order to use tts_say actions. ​ flow_config Optional[FlowConfig] Optional static flow configuration ​ context_strategy Optional[ContextStrategyConfig] Optional configuration for how context should be managed during transitions. Defaults to APPEND strategy if not specified. ​ Methods ​ initialize method Initialize the flow with starting messages. Show Parameters ​ initial_node NodeConfig The initial conversation node (needed for dynamic flows only). If not specified, you’ll need to call set_node_from_config() to kick off the conversation. Show Raises ​ FlowInitializationError If initialization fails ​ set_node method deprecated Set up a new conversation node programmatically (dynamic flows only). In dynamic flows, the application advances the conversation using set_node to set up each next node. In static flows, set_node is triggered under the hood when a node contains a transition_to field. Deprecated: use the following patterns instead of set_node : Prefer “consolidated” function handlers that return a tuple (result, next node), which implicitly sets up the next node Prefer passing your initial node to FlowManager.initialize() If you really need to set a node explicitly, use set_node_from_config() (note: its name will be read from its NodeConfig ) Show Parameters ​ node_id str required Identifier for the new node ​ node_config NodeConfig required Node configuration including messages, functions, and actions Show Raises ​ FlowError If node setup fails ​ set_node_from_config method Set up a new conversation node programmatically (dynamic flows only). Note that this method should only be used in rare circumstances. Most often, you should: Prefer “consolidated” function handlers that return a tuple (result, next node), which implicitly sets up the next node Prefer passing your initial node to FlowManager.initialize() Show Parameters ​ node_config NodeConfig required Node configuration including messages, functions, and actions Show Raises ​ FlowError If node setup fails ​ register_action method Register a handler for a custom action type. Show Parameters ​ action_type str required String identifier for the action ​ handler Callable required Async or sync function that handles the action ​ get_current_context method Get the current conversation context. Returns a list of messages in the current context, including system messages, user messages, and assistant responses. Show Returns ​ messages List[dict] List of messages in the current context Show Raises ​ FlowError If context aggregator is not available Example usage: Copy Ask AI # Access current conversation context context = flow_manager.get_current_context() # Use in handlers async def process_response ( args : FlowArgs) -> tuple[FlowResult, str ]: context = flow_manager.get_current_context() # Process conversation history return { "status" : "success" }, "next" ​ State Management The FlowManager provides a state dictionary for storing conversation data: Access state Access in transitions Copy Ask AI flow_manager.state: Dict[ str , Any] # Store data flow_manager.state[ "user_age" ] = 25 ​ Usage Examples Static Flow Dynamic Flow Copy Ask AI flow_config: FlowConfig = { "initial_node" : "greeting" , "nodes" : { "greeting" : { "role_messages" : [ { "role" : "system" , "content" : "You are a helpful assistant. Your responses will be converted to audio." } ], "task_messages" : [ { "role" : "system" , "content" : "Start by greeting the user and asking for their name." } ], "functions" : [{ "type" : "function" , "function" : { "name" : "collect_name" , "handler" : collect_name_handler, "description" : "Record user's name" , "parameters" : { ... } } }] } } } # Create and initialize the FlowManager flow_manager = FlowManager( task = task, llm = llm, context_aggregator = context_aggregator, flow_config = flow_config ) # Initialize the flow_manager to start the conversation await flow_manager.initialize() ​ Node Functions concept Functions that execute operations within a single conversational state, without switching nodes. Copy Ask AI Copy Ask AI from pipecat_flows import FlowArgs, FlowResult async def process_data ( args : FlowArgs) -> tuple[FlowResult, None ]: """Handle data processing within a node.""" data = args[ "data" ] result = await process(data) return { "status" : "success" , "processed_data" : result }, None # Function configuration { "type" : "function" , "function" : { "name" : "process_data" , "handler" : process_data, "description" : "Process user data" , "parameters" : { "type" : "object" , "properties" : { "data" : { "type" : "string" } } } } } ​ Edge Functions concept Functions that specify a transition between nodes (optionally processing data first). Copy Ask AI Copy Ask AI from pipecat_flows import FlowArgs, FlowResult async def next_step ( args : FlowArgs) -> tuple[ None , str ]: """Specify the next node to transition to.""" return None , "target_node" # Return NodeConfig instead of str for dynamic flows # Function configuration { "type" : "function" , "function" : { "name" : "next_step" , "handler" : next_step, "description" : "Transition to next node" , "parameters" : { "type" : "object" , "properties" : {}} } } ​ Function Properties ​ handler Optional[Callable] Async function that processes data within a node and/or specifies the next node ( more details here ). Can be specified as: Direct function reference Either a Callable function or a string with __function__: prefix (e.g., "__function__:process_data" ) to reference a function in the main script Direct Reference Function Token Copy Ask AI { "type" : "function" , "function" : { "name" : "process_data" , "handler" : process_data, # Callable function "parameters" : { ... } } } ​ transition_callback Optional[Callable] deprecated Handler for dynamic flow transitions. Deprecated: instead of transition_callback , use a “consolidated” handler that returns a tuple (result, next node). Must be an async function with one of these signatures: Copy Ask AI # New style (recommended) async def handle_transition ( args : Dict[ str , Any], result : FlowResult, flow_manager : FlowManager ) -> None : """Handle transition to next node.""" if result.available: # Type-safe access to result await flow_manager.set_node_from_config(create_confirmation_node()) else : await flow_manager.set_node_from_config( create_no_availability_node(result.alternative_times) ) # Legacy style (supported for backwards compatibility) async def handle_transition ( args : Dict[ str , Any], flow_manager : FlowManager ) -> None : """Handle transition to next node.""" await flow_manager.set_node_from_config(create_next_node()) The callback receives: args : Arguments from the function call result : Typed result from the function handler (new style only) flow_manager : Reference to the FlowManager instance Example usage: Copy Ask AI async def handle_availability_check ( args : Dict, result : TimeResult, # Typed result flow_manager : FlowManager ): """Handle availability check and transition based on result.""" if result.available: await flow_manager.set_node_from_config(create_confirmation_node()) else : await flow_manager.set_node_from_config( create_no_availability_node(result.alternative_times) ) # Use in function configuration { "type" : "function" , "function" : { "name" : "check_availability" , "handler" : check_availability, "parameters" : { ... }, "transition_callback" : handle_availability_check } } Note: A function cannot have both transition_to and transition_callback . ​ Handler Signatures Function handlers passed as a handler in a function configuration can be defined with three different signatures: Modern (Args + FlowManager) Legacy (Args Only) No Arguments Copy Ask AI async def handler_with_flow_manager ( args : FlowArgs, flow_manager : FlowManager) -> tuple[FlowResult, NodeConfig]: """Modern handler that receives both arguments and FlowManager access.""" # Access state previous_data = flow_manager.state.get( "stored_data" ) # Access pipeline resources await flow_manager.task.queue_frame(TTSSpeakFrame( "Processing your request..." )) # Store data in state for later flow_manager.state[ "new_data" ] = args[ "input" ] return { "status" : "success" , "result" : "Processed with flow access" }, create_next_node() The framework automatically detects which signature your handler is using and calls it appropriately. If you’re passing your function directly into your NodeConfig rather than as a handler in a function configuration, you’d use a somewhat different signature: Direct Copy Ask AI async def do_something ( flow_manager : FlowManager, foo : int , bar : str = "" ) -> tuple[FlowResult, NodeConfig]: """ Do something interesting. Args: foo (int): The foo to do something interesting with. bar (string): The bar to do something interesting with. Defaults to empty string. """ result = await fetch_data(foo, bar) next_node = create_end_node() return result, next_node ​ Return Types Success Response Error Response Copy Ask AI { "status" : "success" , "data" : "some data" # Optional additional data } ​ Provider-Specific Formats You don’t need to handle these format differences manually - use the standard format and the FlowManager will adapt it for your chosen provider. OpenAI Anthropic Google (Gemini) Copy Ask AI { "type" : "function" , "function" : { "name" : "function_name" , "handler" : handler, "description" : "Description" , "parameters" : { ... } } } ​ Actions pre_actions and post_actions are used to manage conversation flow. They are included in the NodeConfig and executed before and after the LLM completion, respectively. Three kinds of actions are available: Pre-canned actions: These actions perform common tasks and require little configuration. Function actions: These actions run developer-defined functions at the appropriate time. Custom actions: These are fully developer-defined actions, providing flexibility at the expense of complexity. ​ Pre-canned Actions Common actions shipped with Flows for managing conversation flow. To use them, just add them to your NodeConfig . ​ tts_say action Speaks text immediately using the TTS service. Copy Ask AI Copy Ask AI "pre_actions" : [ { "type" : "tts_say" , "text" : "Processing your request..." # Required } ] ​ end_conversation action Ends the conversation and closes the connection. Copy Ask AI Copy Ask AI "post_actions" : [ { "type" : "end_conversation" , "text" : "Goodbye!" # Optional farewell message } ] ​ Function Actions Actions that run developer-defined functions at the appropriate time. For example, if used in post_actions , they’ll run after the bot has finished talking and after any previous post_actions have finished. ​ function action Calls the developer-defined function at the appropriate time. Copy Ask AI Copy Ask AI "post_actions" : [ { "type" : "function" , "handler" : bot_turn_ended # Required } ] ​ Custom Actions Fully developer-defined actions, providing flexibility at the expense of complexity. Here’s the complexity: because these actions aren’t queued in the Pipecat pipeline, they may execute seemingly early if used in post_actions ; they’ll run immediately after the LLM completion is triggered but won’t wait around for the bot to finish talking. Why would you want this behavior? You might be writing an action that: Itself just queues another Frame into the Pipecat pipeline (meaning there would no benefit to waiting around for sequencing purposes) Does work that can be done a bit sooner, like logging that the LLM was updated Custom actions are composed of at least: ​ type str required String identifier for the action ​ handler Callable required Async or sync function that handles the action Example: Copy Ask AI Copy Ask AI # Define custom action handler async def custom_notification ( action : dict , flow_manager : FlowManager): """Custom action handler.""" message = action.get( "message" , "" ) await notify_user(message) # Use in node configuration "pre_actions" : [ { "type" : "notify" , "handler" : send_notification, "message" : "Attention!" , } ] ​ Exceptions ​ FlowError exception Base exception for all flow-related errors. Copy Ask AI Copy Ask AI from pipecat_flows import FlowError try : await flow_manager.set_node_from_config(config) except FlowError as e: print ( f "Flow error: { e } " ) ​ FlowInitializationError exception Raised when flow initialization fails. Copy Ask AI Copy Ask AI from pipecat_flows import FlowInitializationError try : await flow_manager.initialize() except FlowInitializationError as e: print ( f "Initialization failed: { e } " ) ​ FlowTransitionError exception Raised when a state transition fails. Copy Ask AI Copy Ask AI from pipecat_flows import FlowTransitionError try : await flow_manager.set_node_from_config(node_config) except FlowTransitionError as e: print ( f "Transition failed: { e } " ) ​ InvalidFunctionError exception Raised when an invalid or unavailable function is specified. Copy Ask AI Copy Ask AI from pipecat_flows import InvalidFunctionError try : await flow_manager.set_node_from_config({ "functions" : [{ "type" : "function" , "function" : { "name" : "invalid_function" } }] }) except InvalidFunctionError as e: print ( f "Invalid function: { e } " ) RTVI Observer PipelineParams On this page Installation Core Types FlowArgs FlowResult FlowConfig NodeConfig Function Handler Types ContextStrategy ContextStrategyConfig FlowsFunctionSchema FlowManager Methods State Management Usage Examples Function Properties Handler Signatures Return Types Provider-Specific Formats Actions Pre-canned Actions Function Actions Custom Actions Exceptions Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/flows_pipecat-flows_523e5ce5.txt b/flows_pipecat-flows_523e5ce5.txt
new file mode 100644
index 0000000000000000000000000000000000000000..ceee4396dc7d8a2b6b70a7cc7147da88ea08aa76
--- /dev/null
+++ b/flows_pipecat-flows_523e5ce5.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/frameworks/flows/pipecat-flows#param-tts-say
+Title: Pipecat Flows - Pipecat
+==================================================
+
+Pipecat Flows - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Frameworks Pipecat Flows Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline New to building conversational flows? Check out our Pipecat Flows guide first. ​ Installation Existing Pipecat installation Fresh Pipecat installation Copy Ask AI pip install pipecat-ai-flows ​ Core Types ​ FlowArgs ​ FlowArgs Dict[str, Any] Type alias for function handler arguments. ​ FlowResult ​ FlowResult TypedDict Base type for function handler results. Additional fields can be included as needed. Show Fields ​ status str Optional status field ​ error str Optional error message ​ FlowConfig ​ FlowConfig TypedDict Configuration for the entire conversation flow. Show Fields ​ initial_node str required Starting node identifier ​ nodes Dict[str, NodeConfig] required Map of node names to configurations ​ NodeConfig ​ NodeConfig TypedDict Configuration for a single node in the flow. Show Fields ​ name str The name of the node, used in debug logging in dynamic flows. If no name is specified, an automatically-generated UUID is used. Copy Ask AI # Example name "name" : "greeting" ​ role_messages List[dict] Defines the role or persona of the LLM. Required for the initial node and optional for subsequent nodes. Copy Ask AI # Example role messages "role_messages" : [ { "role" : "system" , "content" : "You are a helpful assistant..." } ], ​ task_messages List[dict] required Defines the task for a given node. Required for all nodes. Copy Ask AI # Example task messages "task_messages" : [ { "role" : "system" , # May be `user` depending on the LLM "content" : "Ask the user for their name..." } ], ​ context_strategy ContextStrategyConfig Strategy for managing context during transitions to this node. Copy Ask AI # Example context strategy configuration "context_strategy" : ContextStrategyConfig( strategy = ContextStrategy. RESET_WITH_SUMMARY , summary_prompt = "Summarize the key points discussed so far." ) ​ functions List[Union[dict, FlowsFunctionSchema]] required LLM function / tool call configurations, defined in one of the supported formats . Copy Ask AI # Using provider-specific dictionary format "functions" : [ { "type" : "function" , "function" : { "name" : "get_current_movies" , "handler" : get_movies, "description" : "Fetch movies currently playing" , "parameters" : { ... } }, } ] # Using FlowsFunctionSchema "functions" : [ FlowsFunctionSchema( name = "get_current_movies" , description = "Fetch movies currently playing" , properties = { ... }, required = [ ... ], handler = get_movies ) ] # Using direct functions (auto-configuration) "functions" : [get_movies] ​ pre_actions List[dict] Actions that execute before the LLM inference. For example, you can send a message to the TTS to speak a phrase (e.g. “Hold on a moment…”), which may be effective if an LLM function call takes time to execute. Copy Ask AI # Example pre_actions "pre_actions" : [ { "type" : "tts_say" , "text" : "Hold on a moment..." } ], ​ post_actions List[dict] Actions that execute after the LLM inference. For example, you can end the conversation. Copy Ask AI # Example post_actions "post_actions" : [ { "type" : "end_conversation" } ] ​ respond_immediately bool If set to False , the LLM will not respond immediately when the node is set, but will instead wait for the user to speak first before responding. Defaults to True . Copy Ask AI # Example usage "respond_immediately" : False ​ Function Handler Types ​ LegacyFunctionHandler Callable[[FlowArgs], Awaitable[FlowResult | ConsolidatedFunctionResult]] Legacy function handler that only receives arguments. Returns either: A FlowResult (⚠️ deprecated) A “consolidated” result tuple (result, next node) where: result is an optional FlowResult next node is an optional NodeConfig (for dynamic flows) or string (for static flows) ​ FlowFunctionHandler Callable[[FlowArgs, FlowManager], Awaitable[FlowResult | ConsolidatedFunctionResult]] Modern function handler that receives both arguments and FlowManager . Returns either: A FlowResult (⚠️ deprecated) A “consolidated” result tuple (result, next node) where: result is an optional FlowResult next node is an optional NodeConfig (for dynamic flows) or string (for static flows) ​ DirectFunction DirectFunction Function that is meant to be passed directly into a NodeConfig rather than into the handler field of a function configuration. It must be an async function with flow_manager: FlowManager as its first parameter. It must return a ConsolidatedFunctionResult , which is a tuple (result, next node) where: result is an optional FlowResult next node is an optional NodeConfig (for dynamic flows) or string (for static flows) ​ ContextStrategy ​ ContextStrategy Enum Strategy for managing conversation context during node transitions. Show Values ​ APPEND str Default strategy. Adds new messages to existing context. ​ RESET str Clears context and starts fresh with new messages. ​ RESET_WITH_SUMMARY str Resets context but includes an AI-generated summary. ​ ContextStrategyConfig ​ ContextStrategyConfig dataclass Configuration for context management strategy. Show Fields ​ strategy ContextStrategy required The strategy to use for context management ​ summary_prompt Optional[str] Required when using RESET_WITH_SUMMARY. Prompt text for generating the conversation summary. Copy Ask AI # Example usage config = ContextStrategyConfig( strategy = ContextStrategy. RESET_WITH_SUMMARY , summary_prompt = "Summarize the key points discussed so far." ) ​ FlowsFunctionSchema ​ FlowsFunctionSchema class A standardized schema for defining functions in Pipecat Flows with flow-specific properties. Show Constructor Parameters ​ name str required Name of the function ​ description str required Description of the function’s purpose ​ properties Dict[str, Any] required Dictionary defining properties types and descriptions ​ required List[str] required List of required parameter names ​ handler Optional[FunctionHandler] Function handler to process the function call ​ transition_to Optional[str] deprecated Target node to transition to after function execution Deprecated: instead of transition_to , use a “consolidated” handler that returns a tuple (result, next node). ​ transition_callback Optional[Callable] deprecated Callback function for dynamic transitions Deprecated: instead of transition_callback , use a “consolidated” handler that returns a tuple (result, next node). You cannot specify both transition_to and transition_callback in the same function schema. Example usage: Copy Ask AI from pipecat_flows import FlowsFunctionSchema # Define a function schema collect_name_schema = FlowsFunctionSchema( name = "collect_name" , description = "Record the user's name" , properties = { "name" : { "type" : "string" , "description" : "The user's name" } }, required = [ "name" ], handler = collect_name_handler ) # Use in node configuration node_config = { "name" : "greeting" , "task_messages" : [ { "role" : "system" , "content" : "Ask the user for their name." } ], "functions" : [collect_name_schema] } # Pass to flow manager await flow_manager.set_node_from_config(node_config) ​ FlowManager ​ FlowManager class Main class for managing conversation flows, supporting both static (configuration-driven) and dynamic (runtime-determined) flows. Show Constructor Parameters ​ task PipelineTask required Pipeline task for frame queueing ​ llm LLMService required LLM service instance (OpenAI, Anthropic, or Google). Must be initialized with the corresponding pipecat-ai provider dependency installed. ​ context_aggregator Any required Context aggregator used for pushing messages to the LLM service ​ tts Optional[Any] deprecated Optional TTS service for voice actions. Deprecated: No need to explicitly pass tts to FlowManager in order to use tts_say actions. ​ flow_config Optional[FlowConfig] Optional static flow configuration ​ context_strategy Optional[ContextStrategyConfig] Optional configuration for how context should be managed during transitions. Defaults to APPEND strategy if not specified. ​ Methods ​ initialize method Initialize the flow with starting messages. Show Parameters ​ initial_node NodeConfig The initial conversation node (needed for dynamic flows only). If not specified, you’ll need to call set_node_from_config() to kick off the conversation. Show Raises ​ FlowInitializationError If initialization fails ​ set_node method deprecated Set up a new conversation node programmatically (dynamic flows only). In dynamic flows, the application advances the conversation using set_node to set up each next node. In static flows, set_node is triggered under the hood when a node contains a transition_to field. Deprecated: use the following patterns instead of set_node : Prefer “consolidated” function handlers that return a tuple (result, next node), which implicitly sets up the next node Prefer passing your initial node to FlowManager.initialize() If you really need to set a node explicitly, use set_node_from_config() (note: its name will be read from its NodeConfig ) Show Parameters ​ node_id str required Identifier for the new node ​ node_config NodeConfig required Node configuration including messages, functions, and actions Show Raises ​ FlowError If node setup fails ​ set_node_from_config method Set up a new conversation node programmatically (dynamic flows only). Note that this method should only be used in rare circumstances. Most often, you should: Prefer “consolidated” function handlers that return a tuple (result, next node), which implicitly sets up the next node Prefer passing your initial node to FlowManager.initialize() Show Parameters ​ node_config NodeConfig required Node configuration including messages, functions, and actions Show Raises ​ FlowError If node setup fails ​ register_action method Register a handler for a custom action type. Show Parameters ​ action_type str required String identifier for the action ​ handler Callable required Async or sync function that handles the action ​ get_current_context method Get the current conversation context. Returns a list of messages in the current context, including system messages, user messages, and assistant responses. Show Returns ​ messages List[dict] List of messages in the current context Show Raises ​ FlowError If context aggregator is not available Example usage: Copy Ask AI # Access current conversation context context = flow_manager.get_current_context() # Use in handlers async def process_response ( args : FlowArgs) -> tuple[FlowResult, str ]: context = flow_manager.get_current_context() # Process conversation history return { "status" : "success" }, "next" ​ State Management The FlowManager provides a state dictionary for storing conversation data: Access state Access in transitions Copy Ask AI flow_manager.state: Dict[ str , Any] # Store data flow_manager.state[ "user_age" ] = 25 ​ Usage Examples Static Flow Dynamic Flow Copy Ask AI flow_config: FlowConfig = { "initial_node" : "greeting" , "nodes" : { "greeting" : { "role_messages" : [ { "role" : "system" , "content" : "You are a helpful assistant. Your responses will be converted to audio." } ], "task_messages" : [ { "role" : "system" , "content" : "Start by greeting the user and asking for their name." } ], "functions" : [{ "type" : "function" , "function" : { "name" : "collect_name" , "handler" : collect_name_handler, "description" : "Record user's name" , "parameters" : { ... } } }] } } } # Create and initialize the FlowManager flow_manager = FlowManager( task = task, llm = llm, context_aggregator = context_aggregator, flow_config = flow_config ) # Initialize the flow_manager to start the conversation await flow_manager.initialize() ​ Node Functions concept Functions that execute operations within a single conversational state, without switching nodes. Copy Ask AI Copy Ask AI from pipecat_flows import FlowArgs, FlowResult async def process_data ( args : FlowArgs) -> tuple[FlowResult, None ]: """Handle data processing within a node.""" data = args[ "data" ] result = await process(data) return { "status" : "success" , "processed_data" : result }, None # Function configuration { "type" : "function" , "function" : { "name" : "process_data" , "handler" : process_data, "description" : "Process user data" , "parameters" : { "type" : "object" , "properties" : { "data" : { "type" : "string" } } } } } ​ Edge Functions concept Functions that specify a transition between nodes (optionally processing data first). Copy Ask AI Copy Ask AI from pipecat_flows import FlowArgs, FlowResult async def next_step ( args : FlowArgs) -> tuple[ None , str ]: """Specify the next node to transition to.""" return None , "target_node" # Return NodeConfig instead of str for dynamic flows # Function configuration { "type" : "function" , "function" : { "name" : "next_step" , "handler" : next_step, "description" : "Transition to next node" , "parameters" : { "type" : "object" , "properties" : {}} } } ​ Function Properties ​ handler Optional[Callable] Async function that processes data within a node and/or specifies the next node ( more details here ). Can be specified as: Direct function reference Either a Callable function or a string with __function__: prefix (e.g., "__function__:process_data" ) to reference a function in the main script Direct Reference Function Token Copy Ask AI { "type" : "function" , "function" : { "name" : "process_data" , "handler" : process_data, # Callable function "parameters" : { ... } } } ​ transition_callback Optional[Callable] deprecated Handler for dynamic flow transitions. Deprecated: instead of transition_callback , use a “consolidated” handler that returns a tuple (result, next node). Must be an async function with one of these signatures: Copy Ask AI # New style (recommended) async def handle_transition ( args : Dict[ str , Any], result : FlowResult, flow_manager : FlowManager ) -> None : """Handle transition to next node.""" if result.available: # Type-safe access to result await flow_manager.set_node_from_config(create_confirmation_node()) else : await flow_manager.set_node_from_config( create_no_availability_node(result.alternative_times) ) # Legacy style (supported for backwards compatibility) async def handle_transition ( args : Dict[ str , Any], flow_manager : FlowManager ) -> None : """Handle transition to next node.""" await flow_manager.set_node_from_config(create_next_node()) The callback receives: args : Arguments from the function call result : Typed result from the function handler (new style only) flow_manager : Reference to the FlowManager instance Example usage: Copy Ask AI async def handle_availability_check ( args : Dict, result : TimeResult, # Typed result flow_manager : FlowManager ): """Handle availability check and transition based on result.""" if result.available: await flow_manager.set_node_from_config(create_confirmation_node()) else : await flow_manager.set_node_from_config( create_no_availability_node(result.alternative_times) ) # Use in function configuration { "type" : "function" , "function" : { "name" : "check_availability" , "handler" : check_availability, "parameters" : { ... }, "transition_callback" : handle_availability_check } } Note: A function cannot have both transition_to and transition_callback . ​ Handler Signatures Function handlers passed as a handler in a function configuration can be defined with three different signatures: Modern (Args + FlowManager) Legacy (Args Only) No Arguments Copy Ask AI async def handler_with_flow_manager ( args : FlowArgs, flow_manager : FlowManager) -> tuple[FlowResult, NodeConfig]: """Modern handler that receives both arguments and FlowManager access.""" # Access state previous_data = flow_manager.state.get( "stored_data" ) # Access pipeline resources await flow_manager.task.queue_frame(TTSSpeakFrame( "Processing your request..." )) # Store data in state for later flow_manager.state[ "new_data" ] = args[ "input" ] return { "status" : "success" , "result" : "Processed with flow access" }, create_next_node() The framework automatically detects which signature your handler is using and calls it appropriately. If you’re passing your function directly into your NodeConfig rather than as a handler in a function configuration, you’d use a somewhat different signature: Direct Copy Ask AI async def do_something ( flow_manager : FlowManager, foo : int , bar : str = "" ) -> tuple[FlowResult, NodeConfig]: """ Do something interesting. Args: foo (int): The foo to do something interesting with. bar (string): The bar to do something interesting with. Defaults to empty string. """ result = await fetch_data(foo, bar) next_node = create_end_node() return result, next_node ​ Return Types Success Response Error Response Copy Ask AI { "status" : "success" , "data" : "some data" # Optional additional data } ​ Provider-Specific Formats You don’t need to handle these format differences manually - use the standard format and the FlowManager will adapt it for your chosen provider. OpenAI Anthropic Google (Gemini) Copy Ask AI { "type" : "function" , "function" : { "name" : "function_name" , "handler" : handler, "description" : "Description" , "parameters" : { ... } } } ​ Actions pre_actions and post_actions are used to manage conversation flow. They are included in the NodeConfig and executed before and after the LLM completion, respectively. Three kinds of actions are available: Pre-canned actions: These actions perform common tasks and require little configuration. Function actions: These actions run developer-defined functions at the appropriate time. Custom actions: These are fully developer-defined actions, providing flexibility at the expense of complexity. ​ Pre-canned Actions Common actions shipped with Flows for managing conversation flow. To use them, just add them to your NodeConfig . ​ tts_say action Speaks text immediately using the TTS service. Copy Ask AI Copy Ask AI "pre_actions" : [ { "type" : "tts_say" , "text" : "Processing your request..." # Required } ] ​ end_conversation action Ends the conversation and closes the connection. Copy Ask AI Copy Ask AI "post_actions" : [ { "type" : "end_conversation" , "text" : "Goodbye!" # Optional farewell message } ] ​ Function Actions Actions that run developer-defined functions at the appropriate time. For example, if used in post_actions , they’ll run after the bot has finished talking and after any previous post_actions have finished. ​ function action Calls the developer-defined function at the appropriate time. Copy Ask AI Copy Ask AI "post_actions" : [ { "type" : "function" , "handler" : bot_turn_ended # Required } ] ​ Custom Actions Fully developer-defined actions, providing flexibility at the expense of complexity. Here’s the complexity: because these actions aren’t queued in the Pipecat pipeline, they may execute seemingly early if used in post_actions ; they’ll run immediately after the LLM completion is triggered but won’t wait around for the bot to finish talking. Why would you want this behavior? You might be writing an action that: Itself just queues another Frame into the Pipecat pipeline (meaning there would no benefit to waiting around for sequencing purposes) Does work that can be done a bit sooner, like logging that the LLM was updated Custom actions are composed of at least: ​ type str required String identifier for the action ​ handler Callable required Async or sync function that handles the action Example: Copy Ask AI Copy Ask AI # Define custom action handler async def custom_notification ( action : dict , flow_manager : FlowManager): """Custom action handler.""" message = action.get( "message" , "" ) await notify_user(message) # Use in node configuration "pre_actions" : [ { "type" : "notify" , "handler" : send_notification, "message" : "Attention!" , } ] ​ Exceptions ​ FlowError exception Base exception for all flow-related errors. Copy Ask AI Copy Ask AI from pipecat_flows import FlowError try : await flow_manager.set_node_from_config(config) except FlowError as e: print ( f "Flow error: { e } " ) ​ FlowInitializationError exception Raised when flow initialization fails. Copy Ask AI Copy Ask AI from pipecat_flows import FlowInitializationError try : await flow_manager.initialize() except FlowInitializationError as e: print ( f "Initialization failed: { e } " ) ​ FlowTransitionError exception Raised when a state transition fails. Copy Ask AI Copy Ask AI from pipecat_flows import FlowTransitionError try : await flow_manager.set_node_from_config(node_config) except FlowTransitionError as e: print ( f "Transition failed: { e } " ) ​ InvalidFunctionError exception Raised when an invalid or unavailable function is specified. Copy Ask AI Copy Ask AI from pipecat_flows import InvalidFunctionError try : await flow_manager.set_node_from_config({ "functions" : [{ "type" : "function" , "function" : { "name" : "invalid_function" } }] }) except InvalidFunctionError as e: print ( f "Invalid function: { e } " ) RTVI Observer PipelineParams On this page Installation Core Types FlowArgs FlowResult FlowConfig NodeConfig Function Handler Types ContextStrategy ContextStrategyConfig FlowsFunctionSchema FlowManager Methods State Management Usage Examples Function Properties Handler Signatures Return Types Provider-Specific Formats Actions Pre-canned Actions Function Actions Custom Actions Exceptions Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/flows_pipecat-flows_54a1cc6f.txt b/flows_pipecat-flows_54a1cc6f.txt
new file mode 100644
index 0000000000000000000000000000000000000000..a42e6ff931a55286df0d03aa2241ecc8a7ed03d0
--- /dev/null
+++ b/flows_pipecat-flows_54a1cc6f.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/frameworks/flows/pipecat-flows#param-edge-functions
+Title: Pipecat Flows - Pipecat
+==================================================
+
+Pipecat Flows - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Frameworks Pipecat Flows Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline New to building conversational flows? Check out our Pipecat Flows guide first. ​ Installation Existing Pipecat installation Fresh Pipecat installation Copy Ask AI pip install pipecat-ai-flows ​ Core Types ​ FlowArgs ​ FlowArgs Dict[str, Any] Type alias for function handler arguments. ​ FlowResult ​ FlowResult TypedDict Base type for function handler results. Additional fields can be included as needed. Show Fields ​ status str Optional status field ​ error str Optional error message ​ FlowConfig ​ FlowConfig TypedDict Configuration for the entire conversation flow. Show Fields ​ initial_node str required Starting node identifier ​ nodes Dict[str, NodeConfig] required Map of node names to configurations ​ NodeConfig ​ NodeConfig TypedDict Configuration for a single node in the flow. Show Fields ​ name str The name of the node, used in debug logging in dynamic flows. If no name is specified, an automatically-generated UUID is used. Copy Ask AI # Example name "name" : "greeting" ​ role_messages List[dict] Defines the role or persona of the LLM. Required for the initial node and optional for subsequent nodes. Copy Ask AI # Example role messages "role_messages" : [ { "role" : "system" , "content" : "You are a helpful assistant..." } ], ​ task_messages List[dict] required Defines the task for a given node. Required for all nodes. Copy Ask AI # Example task messages "task_messages" : [ { "role" : "system" , # May be `user` depending on the LLM "content" : "Ask the user for their name..." } ], ​ context_strategy ContextStrategyConfig Strategy for managing context during transitions to this node. Copy Ask AI # Example context strategy configuration "context_strategy" : ContextStrategyConfig( strategy = ContextStrategy. RESET_WITH_SUMMARY , summary_prompt = "Summarize the key points discussed so far." ) ​ functions List[Union[dict, FlowsFunctionSchema]] required LLM function / tool call configurations, defined in one of the supported formats . Copy Ask AI # Using provider-specific dictionary format "functions" : [ { "type" : "function" , "function" : { "name" : "get_current_movies" , "handler" : get_movies, "description" : "Fetch movies currently playing" , "parameters" : { ... } }, } ] # Using FlowsFunctionSchema "functions" : [ FlowsFunctionSchema( name = "get_current_movies" , description = "Fetch movies currently playing" , properties = { ... }, required = [ ... ], handler = get_movies ) ] # Using direct functions (auto-configuration) "functions" : [get_movies] ​ pre_actions List[dict] Actions that execute before the LLM inference. For example, you can send a message to the TTS to speak a phrase (e.g. “Hold on a moment…”), which may be effective if an LLM function call takes time to execute. Copy Ask AI # Example pre_actions "pre_actions" : [ { "type" : "tts_say" , "text" : "Hold on a moment..." } ], ​ post_actions List[dict] Actions that execute after the LLM inference. For example, you can end the conversation. Copy Ask AI # Example post_actions "post_actions" : [ { "type" : "end_conversation" } ] ​ respond_immediately bool If set to False , the LLM will not respond immediately when the node is set, but will instead wait for the user to speak first before responding. Defaults to True . Copy Ask AI # Example usage "respond_immediately" : False ​ Function Handler Types ​ LegacyFunctionHandler Callable[[FlowArgs], Awaitable[FlowResult | ConsolidatedFunctionResult]] Legacy function handler that only receives arguments. Returns either: A FlowResult (⚠️ deprecated) A “consolidated” result tuple (result, next node) where: result is an optional FlowResult next node is an optional NodeConfig (for dynamic flows) or string (for static flows) ​ FlowFunctionHandler Callable[[FlowArgs, FlowManager], Awaitable[FlowResult | ConsolidatedFunctionResult]] Modern function handler that receives both arguments and FlowManager . Returns either: A FlowResult (⚠️ deprecated) A “consolidated” result tuple (result, next node) where: result is an optional FlowResult next node is an optional NodeConfig (for dynamic flows) or string (for static flows) ​ DirectFunction DirectFunction Function that is meant to be passed directly into a NodeConfig rather than into the handler field of a function configuration. It must be an async function with flow_manager: FlowManager as its first parameter. It must return a ConsolidatedFunctionResult , which is a tuple (result, next node) where: result is an optional FlowResult next node is an optional NodeConfig (for dynamic flows) or string (for static flows) ​ ContextStrategy ​ ContextStrategy Enum Strategy for managing conversation context during node transitions. Show Values ​ APPEND str Default strategy. Adds new messages to existing context. ​ RESET str Clears context and starts fresh with new messages. ​ RESET_WITH_SUMMARY str Resets context but includes an AI-generated summary. ​ ContextStrategyConfig ​ ContextStrategyConfig dataclass Configuration for context management strategy. Show Fields ​ strategy ContextStrategy required The strategy to use for context management ​ summary_prompt Optional[str] Required when using RESET_WITH_SUMMARY. Prompt text for generating the conversation summary. Copy Ask AI # Example usage config = ContextStrategyConfig( strategy = ContextStrategy. RESET_WITH_SUMMARY , summary_prompt = "Summarize the key points discussed so far." ) ​ FlowsFunctionSchema ​ FlowsFunctionSchema class A standardized schema for defining functions in Pipecat Flows with flow-specific properties. Show Constructor Parameters ​ name str required Name of the function ​ description str required Description of the function’s purpose ​ properties Dict[str, Any] required Dictionary defining properties types and descriptions ​ required List[str] required List of required parameter names ​ handler Optional[FunctionHandler] Function handler to process the function call ​ transition_to Optional[str] deprecated Target node to transition to after function execution Deprecated: instead of transition_to , use a “consolidated” handler that returns a tuple (result, next node). ​ transition_callback Optional[Callable] deprecated Callback function for dynamic transitions Deprecated: instead of transition_callback , use a “consolidated” handler that returns a tuple (result, next node). You cannot specify both transition_to and transition_callback in the same function schema. Example usage: Copy Ask AI from pipecat_flows import FlowsFunctionSchema # Define a function schema collect_name_schema = FlowsFunctionSchema( name = "collect_name" , description = "Record the user's name" , properties = { "name" : { "type" : "string" , "description" : "The user's name" } }, required = [ "name" ], handler = collect_name_handler ) # Use in node configuration node_config = { "name" : "greeting" , "task_messages" : [ { "role" : "system" , "content" : "Ask the user for their name." } ], "functions" : [collect_name_schema] } # Pass to flow manager await flow_manager.set_node_from_config(node_config) ​ FlowManager ​ FlowManager class Main class for managing conversation flows, supporting both static (configuration-driven) and dynamic (runtime-determined) flows. Show Constructor Parameters ​ task PipelineTask required Pipeline task for frame queueing ​ llm LLMService required LLM service instance (OpenAI, Anthropic, or Google). Must be initialized with the corresponding pipecat-ai provider dependency installed. ​ context_aggregator Any required Context aggregator used for pushing messages to the LLM service ​ tts Optional[Any] deprecated Optional TTS service for voice actions. Deprecated: No need to explicitly pass tts to FlowManager in order to use tts_say actions. ​ flow_config Optional[FlowConfig] Optional static flow configuration ​ context_strategy Optional[ContextStrategyConfig] Optional configuration for how context should be managed during transitions. Defaults to APPEND strategy if not specified. ​ Methods ​ initialize method Initialize the flow with starting messages. Show Parameters ​ initial_node NodeConfig The initial conversation node (needed for dynamic flows only). If not specified, you’ll need to call set_node_from_config() to kick off the conversation. Show Raises ​ FlowInitializationError If initialization fails ​ set_node method deprecated Set up a new conversation node programmatically (dynamic flows only). In dynamic flows, the application advances the conversation using set_node to set up each next node. In static flows, set_node is triggered under the hood when a node contains a transition_to field. Deprecated: use the following patterns instead of set_node : Prefer “consolidated” function handlers that return a tuple (result, next node), which implicitly sets up the next node Prefer passing your initial node to FlowManager.initialize() If you really need to set a node explicitly, use set_node_from_config() (note: its name will be read from its NodeConfig ) Show Parameters ​ node_id str required Identifier for the new node ​ node_config NodeConfig required Node configuration including messages, functions, and actions Show Raises ​ FlowError If node setup fails ​ set_node_from_config method Set up a new conversation node programmatically (dynamic flows only). Note that this method should only be used in rare circumstances. Most often, you should: Prefer “consolidated” function handlers that return a tuple (result, next node), which implicitly sets up the next node Prefer passing your initial node to FlowManager.initialize() Show Parameters ​ node_config NodeConfig required Node configuration including messages, functions, and actions Show Raises ​ FlowError If node setup fails ​ register_action method Register a handler for a custom action type. Show Parameters ​ action_type str required String identifier for the action ​ handler Callable required Async or sync function that handles the action ​ get_current_context method Get the current conversation context. Returns a list of messages in the current context, including system messages, user messages, and assistant responses. Show Returns ​ messages List[dict] List of messages in the current context Show Raises ​ FlowError If context aggregator is not available Example usage: Copy Ask AI # Access current conversation context context = flow_manager.get_current_context() # Use in handlers async def process_response ( args : FlowArgs) -> tuple[FlowResult, str ]: context = flow_manager.get_current_context() # Process conversation history return { "status" : "success" }, "next" ​ State Management The FlowManager provides a state dictionary for storing conversation data: Access state Access in transitions Copy Ask AI flow_manager.state: Dict[ str , Any] # Store data flow_manager.state[ "user_age" ] = 25 ​ Usage Examples Static Flow Dynamic Flow Copy Ask AI flow_config: FlowConfig = { "initial_node" : "greeting" , "nodes" : { "greeting" : { "role_messages" : [ { "role" : "system" , "content" : "You are a helpful assistant. Your responses will be converted to audio." } ], "task_messages" : [ { "role" : "system" , "content" : "Start by greeting the user and asking for their name." } ], "functions" : [{ "type" : "function" , "function" : { "name" : "collect_name" , "handler" : collect_name_handler, "description" : "Record user's name" , "parameters" : { ... } } }] } } } # Create and initialize the FlowManager flow_manager = FlowManager( task = task, llm = llm, context_aggregator = context_aggregator, flow_config = flow_config ) # Initialize the flow_manager to start the conversation await flow_manager.initialize() ​ Node Functions concept Functions that execute operations within a single conversational state, without switching nodes. Copy Ask AI Copy Ask AI from pipecat_flows import FlowArgs, FlowResult async def process_data ( args : FlowArgs) -> tuple[FlowResult, None ]: """Handle data processing within a node.""" data = args[ "data" ] result = await process(data) return { "status" : "success" , "processed_data" : result }, None # Function configuration { "type" : "function" , "function" : { "name" : "process_data" , "handler" : process_data, "description" : "Process user data" , "parameters" : { "type" : "object" , "properties" : { "data" : { "type" : "string" } } } } } ​ Edge Functions concept Functions that specify a transition between nodes (optionally processing data first). Copy Ask AI Copy Ask AI from pipecat_flows import FlowArgs, FlowResult async def next_step ( args : FlowArgs) -> tuple[ None , str ]: """Specify the next node to transition to.""" return None , "target_node" # Return NodeConfig instead of str for dynamic flows # Function configuration { "type" : "function" , "function" : { "name" : "next_step" , "handler" : next_step, "description" : "Transition to next node" , "parameters" : { "type" : "object" , "properties" : {}} } } ​ Function Properties ​ handler Optional[Callable] Async function that processes data within a node and/or specifies the next node ( more details here ). Can be specified as: Direct function reference Either a Callable function or a string with __function__: prefix (e.g., "__function__:process_data" ) to reference a function in the main script Direct Reference Function Token Copy Ask AI { "type" : "function" , "function" : { "name" : "process_data" , "handler" : process_data, # Callable function "parameters" : { ... } } } ​ transition_callback Optional[Callable] deprecated Handler for dynamic flow transitions. Deprecated: instead of transition_callback , use a “consolidated” handler that returns a tuple (result, next node). Must be an async function with one of these signatures: Copy Ask AI # New style (recommended) async def handle_transition ( args : Dict[ str , Any], result : FlowResult, flow_manager : FlowManager ) -> None : """Handle transition to next node.""" if result.available: # Type-safe access to result await flow_manager.set_node_from_config(create_confirmation_node()) else : await flow_manager.set_node_from_config( create_no_availability_node(result.alternative_times) ) # Legacy style (supported for backwards compatibility) async def handle_transition ( args : Dict[ str , Any], flow_manager : FlowManager ) -> None : """Handle transition to next node.""" await flow_manager.set_node_from_config(create_next_node()) The callback receives: args : Arguments from the function call result : Typed result from the function handler (new style only) flow_manager : Reference to the FlowManager instance Example usage: Copy Ask AI async def handle_availability_check ( args : Dict, result : TimeResult, # Typed result flow_manager : FlowManager ): """Handle availability check and transition based on result.""" if result.available: await flow_manager.set_node_from_config(create_confirmation_node()) else : await flow_manager.set_node_from_config( create_no_availability_node(result.alternative_times) ) # Use in function configuration { "type" : "function" , "function" : { "name" : "check_availability" , "handler" : check_availability, "parameters" : { ... }, "transition_callback" : handle_availability_check } } Note: A function cannot have both transition_to and transition_callback . ​ Handler Signatures Function handlers passed as a handler in a function configuration can be defined with three different signatures: Modern (Args + FlowManager) Legacy (Args Only) No Arguments Copy Ask AI async def handler_with_flow_manager ( args : FlowArgs, flow_manager : FlowManager) -> tuple[FlowResult, NodeConfig]: """Modern handler that receives both arguments and FlowManager access.""" # Access state previous_data = flow_manager.state.get( "stored_data" ) # Access pipeline resources await flow_manager.task.queue_frame(TTSSpeakFrame( "Processing your request..." )) # Store data in state for later flow_manager.state[ "new_data" ] = args[ "input" ] return { "status" : "success" , "result" : "Processed with flow access" }, create_next_node() The framework automatically detects which signature your handler is using and calls it appropriately. If you’re passing your function directly into your NodeConfig rather than as a handler in a function configuration, you’d use a somewhat different signature: Direct Copy Ask AI async def do_something ( flow_manager : FlowManager, foo : int , bar : str = "" ) -> tuple[FlowResult, NodeConfig]: """ Do something interesting. Args: foo (int): The foo to do something interesting with. bar (string): The bar to do something interesting with. Defaults to empty string. """ result = await fetch_data(foo, bar) next_node = create_end_node() return result, next_node ​ Return Types Success Response Error Response Copy Ask AI { "status" : "success" , "data" : "some data" # Optional additional data } ​ Provider-Specific Formats You don’t need to handle these format differences manually - use the standard format and the FlowManager will adapt it for your chosen provider. OpenAI Anthropic Google (Gemini) Copy Ask AI { "type" : "function" , "function" : { "name" : "function_name" , "handler" : handler, "description" : "Description" , "parameters" : { ... } } } ​ Actions pre_actions and post_actions are used to manage conversation flow. They are included in the NodeConfig and executed before and after the LLM completion, respectively. Three kinds of actions are available: Pre-canned actions: These actions perform common tasks and require little configuration. Function actions: These actions run developer-defined functions at the appropriate time. Custom actions: These are fully developer-defined actions, providing flexibility at the expense of complexity. ​ Pre-canned Actions Common actions shipped with Flows for managing conversation flow. To use them, just add them to your NodeConfig . ​ tts_say action Speaks text immediately using the TTS service. Copy Ask AI Copy Ask AI "pre_actions" : [ { "type" : "tts_say" , "text" : "Processing your request..." # Required } ] ​ end_conversation action Ends the conversation and closes the connection. Copy Ask AI Copy Ask AI "post_actions" : [ { "type" : "end_conversation" , "text" : "Goodbye!" # Optional farewell message } ] ​ Function Actions Actions that run developer-defined functions at the appropriate time. For example, if used in post_actions , they’ll run after the bot has finished talking and after any previous post_actions have finished. ​ function action Calls the developer-defined function at the appropriate time. Copy Ask AI Copy Ask AI "post_actions" : [ { "type" : "function" , "handler" : bot_turn_ended # Required } ] ​ Custom Actions Fully developer-defined actions, providing flexibility at the expense of complexity. Here’s the complexity: because these actions aren’t queued in the Pipecat pipeline, they may execute seemingly early if used in post_actions ; they’ll run immediately after the LLM completion is triggered but won’t wait around for the bot to finish talking. Why would you want this behavior? You might be writing an action that: Itself just queues another Frame into the Pipecat pipeline (meaning there would no benefit to waiting around for sequencing purposes) Does work that can be done a bit sooner, like logging that the LLM was updated Custom actions are composed of at least: ​ type str required String identifier for the action ​ handler Callable required Async or sync function that handles the action Example: Copy Ask AI Copy Ask AI # Define custom action handler async def custom_notification ( action : dict , flow_manager : FlowManager): """Custom action handler.""" message = action.get( "message" , "" ) await notify_user(message) # Use in node configuration "pre_actions" : [ { "type" : "notify" , "handler" : send_notification, "message" : "Attention!" , } ] ​ Exceptions ​ FlowError exception Base exception for all flow-related errors. Copy Ask AI Copy Ask AI from pipecat_flows import FlowError try : await flow_manager.set_node_from_config(config) except FlowError as e: print ( f "Flow error: { e } " ) ​ FlowInitializationError exception Raised when flow initialization fails. Copy Ask AI Copy Ask AI from pipecat_flows import FlowInitializationError try : await flow_manager.initialize() except FlowInitializationError as e: print ( f "Initialization failed: { e } " ) ​ FlowTransitionError exception Raised when a state transition fails. Copy Ask AI Copy Ask AI from pipecat_flows import FlowTransitionError try : await flow_manager.set_node_from_config(node_config) except FlowTransitionError as e: print ( f "Transition failed: { e } " ) ​ InvalidFunctionError exception Raised when an invalid or unavailable function is specified. Copy Ask AI Copy Ask AI from pipecat_flows import InvalidFunctionError try : await flow_manager.set_node_from_config({ "functions" : [{ "type" : "function" , "function" : { "name" : "invalid_function" } }] }) except InvalidFunctionError as e: print ( f "Invalid function: { e } " ) RTVI Observer PipelineParams On this page Installation Core Types FlowArgs FlowResult FlowConfig NodeConfig Function Handler Types ContextStrategy ContextStrategyConfig FlowsFunctionSchema FlowManager Methods State Management Usage Examples Function Properties Handler Signatures Return Types Provider-Specific Formats Actions Pre-canned Actions Function Actions Custom Actions Exceptions Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/flows_pipecat-flows_5f7c769e.txt b/flows_pipecat-flows_5f7c769e.txt
new file mode 100644
index 0000000000000000000000000000000000000000..58d39e23859689d7477398c16cf899263d54fe81
--- /dev/null
+++ b/flows_pipecat-flows_5f7c769e.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/frameworks/flows/pipecat-flows#flowresult
+Title: Pipecat Flows - Pipecat
+==================================================
+
+Pipecat Flows - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Frameworks Pipecat Flows Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline New to building conversational flows? Check out our Pipecat Flows guide first. ​ Installation Existing Pipecat installation Fresh Pipecat installation Copy Ask AI pip install pipecat-ai-flows ​ Core Types ​ FlowArgs ​ FlowArgs Dict[str, Any] Type alias for function handler arguments. ​ FlowResult ​ FlowResult TypedDict Base type for function handler results. Additional fields can be included as needed. Show Fields ​ status str Optional status field ​ error str Optional error message ​ FlowConfig ​ FlowConfig TypedDict Configuration for the entire conversation flow. Show Fields ​ initial_node str required Starting node identifier ​ nodes Dict[str, NodeConfig] required Map of node names to configurations ​ NodeConfig ​ NodeConfig TypedDict Configuration for a single node in the flow. Show Fields ​ name str The name of the node, used in debug logging in dynamic flows. If no name is specified, an automatically-generated UUID is used. Copy Ask AI # Example name "name" : "greeting" ​ role_messages List[dict] Defines the role or persona of the LLM. Required for the initial node and optional for subsequent nodes. Copy Ask AI # Example role messages "role_messages" : [ { "role" : "system" , "content" : "You are a helpful assistant..." } ], ​ task_messages List[dict] required Defines the task for a given node. Required for all nodes. Copy Ask AI # Example task messages "task_messages" : [ { "role" : "system" , # May be `user` depending on the LLM "content" : "Ask the user for their name..." } ], ​ context_strategy ContextStrategyConfig Strategy for managing context during transitions to this node. Copy Ask AI # Example context strategy configuration "context_strategy" : ContextStrategyConfig( strategy = ContextStrategy. RESET_WITH_SUMMARY , summary_prompt = "Summarize the key points discussed so far." ) ​ functions List[Union[dict, FlowsFunctionSchema]] required LLM function / tool call configurations, defined in one of the supported formats . Copy Ask AI # Using provider-specific dictionary format "functions" : [ { "type" : "function" , "function" : { "name" : "get_current_movies" , "handler" : get_movies, "description" : "Fetch movies currently playing" , "parameters" : { ... } }, } ] # Using FlowsFunctionSchema "functions" : [ FlowsFunctionSchema( name = "get_current_movies" , description = "Fetch movies currently playing" , properties = { ... }, required = [ ... ], handler = get_movies ) ] # Using direct functions (auto-configuration) "functions" : [get_movies] ​ pre_actions List[dict] Actions that execute before the LLM inference. For example, you can send a message to the TTS to speak a phrase (e.g. “Hold on a moment…”), which may be effective if an LLM function call takes time to execute. Copy Ask AI # Example pre_actions "pre_actions" : [ { "type" : "tts_say" , "text" : "Hold on a moment..." } ], ​ post_actions List[dict] Actions that execute after the LLM inference. For example, you can end the conversation. Copy Ask AI # Example post_actions "post_actions" : [ { "type" : "end_conversation" } ] ​ respond_immediately bool If set to False , the LLM will not respond immediately when the node is set, but will instead wait for the user to speak first before responding. Defaults to True . Copy Ask AI # Example usage "respond_immediately" : False ​ Function Handler Types ​ LegacyFunctionHandler Callable[[FlowArgs], Awaitable[FlowResult | ConsolidatedFunctionResult]] Legacy function handler that only receives arguments. Returns either: A FlowResult (⚠️ deprecated) A “consolidated” result tuple (result, next node) where: result is an optional FlowResult next node is an optional NodeConfig (for dynamic flows) or string (for static flows) ​ FlowFunctionHandler Callable[[FlowArgs, FlowManager], Awaitable[FlowResult | ConsolidatedFunctionResult]] Modern function handler that receives both arguments and FlowManager . Returns either: A FlowResult (⚠️ deprecated) A “consolidated” result tuple (result, next node) where: result is an optional FlowResult next node is an optional NodeConfig (for dynamic flows) or string (for static flows) ​ DirectFunction DirectFunction Function that is meant to be passed directly into a NodeConfig rather than into the handler field of a function configuration. It must be an async function with flow_manager: FlowManager as its first parameter. It must return a ConsolidatedFunctionResult , which is a tuple (result, next node) where: result is an optional FlowResult next node is an optional NodeConfig (for dynamic flows) or string (for static flows) ​ ContextStrategy ​ ContextStrategy Enum Strategy for managing conversation context during node transitions. Show Values ​ APPEND str Default strategy. Adds new messages to existing context. ​ RESET str Clears context and starts fresh with new messages. ​ RESET_WITH_SUMMARY str Resets context but includes an AI-generated summary. ​ ContextStrategyConfig ​ ContextStrategyConfig dataclass Configuration for context management strategy. Show Fields ​ strategy ContextStrategy required The strategy to use for context management ​ summary_prompt Optional[str] Required when using RESET_WITH_SUMMARY. Prompt text for generating the conversation summary. Copy Ask AI # Example usage config = ContextStrategyConfig( strategy = ContextStrategy. RESET_WITH_SUMMARY , summary_prompt = "Summarize the key points discussed so far." ) ​ FlowsFunctionSchema ​ FlowsFunctionSchema class A standardized schema for defining functions in Pipecat Flows with flow-specific properties. Show Constructor Parameters ​ name str required Name of the function ​ description str required Description of the function’s purpose ​ properties Dict[str, Any] required Dictionary defining properties types and descriptions ​ required List[str] required List of required parameter names ​ handler Optional[FunctionHandler] Function handler to process the function call ​ transition_to Optional[str] deprecated Target node to transition to after function execution Deprecated: instead of transition_to , use a “consolidated” handler that returns a tuple (result, next node). ​ transition_callback Optional[Callable] deprecated Callback function for dynamic transitions Deprecated: instead of transition_callback , use a “consolidated” handler that returns a tuple (result, next node). You cannot specify both transition_to and transition_callback in the same function schema. Example usage: Copy Ask AI from pipecat_flows import FlowsFunctionSchema # Define a function schema collect_name_schema = FlowsFunctionSchema( name = "collect_name" , description = "Record the user's name" , properties = { "name" : { "type" : "string" , "description" : "The user's name" } }, required = [ "name" ], handler = collect_name_handler ) # Use in node configuration node_config = { "name" : "greeting" , "task_messages" : [ { "role" : "system" , "content" : "Ask the user for their name." } ], "functions" : [collect_name_schema] } # Pass to flow manager await flow_manager.set_node_from_config(node_config) ​ FlowManager ​ FlowManager class Main class for managing conversation flows, supporting both static (configuration-driven) and dynamic (runtime-determined) flows. Show Constructor Parameters ​ task PipelineTask required Pipeline task for frame queueing ​ llm LLMService required LLM service instance (OpenAI, Anthropic, or Google). Must be initialized with the corresponding pipecat-ai provider dependency installed. ​ context_aggregator Any required Context aggregator used for pushing messages to the LLM service ​ tts Optional[Any] deprecated Optional TTS service for voice actions. Deprecated: No need to explicitly pass tts to FlowManager in order to use tts_say actions. ​ flow_config Optional[FlowConfig] Optional static flow configuration ​ context_strategy Optional[ContextStrategyConfig] Optional configuration for how context should be managed during transitions. Defaults to APPEND strategy if not specified. ​ Methods ​ initialize method Initialize the flow with starting messages. Show Parameters ​ initial_node NodeConfig The initial conversation node (needed for dynamic flows only). If not specified, you’ll need to call set_node_from_config() to kick off the conversation. Show Raises ​ FlowInitializationError If initialization fails ​ set_node method deprecated Set up a new conversation node programmatically (dynamic flows only). In dynamic flows, the application advances the conversation using set_node to set up each next node. In static flows, set_node is triggered under the hood when a node contains a transition_to field. Deprecated: use the following patterns instead of set_node : Prefer “consolidated” function handlers that return a tuple (result, next node), which implicitly sets up the next node Prefer passing your initial node to FlowManager.initialize() If you really need to set a node explicitly, use set_node_from_config() (note: its name will be read from its NodeConfig ) Show Parameters ​ node_id str required Identifier for the new node ​ node_config NodeConfig required Node configuration including messages, functions, and actions Show Raises ​ FlowError If node setup fails ​ set_node_from_config method Set up a new conversation node programmatically (dynamic flows only). Note that this method should only be used in rare circumstances. Most often, you should: Prefer “consolidated” function handlers that return a tuple (result, next node), which implicitly sets up the next node Prefer passing your initial node to FlowManager.initialize() Show Parameters ​ node_config NodeConfig required Node configuration including messages, functions, and actions Show Raises ​ FlowError If node setup fails ​ register_action method Register a handler for a custom action type. Show Parameters ​ action_type str required String identifier for the action ​ handler Callable required Async or sync function that handles the action ​ get_current_context method Get the current conversation context. Returns a list of messages in the current context, including system messages, user messages, and assistant responses. Show Returns ​ messages List[dict] List of messages in the current context Show Raises ​ FlowError If context aggregator is not available Example usage: Copy Ask AI # Access current conversation context context = flow_manager.get_current_context() # Use in handlers async def process_response ( args : FlowArgs) -> tuple[FlowResult, str ]: context = flow_manager.get_current_context() # Process conversation history return { "status" : "success" }, "next" ​ State Management The FlowManager provides a state dictionary for storing conversation data: Access state Access in transitions Copy Ask AI flow_manager.state: Dict[ str , Any] # Store data flow_manager.state[ "user_age" ] = 25 ​ Usage Examples Static Flow Dynamic Flow Copy Ask AI flow_config: FlowConfig = { "initial_node" : "greeting" , "nodes" : { "greeting" : { "role_messages" : [ { "role" : "system" , "content" : "You are a helpful assistant. Your responses will be converted to audio." } ], "task_messages" : [ { "role" : "system" , "content" : "Start by greeting the user and asking for their name." } ], "functions" : [{ "type" : "function" , "function" : { "name" : "collect_name" , "handler" : collect_name_handler, "description" : "Record user's name" , "parameters" : { ... } } }] } } } # Create and initialize the FlowManager flow_manager = FlowManager( task = task, llm = llm, context_aggregator = context_aggregator, flow_config = flow_config ) # Initialize the flow_manager to start the conversation await flow_manager.initialize() ​ Node Functions concept Functions that execute operations within a single conversational state, without switching nodes. Copy Ask AI Copy Ask AI from pipecat_flows import FlowArgs, FlowResult async def process_data ( args : FlowArgs) -> tuple[FlowResult, None ]: """Handle data processing within a node.""" data = args[ "data" ] result = await process(data) return { "status" : "success" , "processed_data" : result }, None # Function configuration { "type" : "function" , "function" : { "name" : "process_data" , "handler" : process_data, "description" : "Process user data" , "parameters" : { "type" : "object" , "properties" : { "data" : { "type" : "string" } } } } } ​ Edge Functions concept Functions that specify a transition between nodes (optionally processing data first). Copy Ask AI Copy Ask AI from pipecat_flows import FlowArgs, FlowResult async def next_step ( args : FlowArgs) -> tuple[ None , str ]: """Specify the next node to transition to.""" return None , "target_node" # Return NodeConfig instead of str for dynamic flows # Function configuration { "type" : "function" , "function" : { "name" : "next_step" , "handler" : next_step, "description" : "Transition to next node" , "parameters" : { "type" : "object" , "properties" : {}} } } ​ Function Properties ​ handler Optional[Callable] Async function that processes data within a node and/or specifies the next node ( more details here ). Can be specified as: Direct function reference Either a Callable function or a string with __function__: prefix (e.g., "__function__:process_data" ) to reference a function in the main script Direct Reference Function Token Copy Ask AI { "type" : "function" , "function" : { "name" : "process_data" , "handler" : process_data, # Callable function "parameters" : { ... } } } ​ transition_callback Optional[Callable] deprecated Handler for dynamic flow transitions. Deprecated: instead of transition_callback , use a “consolidated” handler that returns a tuple (result, next node). Must be an async function with one of these signatures: Copy Ask AI # New style (recommended) async def handle_transition ( args : Dict[ str , Any], result : FlowResult, flow_manager : FlowManager ) -> None : """Handle transition to next node.""" if result.available: # Type-safe access to result await flow_manager.set_node_from_config(create_confirmation_node()) else : await flow_manager.set_node_from_config( create_no_availability_node(result.alternative_times) ) # Legacy style (supported for backwards compatibility) async def handle_transition ( args : Dict[ str , Any], flow_manager : FlowManager ) -> None : """Handle transition to next node.""" await flow_manager.set_node_from_config(create_next_node()) The callback receives: args : Arguments from the function call result : Typed result from the function handler (new style only) flow_manager : Reference to the FlowManager instance Example usage: Copy Ask AI async def handle_availability_check ( args : Dict, result : TimeResult, # Typed result flow_manager : FlowManager ): """Handle availability check and transition based on result.""" if result.available: await flow_manager.set_node_from_config(create_confirmation_node()) else : await flow_manager.set_node_from_config( create_no_availability_node(result.alternative_times) ) # Use in function configuration { "type" : "function" , "function" : { "name" : "check_availability" , "handler" : check_availability, "parameters" : { ... }, "transition_callback" : handle_availability_check } } Note: A function cannot have both transition_to and transition_callback . ​ Handler Signatures Function handlers passed as a handler in a function configuration can be defined with three different signatures: Modern (Args + FlowManager) Legacy (Args Only) No Arguments Copy Ask AI async def handler_with_flow_manager ( args : FlowArgs, flow_manager : FlowManager) -> tuple[FlowResult, NodeConfig]: """Modern handler that receives both arguments and FlowManager access.""" # Access state previous_data = flow_manager.state.get( "stored_data" ) # Access pipeline resources await flow_manager.task.queue_frame(TTSSpeakFrame( "Processing your request..." )) # Store data in state for later flow_manager.state[ "new_data" ] = args[ "input" ] return { "status" : "success" , "result" : "Processed with flow access" }, create_next_node() The framework automatically detects which signature your handler is using and calls it appropriately. If you’re passing your function directly into your NodeConfig rather than as a handler in a function configuration, you’d use a somewhat different signature: Direct Copy Ask AI async def do_something ( flow_manager : FlowManager, foo : int , bar : str = "" ) -> tuple[FlowResult, NodeConfig]: """ Do something interesting. Args: foo (int): The foo to do something interesting with. bar (string): The bar to do something interesting with. Defaults to empty string. """ result = await fetch_data(foo, bar) next_node = create_end_node() return result, next_node ​ Return Types Success Response Error Response Copy Ask AI { "status" : "success" , "data" : "some data" # Optional additional data } ​ Provider-Specific Formats You don’t need to handle these format differences manually - use the standard format and the FlowManager will adapt it for your chosen provider. OpenAI Anthropic Google (Gemini) Copy Ask AI { "type" : "function" , "function" : { "name" : "function_name" , "handler" : handler, "description" : "Description" , "parameters" : { ... } } } ​ Actions pre_actions and post_actions are used to manage conversation flow. They are included in the NodeConfig and executed before and after the LLM completion, respectively. Three kinds of actions are available: Pre-canned actions: These actions perform common tasks and require little configuration. Function actions: These actions run developer-defined functions at the appropriate time. Custom actions: These are fully developer-defined actions, providing flexibility at the expense of complexity. ​ Pre-canned Actions Common actions shipped with Flows for managing conversation flow. To use them, just add them to your NodeConfig . ​ tts_say action Speaks text immediately using the TTS service. Copy Ask AI Copy Ask AI "pre_actions" : [ { "type" : "tts_say" , "text" : "Processing your request..." # Required } ] ​ end_conversation action Ends the conversation and closes the connection. Copy Ask AI Copy Ask AI "post_actions" : [ { "type" : "end_conversation" , "text" : "Goodbye!" # Optional farewell message } ] ​ Function Actions Actions that run developer-defined functions at the appropriate time. For example, if used in post_actions , they’ll run after the bot has finished talking and after any previous post_actions have finished. ​ function action Calls the developer-defined function at the appropriate time. Copy Ask AI Copy Ask AI "post_actions" : [ { "type" : "function" , "handler" : bot_turn_ended # Required } ] ​ Custom Actions Fully developer-defined actions, providing flexibility at the expense of complexity. Here’s the complexity: because these actions aren’t queued in the Pipecat pipeline, they may execute seemingly early if used in post_actions ; they’ll run immediately after the LLM completion is triggered but won’t wait around for the bot to finish talking. Why would you want this behavior? You might be writing an action that: Itself just queues another Frame into the Pipecat pipeline (meaning there would no benefit to waiting around for sequencing purposes) Does work that can be done a bit sooner, like logging that the LLM was updated Custom actions are composed of at least: ​ type str required String identifier for the action ​ handler Callable required Async or sync function that handles the action Example: Copy Ask AI Copy Ask AI # Define custom action handler async def custom_notification ( action : dict , flow_manager : FlowManager): """Custom action handler.""" message = action.get( "message" , "" ) await notify_user(message) # Use in node configuration "pre_actions" : [ { "type" : "notify" , "handler" : send_notification, "message" : "Attention!" , } ] ​ Exceptions ​ FlowError exception Base exception for all flow-related errors. Copy Ask AI Copy Ask AI from pipecat_flows import FlowError try : await flow_manager.set_node_from_config(config) except FlowError as e: print ( f "Flow error: { e } " ) ​ FlowInitializationError exception Raised when flow initialization fails. Copy Ask AI Copy Ask AI from pipecat_flows import FlowInitializationError try : await flow_manager.initialize() except FlowInitializationError as e: print ( f "Initialization failed: { e } " ) ​ FlowTransitionError exception Raised when a state transition fails. Copy Ask AI Copy Ask AI from pipecat_flows import FlowTransitionError try : await flow_manager.set_node_from_config(node_config) except FlowTransitionError as e: print ( f "Transition failed: { e } " ) ​ InvalidFunctionError exception Raised when an invalid or unavailable function is specified. Copy Ask AI Copy Ask AI from pipecat_flows import InvalidFunctionError try : await flow_manager.set_node_from_config({ "functions" : [{ "type" : "function" , "function" : { "name" : "invalid_function" } }] }) except InvalidFunctionError as e: print ( f "Invalid function: { e } " ) RTVI Observer PipelineParams On this page Installation Core Types FlowArgs FlowResult FlowConfig NodeConfig Function Handler Types ContextStrategy ContextStrategyConfig FlowsFunctionSchema FlowManager Methods State Management Usage Examples Function Properties Handler Signatures Return Types Provider-Specific Formats Actions Pre-canned Actions Function Actions Custom Actions Exceptions Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/flows_pipecat-flows_6d68771f.txt b/flows_pipecat-flows_6d68771f.txt
new file mode 100644
index 0000000000000000000000000000000000000000..8ce19b8d8f562db235c205ea336ad19b001ca91b
--- /dev/null
+++ b/flows_pipecat-flows_6d68771f.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/frameworks/flows/pipecat-flows#param-task-messages
+Title: Pipecat Flows - Pipecat
+==================================================
+
+Pipecat Flows - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Frameworks Pipecat Flows Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline New to building conversational flows? Check out our Pipecat Flows guide first. ​ Installation Existing Pipecat installation Fresh Pipecat installation Copy Ask AI pip install pipecat-ai-flows ​ Core Types ​ FlowArgs ​ FlowArgs Dict[str, Any] Type alias for function handler arguments. ​ FlowResult ​ FlowResult TypedDict Base type for function handler results. Additional fields can be included as needed. Show Fields ​ status str Optional status field ​ error str Optional error message ​ FlowConfig ​ FlowConfig TypedDict Configuration for the entire conversation flow. Show Fields ​ initial_node str required Starting node identifier ​ nodes Dict[str, NodeConfig] required Map of node names to configurations ​ NodeConfig ​ NodeConfig TypedDict Configuration for a single node in the flow. Show Fields ​ name str The name of the node, used in debug logging in dynamic flows. If no name is specified, an automatically-generated UUID is used. Copy Ask AI # Example name "name" : "greeting" ​ role_messages List[dict] Defines the role or persona of the LLM. Required for the initial node and optional for subsequent nodes. Copy Ask AI # Example role messages "role_messages" : [ { "role" : "system" , "content" : "You are a helpful assistant..." } ], ​ task_messages List[dict] required Defines the task for a given node. Required for all nodes. Copy Ask AI # Example task messages "task_messages" : [ { "role" : "system" , # May be `user` depending on the LLM "content" : "Ask the user for their name..." } ], ​ context_strategy ContextStrategyConfig Strategy for managing context during transitions to this node. Copy Ask AI # Example context strategy configuration "context_strategy" : ContextStrategyConfig( strategy = ContextStrategy. RESET_WITH_SUMMARY , summary_prompt = "Summarize the key points discussed so far." ) ​ functions List[Union[dict, FlowsFunctionSchema]] required LLM function / tool call configurations, defined in one of the supported formats . Copy Ask AI # Using provider-specific dictionary format "functions" : [ { "type" : "function" , "function" : { "name" : "get_current_movies" , "handler" : get_movies, "description" : "Fetch movies currently playing" , "parameters" : { ... } }, } ] # Using FlowsFunctionSchema "functions" : [ FlowsFunctionSchema( name = "get_current_movies" , description = "Fetch movies currently playing" , properties = { ... }, required = [ ... ], handler = get_movies ) ] # Using direct functions (auto-configuration) "functions" : [get_movies] ​ pre_actions List[dict] Actions that execute before the LLM inference. For example, you can send a message to the TTS to speak a phrase (e.g. “Hold on a moment…”), which may be effective if an LLM function call takes time to execute. Copy Ask AI # Example pre_actions "pre_actions" : [ { "type" : "tts_say" , "text" : "Hold on a moment..." } ], ​ post_actions List[dict] Actions that execute after the LLM inference. For example, you can end the conversation. Copy Ask AI # Example post_actions "post_actions" : [ { "type" : "end_conversation" } ] ​ respond_immediately bool If set to False , the LLM will not respond immediately when the node is set, but will instead wait for the user to speak first before responding. Defaults to True . Copy Ask AI # Example usage "respond_immediately" : False ​ Function Handler Types ​ LegacyFunctionHandler Callable[[FlowArgs], Awaitable[FlowResult | ConsolidatedFunctionResult]] Legacy function handler that only receives arguments. Returns either: A FlowResult (⚠️ deprecated) A “consolidated” result tuple (result, next node) where: result is an optional FlowResult next node is an optional NodeConfig (for dynamic flows) or string (for static flows) ​ FlowFunctionHandler Callable[[FlowArgs, FlowManager], Awaitable[FlowResult | ConsolidatedFunctionResult]] Modern function handler that receives both arguments and FlowManager . Returns either: A FlowResult (⚠️ deprecated) A “consolidated” result tuple (result, next node) where: result is an optional FlowResult next node is an optional NodeConfig (for dynamic flows) or string (for static flows) ​ DirectFunction DirectFunction Function that is meant to be passed directly into a NodeConfig rather than into the handler field of a function configuration. It must be an async function with flow_manager: FlowManager as its first parameter. It must return a ConsolidatedFunctionResult , which is a tuple (result, next node) where: result is an optional FlowResult next node is an optional NodeConfig (for dynamic flows) or string (for static flows) ​ ContextStrategy ​ ContextStrategy Enum Strategy for managing conversation context during node transitions. Show Values ​ APPEND str Default strategy. Adds new messages to existing context. ​ RESET str Clears context and starts fresh with new messages. ​ RESET_WITH_SUMMARY str Resets context but includes an AI-generated summary. ​ ContextStrategyConfig ​ ContextStrategyConfig dataclass Configuration for context management strategy. Show Fields ​ strategy ContextStrategy required The strategy to use for context management ​ summary_prompt Optional[str] Required when using RESET_WITH_SUMMARY. Prompt text for generating the conversation summary. Copy Ask AI # Example usage config = ContextStrategyConfig( strategy = ContextStrategy. RESET_WITH_SUMMARY , summary_prompt = "Summarize the key points discussed so far." ) ​ FlowsFunctionSchema ​ FlowsFunctionSchema class A standardized schema for defining functions in Pipecat Flows with flow-specific properties. Show Constructor Parameters ​ name str required Name of the function ​ description str required Description of the function’s purpose ​ properties Dict[str, Any] required Dictionary defining properties types and descriptions ​ required List[str] required List of required parameter names ​ handler Optional[FunctionHandler] Function handler to process the function call ​ transition_to Optional[str] deprecated Target node to transition to after function execution Deprecated: instead of transition_to , use a “consolidated” handler that returns a tuple (result, next node). ​ transition_callback Optional[Callable] deprecated Callback function for dynamic transitions Deprecated: instead of transition_callback , use a “consolidated” handler that returns a tuple (result, next node). You cannot specify both transition_to and transition_callback in the same function schema. Example usage: Copy Ask AI from pipecat_flows import FlowsFunctionSchema # Define a function schema collect_name_schema = FlowsFunctionSchema( name = "collect_name" , description = "Record the user's name" , properties = { "name" : { "type" : "string" , "description" : "The user's name" } }, required = [ "name" ], handler = collect_name_handler ) # Use in node configuration node_config = { "name" : "greeting" , "task_messages" : [ { "role" : "system" , "content" : "Ask the user for their name." } ], "functions" : [collect_name_schema] } # Pass to flow manager await flow_manager.set_node_from_config(node_config) ​ FlowManager ​ FlowManager class Main class for managing conversation flows, supporting both static (configuration-driven) and dynamic (runtime-determined) flows. Show Constructor Parameters ​ task PipelineTask required Pipeline task for frame queueing ​ llm LLMService required LLM service instance (OpenAI, Anthropic, or Google). Must be initialized with the corresponding pipecat-ai provider dependency installed. ​ context_aggregator Any required Context aggregator used for pushing messages to the LLM service ​ tts Optional[Any] deprecated Optional TTS service for voice actions. Deprecated: No need to explicitly pass tts to FlowManager in order to use tts_say actions. ​ flow_config Optional[FlowConfig] Optional static flow configuration ​ context_strategy Optional[ContextStrategyConfig] Optional configuration for how context should be managed during transitions. Defaults to APPEND strategy if not specified. ​ Methods ​ initialize method Initialize the flow with starting messages. Show Parameters ​ initial_node NodeConfig The initial conversation node (needed for dynamic flows only). If not specified, you’ll need to call set_node_from_config() to kick off the conversation. Show Raises ​ FlowInitializationError If initialization fails ​ set_node method deprecated Set up a new conversation node programmatically (dynamic flows only). In dynamic flows, the application advances the conversation using set_node to set up each next node. In static flows, set_node is triggered under the hood when a node contains a transition_to field. Deprecated: use the following patterns instead of set_node : Prefer “consolidated” function handlers that return a tuple (result, next node), which implicitly sets up the next node Prefer passing your initial node to FlowManager.initialize() If you really need to set a node explicitly, use set_node_from_config() (note: its name will be read from its NodeConfig ) Show Parameters ​ node_id str required Identifier for the new node ​ node_config NodeConfig required Node configuration including messages, functions, and actions Show Raises ​ FlowError If node setup fails ​ set_node_from_config method Set up a new conversation node programmatically (dynamic flows only). Note that this method should only be used in rare circumstances. Most often, you should: Prefer “consolidated” function handlers that return a tuple (result, next node), which implicitly sets up the next node Prefer passing your initial node to FlowManager.initialize() Show Parameters ​ node_config NodeConfig required Node configuration including messages, functions, and actions Show Raises ​ FlowError If node setup fails ​ register_action method Register a handler for a custom action type. Show Parameters ​ action_type str required String identifier for the action ​ handler Callable required Async or sync function that handles the action ​ get_current_context method Get the current conversation context. Returns a list of messages in the current context, including system messages, user messages, and assistant responses. Show Returns ​ messages List[dict] List of messages in the current context Show Raises ​ FlowError If context aggregator is not available Example usage: Copy Ask AI # Access current conversation context context = flow_manager.get_current_context() # Use in handlers async def process_response ( args : FlowArgs) -> tuple[FlowResult, str ]: context = flow_manager.get_current_context() # Process conversation history return { "status" : "success" }, "next" ​ State Management The FlowManager provides a state dictionary for storing conversation data: Access state Access in transitions Copy Ask AI flow_manager.state: Dict[ str , Any] # Store data flow_manager.state[ "user_age" ] = 25 ​ Usage Examples Static Flow Dynamic Flow Copy Ask AI flow_config: FlowConfig = { "initial_node" : "greeting" , "nodes" : { "greeting" : { "role_messages" : [ { "role" : "system" , "content" : "You are a helpful assistant. Your responses will be converted to audio." } ], "task_messages" : [ { "role" : "system" , "content" : "Start by greeting the user and asking for their name." } ], "functions" : [{ "type" : "function" , "function" : { "name" : "collect_name" , "handler" : collect_name_handler, "description" : "Record user's name" , "parameters" : { ... } } }] } } } # Create and initialize the FlowManager flow_manager = FlowManager( task = task, llm = llm, context_aggregator = context_aggregator, flow_config = flow_config ) # Initialize the flow_manager to start the conversation await flow_manager.initialize() ​ Node Functions concept Functions that execute operations within a single conversational state, without switching nodes. Copy Ask AI Copy Ask AI from pipecat_flows import FlowArgs, FlowResult async def process_data ( args : FlowArgs) -> tuple[FlowResult, None ]: """Handle data processing within a node.""" data = args[ "data" ] result = await process(data) return { "status" : "success" , "processed_data" : result }, None # Function configuration { "type" : "function" , "function" : { "name" : "process_data" , "handler" : process_data, "description" : "Process user data" , "parameters" : { "type" : "object" , "properties" : { "data" : { "type" : "string" } } } } } ​ Edge Functions concept Functions that specify a transition between nodes (optionally processing data first). Copy Ask AI Copy Ask AI from pipecat_flows import FlowArgs, FlowResult async def next_step ( args : FlowArgs) -> tuple[ None , str ]: """Specify the next node to transition to.""" return None , "target_node" # Return NodeConfig instead of str for dynamic flows # Function configuration { "type" : "function" , "function" : { "name" : "next_step" , "handler" : next_step, "description" : "Transition to next node" , "parameters" : { "type" : "object" , "properties" : {}} } } ​ Function Properties ​ handler Optional[Callable] Async function that processes data within a node and/or specifies the next node ( more details here ). Can be specified as: Direct function reference Either a Callable function or a string with __function__: prefix (e.g., "__function__:process_data" ) to reference a function in the main script Direct Reference Function Token Copy Ask AI { "type" : "function" , "function" : { "name" : "process_data" , "handler" : process_data, # Callable function "parameters" : { ... } } } ​ transition_callback Optional[Callable] deprecated Handler for dynamic flow transitions. Deprecated: instead of transition_callback , use a “consolidated” handler that returns a tuple (result, next node). Must be an async function with one of these signatures: Copy Ask AI # New style (recommended) async def handle_transition ( args : Dict[ str , Any], result : FlowResult, flow_manager : FlowManager ) -> None : """Handle transition to next node.""" if result.available: # Type-safe access to result await flow_manager.set_node_from_config(create_confirmation_node()) else : await flow_manager.set_node_from_config( create_no_availability_node(result.alternative_times) ) # Legacy style (supported for backwards compatibility) async def handle_transition ( args : Dict[ str , Any], flow_manager : FlowManager ) -> None : """Handle transition to next node.""" await flow_manager.set_node_from_config(create_next_node()) The callback receives: args : Arguments from the function call result : Typed result from the function handler (new style only) flow_manager : Reference to the FlowManager instance Example usage: Copy Ask AI async def handle_availability_check ( args : Dict, result : TimeResult, # Typed result flow_manager : FlowManager ): """Handle availability check and transition based on result.""" if result.available: await flow_manager.set_node_from_config(create_confirmation_node()) else : await flow_manager.set_node_from_config( create_no_availability_node(result.alternative_times) ) # Use in function configuration { "type" : "function" , "function" : { "name" : "check_availability" , "handler" : check_availability, "parameters" : { ... }, "transition_callback" : handle_availability_check } } Note: A function cannot have both transition_to and transition_callback . ​ Handler Signatures Function handlers passed as a handler in a function configuration can be defined with three different signatures: Modern (Args + FlowManager) Legacy (Args Only) No Arguments Copy Ask AI async def handler_with_flow_manager ( args : FlowArgs, flow_manager : FlowManager) -> tuple[FlowResult, NodeConfig]: """Modern handler that receives both arguments and FlowManager access.""" # Access state previous_data = flow_manager.state.get( "stored_data" ) # Access pipeline resources await flow_manager.task.queue_frame(TTSSpeakFrame( "Processing your request..." )) # Store data in state for later flow_manager.state[ "new_data" ] = args[ "input" ] return { "status" : "success" , "result" : "Processed with flow access" }, create_next_node() The framework automatically detects which signature your handler is using and calls it appropriately. If you’re passing your function directly into your NodeConfig rather than as a handler in a function configuration, you’d use a somewhat different signature: Direct Copy Ask AI async def do_something ( flow_manager : FlowManager, foo : int , bar : str = "" ) -> tuple[FlowResult, NodeConfig]: """ Do something interesting. Args: foo (int): The foo to do something interesting with. bar (string): The bar to do something interesting with. Defaults to empty string. """ result = await fetch_data(foo, bar) next_node = create_end_node() return result, next_node ​ Return Types Success Response Error Response Copy Ask AI { "status" : "success" , "data" : "some data" # Optional additional data } ​ Provider-Specific Formats You don’t need to handle these format differences manually - use the standard format and the FlowManager will adapt it for your chosen provider. OpenAI Anthropic Google (Gemini) Copy Ask AI { "type" : "function" , "function" : { "name" : "function_name" , "handler" : handler, "description" : "Description" , "parameters" : { ... } } } ​ Actions pre_actions and post_actions are used to manage conversation flow. They are included in the NodeConfig and executed before and after the LLM completion, respectively. Three kinds of actions are available: Pre-canned actions: These actions perform common tasks and require little configuration. Function actions: These actions run developer-defined functions at the appropriate time. Custom actions: These are fully developer-defined actions, providing flexibility at the expense of complexity. ​ Pre-canned Actions Common actions shipped with Flows for managing conversation flow. To use them, just add them to your NodeConfig . ​ tts_say action Speaks text immediately using the TTS service. Copy Ask AI Copy Ask AI "pre_actions" : [ { "type" : "tts_say" , "text" : "Processing your request..." # Required } ] ​ end_conversation action Ends the conversation and closes the connection. Copy Ask AI Copy Ask AI "post_actions" : [ { "type" : "end_conversation" , "text" : "Goodbye!" # Optional farewell message } ] ​ Function Actions Actions that run developer-defined functions at the appropriate time. For example, if used in post_actions , they’ll run after the bot has finished talking and after any previous post_actions have finished. ​ function action Calls the developer-defined function at the appropriate time. Copy Ask AI Copy Ask AI "post_actions" : [ { "type" : "function" , "handler" : bot_turn_ended # Required } ] ​ Custom Actions Fully developer-defined actions, providing flexibility at the expense of complexity. Here’s the complexity: because these actions aren’t queued in the Pipecat pipeline, they may execute seemingly early if used in post_actions ; they’ll run immediately after the LLM completion is triggered but won’t wait around for the bot to finish talking. Why would you want this behavior? You might be writing an action that: Itself just queues another Frame into the Pipecat pipeline (meaning there would no benefit to waiting around for sequencing purposes) Does work that can be done a bit sooner, like logging that the LLM was updated Custom actions are composed of at least: ​ type str required String identifier for the action ​ handler Callable required Async or sync function that handles the action Example: Copy Ask AI Copy Ask AI # Define custom action handler async def custom_notification ( action : dict , flow_manager : FlowManager): """Custom action handler.""" message = action.get( "message" , "" ) await notify_user(message) # Use in node configuration "pre_actions" : [ { "type" : "notify" , "handler" : send_notification, "message" : "Attention!" , } ] ​ Exceptions ​ FlowError exception Base exception for all flow-related errors. Copy Ask AI Copy Ask AI from pipecat_flows import FlowError try : await flow_manager.set_node_from_config(config) except FlowError as e: print ( f "Flow error: { e } " ) ​ FlowInitializationError exception Raised when flow initialization fails. Copy Ask AI Copy Ask AI from pipecat_flows import FlowInitializationError try : await flow_manager.initialize() except FlowInitializationError as e: print ( f "Initialization failed: { e } " ) ​ FlowTransitionError exception Raised when a state transition fails. Copy Ask AI Copy Ask AI from pipecat_flows import FlowTransitionError try : await flow_manager.set_node_from_config(node_config) except FlowTransitionError as e: print ( f "Transition failed: { e } " ) ​ InvalidFunctionError exception Raised when an invalid or unavailable function is specified. Copy Ask AI Copy Ask AI from pipecat_flows import InvalidFunctionError try : await flow_manager.set_node_from_config({ "functions" : [{ "type" : "function" , "function" : { "name" : "invalid_function" } }] }) except InvalidFunctionError as e: print ( f "Invalid function: { e } " ) RTVI Observer PipelineParams On this page Installation Core Types FlowArgs FlowResult FlowConfig NodeConfig Function Handler Types ContextStrategy ContextStrategyConfig FlowsFunctionSchema FlowManager Methods State Management Usage Examples Function Properties Handler Signatures Return Types Provider-Specific Formats Actions Pre-canned Actions Function Actions Custom Actions Exceptions Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/flows_pipecat-flows_923f99a4.txt b/flows_pipecat-flows_923f99a4.txt
new file mode 100644
index 0000000000000000000000000000000000000000..1684e8f4a0d9cbb593e56cf2591540c47cb5e780
--- /dev/null
+++ b/flows_pipecat-flows_923f99a4.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/frameworks/flows/pipecat-flows#param-flow-function-handler
+Title: Pipecat Flows - Pipecat
+==================================================
+
+Pipecat Flows - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Frameworks Pipecat Flows Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline New to building conversational flows? Check out our Pipecat Flows guide first. ​ Installation Existing Pipecat installation Fresh Pipecat installation Copy Ask AI pip install pipecat-ai-flows ​ Core Types ​ FlowArgs ​ FlowArgs Dict[str, Any] Type alias for function handler arguments. ​ FlowResult ​ FlowResult TypedDict Base type for function handler results. Additional fields can be included as needed. Show Fields ​ status str Optional status field ​ error str Optional error message ​ FlowConfig ​ FlowConfig TypedDict Configuration for the entire conversation flow. Show Fields ​ initial_node str required Starting node identifier ​ nodes Dict[str, NodeConfig] required Map of node names to configurations ​ NodeConfig ​ NodeConfig TypedDict Configuration for a single node in the flow. Show Fields ​ name str The name of the node, used in debug logging in dynamic flows. If no name is specified, an automatically-generated UUID is used. Copy Ask AI # Example name "name" : "greeting" ​ role_messages List[dict] Defines the role or persona of the LLM. Required for the initial node and optional for subsequent nodes. Copy Ask AI # Example role messages "role_messages" : [ { "role" : "system" , "content" : "You are a helpful assistant..." } ], ​ task_messages List[dict] required Defines the task for a given node. Required for all nodes. Copy Ask AI # Example task messages "task_messages" : [ { "role" : "system" , # May be `user` depending on the LLM "content" : "Ask the user for their name..." } ], ​ context_strategy ContextStrategyConfig Strategy for managing context during transitions to this node. Copy Ask AI # Example context strategy configuration "context_strategy" : ContextStrategyConfig( strategy = ContextStrategy. RESET_WITH_SUMMARY , summary_prompt = "Summarize the key points discussed so far." ) ​ functions List[Union[dict, FlowsFunctionSchema]] required LLM function / tool call configurations, defined in one of the supported formats . Copy Ask AI # Using provider-specific dictionary format "functions" : [ { "type" : "function" , "function" : { "name" : "get_current_movies" , "handler" : get_movies, "description" : "Fetch movies currently playing" , "parameters" : { ... } }, } ] # Using FlowsFunctionSchema "functions" : [ FlowsFunctionSchema( name = "get_current_movies" , description = "Fetch movies currently playing" , properties = { ... }, required = [ ... ], handler = get_movies ) ] # Using direct functions (auto-configuration) "functions" : [get_movies] ​ pre_actions List[dict] Actions that execute before the LLM inference. For example, you can send a message to the TTS to speak a phrase (e.g. “Hold on a moment…”), which may be effective if an LLM function call takes time to execute. Copy Ask AI # Example pre_actions "pre_actions" : [ { "type" : "tts_say" , "text" : "Hold on a moment..." } ], ​ post_actions List[dict] Actions that execute after the LLM inference. For example, you can end the conversation. Copy Ask AI # Example post_actions "post_actions" : [ { "type" : "end_conversation" } ] ​ respond_immediately bool If set to False , the LLM will not respond immediately when the node is set, but will instead wait for the user to speak first before responding. Defaults to True . Copy Ask AI # Example usage "respond_immediately" : False ​ Function Handler Types ​ LegacyFunctionHandler Callable[[FlowArgs], Awaitable[FlowResult | ConsolidatedFunctionResult]] Legacy function handler that only receives arguments. Returns either: A FlowResult (⚠️ deprecated) A “consolidated” result tuple (result, next node) where: result is an optional FlowResult next node is an optional NodeConfig (for dynamic flows) or string (for static flows) ​ FlowFunctionHandler Callable[[FlowArgs, FlowManager], Awaitable[FlowResult | ConsolidatedFunctionResult]] Modern function handler that receives both arguments and FlowManager . Returns either: A FlowResult (⚠️ deprecated) A “consolidated” result tuple (result, next node) where: result is an optional FlowResult next node is an optional NodeConfig (for dynamic flows) or string (for static flows) ​ DirectFunction DirectFunction Function that is meant to be passed directly into a NodeConfig rather than into the handler field of a function configuration. It must be an async function with flow_manager: FlowManager as its first parameter. It must return a ConsolidatedFunctionResult , which is a tuple (result, next node) where: result is an optional FlowResult next node is an optional NodeConfig (for dynamic flows) or string (for static flows) ​ ContextStrategy ​ ContextStrategy Enum Strategy for managing conversation context during node transitions. Show Values ​ APPEND str Default strategy. Adds new messages to existing context. ​ RESET str Clears context and starts fresh with new messages. ​ RESET_WITH_SUMMARY str Resets context but includes an AI-generated summary. ​ ContextStrategyConfig ​ ContextStrategyConfig dataclass Configuration for context management strategy. Show Fields ​ strategy ContextStrategy required The strategy to use for context management ​ summary_prompt Optional[str] Required when using RESET_WITH_SUMMARY. Prompt text for generating the conversation summary. Copy Ask AI # Example usage config = ContextStrategyConfig( strategy = ContextStrategy. RESET_WITH_SUMMARY , summary_prompt = "Summarize the key points discussed so far." ) ​ FlowsFunctionSchema ​ FlowsFunctionSchema class A standardized schema for defining functions in Pipecat Flows with flow-specific properties. Show Constructor Parameters ​ name str required Name of the function ​ description str required Description of the function’s purpose ​ properties Dict[str, Any] required Dictionary defining properties types and descriptions ​ required List[str] required List of required parameter names ​ handler Optional[FunctionHandler] Function handler to process the function call ​ transition_to Optional[str] deprecated Target node to transition to after function execution Deprecated: instead of transition_to , use a “consolidated” handler that returns a tuple (result, next node). ​ transition_callback Optional[Callable] deprecated Callback function for dynamic transitions Deprecated: instead of transition_callback , use a “consolidated” handler that returns a tuple (result, next node). You cannot specify both transition_to and transition_callback in the same function schema. Example usage: Copy Ask AI from pipecat_flows import FlowsFunctionSchema # Define a function schema collect_name_schema = FlowsFunctionSchema( name = "collect_name" , description = "Record the user's name" , properties = { "name" : { "type" : "string" , "description" : "The user's name" } }, required = [ "name" ], handler = collect_name_handler ) # Use in node configuration node_config = { "name" : "greeting" , "task_messages" : [ { "role" : "system" , "content" : "Ask the user for their name." } ], "functions" : [collect_name_schema] } # Pass to flow manager await flow_manager.set_node_from_config(node_config) ​ FlowManager ​ FlowManager class Main class for managing conversation flows, supporting both static (configuration-driven) and dynamic (runtime-determined) flows. Show Constructor Parameters ​ task PipelineTask required Pipeline task for frame queueing ​ llm LLMService required LLM service instance (OpenAI, Anthropic, or Google). Must be initialized with the corresponding pipecat-ai provider dependency installed. ​ context_aggregator Any required Context aggregator used for pushing messages to the LLM service ​ tts Optional[Any] deprecated Optional TTS service for voice actions. Deprecated: No need to explicitly pass tts to FlowManager in order to use tts_say actions. ​ flow_config Optional[FlowConfig] Optional static flow configuration ​ context_strategy Optional[ContextStrategyConfig] Optional configuration for how context should be managed during transitions. Defaults to APPEND strategy if not specified. ​ Methods ​ initialize method Initialize the flow with starting messages. Show Parameters ​ initial_node NodeConfig The initial conversation node (needed for dynamic flows only). If not specified, you’ll need to call set_node_from_config() to kick off the conversation. Show Raises ​ FlowInitializationError If initialization fails ​ set_node method deprecated Set up a new conversation node programmatically (dynamic flows only). In dynamic flows, the application advances the conversation using set_node to set up each next node. In static flows, set_node is triggered under the hood when a node contains a transition_to field. Deprecated: use the following patterns instead of set_node : Prefer “consolidated” function handlers that return a tuple (result, next node), which implicitly sets up the next node Prefer passing your initial node to FlowManager.initialize() If you really need to set a node explicitly, use set_node_from_config() (note: its name will be read from its NodeConfig ) Show Parameters ​ node_id str required Identifier for the new node ​ node_config NodeConfig required Node configuration including messages, functions, and actions Show Raises ​ FlowError If node setup fails ​ set_node_from_config method Set up a new conversation node programmatically (dynamic flows only). Note that this method should only be used in rare circumstances. Most often, you should: Prefer “consolidated” function handlers that return a tuple (result, next node), which implicitly sets up the next node Prefer passing your initial node to FlowManager.initialize() Show Parameters ​ node_config NodeConfig required Node configuration including messages, functions, and actions Show Raises ​ FlowError If node setup fails ​ register_action method Register a handler for a custom action type. Show Parameters ​ action_type str required String identifier for the action ​ handler Callable required Async or sync function that handles the action ​ get_current_context method Get the current conversation context. Returns a list of messages in the current context, including system messages, user messages, and assistant responses. Show Returns ​ messages List[dict] List of messages in the current context Show Raises ​ FlowError If context aggregator is not available Example usage: Copy Ask AI # Access current conversation context context = flow_manager.get_current_context() # Use in handlers async def process_response ( args : FlowArgs) -> tuple[FlowResult, str ]: context = flow_manager.get_current_context() # Process conversation history return { "status" : "success" }, "next" ​ State Management The FlowManager provides a state dictionary for storing conversation data: Access state Access in transitions Copy Ask AI flow_manager.state: Dict[ str , Any] # Store data flow_manager.state[ "user_age" ] = 25 ​ Usage Examples Static Flow Dynamic Flow Copy Ask AI flow_config: FlowConfig = { "initial_node" : "greeting" , "nodes" : { "greeting" : { "role_messages" : [ { "role" : "system" , "content" : "You are a helpful assistant. Your responses will be converted to audio." } ], "task_messages" : [ { "role" : "system" , "content" : "Start by greeting the user and asking for their name." } ], "functions" : [{ "type" : "function" , "function" : { "name" : "collect_name" , "handler" : collect_name_handler, "description" : "Record user's name" , "parameters" : { ... } } }] } } } # Create and initialize the FlowManager flow_manager = FlowManager( task = task, llm = llm, context_aggregator = context_aggregator, flow_config = flow_config ) # Initialize the flow_manager to start the conversation await flow_manager.initialize() ​ Node Functions concept Functions that execute operations within a single conversational state, without switching nodes. Copy Ask AI Copy Ask AI from pipecat_flows import FlowArgs, FlowResult async def process_data ( args : FlowArgs) -> tuple[FlowResult, None ]: """Handle data processing within a node.""" data = args[ "data" ] result = await process(data) return { "status" : "success" , "processed_data" : result }, None # Function configuration { "type" : "function" , "function" : { "name" : "process_data" , "handler" : process_data, "description" : "Process user data" , "parameters" : { "type" : "object" , "properties" : { "data" : { "type" : "string" } } } } } ​ Edge Functions concept Functions that specify a transition between nodes (optionally processing data first). Copy Ask AI Copy Ask AI from pipecat_flows import FlowArgs, FlowResult async def next_step ( args : FlowArgs) -> tuple[ None , str ]: """Specify the next node to transition to.""" return None , "target_node" # Return NodeConfig instead of str for dynamic flows # Function configuration { "type" : "function" , "function" : { "name" : "next_step" , "handler" : next_step, "description" : "Transition to next node" , "parameters" : { "type" : "object" , "properties" : {}} } } ​ Function Properties ​ handler Optional[Callable] Async function that processes data within a node and/or specifies the next node ( more details here ). Can be specified as: Direct function reference Either a Callable function or a string with __function__: prefix (e.g., "__function__:process_data" ) to reference a function in the main script Direct Reference Function Token Copy Ask AI { "type" : "function" , "function" : { "name" : "process_data" , "handler" : process_data, # Callable function "parameters" : { ... } } } ​ transition_callback Optional[Callable] deprecated Handler for dynamic flow transitions. Deprecated: instead of transition_callback , use a “consolidated” handler that returns a tuple (result, next node). Must be an async function with one of these signatures: Copy Ask AI # New style (recommended) async def handle_transition ( args : Dict[ str , Any], result : FlowResult, flow_manager : FlowManager ) -> None : """Handle transition to next node.""" if result.available: # Type-safe access to result await flow_manager.set_node_from_config(create_confirmation_node()) else : await flow_manager.set_node_from_config( create_no_availability_node(result.alternative_times) ) # Legacy style (supported for backwards compatibility) async def handle_transition ( args : Dict[ str , Any], flow_manager : FlowManager ) -> None : """Handle transition to next node.""" await flow_manager.set_node_from_config(create_next_node()) The callback receives: args : Arguments from the function call result : Typed result from the function handler (new style only) flow_manager : Reference to the FlowManager instance Example usage: Copy Ask AI async def handle_availability_check ( args : Dict, result : TimeResult, # Typed result flow_manager : FlowManager ): """Handle availability check and transition based on result.""" if result.available: await flow_manager.set_node_from_config(create_confirmation_node()) else : await flow_manager.set_node_from_config( create_no_availability_node(result.alternative_times) ) # Use in function configuration { "type" : "function" , "function" : { "name" : "check_availability" , "handler" : check_availability, "parameters" : { ... }, "transition_callback" : handle_availability_check } } Note: A function cannot have both transition_to and transition_callback . ​ Handler Signatures Function handlers passed as a handler in a function configuration can be defined with three different signatures: Modern (Args + FlowManager) Legacy (Args Only) No Arguments Copy Ask AI async def handler_with_flow_manager ( args : FlowArgs, flow_manager : FlowManager) -> tuple[FlowResult, NodeConfig]: """Modern handler that receives both arguments and FlowManager access.""" # Access state previous_data = flow_manager.state.get( "stored_data" ) # Access pipeline resources await flow_manager.task.queue_frame(TTSSpeakFrame( "Processing your request..." )) # Store data in state for later flow_manager.state[ "new_data" ] = args[ "input" ] return { "status" : "success" , "result" : "Processed with flow access" }, create_next_node() The framework automatically detects which signature your handler is using and calls it appropriately. If you’re passing your function directly into your NodeConfig rather than as a handler in a function configuration, you’d use a somewhat different signature: Direct Copy Ask AI async def do_something ( flow_manager : FlowManager, foo : int , bar : str = "" ) -> tuple[FlowResult, NodeConfig]: """ Do something interesting. Args: foo (int): The foo to do something interesting with. bar (string): The bar to do something interesting with. Defaults to empty string. """ result = await fetch_data(foo, bar) next_node = create_end_node() return result, next_node ​ Return Types Success Response Error Response Copy Ask AI { "status" : "success" , "data" : "some data" # Optional additional data } ​ Provider-Specific Formats You don’t need to handle these format differences manually - use the standard format and the FlowManager will adapt it for your chosen provider. OpenAI Anthropic Google (Gemini) Copy Ask AI { "type" : "function" , "function" : { "name" : "function_name" , "handler" : handler, "description" : "Description" , "parameters" : { ... } } } ​ Actions pre_actions and post_actions are used to manage conversation flow. They are included in the NodeConfig and executed before and after the LLM completion, respectively. Three kinds of actions are available: Pre-canned actions: These actions perform common tasks and require little configuration. Function actions: These actions run developer-defined functions at the appropriate time. Custom actions: These are fully developer-defined actions, providing flexibility at the expense of complexity. ​ Pre-canned Actions Common actions shipped with Flows for managing conversation flow. To use them, just add them to your NodeConfig . ​ tts_say action Speaks text immediately using the TTS service. Copy Ask AI Copy Ask AI "pre_actions" : [ { "type" : "tts_say" , "text" : "Processing your request..." # Required } ] ​ end_conversation action Ends the conversation and closes the connection. Copy Ask AI Copy Ask AI "post_actions" : [ { "type" : "end_conversation" , "text" : "Goodbye!" # Optional farewell message } ] ​ Function Actions Actions that run developer-defined functions at the appropriate time. For example, if used in post_actions , they’ll run after the bot has finished talking and after any previous post_actions have finished. ​ function action Calls the developer-defined function at the appropriate time. Copy Ask AI Copy Ask AI "post_actions" : [ { "type" : "function" , "handler" : bot_turn_ended # Required } ] ​ Custom Actions Fully developer-defined actions, providing flexibility at the expense of complexity. Here’s the complexity: because these actions aren’t queued in the Pipecat pipeline, they may execute seemingly early if used in post_actions ; they’ll run immediately after the LLM completion is triggered but won’t wait around for the bot to finish talking. Why would you want this behavior? You might be writing an action that: Itself just queues another Frame into the Pipecat pipeline (meaning there would no benefit to waiting around for sequencing purposes) Does work that can be done a bit sooner, like logging that the LLM was updated Custom actions are composed of at least: ​ type str required String identifier for the action ​ handler Callable required Async or sync function that handles the action Example: Copy Ask AI Copy Ask AI # Define custom action handler async def custom_notification ( action : dict , flow_manager : FlowManager): """Custom action handler.""" message = action.get( "message" , "" ) await notify_user(message) # Use in node configuration "pre_actions" : [ { "type" : "notify" , "handler" : send_notification, "message" : "Attention!" , } ] ​ Exceptions ​ FlowError exception Base exception for all flow-related errors. Copy Ask AI Copy Ask AI from pipecat_flows import FlowError try : await flow_manager.set_node_from_config(config) except FlowError as e: print ( f "Flow error: { e } " ) ​ FlowInitializationError exception Raised when flow initialization fails. Copy Ask AI Copy Ask AI from pipecat_flows import FlowInitializationError try : await flow_manager.initialize() except FlowInitializationError as e: print ( f "Initialization failed: { e } " ) ​ FlowTransitionError exception Raised when a state transition fails. Copy Ask AI Copy Ask AI from pipecat_flows import FlowTransitionError try : await flow_manager.set_node_from_config(node_config) except FlowTransitionError as e: print ( f "Transition failed: { e } " ) ​ InvalidFunctionError exception Raised when an invalid or unavailable function is specified. Copy Ask AI Copy Ask AI from pipecat_flows import InvalidFunctionError try : await flow_manager.set_node_from_config({ "functions" : [{ "type" : "function" , "function" : { "name" : "invalid_function" } }] }) except InvalidFunctionError as e: print ( f "Invalid function: { e } " ) RTVI Observer PipelineParams On this page Installation Core Types FlowArgs FlowResult FlowConfig NodeConfig Function Handler Types ContextStrategy ContextStrategyConfig FlowsFunctionSchema FlowManager Methods State Management Usage Examples Function Properties Handler Signatures Return Types Provider-Specific Formats Actions Pre-canned Actions Function Actions Custom Actions Exceptions Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/flows_pipecat-flows_9c4b8695.txt b/flows_pipecat-flows_9c4b8695.txt
new file mode 100644
index 0000000000000000000000000000000000000000..aa1abe55040cdff62879359283e25e3b57a3b850
--- /dev/null
+++ b/flows_pipecat-flows_9c4b8695.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/frameworks/flows/pipecat-flows#param-get-current-context
+Title: Pipecat Flows - Pipecat
+==================================================
+
+Pipecat Flows - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Frameworks Pipecat Flows Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline New to building conversational flows? Check out our Pipecat Flows guide first. ​ Installation Existing Pipecat installation Fresh Pipecat installation Copy Ask AI pip install pipecat-ai-flows ​ Core Types ​ FlowArgs ​ FlowArgs Dict[str, Any] Type alias for function handler arguments. ​ FlowResult ​ FlowResult TypedDict Base type for function handler results. Additional fields can be included as needed. Show Fields ​ status str Optional status field ​ error str Optional error message ​ FlowConfig ​ FlowConfig TypedDict Configuration for the entire conversation flow. Show Fields ​ initial_node str required Starting node identifier ​ nodes Dict[str, NodeConfig] required Map of node names to configurations ​ NodeConfig ​ NodeConfig TypedDict Configuration for a single node in the flow. Show Fields ​ name str The name of the node, used in debug logging in dynamic flows. If no name is specified, an automatically-generated UUID is used. Copy Ask AI # Example name "name" : "greeting" ​ role_messages List[dict] Defines the role or persona of the LLM. Required for the initial node and optional for subsequent nodes. Copy Ask AI # Example role messages "role_messages" : [ { "role" : "system" , "content" : "You are a helpful assistant..." } ], ​ task_messages List[dict] required Defines the task for a given node. Required for all nodes. Copy Ask AI # Example task messages "task_messages" : [ { "role" : "system" , # May be `user` depending on the LLM "content" : "Ask the user for their name..." } ], ​ context_strategy ContextStrategyConfig Strategy for managing context during transitions to this node. Copy Ask AI # Example context strategy configuration "context_strategy" : ContextStrategyConfig( strategy = ContextStrategy. RESET_WITH_SUMMARY , summary_prompt = "Summarize the key points discussed so far." ) ​ functions List[Union[dict, FlowsFunctionSchema]] required LLM function / tool call configurations, defined in one of the supported formats . Copy Ask AI # Using provider-specific dictionary format "functions" : [ { "type" : "function" , "function" : { "name" : "get_current_movies" , "handler" : get_movies, "description" : "Fetch movies currently playing" , "parameters" : { ... } }, } ] # Using FlowsFunctionSchema "functions" : [ FlowsFunctionSchema( name = "get_current_movies" , description = "Fetch movies currently playing" , properties = { ... }, required = [ ... ], handler = get_movies ) ] # Using direct functions (auto-configuration) "functions" : [get_movies] ​ pre_actions List[dict] Actions that execute before the LLM inference. For example, you can send a message to the TTS to speak a phrase (e.g. “Hold on a moment…”), which may be effective if an LLM function call takes time to execute. Copy Ask AI # Example pre_actions "pre_actions" : [ { "type" : "tts_say" , "text" : "Hold on a moment..." } ], ​ post_actions List[dict] Actions that execute after the LLM inference. For example, you can end the conversation. Copy Ask AI # Example post_actions "post_actions" : [ { "type" : "end_conversation" } ] ​ respond_immediately bool If set to False , the LLM will not respond immediately when the node is set, but will instead wait for the user to speak first before responding. Defaults to True . Copy Ask AI # Example usage "respond_immediately" : False ​ Function Handler Types ​ LegacyFunctionHandler Callable[[FlowArgs], Awaitable[FlowResult | ConsolidatedFunctionResult]] Legacy function handler that only receives arguments. Returns either: A FlowResult (⚠️ deprecated) A “consolidated” result tuple (result, next node) where: result is an optional FlowResult next node is an optional NodeConfig (for dynamic flows) or string (for static flows) ​ FlowFunctionHandler Callable[[FlowArgs, FlowManager], Awaitable[FlowResult | ConsolidatedFunctionResult]] Modern function handler that receives both arguments and FlowManager . Returns either: A FlowResult (⚠️ deprecated) A “consolidated” result tuple (result, next node) where: result is an optional FlowResult next node is an optional NodeConfig (for dynamic flows) or string (for static flows) ​ DirectFunction DirectFunction Function that is meant to be passed directly into a NodeConfig rather than into the handler field of a function configuration. It must be an async function with flow_manager: FlowManager as its first parameter. It must return a ConsolidatedFunctionResult , which is a tuple (result, next node) where: result is an optional FlowResult next node is an optional NodeConfig (for dynamic flows) or string (for static flows) ​ ContextStrategy ​ ContextStrategy Enum Strategy for managing conversation context during node transitions. Show Values ​ APPEND str Default strategy. Adds new messages to existing context. ​ RESET str Clears context and starts fresh with new messages. ​ RESET_WITH_SUMMARY str Resets context but includes an AI-generated summary. ​ ContextStrategyConfig ​ ContextStrategyConfig dataclass Configuration for context management strategy. Show Fields ​ strategy ContextStrategy required The strategy to use for context management ​ summary_prompt Optional[str] Required when using RESET_WITH_SUMMARY. Prompt text for generating the conversation summary. Copy Ask AI # Example usage config = ContextStrategyConfig( strategy = ContextStrategy. RESET_WITH_SUMMARY , summary_prompt = "Summarize the key points discussed so far." ) ​ FlowsFunctionSchema ​ FlowsFunctionSchema class A standardized schema for defining functions in Pipecat Flows with flow-specific properties. Show Constructor Parameters ​ name str required Name of the function ​ description str required Description of the function’s purpose ​ properties Dict[str, Any] required Dictionary defining properties types and descriptions ​ required List[str] required List of required parameter names ​ handler Optional[FunctionHandler] Function handler to process the function call ​ transition_to Optional[str] deprecated Target node to transition to after function execution Deprecated: instead of transition_to , use a “consolidated” handler that returns a tuple (result, next node). ​ transition_callback Optional[Callable] deprecated Callback function for dynamic transitions Deprecated: instead of transition_callback , use a “consolidated” handler that returns a tuple (result, next node). You cannot specify both transition_to and transition_callback in the same function schema. Example usage: Copy Ask AI from pipecat_flows import FlowsFunctionSchema # Define a function schema collect_name_schema = FlowsFunctionSchema( name = "collect_name" , description = "Record the user's name" , properties = { "name" : { "type" : "string" , "description" : "The user's name" } }, required = [ "name" ], handler = collect_name_handler ) # Use in node configuration node_config = { "name" : "greeting" , "task_messages" : [ { "role" : "system" , "content" : "Ask the user for their name." } ], "functions" : [collect_name_schema] } # Pass to flow manager await flow_manager.set_node_from_config(node_config) ​ FlowManager ​ FlowManager class Main class for managing conversation flows, supporting both static (configuration-driven) and dynamic (runtime-determined) flows. Show Constructor Parameters ​ task PipelineTask required Pipeline task for frame queueing ​ llm LLMService required LLM service instance (OpenAI, Anthropic, or Google). Must be initialized with the corresponding pipecat-ai provider dependency installed. ​ context_aggregator Any required Context aggregator used for pushing messages to the LLM service ​ tts Optional[Any] deprecated Optional TTS service for voice actions. Deprecated: No need to explicitly pass tts to FlowManager in order to use tts_say actions. ​ flow_config Optional[FlowConfig] Optional static flow configuration ​ context_strategy Optional[ContextStrategyConfig] Optional configuration for how context should be managed during transitions. Defaults to APPEND strategy if not specified. ​ Methods ​ initialize method Initialize the flow with starting messages. Show Parameters ​ initial_node NodeConfig The initial conversation node (needed for dynamic flows only). If not specified, you’ll need to call set_node_from_config() to kick off the conversation. Show Raises ​ FlowInitializationError If initialization fails ​ set_node method deprecated Set up a new conversation node programmatically (dynamic flows only). In dynamic flows, the application advances the conversation using set_node to set up each next node. In static flows, set_node is triggered under the hood when a node contains a transition_to field. Deprecated: use the following patterns instead of set_node : Prefer “consolidated” function handlers that return a tuple (result, next node), which implicitly sets up the next node Prefer passing your initial node to FlowManager.initialize() If you really need to set a node explicitly, use set_node_from_config() (note: its name will be read from its NodeConfig ) Show Parameters ​ node_id str required Identifier for the new node ​ node_config NodeConfig required Node configuration including messages, functions, and actions Show Raises ​ FlowError If node setup fails ​ set_node_from_config method Set up a new conversation node programmatically (dynamic flows only). Note that this method should only be used in rare circumstances. Most often, you should: Prefer “consolidated” function handlers that return a tuple (result, next node), which implicitly sets up the next node Prefer passing your initial node to FlowManager.initialize() Show Parameters ​ node_config NodeConfig required Node configuration including messages, functions, and actions Show Raises ​ FlowError If node setup fails ​ register_action method Register a handler for a custom action type. Show Parameters ​ action_type str required String identifier for the action ​ handler Callable required Async or sync function that handles the action ​ get_current_context method Get the current conversation context. Returns a list of messages in the current context, including system messages, user messages, and assistant responses. Show Returns ​ messages List[dict] List of messages in the current context Show Raises ​ FlowError If context aggregator is not available Example usage: Copy Ask AI # Access current conversation context context = flow_manager.get_current_context() # Use in handlers async def process_response ( args : FlowArgs) -> tuple[FlowResult, str ]: context = flow_manager.get_current_context() # Process conversation history return { "status" : "success" }, "next" ​ State Management The FlowManager provides a state dictionary for storing conversation data: Access state Access in transitions Copy Ask AI flow_manager.state: Dict[ str , Any] # Store data flow_manager.state[ "user_age" ] = 25 ​ Usage Examples Static Flow Dynamic Flow Copy Ask AI flow_config: FlowConfig = { "initial_node" : "greeting" , "nodes" : { "greeting" : { "role_messages" : [ { "role" : "system" , "content" : "You are a helpful assistant. Your responses will be converted to audio." } ], "task_messages" : [ { "role" : "system" , "content" : "Start by greeting the user and asking for their name." } ], "functions" : [{ "type" : "function" , "function" : { "name" : "collect_name" , "handler" : collect_name_handler, "description" : "Record user's name" , "parameters" : { ... } } }] } } } # Create and initialize the FlowManager flow_manager = FlowManager( task = task, llm = llm, context_aggregator = context_aggregator, flow_config = flow_config ) # Initialize the flow_manager to start the conversation await flow_manager.initialize() ​ Node Functions concept Functions that execute operations within a single conversational state, without switching nodes. Copy Ask AI Copy Ask AI from pipecat_flows import FlowArgs, FlowResult async def process_data ( args : FlowArgs) -> tuple[FlowResult, None ]: """Handle data processing within a node.""" data = args[ "data" ] result = await process(data) return { "status" : "success" , "processed_data" : result }, None # Function configuration { "type" : "function" , "function" : { "name" : "process_data" , "handler" : process_data, "description" : "Process user data" , "parameters" : { "type" : "object" , "properties" : { "data" : { "type" : "string" } } } } } ​ Edge Functions concept Functions that specify a transition between nodes (optionally processing data first). Copy Ask AI Copy Ask AI from pipecat_flows import FlowArgs, FlowResult async def next_step ( args : FlowArgs) -> tuple[ None , str ]: """Specify the next node to transition to.""" return None , "target_node" # Return NodeConfig instead of str for dynamic flows # Function configuration { "type" : "function" , "function" : { "name" : "next_step" , "handler" : next_step, "description" : "Transition to next node" , "parameters" : { "type" : "object" , "properties" : {}} } } ​ Function Properties ​ handler Optional[Callable] Async function that processes data within a node and/or specifies the next node ( more details here ). Can be specified as: Direct function reference Either a Callable function or a string with __function__: prefix (e.g., "__function__:process_data" ) to reference a function in the main script Direct Reference Function Token Copy Ask AI { "type" : "function" , "function" : { "name" : "process_data" , "handler" : process_data, # Callable function "parameters" : { ... } } } ​ transition_callback Optional[Callable] deprecated Handler for dynamic flow transitions. Deprecated: instead of transition_callback , use a “consolidated” handler that returns a tuple (result, next node). Must be an async function with one of these signatures: Copy Ask AI # New style (recommended) async def handle_transition ( args : Dict[ str , Any], result : FlowResult, flow_manager : FlowManager ) -> None : """Handle transition to next node.""" if result.available: # Type-safe access to result await flow_manager.set_node_from_config(create_confirmation_node()) else : await flow_manager.set_node_from_config( create_no_availability_node(result.alternative_times) ) # Legacy style (supported for backwards compatibility) async def handle_transition ( args : Dict[ str , Any], flow_manager : FlowManager ) -> None : """Handle transition to next node.""" await flow_manager.set_node_from_config(create_next_node()) The callback receives: args : Arguments from the function call result : Typed result from the function handler (new style only) flow_manager : Reference to the FlowManager instance Example usage: Copy Ask AI async def handle_availability_check ( args : Dict, result : TimeResult, # Typed result flow_manager : FlowManager ): """Handle availability check and transition based on result.""" if result.available: await flow_manager.set_node_from_config(create_confirmation_node()) else : await flow_manager.set_node_from_config( create_no_availability_node(result.alternative_times) ) # Use in function configuration { "type" : "function" , "function" : { "name" : "check_availability" , "handler" : check_availability, "parameters" : { ... }, "transition_callback" : handle_availability_check } } Note: A function cannot have both transition_to and transition_callback . ​ Handler Signatures Function handlers passed as a handler in a function configuration can be defined with three different signatures: Modern (Args + FlowManager) Legacy (Args Only) No Arguments Copy Ask AI async def handler_with_flow_manager ( args : FlowArgs, flow_manager : FlowManager) -> tuple[FlowResult, NodeConfig]: """Modern handler that receives both arguments and FlowManager access.""" # Access state previous_data = flow_manager.state.get( "stored_data" ) # Access pipeline resources await flow_manager.task.queue_frame(TTSSpeakFrame( "Processing your request..." )) # Store data in state for later flow_manager.state[ "new_data" ] = args[ "input" ] return { "status" : "success" , "result" : "Processed with flow access" }, create_next_node() The framework automatically detects which signature your handler is using and calls it appropriately. If you’re passing your function directly into your NodeConfig rather than as a handler in a function configuration, you’d use a somewhat different signature: Direct Copy Ask AI async def do_something ( flow_manager : FlowManager, foo : int , bar : str = "" ) -> tuple[FlowResult, NodeConfig]: """ Do something interesting. Args: foo (int): The foo to do something interesting with. bar (string): The bar to do something interesting with. Defaults to empty string. """ result = await fetch_data(foo, bar) next_node = create_end_node() return result, next_node ​ Return Types Success Response Error Response Copy Ask AI { "status" : "success" , "data" : "some data" # Optional additional data } ​ Provider-Specific Formats You don’t need to handle these format differences manually - use the standard format and the FlowManager will adapt it for your chosen provider. OpenAI Anthropic Google (Gemini) Copy Ask AI { "type" : "function" , "function" : { "name" : "function_name" , "handler" : handler, "description" : "Description" , "parameters" : { ... } } } ​ Actions pre_actions and post_actions are used to manage conversation flow. They are included in the NodeConfig and executed before and after the LLM completion, respectively. Three kinds of actions are available: Pre-canned actions: These actions perform common tasks and require little configuration. Function actions: These actions run developer-defined functions at the appropriate time. Custom actions: These are fully developer-defined actions, providing flexibility at the expense of complexity. ​ Pre-canned Actions Common actions shipped with Flows for managing conversation flow. To use them, just add them to your NodeConfig . ​ tts_say action Speaks text immediately using the TTS service. Copy Ask AI Copy Ask AI "pre_actions" : [ { "type" : "tts_say" , "text" : "Processing your request..." # Required } ] ​ end_conversation action Ends the conversation and closes the connection. Copy Ask AI Copy Ask AI "post_actions" : [ { "type" : "end_conversation" , "text" : "Goodbye!" # Optional farewell message } ] ​ Function Actions Actions that run developer-defined functions at the appropriate time. For example, if used in post_actions , they’ll run after the bot has finished talking and after any previous post_actions have finished. ​ function action Calls the developer-defined function at the appropriate time. Copy Ask AI Copy Ask AI "post_actions" : [ { "type" : "function" , "handler" : bot_turn_ended # Required } ] ​ Custom Actions Fully developer-defined actions, providing flexibility at the expense of complexity. Here’s the complexity: because these actions aren’t queued in the Pipecat pipeline, they may execute seemingly early if used in post_actions ; they’ll run immediately after the LLM completion is triggered but won’t wait around for the bot to finish talking. Why would you want this behavior? You might be writing an action that: Itself just queues another Frame into the Pipecat pipeline (meaning there would no benefit to waiting around for sequencing purposes) Does work that can be done a bit sooner, like logging that the LLM was updated Custom actions are composed of at least: ​ type str required String identifier for the action ​ handler Callable required Async or sync function that handles the action Example: Copy Ask AI Copy Ask AI # Define custom action handler async def custom_notification ( action : dict , flow_manager : FlowManager): """Custom action handler.""" message = action.get( "message" , "" ) await notify_user(message) # Use in node configuration "pre_actions" : [ { "type" : "notify" , "handler" : send_notification, "message" : "Attention!" , } ] ​ Exceptions ​ FlowError exception Base exception for all flow-related errors. Copy Ask AI Copy Ask AI from pipecat_flows import FlowError try : await flow_manager.set_node_from_config(config) except FlowError as e: print ( f "Flow error: { e } " ) ​ FlowInitializationError exception Raised when flow initialization fails. Copy Ask AI Copy Ask AI from pipecat_flows import FlowInitializationError try : await flow_manager.initialize() except FlowInitializationError as e: print ( f "Initialization failed: { e } " ) ​ FlowTransitionError exception Raised when a state transition fails. Copy Ask AI Copy Ask AI from pipecat_flows import FlowTransitionError try : await flow_manager.set_node_from_config(node_config) except FlowTransitionError as e: print ( f "Transition failed: { e } " ) ​ InvalidFunctionError exception Raised when an invalid or unavailable function is specified. Copy Ask AI Copy Ask AI from pipecat_flows import InvalidFunctionError try : await flow_manager.set_node_from_config({ "functions" : [{ "type" : "function" , "function" : { "name" : "invalid_function" } }] }) except InvalidFunctionError as e: print ( f "Invalid function: { e } " ) RTVI Observer PipelineParams On this page Installation Core Types FlowArgs FlowResult FlowConfig NodeConfig Function Handler Types ContextStrategy ContextStrategyConfig FlowsFunctionSchema FlowManager Methods State Management Usage Examples Function Properties Handler Signatures Return Types Provider-Specific Formats Actions Pre-canned Actions Function Actions Custom Actions Exceptions Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/flows_pipecat-flows_a04e63f0.txt b/flows_pipecat-flows_a04e63f0.txt
new file mode 100644
index 0000000000000000000000000000000000000000..cf5639b3ad2a350c508b201c03036aea76c0a8ca
--- /dev/null
+++ b/flows_pipecat-flows_a04e63f0.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/frameworks/flows/pipecat-flows#param-description
+Title: Pipecat Flows - Pipecat
+==================================================
+
+Pipecat Flows - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Frameworks Pipecat Flows Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline New to building conversational flows? Check out our Pipecat Flows guide first. ​ Installation Existing Pipecat installation Fresh Pipecat installation Copy Ask AI pip install pipecat-ai-flows ​ Core Types ​ FlowArgs ​ FlowArgs Dict[str, Any] Type alias for function handler arguments. ​ FlowResult ​ FlowResult TypedDict Base type for function handler results. Additional fields can be included as needed. Show Fields ​ status str Optional status field ​ error str Optional error message ​ FlowConfig ​ FlowConfig TypedDict Configuration for the entire conversation flow. Show Fields ​ initial_node str required Starting node identifier ​ nodes Dict[str, NodeConfig] required Map of node names to configurations ​ NodeConfig ​ NodeConfig TypedDict Configuration for a single node in the flow. Show Fields ​ name str The name of the node, used in debug logging in dynamic flows. If no name is specified, an automatically-generated UUID is used. Copy Ask AI # Example name "name" : "greeting" ​ role_messages List[dict] Defines the role or persona of the LLM. Required for the initial node and optional for subsequent nodes. Copy Ask AI # Example role messages "role_messages" : [ { "role" : "system" , "content" : "You are a helpful assistant..." } ], ​ task_messages List[dict] required Defines the task for a given node. Required for all nodes. Copy Ask AI # Example task messages "task_messages" : [ { "role" : "system" , # May be `user` depending on the LLM "content" : "Ask the user for their name..." } ], ​ context_strategy ContextStrategyConfig Strategy for managing context during transitions to this node. Copy Ask AI # Example context strategy configuration "context_strategy" : ContextStrategyConfig( strategy = ContextStrategy. RESET_WITH_SUMMARY , summary_prompt = "Summarize the key points discussed so far." ) ​ functions List[Union[dict, FlowsFunctionSchema]] required LLM function / tool call configurations, defined in one of the supported formats . Copy Ask AI # Using provider-specific dictionary format "functions" : [ { "type" : "function" , "function" : { "name" : "get_current_movies" , "handler" : get_movies, "description" : "Fetch movies currently playing" , "parameters" : { ... } }, } ] # Using FlowsFunctionSchema "functions" : [ FlowsFunctionSchema( name = "get_current_movies" , description = "Fetch movies currently playing" , properties = { ... }, required = [ ... ], handler = get_movies ) ] # Using direct functions (auto-configuration) "functions" : [get_movies] ​ pre_actions List[dict] Actions that execute before the LLM inference. For example, you can send a message to the TTS to speak a phrase (e.g. “Hold on a moment…”), which may be effective if an LLM function call takes time to execute. Copy Ask AI # Example pre_actions "pre_actions" : [ { "type" : "tts_say" , "text" : "Hold on a moment..." } ], ​ post_actions List[dict] Actions that execute after the LLM inference. For example, you can end the conversation. Copy Ask AI # Example post_actions "post_actions" : [ { "type" : "end_conversation" } ] ​ respond_immediately bool If set to False , the LLM will not respond immediately when the node is set, but will instead wait for the user to speak first before responding. Defaults to True . Copy Ask AI # Example usage "respond_immediately" : False ​ Function Handler Types ​ LegacyFunctionHandler Callable[[FlowArgs], Awaitable[FlowResult | ConsolidatedFunctionResult]] Legacy function handler that only receives arguments. Returns either: A FlowResult (⚠️ deprecated) A “consolidated” result tuple (result, next node) where: result is an optional FlowResult next node is an optional NodeConfig (for dynamic flows) or string (for static flows) ​ FlowFunctionHandler Callable[[FlowArgs, FlowManager], Awaitable[FlowResult | ConsolidatedFunctionResult]] Modern function handler that receives both arguments and FlowManager . Returns either: A FlowResult (⚠️ deprecated) A “consolidated” result tuple (result, next node) where: result is an optional FlowResult next node is an optional NodeConfig (for dynamic flows) or string (for static flows) ​ DirectFunction DirectFunction Function that is meant to be passed directly into a NodeConfig rather than into the handler field of a function configuration. It must be an async function with flow_manager: FlowManager as its first parameter. It must return a ConsolidatedFunctionResult , which is a tuple (result, next node) where: result is an optional FlowResult next node is an optional NodeConfig (for dynamic flows) or string (for static flows) ​ ContextStrategy ​ ContextStrategy Enum Strategy for managing conversation context during node transitions. Show Values ​ APPEND str Default strategy. Adds new messages to existing context. ​ RESET str Clears context and starts fresh with new messages. ​ RESET_WITH_SUMMARY str Resets context but includes an AI-generated summary. ​ ContextStrategyConfig ​ ContextStrategyConfig dataclass Configuration for context management strategy. Show Fields ​ strategy ContextStrategy required The strategy to use for context management ​ summary_prompt Optional[str] Required when using RESET_WITH_SUMMARY. Prompt text for generating the conversation summary. Copy Ask AI # Example usage config = ContextStrategyConfig( strategy = ContextStrategy. RESET_WITH_SUMMARY , summary_prompt = "Summarize the key points discussed so far." ) ​ FlowsFunctionSchema ​ FlowsFunctionSchema class A standardized schema for defining functions in Pipecat Flows with flow-specific properties. Show Constructor Parameters ​ name str required Name of the function ​ description str required Description of the function’s purpose ​ properties Dict[str, Any] required Dictionary defining properties types and descriptions ​ required List[str] required List of required parameter names ​ handler Optional[FunctionHandler] Function handler to process the function call ​ transition_to Optional[str] deprecated Target node to transition to after function execution Deprecated: instead of transition_to , use a “consolidated” handler that returns a tuple (result, next node). ​ transition_callback Optional[Callable] deprecated Callback function for dynamic transitions Deprecated: instead of transition_callback , use a “consolidated” handler that returns a tuple (result, next node). You cannot specify both transition_to and transition_callback in the same function schema. Example usage: Copy Ask AI from pipecat_flows import FlowsFunctionSchema # Define a function schema collect_name_schema = FlowsFunctionSchema( name = "collect_name" , description = "Record the user's name" , properties = { "name" : { "type" : "string" , "description" : "The user's name" } }, required = [ "name" ], handler = collect_name_handler ) # Use in node configuration node_config = { "name" : "greeting" , "task_messages" : [ { "role" : "system" , "content" : "Ask the user for their name." } ], "functions" : [collect_name_schema] } # Pass to flow manager await flow_manager.set_node_from_config(node_config) ​ FlowManager ​ FlowManager class Main class for managing conversation flows, supporting both static (configuration-driven) and dynamic (runtime-determined) flows. Show Constructor Parameters ​ task PipelineTask required Pipeline task for frame queueing ​ llm LLMService required LLM service instance (OpenAI, Anthropic, or Google). Must be initialized with the corresponding pipecat-ai provider dependency installed. ​ context_aggregator Any required Context aggregator used for pushing messages to the LLM service ​ tts Optional[Any] deprecated Optional TTS service for voice actions. Deprecated: No need to explicitly pass tts to FlowManager in order to use tts_say actions. ​ flow_config Optional[FlowConfig] Optional static flow configuration ​ context_strategy Optional[ContextStrategyConfig] Optional configuration for how context should be managed during transitions. Defaults to APPEND strategy if not specified. ​ Methods ​ initialize method Initialize the flow with starting messages. Show Parameters ​ initial_node NodeConfig The initial conversation node (needed for dynamic flows only). If not specified, you’ll need to call set_node_from_config() to kick off the conversation. Show Raises ​ FlowInitializationError If initialization fails ​ set_node method deprecated Set up a new conversation node programmatically (dynamic flows only). In dynamic flows, the application advances the conversation using set_node to set up each next node. In static flows, set_node is triggered under the hood when a node contains a transition_to field. Deprecated: use the following patterns instead of set_node : Prefer “consolidated” function handlers that return a tuple (result, next node), which implicitly sets up the next node Prefer passing your initial node to FlowManager.initialize() If you really need to set a node explicitly, use set_node_from_config() (note: its name will be read from its NodeConfig ) Show Parameters ​ node_id str required Identifier for the new node ​ node_config NodeConfig required Node configuration including messages, functions, and actions Show Raises ​ FlowError If node setup fails ​ set_node_from_config method Set up a new conversation node programmatically (dynamic flows only). Note that this method should only be used in rare circumstances. Most often, you should: Prefer “consolidated” function handlers that return a tuple (result, next node), which implicitly sets up the next node Prefer passing your initial node to FlowManager.initialize() Show Parameters ​ node_config NodeConfig required Node configuration including messages, functions, and actions Show Raises ​ FlowError If node setup fails ​ register_action method Register a handler for a custom action type. Show Parameters ​ action_type str required String identifier for the action ​ handler Callable required Async or sync function that handles the action ​ get_current_context method Get the current conversation context. Returns a list of messages in the current context, including system messages, user messages, and assistant responses. Show Returns ​ messages List[dict] List of messages in the current context Show Raises ​ FlowError If context aggregator is not available Example usage: Copy Ask AI # Access current conversation context context = flow_manager.get_current_context() # Use in handlers async def process_response ( args : FlowArgs) -> tuple[FlowResult, str ]: context = flow_manager.get_current_context() # Process conversation history return { "status" : "success" }, "next" ​ State Management The FlowManager provides a state dictionary for storing conversation data: Access state Access in transitions Copy Ask AI flow_manager.state: Dict[ str , Any] # Store data flow_manager.state[ "user_age" ] = 25 ​ Usage Examples Static Flow Dynamic Flow Copy Ask AI flow_config: FlowConfig = { "initial_node" : "greeting" , "nodes" : { "greeting" : { "role_messages" : [ { "role" : "system" , "content" : "You are a helpful assistant. Your responses will be converted to audio." } ], "task_messages" : [ { "role" : "system" , "content" : "Start by greeting the user and asking for their name." } ], "functions" : [{ "type" : "function" , "function" : { "name" : "collect_name" , "handler" : collect_name_handler, "description" : "Record user's name" , "parameters" : { ... } } }] } } } # Create and initialize the FlowManager flow_manager = FlowManager( task = task, llm = llm, context_aggregator = context_aggregator, flow_config = flow_config ) # Initialize the flow_manager to start the conversation await flow_manager.initialize() ​ Node Functions concept Functions that execute operations within a single conversational state, without switching nodes. Copy Ask AI Copy Ask AI from pipecat_flows import FlowArgs, FlowResult async def process_data ( args : FlowArgs) -> tuple[FlowResult, None ]: """Handle data processing within a node.""" data = args[ "data" ] result = await process(data) return { "status" : "success" , "processed_data" : result }, None # Function configuration { "type" : "function" , "function" : { "name" : "process_data" , "handler" : process_data, "description" : "Process user data" , "parameters" : { "type" : "object" , "properties" : { "data" : { "type" : "string" } } } } } ​ Edge Functions concept Functions that specify a transition between nodes (optionally processing data first). Copy Ask AI Copy Ask AI from pipecat_flows import FlowArgs, FlowResult async def next_step ( args : FlowArgs) -> tuple[ None , str ]: """Specify the next node to transition to.""" return None , "target_node" # Return NodeConfig instead of str for dynamic flows # Function configuration { "type" : "function" , "function" : { "name" : "next_step" , "handler" : next_step, "description" : "Transition to next node" , "parameters" : { "type" : "object" , "properties" : {}} } } ​ Function Properties ​ handler Optional[Callable] Async function that processes data within a node and/or specifies the next node ( more details here ). Can be specified as: Direct function reference Either a Callable function or a string with __function__: prefix (e.g., "__function__:process_data" ) to reference a function in the main script Direct Reference Function Token Copy Ask AI { "type" : "function" , "function" : { "name" : "process_data" , "handler" : process_data, # Callable function "parameters" : { ... } } } ​ transition_callback Optional[Callable] deprecated Handler for dynamic flow transitions. Deprecated: instead of transition_callback , use a “consolidated” handler that returns a tuple (result, next node). Must be an async function with one of these signatures: Copy Ask AI # New style (recommended) async def handle_transition ( args : Dict[ str , Any], result : FlowResult, flow_manager : FlowManager ) -> None : """Handle transition to next node.""" if result.available: # Type-safe access to result await flow_manager.set_node_from_config(create_confirmation_node()) else : await flow_manager.set_node_from_config( create_no_availability_node(result.alternative_times) ) # Legacy style (supported for backwards compatibility) async def handle_transition ( args : Dict[ str , Any], flow_manager : FlowManager ) -> None : """Handle transition to next node.""" await flow_manager.set_node_from_config(create_next_node()) The callback receives: args : Arguments from the function call result : Typed result from the function handler (new style only) flow_manager : Reference to the FlowManager instance Example usage: Copy Ask AI async def handle_availability_check ( args : Dict, result : TimeResult, # Typed result flow_manager : FlowManager ): """Handle availability check and transition based on result.""" if result.available: await flow_manager.set_node_from_config(create_confirmation_node()) else : await flow_manager.set_node_from_config( create_no_availability_node(result.alternative_times) ) # Use in function configuration { "type" : "function" , "function" : { "name" : "check_availability" , "handler" : check_availability, "parameters" : { ... }, "transition_callback" : handle_availability_check } } Note: A function cannot have both transition_to and transition_callback . ​ Handler Signatures Function handlers passed as a handler in a function configuration can be defined with three different signatures: Modern (Args + FlowManager) Legacy (Args Only) No Arguments Copy Ask AI async def handler_with_flow_manager ( args : FlowArgs, flow_manager : FlowManager) -> tuple[FlowResult, NodeConfig]: """Modern handler that receives both arguments and FlowManager access.""" # Access state previous_data = flow_manager.state.get( "stored_data" ) # Access pipeline resources await flow_manager.task.queue_frame(TTSSpeakFrame( "Processing your request..." )) # Store data in state for later flow_manager.state[ "new_data" ] = args[ "input" ] return { "status" : "success" , "result" : "Processed with flow access" }, create_next_node() The framework automatically detects which signature your handler is using and calls it appropriately. If you’re passing your function directly into your NodeConfig rather than as a handler in a function configuration, you’d use a somewhat different signature: Direct Copy Ask AI async def do_something ( flow_manager : FlowManager, foo : int , bar : str = "" ) -> tuple[FlowResult, NodeConfig]: """ Do something interesting. Args: foo (int): The foo to do something interesting with. bar (string): The bar to do something interesting with. Defaults to empty string. """ result = await fetch_data(foo, bar) next_node = create_end_node() return result, next_node ​ Return Types Success Response Error Response Copy Ask AI { "status" : "success" , "data" : "some data" # Optional additional data } ​ Provider-Specific Formats You don’t need to handle these format differences manually - use the standard format and the FlowManager will adapt it for your chosen provider. OpenAI Anthropic Google (Gemini) Copy Ask AI { "type" : "function" , "function" : { "name" : "function_name" , "handler" : handler, "description" : "Description" , "parameters" : { ... } } } ​ Actions pre_actions and post_actions are used to manage conversation flow. They are included in the NodeConfig and executed before and after the LLM completion, respectively. Three kinds of actions are available: Pre-canned actions: These actions perform common tasks and require little configuration. Function actions: These actions run developer-defined functions at the appropriate time. Custom actions: These are fully developer-defined actions, providing flexibility at the expense of complexity. ​ Pre-canned Actions Common actions shipped with Flows for managing conversation flow. To use them, just add them to your NodeConfig . ​ tts_say action Speaks text immediately using the TTS service. Copy Ask AI Copy Ask AI "pre_actions" : [ { "type" : "tts_say" , "text" : "Processing your request..." # Required } ] ​ end_conversation action Ends the conversation and closes the connection. Copy Ask AI Copy Ask AI "post_actions" : [ { "type" : "end_conversation" , "text" : "Goodbye!" # Optional farewell message } ] ​ Function Actions Actions that run developer-defined functions at the appropriate time. For example, if used in post_actions , they’ll run after the bot has finished talking and after any previous post_actions have finished. ​ function action Calls the developer-defined function at the appropriate time. Copy Ask AI Copy Ask AI "post_actions" : [ { "type" : "function" , "handler" : bot_turn_ended # Required } ] ​ Custom Actions Fully developer-defined actions, providing flexibility at the expense of complexity. Here’s the complexity: because these actions aren’t queued in the Pipecat pipeline, they may execute seemingly early if used in post_actions ; they’ll run immediately after the LLM completion is triggered but won’t wait around for the bot to finish talking. Why would you want this behavior? You might be writing an action that: Itself just queues another Frame into the Pipecat pipeline (meaning there would no benefit to waiting around for sequencing purposes) Does work that can be done a bit sooner, like logging that the LLM was updated Custom actions are composed of at least: ​ type str required String identifier for the action ​ handler Callable required Async or sync function that handles the action Example: Copy Ask AI Copy Ask AI # Define custom action handler async def custom_notification ( action : dict , flow_manager : FlowManager): """Custom action handler.""" message = action.get( "message" , "" ) await notify_user(message) # Use in node configuration "pre_actions" : [ { "type" : "notify" , "handler" : send_notification, "message" : "Attention!" , } ] ​ Exceptions ​ FlowError exception Base exception for all flow-related errors. Copy Ask AI Copy Ask AI from pipecat_flows import FlowError try : await flow_manager.set_node_from_config(config) except FlowError as e: print ( f "Flow error: { e } " ) ​ FlowInitializationError exception Raised when flow initialization fails. Copy Ask AI Copy Ask AI from pipecat_flows import FlowInitializationError try : await flow_manager.initialize() except FlowInitializationError as e: print ( f "Initialization failed: { e } " ) ​ FlowTransitionError exception Raised when a state transition fails. Copy Ask AI Copy Ask AI from pipecat_flows import FlowTransitionError try : await flow_manager.set_node_from_config(node_config) except FlowTransitionError as e: print ( f "Transition failed: { e } " ) ​ InvalidFunctionError exception Raised when an invalid or unavailable function is specified. Copy Ask AI Copy Ask AI from pipecat_flows import InvalidFunctionError try : await flow_manager.set_node_from_config({ "functions" : [{ "type" : "function" , "function" : { "name" : "invalid_function" } }] }) except InvalidFunctionError as e: print ( f "Invalid function: { e } " ) RTVI Observer PipelineParams On this page Installation Core Types FlowArgs FlowResult FlowConfig NodeConfig Function Handler Types ContextStrategy ContextStrategyConfig FlowsFunctionSchema FlowManager Methods State Management Usage Examples Function Properties Handler Signatures Return Types Provider-Specific Formats Actions Pre-canned Actions Function Actions Custom Actions Exceptions Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/flows_pipecat-flows_a8e72ea8.txt b/flows_pipecat-flows_a8e72ea8.txt
new file mode 100644
index 0000000000000000000000000000000000000000..6945d0a212ddce176b21ec33cf681ca00bf01057
--- /dev/null
+++ b/flows_pipecat-flows_a8e72ea8.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/frameworks/flows/pipecat-flows#param-initial-node
+Title: Pipecat Flows - Pipecat
+==================================================
+
+Pipecat Flows - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Frameworks Pipecat Flows Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline New to building conversational flows? Check out our Pipecat Flows guide first. ​ Installation Existing Pipecat installation Fresh Pipecat installation Copy Ask AI pip install pipecat-ai-flows ​ Core Types ​ FlowArgs ​ FlowArgs Dict[str, Any] Type alias for function handler arguments. ​ FlowResult ​ FlowResult TypedDict Base type for function handler results. Additional fields can be included as needed. Show Fields ​ status str Optional status field ​ error str Optional error message ​ FlowConfig ​ FlowConfig TypedDict Configuration for the entire conversation flow. Show Fields ​ initial_node str required Starting node identifier ​ nodes Dict[str, NodeConfig] required Map of node names to configurations ​ NodeConfig ​ NodeConfig TypedDict Configuration for a single node in the flow. Show Fields ​ name str The name of the node, used in debug logging in dynamic flows. If no name is specified, an automatically-generated UUID is used. Copy Ask AI # Example name "name" : "greeting" ​ role_messages List[dict] Defines the role or persona of the LLM. Required for the initial node and optional for subsequent nodes. Copy Ask AI # Example role messages "role_messages" : [ { "role" : "system" , "content" : "You are a helpful assistant..." } ], ​ task_messages List[dict] required Defines the task for a given node. Required for all nodes. Copy Ask AI # Example task messages "task_messages" : [ { "role" : "system" , # May be `user` depending on the LLM "content" : "Ask the user for their name..." } ], ​ context_strategy ContextStrategyConfig Strategy for managing context during transitions to this node. Copy Ask AI # Example context strategy configuration "context_strategy" : ContextStrategyConfig( strategy = ContextStrategy. RESET_WITH_SUMMARY , summary_prompt = "Summarize the key points discussed so far." ) ​ functions List[Union[dict, FlowsFunctionSchema]] required LLM function / tool call configurations, defined in one of the supported formats . Copy Ask AI # Using provider-specific dictionary format "functions" : [ { "type" : "function" , "function" : { "name" : "get_current_movies" , "handler" : get_movies, "description" : "Fetch movies currently playing" , "parameters" : { ... } }, } ] # Using FlowsFunctionSchema "functions" : [ FlowsFunctionSchema( name = "get_current_movies" , description = "Fetch movies currently playing" , properties = { ... }, required = [ ... ], handler = get_movies ) ] # Using direct functions (auto-configuration) "functions" : [get_movies] ​ pre_actions List[dict] Actions that execute before the LLM inference. For example, you can send a message to the TTS to speak a phrase (e.g. “Hold on a moment…”), which may be effective if an LLM function call takes time to execute. Copy Ask AI # Example pre_actions "pre_actions" : [ { "type" : "tts_say" , "text" : "Hold on a moment..." } ], ​ post_actions List[dict] Actions that execute after the LLM inference. For example, you can end the conversation. Copy Ask AI # Example post_actions "post_actions" : [ { "type" : "end_conversation" } ] ​ respond_immediately bool If set to False , the LLM will not respond immediately when the node is set, but will instead wait for the user to speak first before responding. Defaults to True . Copy Ask AI # Example usage "respond_immediately" : False ​ Function Handler Types ​ LegacyFunctionHandler Callable[[FlowArgs], Awaitable[FlowResult | ConsolidatedFunctionResult]] Legacy function handler that only receives arguments. Returns either: A FlowResult (⚠️ deprecated) A “consolidated” result tuple (result, next node) where: result is an optional FlowResult next node is an optional NodeConfig (for dynamic flows) or string (for static flows) ​ FlowFunctionHandler Callable[[FlowArgs, FlowManager], Awaitable[FlowResult | ConsolidatedFunctionResult]] Modern function handler that receives both arguments and FlowManager . Returns either: A FlowResult (⚠️ deprecated) A “consolidated” result tuple (result, next node) where: result is an optional FlowResult next node is an optional NodeConfig (for dynamic flows) or string (for static flows) ​ DirectFunction DirectFunction Function that is meant to be passed directly into a NodeConfig rather than into the handler field of a function configuration. It must be an async function with flow_manager: FlowManager as its first parameter. It must return a ConsolidatedFunctionResult , which is a tuple (result, next node) where: result is an optional FlowResult next node is an optional NodeConfig (for dynamic flows) or string (for static flows) ​ ContextStrategy ​ ContextStrategy Enum Strategy for managing conversation context during node transitions. Show Values ​ APPEND str Default strategy. Adds new messages to existing context. ​ RESET str Clears context and starts fresh with new messages. ​ RESET_WITH_SUMMARY str Resets context but includes an AI-generated summary. ​ ContextStrategyConfig ​ ContextStrategyConfig dataclass Configuration for context management strategy. Show Fields ​ strategy ContextStrategy required The strategy to use for context management ​ summary_prompt Optional[str] Required when using RESET_WITH_SUMMARY. Prompt text for generating the conversation summary. Copy Ask AI # Example usage config = ContextStrategyConfig( strategy = ContextStrategy. RESET_WITH_SUMMARY , summary_prompt = "Summarize the key points discussed so far." ) ​ FlowsFunctionSchema ​ FlowsFunctionSchema class A standardized schema for defining functions in Pipecat Flows with flow-specific properties. Show Constructor Parameters ​ name str required Name of the function ​ description str required Description of the function’s purpose ​ properties Dict[str, Any] required Dictionary defining properties types and descriptions ​ required List[str] required List of required parameter names ​ handler Optional[FunctionHandler] Function handler to process the function call ​ transition_to Optional[str] deprecated Target node to transition to after function execution Deprecated: instead of transition_to , use a “consolidated” handler that returns a tuple (result, next node). ​ transition_callback Optional[Callable] deprecated Callback function for dynamic transitions Deprecated: instead of transition_callback , use a “consolidated” handler that returns a tuple (result, next node). You cannot specify both transition_to and transition_callback in the same function schema. Example usage: Copy Ask AI from pipecat_flows import FlowsFunctionSchema # Define a function schema collect_name_schema = FlowsFunctionSchema( name = "collect_name" , description = "Record the user's name" , properties = { "name" : { "type" : "string" , "description" : "The user's name" } }, required = [ "name" ], handler = collect_name_handler ) # Use in node configuration node_config = { "name" : "greeting" , "task_messages" : [ { "role" : "system" , "content" : "Ask the user for their name." } ], "functions" : [collect_name_schema] } # Pass to flow manager await flow_manager.set_node_from_config(node_config) ​ FlowManager ​ FlowManager class Main class for managing conversation flows, supporting both static (configuration-driven) and dynamic (runtime-determined) flows. Show Constructor Parameters ​ task PipelineTask required Pipeline task for frame queueing ​ llm LLMService required LLM service instance (OpenAI, Anthropic, or Google). Must be initialized with the corresponding pipecat-ai provider dependency installed. ​ context_aggregator Any required Context aggregator used for pushing messages to the LLM service ​ tts Optional[Any] deprecated Optional TTS service for voice actions. Deprecated: No need to explicitly pass tts to FlowManager in order to use tts_say actions. ​ flow_config Optional[FlowConfig] Optional static flow configuration ​ context_strategy Optional[ContextStrategyConfig] Optional configuration for how context should be managed during transitions. Defaults to APPEND strategy if not specified. ​ Methods ​ initialize method Initialize the flow with starting messages. Show Parameters ​ initial_node NodeConfig The initial conversation node (needed for dynamic flows only). If not specified, you’ll need to call set_node_from_config() to kick off the conversation. Show Raises ​ FlowInitializationError If initialization fails ​ set_node method deprecated Set up a new conversation node programmatically (dynamic flows only). In dynamic flows, the application advances the conversation using set_node to set up each next node. In static flows, set_node is triggered under the hood when a node contains a transition_to field. Deprecated: use the following patterns instead of set_node : Prefer “consolidated” function handlers that return a tuple (result, next node), which implicitly sets up the next node Prefer passing your initial node to FlowManager.initialize() If you really need to set a node explicitly, use set_node_from_config() (note: its name will be read from its NodeConfig ) Show Parameters ​ node_id str required Identifier for the new node ​ node_config NodeConfig required Node configuration including messages, functions, and actions Show Raises ​ FlowError If node setup fails ​ set_node_from_config method Set up a new conversation node programmatically (dynamic flows only). Note that this method should only be used in rare circumstances. Most often, you should: Prefer “consolidated” function handlers that return a tuple (result, next node), which implicitly sets up the next node Prefer passing your initial node to FlowManager.initialize() Show Parameters ​ node_config NodeConfig required Node configuration including messages, functions, and actions Show Raises ​ FlowError If node setup fails ​ register_action method Register a handler for a custom action type. Show Parameters ​ action_type str required String identifier for the action ​ handler Callable required Async or sync function that handles the action ​ get_current_context method Get the current conversation context. Returns a list of messages in the current context, including system messages, user messages, and assistant responses. Show Returns ​ messages List[dict] List of messages in the current context Show Raises ​ FlowError If context aggregator is not available Example usage: Copy Ask AI # Access current conversation context context = flow_manager.get_current_context() # Use in handlers async def process_response ( args : FlowArgs) -> tuple[FlowResult, str ]: context = flow_manager.get_current_context() # Process conversation history return { "status" : "success" }, "next" ​ State Management The FlowManager provides a state dictionary for storing conversation data: Access state Access in transitions Copy Ask AI flow_manager.state: Dict[ str , Any] # Store data flow_manager.state[ "user_age" ] = 25 ​ Usage Examples Static Flow Dynamic Flow Copy Ask AI flow_config: FlowConfig = { "initial_node" : "greeting" , "nodes" : { "greeting" : { "role_messages" : [ { "role" : "system" , "content" : "You are a helpful assistant. Your responses will be converted to audio." } ], "task_messages" : [ { "role" : "system" , "content" : "Start by greeting the user and asking for their name." } ], "functions" : [{ "type" : "function" , "function" : { "name" : "collect_name" , "handler" : collect_name_handler, "description" : "Record user's name" , "parameters" : { ... } } }] } } } # Create and initialize the FlowManager flow_manager = FlowManager( task = task, llm = llm, context_aggregator = context_aggregator, flow_config = flow_config ) # Initialize the flow_manager to start the conversation await flow_manager.initialize() ​ Node Functions concept Functions that execute operations within a single conversational state, without switching nodes. Copy Ask AI Copy Ask AI from pipecat_flows import FlowArgs, FlowResult async def process_data ( args : FlowArgs) -> tuple[FlowResult, None ]: """Handle data processing within a node.""" data = args[ "data" ] result = await process(data) return { "status" : "success" , "processed_data" : result }, None # Function configuration { "type" : "function" , "function" : { "name" : "process_data" , "handler" : process_data, "description" : "Process user data" , "parameters" : { "type" : "object" , "properties" : { "data" : { "type" : "string" } } } } } ​ Edge Functions concept Functions that specify a transition between nodes (optionally processing data first). Copy Ask AI Copy Ask AI from pipecat_flows import FlowArgs, FlowResult async def next_step ( args : FlowArgs) -> tuple[ None , str ]: """Specify the next node to transition to.""" return None , "target_node" # Return NodeConfig instead of str for dynamic flows # Function configuration { "type" : "function" , "function" : { "name" : "next_step" , "handler" : next_step, "description" : "Transition to next node" , "parameters" : { "type" : "object" , "properties" : {}} } } ​ Function Properties ​ handler Optional[Callable] Async function that processes data within a node and/or specifies the next node ( more details here ). Can be specified as: Direct function reference Either a Callable function or a string with __function__: prefix (e.g., "__function__:process_data" ) to reference a function in the main script Direct Reference Function Token Copy Ask AI { "type" : "function" , "function" : { "name" : "process_data" , "handler" : process_data, # Callable function "parameters" : { ... } } } ​ transition_callback Optional[Callable] deprecated Handler for dynamic flow transitions. Deprecated: instead of transition_callback , use a “consolidated” handler that returns a tuple (result, next node). Must be an async function with one of these signatures: Copy Ask AI # New style (recommended) async def handle_transition ( args : Dict[ str , Any], result : FlowResult, flow_manager : FlowManager ) -> None : """Handle transition to next node.""" if result.available: # Type-safe access to result await flow_manager.set_node_from_config(create_confirmation_node()) else : await flow_manager.set_node_from_config( create_no_availability_node(result.alternative_times) ) # Legacy style (supported for backwards compatibility) async def handle_transition ( args : Dict[ str , Any], flow_manager : FlowManager ) -> None : """Handle transition to next node.""" await flow_manager.set_node_from_config(create_next_node()) The callback receives: args : Arguments from the function call result : Typed result from the function handler (new style only) flow_manager : Reference to the FlowManager instance Example usage: Copy Ask AI async def handle_availability_check ( args : Dict, result : TimeResult, # Typed result flow_manager : FlowManager ): """Handle availability check and transition based on result.""" if result.available: await flow_manager.set_node_from_config(create_confirmation_node()) else : await flow_manager.set_node_from_config( create_no_availability_node(result.alternative_times) ) # Use in function configuration { "type" : "function" , "function" : { "name" : "check_availability" , "handler" : check_availability, "parameters" : { ... }, "transition_callback" : handle_availability_check } } Note: A function cannot have both transition_to and transition_callback . ​ Handler Signatures Function handlers passed as a handler in a function configuration can be defined with three different signatures: Modern (Args + FlowManager) Legacy (Args Only) No Arguments Copy Ask AI async def handler_with_flow_manager ( args : FlowArgs, flow_manager : FlowManager) -> tuple[FlowResult, NodeConfig]: """Modern handler that receives both arguments and FlowManager access.""" # Access state previous_data = flow_manager.state.get( "stored_data" ) # Access pipeline resources await flow_manager.task.queue_frame(TTSSpeakFrame( "Processing your request..." )) # Store data in state for later flow_manager.state[ "new_data" ] = args[ "input" ] return { "status" : "success" , "result" : "Processed with flow access" }, create_next_node() The framework automatically detects which signature your handler is using and calls it appropriately. If you’re passing your function directly into your NodeConfig rather than as a handler in a function configuration, you’d use a somewhat different signature: Direct Copy Ask AI async def do_something ( flow_manager : FlowManager, foo : int , bar : str = "" ) -> tuple[FlowResult, NodeConfig]: """ Do something interesting. Args: foo (int): The foo to do something interesting with. bar (string): The bar to do something interesting with. Defaults to empty string. """ result = await fetch_data(foo, bar) next_node = create_end_node() return result, next_node ​ Return Types Success Response Error Response Copy Ask AI { "status" : "success" , "data" : "some data" # Optional additional data } ​ Provider-Specific Formats You don’t need to handle these format differences manually - use the standard format and the FlowManager will adapt it for your chosen provider. OpenAI Anthropic Google (Gemini) Copy Ask AI { "type" : "function" , "function" : { "name" : "function_name" , "handler" : handler, "description" : "Description" , "parameters" : { ... } } } ​ Actions pre_actions and post_actions are used to manage conversation flow. They are included in the NodeConfig and executed before and after the LLM completion, respectively. Three kinds of actions are available: Pre-canned actions: These actions perform common tasks and require little configuration. Function actions: These actions run developer-defined functions at the appropriate time. Custom actions: These are fully developer-defined actions, providing flexibility at the expense of complexity. ​ Pre-canned Actions Common actions shipped with Flows for managing conversation flow. To use them, just add them to your NodeConfig . ​ tts_say action Speaks text immediately using the TTS service. Copy Ask AI Copy Ask AI "pre_actions" : [ { "type" : "tts_say" , "text" : "Processing your request..." # Required } ] ​ end_conversation action Ends the conversation and closes the connection. Copy Ask AI Copy Ask AI "post_actions" : [ { "type" : "end_conversation" , "text" : "Goodbye!" # Optional farewell message } ] ​ Function Actions Actions that run developer-defined functions at the appropriate time. For example, if used in post_actions , they’ll run after the bot has finished talking and after any previous post_actions have finished. ​ function action Calls the developer-defined function at the appropriate time. Copy Ask AI Copy Ask AI "post_actions" : [ { "type" : "function" , "handler" : bot_turn_ended # Required } ] ​ Custom Actions Fully developer-defined actions, providing flexibility at the expense of complexity. Here’s the complexity: because these actions aren’t queued in the Pipecat pipeline, they may execute seemingly early if used in post_actions ; they’ll run immediately after the LLM completion is triggered but won’t wait around for the bot to finish talking. Why would you want this behavior? You might be writing an action that: Itself just queues another Frame into the Pipecat pipeline (meaning there would no benefit to waiting around for sequencing purposes) Does work that can be done a bit sooner, like logging that the LLM was updated Custom actions are composed of at least: ​ type str required String identifier for the action ​ handler Callable required Async or sync function that handles the action Example: Copy Ask AI Copy Ask AI # Define custom action handler async def custom_notification ( action : dict , flow_manager : FlowManager): """Custom action handler.""" message = action.get( "message" , "" ) await notify_user(message) # Use in node configuration "pre_actions" : [ { "type" : "notify" , "handler" : send_notification, "message" : "Attention!" , } ] ​ Exceptions ​ FlowError exception Base exception for all flow-related errors. Copy Ask AI Copy Ask AI from pipecat_flows import FlowError try : await flow_manager.set_node_from_config(config) except FlowError as e: print ( f "Flow error: { e } " ) ​ FlowInitializationError exception Raised when flow initialization fails. Copy Ask AI Copy Ask AI from pipecat_flows import FlowInitializationError try : await flow_manager.initialize() except FlowInitializationError as e: print ( f "Initialization failed: { e } " ) ​ FlowTransitionError exception Raised when a state transition fails. Copy Ask AI Copy Ask AI from pipecat_flows import FlowTransitionError try : await flow_manager.set_node_from_config(node_config) except FlowTransitionError as e: print ( f "Transition failed: { e } " ) ​ InvalidFunctionError exception Raised when an invalid or unavailable function is specified. Copy Ask AI Copy Ask AI from pipecat_flows import InvalidFunctionError try : await flow_manager.set_node_from_config({ "functions" : [{ "type" : "function" , "function" : { "name" : "invalid_function" } }] }) except InvalidFunctionError as e: print ( f "Invalid function: { e } " ) RTVI Observer PipelineParams On this page Installation Core Types FlowArgs FlowResult FlowConfig NodeConfig Function Handler Types ContextStrategy ContextStrategyConfig FlowsFunctionSchema FlowManager Methods State Management Usage Examples Function Properties Handler Signatures Return Types Provider-Specific Formats Actions Pre-canned Actions Function Actions Custom Actions Exceptions Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/flows_pipecat-flows_b971005b.txt b/flows_pipecat-flows_b971005b.txt
new file mode 100644
index 0000000000000000000000000000000000000000..6f1214077e33350a978474a9d8fed463d31d393d
--- /dev/null
+++ b/flows_pipecat-flows_b971005b.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/frameworks/flows/pipecat-flows#param-task
+Title: Pipecat Flows - Pipecat
+==================================================
+
+Pipecat Flows - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Frameworks Pipecat Flows Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline New to building conversational flows? Check out our Pipecat Flows guide first. ​ Installation Existing Pipecat installation Fresh Pipecat installation Copy Ask AI pip install pipecat-ai-flows ​ Core Types ​ FlowArgs ​ FlowArgs Dict[str, Any] Type alias for function handler arguments. ​ FlowResult ​ FlowResult TypedDict Base type for function handler results. Additional fields can be included as needed. Show Fields ​ status str Optional status field ​ error str Optional error message ​ FlowConfig ​ FlowConfig TypedDict Configuration for the entire conversation flow. Show Fields ​ initial_node str required Starting node identifier ​ nodes Dict[str, NodeConfig] required Map of node names to configurations ​ NodeConfig ​ NodeConfig TypedDict Configuration for a single node in the flow. Show Fields ​ name str The name of the node, used in debug logging in dynamic flows. If no name is specified, an automatically-generated UUID is used. Copy Ask AI # Example name "name" : "greeting" ​ role_messages List[dict] Defines the role or persona of the LLM. Required for the initial node and optional for subsequent nodes. Copy Ask AI # Example role messages "role_messages" : [ { "role" : "system" , "content" : "You are a helpful assistant..." } ], ​ task_messages List[dict] required Defines the task for a given node. Required for all nodes. Copy Ask AI # Example task messages "task_messages" : [ { "role" : "system" , # May be `user` depending on the LLM "content" : "Ask the user for their name..." } ], ​ context_strategy ContextStrategyConfig Strategy for managing context during transitions to this node. Copy Ask AI # Example context strategy configuration "context_strategy" : ContextStrategyConfig( strategy = ContextStrategy. RESET_WITH_SUMMARY , summary_prompt = "Summarize the key points discussed so far." ) ​ functions List[Union[dict, FlowsFunctionSchema]] required LLM function / tool call configurations, defined in one of the supported formats . Copy Ask AI # Using provider-specific dictionary format "functions" : [ { "type" : "function" , "function" : { "name" : "get_current_movies" , "handler" : get_movies, "description" : "Fetch movies currently playing" , "parameters" : { ... } }, } ] # Using FlowsFunctionSchema "functions" : [ FlowsFunctionSchema( name = "get_current_movies" , description = "Fetch movies currently playing" , properties = { ... }, required = [ ... ], handler = get_movies ) ] # Using direct functions (auto-configuration) "functions" : [get_movies] ​ pre_actions List[dict] Actions that execute before the LLM inference. For example, you can send a message to the TTS to speak a phrase (e.g. “Hold on a moment…”), which may be effective if an LLM function call takes time to execute. Copy Ask AI # Example pre_actions "pre_actions" : [ { "type" : "tts_say" , "text" : "Hold on a moment..." } ], ​ post_actions List[dict] Actions that execute after the LLM inference. For example, you can end the conversation. Copy Ask AI # Example post_actions "post_actions" : [ { "type" : "end_conversation" } ] ​ respond_immediately bool If set to False , the LLM will not respond immediately when the node is set, but will instead wait for the user to speak first before responding. Defaults to True . Copy Ask AI # Example usage "respond_immediately" : False ​ Function Handler Types ​ LegacyFunctionHandler Callable[[FlowArgs], Awaitable[FlowResult | ConsolidatedFunctionResult]] Legacy function handler that only receives arguments. Returns either: A FlowResult (⚠️ deprecated) A “consolidated” result tuple (result, next node) where: result is an optional FlowResult next node is an optional NodeConfig (for dynamic flows) or string (for static flows) ​ FlowFunctionHandler Callable[[FlowArgs, FlowManager], Awaitable[FlowResult | ConsolidatedFunctionResult]] Modern function handler that receives both arguments and FlowManager . Returns either: A FlowResult (⚠️ deprecated) A “consolidated” result tuple (result, next node) where: result is an optional FlowResult next node is an optional NodeConfig (for dynamic flows) or string (for static flows) ​ DirectFunction DirectFunction Function that is meant to be passed directly into a NodeConfig rather than into the handler field of a function configuration. It must be an async function with flow_manager: FlowManager as its first parameter. It must return a ConsolidatedFunctionResult , which is a tuple (result, next node) where: result is an optional FlowResult next node is an optional NodeConfig (for dynamic flows) or string (for static flows) ​ ContextStrategy ​ ContextStrategy Enum Strategy for managing conversation context during node transitions. Show Values ​ APPEND str Default strategy. Adds new messages to existing context. ​ RESET str Clears context and starts fresh with new messages. ​ RESET_WITH_SUMMARY str Resets context but includes an AI-generated summary. ​ ContextStrategyConfig ​ ContextStrategyConfig dataclass Configuration for context management strategy. Show Fields ​ strategy ContextStrategy required The strategy to use for context management ​ summary_prompt Optional[str] Required when using RESET_WITH_SUMMARY. Prompt text for generating the conversation summary. Copy Ask AI # Example usage config = ContextStrategyConfig( strategy = ContextStrategy. RESET_WITH_SUMMARY , summary_prompt = "Summarize the key points discussed so far." ) ​ FlowsFunctionSchema ​ FlowsFunctionSchema class A standardized schema for defining functions in Pipecat Flows with flow-specific properties. Show Constructor Parameters ​ name str required Name of the function ​ description str required Description of the function’s purpose ​ properties Dict[str, Any] required Dictionary defining properties types and descriptions ​ required List[str] required List of required parameter names ​ handler Optional[FunctionHandler] Function handler to process the function call ​ transition_to Optional[str] deprecated Target node to transition to after function execution Deprecated: instead of transition_to , use a “consolidated” handler that returns a tuple (result, next node). ​ transition_callback Optional[Callable] deprecated Callback function for dynamic transitions Deprecated: instead of transition_callback , use a “consolidated” handler that returns a tuple (result, next node). You cannot specify both transition_to and transition_callback in the same function schema. Example usage: Copy Ask AI from pipecat_flows import FlowsFunctionSchema # Define a function schema collect_name_schema = FlowsFunctionSchema( name = "collect_name" , description = "Record the user's name" , properties = { "name" : { "type" : "string" , "description" : "The user's name" } }, required = [ "name" ], handler = collect_name_handler ) # Use in node configuration node_config = { "name" : "greeting" , "task_messages" : [ { "role" : "system" , "content" : "Ask the user for their name." } ], "functions" : [collect_name_schema] } # Pass to flow manager await flow_manager.set_node_from_config(node_config) ​ FlowManager ​ FlowManager class Main class for managing conversation flows, supporting both static (configuration-driven) and dynamic (runtime-determined) flows. Show Constructor Parameters ​ task PipelineTask required Pipeline task for frame queueing ​ llm LLMService required LLM service instance (OpenAI, Anthropic, or Google). Must be initialized with the corresponding pipecat-ai provider dependency installed. ​ context_aggregator Any required Context aggregator used for pushing messages to the LLM service ​ tts Optional[Any] deprecated Optional TTS service for voice actions. Deprecated: No need to explicitly pass tts to FlowManager in order to use tts_say actions. ​ flow_config Optional[FlowConfig] Optional static flow configuration ​ context_strategy Optional[ContextStrategyConfig] Optional configuration for how context should be managed during transitions. Defaults to APPEND strategy if not specified. ​ Methods ​ initialize method Initialize the flow with starting messages. Show Parameters ​ initial_node NodeConfig The initial conversation node (needed for dynamic flows only). If not specified, you’ll need to call set_node_from_config() to kick off the conversation. Show Raises ​ FlowInitializationError If initialization fails ​ set_node method deprecated Set up a new conversation node programmatically (dynamic flows only). In dynamic flows, the application advances the conversation using set_node to set up each next node. In static flows, set_node is triggered under the hood when a node contains a transition_to field. Deprecated: use the following patterns instead of set_node : Prefer “consolidated” function handlers that return a tuple (result, next node), which implicitly sets up the next node Prefer passing your initial node to FlowManager.initialize() If you really need to set a node explicitly, use set_node_from_config() (note: its name will be read from its NodeConfig ) Show Parameters ​ node_id str required Identifier for the new node ​ node_config NodeConfig required Node configuration including messages, functions, and actions Show Raises ​ FlowError If node setup fails ​ set_node_from_config method Set up a new conversation node programmatically (dynamic flows only). Note that this method should only be used in rare circumstances. Most often, you should: Prefer “consolidated” function handlers that return a tuple (result, next node), which implicitly sets up the next node Prefer passing your initial node to FlowManager.initialize() Show Parameters ​ node_config NodeConfig required Node configuration including messages, functions, and actions Show Raises ​ FlowError If node setup fails ​ register_action method Register a handler for a custom action type. Show Parameters ​ action_type str required String identifier for the action ​ handler Callable required Async or sync function that handles the action ​ get_current_context method Get the current conversation context. Returns a list of messages in the current context, including system messages, user messages, and assistant responses. Show Returns ​ messages List[dict] List of messages in the current context Show Raises ​ FlowError If context aggregator is not available Example usage: Copy Ask AI # Access current conversation context context = flow_manager.get_current_context() # Use in handlers async def process_response ( args : FlowArgs) -> tuple[FlowResult, str ]: context = flow_manager.get_current_context() # Process conversation history return { "status" : "success" }, "next" ​ State Management The FlowManager provides a state dictionary for storing conversation data: Access state Access in transitions Copy Ask AI flow_manager.state: Dict[ str , Any] # Store data flow_manager.state[ "user_age" ] = 25 ​ Usage Examples Static Flow Dynamic Flow Copy Ask AI flow_config: FlowConfig = { "initial_node" : "greeting" , "nodes" : { "greeting" : { "role_messages" : [ { "role" : "system" , "content" : "You are a helpful assistant. Your responses will be converted to audio." } ], "task_messages" : [ { "role" : "system" , "content" : "Start by greeting the user and asking for their name." } ], "functions" : [{ "type" : "function" , "function" : { "name" : "collect_name" , "handler" : collect_name_handler, "description" : "Record user's name" , "parameters" : { ... } } }] } } } # Create and initialize the FlowManager flow_manager = FlowManager( task = task, llm = llm, context_aggregator = context_aggregator, flow_config = flow_config ) # Initialize the flow_manager to start the conversation await flow_manager.initialize() ​ Node Functions concept Functions that execute operations within a single conversational state, without switching nodes. Copy Ask AI Copy Ask AI from pipecat_flows import FlowArgs, FlowResult async def process_data ( args : FlowArgs) -> tuple[FlowResult, None ]: """Handle data processing within a node.""" data = args[ "data" ] result = await process(data) return { "status" : "success" , "processed_data" : result }, None # Function configuration { "type" : "function" , "function" : { "name" : "process_data" , "handler" : process_data, "description" : "Process user data" , "parameters" : { "type" : "object" , "properties" : { "data" : { "type" : "string" } } } } } ​ Edge Functions concept Functions that specify a transition between nodes (optionally processing data first). Copy Ask AI Copy Ask AI from pipecat_flows import FlowArgs, FlowResult async def next_step ( args : FlowArgs) -> tuple[ None , str ]: """Specify the next node to transition to.""" return None , "target_node" # Return NodeConfig instead of str for dynamic flows # Function configuration { "type" : "function" , "function" : { "name" : "next_step" , "handler" : next_step, "description" : "Transition to next node" , "parameters" : { "type" : "object" , "properties" : {}} } } ​ Function Properties ​ handler Optional[Callable] Async function that processes data within a node and/or specifies the next node ( more details here ). Can be specified as: Direct function reference Either a Callable function or a string with __function__: prefix (e.g., "__function__:process_data" ) to reference a function in the main script Direct Reference Function Token Copy Ask AI { "type" : "function" , "function" : { "name" : "process_data" , "handler" : process_data, # Callable function "parameters" : { ... } } } ​ transition_callback Optional[Callable] deprecated Handler for dynamic flow transitions. Deprecated: instead of transition_callback , use a “consolidated” handler that returns a tuple (result, next node). Must be an async function with one of these signatures: Copy Ask AI # New style (recommended) async def handle_transition ( args : Dict[ str , Any], result : FlowResult, flow_manager : FlowManager ) -> None : """Handle transition to next node.""" if result.available: # Type-safe access to result await flow_manager.set_node_from_config(create_confirmation_node()) else : await flow_manager.set_node_from_config( create_no_availability_node(result.alternative_times) ) # Legacy style (supported for backwards compatibility) async def handle_transition ( args : Dict[ str , Any], flow_manager : FlowManager ) -> None : """Handle transition to next node.""" await flow_manager.set_node_from_config(create_next_node()) The callback receives: args : Arguments from the function call result : Typed result from the function handler (new style only) flow_manager : Reference to the FlowManager instance Example usage: Copy Ask AI async def handle_availability_check ( args : Dict, result : TimeResult, # Typed result flow_manager : FlowManager ): """Handle availability check and transition based on result.""" if result.available: await flow_manager.set_node_from_config(create_confirmation_node()) else : await flow_manager.set_node_from_config( create_no_availability_node(result.alternative_times) ) # Use in function configuration { "type" : "function" , "function" : { "name" : "check_availability" , "handler" : check_availability, "parameters" : { ... }, "transition_callback" : handle_availability_check } } Note: A function cannot have both transition_to and transition_callback . ​ Handler Signatures Function handlers passed as a handler in a function configuration can be defined with three different signatures: Modern (Args + FlowManager) Legacy (Args Only) No Arguments Copy Ask AI async def handler_with_flow_manager ( args : FlowArgs, flow_manager : FlowManager) -> tuple[FlowResult, NodeConfig]: """Modern handler that receives both arguments and FlowManager access.""" # Access state previous_data = flow_manager.state.get( "stored_data" ) # Access pipeline resources await flow_manager.task.queue_frame(TTSSpeakFrame( "Processing your request..." )) # Store data in state for later flow_manager.state[ "new_data" ] = args[ "input" ] return { "status" : "success" , "result" : "Processed with flow access" }, create_next_node() The framework automatically detects which signature your handler is using and calls it appropriately. If you’re passing your function directly into your NodeConfig rather than as a handler in a function configuration, you’d use a somewhat different signature: Direct Copy Ask AI async def do_something ( flow_manager : FlowManager, foo : int , bar : str = "" ) -> tuple[FlowResult, NodeConfig]: """ Do something interesting. Args: foo (int): The foo to do something interesting with. bar (string): The bar to do something interesting with. Defaults to empty string. """ result = await fetch_data(foo, bar) next_node = create_end_node() return result, next_node ​ Return Types Success Response Error Response Copy Ask AI { "status" : "success" , "data" : "some data" # Optional additional data } ​ Provider-Specific Formats You don’t need to handle these format differences manually - use the standard format and the FlowManager will adapt it for your chosen provider. OpenAI Anthropic Google (Gemini) Copy Ask AI { "type" : "function" , "function" : { "name" : "function_name" , "handler" : handler, "description" : "Description" , "parameters" : { ... } } } ​ Actions pre_actions and post_actions are used to manage conversation flow. They are included in the NodeConfig and executed before and after the LLM completion, respectively. Three kinds of actions are available: Pre-canned actions: These actions perform common tasks and require little configuration. Function actions: These actions run developer-defined functions at the appropriate time. Custom actions: These are fully developer-defined actions, providing flexibility at the expense of complexity. ​ Pre-canned Actions Common actions shipped with Flows for managing conversation flow. To use them, just add them to your NodeConfig . ​ tts_say action Speaks text immediately using the TTS service. Copy Ask AI Copy Ask AI "pre_actions" : [ { "type" : "tts_say" , "text" : "Processing your request..." # Required } ] ​ end_conversation action Ends the conversation and closes the connection. Copy Ask AI Copy Ask AI "post_actions" : [ { "type" : "end_conversation" , "text" : "Goodbye!" # Optional farewell message } ] ​ Function Actions Actions that run developer-defined functions at the appropriate time. For example, if used in post_actions , they’ll run after the bot has finished talking and after any previous post_actions have finished. ​ function action Calls the developer-defined function at the appropriate time. Copy Ask AI Copy Ask AI "post_actions" : [ { "type" : "function" , "handler" : bot_turn_ended # Required } ] ​ Custom Actions Fully developer-defined actions, providing flexibility at the expense of complexity. Here’s the complexity: because these actions aren’t queued in the Pipecat pipeline, they may execute seemingly early if used in post_actions ; they’ll run immediately after the LLM completion is triggered but won’t wait around for the bot to finish talking. Why would you want this behavior? You might be writing an action that: Itself just queues another Frame into the Pipecat pipeline (meaning there would no benefit to waiting around for sequencing purposes) Does work that can be done a bit sooner, like logging that the LLM was updated Custom actions are composed of at least: ​ type str required String identifier for the action ​ handler Callable required Async or sync function that handles the action Example: Copy Ask AI Copy Ask AI # Define custom action handler async def custom_notification ( action : dict , flow_manager : FlowManager): """Custom action handler.""" message = action.get( "message" , "" ) await notify_user(message) # Use in node configuration "pre_actions" : [ { "type" : "notify" , "handler" : send_notification, "message" : "Attention!" , } ] ​ Exceptions ​ FlowError exception Base exception for all flow-related errors. Copy Ask AI Copy Ask AI from pipecat_flows import FlowError try : await flow_manager.set_node_from_config(config) except FlowError as e: print ( f "Flow error: { e } " ) ​ FlowInitializationError exception Raised when flow initialization fails. Copy Ask AI Copy Ask AI from pipecat_flows import FlowInitializationError try : await flow_manager.initialize() except FlowInitializationError as e: print ( f "Initialization failed: { e } " ) ​ FlowTransitionError exception Raised when a state transition fails. Copy Ask AI Copy Ask AI from pipecat_flows import FlowTransitionError try : await flow_manager.set_node_from_config(node_config) except FlowTransitionError as e: print ( f "Transition failed: { e } " ) ​ InvalidFunctionError exception Raised when an invalid or unavailable function is specified. Copy Ask AI Copy Ask AI from pipecat_flows import InvalidFunctionError try : await flow_manager.set_node_from_config({ "functions" : [{ "type" : "function" , "function" : { "name" : "invalid_function" } }] }) except InvalidFunctionError as e: print ( f "Invalid function: { e } " ) RTVI Observer PipelineParams On this page Installation Core Types FlowArgs FlowResult FlowConfig NodeConfig Function Handler Types ContextStrategy ContextStrategyConfig FlowsFunctionSchema FlowManager Methods State Management Usage Examples Function Properties Handler Signatures Return Types Provider-Specific Formats Actions Pre-canned Actions Function Actions Custom Actions Exceptions Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/flows_pipecat-flows_b9ab2790.txt b/flows_pipecat-flows_b9ab2790.txt
new file mode 100644
index 0000000000000000000000000000000000000000..013d848590907fb53e8d55b8ec79b03a776a26b6
--- /dev/null
+++ b/flows_pipecat-flows_b9ab2790.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/frameworks/flows/pipecat-flows#param-transition-callback-1
+Title: Pipecat Flows - Pipecat
+==================================================
+
+Pipecat Flows - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Frameworks Pipecat Flows Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline New to building conversational flows? Check out our Pipecat Flows guide first. ​ Installation Existing Pipecat installation Fresh Pipecat installation Copy Ask AI pip install pipecat-ai-flows ​ Core Types ​ FlowArgs ​ FlowArgs Dict[str, Any] Type alias for function handler arguments. ​ FlowResult ​ FlowResult TypedDict Base type for function handler results. Additional fields can be included as needed. Show Fields ​ status str Optional status field ​ error str Optional error message ​ FlowConfig ​ FlowConfig TypedDict Configuration for the entire conversation flow. Show Fields ​ initial_node str required Starting node identifier ​ nodes Dict[str, NodeConfig] required Map of node names to configurations ​ NodeConfig ​ NodeConfig TypedDict Configuration for a single node in the flow. Show Fields ​ name str The name of the node, used in debug logging in dynamic flows. If no name is specified, an automatically-generated UUID is used. Copy Ask AI # Example name "name" : "greeting" ​ role_messages List[dict] Defines the role or persona of the LLM. Required for the initial node and optional for subsequent nodes. Copy Ask AI # Example role messages "role_messages" : [ { "role" : "system" , "content" : "You are a helpful assistant..." } ], ​ task_messages List[dict] required Defines the task for a given node. Required for all nodes. Copy Ask AI # Example task messages "task_messages" : [ { "role" : "system" , # May be `user` depending on the LLM "content" : "Ask the user for their name..." } ], ​ context_strategy ContextStrategyConfig Strategy for managing context during transitions to this node. Copy Ask AI # Example context strategy configuration "context_strategy" : ContextStrategyConfig( strategy = ContextStrategy. RESET_WITH_SUMMARY , summary_prompt = "Summarize the key points discussed so far." ) ​ functions List[Union[dict, FlowsFunctionSchema]] required LLM function / tool call configurations, defined in one of the supported formats . Copy Ask AI # Using provider-specific dictionary format "functions" : [ { "type" : "function" , "function" : { "name" : "get_current_movies" , "handler" : get_movies, "description" : "Fetch movies currently playing" , "parameters" : { ... } }, } ] # Using FlowsFunctionSchema "functions" : [ FlowsFunctionSchema( name = "get_current_movies" , description = "Fetch movies currently playing" , properties = { ... }, required = [ ... ], handler = get_movies ) ] # Using direct functions (auto-configuration) "functions" : [get_movies] ​ pre_actions List[dict] Actions that execute before the LLM inference. For example, you can send a message to the TTS to speak a phrase (e.g. “Hold on a moment…”), which may be effective if an LLM function call takes time to execute. Copy Ask AI # Example pre_actions "pre_actions" : [ { "type" : "tts_say" , "text" : "Hold on a moment..." } ], ​ post_actions List[dict] Actions that execute after the LLM inference. For example, you can end the conversation. Copy Ask AI # Example post_actions "post_actions" : [ { "type" : "end_conversation" } ] ​ respond_immediately bool If set to False , the LLM will not respond immediately when the node is set, but will instead wait for the user to speak first before responding. Defaults to True . Copy Ask AI # Example usage "respond_immediately" : False ​ Function Handler Types ​ LegacyFunctionHandler Callable[[FlowArgs], Awaitable[FlowResult | ConsolidatedFunctionResult]] Legacy function handler that only receives arguments. Returns either: A FlowResult (⚠️ deprecated) A “consolidated” result tuple (result, next node) where: result is an optional FlowResult next node is an optional NodeConfig (for dynamic flows) or string (for static flows) ​ FlowFunctionHandler Callable[[FlowArgs, FlowManager], Awaitable[FlowResult | ConsolidatedFunctionResult]] Modern function handler that receives both arguments and FlowManager . Returns either: A FlowResult (⚠️ deprecated) A “consolidated” result tuple (result, next node) where: result is an optional FlowResult next node is an optional NodeConfig (for dynamic flows) or string (for static flows) ​ DirectFunction DirectFunction Function that is meant to be passed directly into a NodeConfig rather than into the handler field of a function configuration. It must be an async function with flow_manager: FlowManager as its first parameter. It must return a ConsolidatedFunctionResult , which is a tuple (result, next node) where: result is an optional FlowResult next node is an optional NodeConfig (for dynamic flows) or string (for static flows) ​ ContextStrategy ​ ContextStrategy Enum Strategy for managing conversation context during node transitions. Show Values ​ APPEND str Default strategy. Adds new messages to existing context. ​ RESET str Clears context and starts fresh with new messages. ​ RESET_WITH_SUMMARY str Resets context but includes an AI-generated summary. ​ ContextStrategyConfig ​ ContextStrategyConfig dataclass Configuration for context management strategy. Show Fields ​ strategy ContextStrategy required The strategy to use for context management ​ summary_prompt Optional[str] Required when using RESET_WITH_SUMMARY. Prompt text for generating the conversation summary. Copy Ask AI # Example usage config = ContextStrategyConfig( strategy = ContextStrategy. RESET_WITH_SUMMARY , summary_prompt = "Summarize the key points discussed so far." ) ​ FlowsFunctionSchema ​ FlowsFunctionSchema class A standardized schema for defining functions in Pipecat Flows with flow-specific properties. Show Constructor Parameters ​ name str required Name of the function ​ description str required Description of the function’s purpose ​ properties Dict[str, Any] required Dictionary defining properties types and descriptions ​ required List[str] required List of required parameter names ​ handler Optional[FunctionHandler] Function handler to process the function call ​ transition_to Optional[str] deprecated Target node to transition to after function execution Deprecated: instead of transition_to , use a “consolidated” handler that returns a tuple (result, next node). ​ transition_callback Optional[Callable] deprecated Callback function for dynamic transitions Deprecated: instead of transition_callback , use a “consolidated” handler that returns a tuple (result, next node). You cannot specify both transition_to and transition_callback in the same function schema. Example usage: Copy Ask AI from pipecat_flows import FlowsFunctionSchema # Define a function schema collect_name_schema = FlowsFunctionSchema( name = "collect_name" , description = "Record the user's name" , properties = { "name" : { "type" : "string" , "description" : "The user's name" } }, required = [ "name" ], handler = collect_name_handler ) # Use in node configuration node_config = { "name" : "greeting" , "task_messages" : [ { "role" : "system" , "content" : "Ask the user for their name." } ], "functions" : [collect_name_schema] } # Pass to flow manager await flow_manager.set_node_from_config(node_config) ​ FlowManager ​ FlowManager class Main class for managing conversation flows, supporting both static (configuration-driven) and dynamic (runtime-determined) flows. Show Constructor Parameters ​ task PipelineTask required Pipeline task for frame queueing ​ llm LLMService required LLM service instance (OpenAI, Anthropic, or Google). Must be initialized with the corresponding pipecat-ai provider dependency installed. ​ context_aggregator Any required Context aggregator used for pushing messages to the LLM service ​ tts Optional[Any] deprecated Optional TTS service for voice actions. Deprecated: No need to explicitly pass tts to FlowManager in order to use tts_say actions. ​ flow_config Optional[FlowConfig] Optional static flow configuration ​ context_strategy Optional[ContextStrategyConfig] Optional configuration for how context should be managed during transitions. Defaults to APPEND strategy if not specified. ​ Methods ​ initialize method Initialize the flow with starting messages. Show Parameters ​ initial_node NodeConfig The initial conversation node (needed for dynamic flows only). If not specified, you’ll need to call set_node_from_config() to kick off the conversation. Show Raises ​ FlowInitializationError If initialization fails ​ set_node method deprecated Set up a new conversation node programmatically (dynamic flows only). In dynamic flows, the application advances the conversation using set_node to set up each next node. In static flows, set_node is triggered under the hood when a node contains a transition_to field. Deprecated: use the following patterns instead of set_node : Prefer “consolidated” function handlers that return a tuple (result, next node), which implicitly sets up the next node Prefer passing your initial node to FlowManager.initialize() If you really need to set a node explicitly, use set_node_from_config() (note: its name will be read from its NodeConfig ) Show Parameters ​ node_id str required Identifier for the new node ​ node_config NodeConfig required Node configuration including messages, functions, and actions Show Raises ​ FlowError If node setup fails ​ set_node_from_config method Set up a new conversation node programmatically (dynamic flows only). Note that this method should only be used in rare circumstances. Most often, you should: Prefer “consolidated” function handlers that return a tuple (result, next node), which implicitly sets up the next node Prefer passing your initial node to FlowManager.initialize() Show Parameters ​ node_config NodeConfig required Node configuration including messages, functions, and actions Show Raises ​ FlowError If node setup fails ​ register_action method Register a handler for a custom action type. Show Parameters ​ action_type str required String identifier for the action ​ handler Callable required Async or sync function that handles the action ​ get_current_context method Get the current conversation context. Returns a list of messages in the current context, including system messages, user messages, and assistant responses. Show Returns ​ messages List[dict] List of messages in the current context Show Raises ​ FlowError If context aggregator is not available Example usage: Copy Ask AI # Access current conversation context context = flow_manager.get_current_context() # Use in handlers async def process_response ( args : FlowArgs) -> tuple[FlowResult, str ]: context = flow_manager.get_current_context() # Process conversation history return { "status" : "success" }, "next" ​ State Management The FlowManager provides a state dictionary for storing conversation data: Access state Access in transitions Copy Ask AI flow_manager.state: Dict[ str , Any] # Store data flow_manager.state[ "user_age" ] = 25 ​ Usage Examples Static Flow Dynamic Flow Copy Ask AI flow_config: FlowConfig = { "initial_node" : "greeting" , "nodes" : { "greeting" : { "role_messages" : [ { "role" : "system" , "content" : "You are a helpful assistant. Your responses will be converted to audio." } ], "task_messages" : [ { "role" : "system" , "content" : "Start by greeting the user and asking for their name." } ], "functions" : [{ "type" : "function" , "function" : { "name" : "collect_name" , "handler" : collect_name_handler, "description" : "Record user's name" , "parameters" : { ... } } }] } } } # Create and initialize the FlowManager flow_manager = FlowManager( task = task, llm = llm, context_aggregator = context_aggregator, flow_config = flow_config ) # Initialize the flow_manager to start the conversation await flow_manager.initialize() ​ Node Functions concept Functions that execute operations within a single conversational state, without switching nodes. Copy Ask AI Copy Ask AI from pipecat_flows import FlowArgs, FlowResult async def process_data ( args : FlowArgs) -> tuple[FlowResult, None ]: """Handle data processing within a node.""" data = args[ "data" ] result = await process(data) return { "status" : "success" , "processed_data" : result }, None # Function configuration { "type" : "function" , "function" : { "name" : "process_data" , "handler" : process_data, "description" : "Process user data" , "parameters" : { "type" : "object" , "properties" : { "data" : { "type" : "string" } } } } } ​ Edge Functions concept Functions that specify a transition between nodes (optionally processing data first). Copy Ask AI Copy Ask AI from pipecat_flows import FlowArgs, FlowResult async def next_step ( args : FlowArgs) -> tuple[ None , str ]: """Specify the next node to transition to.""" return None , "target_node" # Return NodeConfig instead of str for dynamic flows # Function configuration { "type" : "function" , "function" : { "name" : "next_step" , "handler" : next_step, "description" : "Transition to next node" , "parameters" : { "type" : "object" , "properties" : {}} } } ​ Function Properties ​ handler Optional[Callable] Async function that processes data within a node and/or specifies the next node ( more details here ). Can be specified as: Direct function reference Either a Callable function or a string with __function__: prefix (e.g., "__function__:process_data" ) to reference a function in the main script Direct Reference Function Token Copy Ask AI { "type" : "function" , "function" : { "name" : "process_data" , "handler" : process_data, # Callable function "parameters" : { ... } } } ​ transition_callback Optional[Callable] deprecated Handler for dynamic flow transitions. Deprecated: instead of transition_callback , use a “consolidated” handler that returns a tuple (result, next node). Must be an async function with one of these signatures: Copy Ask AI # New style (recommended) async def handle_transition ( args : Dict[ str , Any], result : FlowResult, flow_manager : FlowManager ) -> None : """Handle transition to next node.""" if result.available: # Type-safe access to result await flow_manager.set_node_from_config(create_confirmation_node()) else : await flow_manager.set_node_from_config( create_no_availability_node(result.alternative_times) ) # Legacy style (supported for backwards compatibility) async def handle_transition ( args : Dict[ str , Any], flow_manager : FlowManager ) -> None : """Handle transition to next node.""" await flow_manager.set_node_from_config(create_next_node()) The callback receives: args : Arguments from the function call result : Typed result from the function handler (new style only) flow_manager : Reference to the FlowManager instance Example usage: Copy Ask AI async def handle_availability_check ( args : Dict, result : TimeResult, # Typed result flow_manager : FlowManager ): """Handle availability check and transition based on result.""" if result.available: await flow_manager.set_node_from_config(create_confirmation_node()) else : await flow_manager.set_node_from_config( create_no_availability_node(result.alternative_times) ) # Use in function configuration { "type" : "function" , "function" : { "name" : "check_availability" , "handler" : check_availability, "parameters" : { ... }, "transition_callback" : handle_availability_check } } Note: A function cannot have both transition_to and transition_callback . ​ Handler Signatures Function handlers passed as a handler in a function configuration can be defined with three different signatures: Modern (Args + FlowManager) Legacy (Args Only) No Arguments Copy Ask AI async def handler_with_flow_manager ( args : FlowArgs, flow_manager : FlowManager) -> tuple[FlowResult, NodeConfig]: """Modern handler that receives both arguments and FlowManager access.""" # Access state previous_data = flow_manager.state.get( "stored_data" ) # Access pipeline resources await flow_manager.task.queue_frame(TTSSpeakFrame( "Processing your request..." )) # Store data in state for later flow_manager.state[ "new_data" ] = args[ "input" ] return { "status" : "success" , "result" : "Processed with flow access" }, create_next_node() The framework automatically detects which signature your handler is using and calls it appropriately. If you’re passing your function directly into your NodeConfig rather than as a handler in a function configuration, you’d use a somewhat different signature: Direct Copy Ask AI async def do_something ( flow_manager : FlowManager, foo : int , bar : str = "" ) -> tuple[FlowResult, NodeConfig]: """ Do something interesting. Args: foo (int): The foo to do something interesting with. bar (string): The bar to do something interesting with. Defaults to empty string. """ result = await fetch_data(foo, bar) next_node = create_end_node() return result, next_node ​ Return Types Success Response Error Response Copy Ask AI { "status" : "success" , "data" : "some data" # Optional additional data } ​ Provider-Specific Formats You don’t need to handle these format differences manually - use the standard format and the FlowManager will adapt it for your chosen provider. OpenAI Anthropic Google (Gemini) Copy Ask AI { "type" : "function" , "function" : { "name" : "function_name" , "handler" : handler, "description" : "Description" , "parameters" : { ... } } } ​ Actions pre_actions and post_actions are used to manage conversation flow. They are included in the NodeConfig and executed before and after the LLM completion, respectively. Three kinds of actions are available: Pre-canned actions: These actions perform common tasks and require little configuration. Function actions: These actions run developer-defined functions at the appropriate time. Custom actions: These are fully developer-defined actions, providing flexibility at the expense of complexity. ​ Pre-canned Actions Common actions shipped with Flows for managing conversation flow. To use them, just add them to your NodeConfig . ​ tts_say action Speaks text immediately using the TTS service. Copy Ask AI Copy Ask AI "pre_actions" : [ { "type" : "tts_say" , "text" : "Processing your request..." # Required } ] ​ end_conversation action Ends the conversation and closes the connection. Copy Ask AI Copy Ask AI "post_actions" : [ { "type" : "end_conversation" , "text" : "Goodbye!" # Optional farewell message } ] ​ Function Actions Actions that run developer-defined functions at the appropriate time. For example, if used in post_actions , they’ll run after the bot has finished talking and after any previous post_actions have finished. ​ function action Calls the developer-defined function at the appropriate time. Copy Ask AI Copy Ask AI "post_actions" : [ { "type" : "function" , "handler" : bot_turn_ended # Required } ] ​ Custom Actions Fully developer-defined actions, providing flexibility at the expense of complexity. Here’s the complexity: because these actions aren’t queued in the Pipecat pipeline, they may execute seemingly early if used in post_actions ; they’ll run immediately after the LLM completion is triggered but won’t wait around for the bot to finish talking. Why would you want this behavior? You might be writing an action that: Itself just queues another Frame into the Pipecat pipeline (meaning there would no benefit to waiting around for sequencing purposes) Does work that can be done a bit sooner, like logging that the LLM was updated Custom actions are composed of at least: ​ type str required String identifier for the action ​ handler Callable required Async or sync function that handles the action Example: Copy Ask AI Copy Ask AI # Define custom action handler async def custom_notification ( action : dict , flow_manager : FlowManager): """Custom action handler.""" message = action.get( "message" , "" ) await notify_user(message) # Use in node configuration "pre_actions" : [ { "type" : "notify" , "handler" : send_notification, "message" : "Attention!" , } ] ​ Exceptions ​ FlowError exception Base exception for all flow-related errors. Copy Ask AI Copy Ask AI from pipecat_flows import FlowError try : await flow_manager.set_node_from_config(config) except FlowError as e: print ( f "Flow error: { e } " ) ​ FlowInitializationError exception Raised when flow initialization fails. Copy Ask AI Copy Ask AI from pipecat_flows import FlowInitializationError try : await flow_manager.initialize() except FlowInitializationError as e: print ( f "Initialization failed: { e } " ) ​ FlowTransitionError exception Raised when a state transition fails. Copy Ask AI Copy Ask AI from pipecat_flows import FlowTransitionError try : await flow_manager.set_node_from_config(node_config) except FlowTransitionError as e: print ( f "Transition failed: { e } " ) ​ InvalidFunctionError exception Raised when an invalid or unavailable function is specified. Copy Ask AI Copy Ask AI from pipecat_flows import InvalidFunctionError try : await flow_manager.set_node_from_config({ "functions" : [{ "type" : "function" , "function" : { "name" : "invalid_function" } }] }) except InvalidFunctionError as e: print ( f "Invalid function: { e } " ) RTVI Observer PipelineParams On this page Installation Core Types FlowArgs FlowResult FlowConfig NodeConfig Function Handler Types ContextStrategy ContextStrategyConfig FlowsFunctionSchema FlowManager Methods State Management Usage Examples Function Properties Handler Signatures Return Types Provider-Specific Formats Actions Pre-canned Actions Function Actions Custom Actions Exceptions Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/flows_pipecat-flows_ce044d94.txt b/flows_pipecat-flows_ce044d94.txt
new file mode 100644
index 0000000000000000000000000000000000000000..74024ad0ea5051032e39d5cc26c1c4410ce6b184
--- /dev/null
+++ b/flows_pipecat-flows_ce044d94.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/frameworks/flows/pipecat-flows#installation
+Title: Pipecat Flows - Pipecat
+==================================================
+
+Pipecat Flows - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Frameworks Pipecat Flows Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline New to building conversational flows? Check out our Pipecat Flows guide first. ​ Installation Existing Pipecat installation Fresh Pipecat installation Copy Ask AI pip install pipecat-ai-flows ​ Core Types ​ FlowArgs ​ FlowArgs Dict[str, Any] Type alias for function handler arguments. ​ FlowResult ​ FlowResult TypedDict Base type for function handler results. Additional fields can be included as needed. Show Fields ​ status str Optional status field ​ error str Optional error message ​ FlowConfig ​ FlowConfig TypedDict Configuration for the entire conversation flow. Show Fields ​ initial_node str required Starting node identifier ​ nodes Dict[str, NodeConfig] required Map of node names to configurations ​ NodeConfig ​ NodeConfig TypedDict Configuration for a single node in the flow. Show Fields ​ name str The name of the node, used in debug logging in dynamic flows. If no name is specified, an automatically-generated UUID is used. Copy Ask AI # Example name "name" : "greeting" ​ role_messages List[dict] Defines the role or persona of the LLM. Required for the initial node and optional for subsequent nodes. Copy Ask AI # Example role messages "role_messages" : [ { "role" : "system" , "content" : "You are a helpful assistant..." } ], ​ task_messages List[dict] required Defines the task for a given node. Required for all nodes. Copy Ask AI # Example task messages "task_messages" : [ { "role" : "system" , # May be `user` depending on the LLM "content" : "Ask the user for their name..." } ], ​ context_strategy ContextStrategyConfig Strategy for managing context during transitions to this node. Copy Ask AI # Example context strategy configuration "context_strategy" : ContextStrategyConfig( strategy = ContextStrategy. RESET_WITH_SUMMARY , summary_prompt = "Summarize the key points discussed so far." ) ​ functions List[Union[dict, FlowsFunctionSchema]] required LLM function / tool call configurations, defined in one of the supported formats . Copy Ask AI # Using provider-specific dictionary format "functions" : [ { "type" : "function" , "function" : { "name" : "get_current_movies" , "handler" : get_movies, "description" : "Fetch movies currently playing" , "parameters" : { ... } }, } ] # Using FlowsFunctionSchema "functions" : [ FlowsFunctionSchema( name = "get_current_movies" , description = "Fetch movies currently playing" , properties = { ... }, required = [ ... ], handler = get_movies ) ] # Using direct functions (auto-configuration) "functions" : [get_movies] ​ pre_actions List[dict] Actions that execute before the LLM inference. For example, you can send a message to the TTS to speak a phrase (e.g. “Hold on a moment…”), which may be effective if an LLM function call takes time to execute. Copy Ask AI # Example pre_actions "pre_actions" : [ { "type" : "tts_say" , "text" : "Hold on a moment..." } ], ​ post_actions List[dict] Actions that execute after the LLM inference. For example, you can end the conversation. Copy Ask AI # Example post_actions "post_actions" : [ { "type" : "end_conversation" } ] ​ respond_immediately bool If set to False , the LLM will not respond immediately when the node is set, but will instead wait for the user to speak first before responding. Defaults to True . Copy Ask AI # Example usage "respond_immediately" : False ​ Function Handler Types ​ LegacyFunctionHandler Callable[[FlowArgs], Awaitable[FlowResult | ConsolidatedFunctionResult]] Legacy function handler that only receives arguments. Returns either: A FlowResult (⚠️ deprecated) A “consolidated” result tuple (result, next node) where: result is an optional FlowResult next node is an optional NodeConfig (for dynamic flows) or string (for static flows) ​ FlowFunctionHandler Callable[[FlowArgs, FlowManager], Awaitable[FlowResult | ConsolidatedFunctionResult]] Modern function handler that receives both arguments and FlowManager . Returns either: A FlowResult (⚠️ deprecated) A “consolidated” result tuple (result, next node) where: result is an optional FlowResult next node is an optional NodeConfig (for dynamic flows) or string (for static flows) ​ DirectFunction DirectFunction Function that is meant to be passed directly into a NodeConfig rather than into the handler field of a function configuration. It must be an async function with flow_manager: FlowManager as its first parameter. It must return a ConsolidatedFunctionResult , which is a tuple (result, next node) where: result is an optional FlowResult next node is an optional NodeConfig (for dynamic flows) or string (for static flows) ​ ContextStrategy ​ ContextStrategy Enum Strategy for managing conversation context during node transitions. Show Values ​ APPEND str Default strategy. Adds new messages to existing context. ​ RESET str Clears context and starts fresh with new messages. ​ RESET_WITH_SUMMARY str Resets context but includes an AI-generated summary. ​ ContextStrategyConfig ​ ContextStrategyConfig dataclass Configuration for context management strategy. Show Fields ​ strategy ContextStrategy required The strategy to use for context management ​ summary_prompt Optional[str] Required when using RESET_WITH_SUMMARY. Prompt text for generating the conversation summary. Copy Ask AI # Example usage config = ContextStrategyConfig( strategy = ContextStrategy. RESET_WITH_SUMMARY , summary_prompt = "Summarize the key points discussed so far." ) ​ FlowsFunctionSchema ​ FlowsFunctionSchema class A standardized schema for defining functions in Pipecat Flows with flow-specific properties. Show Constructor Parameters ​ name str required Name of the function ​ description str required Description of the function’s purpose ​ properties Dict[str, Any] required Dictionary defining properties types and descriptions ​ required List[str] required List of required parameter names ​ handler Optional[FunctionHandler] Function handler to process the function call ​ transition_to Optional[str] deprecated Target node to transition to after function execution Deprecated: instead of transition_to , use a “consolidated” handler that returns a tuple (result, next node). ​ transition_callback Optional[Callable] deprecated Callback function for dynamic transitions Deprecated: instead of transition_callback , use a “consolidated” handler that returns a tuple (result, next node). You cannot specify both transition_to and transition_callback in the same function schema. Example usage: Copy Ask AI from pipecat_flows import FlowsFunctionSchema # Define a function schema collect_name_schema = FlowsFunctionSchema( name = "collect_name" , description = "Record the user's name" , properties = { "name" : { "type" : "string" , "description" : "The user's name" } }, required = [ "name" ], handler = collect_name_handler ) # Use in node configuration node_config = { "name" : "greeting" , "task_messages" : [ { "role" : "system" , "content" : "Ask the user for their name." } ], "functions" : [collect_name_schema] } # Pass to flow manager await flow_manager.set_node_from_config(node_config) ​ FlowManager ​ FlowManager class Main class for managing conversation flows, supporting both static (configuration-driven) and dynamic (runtime-determined) flows. Show Constructor Parameters ​ task PipelineTask required Pipeline task for frame queueing ​ llm LLMService required LLM service instance (OpenAI, Anthropic, or Google). Must be initialized with the corresponding pipecat-ai provider dependency installed. ​ context_aggregator Any required Context aggregator used for pushing messages to the LLM service ​ tts Optional[Any] deprecated Optional TTS service for voice actions. Deprecated: No need to explicitly pass tts to FlowManager in order to use tts_say actions. ​ flow_config Optional[FlowConfig] Optional static flow configuration ​ context_strategy Optional[ContextStrategyConfig] Optional configuration for how context should be managed during transitions. Defaults to APPEND strategy if not specified. ​ Methods ​ initialize method Initialize the flow with starting messages. Show Parameters ​ initial_node NodeConfig The initial conversation node (needed for dynamic flows only). If not specified, you’ll need to call set_node_from_config() to kick off the conversation. Show Raises ​ FlowInitializationError If initialization fails ​ set_node method deprecated Set up a new conversation node programmatically (dynamic flows only). In dynamic flows, the application advances the conversation using set_node to set up each next node. In static flows, set_node is triggered under the hood when a node contains a transition_to field. Deprecated: use the following patterns instead of set_node : Prefer “consolidated” function handlers that return a tuple (result, next node), which implicitly sets up the next node Prefer passing your initial node to FlowManager.initialize() If you really need to set a node explicitly, use set_node_from_config() (note: its name will be read from its NodeConfig ) Show Parameters ​ node_id str required Identifier for the new node ​ node_config NodeConfig required Node configuration including messages, functions, and actions Show Raises ​ FlowError If node setup fails ​ set_node_from_config method Set up a new conversation node programmatically (dynamic flows only). Note that this method should only be used in rare circumstances. Most often, you should: Prefer “consolidated” function handlers that return a tuple (result, next node), which implicitly sets up the next node Prefer passing your initial node to FlowManager.initialize() Show Parameters ​ node_config NodeConfig required Node configuration including messages, functions, and actions Show Raises ​ FlowError If node setup fails ​ register_action method Register a handler for a custom action type. Show Parameters ​ action_type str required String identifier for the action ​ handler Callable required Async or sync function that handles the action ​ get_current_context method Get the current conversation context. Returns a list of messages in the current context, including system messages, user messages, and assistant responses. Show Returns ​ messages List[dict] List of messages in the current context Show Raises ​ FlowError If context aggregator is not available Example usage: Copy Ask AI # Access current conversation context context = flow_manager.get_current_context() # Use in handlers async def process_response ( args : FlowArgs) -> tuple[FlowResult, str ]: context = flow_manager.get_current_context() # Process conversation history return { "status" : "success" }, "next" ​ State Management The FlowManager provides a state dictionary for storing conversation data: Access state Access in transitions Copy Ask AI flow_manager.state: Dict[ str , Any] # Store data flow_manager.state[ "user_age" ] = 25 ​ Usage Examples Static Flow Dynamic Flow Copy Ask AI flow_config: FlowConfig = { "initial_node" : "greeting" , "nodes" : { "greeting" : { "role_messages" : [ { "role" : "system" , "content" : "You are a helpful assistant. Your responses will be converted to audio." } ], "task_messages" : [ { "role" : "system" , "content" : "Start by greeting the user and asking for their name." } ], "functions" : [{ "type" : "function" , "function" : { "name" : "collect_name" , "handler" : collect_name_handler, "description" : "Record user's name" , "parameters" : { ... } } }] } } } # Create and initialize the FlowManager flow_manager = FlowManager( task = task, llm = llm, context_aggregator = context_aggregator, flow_config = flow_config ) # Initialize the flow_manager to start the conversation await flow_manager.initialize() ​ Node Functions concept Functions that execute operations within a single conversational state, without switching nodes. Copy Ask AI Copy Ask AI from pipecat_flows import FlowArgs, FlowResult async def process_data ( args : FlowArgs) -> tuple[FlowResult, None ]: """Handle data processing within a node.""" data = args[ "data" ] result = await process(data) return { "status" : "success" , "processed_data" : result }, None # Function configuration { "type" : "function" , "function" : { "name" : "process_data" , "handler" : process_data, "description" : "Process user data" , "parameters" : { "type" : "object" , "properties" : { "data" : { "type" : "string" } } } } } ​ Edge Functions concept Functions that specify a transition between nodes (optionally processing data first). Copy Ask AI Copy Ask AI from pipecat_flows import FlowArgs, FlowResult async def next_step ( args : FlowArgs) -> tuple[ None , str ]: """Specify the next node to transition to.""" return None , "target_node" # Return NodeConfig instead of str for dynamic flows # Function configuration { "type" : "function" , "function" : { "name" : "next_step" , "handler" : next_step, "description" : "Transition to next node" , "parameters" : { "type" : "object" , "properties" : {}} } } ​ Function Properties ​ handler Optional[Callable] Async function that processes data within a node and/or specifies the next node ( more details here ). Can be specified as: Direct function reference Either a Callable function or a string with __function__: prefix (e.g., "__function__:process_data" ) to reference a function in the main script Direct Reference Function Token Copy Ask AI { "type" : "function" , "function" : { "name" : "process_data" , "handler" : process_data, # Callable function "parameters" : { ... } } } ​ transition_callback Optional[Callable] deprecated Handler for dynamic flow transitions. Deprecated: instead of transition_callback , use a “consolidated” handler that returns a tuple (result, next node). Must be an async function with one of these signatures: Copy Ask AI # New style (recommended) async def handle_transition ( args : Dict[ str , Any], result : FlowResult, flow_manager : FlowManager ) -> None : """Handle transition to next node.""" if result.available: # Type-safe access to result await flow_manager.set_node_from_config(create_confirmation_node()) else : await flow_manager.set_node_from_config( create_no_availability_node(result.alternative_times) ) # Legacy style (supported for backwards compatibility) async def handle_transition ( args : Dict[ str , Any], flow_manager : FlowManager ) -> None : """Handle transition to next node.""" await flow_manager.set_node_from_config(create_next_node()) The callback receives: args : Arguments from the function call result : Typed result from the function handler (new style only) flow_manager : Reference to the FlowManager instance Example usage: Copy Ask AI async def handle_availability_check ( args : Dict, result : TimeResult, # Typed result flow_manager : FlowManager ): """Handle availability check and transition based on result.""" if result.available: await flow_manager.set_node_from_config(create_confirmation_node()) else : await flow_manager.set_node_from_config( create_no_availability_node(result.alternative_times) ) # Use in function configuration { "type" : "function" , "function" : { "name" : "check_availability" , "handler" : check_availability, "parameters" : { ... }, "transition_callback" : handle_availability_check } } Note: A function cannot have both transition_to and transition_callback . ​ Handler Signatures Function handlers passed as a handler in a function configuration can be defined with three different signatures: Modern (Args + FlowManager) Legacy (Args Only) No Arguments Copy Ask AI async def handler_with_flow_manager ( args : FlowArgs, flow_manager : FlowManager) -> tuple[FlowResult, NodeConfig]: """Modern handler that receives both arguments and FlowManager access.""" # Access state previous_data = flow_manager.state.get( "stored_data" ) # Access pipeline resources await flow_manager.task.queue_frame(TTSSpeakFrame( "Processing your request..." )) # Store data in state for later flow_manager.state[ "new_data" ] = args[ "input" ] return { "status" : "success" , "result" : "Processed with flow access" }, create_next_node() The framework automatically detects which signature your handler is using and calls it appropriately. If you’re passing your function directly into your NodeConfig rather than as a handler in a function configuration, you’d use a somewhat different signature: Direct Copy Ask AI async def do_something ( flow_manager : FlowManager, foo : int , bar : str = "" ) -> tuple[FlowResult, NodeConfig]: """ Do something interesting. Args: foo (int): The foo to do something interesting with. bar (string): The bar to do something interesting with. Defaults to empty string. """ result = await fetch_data(foo, bar) next_node = create_end_node() return result, next_node ​ Return Types Success Response Error Response Copy Ask AI { "status" : "success" , "data" : "some data" # Optional additional data } ​ Provider-Specific Formats You don’t need to handle these format differences manually - use the standard format and the FlowManager will adapt it for your chosen provider. OpenAI Anthropic Google (Gemini) Copy Ask AI { "type" : "function" , "function" : { "name" : "function_name" , "handler" : handler, "description" : "Description" , "parameters" : { ... } } } ​ Actions pre_actions and post_actions are used to manage conversation flow. They are included in the NodeConfig and executed before and after the LLM completion, respectively. Three kinds of actions are available: Pre-canned actions: These actions perform common tasks and require little configuration. Function actions: These actions run developer-defined functions at the appropriate time. Custom actions: These are fully developer-defined actions, providing flexibility at the expense of complexity. ​ Pre-canned Actions Common actions shipped with Flows for managing conversation flow. To use them, just add them to your NodeConfig . ​ tts_say action Speaks text immediately using the TTS service. Copy Ask AI Copy Ask AI "pre_actions" : [ { "type" : "tts_say" , "text" : "Processing your request..." # Required } ] ​ end_conversation action Ends the conversation and closes the connection. Copy Ask AI Copy Ask AI "post_actions" : [ { "type" : "end_conversation" , "text" : "Goodbye!" # Optional farewell message } ] ​ Function Actions Actions that run developer-defined functions at the appropriate time. For example, if used in post_actions , they’ll run after the bot has finished talking and after any previous post_actions have finished. ​ function action Calls the developer-defined function at the appropriate time. Copy Ask AI Copy Ask AI "post_actions" : [ { "type" : "function" , "handler" : bot_turn_ended # Required } ] ​ Custom Actions Fully developer-defined actions, providing flexibility at the expense of complexity. Here’s the complexity: because these actions aren’t queued in the Pipecat pipeline, they may execute seemingly early if used in post_actions ; they’ll run immediately after the LLM completion is triggered but won’t wait around for the bot to finish talking. Why would you want this behavior? You might be writing an action that: Itself just queues another Frame into the Pipecat pipeline (meaning there would no benefit to waiting around for sequencing purposes) Does work that can be done a bit sooner, like logging that the LLM was updated Custom actions are composed of at least: ​ type str required String identifier for the action ​ handler Callable required Async or sync function that handles the action Example: Copy Ask AI Copy Ask AI # Define custom action handler async def custom_notification ( action : dict , flow_manager : FlowManager): """Custom action handler.""" message = action.get( "message" , "" ) await notify_user(message) # Use in node configuration "pre_actions" : [ { "type" : "notify" , "handler" : send_notification, "message" : "Attention!" , } ] ​ Exceptions ​ FlowError exception Base exception for all flow-related errors. Copy Ask AI Copy Ask AI from pipecat_flows import FlowError try : await flow_manager.set_node_from_config(config) except FlowError as e: print ( f "Flow error: { e } " ) ​ FlowInitializationError exception Raised when flow initialization fails. Copy Ask AI Copy Ask AI from pipecat_flows import FlowInitializationError try : await flow_manager.initialize() except FlowInitializationError as e: print ( f "Initialization failed: { e } " ) ​ FlowTransitionError exception Raised when a state transition fails. Copy Ask AI Copy Ask AI from pipecat_flows import FlowTransitionError try : await flow_manager.set_node_from_config(node_config) except FlowTransitionError as e: print ( f "Transition failed: { e } " ) ​ InvalidFunctionError exception Raised when an invalid or unavailable function is specified. Copy Ask AI Copy Ask AI from pipecat_flows import InvalidFunctionError try : await flow_manager.set_node_from_config({ "functions" : [{ "type" : "function" , "function" : { "name" : "invalid_function" } }] }) except InvalidFunctionError as e: print ( f "Invalid function: { e } " ) RTVI Observer PipelineParams On this page Installation Core Types FlowArgs FlowResult FlowConfig NodeConfig Function Handler Types ContextStrategy ContextStrategyConfig FlowsFunctionSchema FlowManager Methods State Management Usage Examples Function Properties Handler Signatures Return Types Provider-Specific Formats Actions Pre-canned Actions Function Actions Custom Actions Exceptions Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/flows_pipecat-flows_d0032234.txt b/flows_pipecat-flows_d0032234.txt
new file mode 100644
index 0000000000000000000000000000000000000000..4e1371432a4b341b40019b511e9d945307a7d440
--- /dev/null
+++ b/flows_pipecat-flows_d0032234.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/frameworks/flows/pipecat-flows#methods
+Title: Pipecat Flows - Pipecat
+==================================================
+
+Pipecat Flows - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Frameworks Pipecat Flows Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline New to building conversational flows? Check out our Pipecat Flows guide first. ​ Installation Existing Pipecat installation Fresh Pipecat installation Copy Ask AI pip install pipecat-ai-flows ​ Core Types ​ FlowArgs ​ FlowArgs Dict[str, Any] Type alias for function handler arguments. ​ FlowResult ​ FlowResult TypedDict Base type for function handler results. Additional fields can be included as needed. Show Fields ​ status str Optional status field ​ error str Optional error message ​ FlowConfig ​ FlowConfig TypedDict Configuration for the entire conversation flow. Show Fields ​ initial_node str required Starting node identifier ​ nodes Dict[str, NodeConfig] required Map of node names to configurations ​ NodeConfig ​ NodeConfig TypedDict Configuration for a single node in the flow. Show Fields ​ name str The name of the node, used in debug logging in dynamic flows. If no name is specified, an automatically-generated UUID is used. Copy Ask AI # Example name "name" : "greeting" ​ role_messages List[dict] Defines the role or persona of the LLM. Required for the initial node and optional for subsequent nodes. Copy Ask AI # Example role messages "role_messages" : [ { "role" : "system" , "content" : "You are a helpful assistant..." } ], ​ task_messages List[dict] required Defines the task for a given node. Required for all nodes. Copy Ask AI # Example task messages "task_messages" : [ { "role" : "system" , # May be `user` depending on the LLM "content" : "Ask the user for their name..." } ], ​ context_strategy ContextStrategyConfig Strategy for managing context during transitions to this node. Copy Ask AI # Example context strategy configuration "context_strategy" : ContextStrategyConfig( strategy = ContextStrategy. RESET_WITH_SUMMARY , summary_prompt = "Summarize the key points discussed so far." ) ​ functions List[Union[dict, FlowsFunctionSchema]] required LLM function / tool call configurations, defined in one of the supported formats . Copy Ask AI # Using provider-specific dictionary format "functions" : [ { "type" : "function" , "function" : { "name" : "get_current_movies" , "handler" : get_movies, "description" : "Fetch movies currently playing" , "parameters" : { ... } }, } ] # Using FlowsFunctionSchema "functions" : [ FlowsFunctionSchema( name = "get_current_movies" , description = "Fetch movies currently playing" , properties = { ... }, required = [ ... ], handler = get_movies ) ] # Using direct functions (auto-configuration) "functions" : [get_movies] ​ pre_actions List[dict] Actions that execute before the LLM inference. For example, you can send a message to the TTS to speak a phrase (e.g. “Hold on a moment…”), which may be effective if an LLM function call takes time to execute. Copy Ask AI # Example pre_actions "pre_actions" : [ { "type" : "tts_say" , "text" : "Hold on a moment..." } ], ​ post_actions List[dict] Actions that execute after the LLM inference. For example, you can end the conversation. Copy Ask AI # Example post_actions "post_actions" : [ { "type" : "end_conversation" } ] ​ respond_immediately bool If set to False , the LLM will not respond immediately when the node is set, but will instead wait for the user to speak first before responding. Defaults to True . Copy Ask AI # Example usage "respond_immediately" : False ​ Function Handler Types ​ LegacyFunctionHandler Callable[[FlowArgs], Awaitable[FlowResult | ConsolidatedFunctionResult]] Legacy function handler that only receives arguments. Returns either: A FlowResult (⚠️ deprecated) A “consolidated” result tuple (result, next node) where: result is an optional FlowResult next node is an optional NodeConfig (for dynamic flows) or string (for static flows) ​ FlowFunctionHandler Callable[[FlowArgs, FlowManager], Awaitable[FlowResult | ConsolidatedFunctionResult]] Modern function handler that receives both arguments and FlowManager . Returns either: A FlowResult (⚠️ deprecated) A “consolidated” result tuple (result, next node) where: result is an optional FlowResult next node is an optional NodeConfig (for dynamic flows) or string (for static flows) ​ DirectFunction DirectFunction Function that is meant to be passed directly into a NodeConfig rather than into the handler field of a function configuration. It must be an async function with flow_manager: FlowManager as its first parameter. It must return a ConsolidatedFunctionResult , which is a tuple (result, next node) where: result is an optional FlowResult next node is an optional NodeConfig (for dynamic flows) or string (for static flows) ​ ContextStrategy ​ ContextStrategy Enum Strategy for managing conversation context during node transitions. Show Values ​ APPEND str Default strategy. Adds new messages to existing context. ​ RESET str Clears context and starts fresh with new messages. ​ RESET_WITH_SUMMARY str Resets context but includes an AI-generated summary. ​ ContextStrategyConfig ​ ContextStrategyConfig dataclass Configuration for context management strategy. Show Fields ​ strategy ContextStrategy required The strategy to use for context management ​ summary_prompt Optional[str] Required when using RESET_WITH_SUMMARY. Prompt text for generating the conversation summary. Copy Ask AI # Example usage config = ContextStrategyConfig( strategy = ContextStrategy. RESET_WITH_SUMMARY , summary_prompt = "Summarize the key points discussed so far." ) ​ FlowsFunctionSchema ​ FlowsFunctionSchema class A standardized schema for defining functions in Pipecat Flows with flow-specific properties. Show Constructor Parameters ​ name str required Name of the function ​ description str required Description of the function’s purpose ​ properties Dict[str, Any] required Dictionary defining properties types and descriptions ​ required List[str] required List of required parameter names ​ handler Optional[FunctionHandler] Function handler to process the function call ​ transition_to Optional[str] deprecated Target node to transition to after function execution Deprecated: instead of transition_to , use a “consolidated” handler that returns a tuple (result, next node). ​ transition_callback Optional[Callable] deprecated Callback function for dynamic transitions Deprecated: instead of transition_callback , use a “consolidated” handler that returns a tuple (result, next node). You cannot specify both transition_to and transition_callback in the same function schema. Example usage: Copy Ask AI from pipecat_flows import FlowsFunctionSchema # Define a function schema collect_name_schema = FlowsFunctionSchema( name = "collect_name" , description = "Record the user's name" , properties = { "name" : { "type" : "string" , "description" : "The user's name" } }, required = [ "name" ], handler = collect_name_handler ) # Use in node configuration node_config = { "name" : "greeting" , "task_messages" : [ { "role" : "system" , "content" : "Ask the user for their name." } ], "functions" : [collect_name_schema] } # Pass to flow manager await flow_manager.set_node_from_config(node_config) ​ FlowManager ​ FlowManager class Main class for managing conversation flows, supporting both static (configuration-driven) and dynamic (runtime-determined) flows. Show Constructor Parameters ​ task PipelineTask required Pipeline task for frame queueing ​ llm LLMService required LLM service instance (OpenAI, Anthropic, or Google). Must be initialized with the corresponding pipecat-ai provider dependency installed. ​ context_aggregator Any required Context aggregator used for pushing messages to the LLM service ​ tts Optional[Any] deprecated Optional TTS service for voice actions. Deprecated: No need to explicitly pass tts to FlowManager in order to use tts_say actions. ​ flow_config Optional[FlowConfig] Optional static flow configuration ​ context_strategy Optional[ContextStrategyConfig] Optional configuration for how context should be managed during transitions. Defaults to APPEND strategy if not specified. ​ Methods ​ initialize method Initialize the flow with starting messages. Show Parameters ​ initial_node NodeConfig The initial conversation node (needed for dynamic flows only). If not specified, you’ll need to call set_node_from_config() to kick off the conversation. Show Raises ​ FlowInitializationError If initialization fails ​ set_node method deprecated Set up a new conversation node programmatically (dynamic flows only). In dynamic flows, the application advances the conversation using set_node to set up each next node. In static flows, set_node is triggered under the hood when a node contains a transition_to field. Deprecated: use the following patterns instead of set_node : Prefer “consolidated” function handlers that return a tuple (result, next node), which implicitly sets up the next node Prefer passing your initial node to FlowManager.initialize() If you really need to set a node explicitly, use set_node_from_config() (note: its name will be read from its NodeConfig ) Show Parameters ​ node_id str required Identifier for the new node ​ node_config NodeConfig required Node configuration including messages, functions, and actions Show Raises ​ FlowError If node setup fails ​ set_node_from_config method Set up a new conversation node programmatically (dynamic flows only). Note that this method should only be used in rare circumstances. Most often, you should: Prefer “consolidated” function handlers that return a tuple (result, next node), which implicitly sets up the next node Prefer passing your initial node to FlowManager.initialize() Show Parameters ​ node_config NodeConfig required Node configuration including messages, functions, and actions Show Raises ​ FlowError If node setup fails ​ register_action method Register a handler for a custom action type. Show Parameters ​ action_type str required String identifier for the action ​ handler Callable required Async or sync function that handles the action ​ get_current_context method Get the current conversation context. Returns a list of messages in the current context, including system messages, user messages, and assistant responses. Show Returns ​ messages List[dict] List of messages in the current context Show Raises ​ FlowError If context aggregator is not available Example usage: Copy Ask AI # Access current conversation context context = flow_manager.get_current_context() # Use in handlers async def process_response ( args : FlowArgs) -> tuple[FlowResult, str ]: context = flow_manager.get_current_context() # Process conversation history return { "status" : "success" }, "next" ​ State Management The FlowManager provides a state dictionary for storing conversation data: Access state Access in transitions Copy Ask AI flow_manager.state: Dict[ str , Any] # Store data flow_manager.state[ "user_age" ] = 25 ​ Usage Examples Static Flow Dynamic Flow Copy Ask AI flow_config: FlowConfig = { "initial_node" : "greeting" , "nodes" : { "greeting" : { "role_messages" : [ { "role" : "system" , "content" : "You are a helpful assistant. Your responses will be converted to audio." } ], "task_messages" : [ { "role" : "system" , "content" : "Start by greeting the user and asking for their name." } ], "functions" : [{ "type" : "function" , "function" : { "name" : "collect_name" , "handler" : collect_name_handler, "description" : "Record user's name" , "parameters" : { ... } } }] } } } # Create and initialize the FlowManager flow_manager = FlowManager( task = task, llm = llm, context_aggregator = context_aggregator, flow_config = flow_config ) # Initialize the flow_manager to start the conversation await flow_manager.initialize() ​ Node Functions concept Functions that execute operations within a single conversational state, without switching nodes. Copy Ask AI Copy Ask AI from pipecat_flows import FlowArgs, FlowResult async def process_data ( args : FlowArgs) -> tuple[FlowResult, None ]: """Handle data processing within a node.""" data = args[ "data" ] result = await process(data) return { "status" : "success" , "processed_data" : result }, None # Function configuration { "type" : "function" , "function" : { "name" : "process_data" , "handler" : process_data, "description" : "Process user data" , "parameters" : { "type" : "object" , "properties" : { "data" : { "type" : "string" } } } } } ​ Edge Functions concept Functions that specify a transition between nodes (optionally processing data first). Copy Ask AI Copy Ask AI from pipecat_flows import FlowArgs, FlowResult async def next_step ( args : FlowArgs) -> tuple[ None , str ]: """Specify the next node to transition to.""" return None , "target_node" # Return NodeConfig instead of str for dynamic flows # Function configuration { "type" : "function" , "function" : { "name" : "next_step" , "handler" : next_step, "description" : "Transition to next node" , "parameters" : { "type" : "object" , "properties" : {}} } } ​ Function Properties ​ handler Optional[Callable] Async function that processes data within a node and/or specifies the next node ( more details here ). Can be specified as: Direct function reference Either a Callable function or a string with __function__: prefix (e.g., "__function__:process_data" ) to reference a function in the main script Direct Reference Function Token Copy Ask AI { "type" : "function" , "function" : { "name" : "process_data" , "handler" : process_data, # Callable function "parameters" : { ... } } } ​ transition_callback Optional[Callable] deprecated Handler for dynamic flow transitions. Deprecated: instead of transition_callback , use a “consolidated” handler that returns a tuple (result, next node). Must be an async function with one of these signatures: Copy Ask AI # New style (recommended) async def handle_transition ( args : Dict[ str , Any], result : FlowResult, flow_manager : FlowManager ) -> None : """Handle transition to next node.""" if result.available: # Type-safe access to result await flow_manager.set_node_from_config(create_confirmation_node()) else : await flow_manager.set_node_from_config( create_no_availability_node(result.alternative_times) ) # Legacy style (supported for backwards compatibility) async def handle_transition ( args : Dict[ str , Any], flow_manager : FlowManager ) -> None : """Handle transition to next node.""" await flow_manager.set_node_from_config(create_next_node()) The callback receives: args : Arguments from the function call result : Typed result from the function handler (new style only) flow_manager : Reference to the FlowManager instance Example usage: Copy Ask AI async def handle_availability_check ( args : Dict, result : TimeResult, # Typed result flow_manager : FlowManager ): """Handle availability check and transition based on result.""" if result.available: await flow_manager.set_node_from_config(create_confirmation_node()) else : await flow_manager.set_node_from_config( create_no_availability_node(result.alternative_times) ) # Use in function configuration { "type" : "function" , "function" : { "name" : "check_availability" , "handler" : check_availability, "parameters" : { ... }, "transition_callback" : handle_availability_check } } Note: A function cannot have both transition_to and transition_callback . ​ Handler Signatures Function handlers passed as a handler in a function configuration can be defined with three different signatures: Modern (Args + FlowManager) Legacy (Args Only) No Arguments Copy Ask AI async def handler_with_flow_manager ( args : FlowArgs, flow_manager : FlowManager) -> tuple[FlowResult, NodeConfig]: """Modern handler that receives both arguments and FlowManager access.""" # Access state previous_data = flow_manager.state.get( "stored_data" ) # Access pipeline resources await flow_manager.task.queue_frame(TTSSpeakFrame( "Processing your request..." )) # Store data in state for later flow_manager.state[ "new_data" ] = args[ "input" ] return { "status" : "success" , "result" : "Processed with flow access" }, create_next_node() The framework automatically detects which signature your handler is using and calls it appropriately. If you’re passing your function directly into your NodeConfig rather than as a handler in a function configuration, you’d use a somewhat different signature: Direct Copy Ask AI async def do_something ( flow_manager : FlowManager, foo : int , bar : str = "" ) -> tuple[FlowResult, NodeConfig]: """ Do something interesting. Args: foo (int): The foo to do something interesting with. bar (string): The bar to do something interesting with. Defaults to empty string. """ result = await fetch_data(foo, bar) next_node = create_end_node() return result, next_node ​ Return Types Success Response Error Response Copy Ask AI { "status" : "success" , "data" : "some data" # Optional additional data } ​ Provider-Specific Formats You don’t need to handle these format differences manually - use the standard format and the FlowManager will adapt it for your chosen provider. OpenAI Anthropic Google (Gemini) Copy Ask AI { "type" : "function" , "function" : { "name" : "function_name" , "handler" : handler, "description" : "Description" , "parameters" : { ... } } } ​ Actions pre_actions and post_actions are used to manage conversation flow. They are included in the NodeConfig and executed before and after the LLM completion, respectively. Three kinds of actions are available: Pre-canned actions: These actions perform common tasks and require little configuration. Function actions: These actions run developer-defined functions at the appropriate time. Custom actions: These are fully developer-defined actions, providing flexibility at the expense of complexity. ​ Pre-canned Actions Common actions shipped with Flows for managing conversation flow. To use them, just add them to your NodeConfig . ​ tts_say action Speaks text immediately using the TTS service. Copy Ask AI Copy Ask AI "pre_actions" : [ { "type" : "tts_say" , "text" : "Processing your request..." # Required } ] ​ end_conversation action Ends the conversation and closes the connection. Copy Ask AI Copy Ask AI "post_actions" : [ { "type" : "end_conversation" , "text" : "Goodbye!" # Optional farewell message } ] ​ Function Actions Actions that run developer-defined functions at the appropriate time. For example, if used in post_actions , they’ll run after the bot has finished talking and after any previous post_actions have finished. ​ function action Calls the developer-defined function at the appropriate time. Copy Ask AI Copy Ask AI "post_actions" : [ { "type" : "function" , "handler" : bot_turn_ended # Required } ] ​ Custom Actions Fully developer-defined actions, providing flexibility at the expense of complexity. Here’s the complexity: because these actions aren’t queued in the Pipecat pipeline, they may execute seemingly early if used in post_actions ; they’ll run immediately after the LLM completion is triggered but won’t wait around for the bot to finish talking. Why would you want this behavior? You might be writing an action that: Itself just queues another Frame into the Pipecat pipeline (meaning there would no benefit to waiting around for sequencing purposes) Does work that can be done a bit sooner, like logging that the LLM was updated Custom actions are composed of at least: ​ type str required String identifier for the action ​ handler Callable required Async or sync function that handles the action Example: Copy Ask AI Copy Ask AI # Define custom action handler async def custom_notification ( action : dict , flow_manager : FlowManager): """Custom action handler.""" message = action.get( "message" , "" ) await notify_user(message) # Use in node configuration "pre_actions" : [ { "type" : "notify" , "handler" : send_notification, "message" : "Attention!" , } ] ​ Exceptions ​ FlowError exception Base exception for all flow-related errors. Copy Ask AI Copy Ask AI from pipecat_flows import FlowError try : await flow_manager.set_node_from_config(config) except FlowError as e: print ( f "Flow error: { e } " ) ​ FlowInitializationError exception Raised when flow initialization fails. Copy Ask AI Copy Ask AI from pipecat_flows import FlowInitializationError try : await flow_manager.initialize() except FlowInitializationError as e: print ( f "Initialization failed: { e } " ) ​ FlowTransitionError exception Raised when a state transition fails. Copy Ask AI Copy Ask AI from pipecat_flows import FlowTransitionError try : await flow_manager.set_node_from_config(node_config) except FlowTransitionError as e: print ( f "Transition failed: { e } " ) ​ InvalidFunctionError exception Raised when an invalid or unavailable function is specified. Copy Ask AI Copy Ask AI from pipecat_flows import InvalidFunctionError try : await flow_manager.set_node_from_config({ "functions" : [{ "type" : "function" , "function" : { "name" : "invalid_function" } }] }) except InvalidFunctionError as e: print ( f "Invalid function: { e } " ) RTVI Observer PipelineParams On this page Installation Core Types FlowArgs FlowResult FlowConfig NodeConfig Function Handler Types ContextStrategy ContextStrategyConfig FlowsFunctionSchema FlowManager Methods State Management Usage Examples Function Properties Handler Signatures Return Types Provider-Specific Formats Actions Pre-canned Actions Function Actions Custom Actions Exceptions Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/flows_pipecat-flows_d84707ca.txt b/flows_pipecat-flows_d84707ca.txt
new file mode 100644
index 0000000000000000000000000000000000000000..193619134bfeb1899ada08a75185d41355ee4029
--- /dev/null
+++ b/flows_pipecat-flows_d84707ca.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/frameworks/flows/pipecat-flows#exceptions
+Title: Pipecat Flows - Pipecat
+==================================================
+
+Pipecat Flows - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Frameworks Pipecat Flows Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline New to building conversational flows? Check out our Pipecat Flows guide first. ​ Installation Existing Pipecat installation Fresh Pipecat installation Copy Ask AI pip install pipecat-ai-flows ​ Core Types ​ FlowArgs ​ FlowArgs Dict[str, Any] Type alias for function handler arguments. ​ FlowResult ​ FlowResult TypedDict Base type for function handler results. Additional fields can be included as needed. Show Fields ​ status str Optional status field ​ error str Optional error message ​ FlowConfig ​ FlowConfig TypedDict Configuration for the entire conversation flow. Show Fields ​ initial_node str required Starting node identifier ​ nodes Dict[str, NodeConfig] required Map of node names to configurations ​ NodeConfig ​ NodeConfig TypedDict Configuration for a single node in the flow. Show Fields ​ name str The name of the node, used in debug logging in dynamic flows. If no name is specified, an automatically-generated UUID is used. Copy Ask AI # Example name "name" : "greeting" ​ role_messages List[dict] Defines the role or persona of the LLM. Required for the initial node and optional for subsequent nodes. Copy Ask AI # Example role messages "role_messages" : [ { "role" : "system" , "content" : "You are a helpful assistant..." } ], ​ task_messages List[dict] required Defines the task for a given node. Required for all nodes. Copy Ask AI # Example task messages "task_messages" : [ { "role" : "system" , # May be `user` depending on the LLM "content" : "Ask the user for their name..." } ], ​ context_strategy ContextStrategyConfig Strategy for managing context during transitions to this node. Copy Ask AI # Example context strategy configuration "context_strategy" : ContextStrategyConfig( strategy = ContextStrategy. RESET_WITH_SUMMARY , summary_prompt = "Summarize the key points discussed so far." ) ​ functions List[Union[dict, FlowsFunctionSchema]] required LLM function / tool call configurations, defined in one of the supported formats . Copy Ask AI # Using provider-specific dictionary format "functions" : [ { "type" : "function" , "function" : { "name" : "get_current_movies" , "handler" : get_movies, "description" : "Fetch movies currently playing" , "parameters" : { ... } }, } ] # Using FlowsFunctionSchema "functions" : [ FlowsFunctionSchema( name = "get_current_movies" , description = "Fetch movies currently playing" , properties = { ... }, required = [ ... ], handler = get_movies ) ] # Using direct functions (auto-configuration) "functions" : [get_movies] ​ pre_actions List[dict] Actions that execute before the LLM inference. For example, you can send a message to the TTS to speak a phrase (e.g. “Hold on a moment…”), which may be effective if an LLM function call takes time to execute. Copy Ask AI # Example pre_actions "pre_actions" : [ { "type" : "tts_say" , "text" : "Hold on a moment..." } ], ​ post_actions List[dict] Actions that execute after the LLM inference. For example, you can end the conversation. Copy Ask AI # Example post_actions "post_actions" : [ { "type" : "end_conversation" } ] ​ respond_immediately bool If set to False , the LLM will not respond immediately when the node is set, but will instead wait for the user to speak first before responding. Defaults to True . Copy Ask AI # Example usage "respond_immediately" : False ​ Function Handler Types ​ LegacyFunctionHandler Callable[[FlowArgs], Awaitable[FlowResult | ConsolidatedFunctionResult]] Legacy function handler that only receives arguments. Returns either: A FlowResult (⚠️ deprecated) A “consolidated” result tuple (result, next node) where: result is an optional FlowResult next node is an optional NodeConfig (for dynamic flows) or string (for static flows) ​ FlowFunctionHandler Callable[[FlowArgs, FlowManager], Awaitable[FlowResult | ConsolidatedFunctionResult]] Modern function handler that receives both arguments and FlowManager . Returns either: A FlowResult (⚠️ deprecated) A “consolidated” result tuple (result, next node) where: result is an optional FlowResult next node is an optional NodeConfig (for dynamic flows) or string (for static flows) ​ DirectFunction DirectFunction Function that is meant to be passed directly into a NodeConfig rather than into the handler field of a function configuration. It must be an async function with flow_manager: FlowManager as its first parameter. It must return a ConsolidatedFunctionResult , which is a tuple (result, next node) where: result is an optional FlowResult next node is an optional NodeConfig (for dynamic flows) or string (for static flows) ​ ContextStrategy ​ ContextStrategy Enum Strategy for managing conversation context during node transitions. Show Values ​ APPEND str Default strategy. Adds new messages to existing context. ​ RESET str Clears context and starts fresh with new messages. ​ RESET_WITH_SUMMARY str Resets context but includes an AI-generated summary. ​ ContextStrategyConfig ​ ContextStrategyConfig dataclass Configuration for context management strategy. Show Fields ​ strategy ContextStrategy required The strategy to use for context management ​ summary_prompt Optional[str] Required when using RESET_WITH_SUMMARY. Prompt text for generating the conversation summary. Copy Ask AI # Example usage config = ContextStrategyConfig( strategy = ContextStrategy. RESET_WITH_SUMMARY , summary_prompt = "Summarize the key points discussed so far." ) ​ FlowsFunctionSchema ​ FlowsFunctionSchema class A standardized schema for defining functions in Pipecat Flows with flow-specific properties. Show Constructor Parameters ​ name str required Name of the function ​ description str required Description of the function’s purpose ​ properties Dict[str, Any] required Dictionary defining properties types and descriptions ​ required List[str] required List of required parameter names ​ handler Optional[FunctionHandler] Function handler to process the function call ​ transition_to Optional[str] deprecated Target node to transition to after function execution Deprecated: instead of transition_to , use a “consolidated” handler that returns a tuple (result, next node). ​ transition_callback Optional[Callable] deprecated Callback function for dynamic transitions Deprecated: instead of transition_callback , use a “consolidated” handler that returns a tuple (result, next node). You cannot specify both transition_to and transition_callback in the same function schema. Example usage: Copy Ask AI from pipecat_flows import FlowsFunctionSchema # Define a function schema collect_name_schema = FlowsFunctionSchema( name = "collect_name" , description = "Record the user's name" , properties = { "name" : { "type" : "string" , "description" : "The user's name" } }, required = [ "name" ], handler = collect_name_handler ) # Use in node configuration node_config = { "name" : "greeting" , "task_messages" : [ { "role" : "system" , "content" : "Ask the user for their name." } ], "functions" : [collect_name_schema] } # Pass to flow manager await flow_manager.set_node_from_config(node_config) ​ FlowManager ​ FlowManager class Main class for managing conversation flows, supporting both static (configuration-driven) and dynamic (runtime-determined) flows. Show Constructor Parameters ​ task PipelineTask required Pipeline task for frame queueing ​ llm LLMService required LLM service instance (OpenAI, Anthropic, or Google). Must be initialized with the corresponding pipecat-ai provider dependency installed. ​ context_aggregator Any required Context aggregator used for pushing messages to the LLM service ​ tts Optional[Any] deprecated Optional TTS service for voice actions. Deprecated: No need to explicitly pass tts to FlowManager in order to use tts_say actions. ​ flow_config Optional[FlowConfig] Optional static flow configuration ​ context_strategy Optional[ContextStrategyConfig] Optional configuration for how context should be managed during transitions. Defaults to APPEND strategy if not specified. ​ Methods ​ initialize method Initialize the flow with starting messages. Show Parameters ​ initial_node NodeConfig The initial conversation node (needed for dynamic flows only). If not specified, you’ll need to call set_node_from_config() to kick off the conversation. Show Raises ​ FlowInitializationError If initialization fails ​ set_node method deprecated Set up a new conversation node programmatically (dynamic flows only). In dynamic flows, the application advances the conversation using set_node to set up each next node. In static flows, set_node is triggered under the hood when a node contains a transition_to field. Deprecated: use the following patterns instead of set_node : Prefer “consolidated” function handlers that return a tuple (result, next node), which implicitly sets up the next node Prefer passing your initial node to FlowManager.initialize() If you really need to set a node explicitly, use set_node_from_config() (note: its name will be read from its NodeConfig ) Show Parameters ​ node_id str required Identifier for the new node ​ node_config NodeConfig required Node configuration including messages, functions, and actions Show Raises ​ FlowError If node setup fails ​ set_node_from_config method Set up a new conversation node programmatically (dynamic flows only). Note that this method should only be used in rare circumstances. Most often, you should: Prefer “consolidated” function handlers that return a tuple (result, next node), which implicitly sets up the next node Prefer passing your initial node to FlowManager.initialize() Show Parameters ​ node_config NodeConfig required Node configuration including messages, functions, and actions Show Raises ​ FlowError If node setup fails ​ register_action method Register a handler for a custom action type. Show Parameters ​ action_type str required String identifier for the action ​ handler Callable required Async or sync function that handles the action ​ get_current_context method Get the current conversation context. Returns a list of messages in the current context, including system messages, user messages, and assistant responses. Show Returns ​ messages List[dict] List of messages in the current context Show Raises ​ FlowError If context aggregator is not available Example usage: Copy Ask AI # Access current conversation context context = flow_manager.get_current_context() # Use in handlers async def process_response ( args : FlowArgs) -> tuple[FlowResult, str ]: context = flow_manager.get_current_context() # Process conversation history return { "status" : "success" }, "next" ​ State Management The FlowManager provides a state dictionary for storing conversation data: Access state Access in transitions Copy Ask AI flow_manager.state: Dict[ str , Any] # Store data flow_manager.state[ "user_age" ] = 25 ​ Usage Examples Static Flow Dynamic Flow Copy Ask AI flow_config: FlowConfig = { "initial_node" : "greeting" , "nodes" : { "greeting" : { "role_messages" : [ { "role" : "system" , "content" : "You are a helpful assistant. Your responses will be converted to audio." } ], "task_messages" : [ { "role" : "system" , "content" : "Start by greeting the user and asking for their name." } ], "functions" : [{ "type" : "function" , "function" : { "name" : "collect_name" , "handler" : collect_name_handler, "description" : "Record user's name" , "parameters" : { ... } } }] } } } # Create and initialize the FlowManager flow_manager = FlowManager( task = task, llm = llm, context_aggregator = context_aggregator, flow_config = flow_config ) # Initialize the flow_manager to start the conversation await flow_manager.initialize() ​ Node Functions concept Functions that execute operations within a single conversational state, without switching nodes. Copy Ask AI Copy Ask AI from pipecat_flows import FlowArgs, FlowResult async def process_data ( args : FlowArgs) -> tuple[FlowResult, None ]: """Handle data processing within a node.""" data = args[ "data" ] result = await process(data) return { "status" : "success" , "processed_data" : result }, None # Function configuration { "type" : "function" , "function" : { "name" : "process_data" , "handler" : process_data, "description" : "Process user data" , "parameters" : { "type" : "object" , "properties" : { "data" : { "type" : "string" } } } } } ​ Edge Functions concept Functions that specify a transition between nodes (optionally processing data first). Copy Ask AI Copy Ask AI from pipecat_flows import FlowArgs, FlowResult async def next_step ( args : FlowArgs) -> tuple[ None , str ]: """Specify the next node to transition to.""" return None , "target_node" # Return NodeConfig instead of str for dynamic flows # Function configuration { "type" : "function" , "function" : { "name" : "next_step" , "handler" : next_step, "description" : "Transition to next node" , "parameters" : { "type" : "object" , "properties" : {}} } } ​ Function Properties ​ handler Optional[Callable] Async function that processes data within a node and/or specifies the next node ( more details here ). Can be specified as: Direct function reference Either a Callable function or a string with __function__: prefix (e.g., "__function__:process_data" ) to reference a function in the main script Direct Reference Function Token Copy Ask AI { "type" : "function" , "function" : { "name" : "process_data" , "handler" : process_data, # Callable function "parameters" : { ... } } } ​ transition_callback Optional[Callable] deprecated Handler for dynamic flow transitions. Deprecated: instead of transition_callback , use a “consolidated” handler that returns a tuple (result, next node). Must be an async function with one of these signatures: Copy Ask AI # New style (recommended) async def handle_transition ( args : Dict[ str , Any], result : FlowResult, flow_manager : FlowManager ) -> None : """Handle transition to next node.""" if result.available: # Type-safe access to result await flow_manager.set_node_from_config(create_confirmation_node()) else : await flow_manager.set_node_from_config( create_no_availability_node(result.alternative_times) ) # Legacy style (supported for backwards compatibility) async def handle_transition ( args : Dict[ str , Any], flow_manager : FlowManager ) -> None : """Handle transition to next node.""" await flow_manager.set_node_from_config(create_next_node()) The callback receives: args : Arguments from the function call result : Typed result from the function handler (new style only) flow_manager : Reference to the FlowManager instance Example usage: Copy Ask AI async def handle_availability_check ( args : Dict, result : TimeResult, # Typed result flow_manager : FlowManager ): """Handle availability check and transition based on result.""" if result.available: await flow_manager.set_node_from_config(create_confirmation_node()) else : await flow_manager.set_node_from_config( create_no_availability_node(result.alternative_times) ) # Use in function configuration { "type" : "function" , "function" : { "name" : "check_availability" , "handler" : check_availability, "parameters" : { ... }, "transition_callback" : handle_availability_check } } Note: A function cannot have both transition_to and transition_callback . ​ Handler Signatures Function handlers passed as a handler in a function configuration can be defined with three different signatures: Modern (Args + FlowManager) Legacy (Args Only) No Arguments Copy Ask AI async def handler_with_flow_manager ( args : FlowArgs, flow_manager : FlowManager) -> tuple[FlowResult, NodeConfig]: """Modern handler that receives both arguments and FlowManager access.""" # Access state previous_data = flow_manager.state.get( "stored_data" ) # Access pipeline resources await flow_manager.task.queue_frame(TTSSpeakFrame( "Processing your request..." )) # Store data in state for later flow_manager.state[ "new_data" ] = args[ "input" ] return { "status" : "success" , "result" : "Processed with flow access" }, create_next_node() The framework automatically detects which signature your handler is using and calls it appropriately. If you’re passing your function directly into your NodeConfig rather than as a handler in a function configuration, you’d use a somewhat different signature: Direct Copy Ask AI async def do_something ( flow_manager : FlowManager, foo : int , bar : str = "" ) -> tuple[FlowResult, NodeConfig]: """ Do something interesting. Args: foo (int): The foo to do something interesting with. bar (string): The bar to do something interesting with. Defaults to empty string. """ result = await fetch_data(foo, bar) next_node = create_end_node() return result, next_node ​ Return Types Success Response Error Response Copy Ask AI { "status" : "success" , "data" : "some data" # Optional additional data } ​ Provider-Specific Formats You don’t need to handle these format differences manually - use the standard format and the FlowManager will adapt it for your chosen provider. OpenAI Anthropic Google (Gemini) Copy Ask AI { "type" : "function" , "function" : { "name" : "function_name" , "handler" : handler, "description" : "Description" , "parameters" : { ... } } } ​ Actions pre_actions and post_actions are used to manage conversation flow. They are included in the NodeConfig and executed before and after the LLM completion, respectively. Three kinds of actions are available: Pre-canned actions: These actions perform common tasks and require little configuration. Function actions: These actions run developer-defined functions at the appropriate time. Custom actions: These are fully developer-defined actions, providing flexibility at the expense of complexity. ​ Pre-canned Actions Common actions shipped with Flows for managing conversation flow. To use them, just add them to your NodeConfig . ​ tts_say action Speaks text immediately using the TTS service. Copy Ask AI Copy Ask AI "pre_actions" : [ { "type" : "tts_say" , "text" : "Processing your request..." # Required } ] ​ end_conversation action Ends the conversation and closes the connection. Copy Ask AI Copy Ask AI "post_actions" : [ { "type" : "end_conversation" , "text" : "Goodbye!" # Optional farewell message } ] ​ Function Actions Actions that run developer-defined functions at the appropriate time. For example, if used in post_actions , they’ll run after the bot has finished talking and after any previous post_actions have finished. ​ function action Calls the developer-defined function at the appropriate time. Copy Ask AI Copy Ask AI "post_actions" : [ { "type" : "function" , "handler" : bot_turn_ended # Required } ] ​ Custom Actions Fully developer-defined actions, providing flexibility at the expense of complexity. Here’s the complexity: because these actions aren’t queued in the Pipecat pipeline, they may execute seemingly early if used in post_actions ; they’ll run immediately after the LLM completion is triggered but won’t wait around for the bot to finish talking. Why would you want this behavior? You might be writing an action that: Itself just queues another Frame into the Pipecat pipeline (meaning there would no benefit to waiting around for sequencing purposes) Does work that can be done a bit sooner, like logging that the LLM was updated Custom actions are composed of at least: ​ type str required String identifier for the action ​ handler Callable required Async or sync function that handles the action Example: Copy Ask AI Copy Ask AI # Define custom action handler async def custom_notification ( action : dict , flow_manager : FlowManager): """Custom action handler.""" message = action.get( "message" , "" ) await notify_user(message) # Use in node configuration "pre_actions" : [ { "type" : "notify" , "handler" : send_notification, "message" : "Attention!" , } ] ​ Exceptions ​ FlowError exception Base exception for all flow-related errors. Copy Ask AI Copy Ask AI from pipecat_flows import FlowError try : await flow_manager.set_node_from_config(config) except FlowError as e: print ( f "Flow error: { e } " ) ​ FlowInitializationError exception Raised when flow initialization fails. Copy Ask AI Copy Ask AI from pipecat_flows import FlowInitializationError try : await flow_manager.initialize() except FlowInitializationError as e: print ( f "Initialization failed: { e } " ) ​ FlowTransitionError exception Raised when a state transition fails. Copy Ask AI Copy Ask AI from pipecat_flows import FlowTransitionError try : await flow_manager.set_node_from_config(node_config) except FlowTransitionError as e: print ( f "Transition failed: { e } " ) ​ InvalidFunctionError exception Raised when an invalid or unavailable function is specified. Copy Ask AI Copy Ask AI from pipecat_flows import InvalidFunctionError try : await flow_manager.set_node_from_config({ "functions" : [{ "type" : "function" , "function" : { "name" : "invalid_function" } }] }) except InvalidFunctionError as e: print ( f "Invalid function: { e } " ) RTVI Observer PipelineParams On this page Installation Core Types FlowArgs FlowResult FlowConfig NodeConfig Function Handler Types ContextStrategy ContextStrategyConfig FlowsFunctionSchema FlowManager Methods State Management Usage Examples Function Properties Handler Signatures Return Types Provider-Specific Formats Actions Pre-canned Actions Function Actions Custom Actions Exceptions Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/flows_pipecat-flows_d96b49fb.txt b/flows_pipecat-flows_d96b49fb.txt
new file mode 100644
index 0000000000000000000000000000000000000000..fda4bcc99b228a87c15a3a930d510182f53934c2
--- /dev/null
+++ b/flows_pipecat-flows_d96b49fb.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/frameworks/flows/pipecat-flows#function-handler-types
+Title: Pipecat Flows - Pipecat
+==================================================
+
+Pipecat Flows - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Frameworks Pipecat Flows Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline New to building conversational flows? Check out our Pipecat Flows guide first. ​ Installation Existing Pipecat installation Fresh Pipecat installation Copy Ask AI pip install pipecat-ai-flows ​ Core Types ​ FlowArgs ​ FlowArgs Dict[str, Any] Type alias for function handler arguments. ​ FlowResult ​ FlowResult TypedDict Base type for function handler results. Additional fields can be included as needed. Show Fields ​ status str Optional status field ​ error str Optional error message ​ FlowConfig ​ FlowConfig TypedDict Configuration for the entire conversation flow. Show Fields ​ initial_node str required Starting node identifier ​ nodes Dict[str, NodeConfig] required Map of node names to configurations ​ NodeConfig ​ NodeConfig TypedDict Configuration for a single node in the flow. Show Fields ​ name str The name of the node, used in debug logging in dynamic flows. If no name is specified, an automatically-generated UUID is used. Copy Ask AI # Example name "name" : "greeting" ​ role_messages List[dict] Defines the role or persona of the LLM. Required for the initial node and optional for subsequent nodes. Copy Ask AI # Example role messages "role_messages" : [ { "role" : "system" , "content" : "You are a helpful assistant..." } ], ​ task_messages List[dict] required Defines the task for a given node. Required for all nodes. Copy Ask AI # Example task messages "task_messages" : [ { "role" : "system" , # May be `user` depending on the LLM "content" : "Ask the user for their name..." } ], ​ context_strategy ContextStrategyConfig Strategy for managing context during transitions to this node. Copy Ask AI # Example context strategy configuration "context_strategy" : ContextStrategyConfig( strategy = ContextStrategy. RESET_WITH_SUMMARY , summary_prompt = "Summarize the key points discussed so far." ) ​ functions List[Union[dict, FlowsFunctionSchema]] required LLM function / tool call configurations, defined in one of the supported formats . Copy Ask AI # Using provider-specific dictionary format "functions" : [ { "type" : "function" , "function" : { "name" : "get_current_movies" , "handler" : get_movies, "description" : "Fetch movies currently playing" , "parameters" : { ... } }, } ] # Using FlowsFunctionSchema "functions" : [ FlowsFunctionSchema( name = "get_current_movies" , description = "Fetch movies currently playing" , properties = { ... }, required = [ ... ], handler = get_movies ) ] # Using direct functions (auto-configuration) "functions" : [get_movies] ​ pre_actions List[dict] Actions that execute before the LLM inference. For example, you can send a message to the TTS to speak a phrase (e.g. “Hold on a moment…”), which may be effective if an LLM function call takes time to execute. Copy Ask AI # Example pre_actions "pre_actions" : [ { "type" : "tts_say" , "text" : "Hold on a moment..." } ], ​ post_actions List[dict] Actions that execute after the LLM inference. For example, you can end the conversation. Copy Ask AI # Example post_actions "post_actions" : [ { "type" : "end_conversation" } ] ​ respond_immediately bool If set to False , the LLM will not respond immediately when the node is set, but will instead wait for the user to speak first before responding. Defaults to True . Copy Ask AI # Example usage "respond_immediately" : False ​ Function Handler Types ​ LegacyFunctionHandler Callable[[FlowArgs], Awaitable[FlowResult | ConsolidatedFunctionResult]] Legacy function handler that only receives arguments. Returns either: A FlowResult (⚠️ deprecated) A “consolidated” result tuple (result, next node) where: result is an optional FlowResult next node is an optional NodeConfig (for dynamic flows) or string (for static flows) ​ FlowFunctionHandler Callable[[FlowArgs, FlowManager], Awaitable[FlowResult | ConsolidatedFunctionResult]] Modern function handler that receives both arguments and FlowManager . Returns either: A FlowResult (⚠️ deprecated) A “consolidated” result tuple (result, next node) where: result is an optional FlowResult next node is an optional NodeConfig (for dynamic flows) or string (for static flows) ​ DirectFunction DirectFunction Function that is meant to be passed directly into a NodeConfig rather than into the handler field of a function configuration. It must be an async function with flow_manager: FlowManager as its first parameter. It must return a ConsolidatedFunctionResult , which is a tuple (result, next node) where: result is an optional FlowResult next node is an optional NodeConfig (for dynamic flows) or string (for static flows) ​ ContextStrategy ​ ContextStrategy Enum Strategy for managing conversation context during node transitions. Show Values ​ APPEND str Default strategy. Adds new messages to existing context. ​ RESET str Clears context and starts fresh with new messages. ​ RESET_WITH_SUMMARY str Resets context but includes an AI-generated summary. ​ ContextStrategyConfig ​ ContextStrategyConfig dataclass Configuration for context management strategy. Show Fields ​ strategy ContextStrategy required The strategy to use for context management ​ summary_prompt Optional[str] Required when using RESET_WITH_SUMMARY. Prompt text for generating the conversation summary. Copy Ask AI # Example usage config = ContextStrategyConfig( strategy = ContextStrategy. RESET_WITH_SUMMARY , summary_prompt = "Summarize the key points discussed so far." ) ​ FlowsFunctionSchema ​ FlowsFunctionSchema class A standardized schema for defining functions in Pipecat Flows with flow-specific properties. Show Constructor Parameters ​ name str required Name of the function ​ description str required Description of the function’s purpose ​ properties Dict[str, Any] required Dictionary defining properties types and descriptions ​ required List[str] required List of required parameter names ​ handler Optional[FunctionHandler] Function handler to process the function call ​ transition_to Optional[str] deprecated Target node to transition to after function execution Deprecated: instead of transition_to , use a “consolidated” handler that returns a tuple (result, next node). ​ transition_callback Optional[Callable] deprecated Callback function for dynamic transitions Deprecated: instead of transition_callback , use a “consolidated” handler that returns a tuple (result, next node). You cannot specify both transition_to and transition_callback in the same function schema. Example usage: Copy Ask AI from pipecat_flows import FlowsFunctionSchema # Define a function schema collect_name_schema = FlowsFunctionSchema( name = "collect_name" , description = "Record the user's name" , properties = { "name" : { "type" : "string" , "description" : "The user's name" } }, required = [ "name" ], handler = collect_name_handler ) # Use in node configuration node_config = { "name" : "greeting" , "task_messages" : [ { "role" : "system" , "content" : "Ask the user for their name." } ], "functions" : [collect_name_schema] } # Pass to flow manager await flow_manager.set_node_from_config(node_config) ​ FlowManager ​ FlowManager class Main class for managing conversation flows, supporting both static (configuration-driven) and dynamic (runtime-determined) flows. Show Constructor Parameters ​ task PipelineTask required Pipeline task for frame queueing ​ llm LLMService required LLM service instance (OpenAI, Anthropic, or Google). Must be initialized with the corresponding pipecat-ai provider dependency installed. ​ context_aggregator Any required Context aggregator used for pushing messages to the LLM service ​ tts Optional[Any] deprecated Optional TTS service for voice actions. Deprecated: No need to explicitly pass tts to FlowManager in order to use tts_say actions. ​ flow_config Optional[FlowConfig] Optional static flow configuration ​ context_strategy Optional[ContextStrategyConfig] Optional configuration for how context should be managed during transitions. Defaults to APPEND strategy if not specified. ​ Methods ​ initialize method Initialize the flow with starting messages. Show Parameters ​ initial_node NodeConfig The initial conversation node (needed for dynamic flows only). If not specified, you’ll need to call set_node_from_config() to kick off the conversation. Show Raises ​ FlowInitializationError If initialization fails ​ set_node method deprecated Set up a new conversation node programmatically (dynamic flows only). In dynamic flows, the application advances the conversation using set_node to set up each next node. In static flows, set_node is triggered under the hood when a node contains a transition_to field. Deprecated: use the following patterns instead of set_node : Prefer “consolidated” function handlers that return a tuple (result, next node), which implicitly sets up the next node Prefer passing your initial node to FlowManager.initialize() If you really need to set a node explicitly, use set_node_from_config() (note: its name will be read from its NodeConfig ) Show Parameters ​ node_id str required Identifier for the new node ​ node_config NodeConfig required Node configuration including messages, functions, and actions Show Raises ​ FlowError If node setup fails ​ set_node_from_config method Set up a new conversation node programmatically (dynamic flows only). Note that this method should only be used in rare circumstances. Most often, you should: Prefer “consolidated” function handlers that return a tuple (result, next node), which implicitly sets up the next node Prefer passing your initial node to FlowManager.initialize() Show Parameters ​ node_config NodeConfig required Node configuration including messages, functions, and actions Show Raises ​ FlowError If node setup fails ​ register_action method Register a handler for a custom action type. Show Parameters ​ action_type str required String identifier for the action ​ handler Callable required Async or sync function that handles the action ​ get_current_context method Get the current conversation context. Returns a list of messages in the current context, including system messages, user messages, and assistant responses. Show Returns ​ messages List[dict] List of messages in the current context Show Raises ​ FlowError If context aggregator is not available Example usage: Copy Ask AI # Access current conversation context context = flow_manager.get_current_context() # Use in handlers async def process_response ( args : FlowArgs) -> tuple[FlowResult, str ]: context = flow_manager.get_current_context() # Process conversation history return { "status" : "success" }, "next" ​ State Management The FlowManager provides a state dictionary for storing conversation data: Access state Access in transitions Copy Ask AI flow_manager.state: Dict[ str , Any] # Store data flow_manager.state[ "user_age" ] = 25 ​ Usage Examples Static Flow Dynamic Flow Copy Ask AI flow_config: FlowConfig = { "initial_node" : "greeting" , "nodes" : { "greeting" : { "role_messages" : [ { "role" : "system" , "content" : "You are a helpful assistant. Your responses will be converted to audio." } ], "task_messages" : [ { "role" : "system" , "content" : "Start by greeting the user and asking for their name." } ], "functions" : [{ "type" : "function" , "function" : { "name" : "collect_name" , "handler" : collect_name_handler, "description" : "Record user's name" , "parameters" : { ... } } }] } } } # Create and initialize the FlowManager flow_manager = FlowManager( task = task, llm = llm, context_aggregator = context_aggregator, flow_config = flow_config ) # Initialize the flow_manager to start the conversation await flow_manager.initialize() ​ Node Functions concept Functions that execute operations within a single conversational state, without switching nodes. Copy Ask AI Copy Ask AI from pipecat_flows import FlowArgs, FlowResult async def process_data ( args : FlowArgs) -> tuple[FlowResult, None ]: """Handle data processing within a node.""" data = args[ "data" ] result = await process(data) return { "status" : "success" , "processed_data" : result }, None # Function configuration { "type" : "function" , "function" : { "name" : "process_data" , "handler" : process_data, "description" : "Process user data" , "parameters" : { "type" : "object" , "properties" : { "data" : { "type" : "string" } } } } } ​ Edge Functions concept Functions that specify a transition between nodes (optionally processing data first). Copy Ask AI Copy Ask AI from pipecat_flows import FlowArgs, FlowResult async def next_step ( args : FlowArgs) -> tuple[ None , str ]: """Specify the next node to transition to.""" return None , "target_node" # Return NodeConfig instead of str for dynamic flows # Function configuration { "type" : "function" , "function" : { "name" : "next_step" , "handler" : next_step, "description" : "Transition to next node" , "parameters" : { "type" : "object" , "properties" : {}} } } ​ Function Properties ​ handler Optional[Callable] Async function that processes data within a node and/or specifies the next node ( more details here ). Can be specified as: Direct function reference Either a Callable function or a string with __function__: prefix (e.g., "__function__:process_data" ) to reference a function in the main script Direct Reference Function Token Copy Ask AI { "type" : "function" , "function" : { "name" : "process_data" , "handler" : process_data, # Callable function "parameters" : { ... } } } ​ transition_callback Optional[Callable] deprecated Handler for dynamic flow transitions. Deprecated: instead of transition_callback , use a “consolidated” handler that returns a tuple (result, next node). Must be an async function with one of these signatures: Copy Ask AI # New style (recommended) async def handle_transition ( args : Dict[ str , Any], result : FlowResult, flow_manager : FlowManager ) -> None : """Handle transition to next node.""" if result.available: # Type-safe access to result await flow_manager.set_node_from_config(create_confirmation_node()) else : await flow_manager.set_node_from_config( create_no_availability_node(result.alternative_times) ) # Legacy style (supported for backwards compatibility) async def handle_transition ( args : Dict[ str , Any], flow_manager : FlowManager ) -> None : """Handle transition to next node.""" await flow_manager.set_node_from_config(create_next_node()) The callback receives: args : Arguments from the function call result : Typed result from the function handler (new style only) flow_manager : Reference to the FlowManager instance Example usage: Copy Ask AI async def handle_availability_check ( args : Dict, result : TimeResult, # Typed result flow_manager : FlowManager ): """Handle availability check and transition based on result.""" if result.available: await flow_manager.set_node_from_config(create_confirmation_node()) else : await flow_manager.set_node_from_config( create_no_availability_node(result.alternative_times) ) # Use in function configuration { "type" : "function" , "function" : { "name" : "check_availability" , "handler" : check_availability, "parameters" : { ... }, "transition_callback" : handle_availability_check } } Note: A function cannot have both transition_to and transition_callback . ​ Handler Signatures Function handlers passed as a handler in a function configuration can be defined with three different signatures: Modern (Args + FlowManager) Legacy (Args Only) No Arguments Copy Ask AI async def handler_with_flow_manager ( args : FlowArgs, flow_manager : FlowManager) -> tuple[FlowResult, NodeConfig]: """Modern handler that receives both arguments and FlowManager access.""" # Access state previous_data = flow_manager.state.get( "stored_data" ) # Access pipeline resources await flow_manager.task.queue_frame(TTSSpeakFrame( "Processing your request..." )) # Store data in state for later flow_manager.state[ "new_data" ] = args[ "input" ] return { "status" : "success" , "result" : "Processed with flow access" }, create_next_node() The framework automatically detects which signature your handler is using and calls it appropriately. If you’re passing your function directly into your NodeConfig rather than as a handler in a function configuration, you’d use a somewhat different signature: Direct Copy Ask AI async def do_something ( flow_manager : FlowManager, foo : int , bar : str = "" ) -> tuple[FlowResult, NodeConfig]: """ Do something interesting. Args: foo (int): The foo to do something interesting with. bar (string): The bar to do something interesting with. Defaults to empty string. """ result = await fetch_data(foo, bar) next_node = create_end_node() return result, next_node ​ Return Types Success Response Error Response Copy Ask AI { "status" : "success" , "data" : "some data" # Optional additional data } ​ Provider-Specific Formats You don’t need to handle these format differences manually - use the standard format and the FlowManager will adapt it for your chosen provider. OpenAI Anthropic Google (Gemini) Copy Ask AI { "type" : "function" , "function" : { "name" : "function_name" , "handler" : handler, "description" : "Description" , "parameters" : { ... } } } ​ Actions pre_actions and post_actions are used to manage conversation flow. They are included in the NodeConfig and executed before and after the LLM completion, respectively. Three kinds of actions are available: Pre-canned actions: These actions perform common tasks and require little configuration. Function actions: These actions run developer-defined functions at the appropriate time. Custom actions: These are fully developer-defined actions, providing flexibility at the expense of complexity. ​ Pre-canned Actions Common actions shipped with Flows for managing conversation flow. To use them, just add them to your NodeConfig . ​ tts_say action Speaks text immediately using the TTS service. Copy Ask AI Copy Ask AI "pre_actions" : [ { "type" : "tts_say" , "text" : "Processing your request..." # Required } ] ​ end_conversation action Ends the conversation and closes the connection. Copy Ask AI Copy Ask AI "post_actions" : [ { "type" : "end_conversation" , "text" : "Goodbye!" # Optional farewell message } ] ​ Function Actions Actions that run developer-defined functions at the appropriate time. For example, if used in post_actions , they’ll run after the bot has finished talking and after any previous post_actions have finished. ​ function action Calls the developer-defined function at the appropriate time. Copy Ask AI Copy Ask AI "post_actions" : [ { "type" : "function" , "handler" : bot_turn_ended # Required } ] ​ Custom Actions Fully developer-defined actions, providing flexibility at the expense of complexity. Here’s the complexity: because these actions aren’t queued in the Pipecat pipeline, they may execute seemingly early if used in post_actions ; they’ll run immediately after the LLM completion is triggered but won’t wait around for the bot to finish talking. Why would you want this behavior? You might be writing an action that: Itself just queues another Frame into the Pipecat pipeline (meaning there would no benefit to waiting around for sequencing purposes) Does work that can be done a bit sooner, like logging that the LLM was updated Custom actions are composed of at least: ​ type str required String identifier for the action ​ handler Callable required Async or sync function that handles the action Example: Copy Ask AI Copy Ask AI # Define custom action handler async def custom_notification ( action : dict , flow_manager : FlowManager): """Custom action handler.""" message = action.get( "message" , "" ) await notify_user(message) # Use in node configuration "pre_actions" : [ { "type" : "notify" , "handler" : send_notification, "message" : "Attention!" , } ] ​ Exceptions ​ FlowError exception Base exception for all flow-related errors. Copy Ask AI Copy Ask AI from pipecat_flows import FlowError try : await flow_manager.set_node_from_config(config) except FlowError as e: print ( f "Flow error: { e } " ) ​ FlowInitializationError exception Raised when flow initialization fails. Copy Ask AI Copy Ask AI from pipecat_flows import FlowInitializationError try : await flow_manager.initialize() except FlowInitializationError as e: print ( f "Initialization failed: { e } " ) ​ FlowTransitionError exception Raised when a state transition fails. Copy Ask AI Copy Ask AI from pipecat_flows import FlowTransitionError try : await flow_manager.set_node_from_config(node_config) except FlowTransitionError as e: print ( f "Transition failed: { e } " ) ​ InvalidFunctionError exception Raised when an invalid or unavailable function is specified. Copy Ask AI Copy Ask AI from pipecat_flows import InvalidFunctionError try : await flow_manager.set_node_from_config({ "functions" : [{ "type" : "function" , "function" : { "name" : "invalid_function" } }] }) except InvalidFunctionError as e: print ( f "Invalid function: { e } " ) RTVI Observer PipelineParams On this page Installation Core Types FlowArgs FlowResult FlowConfig NodeConfig Function Handler Types ContextStrategy ContextStrategyConfig FlowsFunctionSchema FlowManager Methods State Management Usage Examples Function Properties Handler Signatures Return Types Provider-Specific Formats Actions Pre-canned Actions Function Actions Custom Actions Exceptions Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/flows_pipecat-flows_d9cf493f.txt b/flows_pipecat-flows_d9cf493f.txt
new file mode 100644
index 0000000000000000000000000000000000000000..e1207cdef71714bafba8c9bdb15771e5db5cb2a9
--- /dev/null
+++ b/flows_pipecat-flows_d9cf493f.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/frameworks/flows/pipecat-flows#flowconfig
+Title: Pipecat Flows - Pipecat
+==================================================
+
+Pipecat Flows - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Frameworks Pipecat Flows Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline New to building conversational flows? Check out our Pipecat Flows guide first. ​ Installation Existing Pipecat installation Fresh Pipecat installation Copy Ask AI pip install pipecat-ai-flows ​ Core Types ​ FlowArgs ​ FlowArgs Dict[str, Any] Type alias for function handler arguments. ​ FlowResult ​ FlowResult TypedDict Base type for function handler results. Additional fields can be included as needed. Show Fields ​ status str Optional status field ​ error str Optional error message ​ FlowConfig ​ FlowConfig TypedDict Configuration for the entire conversation flow. Show Fields ​ initial_node str required Starting node identifier ​ nodes Dict[str, NodeConfig] required Map of node names to configurations ​ NodeConfig ​ NodeConfig TypedDict Configuration for a single node in the flow. Show Fields ​ name str The name of the node, used in debug logging in dynamic flows. If no name is specified, an automatically-generated UUID is used. Copy Ask AI # Example name "name" : "greeting" ​ role_messages List[dict] Defines the role or persona of the LLM. Required for the initial node and optional for subsequent nodes. Copy Ask AI # Example role messages "role_messages" : [ { "role" : "system" , "content" : "You are a helpful assistant..." } ], ​ task_messages List[dict] required Defines the task for a given node. Required for all nodes. Copy Ask AI # Example task messages "task_messages" : [ { "role" : "system" , # May be `user` depending on the LLM "content" : "Ask the user for their name..." } ], ​ context_strategy ContextStrategyConfig Strategy for managing context during transitions to this node. Copy Ask AI # Example context strategy configuration "context_strategy" : ContextStrategyConfig( strategy = ContextStrategy. RESET_WITH_SUMMARY , summary_prompt = "Summarize the key points discussed so far." ) ​ functions List[Union[dict, FlowsFunctionSchema]] required LLM function / tool call configurations, defined in one of the supported formats . Copy Ask AI # Using provider-specific dictionary format "functions" : [ { "type" : "function" , "function" : { "name" : "get_current_movies" , "handler" : get_movies, "description" : "Fetch movies currently playing" , "parameters" : { ... } }, } ] # Using FlowsFunctionSchema "functions" : [ FlowsFunctionSchema( name = "get_current_movies" , description = "Fetch movies currently playing" , properties = { ... }, required = [ ... ], handler = get_movies ) ] # Using direct functions (auto-configuration) "functions" : [get_movies] ​ pre_actions List[dict] Actions that execute before the LLM inference. For example, you can send a message to the TTS to speak a phrase (e.g. “Hold on a moment…”), which may be effective if an LLM function call takes time to execute. Copy Ask AI # Example pre_actions "pre_actions" : [ { "type" : "tts_say" , "text" : "Hold on a moment..." } ], ​ post_actions List[dict] Actions that execute after the LLM inference. For example, you can end the conversation. Copy Ask AI # Example post_actions "post_actions" : [ { "type" : "end_conversation" } ] ​ respond_immediately bool If set to False , the LLM will not respond immediately when the node is set, but will instead wait for the user to speak first before responding. Defaults to True . Copy Ask AI # Example usage "respond_immediately" : False ​ Function Handler Types ​ LegacyFunctionHandler Callable[[FlowArgs], Awaitable[FlowResult | ConsolidatedFunctionResult]] Legacy function handler that only receives arguments. Returns either: A FlowResult (⚠️ deprecated) A “consolidated” result tuple (result, next node) where: result is an optional FlowResult next node is an optional NodeConfig (for dynamic flows) or string (for static flows) ​ FlowFunctionHandler Callable[[FlowArgs, FlowManager], Awaitable[FlowResult | ConsolidatedFunctionResult]] Modern function handler that receives both arguments and FlowManager . Returns either: A FlowResult (⚠️ deprecated) A “consolidated” result tuple (result, next node) where: result is an optional FlowResult next node is an optional NodeConfig (for dynamic flows) or string (for static flows) ​ DirectFunction DirectFunction Function that is meant to be passed directly into a NodeConfig rather than into the handler field of a function configuration. It must be an async function with flow_manager: FlowManager as its first parameter. It must return a ConsolidatedFunctionResult , which is a tuple (result, next node) where: result is an optional FlowResult next node is an optional NodeConfig (for dynamic flows) or string (for static flows) ​ ContextStrategy ​ ContextStrategy Enum Strategy for managing conversation context during node transitions. Show Values ​ APPEND str Default strategy. Adds new messages to existing context. ​ RESET str Clears context and starts fresh with new messages. ​ RESET_WITH_SUMMARY str Resets context but includes an AI-generated summary. ​ ContextStrategyConfig ​ ContextStrategyConfig dataclass Configuration for context management strategy. Show Fields ​ strategy ContextStrategy required The strategy to use for context management ​ summary_prompt Optional[str] Required when using RESET_WITH_SUMMARY. Prompt text for generating the conversation summary. Copy Ask AI # Example usage config = ContextStrategyConfig( strategy = ContextStrategy. RESET_WITH_SUMMARY , summary_prompt = "Summarize the key points discussed so far." ) ​ FlowsFunctionSchema ​ FlowsFunctionSchema class A standardized schema for defining functions in Pipecat Flows with flow-specific properties. Show Constructor Parameters ​ name str required Name of the function ​ description str required Description of the function’s purpose ​ properties Dict[str, Any] required Dictionary defining properties types and descriptions ​ required List[str] required List of required parameter names ​ handler Optional[FunctionHandler] Function handler to process the function call ​ transition_to Optional[str] deprecated Target node to transition to after function execution Deprecated: instead of transition_to , use a “consolidated” handler that returns a tuple (result, next node). ​ transition_callback Optional[Callable] deprecated Callback function for dynamic transitions Deprecated: instead of transition_callback , use a “consolidated” handler that returns a tuple (result, next node). You cannot specify both transition_to and transition_callback in the same function schema. Example usage: Copy Ask AI from pipecat_flows import FlowsFunctionSchema # Define a function schema collect_name_schema = FlowsFunctionSchema( name = "collect_name" , description = "Record the user's name" , properties = { "name" : { "type" : "string" , "description" : "The user's name" } }, required = [ "name" ], handler = collect_name_handler ) # Use in node configuration node_config = { "name" : "greeting" , "task_messages" : [ { "role" : "system" , "content" : "Ask the user for their name." } ], "functions" : [collect_name_schema] } # Pass to flow manager await flow_manager.set_node_from_config(node_config) ​ FlowManager ​ FlowManager class Main class for managing conversation flows, supporting both static (configuration-driven) and dynamic (runtime-determined) flows. Show Constructor Parameters ​ task PipelineTask required Pipeline task for frame queueing ​ llm LLMService required LLM service instance (OpenAI, Anthropic, or Google). Must be initialized with the corresponding pipecat-ai provider dependency installed. ​ context_aggregator Any required Context aggregator used for pushing messages to the LLM service ​ tts Optional[Any] deprecated Optional TTS service for voice actions. Deprecated: No need to explicitly pass tts to FlowManager in order to use tts_say actions. ​ flow_config Optional[FlowConfig] Optional static flow configuration ​ context_strategy Optional[ContextStrategyConfig] Optional configuration for how context should be managed during transitions. Defaults to APPEND strategy if not specified. ​ Methods ​ initialize method Initialize the flow with starting messages. Show Parameters ​ initial_node NodeConfig The initial conversation node (needed for dynamic flows only). If not specified, you’ll need to call set_node_from_config() to kick off the conversation. Show Raises ​ FlowInitializationError If initialization fails ​ set_node method deprecated Set up a new conversation node programmatically (dynamic flows only). In dynamic flows, the application advances the conversation using set_node to set up each next node. In static flows, set_node is triggered under the hood when a node contains a transition_to field. Deprecated: use the following patterns instead of set_node : Prefer “consolidated” function handlers that return a tuple (result, next node), which implicitly sets up the next node Prefer passing your initial node to FlowManager.initialize() If you really need to set a node explicitly, use set_node_from_config() (note: its name will be read from its NodeConfig ) Show Parameters ​ node_id str required Identifier for the new node ​ node_config NodeConfig required Node configuration including messages, functions, and actions Show Raises ​ FlowError If node setup fails ​ set_node_from_config method Set up a new conversation node programmatically (dynamic flows only). Note that this method should only be used in rare circumstances. Most often, you should: Prefer “consolidated” function handlers that return a tuple (result, next node), which implicitly sets up the next node Prefer passing your initial node to FlowManager.initialize() Show Parameters ​ node_config NodeConfig required Node configuration including messages, functions, and actions Show Raises ​ FlowError If node setup fails ​ register_action method Register a handler for a custom action type. Show Parameters ​ action_type str required String identifier for the action ​ handler Callable required Async or sync function that handles the action ​ get_current_context method Get the current conversation context. Returns a list of messages in the current context, including system messages, user messages, and assistant responses. Show Returns ​ messages List[dict] List of messages in the current context Show Raises ​ FlowError If context aggregator is not available Example usage: Copy Ask AI # Access current conversation context context = flow_manager.get_current_context() # Use in handlers async def process_response ( args : FlowArgs) -> tuple[FlowResult, str ]: context = flow_manager.get_current_context() # Process conversation history return { "status" : "success" }, "next" ​ State Management The FlowManager provides a state dictionary for storing conversation data: Access state Access in transitions Copy Ask AI flow_manager.state: Dict[ str , Any] # Store data flow_manager.state[ "user_age" ] = 25 ​ Usage Examples Static Flow Dynamic Flow Copy Ask AI flow_config: FlowConfig = { "initial_node" : "greeting" , "nodes" : { "greeting" : { "role_messages" : [ { "role" : "system" , "content" : "You are a helpful assistant. Your responses will be converted to audio." } ], "task_messages" : [ { "role" : "system" , "content" : "Start by greeting the user and asking for their name." } ], "functions" : [{ "type" : "function" , "function" : { "name" : "collect_name" , "handler" : collect_name_handler, "description" : "Record user's name" , "parameters" : { ... } } }] } } } # Create and initialize the FlowManager flow_manager = FlowManager( task = task, llm = llm, context_aggregator = context_aggregator, flow_config = flow_config ) # Initialize the flow_manager to start the conversation await flow_manager.initialize() ​ Node Functions concept Functions that execute operations within a single conversational state, without switching nodes. Copy Ask AI Copy Ask AI from pipecat_flows import FlowArgs, FlowResult async def process_data ( args : FlowArgs) -> tuple[FlowResult, None ]: """Handle data processing within a node.""" data = args[ "data" ] result = await process(data) return { "status" : "success" , "processed_data" : result }, None # Function configuration { "type" : "function" , "function" : { "name" : "process_data" , "handler" : process_data, "description" : "Process user data" , "parameters" : { "type" : "object" , "properties" : { "data" : { "type" : "string" } } } } } ​ Edge Functions concept Functions that specify a transition between nodes (optionally processing data first). Copy Ask AI Copy Ask AI from pipecat_flows import FlowArgs, FlowResult async def next_step ( args : FlowArgs) -> tuple[ None , str ]: """Specify the next node to transition to.""" return None , "target_node" # Return NodeConfig instead of str for dynamic flows # Function configuration { "type" : "function" , "function" : { "name" : "next_step" , "handler" : next_step, "description" : "Transition to next node" , "parameters" : { "type" : "object" , "properties" : {}} } } ​ Function Properties ​ handler Optional[Callable] Async function that processes data within a node and/or specifies the next node ( more details here ). Can be specified as: Direct function reference Either a Callable function or a string with __function__: prefix (e.g., "__function__:process_data" ) to reference a function in the main script Direct Reference Function Token Copy Ask AI { "type" : "function" , "function" : { "name" : "process_data" , "handler" : process_data, # Callable function "parameters" : { ... } } } ​ transition_callback Optional[Callable] deprecated Handler for dynamic flow transitions. Deprecated: instead of transition_callback , use a “consolidated” handler that returns a tuple (result, next node). Must be an async function with one of these signatures: Copy Ask AI # New style (recommended) async def handle_transition ( args : Dict[ str , Any], result : FlowResult, flow_manager : FlowManager ) -> None : """Handle transition to next node.""" if result.available: # Type-safe access to result await flow_manager.set_node_from_config(create_confirmation_node()) else : await flow_manager.set_node_from_config( create_no_availability_node(result.alternative_times) ) # Legacy style (supported for backwards compatibility) async def handle_transition ( args : Dict[ str , Any], flow_manager : FlowManager ) -> None : """Handle transition to next node.""" await flow_manager.set_node_from_config(create_next_node()) The callback receives: args : Arguments from the function call result : Typed result from the function handler (new style only) flow_manager : Reference to the FlowManager instance Example usage: Copy Ask AI async def handle_availability_check ( args : Dict, result : TimeResult, # Typed result flow_manager : FlowManager ): """Handle availability check and transition based on result.""" if result.available: await flow_manager.set_node_from_config(create_confirmation_node()) else : await flow_manager.set_node_from_config( create_no_availability_node(result.alternative_times) ) # Use in function configuration { "type" : "function" , "function" : { "name" : "check_availability" , "handler" : check_availability, "parameters" : { ... }, "transition_callback" : handle_availability_check } } Note: A function cannot have both transition_to and transition_callback . ​ Handler Signatures Function handlers passed as a handler in a function configuration can be defined with three different signatures: Modern (Args + FlowManager) Legacy (Args Only) No Arguments Copy Ask AI async def handler_with_flow_manager ( args : FlowArgs, flow_manager : FlowManager) -> tuple[FlowResult, NodeConfig]: """Modern handler that receives both arguments and FlowManager access.""" # Access state previous_data = flow_manager.state.get( "stored_data" ) # Access pipeline resources await flow_manager.task.queue_frame(TTSSpeakFrame( "Processing your request..." )) # Store data in state for later flow_manager.state[ "new_data" ] = args[ "input" ] return { "status" : "success" , "result" : "Processed with flow access" }, create_next_node() The framework automatically detects which signature your handler is using and calls it appropriately. If you’re passing your function directly into your NodeConfig rather than as a handler in a function configuration, you’d use a somewhat different signature: Direct Copy Ask AI async def do_something ( flow_manager : FlowManager, foo : int , bar : str = "" ) -> tuple[FlowResult, NodeConfig]: """ Do something interesting. Args: foo (int): The foo to do something interesting with. bar (string): The bar to do something interesting with. Defaults to empty string. """ result = await fetch_data(foo, bar) next_node = create_end_node() return result, next_node ​ Return Types Success Response Error Response Copy Ask AI { "status" : "success" , "data" : "some data" # Optional additional data } ​ Provider-Specific Formats You don’t need to handle these format differences manually - use the standard format and the FlowManager will adapt it for your chosen provider. OpenAI Anthropic Google (Gemini) Copy Ask AI { "type" : "function" , "function" : { "name" : "function_name" , "handler" : handler, "description" : "Description" , "parameters" : { ... } } } ​ Actions pre_actions and post_actions are used to manage conversation flow. They are included in the NodeConfig and executed before and after the LLM completion, respectively. Three kinds of actions are available: Pre-canned actions: These actions perform common tasks and require little configuration. Function actions: These actions run developer-defined functions at the appropriate time. Custom actions: These are fully developer-defined actions, providing flexibility at the expense of complexity. ​ Pre-canned Actions Common actions shipped with Flows for managing conversation flow. To use them, just add them to your NodeConfig . ​ tts_say action Speaks text immediately using the TTS service. Copy Ask AI Copy Ask AI "pre_actions" : [ { "type" : "tts_say" , "text" : "Processing your request..." # Required } ] ​ end_conversation action Ends the conversation and closes the connection. Copy Ask AI Copy Ask AI "post_actions" : [ { "type" : "end_conversation" , "text" : "Goodbye!" # Optional farewell message } ] ​ Function Actions Actions that run developer-defined functions at the appropriate time. For example, if used in post_actions , they’ll run after the bot has finished talking and after any previous post_actions have finished. ​ function action Calls the developer-defined function at the appropriate time. Copy Ask AI Copy Ask AI "post_actions" : [ { "type" : "function" , "handler" : bot_turn_ended # Required } ] ​ Custom Actions Fully developer-defined actions, providing flexibility at the expense of complexity. Here’s the complexity: because these actions aren’t queued in the Pipecat pipeline, they may execute seemingly early if used in post_actions ; they’ll run immediately after the LLM completion is triggered but won’t wait around for the bot to finish talking. Why would you want this behavior? You might be writing an action that: Itself just queues another Frame into the Pipecat pipeline (meaning there would no benefit to waiting around for sequencing purposes) Does work that can be done a bit sooner, like logging that the LLM was updated Custom actions are composed of at least: ​ type str required String identifier for the action ​ handler Callable required Async or sync function that handles the action Example: Copy Ask AI Copy Ask AI # Define custom action handler async def custom_notification ( action : dict , flow_manager : FlowManager): """Custom action handler.""" message = action.get( "message" , "" ) await notify_user(message) # Use in node configuration "pre_actions" : [ { "type" : "notify" , "handler" : send_notification, "message" : "Attention!" , } ] ​ Exceptions ​ FlowError exception Base exception for all flow-related errors. Copy Ask AI Copy Ask AI from pipecat_flows import FlowError try : await flow_manager.set_node_from_config(config) except FlowError as e: print ( f "Flow error: { e } " ) ​ FlowInitializationError exception Raised when flow initialization fails. Copy Ask AI Copy Ask AI from pipecat_flows import FlowInitializationError try : await flow_manager.initialize() except FlowInitializationError as e: print ( f "Initialization failed: { e } " ) ​ FlowTransitionError exception Raised when a state transition fails. Copy Ask AI Copy Ask AI from pipecat_flows import FlowTransitionError try : await flow_manager.set_node_from_config(node_config) except FlowTransitionError as e: print ( f "Transition failed: { e } " ) ​ InvalidFunctionError exception Raised when an invalid or unavailable function is specified. Copy Ask AI Copy Ask AI from pipecat_flows import InvalidFunctionError try : await flow_manager.set_node_from_config({ "functions" : [{ "type" : "function" , "function" : { "name" : "invalid_function" } }] }) except InvalidFunctionError as e: print ( f "Invalid function: { e } " ) RTVI Observer PipelineParams On this page Installation Core Types FlowArgs FlowResult FlowConfig NodeConfig Function Handler Types ContextStrategy ContextStrategyConfig FlowsFunctionSchema FlowManager Methods State Management Usage Examples Function Properties Handler Signatures Return Types Provider-Specific Formats Actions Pre-canned Actions Function Actions Custom Actions Exceptions Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/flows_pipecat-flows_dc6c577f.txt b/flows_pipecat-flows_dc6c577f.txt
new file mode 100644
index 0000000000000000000000000000000000000000..88da2b2633e0dc3777ef22acbc3315a6d4f426cf
--- /dev/null
+++ b/flows_pipecat-flows_dc6c577f.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/frameworks/flows/pipecat-flows#param-flow-config
+Title: Pipecat Flows - Pipecat
+==================================================
+
+Pipecat Flows - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Frameworks Pipecat Flows Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline New to building conversational flows? Check out our Pipecat Flows guide first. ​ Installation Existing Pipecat installation Fresh Pipecat installation Copy Ask AI pip install pipecat-ai-flows ​ Core Types ​ FlowArgs ​ FlowArgs Dict[str, Any] Type alias for function handler arguments. ​ FlowResult ​ FlowResult TypedDict Base type for function handler results. Additional fields can be included as needed. Show Fields ​ status str Optional status field ​ error str Optional error message ​ FlowConfig ​ FlowConfig TypedDict Configuration for the entire conversation flow. Show Fields ​ initial_node str required Starting node identifier ​ nodes Dict[str, NodeConfig] required Map of node names to configurations ​ NodeConfig ​ NodeConfig TypedDict Configuration for a single node in the flow. Show Fields ​ name str The name of the node, used in debug logging in dynamic flows. If no name is specified, an automatically-generated UUID is used. Copy Ask AI # Example name "name" : "greeting" ​ role_messages List[dict] Defines the role or persona of the LLM. Required for the initial node and optional for subsequent nodes. Copy Ask AI # Example role messages "role_messages" : [ { "role" : "system" , "content" : "You are a helpful assistant..." } ], ​ task_messages List[dict] required Defines the task for a given node. Required for all nodes. Copy Ask AI # Example task messages "task_messages" : [ { "role" : "system" , # May be `user` depending on the LLM "content" : "Ask the user for their name..." } ], ​ context_strategy ContextStrategyConfig Strategy for managing context during transitions to this node. Copy Ask AI # Example context strategy configuration "context_strategy" : ContextStrategyConfig( strategy = ContextStrategy. RESET_WITH_SUMMARY , summary_prompt = "Summarize the key points discussed so far." ) ​ functions List[Union[dict, FlowsFunctionSchema]] required LLM function / tool call configurations, defined in one of the supported formats . Copy Ask AI # Using provider-specific dictionary format "functions" : [ { "type" : "function" , "function" : { "name" : "get_current_movies" , "handler" : get_movies, "description" : "Fetch movies currently playing" , "parameters" : { ... } }, } ] # Using FlowsFunctionSchema "functions" : [ FlowsFunctionSchema( name = "get_current_movies" , description = "Fetch movies currently playing" , properties = { ... }, required = [ ... ], handler = get_movies ) ] # Using direct functions (auto-configuration) "functions" : [get_movies] ​ pre_actions List[dict] Actions that execute before the LLM inference. For example, you can send a message to the TTS to speak a phrase (e.g. “Hold on a moment…”), which may be effective if an LLM function call takes time to execute. Copy Ask AI # Example pre_actions "pre_actions" : [ { "type" : "tts_say" , "text" : "Hold on a moment..." } ], ​ post_actions List[dict] Actions that execute after the LLM inference. For example, you can end the conversation. Copy Ask AI # Example post_actions "post_actions" : [ { "type" : "end_conversation" } ] ​ respond_immediately bool If set to False , the LLM will not respond immediately when the node is set, but will instead wait for the user to speak first before responding. Defaults to True . Copy Ask AI # Example usage "respond_immediately" : False ​ Function Handler Types ​ LegacyFunctionHandler Callable[[FlowArgs], Awaitable[FlowResult | ConsolidatedFunctionResult]] Legacy function handler that only receives arguments. Returns either: A FlowResult (⚠️ deprecated) A “consolidated” result tuple (result, next node) where: result is an optional FlowResult next node is an optional NodeConfig (for dynamic flows) or string (for static flows) ​ FlowFunctionHandler Callable[[FlowArgs, FlowManager], Awaitable[FlowResult | ConsolidatedFunctionResult]] Modern function handler that receives both arguments and FlowManager . Returns either: A FlowResult (⚠️ deprecated) A “consolidated” result tuple (result, next node) where: result is an optional FlowResult next node is an optional NodeConfig (for dynamic flows) or string (for static flows) ​ DirectFunction DirectFunction Function that is meant to be passed directly into a NodeConfig rather than into the handler field of a function configuration. It must be an async function with flow_manager: FlowManager as its first parameter. It must return a ConsolidatedFunctionResult , which is a tuple (result, next node) where: result is an optional FlowResult next node is an optional NodeConfig (for dynamic flows) or string (for static flows) ​ ContextStrategy ​ ContextStrategy Enum Strategy for managing conversation context during node transitions. Show Values ​ APPEND str Default strategy. Adds new messages to existing context. ​ RESET str Clears context and starts fresh with new messages. ​ RESET_WITH_SUMMARY str Resets context but includes an AI-generated summary. ​ ContextStrategyConfig ​ ContextStrategyConfig dataclass Configuration for context management strategy. Show Fields ​ strategy ContextStrategy required The strategy to use for context management ​ summary_prompt Optional[str] Required when using RESET_WITH_SUMMARY. Prompt text for generating the conversation summary. Copy Ask AI # Example usage config = ContextStrategyConfig( strategy = ContextStrategy. RESET_WITH_SUMMARY , summary_prompt = "Summarize the key points discussed so far." ) ​ FlowsFunctionSchema ​ FlowsFunctionSchema class A standardized schema for defining functions in Pipecat Flows with flow-specific properties. Show Constructor Parameters ​ name str required Name of the function ​ description str required Description of the function’s purpose ​ properties Dict[str, Any] required Dictionary defining properties types and descriptions ​ required List[str] required List of required parameter names ​ handler Optional[FunctionHandler] Function handler to process the function call ​ transition_to Optional[str] deprecated Target node to transition to after function execution Deprecated: instead of transition_to , use a “consolidated” handler that returns a tuple (result, next node). ​ transition_callback Optional[Callable] deprecated Callback function for dynamic transitions Deprecated: instead of transition_callback , use a “consolidated” handler that returns a tuple (result, next node). You cannot specify both transition_to and transition_callback in the same function schema. Example usage: Copy Ask AI from pipecat_flows import FlowsFunctionSchema # Define a function schema collect_name_schema = FlowsFunctionSchema( name = "collect_name" , description = "Record the user's name" , properties = { "name" : { "type" : "string" , "description" : "The user's name" } }, required = [ "name" ], handler = collect_name_handler ) # Use in node configuration node_config = { "name" : "greeting" , "task_messages" : [ { "role" : "system" , "content" : "Ask the user for their name." } ], "functions" : [collect_name_schema] } # Pass to flow manager await flow_manager.set_node_from_config(node_config) ​ FlowManager ​ FlowManager class Main class for managing conversation flows, supporting both static (configuration-driven) and dynamic (runtime-determined) flows. Show Constructor Parameters ​ task PipelineTask required Pipeline task for frame queueing ​ llm LLMService required LLM service instance (OpenAI, Anthropic, or Google). Must be initialized with the corresponding pipecat-ai provider dependency installed. ​ context_aggregator Any required Context aggregator used for pushing messages to the LLM service ​ tts Optional[Any] deprecated Optional TTS service for voice actions. Deprecated: No need to explicitly pass tts to FlowManager in order to use tts_say actions. ​ flow_config Optional[FlowConfig] Optional static flow configuration ​ context_strategy Optional[ContextStrategyConfig] Optional configuration for how context should be managed during transitions. Defaults to APPEND strategy if not specified. ​ Methods ​ initialize method Initialize the flow with starting messages. Show Parameters ​ initial_node NodeConfig The initial conversation node (needed for dynamic flows only). If not specified, you’ll need to call set_node_from_config() to kick off the conversation. Show Raises ​ FlowInitializationError If initialization fails ​ set_node method deprecated Set up a new conversation node programmatically (dynamic flows only). In dynamic flows, the application advances the conversation using set_node to set up each next node. In static flows, set_node is triggered under the hood when a node contains a transition_to field. Deprecated: use the following patterns instead of set_node : Prefer “consolidated” function handlers that return a tuple (result, next node), which implicitly sets up the next node Prefer passing your initial node to FlowManager.initialize() If you really need to set a node explicitly, use set_node_from_config() (note: its name will be read from its NodeConfig ) Show Parameters ​ node_id str required Identifier for the new node ​ node_config NodeConfig required Node configuration including messages, functions, and actions Show Raises ​ FlowError If node setup fails ​ set_node_from_config method Set up a new conversation node programmatically (dynamic flows only). Note that this method should only be used in rare circumstances. Most often, you should: Prefer “consolidated” function handlers that return a tuple (result, next node), which implicitly sets up the next node Prefer passing your initial node to FlowManager.initialize() Show Parameters ​ node_config NodeConfig required Node configuration including messages, functions, and actions Show Raises ​ FlowError If node setup fails ​ register_action method Register a handler for a custom action type. Show Parameters ​ action_type str required String identifier for the action ​ handler Callable required Async or sync function that handles the action ​ get_current_context method Get the current conversation context. Returns a list of messages in the current context, including system messages, user messages, and assistant responses. Show Returns ​ messages List[dict] List of messages in the current context Show Raises ​ FlowError If context aggregator is not available Example usage: Copy Ask AI # Access current conversation context context = flow_manager.get_current_context() # Use in handlers async def process_response ( args : FlowArgs) -> tuple[FlowResult, str ]: context = flow_manager.get_current_context() # Process conversation history return { "status" : "success" }, "next" ​ State Management The FlowManager provides a state dictionary for storing conversation data: Access state Access in transitions Copy Ask AI flow_manager.state: Dict[ str , Any] # Store data flow_manager.state[ "user_age" ] = 25 ​ Usage Examples Static Flow Dynamic Flow Copy Ask AI flow_config: FlowConfig = { "initial_node" : "greeting" , "nodes" : { "greeting" : { "role_messages" : [ { "role" : "system" , "content" : "You are a helpful assistant. Your responses will be converted to audio." } ], "task_messages" : [ { "role" : "system" , "content" : "Start by greeting the user and asking for their name." } ], "functions" : [{ "type" : "function" , "function" : { "name" : "collect_name" , "handler" : collect_name_handler, "description" : "Record user's name" , "parameters" : { ... } } }] } } } # Create and initialize the FlowManager flow_manager = FlowManager( task = task, llm = llm, context_aggregator = context_aggregator, flow_config = flow_config ) # Initialize the flow_manager to start the conversation await flow_manager.initialize() ​ Node Functions concept Functions that execute operations within a single conversational state, without switching nodes. Copy Ask AI Copy Ask AI from pipecat_flows import FlowArgs, FlowResult async def process_data ( args : FlowArgs) -> tuple[FlowResult, None ]: """Handle data processing within a node.""" data = args[ "data" ] result = await process(data) return { "status" : "success" , "processed_data" : result }, None # Function configuration { "type" : "function" , "function" : { "name" : "process_data" , "handler" : process_data, "description" : "Process user data" , "parameters" : { "type" : "object" , "properties" : { "data" : { "type" : "string" } } } } } ​ Edge Functions concept Functions that specify a transition between nodes (optionally processing data first). Copy Ask AI Copy Ask AI from pipecat_flows import FlowArgs, FlowResult async def next_step ( args : FlowArgs) -> tuple[ None , str ]: """Specify the next node to transition to.""" return None , "target_node" # Return NodeConfig instead of str for dynamic flows # Function configuration { "type" : "function" , "function" : { "name" : "next_step" , "handler" : next_step, "description" : "Transition to next node" , "parameters" : { "type" : "object" , "properties" : {}} } } ​ Function Properties ​ handler Optional[Callable] Async function that processes data within a node and/or specifies the next node ( more details here ). Can be specified as: Direct function reference Either a Callable function or a string with __function__: prefix (e.g., "__function__:process_data" ) to reference a function in the main script Direct Reference Function Token Copy Ask AI { "type" : "function" , "function" : { "name" : "process_data" , "handler" : process_data, # Callable function "parameters" : { ... } } } ​ transition_callback Optional[Callable] deprecated Handler for dynamic flow transitions. Deprecated: instead of transition_callback , use a “consolidated” handler that returns a tuple (result, next node). Must be an async function with one of these signatures: Copy Ask AI # New style (recommended) async def handle_transition ( args : Dict[ str , Any], result : FlowResult, flow_manager : FlowManager ) -> None : """Handle transition to next node.""" if result.available: # Type-safe access to result await flow_manager.set_node_from_config(create_confirmation_node()) else : await flow_manager.set_node_from_config( create_no_availability_node(result.alternative_times) ) # Legacy style (supported for backwards compatibility) async def handle_transition ( args : Dict[ str , Any], flow_manager : FlowManager ) -> None : """Handle transition to next node.""" await flow_manager.set_node_from_config(create_next_node()) The callback receives: args : Arguments from the function call result : Typed result from the function handler (new style only) flow_manager : Reference to the FlowManager instance Example usage: Copy Ask AI async def handle_availability_check ( args : Dict, result : TimeResult, # Typed result flow_manager : FlowManager ): """Handle availability check and transition based on result.""" if result.available: await flow_manager.set_node_from_config(create_confirmation_node()) else : await flow_manager.set_node_from_config( create_no_availability_node(result.alternative_times) ) # Use in function configuration { "type" : "function" , "function" : { "name" : "check_availability" , "handler" : check_availability, "parameters" : { ... }, "transition_callback" : handle_availability_check } } Note: A function cannot have both transition_to and transition_callback . ​ Handler Signatures Function handlers passed as a handler in a function configuration can be defined with three different signatures: Modern (Args + FlowManager) Legacy (Args Only) No Arguments Copy Ask AI async def handler_with_flow_manager ( args : FlowArgs, flow_manager : FlowManager) -> tuple[FlowResult, NodeConfig]: """Modern handler that receives both arguments and FlowManager access.""" # Access state previous_data = flow_manager.state.get( "stored_data" ) # Access pipeline resources await flow_manager.task.queue_frame(TTSSpeakFrame( "Processing your request..." )) # Store data in state for later flow_manager.state[ "new_data" ] = args[ "input" ] return { "status" : "success" , "result" : "Processed with flow access" }, create_next_node() The framework automatically detects which signature your handler is using and calls it appropriately. If you’re passing your function directly into your NodeConfig rather than as a handler in a function configuration, you’d use a somewhat different signature: Direct Copy Ask AI async def do_something ( flow_manager : FlowManager, foo : int , bar : str = "" ) -> tuple[FlowResult, NodeConfig]: """ Do something interesting. Args: foo (int): The foo to do something interesting with. bar (string): The bar to do something interesting with. Defaults to empty string. """ result = await fetch_data(foo, bar) next_node = create_end_node() return result, next_node ​ Return Types Success Response Error Response Copy Ask AI { "status" : "success" , "data" : "some data" # Optional additional data } ​ Provider-Specific Formats You don’t need to handle these format differences manually - use the standard format and the FlowManager will adapt it for your chosen provider. OpenAI Anthropic Google (Gemini) Copy Ask AI { "type" : "function" , "function" : { "name" : "function_name" , "handler" : handler, "description" : "Description" , "parameters" : { ... } } } ​ Actions pre_actions and post_actions are used to manage conversation flow. They are included in the NodeConfig and executed before and after the LLM completion, respectively. Three kinds of actions are available: Pre-canned actions: These actions perform common tasks and require little configuration. Function actions: These actions run developer-defined functions at the appropriate time. Custom actions: These are fully developer-defined actions, providing flexibility at the expense of complexity. ​ Pre-canned Actions Common actions shipped with Flows for managing conversation flow. To use them, just add them to your NodeConfig . ​ tts_say action Speaks text immediately using the TTS service. Copy Ask AI Copy Ask AI "pre_actions" : [ { "type" : "tts_say" , "text" : "Processing your request..." # Required } ] ​ end_conversation action Ends the conversation and closes the connection. Copy Ask AI Copy Ask AI "post_actions" : [ { "type" : "end_conversation" , "text" : "Goodbye!" # Optional farewell message } ] ​ Function Actions Actions that run developer-defined functions at the appropriate time. For example, if used in post_actions , they’ll run after the bot has finished talking and after any previous post_actions have finished. ​ function action Calls the developer-defined function at the appropriate time. Copy Ask AI Copy Ask AI "post_actions" : [ { "type" : "function" , "handler" : bot_turn_ended # Required } ] ​ Custom Actions Fully developer-defined actions, providing flexibility at the expense of complexity. Here’s the complexity: because these actions aren’t queued in the Pipecat pipeline, they may execute seemingly early if used in post_actions ; they’ll run immediately after the LLM completion is triggered but won’t wait around for the bot to finish talking. Why would you want this behavior? You might be writing an action that: Itself just queues another Frame into the Pipecat pipeline (meaning there would no benefit to waiting around for sequencing purposes) Does work that can be done a bit sooner, like logging that the LLM was updated Custom actions are composed of at least: ​ type str required String identifier for the action ​ handler Callable required Async or sync function that handles the action Example: Copy Ask AI Copy Ask AI # Define custom action handler async def custom_notification ( action : dict , flow_manager : FlowManager): """Custom action handler.""" message = action.get( "message" , "" ) await notify_user(message) # Use in node configuration "pre_actions" : [ { "type" : "notify" , "handler" : send_notification, "message" : "Attention!" , } ] ​ Exceptions ​ FlowError exception Base exception for all flow-related errors. Copy Ask AI Copy Ask AI from pipecat_flows import FlowError try : await flow_manager.set_node_from_config(config) except FlowError as e: print ( f "Flow error: { e } " ) ​ FlowInitializationError exception Raised when flow initialization fails. Copy Ask AI Copy Ask AI from pipecat_flows import FlowInitializationError try : await flow_manager.initialize() except FlowInitializationError as e: print ( f "Initialization failed: { e } " ) ​ FlowTransitionError exception Raised when a state transition fails. Copy Ask AI Copy Ask AI from pipecat_flows import FlowTransitionError try : await flow_manager.set_node_from_config(node_config) except FlowTransitionError as e: print ( f "Transition failed: { e } " ) ​ InvalidFunctionError exception Raised when an invalid or unavailable function is specified. Copy Ask AI Copy Ask AI from pipecat_flows import InvalidFunctionError try : await flow_manager.set_node_from_config({ "functions" : [{ "type" : "function" , "function" : { "name" : "invalid_function" } }] }) except InvalidFunctionError as e: print ( f "Invalid function: { e } " ) RTVI Observer PipelineParams On this page Installation Core Types FlowArgs FlowResult FlowConfig NodeConfig Function Handler Types ContextStrategy ContextStrategyConfig FlowsFunctionSchema FlowManager Methods State Management Usage Examples Function Properties Handler Signatures Return Types Provider-Specific Formats Actions Pre-canned Actions Function Actions Custom Actions Exceptions Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/flows_pipecat-flows_e3742b21.txt b/flows_pipecat-flows_e3742b21.txt
new file mode 100644
index 0000000000000000000000000000000000000000..e7987e9ee755840fe454493b25d78337dad5c966
--- /dev/null
+++ b/flows_pipecat-flows_e3742b21.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/frameworks/flows/pipecat-flows#function-properties
+Title: Pipecat Flows - Pipecat
+==================================================
+
+Pipecat Flows - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Frameworks Pipecat Flows Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline New to building conversational flows? Check out our Pipecat Flows guide first. ​ Installation Existing Pipecat installation Fresh Pipecat installation Copy Ask AI pip install pipecat-ai-flows ​ Core Types ​ FlowArgs ​ FlowArgs Dict[str, Any] Type alias for function handler arguments. ​ FlowResult ​ FlowResult TypedDict Base type for function handler results. Additional fields can be included as needed. Show Fields ​ status str Optional status field ​ error str Optional error message ​ FlowConfig ​ FlowConfig TypedDict Configuration for the entire conversation flow. Show Fields ​ initial_node str required Starting node identifier ​ nodes Dict[str, NodeConfig] required Map of node names to configurations ​ NodeConfig ​ NodeConfig TypedDict Configuration for a single node in the flow. Show Fields ​ name str The name of the node, used in debug logging in dynamic flows. If no name is specified, an automatically-generated UUID is used. Copy Ask AI # Example name "name" : "greeting" ​ role_messages List[dict] Defines the role or persona of the LLM. Required for the initial node and optional for subsequent nodes. Copy Ask AI # Example role messages "role_messages" : [ { "role" : "system" , "content" : "You are a helpful assistant..." } ], ​ task_messages List[dict] required Defines the task for a given node. Required for all nodes. Copy Ask AI # Example task messages "task_messages" : [ { "role" : "system" , # May be `user` depending on the LLM "content" : "Ask the user for their name..." } ], ​ context_strategy ContextStrategyConfig Strategy for managing context during transitions to this node. Copy Ask AI # Example context strategy configuration "context_strategy" : ContextStrategyConfig( strategy = ContextStrategy. RESET_WITH_SUMMARY , summary_prompt = "Summarize the key points discussed so far." ) ​ functions List[Union[dict, FlowsFunctionSchema]] required LLM function / tool call configurations, defined in one of the supported formats . Copy Ask AI # Using provider-specific dictionary format "functions" : [ { "type" : "function" , "function" : { "name" : "get_current_movies" , "handler" : get_movies, "description" : "Fetch movies currently playing" , "parameters" : { ... } }, } ] # Using FlowsFunctionSchema "functions" : [ FlowsFunctionSchema( name = "get_current_movies" , description = "Fetch movies currently playing" , properties = { ... }, required = [ ... ], handler = get_movies ) ] # Using direct functions (auto-configuration) "functions" : [get_movies] ​ pre_actions List[dict] Actions that execute before the LLM inference. For example, you can send a message to the TTS to speak a phrase (e.g. “Hold on a moment…”), which may be effective if an LLM function call takes time to execute. Copy Ask AI # Example pre_actions "pre_actions" : [ { "type" : "tts_say" , "text" : "Hold on a moment..." } ], ​ post_actions List[dict] Actions that execute after the LLM inference. For example, you can end the conversation. Copy Ask AI # Example post_actions "post_actions" : [ { "type" : "end_conversation" } ] ​ respond_immediately bool If set to False , the LLM will not respond immediately when the node is set, but will instead wait for the user to speak first before responding. Defaults to True . Copy Ask AI # Example usage "respond_immediately" : False ​ Function Handler Types ​ LegacyFunctionHandler Callable[[FlowArgs], Awaitable[FlowResult | ConsolidatedFunctionResult]] Legacy function handler that only receives arguments. Returns either: A FlowResult (⚠️ deprecated) A “consolidated” result tuple (result, next node) where: result is an optional FlowResult next node is an optional NodeConfig (for dynamic flows) or string (for static flows) ​ FlowFunctionHandler Callable[[FlowArgs, FlowManager], Awaitable[FlowResult | ConsolidatedFunctionResult]] Modern function handler that receives both arguments and FlowManager . Returns either: A FlowResult (⚠️ deprecated) A “consolidated” result tuple (result, next node) where: result is an optional FlowResult next node is an optional NodeConfig (for dynamic flows) or string (for static flows) ​ DirectFunction DirectFunction Function that is meant to be passed directly into a NodeConfig rather than into the handler field of a function configuration. It must be an async function with flow_manager: FlowManager as its first parameter. It must return a ConsolidatedFunctionResult , which is a tuple (result, next node) where: result is an optional FlowResult next node is an optional NodeConfig (for dynamic flows) or string (for static flows) ​ ContextStrategy ​ ContextStrategy Enum Strategy for managing conversation context during node transitions. Show Values ​ APPEND str Default strategy. Adds new messages to existing context. ​ RESET str Clears context and starts fresh with new messages. ​ RESET_WITH_SUMMARY str Resets context but includes an AI-generated summary. ​ ContextStrategyConfig ​ ContextStrategyConfig dataclass Configuration for context management strategy. Show Fields ​ strategy ContextStrategy required The strategy to use for context management ​ summary_prompt Optional[str] Required when using RESET_WITH_SUMMARY. Prompt text for generating the conversation summary. Copy Ask AI # Example usage config = ContextStrategyConfig( strategy = ContextStrategy. RESET_WITH_SUMMARY , summary_prompt = "Summarize the key points discussed so far." ) ​ FlowsFunctionSchema ​ FlowsFunctionSchema class A standardized schema for defining functions in Pipecat Flows with flow-specific properties. Show Constructor Parameters ​ name str required Name of the function ​ description str required Description of the function’s purpose ​ properties Dict[str, Any] required Dictionary defining properties types and descriptions ​ required List[str] required List of required parameter names ​ handler Optional[FunctionHandler] Function handler to process the function call ​ transition_to Optional[str] deprecated Target node to transition to after function execution Deprecated: instead of transition_to , use a “consolidated” handler that returns a tuple (result, next node). ​ transition_callback Optional[Callable] deprecated Callback function for dynamic transitions Deprecated: instead of transition_callback , use a “consolidated” handler that returns a tuple (result, next node). You cannot specify both transition_to and transition_callback in the same function schema. Example usage: Copy Ask AI from pipecat_flows import FlowsFunctionSchema # Define a function schema collect_name_schema = FlowsFunctionSchema( name = "collect_name" , description = "Record the user's name" , properties = { "name" : { "type" : "string" , "description" : "The user's name" } }, required = [ "name" ], handler = collect_name_handler ) # Use in node configuration node_config = { "name" : "greeting" , "task_messages" : [ { "role" : "system" , "content" : "Ask the user for their name." } ], "functions" : [collect_name_schema] } # Pass to flow manager await flow_manager.set_node_from_config(node_config) ​ FlowManager ​ FlowManager class Main class for managing conversation flows, supporting both static (configuration-driven) and dynamic (runtime-determined) flows. Show Constructor Parameters ​ task PipelineTask required Pipeline task for frame queueing ​ llm LLMService required LLM service instance (OpenAI, Anthropic, or Google). Must be initialized with the corresponding pipecat-ai provider dependency installed. ​ context_aggregator Any required Context aggregator used for pushing messages to the LLM service ​ tts Optional[Any] deprecated Optional TTS service for voice actions. Deprecated: No need to explicitly pass tts to FlowManager in order to use tts_say actions. ​ flow_config Optional[FlowConfig] Optional static flow configuration ​ context_strategy Optional[ContextStrategyConfig] Optional configuration for how context should be managed during transitions. Defaults to APPEND strategy if not specified. ​ Methods ​ initialize method Initialize the flow with starting messages. Show Parameters ​ initial_node NodeConfig The initial conversation node (needed for dynamic flows only). If not specified, you’ll need to call set_node_from_config() to kick off the conversation. Show Raises ​ FlowInitializationError If initialization fails ​ set_node method deprecated Set up a new conversation node programmatically (dynamic flows only). In dynamic flows, the application advances the conversation using set_node to set up each next node. In static flows, set_node is triggered under the hood when a node contains a transition_to field. Deprecated: use the following patterns instead of set_node : Prefer “consolidated” function handlers that return a tuple (result, next node), which implicitly sets up the next node Prefer passing your initial node to FlowManager.initialize() If you really need to set a node explicitly, use set_node_from_config() (note: its name will be read from its NodeConfig ) Show Parameters ​ node_id str required Identifier for the new node ​ node_config NodeConfig required Node configuration including messages, functions, and actions Show Raises ​ FlowError If node setup fails ​ set_node_from_config method Set up a new conversation node programmatically (dynamic flows only). Note that this method should only be used in rare circumstances. Most often, you should: Prefer “consolidated” function handlers that return a tuple (result, next node), which implicitly sets up the next node Prefer passing your initial node to FlowManager.initialize() Show Parameters ​ node_config NodeConfig required Node configuration including messages, functions, and actions Show Raises ​ FlowError If node setup fails ​ register_action method Register a handler for a custom action type. Show Parameters ​ action_type str required String identifier for the action ​ handler Callable required Async or sync function that handles the action ​ get_current_context method Get the current conversation context. Returns a list of messages in the current context, including system messages, user messages, and assistant responses. Show Returns ​ messages List[dict] List of messages in the current context Show Raises ​ FlowError If context aggregator is not available Example usage: Copy Ask AI # Access current conversation context context = flow_manager.get_current_context() # Use in handlers async def process_response ( args : FlowArgs) -> tuple[FlowResult, str ]: context = flow_manager.get_current_context() # Process conversation history return { "status" : "success" }, "next" ​ State Management The FlowManager provides a state dictionary for storing conversation data: Access state Access in transitions Copy Ask AI flow_manager.state: Dict[ str , Any] # Store data flow_manager.state[ "user_age" ] = 25 ​ Usage Examples Static Flow Dynamic Flow Copy Ask AI flow_config: FlowConfig = { "initial_node" : "greeting" , "nodes" : { "greeting" : { "role_messages" : [ { "role" : "system" , "content" : "You are a helpful assistant. Your responses will be converted to audio." } ], "task_messages" : [ { "role" : "system" , "content" : "Start by greeting the user and asking for their name." } ], "functions" : [{ "type" : "function" , "function" : { "name" : "collect_name" , "handler" : collect_name_handler, "description" : "Record user's name" , "parameters" : { ... } } }] } } } # Create and initialize the FlowManager flow_manager = FlowManager( task = task, llm = llm, context_aggregator = context_aggregator, flow_config = flow_config ) # Initialize the flow_manager to start the conversation await flow_manager.initialize() ​ Node Functions concept Functions that execute operations within a single conversational state, without switching nodes. Copy Ask AI Copy Ask AI from pipecat_flows import FlowArgs, FlowResult async def process_data ( args : FlowArgs) -> tuple[FlowResult, None ]: """Handle data processing within a node.""" data = args[ "data" ] result = await process(data) return { "status" : "success" , "processed_data" : result }, None # Function configuration { "type" : "function" , "function" : { "name" : "process_data" , "handler" : process_data, "description" : "Process user data" , "parameters" : { "type" : "object" , "properties" : { "data" : { "type" : "string" } } } } } ​ Edge Functions concept Functions that specify a transition between nodes (optionally processing data first). Copy Ask AI Copy Ask AI from pipecat_flows import FlowArgs, FlowResult async def next_step ( args : FlowArgs) -> tuple[ None , str ]: """Specify the next node to transition to.""" return None , "target_node" # Return NodeConfig instead of str for dynamic flows # Function configuration { "type" : "function" , "function" : { "name" : "next_step" , "handler" : next_step, "description" : "Transition to next node" , "parameters" : { "type" : "object" , "properties" : {}} } } ​ Function Properties ​ handler Optional[Callable] Async function that processes data within a node and/or specifies the next node ( more details here ). Can be specified as: Direct function reference Either a Callable function or a string with __function__: prefix (e.g., "__function__:process_data" ) to reference a function in the main script Direct Reference Function Token Copy Ask AI { "type" : "function" , "function" : { "name" : "process_data" , "handler" : process_data, # Callable function "parameters" : { ... } } } ​ transition_callback Optional[Callable] deprecated Handler for dynamic flow transitions. Deprecated: instead of transition_callback , use a “consolidated” handler that returns a tuple (result, next node). Must be an async function with one of these signatures: Copy Ask AI # New style (recommended) async def handle_transition ( args : Dict[ str , Any], result : FlowResult, flow_manager : FlowManager ) -> None : """Handle transition to next node.""" if result.available: # Type-safe access to result await flow_manager.set_node_from_config(create_confirmation_node()) else : await flow_manager.set_node_from_config( create_no_availability_node(result.alternative_times) ) # Legacy style (supported for backwards compatibility) async def handle_transition ( args : Dict[ str , Any], flow_manager : FlowManager ) -> None : """Handle transition to next node.""" await flow_manager.set_node_from_config(create_next_node()) The callback receives: args : Arguments from the function call result : Typed result from the function handler (new style only) flow_manager : Reference to the FlowManager instance Example usage: Copy Ask AI async def handle_availability_check ( args : Dict, result : TimeResult, # Typed result flow_manager : FlowManager ): """Handle availability check and transition based on result.""" if result.available: await flow_manager.set_node_from_config(create_confirmation_node()) else : await flow_manager.set_node_from_config( create_no_availability_node(result.alternative_times) ) # Use in function configuration { "type" : "function" , "function" : { "name" : "check_availability" , "handler" : check_availability, "parameters" : { ... }, "transition_callback" : handle_availability_check } } Note: A function cannot have both transition_to and transition_callback . ​ Handler Signatures Function handlers passed as a handler in a function configuration can be defined with three different signatures: Modern (Args + FlowManager) Legacy (Args Only) No Arguments Copy Ask AI async def handler_with_flow_manager ( args : FlowArgs, flow_manager : FlowManager) -> tuple[FlowResult, NodeConfig]: """Modern handler that receives both arguments and FlowManager access.""" # Access state previous_data = flow_manager.state.get( "stored_data" ) # Access pipeline resources await flow_manager.task.queue_frame(TTSSpeakFrame( "Processing your request..." )) # Store data in state for later flow_manager.state[ "new_data" ] = args[ "input" ] return { "status" : "success" , "result" : "Processed with flow access" }, create_next_node() The framework automatically detects which signature your handler is using and calls it appropriately. If you’re passing your function directly into your NodeConfig rather than as a handler in a function configuration, you’d use a somewhat different signature: Direct Copy Ask AI async def do_something ( flow_manager : FlowManager, foo : int , bar : str = "" ) -> tuple[FlowResult, NodeConfig]: """ Do something interesting. Args: foo (int): The foo to do something interesting with. bar (string): The bar to do something interesting with. Defaults to empty string. """ result = await fetch_data(foo, bar) next_node = create_end_node() return result, next_node ​ Return Types Success Response Error Response Copy Ask AI { "status" : "success" , "data" : "some data" # Optional additional data } ​ Provider-Specific Formats You don’t need to handle these format differences manually - use the standard format and the FlowManager will adapt it for your chosen provider. OpenAI Anthropic Google (Gemini) Copy Ask AI { "type" : "function" , "function" : { "name" : "function_name" , "handler" : handler, "description" : "Description" , "parameters" : { ... } } } ​ Actions pre_actions and post_actions are used to manage conversation flow. They are included in the NodeConfig and executed before and after the LLM completion, respectively. Three kinds of actions are available: Pre-canned actions: These actions perform common tasks and require little configuration. Function actions: These actions run developer-defined functions at the appropriate time. Custom actions: These are fully developer-defined actions, providing flexibility at the expense of complexity. ​ Pre-canned Actions Common actions shipped with Flows for managing conversation flow. To use them, just add them to your NodeConfig . ​ tts_say action Speaks text immediately using the TTS service. Copy Ask AI Copy Ask AI "pre_actions" : [ { "type" : "tts_say" , "text" : "Processing your request..." # Required } ] ​ end_conversation action Ends the conversation and closes the connection. Copy Ask AI Copy Ask AI "post_actions" : [ { "type" : "end_conversation" , "text" : "Goodbye!" # Optional farewell message } ] ​ Function Actions Actions that run developer-defined functions at the appropriate time. For example, if used in post_actions , they’ll run after the bot has finished talking and after any previous post_actions have finished. ​ function action Calls the developer-defined function at the appropriate time. Copy Ask AI Copy Ask AI "post_actions" : [ { "type" : "function" , "handler" : bot_turn_ended # Required } ] ​ Custom Actions Fully developer-defined actions, providing flexibility at the expense of complexity. Here’s the complexity: because these actions aren’t queued in the Pipecat pipeline, they may execute seemingly early if used in post_actions ; they’ll run immediately after the LLM completion is triggered but won’t wait around for the bot to finish talking. Why would you want this behavior? You might be writing an action that: Itself just queues another Frame into the Pipecat pipeline (meaning there would no benefit to waiting around for sequencing purposes) Does work that can be done a bit sooner, like logging that the LLM was updated Custom actions are composed of at least: ​ type str required String identifier for the action ​ handler Callable required Async or sync function that handles the action Example: Copy Ask AI Copy Ask AI # Define custom action handler async def custom_notification ( action : dict , flow_manager : FlowManager): """Custom action handler.""" message = action.get( "message" , "" ) await notify_user(message) # Use in node configuration "pre_actions" : [ { "type" : "notify" , "handler" : send_notification, "message" : "Attention!" , } ] ​ Exceptions ​ FlowError exception Base exception for all flow-related errors. Copy Ask AI Copy Ask AI from pipecat_flows import FlowError try : await flow_manager.set_node_from_config(config) except FlowError as e: print ( f "Flow error: { e } " ) ​ FlowInitializationError exception Raised when flow initialization fails. Copy Ask AI Copy Ask AI from pipecat_flows import FlowInitializationError try : await flow_manager.initialize() except FlowInitializationError as e: print ( f "Initialization failed: { e } " ) ​ FlowTransitionError exception Raised when a state transition fails. Copy Ask AI Copy Ask AI from pipecat_flows import FlowTransitionError try : await flow_manager.set_node_from_config(node_config) except FlowTransitionError as e: print ( f "Transition failed: { e } " ) ​ InvalidFunctionError exception Raised when an invalid or unavailable function is specified. Copy Ask AI Copy Ask AI from pipecat_flows import InvalidFunctionError try : await flow_manager.set_node_from_config({ "functions" : [{ "type" : "function" , "function" : { "name" : "invalid_function" } }] }) except InvalidFunctionError as e: print ( f "Invalid function: { e } " ) RTVI Observer PipelineParams On this page Installation Core Types FlowArgs FlowResult FlowConfig NodeConfig Function Handler Types ContextStrategy ContextStrategyConfig FlowsFunctionSchema FlowManager Methods State Management Usage Examples Function Properties Handler Signatures Return Types Provider-Specific Formats Actions Pre-canned Actions Function Actions Custom Actions Exceptions Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/frame_producer-consumer_11a802fa.txt b/frame_producer-consumer_11a802fa.txt
new file mode 100644
index 0000000000000000000000000000000000000000..6a2b408145570f5ce902658633236fabcdbc55e9
--- /dev/null
+++ b/frame_producer-consumer_11a802fa.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/frame/producer-consumer#overview
+Title: Producer & Consumer Processors - Pipecat
+==================================================
+
+Producer & Consumer Processors - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Advanced Frame Processors Producer & Consumer Processors Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Producer & Consumer Processors UserIdleProcessor Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview The Producer and Consumer processors work as a pair to route frames between different parts of a pipeline, particularly useful when working with ParallelPipeline . They allow you to selectively capture frames from one pipeline branch and inject them into another. ​ ProducerProcessor ProducerProcessor examines frames flowing through the pipeline, applies a filter to decide which frames to share, and optionally transforms these frames before sending them to connected consumers. ​ Constructor Parameters ​ filter Callable[[Frame], Awaitable[bool]] required An async function that determines which frames should be sent to consumers. Should return True for frames to be shared. ​ transformer Callable[[Frame], Awaitable[Frame]] default: "identity_transformer" Optional async function that transforms frames before sending to consumers. By default, passes frames unchanged. ​ passthrough bool default: "True" When True , passes all frames through the normal pipeline flow. When False , only passes through frames that don’t match the filter. ​ ConsumerProcessor ConsumerProcessor receives frames from a ProducerProcessor and injects them into its pipeline branch. ​ Constructor Parameters ​ producer ProducerProcessor required The producer processor that will send frames to this consumer. ​ transformer Callable[[Frame], Awaitable[Frame]] default: "identity_transformer" Optional async function that transforms frames before injecting them into the pipeline. ​ direction FrameDirection default: "FrameDirection.DOWNSTREAM" The direction in which to push received frames. Usually DOWNSTREAM to send frames forward in the pipeline. ​ Usage Examples ​ Basic Usage: Moving TTS Audio Between Branches Copy Ask AI # Create a producer that captures TTS audio frames async def is_tts_audio ( frame : Frame) -> bool : return isinstance (frame, TTSAudioRawFrame) # Define an async transformer function async def tts_to_input_audio_transformer ( frame : Frame) -> Frame: if isinstance (frame, TTSAudioRawFrame): # Convert TTS audio to input audio format return InputAudioRawFrame( audio = frame.audio, sample_rate = frame.sample_rate, num_channels = frame.num_channels ) return frame producer = ProducerProcessor( filter = is_tts_audio, transformer = tts_to_input_audio_transformer passthrough = True # Keep these frames in original pipeline ) # Create a consumer to receive the frames consumer = ConsumerProcessor( producer = producer, direction = FrameDirection. DOWNSTREAM ) # Use in a ParallelPipeline pipeline = Pipeline([ transport.input(), ParallelPipeline( # Branch 1: LLM for bot responses [ llm, tts, producer, # Capture TTS audio here ], # Branch 2: Audio processing branch [ consumer, # Receive TTS audio here llm, # Speech-to-Speech LLM (audio in) ] ), transport.output(), ]) Sentry Metrics UserIdleProcessor On this page Overview ProducerProcessor Constructor Parameters ConsumerProcessor Constructor Parameters Usage Examples Basic Usage: Moving TTS Audio Between Branches Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/frame_producer-consumer_12a355af.txt b/frame_producer-consumer_12a355af.txt
new file mode 100644
index 0000000000000000000000000000000000000000..114ef6548fbec3d46b1e180860daee38fd980c24
--- /dev/null
+++ b/frame_producer-consumer_12a355af.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/frame/producer-consumer#param-producer
+Title: Producer & Consumer Processors - Pipecat
+==================================================
+
+Producer & Consumer Processors - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Advanced Frame Processors Producer & Consumer Processors Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Producer & Consumer Processors UserIdleProcessor Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview The Producer and Consumer processors work as a pair to route frames between different parts of a pipeline, particularly useful when working with ParallelPipeline . They allow you to selectively capture frames from one pipeline branch and inject them into another. ​ ProducerProcessor ProducerProcessor examines frames flowing through the pipeline, applies a filter to decide which frames to share, and optionally transforms these frames before sending them to connected consumers. ​ Constructor Parameters ​ filter Callable[[Frame], Awaitable[bool]] required An async function that determines which frames should be sent to consumers. Should return True for frames to be shared. ​ transformer Callable[[Frame], Awaitable[Frame]] default: "identity_transformer" Optional async function that transforms frames before sending to consumers. By default, passes frames unchanged. ​ passthrough bool default: "True" When True , passes all frames through the normal pipeline flow. When False , only passes through frames that don’t match the filter. ​ ConsumerProcessor ConsumerProcessor receives frames from a ProducerProcessor and injects them into its pipeline branch. ​ Constructor Parameters ​ producer ProducerProcessor required The producer processor that will send frames to this consumer. ​ transformer Callable[[Frame], Awaitable[Frame]] default: "identity_transformer" Optional async function that transforms frames before injecting them into the pipeline. ​ direction FrameDirection default: "FrameDirection.DOWNSTREAM" The direction in which to push received frames. Usually DOWNSTREAM to send frames forward in the pipeline. ​ Usage Examples ​ Basic Usage: Moving TTS Audio Between Branches Copy Ask AI # Create a producer that captures TTS audio frames async def is_tts_audio ( frame : Frame) -> bool : return isinstance (frame, TTSAudioRawFrame) # Define an async transformer function async def tts_to_input_audio_transformer ( frame : Frame) -> Frame: if isinstance (frame, TTSAudioRawFrame): # Convert TTS audio to input audio format return InputAudioRawFrame( audio = frame.audio, sample_rate = frame.sample_rate, num_channels = frame.num_channels ) return frame producer = ProducerProcessor( filter = is_tts_audio, transformer = tts_to_input_audio_transformer passthrough = True # Keep these frames in original pipeline ) # Create a consumer to receive the frames consumer = ConsumerProcessor( producer = producer, direction = FrameDirection. DOWNSTREAM ) # Use in a ParallelPipeline pipeline = Pipeline([ transport.input(), ParallelPipeline( # Branch 1: LLM for bot responses [ llm, tts, producer, # Capture TTS audio here ], # Branch 2: Audio processing branch [ consumer, # Receive TTS audio here llm, # Speech-to-Speech LLM (audio in) ] ), transport.output(), ]) Sentry Metrics UserIdleProcessor On this page Overview ProducerProcessor Constructor Parameters ConsumerProcessor Constructor Parameters Usage Examples Basic Usage: Moving TTS Audio Between Branches Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/frame_producer-consumer_93e44eb1.txt b/frame_producer-consumer_93e44eb1.txt
new file mode 100644
index 0000000000000000000000000000000000000000..6fdc2d0c1a4e541ce130504b71a3d87057d0704f
--- /dev/null
+++ b/frame_producer-consumer_93e44eb1.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/frame/producer-consumer#constructor-parameters-2
+Title: Producer & Consumer Processors - Pipecat
+==================================================
+
+Producer & Consumer Processors - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Advanced Frame Processors Producer & Consumer Processors Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Producer & Consumer Processors UserIdleProcessor Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview The Producer and Consumer processors work as a pair to route frames between different parts of a pipeline, particularly useful when working with ParallelPipeline . They allow you to selectively capture frames from one pipeline branch and inject them into another. ​ ProducerProcessor ProducerProcessor examines frames flowing through the pipeline, applies a filter to decide which frames to share, and optionally transforms these frames before sending them to connected consumers. ​ Constructor Parameters ​ filter Callable[[Frame], Awaitable[bool]] required An async function that determines which frames should be sent to consumers. Should return True for frames to be shared. ​ transformer Callable[[Frame], Awaitable[Frame]] default: "identity_transformer" Optional async function that transforms frames before sending to consumers. By default, passes frames unchanged. ​ passthrough bool default: "True" When True , passes all frames through the normal pipeline flow. When False , only passes through frames that don’t match the filter. ​ ConsumerProcessor ConsumerProcessor receives frames from a ProducerProcessor and injects them into its pipeline branch. ​ Constructor Parameters ​ producer ProducerProcessor required The producer processor that will send frames to this consumer. ​ transformer Callable[[Frame], Awaitable[Frame]] default: "identity_transformer" Optional async function that transforms frames before injecting them into the pipeline. ​ direction FrameDirection default: "FrameDirection.DOWNSTREAM" The direction in which to push received frames. Usually DOWNSTREAM to send frames forward in the pipeline. ​ Usage Examples ​ Basic Usage: Moving TTS Audio Between Branches Copy Ask AI # Create a producer that captures TTS audio frames async def is_tts_audio ( frame : Frame) -> bool : return isinstance (frame, TTSAudioRawFrame) # Define an async transformer function async def tts_to_input_audio_transformer ( frame : Frame) -> Frame: if isinstance (frame, TTSAudioRawFrame): # Convert TTS audio to input audio format return InputAudioRawFrame( audio = frame.audio, sample_rate = frame.sample_rate, num_channels = frame.num_channels ) return frame producer = ProducerProcessor( filter = is_tts_audio, transformer = tts_to_input_audio_transformer passthrough = True # Keep these frames in original pipeline ) # Create a consumer to receive the frames consumer = ConsumerProcessor( producer = producer, direction = FrameDirection. DOWNSTREAM ) # Use in a ParallelPipeline pipeline = Pipeline([ transport.input(), ParallelPipeline( # Branch 1: LLM for bot responses [ llm, tts, producer, # Capture TTS audio here ], # Branch 2: Audio processing branch [ consumer, # Receive TTS audio here llm, # Speech-to-Speech LLM (audio in) ] ), transport.output(), ]) Sentry Metrics UserIdleProcessor On this page Overview ProducerProcessor Constructor Parameters ConsumerProcessor Constructor Parameters Usage Examples Basic Usage: Moving TTS Audio Between Branches Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/fundamentals_context-management_1d1ce99c.txt b/fundamentals_context-management_1d1ce99c.txt
new file mode 100644
index 0000000000000000000000000000000000000000..7cdd2f40040f3c9598bdc33e2ebda035b317b197
--- /dev/null
+++ b/fundamentals_context-management_1d1ce99c.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/guides/fundamentals/context-management
+Title: Context Management - Pipecat
+==================================================
+
+Context Management - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Fundamentals Context Management Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Guides Fundamentals Context Management Custom FrameProcessor Detecting Idle Users Ending a Pipeline Function Calling Muting User Input Recording Audio Recording Transcripts Features Gemini Multimodal Live Metrics Noise cancellation with Krisp OpenAI Audio Models and APIs Pipecat Flows Telephony Overview Dial-in: WebRTC (Daily) Dial-in: WebRTC (Twilio + Daily) Dial-in: Twilio (Media Streams) Dialout: WebRTC (Daily) Deploying your bot Overview Deployment pattern Example: Pipecat Cloud Example: Fly.io Example: Cerebrium Example: Modal ​ What is Context in Pipecat? In Pipecat, context refers to the text that the LLM uses to perform an inference. Commonly, this is the text inputted to the LLM and outputted from the LLM. The context consists of a list of alternating user/assistant messages that represents the information you want an LLM to respond to. Since Pipecat is a real-time voice (and multimodal) AI framework, the context serves as the collective history of the entire conversation. ​ How Context Updates During Conversations After every user and bot turn in the conversation, processors in the pipeline push frames to update the context: STT Service : Pushes TranscriptionFrame objects that represent what the user says. LLM and TTS Services : Work together to represent what the bot says. The LLM streams tokens (as LLMTextFrame s) to the TTS service, which outputs TTSTextFrame s representing the bot’s spoken words. ​ Setting Up Context Management Pipecat includes a context aggregator class that creates and manages context for both user and assistant messages. Here’s how to set it up: ​ 1. Create the Context and Context Aggregator Copy Ask AI # Create LLM service llm = OpenAILLMService( api_key = os.getenv( "OPENAI_API_KEY" )) # Create context context = OpenAILLMContext(messages, tools) # Create context aggregator instance context_aggregator = llm.create_context_aggregator(context) The context (which represents the conversation) is passed to the context aggregator. This ensures that both user and assistant instances of the context aggregators have access to the shared conversation context. ​ 2. Add Context Aggregators to Your Pipeline Copy Ask AI pipeline = Pipeline([ transport.input(), stt, context_aggregator.user(), # User context aggregator llm, tts, transport.output(), context_aggregator.assistant(), # Assistant context aggregator ]) ​ Context Aggregator Placement The placement of context aggregator instances in your pipeline is crucial for proper operation: ​ User Context Aggregator Place the user context aggregator downstream from the STT service . Since the user’s speech results in TranscriptionFrame objects pushed by the STT service, the user aggregator needs to be positioned to collect these frames. ​ Assistant Context Aggregator Place the assistant context aggregator after transport.output() . This positioning is important because: The TTS service outputs spoken words in addition to audio The assistant aggregator must be downstream to collect those frames It ensures context updates happen word-by-word for specific services (e.g. Cartesia, ElevenLabs, and Rime) Your context stays updated at the word level in case an interruption occurs Always place the assistant context aggregator after transport.output() to ensure proper word-level context updates during interruptions. ​ Manually Managing Context You can programmatically add new messages to the context by pushing or queueing specific frames: ​ Adding Messages LLMMessagesAppendFrame : Appends a new message to the existing context LLMMessagesUpdateFrame : Completely replaces the existing context with new context provided in the frame ​ Retrieving Current Context The context aggregator provides a get_context_frame() method to obtain the latest context: Copy Ask AI await task.queue_frames([context_aggregator.user().get_context_frame()]) ​ Triggering Bot Responses You’ll commonly use this manual mechanism—obtaining the current context and pushing/queueing it—to trigger the bot to speak in two scenarios: Starting a pipeline where the bot should speak first After pushing new context frames using LLMMessagesAppendFrame or LLMMessagesUpdateFrame This gives you fine-grained control over when and how the bot responds during the conversation flow. Guides Custom FrameProcessor On this page What is Context in Pipecat? How Context Updates During Conversations Setting Up Context Management 1. Create the Context and Context Aggregator 2. Add Context Aggregators to Your Pipeline Context Aggregator Placement User Context Aggregator Assistant Context Aggregator Manually Managing Context Adding Messages Retrieving Current Context Triggering Bot Responses Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/fundamentals_context-management_40ddd2ea.txt b/fundamentals_context-management_40ddd2ea.txt
new file mode 100644
index 0000000000000000000000000000000000000000..0a5432b52737abdf6670f4893caa297fca2e55f2
--- /dev/null
+++ b/fundamentals_context-management_40ddd2ea.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/guides/fundamentals/context-management#2-add-context-aggregators-to-your-pipeline
+Title: Context Management - Pipecat
+==================================================
+
+Context Management - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Fundamentals Context Management Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Guides Fundamentals Context Management Custom FrameProcessor Detecting Idle Users Ending a Pipeline Function Calling Muting User Input Recording Audio Recording Transcripts Features Gemini Multimodal Live Metrics Noise cancellation with Krisp OpenAI Audio Models and APIs Pipecat Flows Telephony Overview Dial-in: WebRTC (Daily) Dial-in: WebRTC (Twilio + Daily) Dial-in: Twilio (Media Streams) Dialout: WebRTC (Daily) Deploying your bot Overview Deployment pattern Example: Pipecat Cloud Example: Fly.io Example: Cerebrium Example: Modal ​ What is Context in Pipecat? In Pipecat, context refers to the text that the LLM uses to perform an inference. Commonly, this is the text inputted to the LLM and outputted from the LLM. The context consists of a list of alternating user/assistant messages that represents the information you want an LLM to respond to. Since Pipecat is a real-time voice (and multimodal) AI framework, the context serves as the collective history of the entire conversation. ​ How Context Updates During Conversations After every user and bot turn in the conversation, processors in the pipeline push frames to update the context: STT Service : Pushes TranscriptionFrame objects that represent what the user says. LLM and TTS Services : Work together to represent what the bot says. The LLM streams tokens (as LLMTextFrame s) to the TTS service, which outputs TTSTextFrame s representing the bot’s spoken words. ​ Setting Up Context Management Pipecat includes a context aggregator class that creates and manages context for both user and assistant messages. Here’s how to set it up: ​ 1. Create the Context and Context Aggregator Copy Ask AI # Create LLM service llm = OpenAILLMService( api_key = os.getenv( "OPENAI_API_KEY" )) # Create context context = OpenAILLMContext(messages, tools) # Create context aggregator instance context_aggregator = llm.create_context_aggregator(context) The context (which represents the conversation) is passed to the context aggregator. This ensures that both user and assistant instances of the context aggregators have access to the shared conversation context. ​ 2. Add Context Aggregators to Your Pipeline Copy Ask AI pipeline = Pipeline([ transport.input(), stt, context_aggregator.user(), # User context aggregator llm, tts, transport.output(), context_aggregator.assistant(), # Assistant context aggregator ]) ​ Context Aggregator Placement The placement of context aggregator instances in your pipeline is crucial for proper operation: ​ User Context Aggregator Place the user context aggregator downstream from the STT service . Since the user’s speech results in TranscriptionFrame objects pushed by the STT service, the user aggregator needs to be positioned to collect these frames. ​ Assistant Context Aggregator Place the assistant context aggregator after transport.output() . This positioning is important because: The TTS service outputs spoken words in addition to audio The assistant aggregator must be downstream to collect those frames It ensures context updates happen word-by-word for specific services (e.g. Cartesia, ElevenLabs, and Rime) Your context stays updated at the word level in case an interruption occurs Always place the assistant context aggregator after transport.output() to ensure proper word-level context updates during interruptions. ​ Manually Managing Context You can programmatically add new messages to the context by pushing or queueing specific frames: ​ Adding Messages LLMMessagesAppendFrame : Appends a new message to the existing context LLMMessagesUpdateFrame : Completely replaces the existing context with new context provided in the frame ​ Retrieving Current Context The context aggregator provides a get_context_frame() method to obtain the latest context: Copy Ask AI await task.queue_frames([context_aggregator.user().get_context_frame()]) ​ Triggering Bot Responses You’ll commonly use this manual mechanism—obtaining the current context and pushing/queueing it—to trigger the bot to speak in two scenarios: Starting a pipeline where the bot should speak first After pushing new context frames using LLMMessagesAppendFrame or LLMMessagesUpdateFrame This gives you fine-grained control over when and how the bot responds during the conversation flow. Guides Custom FrameProcessor On this page What is Context in Pipecat? How Context Updates During Conversations Setting Up Context Management 1. Create the Context and Context Aggregator 2. Add Context Aggregators to Your Pipeline Context Aggregator Placement User Context Aggregator Assistant Context Aggregator Manually Managing Context Adding Messages Retrieving Current Context Triggering Bot Responses Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/fundamentals_context-management_88103605.txt b/fundamentals_context-management_88103605.txt
new file mode 100644
index 0000000000000000000000000000000000000000..4daaedbd676797f5988d571d726de2846e2914a5
--- /dev/null
+++ b/fundamentals_context-management_88103605.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/guides/fundamentals/context-management#manually-managing-context
+Title: Context Management - Pipecat
+==================================================
+
+Context Management - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Fundamentals Context Management Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Guides Fundamentals Context Management Custom FrameProcessor Detecting Idle Users Ending a Pipeline Function Calling Muting User Input Recording Audio Recording Transcripts Features Gemini Multimodal Live Metrics Noise cancellation with Krisp OpenAI Audio Models and APIs Pipecat Flows Telephony Overview Dial-in: WebRTC (Daily) Dial-in: WebRTC (Twilio + Daily) Dial-in: Twilio (Media Streams) Dialout: WebRTC (Daily) Deploying your bot Overview Deployment pattern Example: Pipecat Cloud Example: Fly.io Example: Cerebrium Example: Modal ​ What is Context in Pipecat? In Pipecat, context refers to the text that the LLM uses to perform an inference. Commonly, this is the text inputted to the LLM and outputted from the LLM. The context consists of a list of alternating user/assistant messages that represents the information you want an LLM to respond to. Since Pipecat is a real-time voice (and multimodal) AI framework, the context serves as the collective history of the entire conversation. ​ How Context Updates During Conversations After every user and bot turn in the conversation, processors in the pipeline push frames to update the context: STT Service : Pushes TranscriptionFrame objects that represent what the user says. LLM and TTS Services : Work together to represent what the bot says. The LLM streams tokens (as LLMTextFrame s) to the TTS service, which outputs TTSTextFrame s representing the bot’s spoken words. ​ Setting Up Context Management Pipecat includes a context aggregator class that creates and manages context for both user and assistant messages. Here’s how to set it up: ​ 1. Create the Context and Context Aggregator Copy Ask AI # Create LLM service llm = OpenAILLMService( api_key = os.getenv( "OPENAI_API_KEY" )) # Create context context = OpenAILLMContext(messages, tools) # Create context aggregator instance context_aggregator = llm.create_context_aggregator(context) The context (which represents the conversation) is passed to the context aggregator. This ensures that both user and assistant instances of the context aggregators have access to the shared conversation context. ​ 2. Add Context Aggregators to Your Pipeline Copy Ask AI pipeline = Pipeline([ transport.input(), stt, context_aggregator.user(), # User context aggregator llm, tts, transport.output(), context_aggregator.assistant(), # Assistant context aggregator ]) ​ Context Aggregator Placement The placement of context aggregator instances in your pipeline is crucial for proper operation: ​ User Context Aggregator Place the user context aggregator downstream from the STT service . Since the user’s speech results in TranscriptionFrame objects pushed by the STT service, the user aggregator needs to be positioned to collect these frames. ​ Assistant Context Aggregator Place the assistant context aggregator after transport.output() . This positioning is important because: The TTS service outputs spoken words in addition to audio The assistant aggregator must be downstream to collect those frames It ensures context updates happen word-by-word for specific services (e.g. Cartesia, ElevenLabs, and Rime) Your context stays updated at the word level in case an interruption occurs Always place the assistant context aggregator after transport.output() to ensure proper word-level context updates during interruptions. ​ Manually Managing Context You can programmatically add new messages to the context by pushing or queueing specific frames: ​ Adding Messages LLMMessagesAppendFrame : Appends a new message to the existing context LLMMessagesUpdateFrame : Completely replaces the existing context with new context provided in the frame ​ Retrieving Current Context The context aggregator provides a get_context_frame() method to obtain the latest context: Copy Ask AI await task.queue_frames([context_aggregator.user().get_context_frame()]) ​ Triggering Bot Responses You’ll commonly use this manual mechanism—obtaining the current context and pushing/queueing it—to trigger the bot to speak in two scenarios: Starting a pipeline where the bot should speak first After pushing new context frames using LLMMessagesAppendFrame or LLMMessagesUpdateFrame This gives you fine-grained control over when and how the bot responds during the conversation flow. Guides Custom FrameProcessor On this page What is Context in Pipecat? How Context Updates During Conversations Setting Up Context Management 1. Create the Context and Context Aggregator 2. Add Context Aggregators to Your Pipeline Context Aggregator Placement User Context Aggregator Assistant Context Aggregator Manually Managing Context Adding Messages Retrieving Current Context Triggering Bot Responses Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/fundamentals_context-management_ebd29b31.txt b/fundamentals_context-management_ebd29b31.txt
new file mode 100644
index 0000000000000000000000000000000000000000..d948d2f6cb61b7014ece0055ba70a09a539a333d
--- /dev/null
+++ b/fundamentals_context-management_ebd29b31.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/guides/fundamentals/context-management#setting-up-context-management
+Title: Context Management - Pipecat
+==================================================
+
+Context Management - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Fundamentals Context Management Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Guides Fundamentals Context Management Custom FrameProcessor Detecting Idle Users Ending a Pipeline Function Calling Muting User Input Recording Audio Recording Transcripts Features Gemini Multimodal Live Metrics Noise cancellation with Krisp OpenAI Audio Models and APIs Pipecat Flows Telephony Overview Dial-in: WebRTC (Daily) Dial-in: WebRTC (Twilio + Daily) Dial-in: Twilio (Media Streams) Dialout: WebRTC (Daily) Deploying your bot Overview Deployment pattern Example: Pipecat Cloud Example: Fly.io Example: Cerebrium Example: Modal ​ What is Context in Pipecat? In Pipecat, context refers to the text that the LLM uses to perform an inference. Commonly, this is the text inputted to the LLM and outputted from the LLM. The context consists of a list of alternating user/assistant messages that represents the information you want an LLM to respond to. Since Pipecat is a real-time voice (and multimodal) AI framework, the context serves as the collective history of the entire conversation. ​ How Context Updates During Conversations After every user and bot turn in the conversation, processors in the pipeline push frames to update the context: STT Service : Pushes TranscriptionFrame objects that represent what the user says. LLM and TTS Services : Work together to represent what the bot says. The LLM streams tokens (as LLMTextFrame s) to the TTS service, which outputs TTSTextFrame s representing the bot’s spoken words. ​ Setting Up Context Management Pipecat includes a context aggregator class that creates and manages context for both user and assistant messages. Here’s how to set it up: ​ 1. Create the Context and Context Aggregator Copy Ask AI # Create LLM service llm = OpenAILLMService( api_key = os.getenv( "OPENAI_API_KEY" )) # Create context context = OpenAILLMContext(messages, tools) # Create context aggregator instance context_aggregator = llm.create_context_aggregator(context) The context (which represents the conversation) is passed to the context aggregator. This ensures that both user and assistant instances of the context aggregators have access to the shared conversation context. ​ 2. Add Context Aggregators to Your Pipeline Copy Ask AI pipeline = Pipeline([ transport.input(), stt, context_aggregator.user(), # User context aggregator llm, tts, transport.output(), context_aggregator.assistant(), # Assistant context aggregator ]) ​ Context Aggregator Placement The placement of context aggregator instances in your pipeline is crucial for proper operation: ​ User Context Aggregator Place the user context aggregator downstream from the STT service . Since the user’s speech results in TranscriptionFrame objects pushed by the STT service, the user aggregator needs to be positioned to collect these frames. ​ Assistant Context Aggregator Place the assistant context aggregator after transport.output() . This positioning is important because: The TTS service outputs spoken words in addition to audio The assistant aggregator must be downstream to collect those frames It ensures context updates happen word-by-word for specific services (e.g. Cartesia, ElevenLabs, and Rime) Your context stays updated at the word level in case an interruption occurs Always place the assistant context aggregator after transport.output() to ensure proper word-level context updates during interruptions. ​ Manually Managing Context You can programmatically add new messages to the context by pushing or queueing specific frames: ​ Adding Messages LLMMessagesAppendFrame : Appends a new message to the existing context LLMMessagesUpdateFrame : Completely replaces the existing context with new context provided in the frame ​ Retrieving Current Context The context aggregator provides a get_context_frame() method to obtain the latest context: Copy Ask AI await task.queue_frames([context_aggregator.user().get_context_frame()]) ​ Triggering Bot Responses You’ll commonly use this manual mechanism—obtaining the current context and pushing/queueing it—to trigger the bot to speak in two scenarios: Starting a pipeline where the bot should speak first After pushing new context frames using LLMMessagesAppendFrame or LLMMessagesUpdateFrame This gives you fine-grained control over when and how the bot responds during the conversation flow. Guides Custom FrameProcessor On this page What is Context in Pipecat? How Context Updates During Conversations Setting Up Context Management 1. Create the Context and Context Aggregator 2. Add Context Aggregators to Your Pipeline Context Aggregator Placement User Context Aggregator Assistant Context Aggregator Manually Managing Context Adding Messages Retrieving Current Context Triggering Bot Responses Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/fundamentals_custom-frame-processor_c249fb20.txt b/fundamentals_custom-frame-processor_c249fb20.txt
new file mode 100644
index 0000000000000000000000000000000000000000..3b6d7107a6b7f0e36570a80701c86fb83f4461f0
--- /dev/null
+++ b/fundamentals_custom-frame-processor_c249fb20.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/guides/fundamentals/custom-frame-processor#best-practices
+Title: Custom FrameProcessor - Pipecat
+==================================================
+
+Custom FrameProcessor - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Fundamentals Custom FrameProcessor Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Guides Fundamentals Context Management Custom FrameProcessor Detecting Idle Users Ending a Pipeline Function Calling Muting User Input Recording Audio Recording Transcripts Features Gemini Multimodal Live Metrics Noise cancellation with Krisp OpenAI Audio Models and APIs Pipecat Flows Telephony Overview Dial-in: WebRTC (Daily) Dial-in: WebRTC (Twilio + Daily) Dial-in: Twilio (Media Streams) Dialout: WebRTC (Daily) Deploying your bot Overview Deployment pattern Example: Pipecat Cloud Example: Fly.io Example: Cerebrium Example: Modal Pipecat’s architecture is made up of a Pipeline, FrameProcessors, and Frames. See the Core Concepts for a full review. From that architecture, recall that FrameProcessors are the workers in the pipeline that receive frames and complete actions based on the frames received. Pipecat comes with many FrameProcessors built in. These consist of services, like OpenAILLMService or CartesiaTTSService , utilities, like UserIdleProcessor , and other things. Largely, you can build most of your application with these built-in FrameProcessors, but commonly, your application code may require custom frame processing logic. For example, you may want to perform an action as a result of a frame that’s pushed in the pipeline. ​ Example: ImageSyncAggregator Let’s look at an example custom FrameProcessor that synchronizes images with bot speech: Copy Ask AI class ImageSyncAggregator ( FrameProcessor ): def __init__ ( self , speaking_path : str , waiting_path : str ): super (). __init__ () self ._speaking_image = Image.open(speaking_path) self ._speaking_image_format = self ._speaking_image.format self ._speaking_image_bytes = self ._speaking_image.tobytes() self ._waiting_image = Image.open(waiting_path) self ._waiting_image_format = self ._waiting_image.format self ._waiting_image_bytes = self ._waiting_image.tobytes() async def process_frame ( self , frame : Frame, direction : FrameDirection): await super ().process_frame(frame, direction) if isinstance (frame, BotStartedSpeakingFrame): await self .push_frame( OutputImageRawFrame( image = self ._speaking_image_bytes, size = ( 1024 , 1024 ), format = self ._speaking_image_format, ) ) elif isinstance (frame, BotStoppedSpeakingFrame): await self .push_frame( OutputImageRawFrame( image = self ._waiting_image_bytes, size = ( 1024 , 1024 ), format = self ._waiting_image_format, ) ) await self .push_frame(frame) This example custom FrameProcessor looks for BotStartedSpeakingFrame and BotStoppedSpeakingFrame . When it sees a BotStartedSpeakingFrame , it will show an image that says the bot is speaking. When it sees a BotStoppedSpeakingFrame , it will show an image that says the bot is not speaking. See this working example using the ImageSyncAggregator FrameProcessor ​ Adding to a Pipeline This custom FrameProcessor can be added to a Pipeline just before the transport output: Copy Ask AI # Create and initialize the custom FrameProcessor image_sync_aggregator = ImageSyncAggregator( os.path.join(os.path.dirname( __file__ ), "assets" , "speaking.png" ), os.path.join(os.path.dirname( __file__ ), "assets" , "waiting.png" ), ) pipeline = Pipeline( [ transport.input(), stt, context_aggregator.user(), llm, tts, image_sync_aggregator, # Our custom FrameProcessor transport.output(), context_aggregator.assistant(), ] ) With this positioning, the ImageSyncAggregator FrameProcessor will receive the BotStartedSpeakingFrame and BotStoppedSpeakingFrame outputted by the TTS processor and then push its own frame— OutputImageRawFrame —to the output transport. ​ Key Requirements FrameProcessors must inherit from the base FrameProcessor class. This ensures that your custom FrameProcessor will correctly handle frames like StartFrame , EndFrame , StartInterruptionFrame without having to write custom logic for those frames. This inheritance also provides it with the ability to process_frame() and push_frame() : process_frame() is what allows the FrameProcessor to receive frames and add custom conditional logic based on the frames that are received. push_frame() allows the FrameProcessor to push frames to the pipeline. Normally, frames are pushed DOWNSTREAM, but based on which processors need the output, you can also push UPSTREAM or in both directions. ​ Essential Implementation Details To ensure proper base class inheritance, it’s critical to include: super().__init__() in your __init__ method await super().process_frame(frame, direction) in your process_frame() method Copy Ask AI class MyCustomProcessor ( FrameProcessor ): def __init__ ( self , ** kwargs ): super (). __init__ ( ** kwargs) # ✅ Required # Your initialization code here async def process_frame ( self , frame : Frame, direction : FrameDirection): await super ().process_frame(frame, direction) # ✅ Required # Your custom frame processing logic here if isinstance (frame, SomeSpecificFrame): # Handle the frame pass await self .push_frame(frame) # ✅ Required - pass frame through ​ Critical Responsibility: Frame Forwarding FrameProcessors receive all frames that are pushed through the pipeline. This gives them a lot of power, but also a great responsibility. Critically, they must push all frames through the pipeline; if they don’t, they block frames from moving through the Pipeline, which will cause issues in how your application functions. You can see this at work in the ImageSyncAggregator ’s process_frame() method. It handles both bot speaking frames and also has an await self.push_frame(frame) which pushes the frame through to the next processor in the pipeline. ​ Frame Direction When pushing frames, you can specify the direction: Copy Ask AI # Push downstream (default) await self .push_frame(frame) await self .push_frame(frame, FrameDirection. DOWNSTREAM ) # Push upstream await self .push_frame(frame, FrameDirection. UPSTREAM ) Most custom FrameProcessors will push frames downstream, but upstream can be useful for sending control frames or error notifications back up the pipeline. ​ Best Practices Always call the parent methods : Use super().__init__() and await super().process_frame() Forward all frames : Make sure every frame is pushed through with await self.push_frame(frame) Handle frames conditionally : Use isinstance() checks to handle specific frame types Use proper error handling : Wrap risky operations in try/catch blocks Position carefully in pipeline : Consider where in the pipeline your processor needs to be to receive the right frames With these patterns, you can create powerful custom FrameProcessors that extend Pipecat’s capabilities for your specific use case. Context Management Detecting Idle Users On this page Example: ImageSyncAggregator Adding to a Pipeline Key Requirements Essential Implementation Details Critical Responsibility: Frame Forwarding Frame Direction Best Practices Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/fundamentals_detecting-user-idle_00905d86.txt b/fundamentals_detecting-user-idle_00905d86.txt
new file mode 100644
index 0000000000000000000000000000000000000000..b478fdc89f2d6bf02a4febd82f25540117d7b200
--- /dev/null
+++ b/fundamentals_detecting-user-idle_00905d86.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/guides/fundamentals/detecting-user-idle#best-practices
+Title: Detecting Idle Users - Pipecat
+==================================================
+
+Detecting Idle Users - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Fundamentals Detecting Idle Users Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Guides Fundamentals Context Management Custom FrameProcessor Detecting Idle Users Ending a Pipeline Function Calling Muting User Input Recording Audio Recording Transcripts Features Gemini Multimodal Live Metrics Noise cancellation with Krisp OpenAI Audio Models and APIs Pipecat Flows Telephony Overview Dial-in: WebRTC (Daily) Dial-in: WebRTC (Twilio + Daily) Dial-in: Twilio (Media Streams) Dialout: WebRTC (Daily) Deploying your bot Overview Deployment pattern Example: Pipecat Cloud Example: Fly.io Example: Cerebrium Example: Modal ​ Overview In conversational applications, it’s important to handle situations where users go silent or inactive. Pipecat’s UserIdleProcessor helps you detect when users haven’t spoken for a defined period, allowing your bot to respond appropriately. ​ How It Works The UserIdleProcessor monitors user activity and: Starts tracking after the first interaction (user or bot speaking) Resets a timer whenever the user speaks Calls your custom callback function when the user is idle for longer than the timeout period Provides a retry counter to track consecutive idle periods Allows you to implement escalating responses or gracefully end the conversation The processor uses speech events (not audio frames) to detect activity, so it requires an active speech-to-text service or a transport with built-in speech detection. ​ Basic Implementation ​ Step 1: Create a Handler Function First, create a callback function that will be triggered when the user is idle: Copy Ask AI # Simple handler that doesn't use retry count async def handle_user_idle ( processor ): # Send a reminder to the user await processor.push_frame(TTSSpeakFrame( "Are you still there?" )) # OR # Advanced handler with retry logic async def handle_user_idle ( processor , retry_count ): if retry_count == 1 : # First attempt - gentle reminder await processor.push_frame(TTSSpeakFrame( "Are you still there?" )) return True # Continue monitoring elif retry_count == 2 : # Second attempt - more direct prompt await processor.push_frame(TTSSpeakFrame( "Would you like to continue our conversation?" )) return True # Continue monitoring else : # Third attempt - end conversation await processor.push_frame(TTSSpeakFrame( "I'll leave you for now. Have a nice day!" )) await processor.push_frame(EndFrame(), FrameDirection. UPSTREAM ) return False # Stop monitoring ​ Step 2: Create the Idle Processor Initialize the processor with your callback and desired timeout: Copy Ask AI from pipecat.processors.user_idle_processor import UserIdleProcessor user_idle = UserIdleProcessor( callback = handle_user_idle, # Your callback function timeout = 5.0 , # Seconds of inactivity before triggering ) ​ Step 3: Add to Your Pipeline Place the processor after speech detection but before context handling: Copy Ask AI pipeline = Pipeline( [ transport.input(), stt, # Speech-to-text user_idle, # Add idle detection here context_aggregator.user(), llm, tts, transport.output(), context_aggregator.assistant(), ] ) ​ Best Practices Set appropriate timeouts : Shorter timeouts (5-10 seconds) work well for voice conversations Use escalating responses : Start with gentle reminders and gradually become more direct Limit retry attempts : After 2-3 unsuccessful attempts, consider ending the conversation gracefully by pushing an EndFrame ​ Next Steps Try the User Idle Example Explore a complete working example that demonstrates how to detect and respond to user inactivity in Pipecat. UserIdleProcessor Reference Read the complete API reference documentation for advanced configuration options and callback patterns. Implementing idle user detection improves the conversational experience by ensuring your bot can handle periods of user inactivity gracefully, either by prompting for re-engagement or politely ending the conversation when appropriate. Custom FrameProcessor Ending a Pipeline On this page Overview How It Works Basic Implementation Step 1: Create a Handler Function Step 2: Create the Idle Processor Step 3: Add to Your Pipeline Best Practices Next Steps Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/fundamentals_detecting-user-idle_0e0bc2e8.txt b/fundamentals_detecting-user-idle_0e0bc2e8.txt
new file mode 100644
index 0000000000000000000000000000000000000000..f7f8798e8d957d1409c3a7a485541beb99554947
--- /dev/null
+++ b/fundamentals_detecting-user-idle_0e0bc2e8.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/guides/fundamentals/detecting-user-idle
+Title: Detecting Idle Users - Pipecat
+==================================================
+
+Detecting Idle Users - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Fundamentals Detecting Idle Users Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Guides Fundamentals Context Management Custom FrameProcessor Detecting Idle Users Ending a Pipeline Function Calling Muting User Input Recording Audio Recording Transcripts Features Gemini Multimodal Live Metrics Noise cancellation with Krisp OpenAI Audio Models and APIs Pipecat Flows Telephony Overview Dial-in: WebRTC (Daily) Dial-in: WebRTC (Twilio + Daily) Dial-in: Twilio (Media Streams) Dialout: WebRTC (Daily) Deploying your bot Overview Deployment pattern Example: Pipecat Cloud Example: Fly.io Example: Cerebrium Example: Modal ​ Overview In conversational applications, it’s important to handle situations where users go silent or inactive. Pipecat’s UserIdleProcessor helps you detect when users haven’t spoken for a defined period, allowing your bot to respond appropriately. ​ How It Works The UserIdleProcessor monitors user activity and: Starts tracking after the first interaction (user or bot speaking) Resets a timer whenever the user speaks Calls your custom callback function when the user is idle for longer than the timeout period Provides a retry counter to track consecutive idle periods Allows you to implement escalating responses or gracefully end the conversation The processor uses speech events (not audio frames) to detect activity, so it requires an active speech-to-text service or a transport with built-in speech detection. ​ Basic Implementation ​ Step 1: Create a Handler Function First, create a callback function that will be triggered when the user is idle: Copy Ask AI # Simple handler that doesn't use retry count async def handle_user_idle ( processor ): # Send a reminder to the user await processor.push_frame(TTSSpeakFrame( "Are you still there?" )) # OR # Advanced handler with retry logic async def handle_user_idle ( processor , retry_count ): if retry_count == 1 : # First attempt - gentle reminder await processor.push_frame(TTSSpeakFrame( "Are you still there?" )) return True # Continue monitoring elif retry_count == 2 : # Second attempt - more direct prompt await processor.push_frame(TTSSpeakFrame( "Would you like to continue our conversation?" )) return True # Continue monitoring else : # Third attempt - end conversation await processor.push_frame(TTSSpeakFrame( "I'll leave you for now. Have a nice day!" )) await processor.push_frame(EndFrame(), FrameDirection. UPSTREAM ) return False # Stop monitoring ​ Step 2: Create the Idle Processor Initialize the processor with your callback and desired timeout: Copy Ask AI from pipecat.processors.user_idle_processor import UserIdleProcessor user_idle = UserIdleProcessor( callback = handle_user_idle, # Your callback function timeout = 5.0 , # Seconds of inactivity before triggering ) ​ Step 3: Add to Your Pipeline Place the processor after speech detection but before context handling: Copy Ask AI pipeline = Pipeline( [ transport.input(), stt, # Speech-to-text user_idle, # Add idle detection here context_aggregator.user(), llm, tts, transport.output(), context_aggregator.assistant(), ] ) ​ Best Practices Set appropriate timeouts : Shorter timeouts (5-10 seconds) work well for voice conversations Use escalating responses : Start with gentle reminders and gradually become more direct Limit retry attempts : After 2-3 unsuccessful attempts, consider ending the conversation gracefully by pushing an EndFrame ​ Next Steps Try the User Idle Example Explore a complete working example that demonstrates how to detect and respond to user inactivity in Pipecat. UserIdleProcessor Reference Read the complete API reference documentation for advanced configuration options and callback patterns. Implementing idle user detection improves the conversational experience by ensuring your bot can handle periods of user inactivity gracefully, either by prompting for re-engagement or politely ending the conversation when appropriate. Custom FrameProcessor Ending a Pipeline On this page Overview How It Works Basic Implementation Step 1: Create a Handler Function Step 2: Create the Idle Processor Step 3: Add to Your Pipeline Best Practices Next Steps Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/fundamentals_detecting-user-idle_346c1f56.txt b/fundamentals_detecting-user-idle_346c1f56.txt
new file mode 100644
index 0000000000000000000000000000000000000000..b9cf3e8c732af1f9ca2c87d5d169752af5745ff8
--- /dev/null
+++ b/fundamentals_detecting-user-idle_346c1f56.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/guides/fundamentals/detecting-user-idle#basic-implementation
+Title: Detecting Idle Users - Pipecat
+==================================================
+
+Detecting Idle Users - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Fundamentals Detecting Idle Users Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Guides Fundamentals Context Management Custom FrameProcessor Detecting Idle Users Ending a Pipeline Function Calling Muting User Input Recording Audio Recording Transcripts Features Gemini Multimodal Live Metrics Noise cancellation with Krisp OpenAI Audio Models and APIs Pipecat Flows Telephony Overview Dial-in: WebRTC (Daily) Dial-in: WebRTC (Twilio + Daily) Dial-in: Twilio (Media Streams) Dialout: WebRTC (Daily) Deploying your bot Overview Deployment pattern Example: Pipecat Cloud Example: Fly.io Example: Cerebrium Example: Modal ​ Overview In conversational applications, it’s important to handle situations where users go silent or inactive. Pipecat’s UserIdleProcessor helps you detect when users haven’t spoken for a defined period, allowing your bot to respond appropriately. ​ How It Works The UserIdleProcessor monitors user activity and: Starts tracking after the first interaction (user or bot speaking) Resets a timer whenever the user speaks Calls your custom callback function when the user is idle for longer than the timeout period Provides a retry counter to track consecutive idle periods Allows you to implement escalating responses or gracefully end the conversation The processor uses speech events (not audio frames) to detect activity, so it requires an active speech-to-text service or a transport with built-in speech detection. ​ Basic Implementation ​ Step 1: Create a Handler Function First, create a callback function that will be triggered when the user is idle: Copy Ask AI # Simple handler that doesn't use retry count async def handle_user_idle ( processor ): # Send a reminder to the user await processor.push_frame(TTSSpeakFrame( "Are you still there?" )) # OR # Advanced handler with retry logic async def handle_user_idle ( processor , retry_count ): if retry_count == 1 : # First attempt - gentle reminder await processor.push_frame(TTSSpeakFrame( "Are you still there?" )) return True # Continue monitoring elif retry_count == 2 : # Second attempt - more direct prompt await processor.push_frame(TTSSpeakFrame( "Would you like to continue our conversation?" )) return True # Continue monitoring else : # Third attempt - end conversation await processor.push_frame(TTSSpeakFrame( "I'll leave you for now. Have a nice day!" )) await processor.push_frame(EndFrame(), FrameDirection. UPSTREAM ) return False # Stop monitoring ​ Step 2: Create the Idle Processor Initialize the processor with your callback and desired timeout: Copy Ask AI from pipecat.processors.user_idle_processor import UserIdleProcessor user_idle = UserIdleProcessor( callback = handle_user_idle, # Your callback function timeout = 5.0 , # Seconds of inactivity before triggering ) ​ Step 3: Add to Your Pipeline Place the processor after speech detection but before context handling: Copy Ask AI pipeline = Pipeline( [ transport.input(), stt, # Speech-to-text user_idle, # Add idle detection here context_aggregator.user(), llm, tts, transport.output(), context_aggregator.assistant(), ] ) ​ Best Practices Set appropriate timeouts : Shorter timeouts (5-10 seconds) work well for voice conversations Use escalating responses : Start with gentle reminders and gradually become more direct Limit retry attempts : After 2-3 unsuccessful attempts, consider ending the conversation gracefully by pushing an EndFrame ​ Next Steps Try the User Idle Example Explore a complete working example that demonstrates how to detect and respond to user inactivity in Pipecat. UserIdleProcessor Reference Read the complete API reference documentation for advanced configuration options and callback patterns. Implementing idle user detection improves the conversational experience by ensuring your bot can handle periods of user inactivity gracefully, either by prompting for re-engagement or politely ending the conversation when appropriate. Custom FrameProcessor Ending a Pipeline On this page Overview How It Works Basic Implementation Step 1: Create a Handler Function Step 2: Create the Idle Processor Step 3: Add to Your Pipeline Best Practices Next Steps Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/fundamentals_detecting-user-idle_3c954034.txt b/fundamentals_detecting-user-idle_3c954034.txt
new file mode 100644
index 0000000000000000000000000000000000000000..bcad72a501f7a43d2939325489b87d146c285be9
--- /dev/null
+++ b/fundamentals_detecting-user-idle_3c954034.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/guides/fundamentals/detecting-user-idle#step-3%3A-add-to-your-pipeline
+Title: Detecting Idle Users - Pipecat
+==================================================
+
+Detecting Idle Users - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Fundamentals Detecting Idle Users Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Guides Fundamentals Context Management Custom FrameProcessor Detecting Idle Users Ending a Pipeline Function Calling Muting User Input Recording Audio Recording Transcripts Features Gemini Multimodal Live Metrics Noise cancellation with Krisp OpenAI Audio Models and APIs Pipecat Flows Telephony Overview Dial-in: WebRTC (Daily) Dial-in: WebRTC (Twilio + Daily) Dial-in: Twilio (Media Streams) Dialout: WebRTC (Daily) Deploying your bot Overview Deployment pattern Example: Pipecat Cloud Example: Fly.io Example: Cerebrium Example: Modal ​ Overview In conversational applications, it’s important to handle situations where users go silent or inactive. Pipecat’s UserIdleProcessor helps you detect when users haven’t spoken for a defined period, allowing your bot to respond appropriately. ​ How It Works The UserIdleProcessor monitors user activity and: Starts tracking after the first interaction (user or bot speaking) Resets a timer whenever the user speaks Calls your custom callback function when the user is idle for longer than the timeout period Provides a retry counter to track consecutive idle periods Allows you to implement escalating responses or gracefully end the conversation The processor uses speech events (not audio frames) to detect activity, so it requires an active speech-to-text service or a transport with built-in speech detection. ​ Basic Implementation ​ Step 1: Create a Handler Function First, create a callback function that will be triggered when the user is idle: Copy Ask AI # Simple handler that doesn't use retry count async def handle_user_idle ( processor ): # Send a reminder to the user await processor.push_frame(TTSSpeakFrame( "Are you still there?" )) # OR # Advanced handler with retry logic async def handle_user_idle ( processor , retry_count ): if retry_count == 1 : # First attempt - gentle reminder await processor.push_frame(TTSSpeakFrame( "Are you still there?" )) return True # Continue monitoring elif retry_count == 2 : # Second attempt - more direct prompt await processor.push_frame(TTSSpeakFrame( "Would you like to continue our conversation?" )) return True # Continue monitoring else : # Third attempt - end conversation await processor.push_frame(TTSSpeakFrame( "I'll leave you for now. Have a nice day!" )) await processor.push_frame(EndFrame(), FrameDirection. UPSTREAM ) return False # Stop monitoring ​ Step 2: Create the Idle Processor Initialize the processor with your callback and desired timeout: Copy Ask AI from pipecat.processors.user_idle_processor import UserIdleProcessor user_idle = UserIdleProcessor( callback = handle_user_idle, # Your callback function timeout = 5.0 , # Seconds of inactivity before triggering ) ​ Step 3: Add to Your Pipeline Place the processor after speech detection but before context handling: Copy Ask AI pipeline = Pipeline( [ transport.input(), stt, # Speech-to-text user_idle, # Add idle detection here context_aggregator.user(), llm, tts, transport.output(), context_aggregator.assistant(), ] ) ​ Best Practices Set appropriate timeouts : Shorter timeouts (5-10 seconds) work well for voice conversations Use escalating responses : Start with gentle reminders and gradually become more direct Limit retry attempts : After 2-3 unsuccessful attempts, consider ending the conversation gracefully by pushing an EndFrame ​ Next Steps Try the User Idle Example Explore a complete working example that demonstrates how to detect and respond to user inactivity in Pipecat. UserIdleProcessor Reference Read the complete API reference documentation for advanced configuration options and callback patterns. Implementing idle user detection improves the conversational experience by ensuring your bot can handle periods of user inactivity gracefully, either by prompting for re-engagement or politely ending the conversation when appropriate. Custom FrameProcessor Ending a Pipeline On this page Overview How It Works Basic Implementation Step 1: Create a Handler Function Step 2: Create the Idle Processor Step 3: Add to Your Pipeline Best Practices Next Steps Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/fundamentals_detecting-user-idle_97729173.txt b/fundamentals_detecting-user-idle_97729173.txt
new file mode 100644
index 0000000000000000000000000000000000000000..71e0f275435367bb8f654deabf8ae13804fe95d3
--- /dev/null
+++ b/fundamentals_detecting-user-idle_97729173.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/guides/fundamentals/detecting-user-idle#overview
+Title: Detecting Idle Users - Pipecat
+==================================================
+
+Detecting Idle Users - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Fundamentals Detecting Idle Users Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Guides Fundamentals Context Management Custom FrameProcessor Detecting Idle Users Ending a Pipeline Function Calling Muting User Input Recording Audio Recording Transcripts Features Gemini Multimodal Live Metrics Noise cancellation with Krisp OpenAI Audio Models and APIs Pipecat Flows Telephony Overview Dial-in: WebRTC (Daily) Dial-in: WebRTC (Twilio + Daily) Dial-in: Twilio (Media Streams) Dialout: WebRTC (Daily) Deploying your bot Overview Deployment pattern Example: Pipecat Cloud Example: Fly.io Example: Cerebrium Example: Modal ​ Overview In conversational applications, it’s important to handle situations where users go silent or inactive. Pipecat’s UserIdleProcessor helps you detect when users haven’t spoken for a defined period, allowing your bot to respond appropriately. ​ How It Works The UserIdleProcessor monitors user activity and: Starts tracking after the first interaction (user or bot speaking) Resets a timer whenever the user speaks Calls your custom callback function when the user is idle for longer than the timeout period Provides a retry counter to track consecutive idle periods Allows you to implement escalating responses or gracefully end the conversation The processor uses speech events (not audio frames) to detect activity, so it requires an active speech-to-text service or a transport with built-in speech detection. ​ Basic Implementation ​ Step 1: Create a Handler Function First, create a callback function that will be triggered when the user is idle: Copy Ask AI # Simple handler that doesn't use retry count async def handle_user_idle ( processor ): # Send a reminder to the user await processor.push_frame(TTSSpeakFrame( "Are you still there?" )) # OR # Advanced handler with retry logic async def handle_user_idle ( processor , retry_count ): if retry_count == 1 : # First attempt - gentle reminder await processor.push_frame(TTSSpeakFrame( "Are you still there?" )) return True # Continue monitoring elif retry_count == 2 : # Second attempt - more direct prompt await processor.push_frame(TTSSpeakFrame( "Would you like to continue our conversation?" )) return True # Continue monitoring else : # Third attempt - end conversation await processor.push_frame(TTSSpeakFrame( "I'll leave you for now. Have a nice day!" )) await processor.push_frame(EndFrame(), FrameDirection. UPSTREAM ) return False # Stop monitoring ​ Step 2: Create the Idle Processor Initialize the processor with your callback and desired timeout: Copy Ask AI from pipecat.processors.user_idle_processor import UserIdleProcessor user_idle = UserIdleProcessor( callback = handle_user_idle, # Your callback function timeout = 5.0 , # Seconds of inactivity before triggering ) ​ Step 3: Add to Your Pipeline Place the processor after speech detection but before context handling: Copy Ask AI pipeline = Pipeline( [ transport.input(), stt, # Speech-to-text user_idle, # Add idle detection here context_aggregator.user(), llm, tts, transport.output(), context_aggregator.assistant(), ] ) ​ Best Practices Set appropriate timeouts : Shorter timeouts (5-10 seconds) work well for voice conversations Use escalating responses : Start with gentle reminders and gradually become more direct Limit retry attempts : After 2-3 unsuccessful attempts, consider ending the conversation gracefully by pushing an EndFrame ​ Next Steps Try the User Idle Example Explore a complete working example that demonstrates how to detect and respond to user inactivity in Pipecat. UserIdleProcessor Reference Read the complete API reference documentation for advanced configuration options and callback patterns. Implementing idle user detection improves the conversational experience by ensuring your bot can handle periods of user inactivity gracefully, either by prompting for re-engagement or politely ending the conversation when appropriate. Custom FrameProcessor Ending a Pipeline On this page Overview How It Works Basic Implementation Step 1: Create a Handler Function Step 2: Create the Idle Processor Step 3: Add to Your Pipeline Best Practices Next Steps Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/fundamentals_end-pipeline_fc401a2f.txt b/fundamentals_end-pipeline_fc401a2f.txt
new file mode 100644
index 0000000000000000000000000000000000000000..af7b3f14a6e47ab0e1a515ebb1766fc725fe2c69
--- /dev/null
+++ b/fundamentals_end-pipeline_fc401a2f.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/guides/fundamentals/end-pipeline#best-practices
+Title: Ending a Pipeline - Pipecat
+==================================================
+
+Ending a Pipeline - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Fundamentals Ending a Pipeline Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Guides Fundamentals Context Management Custom FrameProcessor Detecting Idle Users Ending a Pipeline Function Calling Muting User Input Recording Audio Recording Transcripts Features Gemini Multimodal Live Metrics Noise cancellation with Krisp OpenAI Audio Models and APIs Pipecat Flows Telephony Overview Dial-in: WebRTC (Daily) Dial-in: WebRTC (Twilio + Daily) Dial-in: Twilio (Media Streams) Dialout: WebRTC (Daily) Deploying your bot Overview Deployment pattern Example: Pipecat Cloud Example: Fly.io Example: Cerebrium Example: Modal ​ Overview Properly ending a Pipecat pipeline is essential to prevent hanging processes and ensure clean shutdown of your session and related infrastructure. This guide covers different approaches to pipeline termination and provides best practices for each scenario. ​ Shutdown Approaches Pipecat provides two primary methods for shutting down a pipeline: Graceful Shutdown : Allows completion of in-progress processing before termination Immediate Shutdown : Cancels all tasks immediately Each approach is designed for different use cases, as detailed below. ​ Graceful Shutdown A graceful shutdown is ideal when you want the bot to properly end a conversation. For example, you might want to terminate a session after the bot has completed a specific task or reached a natural conclusion. This approach ensures that any final messages from the bot are processed and delivered before the pipeline terminates. ​ Implementation To implement a graceful shutdown, there are two options: Push an EndFrame from outside your pipeline using the pipeline task: Copy Ask AI await task.queue_frame(EndFrame()) Push an EndTaskFrame upstream from inside your pipeline. For example, inside a function call: Copy Ask AI async def end_conversation ( params : FunctionCallParams): await params.llm.push_frame(TTSSpeakFrame( "Have a nice day!" )) # Signal that the task should end after processing this frame await params.llm.push_frame(EndTaskFrame(), FrameDirection. UPSTREAM ) ​ How Graceful Shutdown Works In both cases, an EndFrame is pushed downstream from the beginning of the pipeline: EndFrame s are queued, so they’ll process after any pending frames (like goodbye messages) All processors in the pipeline will shutdown when processing the EndFrame Once the EndFrame reaches the sink of the PipelineTask , the Pipeline is ready to shut down The Pipecat processor terminates and related resources are released Graceful shutdowns allow your bot to say goodbye and complete any final actions before terminating. ​ Immediate Shutdown An immediate shutdown is appropriate when the human participant is no longer active in the conversation. For example: In a client/server app, when the user closes the browser tab or ends the session In a phone call, when the user hangs up When an error occurs that requires immediate termination In these scenarios, there’s no value in having the bot complete its current turn. ​ Implementation To implement an immediate shutdown, you can use event handlers to, for example, detect disconnections and then push a CancelFrame : Copy Ask AI @transport.event_handler ( "on_client_closed" ) async def on_client_closed ( transport , client ): logger.info( f "Client closed connection" ) await task.cancel() ​ How Immediate Shutdown Works An event triggers the cancellation (like a client disconnection) task.cancel() is called, which pushes a CancelFrame downstream from the PipelineTask CancelFrame s are SystemFrame s and are not queued Processors that handle the CancelFrame immediate shutdown and push the frame downstream Once the CancelFrame reaches the sink of the PipelineTask , the Pipeline is ready to shut down Immediate shutdowns will discard any pending frames in the pipeline. Use this approach when completing the conversation is no longer necessary. ​ Pipeline Idle Detection In addition to the two explicit shutdown mechanisms, Pipecat includes a backup mechanism to prevent hanging pipelines—Pipeline Idle Detection. This feature monitors activity in your pipeline and can automatically cancel tasks when no meaningful bot interactions are occurring for an extended period. It serves as a safety net to conditionally terminate the pipeline if anomalous behavior occurs. Pipeline Idle Detection is enabled by default and helps prevent resources from being wasted on inactive conversations. For more information on configuring and customizing this feature, see the Pipeline Idle Detection documentation. ​ Best Practices Use graceful shutdowns when you want to let the bot complete its conversation Use immediate shutdowns when the human participant has already disconnected Implement error handling to ensure pipelines can terminate even when exceptions occur Configure idle detection timeouts appropriate for your use case By following these practices, you’ll ensure that your Pipecat pipelines terminate properly and efficiently, preventing resource leaks and improving overall system reliability. Detecting Idle Users Function Calling On this page Overview Shutdown Approaches Graceful Shutdown Implementation How Graceful Shutdown Works Immediate Shutdown Implementation How Immediate Shutdown Works Pipeline Idle Detection Best Practices Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/fundamentals_function-calling_101b28b3.txt b/fundamentals_function-calling_101b28b3.txt
new file mode 100644
index 0000000000000000000000000000000000000000..7f0028fccbc11f603fae75a2d64d835ce9f8ab8b
--- /dev/null
+++ b/fundamentals_function-calling_101b28b3.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/guides/fundamentals/function-calling#properties
+Title: Function Calling - Pipecat
+==================================================
+
+Function Calling - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Fundamentals Function Calling Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Guides Fundamentals Context Management Custom FrameProcessor Detecting Idle Users Ending a Pipeline Function Calling Muting User Input Recording Audio Recording Transcripts Features Gemini Multimodal Live Metrics Noise cancellation with Krisp OpenAI Audio Models and APIs Pipecat Flows Telephony Overview Dial-in: WebRTC (Daily) Dial-in: WebRTC (Twilio + Daily) Dial-in: Twilio (Media Streams) Dialout: WebRTC (Daily) Deploying your bot Overview Deployment pattern Example: Pipecat Cloud Example: Fly.io Example: Cerebrium Example: Modal ​ Understanding Function Calling Function calling (also known as tool calling) allows LLMs to request information from external services and APIs. This enables your bot to access real-time data and perform actions that aren’t part of its training data. For example, you could give your bot the ability to: Check current weather conditions Look up stock prices Query a database Control smart home devices Schedule appointments Here’s how it works: You define functions the LLM can use and register them to the LLM service used in your pipeline When needed, the LLM requests a function call Your application executes any corresponding functions The result is sent back to the LLM The LLM uses this information in its response ​ Implementation ​ 1. Define Functions Pipecat provides a standardized FunctionSchema that works across all supported LLM providers. This makes it easy to define functions once and use them with any provider. As a shorthand, you could also bypass specifying a function configuration at all and instead use “direct” functions. Under the hood, these are converted to FunctionSchema s. ​ Using the Standard Schema (Recommended) Copy Ask AI from pipecat.adapters.schemas.function_schema import FunctionSchema from pipecat.adapters.schemas.tools_schema import ToolsSchema # Define a function using the standard schema weather_function = FunctionSchema( name = "get_current_weather" , description = "Get the current weather in a location" , properties = { "location" : { "type" : "string" , "description" : "The city and state, e.g. San Francisco, CA" , }, "format" : { "type" : "string" , "enum" : [ "celsius" , "fahrenheit" ], "description" : "The temperature unit to use." , }, }, required = [ "location" , "format" ] ) # Create a tools schema with your functions tools = ToolsSchema( standard_tools = [weather_function]) # Pass this to your LLM context context = OpenAILLMContext( messages = [{ "role" : "system" , "content" : "You are a helpful assistant." }], tools = tools ) The ToolsSchema will be automatically converted to the correct format for your LLM provider through adapters. ​ Using Direct Functions (Shorthand) You can bypass specifying a function configuration (as a FunctionSchema or in a provider-specific format) and instead pass the function directly to your ToolsSchema . Pipecat will auto-configure the function, gathering relevant metadata from its signature and docstring. Metadata includes: name description properties (including individual property descriptions) list of required properties Note that the function signature is a bit different when using direct functions. The first parameter is FunctionCallParams , followed by any others necessary for the function. Copy Ask AI from pipecat.adapters.schemas.tools_schema import ToolsSchema from pipecat.services.llm_service import FunctionCallParams # Define a direct function async def get_current_weather ( params : FunctionCallParams, location : str , format : str ): """Get the current weather. Args: location: The city and state, e.g. "San Francisco, CA". format: The temperature unit to use. Must be either "celsius" or "fahrenheit". """ weather_data = { "conditions" : "sunny" , "temperature" : "75" } await params.result_callback(weather_data) # Create a tools schema, passing your function directly to it tools = ToolsSchema( standard_tools = [get_current_weather]) # Pass this to your LLM context context = OpenAILLMContext( messages = [{ "role" : "system" , "content" : "You are a helpful assistant." }], tools = tools ) ​ Using Provider-Specific Formats (Alternative) You can also define functions in the provider-specific format if needed: OpenAI Anthropic Gemini Copy Ask AI from openai.types.chat import ChatCompletionToolParam # OpenAI native format tools = [ ChatCompletionToolParam( type = "function" , function = { "name" : "get_current_weather" , "description" : "Get the current weather" , "parameters" : { "type" : "object" , "properties" : { "location" : { "type" : "string" , "description" : "The city and state, e.g. San Francisco, CA" , }, "format" : { "type" : "string" , "enum" : [ "celsius" , "fahrenheit" ], "description" : "The temperature unit to use." , }, }, "required" : [ "location" , "format" ], }, }, ) ] ​ Provider-Specific Custom Tools Some providers support unique tools that don’t fit the standard function schema. For these cases, you can add custom tools: Copy Ask AI from pipecat.adapters.schemas.tools_schema import AdapterType, ToolsSchema # Standard functions weather_function = FunctionSchema( name = "get_current_weather" , description = "Get the current weather" , properties = { "location" : { "type" : "string" }}, required = [ "location" ] ) # Custom Gemini search tool gemini_search_tool = { "web_search" : { "description" : "Search the web for information" } } # Create a tools schema with both standard and custom tools tools = ToolsSchema( standard_tools = [weather_function], custom_tools = { AdapterType. GEMINI : [gemini_search_tool] } ) See the provider-specific documentation for details on custom tools and their formats. ​ 2. Register Function Handlers Register handlers for your functions using one of these LLM service methods : register_function register_direct_function Which one you use depends on whether your function is a “direct” function . Non-Direct Function Direct Function Copy Ask AI from pipecat.services.llm_service import FunctionCallParams llm = OpenAILLMService( api_key = "your-api-key" ) # Main function handler - called to execute the function async def fetch_weather_from_api ( params : FunctionCallParams): # Fetch weather data from your API weather_data = { "conditions" : "sunny" , "temperature" : "75" } await params.result_callback(weather_data) # Register the function llm.register_function( "get_current_weather" , fetch_weather_from_api, ) ​ 3. Create the Pipeline Include your LLM service in your pipeline with the registered functions: Copy Ask AI # Initialize the LLM context with your function schemas context = OpenAILLMContext( messages = [{ "role" : "system" , "content" : "You are a helpful assistant." }], tools = tools ) # Create the context aggregator to collect the user and assistant context context_aggregator = llm.create_context_aggregator(context) # Create the pipeline pipeline = Pipeline([ transport.input(), # Input from the transport stt, # STT processing context_aggregator.user(), # User context aggregation llm, # LLM processing tts, # TTS processing transport.output(), # Output to the transport context_aggregator.assistant(), # Assistant context aggregation ]) ​ Function Handler Details ​ FunctionCallParams The FunctionCallParams object contains all the information needed for handling function calls: params : FunctionCallParams function_name : Name of the called function arguments : Arguments passed by the LLM tool_call_id : Unique identifier for the function call llm : Reference to the LLM service context : Current conversation context result_callback : Async function to return results ​ function_name str Name of the function being called ​ tool_call_id str Unique identifier for the function call ​ arguments Mapping[str, Any] Arguments passed by the LLM to the function ​ llm LLMService Reference to the LLM service that initiated the call ​ context OpenAILLMContext Current conversation context ​ result_callback FunctionCallResultCallback Async callback function to return results ​ Handler Structure Your function handler should: Receive necessary arguments, either: From params.arguments Directly From function arguments, if using direct functions Process data or call external services Return results via params.result_callback(result) Non-Direct Function Direct Function Copy Ask AI async def fetch_weather_from_api ( params : FunctionCallParams): try : # Extract arguments location = params.arguments.get( "location" ) format_type = params.arguments.get( "format" , "celsius" ) # Call external API api_result = await weather_api.get_weather(location, format_type) # Return formatted result await params.result_callback({ "location" : location, "temperature" : api_result[ "temp" ], "conditions" : api_result[ "conditions" ], "unit" : format_type }) except Exception as e: # Handle errors await params.result_callback({ "error" : f "Failed to get weather: { str (e) } " }) ​ Controlling Function Call Behavior (Advanced) When returning results from a function handler, you can control how the LLM processes those results using a FunctionCallResultProperties object passed to the result callback. It can be handy to skip a completion when you have back-to-back function calls. Note, if you skip a completion, you must manually trigger one from the context. ​ Properties ​ run_llm Optional[bool] Controls whether the LLM should generate a response after the function call: True : Run LLM after function call (default if no other function calls in progress) False : Don’t run LLM after function call None : Use default behavior ​ on_context_updated Optional[Callable[[], Awaitable[None]]] Optional callback that runs after the function result is added to the context ​ Example Usage Copy Ask AI from pipecat.frames.frames import FunctionCallResultProperties from pipecat.services.llm_service import FunctionCallParams async def fetch_weather_from_api ( params : FunctionCallParams): # Fetch weather data weather_data = { "conditions" : "sunny" , "temperature" : "75" } # Don't run LLM after this function call properties = FunctionCallResultProperties( run_llm = False ) await params.result_callback(weather_data, properties = properties) async def query_database ( params : FunctionCallParams): # Query database results = await db.query(params.arguments[ "query" ]) async def on_update (): await notify_system( "Database query complete" ) # Run LLM after function call and notify when context is updated properties = FunctionCallResultProperties( run_llm = True , on_context_updated = on_update ) await params.result_callback(results, properties = properties) ​ Next steps Check out the function calling examples to see a complete example for specific LLM providers. Refer to your LLM provider’s documentation to learn more about their function calling capabilities. Ending a Pipeline Muting User Input On this page Understanding Function Calling Implementation 1. Define Functions Using the Standard Schema (Recommended) Using Direct Functions (Shorthand) Using Provider-Specific Formats (Alternative) Provider-Specific Custom Tools 2. Register Function Handlers 3. Create the Pipeline Function Handler Details FunctionCallParams Handler Structure Controlling Function Call Behavior (Advanced) Properties Example Usage Next steps Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/fundamentals_function-calling_13f3c4fc.txt b/fundamentals_function-calling_13f3c4fc.txt
new file mode 100644
index 0000000000000000000000000000000000000000..d9c77e5e2b829cd56a1d76d34abe7ffdaf8b9c20
--- /dev/null
+++ b/fundamentals_function-calling_13f3c4fc.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/guides/fundamentals/function-calling
+Title: Function Calling - Pipecat
+==================================================
+
+Function Calling - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Fundamentals Function Calling Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Guides Fundamentals Context Management Custom FrameProcessor Detecting Idle Users Ending a Pipeline Function Calling Muting User Input Recording Audio Recording Transcripts Features Gemini Multimodal Live Metrics Noise cancellation with Krisp OpenAI Audio Models and APIs Pipecat Flows Telephony Overview Dial-in: WebRTC (Daily) Dial-in: WebRTC (Twilio + Daily) Dial-in: Twilio (Media Streams) Dialout: WebRTC (Daily) Deploying your bot Overview Deployment pattern Example: Pipecat Cloud Example: Fly.io Example: Cerebrium Example: Modal ​ Understanding Function Calling Function calling (also known as tool calling) allows LLMs to request information from external services and APIs. This enables your bot to access real-time data and perform actions that aren’t part of its training data. For example, you could give your bot the ability to: Check current weather conditions Look up stock prices Query a database Control smart home devices Schedule appointments Here’s how it works: You define functions the LLM can use and register them to the LLM service used in your pipeline When needed, the LLM requests a function call Your application executes any corresponding functions The result is sent back to the LLM The LLM uses this information in its response ​ Implementation ​ 1. Define Functions Pipecat provides a standardized FunctionSchema that works across all supported LLM providers. This makes it easy to define functions once and use them with any provider. As a shorthand, you could also bypass specifying a function configuration at all and instead use “direct” functions. Under the hood, these are converted to FunctionSchema s. ​ Using the Standard Schema (Recommended) Copy Ask AI from pipecat.adapters.schemas.function_schema import FunctionSchema from pipecat.adapters.schemas.tools_schema import ToolsSchema # Define a function using the standard schema weather_function = FunctionSchema( name = "get_current_weather" , description = "Get the current weather in a location" , properties = { "location" : { "type" : "string" , "description" : "The city and state, e.g. San Francisco, CA" , }, "format" : { "type" : "string" , "enum" : [ "celsius" , "fahrenheit" ], "description" : "The temperature unit to use." , }, }, required = [ "location" , "format" ] ) # Create a tools schema with your functions tools = ToolsSchema( standard_tools = [weather_function]) # Pass this to your LLM context context = OpenAILLMContext( messages = [{ "role" : "system" , "content" : "You are a helpful assistant." }], tools = tools ) The ToolsSchema will be automatically converted to the correct format for your LLM provider through adapters. ​ Using Direct Functions (Shorthand) You can bypass specifying a function configuration (as a FunctionSchema or in a provider-specific format) and instead pass the function directly to your ToolsSchema . Pipecat will auto-configure the function, gathering relevant metadata from its signature and docstring. Metadata includes: name description properties (including individual property descriptions) list of required properties Note that the function signature is a bit different when using direct functions. The first parameter is FunctionCallParams , followed by any others necessary for the function. Copy Ask AI from pipecat.adapters.schemas.tools_schema import ToolsSchema from pipecat.services.llm_service import FunctionCallParams # Define a direct function async def get_current_weather ( params : FunctionCallParams, location : str , format : str ): """Get the current weather. Args: location: The city and state, e.g. "San Francisco, CA". format: The temperature unit to use. Must be either "celsius" or "fahrenheit". """ weather_data = { "conditions" : "sunny" , "temperature" : "75" } await params.result_callback(weather_data) # Create a tools schema, passing your function directly to it tools = ToolsSchema( standard_tools = [get_current_weather]) # Pass this to your LLM context context = OpenAILLMContext( messages = [{ "role" : "system" , "content" : "You are a helpful assistant." }], tools = tools ) ​ Using Provider-Specific Formats (Alternative) You can also define functions in the provider-specific format if needed: OpenAI Anthropic Gemini Copy Ask AI from openai.types.chat import ChatCompletionToolParam # OpenAI native format tools = [ ChatCompletionToolParam( type = "function" , function = { "name" : "get_current_weather" , "description" : "Get the current weather" , "parameters" : { "type" : "object" , "properties" : { "location" : { "type" : "string" , "description" : "The city and state, e.g. San Francisco, CA" , }, "format" : { "type" : "string" , "enum" : [ "celsius" , "fahrenheit" ], "description" : "The temperature unit to use." , }, }, "required" : [ "location" , "format" ], }, }, ) ] ​ Provider-Specific Custom Tools Some providers support unique tools that don’t fit the standard function schema. For these cases, you can add custom tools: Copy Ask AI from pipecat.adapters.schemas.tools_schema import AdapterType, ToolsSchema # Standard functions weather_function = FunctionSchema( name = "get_current_weather" , description = "Get the current weather" , properties = { "location" : { "type" : "string" }}, required = [ "location" ] ) # Custom Gemini search tool gemini_search_tool = { "web_search" : { "description" : "Search the web for information" } } # Create a tools schema with both standard and custom tools tools = ToolsSchema( standard_tools = [weather_function], custom_tools = { AdapterType. GEMINI : [gemini_search_tool] } ) See the provider-specific documentation for details on custom tools and their formats. ​ 2. Register Function Handlers Register handlers for your functions using one of these LLM service methods : register_function register_direct_function Which one you use depends on whether your function is a “direct” function . Non-Direct Function Direct Function Copy Ask AI from pipecat.services.llm_service import FunctionCallParams llm = OpenAILLMService( api_key = "your-api-key" ) # Main function handler - called to execute the function async def fetch_weather_from_api ( params : FunctionCallParams): # Fetch weather data from your API weather_data = { "conditions" : "sunny" , "temperature" : "75" } await params.result_callback(weather_data) # Register the function llm.register_function( "get_current_weather" , fetch_weather_from_api, ) ​ 3. Create the Pipeline Include your LLM service in your pipeline with the registered functions: Copy Ask AI # Initialize the LLM context with your function schemas context = OpenAILLMContext( messages = [{ "role" : "system" , "content" : "You are a helpful assistant." }], tools = tools ) # Create the context aggregator to collect the user and assistant context context_aggregator = llm.create_context_aggregator(context) # Create the pipeline pipeline = Pipeline([ transport.input(), # Input from the transport stt, # STT processing context_aggregator.user(), # User context aggregation llm, # LLM processing tts, # TTS processing transport.output(), # Output to the transport context_aggregator.assistant(), # Assistant context aggregation ]) ​ Function Handler Details ​ FunctionCallParams The FunctionCallParams object contains all the information needed for handling function calls: params : FunctionCallParams function_name : Name of the called function arguments : Arguments passed by the LLM tool_call_id : Unique identifier for the function call llm : Reference to the LLM service context : Current conversation context result_callback : Async function to return results ​ function_name str Name of the function being called ​ tool_call_id str Unique identifier for the function call ​ arguments Mapping[str, Any] Arguments passed by the LLM to the function ​ llm LLMService Reference to the LLM service that initiated the call ​ context OpenAILLMContext Current conversation context ​ result_callback FunctionCallResultCallback Async callback function to return results ​ Handler Structure Your function handler should: Receive necessary arguments, either: From params.arguments Directly From function arguments, if using direct functions Process data or call external services Return results via params.result_callback(result) Non-Direct Function Direct Function Copy Ask AI async def fetch_weather_from_api ( params : FunctionCallParams): try : # Extract arguments location = params.arguments.get( "location" ) format_type = params.arguments.get( "format" , "celsius" ) # Call external API api_result = await weather_api.get_weather(location, format_type) # Return formatted result await params.result_callback({ "location" : location, "temperature" : api_result[ "temp" ], "conditions" : api_result[ "conditions" ], "unit" : format_type }) except Exception as e: # Handle errors await params.result_callback({ "error" : f "Failed to get weather: { str (e) } " }) ​ Controlling Function Call Behavior (Advanced) When returning results from a function handler, you can control how the LLM processes those results using a FunctionCallResultProperties object passed to the result callback. It can be handy to skip a completion when you have back-to-back function calls. Note, if you skip a completion, you must manually trigger one from the context. ​ Properties ​ run_llm Optional[bool] Controls whether the LLM should generate a response after the function call: True : Run LLM after function call (default if no other function calls in progress) False : Don’t run LLM after function call None : Use default behavior ​ on_context_updated Optional[Callable[[], Awaitable[None]]] Optional callback that runs after the function result is added to the context ​ Example Usage Copy Ask AI from pipecat.frames.frames import FunctionCallResultProperties from pipecat.services.llm_service import FunctionCallParams async def fetch_weather_from_api ( params : FunctionCallParams): # Fetch weather data weather_data = { "conditions" : "sunny" , "temperature" : "75" } # Don't run LLM after this function call properties = FunctionCallResultProperties( run_llm = False ) await params.result_callback(weather_data, properties = properties) async def query_database ( params : FunctionCallParams): # Query database results = await db.query(params.arguments[ "query" ]) async def on_update (): await notify_system( "Database query complete" ) # Run LLM after function call and notify when context is updated properties = FunctionCallResultProperties( run_llm = True , on_context_updated = on_update ) await params.result_callback(results, properties = properties) ​ Next steps Check out the function calling examples to see a complete example for specific LLM providers. Refer to your LLM provider’s documentation to learn more about their function calling capabilities. Ending a Pipeline Muting User Input On this page Understanding Function Calling Implementation 1. Define Functions Using the Standard Schema (Recommended) Using Direct Functions (Shorthand) Using Provider-Specific Formats (Alternative) Provider-Specific Custom Tools 2. Register Function Handlers 3. Create the Pipeline Function Handler Details FunctionCallParams Handler Structure Controlling Function Call Behavior (Advanced) Properties Example Usage Next steps Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/fundamentals_function-calling_2c29898d.txt b/fundamentals_function-calling_2c29898d.txt
new file mode 100644
index 0000000000000000000000000000000000000000..475fa8e487f92ae8e23adf5065d77635d5f05c55
--- /dev/null
+++ b/fundamentals_function-calling_2c29898d.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/guides/fundamentals/function-calling#using-the-standard-schema-recommended
+Title: Function Calling - Pipecat
+==================================================
+
+Function Calling - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Fundamentals Function Calling Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Guides Fundamentals Context Management Custom FrameProcessor Detecting Idle Users Ending a Pipeline Function Calling Muting User Input Recording Audio Recording Transcripts Features Gemini Multimodal Live Metrics Noise cancellation with Krisp OpenAI Audio Models and APIs Pipecat Flows Telephony Overview Dial-in: WebRTC (Daily) Dial-in: WebRTC (Twilio + Daily) Dial-in: Twilio (Media Streams) Dialout: WebRTC (Daily) Deploying your bot Overview Deployment pattern Example: Pipecat Cloud Example: Fly.io Example: Cerebrium Example: Modal ​ Understanding Function Calling Function calling (also known as tool calling) allows LLMs to request information from external services and APIs. This enables your bot to access real-time data and perform actions that aren’t part of its training data. For example, you could give your bot the ability to: Check current weather conditions Look up stock prices Query a database Control smart home devices Schedule appointments Here’s how it works: You define functions the LLM can use and register them to the LLM service used in your pipeline When needed, the LLM requests a function call Your application executes any corresponding functions The result is sent back to the LLM The LLM uses this information in its response ​ Implementation ​ 1. Define Functions Pipecat provides a standardized FunctionSchema that works across all supported LLM providers. This makes it easy to define functions once and use them with any provider. As a shorthand, you could also bypass specifying a function configuration at all and instead use “direct” functions. Under the hood, these are converted to FunctionSchema s. ​ Using the Standard Schema (Recommended) Copy Ask AI from pipecat.adapters.schemas.function_schema import FunctionSchema from pipecat.adapters.schemas.tools_schema import ToolsSchema # Define a function using the standard schema weather_function = FunctionSchema( name = "get_current_weather" , description = "Get the current weather in a location" , properties = { "location" : { "type" : "string" , "description" : "The city and state, e.g. San Francisco, CA" , }, "format" : { "type" : "string" , "enum" : [ "celsius" , "fahrenheit" ], "description" : "The temperature unit to use." , }, }, required = [ "location" , "format" ] ) # Create a tools schema with your functions tools = ToolsSchema( standard_tools = [weather_function]) # Pass this to your LLM context context = OpenAILLMContext( messages = [{ "role" : "system" , "content" : "You are a helpful assistant." }], tools = tools ) The ToolsSchema will be automatically converted to the correct format for your LLM provider through adapters. ​ Using Direct Functions (Shorthand) You can bypass specifying a function configuration (as a FunctionSchema or in a provider-specific format) and instead pass the function directly to your ToolsSchema . Pipecat will auto-configure the function, gathering relevant metadata from its signature and docstring. Metadata includes: name description properties (including individual property descriptions) list of required properties Note that the function signature is a bit different when using direct functions. The first parameter is FunctionCallParams , followed by any others necessary for the function. Copy Ask AI from pipecat.adapters.schemas.tools_schema import ToolsSchema from pipecat.services.llm_service import FunctionCallParams # Define a direct function async def get_current_weather ( params : FunctionCallParams, location : str , format : str ): """Get the current weather. Args: location: The city and state, e.g. "San Francisco, CA". format: The temperature unit to use. Must be either "celsius" or "fahrenheit". """ weather_data = { "conditions" : "sunny" , "temperature" : "75" } await params.result_callback(weather_data) # Create a tools schema, passing your function directly to it tools = ToolsSchema( standard_tools = [get_current_weather]) # Pass this to your LLM context context = OpenAILLMContext( messages = [{ "role" : "system" , "content" : "You are a helpful assistant." }], tools = tools ) ​ Using Provider-Specific Formats (Alternative) You can also define functions in the provider-specific format if needed: OpenAI Anthropic Gemini Copy Ask AI from openai.types.chat import ChatCompletionToolParam # OpenAI native format tools = [ ChatCompletionToolParam( type = "function" , function = { "name" : "get_current_weather" , "description" : "Get the current weather" , "parameters" : { "type" : "object" , "properties" : { "location" : { "type" : "string" , "description" : "The city and state, e.g. San Francisco, CA" , }, "format" : { "type" : "string" , "enum" : [ "celsius" , "fahrenheit" ], "description" : "The temperature unit to use." , }, }, "required" : [ "location" , "format" ], }, }, ) ] ​ Provider-Specific Custom Tools Some providers support unique tools that don’t fit the standard function schema. For these cases, you can add custom tools: Copy Ask AI from pipecat.adapters.schemas.tools_schema import AdapterType, ToolsSchema # Standard functions weather_function = FunctionSchema( name = "get_current_weather" , description = "Get the current weather" , properties = { "location" : { "type" : "string" }}, required = [ "location" ] ) # Custom Gemini search tool gemini_search_tool = { "web_search" : { "description" : "Search the web for information" } } # Create a tools schema with both standard and custom tools tools = ToolsSchema( standard_tools = [weather_function], custom_tools = { AdapterType. GEMINI : [gemini_search_tool] } ) See the provider-specific documentation for details on custom tools and their formats. ​ 2. Register Function Handlers Register handlers for your functions using one of these LLM service methods : register_function register_direct_function Which one you use depends on whether your function is a “direct” function . Non-Direct Function Direct Function Copy Ask AI from pipecat.services.llm_service import FunctionCallParams llm = OpenAILLMService( api_key = "your-api-key" ) # Main function handler - called to execute the function async def fetch_weather_from_api ( params : FunctionCallParams): # Fetch weather data from your API weather_data = { "conditions" : "sunny" , "temperature" : "75" } await params.result_callback(weather_data) # Register the function llm.register_function( "get_current_weather" , fetch_weather_from_api, ) ​ 3. Create the Pipeline Include your LLM service in your pipeline with the registered functions: Copy Ask AI # Initialize the LLM context with your function schemas context = OpenAILLMContext( messages = [{ "role" : "system" , "content" : "You are a helpful assistant." }], tools = tools ) # Create the context aggregator to collect the user and assistant context context_aggregator = llm.create_context_aggregator(context) # Create the pipeline pipeline = Pipeline([ transport.input(), # Input from the transport stt, # STT processing context_aggregator.user(), # User context aggregation llm, # LLM processing tts, # TTS processing transport.output(), # Output to the transport context_aggregator.assistant(), # Assistant context aggregation ]) ​ Function Handler Details ​ FunctionCallParams The FunctionCallParams object contains all the information needed for handling function calls: params : FunctionCallParams function_name : Name of the called function arguments : Arguments passed by the LLM tool_call_id : Unique identifier for the function call llm : Reference to the LLM service context : Current conversation context result_callback : Async function to return results ​ function_name str Name of the function being called ​ tool_call_id str Unique identifier for the function call ​ arguments Mapping[str, Any] Arguments passed by the LLM to the function ​ llm LLMService Reference to the LLM service that initiated the call ​ context OpenAILLMContext Current conversation context ​ result_callback FunctionCallResultCallback Async callback function to return results ​ Handler Structure Your function handler should: Receive necessary arguments, either: From params.arguments Directly From function arguments, if using direct functions Process data or call external services Return results via params.result_callback(result) Non-Direct Function Direct Function Copy Ask AI async def fetch_weather_from_api ( params : FunctionCallParams): try : # Extract arguments location = params.arguments.get( "location" ) format_type = params.arguments.get( "format" , "celsius" ) # Call external API api_result = await weather_api.get_weather(location, format_type) # Return formatted result await params.result_callback({ "location" : location, "temperature" : api_result[ "temp" ], "conditions" : api_result[ "conditions" ], "unit" : format_type }) except Exception as e: # Handle errors await params.result_callback({ "error" : f "Failed to get weather: { str (e) } " }) ​ Controlling Function Call Behavior (Advanced) When returning results from a function handler, you can control how the LLM processes those results using a FunctionCallResultProperties object passed to the result callback. It can be handy to skip a completion when you have back-to-back function calls. Note, if you skip a completion, you must manually trigger one from the context. ​ Properties ​ run_llm Optional[bool] Controls whether the LLM should generate a response after the function call: True : Run LLM after function call (default if no other function calls in progress) False : Don’t run LLM after function call None : Use default behavior ​ on_context_updated Optional[Callable[[], Awaitable[None]]] Optional callback that runs after the function result is added to the context ​ Example Usage Copy Ask AI from pipecat.frames.frames import FunctionCallResultProperties from pipecat.services.llm_service import FunctionCallParams async def fetch_weather_from_api ( params : FunctionCallParams): # Fetch weather data weather_data = { "conditions" : "sunny" , "temperature" : "75" } # Don't run LLM after this function call properties = FunctionCallResultProperties( run_llm = False ) await params.result_callback(weather_data, properties = properties) async def query_database ( params : FunctionCallParams): # Query database results = await db.query(params.arguments[ "query" ]) async def on_update (): await notify_system( "Database query complete" ) # Run LLM after function call and notify when context is updated properties = FunctionCallResultProperties( run_llm = True , on_context_updated = on_update ) await params.result_callback(results, properties = properties) ​ Next steps Check out the function calling examples to see a complete example for specific LLM providers. Refer to your LLM provider’s documentation to learn more about their function calling capabilities. Ending a Pipeline Muting User Input On this page Understanding Function Calling Implementation 1. Define Functions Using the Standard Schema (Recommended) Using Direct Functions (Shorthand) Using Provider-Specific Formats (Alternative) Provider-Specific Custom Tools 2. Register Function Handlers 3. Create the Pipeline Function Handler Details FunctionCallParams Handler Structure Controlling Function Call Behavior (Advanced) Properties Example Usage Next steps Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/fundamentals_function-calling_33c71388.txt b/fundamentals_function-calling_33c71388.txt
new file mode 100644
index 0000000000000000000000000000000000000000..52a9a968d978cb1dc64b4239a4f76b1918434dbd
--- /dev/null
+++ b/fundamentals_function-calling_33c71388.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/_sites/docs.pipecat.ai/guides/fundamentals/function-calling#next-steps
+Title: Overview - Pipecat
+==================================================
+
+Overview - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Get Started Overview Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Get Started Overview Installation & Setup Quickstart Core Concepts Next Steps & Examples Pipecat is an open source Python framework that handles the complex orchestration of AI services, network transport, audio processing, and multimodal interactions. “Multimodal” means you can use any combination of audio, video, images, and/or text in your interactions. And “real-time” means that things are happening quickly enough that it feels conversational—a “back-and-forth” with a bot, not submitting a query and waiting for results. ​ What You Can Build Voice Assistants Natural, real-time conversations with AI using speech recognition and synthesis Interactive Agents Personal coaches and meeting assistants that can understand context and provide guidance Multimodal Apps Applications that combine voice, video, images, and text for rich interactions Creative Tools Storytelling experiences and social companions that engage users Business Solutions Customer intake flows and support bots for automated business processes Complex Flows Structured conversations using Pipecat Flows for managing complex interactions ​ How It Works The flow of interactions in a Pipecat application is typically straightforward: The bot says something The user says something The bot says something The user says something This continues until the conversation naturally ends. While this flow seems simple, making it feel natural requires sophisticated real-time processing. ​ Real-time Processing Pipecat’s pipeline architecture handles both simple voice interactions and complex multimodal processing. Let’s look at how data flows through the system: Voice app Multimodal app 1 Send Audio Transmit and capture streamed audio from the user 2 Transcribe Speech Convert speech to text as the user is talking 3 Process with LLM Generate responses using a large language model 4 Convert to Speech Transform text responses into natural speech 5 Play Audio Stream the audio response back to the user 1 Send Audio Transmit and capture streamed audio from the user 2 Transcribe Speech Convert speech to text as the user is talking 3 Process with LLM Generate responses using a large language model 4 Convert to Speech Transform text responses into natural speech 5 Play Audio Stream the audio response back to the user 1 Send Audio and Video Transmit and capture audio, video, and image inputs simultaneously 2 Process Streams Handle multiple input streams in parallel 3 Model Processing Send combined inputs to multimodal models (like GPT-4V) 4 Generate Outputs Create various outputs (text, images, audio, etc.) 5 Coordinate Presentation Synchronize and present multiple output types In both cases, Pipecat: Processes responses as they stream in Handles multiple input/output modalities concurrently Manages resource allocation and synchronization Coordinates parallel processing tasks This architecture creates fluid, natural interactions without noticeable delays, whether you’re building a simple voice assistant or a complex multimodal application. Pipecat’s pipeline architecture is particularly valuable for managing the complexity of real-time, multimodal interactions, ensuring smooth data flow and proper synchronization regardless of the input/output types involved. Pipecat handles all this complexity for you, letting you focus on building your application rather than managing the underlying infrastructure. ​ Next Steps Ready to build your first Pipecat application? Installation & Setup Prepare your environment and install required dependencies Quickstart Build and run your first Pipecat application Core Concepts Learn about pipelines, frames, and real-time processing Use Cases Explore example implementations and patterns ​ Join Our Community Discord Community Connect with other developers, share your projects, and get support from the Pipecat team. Installation & Setup On this page What You Can Build How It Works Real-time Processing Next Steps Join Our Community Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/fundamentals_function-calling_37461605.txt b/fundamentals_function-calling_37461605.txt
new file mode 100644
index 0000000000000000000000000000000000000000..aee736dc6741dbb0b2d88f594b3fdcd8bd96797e
--- /dev/null
+++ b/fundamentals_function-calling_37461605.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/guides/fundamentals/function-calling#1-define-functions
+Title: Function Calling - Pipecat
+==================================================
+
+Function Calling - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Fundamentals Function Calling Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Guides Fundamentals Context Management Custom FrameProcessor Detecting Idle Users Ending a Pipeline Function Calling Muting User Input Recording Audio Recording Transcripts Features Gemini Multimodal Live Metrics Noise cancellation with Krisp OpenAI Audio Models and APIs Pipecat Flows Telephony Overview Dial-in: WebRTC (Daily) Dial-in: WebRTC (Twilio + Daily) Dial-in: Twilio (Media Streams) Dialout: WebRTC (Daily) Deploying your bot Overview Deployment pattern Example: Pipecat Cloud Example: Fly.io Example: Cerebrium Example: Modal ​ Understanding Function Calling Function calling (also known as tool calling) allows LLMs to request information from external services and APIs. This enables your bot to access real-time data and perform actions that aren’t part of its training data. For example, you could give your bot the ability to: Check current weather conditions Look up stock prices Query a database Control smart home devices Schedule appointments Here’s how it works: You define functions the LLM can use and register them to the LLM service used in your pipeline When needed, the LLM requests a function call Your application executes any corresponding functions The result is sent back to the LLM The LLM uses this information in its response ​ Implementation ​ 1. Define Functions Pipecat provides a standardized FunctionSchema that works across all supported LLM providers. This makes it easy to define functions once and use them with any provider. As a shorthand, you could also bypass specifying a function configuration at all and instead use “direct” functions. Under the hood, these are converted to FunctionSchema s. ​ Using the Standard Schema (Recommended) Copy Ask AI from pipecat.adapters.schemas.function_schema import FunctionSchema from pipecat.adapters.schemas.tools_schema import ToolsSchema # Define a function using the standard schema weather_function = FunctionSchema( name = "get_current_weather" , description = "Get the current weather in a location" , properties = { "location" : { "type" : "string" , "description" : "The city and state, e.g. San Francisco, CA" , }, "format" : { "type" : "string" , "enum" : [ "celsius" , "fahrenheit" ], "description" : "The temperature unit to use." , }, }, required = [ "location" , "format" ] ) # Create a tools schema with your functions tools = ToolsSchema( standard_tools = [weather_function]) # Pass this to your LLM context context = OpenAILLMContext( messages = [{ "role" : "system" , "content" : "You are a helpful assistant." }], tools = tools ) The ToolsSchema will be automatically converted to the correct format for your LLM provider through adapters. ​ Using Direct Functions (Shorthand) You can bypass specifying a function configuration (as a FunctionSchema or in a provider-specific format) and instead pass the function directly to your ToolsSchema . Pipecat will auto-configure the function, gathering relevant metadata from its signature and docstring. Metadata includes: name description properties (including individual property descriptions) list of required properties Note that the function signature is a bit different when using direct functions. The first parameter is FunctionCallParams , followed by any others necessary for the function. Copy Ask AI from pipecat.adapters.schemas.tools_schema import ToolsSchema from pipecat.services.llm_service import FunctionCallParams # Define a direct function async def get_current_weather ( params : FunctionCallParams, location : str , format : str ): """Get the current weather. Args: location: The city and state, e.g. "San Francisco, CA". format: The temperature unit to use. Must be either "celsius" or "fahrenheit". """ weather_data = { "conditions" : "sunny" , "temperature" : "75" } await params.result_callback(weather_data) # Create a tools schema, passing your function directly to it tools = ToolsSchema( standard_tools = [get_current_weather]) # Pass this to your LLM context context = OpenAILLMContext( messages = [{ "role" : "system" , "content" : "You are a helpful assistant." }], tools = tools ) ​ Using Provider-Specific Formats (Alternative) You can also define functions in the provider-specific format if needed: OpenAI Anthropic Gemini Copy Ask AI from openai.types.chat import ChatCompletionToolParam # OpenAI native format tools = [ ChatCompletionToolParam( type = "function" , function = { "name" : "get_current_weather" , "description" : "Get the current weather" , "parameters" : { "type" : "object" , "properties" : { "location" : { "type" : "string" , "description" : "The city and state, e.g. San Francisco, CA" , }, "format" : { "type" : "string" , "enum" : [ "celsius" , "fahrenheit" ], "description" : "The temperature unit to use." , }, }, "required" : [ "location" , "format" ], }, }, ) ] ​ Provider-Specific Custom Tools Some providers support unique tools that don’t fit the standard function schema. For these cases, you can add custom tools: Copy Ask AI from pipecat.adapters.schemas.tools_schema import AdapterType, ToolsSchema # Standard functions weather_function = FunctionSchema( name = "get_current_weather" , description = "Get the current weather" , properties = { "location" : { "type" : "string" }}, required = [ "location" ] ) # Custom Gemini search tool gemini_search_tool = { "web_search" : { "description" : "Search the web for information" } } # Create a tools schema with both standard and custom tools tools = ToolsSchema( standard_tools = [weather_function], custom_tools = { AdapterType. GEMINI : [gemini_search_tool] } ) See the provider-specific documentation for details on custom tools and their formats. ​ 2. Register Function Handlers Register handlers for your functions using one of these LLM service methods : register_function register_direct_function Which one you use depends on whether your function is a “direct” function . Non-Direct Function Direct Function Copy Ask AI from pipecat.services.llm_service import FunctionCallParams llm = OpenAILLMService( api_key = "your-api-key" ) # Main function handler - called to execute the function async def fetch_weather_from_api ( params : FunctionCallParams): # Fetch weather data from your API weather_data = { "conditions" : "sunny" , "temperature" : "75" } await params.result_callback(weather_data) # Register the function llm.register_function( "get_current_weather" , fetch_weather_from_api, ) ​ 3. Create the Pipeline Include your LLM service in your pipeline with the registered functions: Copy Ask AI # Initialize the LLM context with your function schemas context = OpenAILLMContext( messages = [{ "role" : "system" , "content" : "You are a helpful assistant." }], tools = tools ) # Create the context aggregator to collect the user and assistant context context_aggregator = llm.create_context_aggregator(context) # Create the pipeline pipeline = Pipeline([ transport.input(), # Input from the transport stt, # STT processing context_aggregator.user(), # User context aggregation llm, # LLM processing tts, # TTS processing transport.output(), # Output to the transport context_aggregator.assistant(), # Assistant context aggregation ]) ​ Function Handler Details ​ FunctionCallParams The FunctionCallParams object contains all the information needed for handling function calls: params : FunctionCallParams function_name : Name of the called function arguments : Arguments passed by the LLM tool_call_id : Unique identifier for the function call llm : Reference to the LLM service context : Current conversation context result_callback : Async function to return results ​ function_name str Name of the function being called ​ tool_call_id str Unique identifier for the function call ​ arguments Mapping[str, Any] Arguments passed by the LLM to the function ​ llm LLMService Reference to the LLM service that initiated the call ​ context OpenAILLMContext Current conversation context ​ result_callback FunctionCallResultCallback Async callback function to return results ​ Handler Structure Your function handler should: Receive necessary arguments, either: From params.arguments Directly From function arguments, if using direct functions Process data or call external services Return results via params.result_callback(result) Non-Direct Function Direct Function Copy Ask AI async def fetch_weather_from_api ( params : FunctionCallParams): try : # Extract arguments location = params.arguments.get( "location" ) format_type = params.arguments.get( "format" , "celsius" ) # Call external API api_result = await weather_api.get_weather(location, format_type) # Return formatted result await params.result_callback({ "location" : location, "temperature" : api_result[ "temp" ], "conditions" : api_result[ "conditions" ], "unit" : format_type }) except Exception as e: # Handle errors await params.result_callback({ "error" : f "Failed to get weather: { str (e) } " }) ​ Controlling Function Call Behavior (Advanced) When returning results from a function handler, you can control how the LLM processes those results using a FunctionCallResultProperties object passed to the result callback. It can be handy to skip a completion when you have back-to-back function calls. Note, if you skip a completion, you must manually trigger one from the context. ​ Properties ​ run_llm Optional[bool] Controls whether the LLM should generate a response after the function call: True : Run LLM after function call (default if no other function calls in progress) False : Don’t run LLM after function call None : Use default behavior ​ on_context_updated Optional[Callable[[], Awaitable[None]]] Optional callback that runs after the function result is added to the context ​ Example Usage Copy Ask AI from pipecat.frames.frames import FunctionCallResultProperties from pipecat.services.llm_service import FunctionCallParams async def fetch_weather_from_api ( params : FunctionCallParams): # Fetch weather data weather_data = { "conditions" : "sunny" , "temperature" : "75" } # Don't run LLM after this function call properties = FunctionCallResultProperties( run_llm = False ) await params.result_callback(weather_data, properties = properties) async def query_database ( params : FunctionCallParams): # Query database results = await db.query(params.arguments[ "query" ]) async def on_update (): await notify_system( "Database query complete" ) # Run LLM after function call and notify when context is updated properties = FunctionCallResultProperties( run_llm = True , on_context_updated = on_update ) await params.result_callback(results, properties = properties) ​ Next steps Check out the function calling examples to see a complete example for specific LLM providers. Refer to your LLM provider’s documentation to learn more about their function calling capabilities. Ending a Pipeline Muting User Input On this page Understanding Function Calling Implementation 1. Define Functions Using the Standard Schema (Recommended) Using Direct Functions (Shorthand) Using Provider-Specific Formats (Alternative) Provider-Specific Custom Tools 2. Register Function Handlers 3. Create the Pipeline Function Handler Details FunctionCallParams Handler Structure Controlling Function Call Behavior (Advanced) Properties Example Usage Next steps Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/fundamentals_function-calling_7c3dda4e.txt b/fundamentals_function-calling_7c3dda4e.txt
new file mode 100644
index 0000000000000000000000000000000000000000..03fc7c82688990beff187d97f7bace032d24f4c7
--- /dev/null
+++ b/fundamentals_function-calling_7c3dda4e.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/guides/fundamentals/function-calling#param-arguments
+Title: Function Calling - Pipecat
+==================================================
+
+Function Calling - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Fundamentals Function Calling Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Guides Fundamentals Context Management Custom FrameProcessor Detecting Idle Users Ending a Pipeline Function Calling Muting User Input Recording Audio Recording Transcripts Features Gemini Multimodal Live Metrics Noise cancellation with Krisp OpenAI Audio Models and APIs Pipecat Flows Telephony Overview Dial-in: WebRTC (Daily) Dial-in: WebRTC (Twilio + Daily) Dial-in: Twilio (Media Streams) Dialout: WebRTC (Daily) Deploying your bot Overview Deployment pattern Example: Pipecat Cloud Example: Fly.io Example: Cerebrium Example: Modal ​ Understanding Function Calling Function calling (also known as tool calling) allows LLMs to request information from external services and APIs. This enables your bot to access real-time data and perform actions that aren’t part of its training data. For example, you could give your bot the ability to: Check current weather conditions Look up stock prices Query a database Control smart home devices Schedule appointments Here’s how it works: You define functions the LLM can use and register them to the LLM service used in your pipeline When needed, the LLM requests a function call Your application executes any corresponding functions The result is sent back to the LLM The LLM uses this information in its response ​ Implementation ​ 1. Define Functions Pipecat provides a standardized FunctionSchema that works across all supported LLM providers. This makes it easy to define functions once and use them with any provider. As a shorthand, you could also bypass specifying a function configuration at all and instead use “direct” functions. Under the hood, these are converted to FunctionSchema s. ​ Using the Standard Schema (Recommended) Copy Ask AI from pipecat.adapters.schemas.function_schema import FunctionSchema from pipecat.adapters.schemas.tools_schema import ToolsSchema # Define a function using the standard schema weather_function = FunctionSchema( name = "get_current_weather" , description = "Get the current weather in a location" , properties = { "location" : { "type" : "string" , "description" : "The city and state, e.g. San Francisco, CA" , }, "format" : { "type" : "string" , "enum" : [ "celsius" , "fahrenheit" ], "description" : "The temperature unit to use." , }, }, required = [ "location" , "format" ] ) # Create a tools schema with your functions tools = ToolsSchema( standard_tools = [weather_function]) # Pass this to your LLM context context = OpenAILLMContext( messages = [{ "role" : "system" , "content" : "You are a helpful assistant." }], tools = tools ) The ToolsSchema will be automatically converted to the correct format for your LLM provider through adapters. ​ Using Direct Functions (Shorthand) You can bypass specifying a function configuration (as a FunctionSchema or in a provider-specific format) and instead pass the function directly to your ToolsSchema . Pipecat will auto-configure the function, gathering relevant metadata from its signature and docstring. Metadata includes: name description properties (including individual property descriptions) list of required properties Note that the function signature is a bit different when using direct functions. The first parameter is FunctionCallParams , followed by any others necessary for the function. Copy Ask AI from pipecat.adapters.schemas.tools_schema import ToolsSchema from pipecat.services.llm_service import FunctionCallParams # Define a direct function async def get_current_weather ( params : FunctionCallParams, location : str , format : str ): """Get the current weather. Args: location: The city and state, e.g. "San Francisco, CA". format: The temperature unit to use. Must be either "celsius" or "fahrenheit". """ weather_data = { "conditions" : "sunny" , "temperature" : "75" } await params.result_callback(weather_data) # Create a tools schema, passing your function directly to it tools = ToolsSchema( standard_tools = [get_current_weather]) # Pass this to your LLM context context = OpenAILLMContext( messages = [{ "role" : "system" , "content" : "You are a helpful assistant." }], tools = tools ) ​ Using Provider-Specific Formats (Alternative) You can also define functions in the provider-specific format if needed: OpenAI Anthropic Gemini Copy Ask AI from openai.types.chat import ChatCompletionToolParam # OpenAI native format tools = [ ChatCompletionToolParam( type = "function" , function = { "name" : "get_current_weather" , "description" : "Get the current weather" , "parameters" : { "type" : "object" , "properties" : { "location" : { "type" : "string" , "description" : "The city and state, e.g. San Francisco, CA" , }, "format" : { "type" : "string" , "enum" : [ "celsius" , "fahrenheit" ], "description" : "The temperature unit to use." , }, }, "required" : [ "location" , "format" ], }, }, ) ] ​ Provider-Specific Custom Tools Some providers support unique tools that don’t fit the standard function schema. For these cases, you can add custom tools: Copy Ask AI from pipecat.adapters.schemas.tools_schema import AdapterType, ToolsSchema # Standard functions weather_function = FunctionSchema( name = "get_current_weather" , description = "Get the current weather" , properties = { "location" : { "type" : "string" }}, required = [ "location" ] ) # Custom Gemini search tool gemini_search_tool = { "web_search" : { "description" : "Search the web for information" } } # Create a tools schema with both standard and custom tools tools = ToolsSchema( standard_tools = [weather_function], custom_tools = { AdapterType. GEMINI : [gemini_search_tool] } ) See the provider-specific documentation for details on custom tools and their formats. ​ 2. Register Function Handlers Register handlers for your functions using one of these LLM service methods : register_function register_direct_function Which one you use depends on whether your function is a “direct” function . Non-Direct Function Direct Function Copy Ask AI from pipecat.services.llm_service import FunctionCallParams llm = OpenAILLMService( api_key = "your-api-key" ) # Main function handler - called to execute the function async def fetch_weather_from_api ( params : FunctionCallParams): # Fetch weather data from your API weather_data = { "conditions" : "sunny" , "temperature" : "75" } await params.result_callback(weather_data) # Register the function llm.register_function( "get_current_weather" , fetch_weather_from_api, ) ​ 3. Create the Pipeline Include your LLM service in your pipeline with the registered functions: Copy Ask AI # Initialize the LLM context with your function schemas context = OpenAILLMContext( messages = [{ "role" : "system" , "content" : "You are a helpful assistant." }], tools = tools ) # Create the context aggregator to collect the user and assistant context context_aggregator = llm.create_context_aggregator(context) # Create the pipeline pipeline = Pipeline([ transport.input(), # Input from the transport stt, # STT processing context_aggregator.user(), # User context aggregation llm, # LLM processing tts, # TTS processing transport.output(), # Output to the transport context_aggregator.assistant(), # Assistant context aggregation ]) ​ Function Handler Details ​ FunctionCallParams The FunctionCallParams object contains all the information needed for handling function calls: params : FunctionCallParams function_name : Name of the called function arguments : Arguments passed by the LLM tool_call_id : Unique identifier for the function call llm : Reference to the LLM service context : Current conversation context result_callback : Async function to return results ​ function_name str Name of the function being called ​ tool_call_id str Unique identifier for the function call ​ arguments Mapping[str, Any] Arguments passed by the LLM to the function ​ llm LLMService Reference to the LLM service that initiated the call ​ context OpenAILLMContext Current conversation context ​ result_callback FunctionCallResultCallback Async callback function to return results ​ Handler Structure Your function handler should: Receive necessary arguments, either: From params.arguments Directly From function arguments, if using direct functions Process data or call external services Return results via params.result_callback(result) Non-Direct Function Direct Function Copy Ask AI async def fetch_weather_from_api ( params : FunctionCallParams): try : # Extract arguments location = params.arguments.get( "location" ) format_type = params.arguments.get( "format" , "celsius" ) # Call external API api_result = await weather_api.get_weather(location, format_type) # Return formatted result await params.result_callback({ "location" : location, "temperature" : api_result[ "temp" ], "conditions" : api_result[ "conditions" ], "unit" : format_type }) except Exception as e: # Handle errors await params.result_callback({ "error" : f "Failed to get weather: { str (e) } " }) ​ Controlling Function Call Behavior (Advanced) When returning results from a function handler, you can control how the LLM processes those results using a FunctionCallResultProperties object passed to the result callback. It can be handy to skip a completion when you have back-to-back function calls. Note, if you skip a completion, you must manually trigger one from the context. ​ Properties ​ run_llm Optional[bool] Controls whether the LLM should generate a response after the function call: True : Run LLM after function call (default if no other function calls in progress) False : Don’t run LLM after function call None : Use default behavior ​ on_context_updated Optional[Callable[[], Awaitable[None]]] Optional callback that runs after the function result is added to the context ​ Example Usage Copy Ask AI from pipecat.frames.frames import FunctionCallResultProperties from pipecat.services.llm_service import FunctionCallParams async def fetch_weather_from_api ( params : FunctionCallParams): # Fetch weather data weather_data = { "conditions" : "sunny" , "temperature" : "75" } # Don't run LLM after this function call properties = FunctionCallResultProperties( run_llm = False ) await params.result_callback(weather_data, properties = properties) async def query_database ( params : FunctionCallParams): # Query database results = await db.query(params.arguments[ "query" ]) async def on_update (): await notify_system( "Database query complete" ) # Run LLM after function call and notify when context is updated properties = FunctionCallResultProperties( run_llm = True , on_context_updated = on_update ) await params.result_callback(results, properties = properties) ​ Next steps Check out the function calling examples to see a complete example for specific LLM providers. Refer to your LLM provider’s documentation to learn more about their function calling capabilities. Ending a Pipeline Muting User Input On this page Understanding Function Calling Implementation 1. Define Functions Using the Standard Schema (Recommended) Using Direct Functions (Shorthand) Using Provider-Specific Formats (Alternative) Provider-Specific Custom Tools 2. Register Function Handlers 3. Create the Pipeline Function Handler Details FunctionCallParams Handler Structure Controlling Function Call Behavior (Advanced) Properties Example Usage Next steps Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/fundamentals_function-calling_cf1f8e42.txt b/fundamentals_function-calling_cf1f8e42.txt
new file mode 100644
index 0000000000000000000000000000000000000000..49ff27d10385578c1cd3dd166772d3e1aa4de346
--- /dev/null
+++ b/fundamentals_function-calling_cf1f8e42.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/guides/fundamentals/function-calling#functioncallparams
+Title: Function Calling - Pipecat
+==================================================
+
+Function Calling - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Fundamentals Function Calling Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Guides Fundamentals Context Management Custom FrameProcessor Detecting Idle Users Ending a Pipeline Function Calling Muting User Input Recording Audio Recording Transcripts Features Gemini Multimodal Live Metrics Noise cancellation with Krisp OpenAI Audio Models and APIs Pipecat Flows Telephony Overview Dial-in: WebRTC (Daily) Dial-in: WebRTC (Twilio + Daily) Dial-in: Twilio (Media Streams) Dialout: WebRTC (Daily) Deploying your bot Overview Deployment pattern Example: Pipecat Cloud Example: Fly.io Example: Cerebrium Example: Modal ​ Understanding Function Calling Function calling (also known as tool calling) allows LLMs to request information from external services and APIs. This enables your bot to access real-time data and perform actions that aren’t part of its training data. For example, you could give your bot the ability to: Check current weather conditions Look up stock prices Query a database Control smart home devices Schedule appointments Here’s how it works: You define functions the LLM can use and register them to the LLM service used in your pipeline When needed, the LLM requests a function call Your application executes any corresponding functions The result is sent back to the LLM The LLM uses this information in its response ​ Implementation ​ 1. Define Functions Pipecat provides a standardized FunctionSchema that works across all supported LLM providers. This makes it easy to define functions once and use them with any provider. As a shorthand, you could also bypass specifying a function configuration at all and instead use “direct” functions. Under the hood, these are converted to FunctionSchema s. ​ Using the Standard Schema (Recommended) Copy Ask AI from pipecat.adapters.schemas.function_schema import FunctionSchema from pipecat.adapters.schemas.tools_schema import ToolsSchema # Define a function using the standard schema weather_function = FunctionSchema( name = "get_current_weather" , description = "Get the current weather in a location" , properties = { "location" : { "type" : "string" , "description" : "The city and state, e.g. San Francisco, CA" , }, "format" : { "type" : "string" , "enum" : [ "celsius" , "fahrenheit" ], "description" : "The temperature unit to use." , }, }, required = [ "location" , "format" ] ) # Create a tools schema with your functions tools = ToolsSchema( standard_tools = [weather_function]) # Pass this to your LLM context context = OpenAILLMContext( messages = [{ "role" : "system" , "content" : "You are a helpful assistant." }], tools = tools ) The ToolsSchema will be automatically converted to the correct format for your LLM provider through adapters. ​ Using Direct Functions (Shorthand) You can bypass specifying a function configuration (as a FunctionSchema or in a provider-specific format) and instead pass the function directly to your ToolsSchema . Pipecat will auto-configure the function, gathering relevant metadata from its signature and docstring. Metadata includes: name description properties (including individual property descriptions) list of required properties Note that the function signature is a bit different when using direct functions. The first parameter is FunctionCallParams , followed by any others necessary for the function. Copy Ask AI from pipecat.adapters.schemas.tools_schema import ToolsSchema from pipecat.services.llm_service import FunctionCallParams # Define a direct function async def get_current_weather ( params : FunctionCallParams, location : str , format : str ): """Get the current weather. Args: location: The city and state, e.g. "San Francisco, CA". format: The temperature unit to use. Must be either "celsius" or "fahrenheit". """ weather_data = { "conditions" : "sunny" , "temperature" : "75" } await params.result_callback(weather_data) # Create a tools schema, passing your function directly to it tools = ToolsSchema( standard_tools = [get_current_weather]) # Pass this to your LLM context context = OpenAILLMContext( messages = [{ "role" : "system" , "content" : "You are a helpful assistant." }], tools = tools ) ​ Using Provider-Specific Formats (Alternative) You can also define functions in the provider-specific format if needed: OpenAI Anthropic Gemini Copy Ask AI from openai.types.chat import ChatCompletionToolParam # OpenAI native format tools = [ ChatCompletionToolParam( type = "function" , function = { "name" : "get_current_weather" , "description" : "Get the current weather" , "parameters" : { "type" : "object" , "properties" : { "location" : { "type" : "string" , "description" : "The city and state, e.g. San Francisco, CA" , }, "format" : { "type" : "string" , "enum" : [ "celsius" , "fahrenheit" ], "description" : "The temperature unit to use." , }, }, "required" : [ "location" , "format" ], }, }, ) ] ​ Provider-Specific Custom Tools Some providers support unique tools that don’t fit the standard function schema. For these cases, you can add custom tools: Copy Ask AI from pipecat.adapters.schemas.tools_schema import AdapterType, ToolsSchema # Standard functions weather_function = FunctionSchema( name = "get_current_weather" , description = "Get the current weather" , properties = { "location" : { "type" : "string" }}, required = [ "location" ] ) # Custom Gemini search tool gemini_search_tool = { "web_search" : { "description" : "Search the web for information" } } # Create a tools schema with both standard and custom tools tools = ToolsSchema( standard_tools = [weather_function], custom_tools = { AdapterType. GEMINI : [gemini_search_tool] } ) See the provider-specific documentation for details on custom tools and their formats. ​ 2. Register Function Handlers Register handlers for your functions using one of these LLM service methods : register_function register_direct_function Which one you use depends on whether your function is a “direct” function . Non-Direct Function Direct Function Copy Ask AI from pipecat.services.llm_service import FunctionCallParams llm = OpenAILLMService( api_key = "your-api-key" ) # Main function handler - called to execute the function async def fetch_weather_from_api ( params : FunctionCallParams): # Fetch weather data from your API weather_data = { "conditions" : "sunny" , "temperature" : "75" } await params.result_callback(weather_data) # Register the function llm.register_function( "get_current_weather" , fetch_weather_from_api, ) ​ 3. Create the Pipeline Include your LLM service in your pipeline with the registered functions: Copy Ask AI # Initialize the LLM context with your function schemas context = OpenAILLMContext( messages = [{ "role" : "system" , "content" : "You are a helpful assistant." }], tools = tools ) # Create the context aggregator to collect the user and assistant context context_aggregator = llm.create_context_aggregator(context) # Create the pipeline pipeline = Pipeline([ transport.input(), # Input from the transport stt, # STT processing context_aggregator.user(), # User context aggregation llm, # LLM processing tts, # TTS processing transport.output(), # Output to the transport context_aggregator.assistant(), # Assistant context aggregation ]) ​ Function Handler Details ​ FunctionCallParams The FunctionCallParams object contains all the information needed for handling function calls: params : FunctionCallParams function_name : Name of the called function arguments : Arguments passed by the LLM tool_call_id : Unique identifier for the function call llm : Reference to the LLM service context : Current conversation context result_callback : Async function to return results ​ function_name str Name of the function being called ​ tool_call_id str Unique identifier for the function call ​ arguments Mapping[str, Any] Arguments passed by the LLM to the function ​ llm LLMService Reference to the LLM service that initiated the call ​ context OpenAILLMContext Current conversation context ​ result_callback FunctionCallResultCallback Async callback function to return results ​ Handler Structure Your function handler should: Receive necessary arguments, either: From params.arguments Directly From function arguments, if using direct functions Process data or call external services Return results via params.result_callback(result) Non-Direct Function Direct Function Copy Ask AI async def fetch_weather_from_api ( params : FunctionCallParams): try : # Extract arguments location = params.arguments.get( "location" ) format_type = params.arguments.get( "format" , "celsius" ) # Call external API api_result = await weather_api.get_weather(location, format_type) # Return formatted result await params.result_callback({ "location" : location, "temperature" : api_result[ "temp" ], "conditions" : api_result[ "conditions" ], "unit" : format_type }) except Exception as e: # Handle errors await params.result_callback({ "error" : f "Failed to get weather: { str (e) } " }) ​ Controlling Function Call Behavior (Advanced) When returning results from a function handler, you can control how the LLM processes those results using a FunctionCallResultProperties object passed to the result callback. It can be handy to skip a completion when you have back-to-back function calls. Note, if you skip a completion, you must manually trigger one from the context. ​ Properties ​ run_llm Optional[bool] Controls whether the LLM should generate a response after the function call: True : Run LLM after function call (default if no other function calls in progress) False : Don’t run LLM after function call None : Use default behavior ​ on_context_updated Optional[Callable[[], Awaitable[None]]] Optional callback that runs after the function result is added to the context ​ Example Usage Copy Ask AI from pipecat.frames.frames import FunctionCallResultProperties from pipecat.services.llm_service import FunctionCallParams async def fetch_weather_from_api ( params : FunctionCallParams): # Fetch weather data weather_data = { "conditions" : "sunny" , "temperature" : "75" } # Don't run LLM after this function call properties = FunctionCallResultProperties( run_llm = False ) await params.result_callback(weather_data, properties = properties) async def query_database ( params : FunctionCallParams): # Query database results = await db.query(params.arguments[ "query" ]) async def on_update (): await notify_system( "Database query complete" ) # Run LLM after function call and notify when context is updated properties = FunctionCallResultProperties( run_llm = True , on_context_updated = on_update ) await params.result_callback(results, properties = properties) ​ Next steps Check out the function calling examples to see a complete example for specific LLM providers. Refer to your LLM provider’s documentation to learn more about their function calling capabilities. Ending a Pipeline Muting User Input On this page Understanding Function Calling Implementation 1. Define Functions Using the Standard Schema (Recommended) Using Direct Functions (Shorthand) Using Provider-Specific Formats (Alternative) Provider-Specific Custom Tools 2. Register Function Handlers 3. Create the Pipeline Function Handler Details FunctionCallParams Handler Structure Controlling Function Call Behavior (Advanced) Properties Example Usage Next steps Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/fundamentals_function-calling_efaefd3d.txt b/fundamentals_function-calling_efaefd3d.txt
new file mode 100644
index 0000000000000000000000000000000000000000..52ae6dd85d9b806479a9f3efaac7bbedd30acac9
--- /dev/null
+++ b/fundamentals_function-calling_efaefd3d.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/guides/fundamentals/function-calling#3-create-the-pipeline
+Title: Function Calling - Pipecat
+==================================================
+
+Function Calling - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Fundamentals Function Calling Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Guides Fundamentals Context Management Custom FrameProcessor Detecting Idle Users Ending a Pipeline Function Calling Muting User Input Recording Audio Recording Transcripts Features Gemini Multimodal Live Metrics Noise cancellation with Krisp OpenAI Audio Models and APIs Pipecat Flows Telephony Overview Dial-in: WebRTC (Daily) Dial-in: WebRTC (Twilio + Daily) Dial-in: Twilio (Media Streams) Dialout: WebRTC (Daily) Deploying your bot Overview Deployment pattern Example: Pipecat Cloud Example: Fly.io Example: Cerebrium Example: Modal ​ Understanding Function Calling Function calling (also known as tool calling) allows LLMs to request information from external services and APIs. This enables your bot to access real-time data and perform actions that aren’t part of its training data. For example, you could give your bot the ability to: Check current weather conditions Look up stock prices Query a database Control smart home devices Schedule appointments Here’s how it works: You define functions the LLM can use and register them to the LLM service used in your pipeline When needed, the LLM requests a function call Your application executes any corresponding functions The result is sent back to the LLM The LLM uses this information in its response ​ Implementation ​ 1. Define Functions Pipecat provides a standardized FunctionSchema that works across all supported LLM providers. This makes it easy to define functions once and use them with any provider. As a shorthand, you could also bypass specifying a function configuration at all and instead use “direct” functions. Under the hood, these are converted to FunctionSchema s. ​ Using the Standard Schema (Recommended) Copy Ask AI from pipecat.adapters.schemas.function_schema import FunctionSchema from pipecat.adapters.schemas.tools_schema import ToolsSchema # Define a function using the standard schema weather_function = FunctionSchema( name = "get_current_weather" , description = "Get the current weather in a location" , properties = { "location" : { "type" : "string" , "description" : "The city and state, e.g. San Francisco, CA" , }, "format" : { "type" : "string" , "enum" : [ "celsius" , "fahrenheit" ], "description" : "The temperature unit to use." , }, }, required = [ "location" , "format" ] ) # Create a tools schema with your functions tools = ToolsSchema( standard_tools = [weather_function]) # Pass this to your LLM context context = OpenAILLMContext( messages = [{ "role" : "system" , "content" : "You are a helpful assistant." }], tools = tools ) The ToolsSchema will be automatically converted to the correct format for your LLM provider through adapters. ​ Using Direct Functions (Shorthand) You can bypass specifying a function configuration (as a FunctionSchema or in a provider-specific format) and instead pass the function directly to your ToolsSchema . Pipecat will auto-configure the function, gathering relevant metadata from its signature and docstring. Metadata includes: name description properties (including individual property descriptions) list of required properties Note that the function signature is a bit different when using direct functions. The first parameter is FunctionCallParams , followed by any others necessary for the function. Copy Ask AI from pipecat.adapters.schemas.tools_schema import ToolsSchema from pipecat.services.llm_service import FunctionCallParams # Define a direct function async def get_current_weather ( params : FunctionCallParams, location : str , format : str ): """Get the current weather. Args: location: The city and state, e.g. "San Francisco, CA". format: The temperature unit to use. Must be either "celsius" or "fahrenheit". """ weather_data = { "conditions" : "sunny" , "temperature" : "75" } await params.result_callback(weather_data) # Create a tools schema, passing your function directly to it tools = ToolsSchema( standard_tools = [get_current_weather]) # Pass this to your LLM context context = OpenAILLMContext( messages = [{ "role" : "system" , "content" : "You are a helpful assistant." }], tools = tools ) ​ Using Provider-Specific Formats (Alternative) You can also define functions in the provider-specific format if needed: OpenAI Anthropic Gemini Copy Ask AI from openai.types.chat import ChatCompletionToolParam # OpenAI native format tools = [ ChatCompletionToolParam( type = "function" , function = { "name" : "get_current_weather" , "description" : "Get the current weather" , "parameters" : { "type" : "object" , "properties" : { "location" : { "type" : "string" , "description" : "The city and state, e.g. San Francisco, CA" , }, "format" : { "type" : "string" , "enum" : [ "celsius" , "fahrenheit" ], "description" : "The temperature unit to use." , }, }, "required" : [ "location" , "format" ], }, }, ) ] ​ Provider-Specific Custom Tools Some providers support unique tools that don’t fit the standard function schema. For these cases, you can add custom tools: Copy Ask AI from pipecat.adapters.schemas.tools_schema import AdapterType, ToolsSchema # Standard functions weather_function = FunctionSchema( name = "get_current_weather" , description = "Get the current weather" , properties = { "location" : { "type" : "string" }}, required = [ "location" ] ) # Custom Gemini search tool gemini_search_tool = { "web_search" : { "description" : "Search the web for information" } } # Create a tools schema with both standard and custom tools tools = ToolsSchema( standard_tools = [weather_function], custom_tools = { AdapterType. GEMINI : [gemini_search_tool] } ) See the provider-specific documentation for details on custom tools and their formats. ​ 2. Register Function Handlers Register handlers for your functions using one of these LLM service methods : register_function register_direct_function Which one you use depends on whether your function is a “direct” function . Non-Direct Function Direct Function Copy Ask AI from pipecat.services.llm_service import FunctionCallParams llm = OpenAILLMService( api_key = "your-api-key" ) # Main function handler - called to execute the function async def fetch_weather_from_api ( params : FunctionCallParams): # Fetch weather data from your API weather_data = { "conditions" : "sunny" , "temperature" : "75" } await params.result_callback(weather_data) # Register the function llm.register_function( "get_current_weather" , fetch_weather_from_api, ) ​ 3. Create the Pipeline Include your LLM service in your pipeline with the registered functions: Copy Ask AI # Initialize the LLM context with your function schemas context = OpenAILLMContext( messages = [{ "role" : "system" , "content" : "You are a helpful assistant." }], tools = tools ) # Create the context aggregator to collect the user and assistant context context_aggregator = llm.create_context_aggregator(context) # Create the pipeline pipeline = Pipeline([ transport.input(), # Input from the transport stt, # STT processing context_aggregator.user(), # User context aggregation llm, # LLM processing tts, # TTS processing transport.output(), # Output to the transport context_aggregator.assistant(), # Assistant context aggregation ]) ​ Function Handler Details ​ FunctionCallParams The FunctionCallParams object contains all the information needed for handling function calls: params : FunctionCallParams function_name : Name of the called function arguments : Arguments passed by the LLM tool_call_id : Unique identifier for the function call llm : Reference to the LLM service context : Current conversation context result_callback : Async function to return results ​ function_name str Name of the function being called ​ tool_call_id str Unique identifier for the function call ​ arguments Mapping[str, Any] Arguments passed by the LLM to the function ​ llm LLMService Reference to the LLM service that initiated the call ​ context OpenAILLMContext Current conversation context ​ result_callback FunctionCallResultCallback Async callback function to return results ​ Handler Structure Your function handler should: Receive necessary arguments, either: From params.arguments Directly From function arguments, if using direct functions Process data or call external services Return results via params.result_callback(result) Non-Direct Function Direct Function Copy Ask AI async def fetch_weather_from_api ( params : FunctionCallParams): try : # Extract arguments location = params.arguments.get( "location" ) format_type = params.arguments.get( "format" , "celsius" ) # Call external API api_result = await weather_api.get_weather(location, format_type) # Return formatted result await params.result_callback({ "location" : location, "temperature" : api_result[ "temp" ], "conditions" : api_result[ "conditions" ], "unit" : format_type }) except Exception as e: # Handle errors await params.result_callback({ "error" : f "Failed to get weather: { str (e) } " }) ​ Controlling Function Call Behavior (Advanced) When returning results from a function handler, you can control how the LLM processes those results using a FunctionCallResultProperties object passed to the result callback. It can be handy to skip a completion when you have back-to-back function calls. Note, if you skip a completion, you must manually trigger one from the context. ​ Properties ​ run_llm Optional[bool] Controls whether the LLM should generate a response after the function call: True : Run LLM after function call (default if no other function calls in progress) False : Don’t run LLM after function call None : Use default behavior ​ on_context_updated Optional[Callable[[], Awaitable[None]]] Optional callback that runs after the function result is added to the context ​ Example Usage Copy Ask AI from pipecat.frames.frames import FunctionCallResultProperties from pipecat.services.llm_service import FunctionCallParams async def fetch_weather_from_api ( params : FunctionCallParams): # Fetch weather data weather_data = { "conditions" : "sunny" , "temperature" : "75" } # Don't run LLM after this function call properties = FunctionCallResultProperties( run_llm = False ) await params.result_callback(weather_data, properties = properties) async def query_database ( params : FunctionCallParams): # Query database results = await db.query(params.arguments[ "query" ]) async def on_update (): await notify_system( "Database query complete" ) # Run LLM after function call and notify when context is updated properties = FunctionCallResultProperties( run_llm = True , on_context_updated = on_update ) await params.result_callback(results, properties = properties) ​ Next steps Check out the function calling examples to see a complete example for specific LLM providers. Refer to your LLM provider’s documentation to learn more about their function calling capabilities. Ending a Pipeline Muting User Input On this page Understanding Function Calling Implementation 1. Define Functions Using the Standard Schema (Recommended) Using Direct Functions (Shorthand) Using Provider-Specific Formats (Alternative) Provider-Specific Custom Tools 2. Register Function Handlers 3. Create the Pipeline Function Handler Details FunctionCallParams Handler Structure Controlling Function Call Behavior (Advanced) Properties Example Usage Next steps Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/fundamentals_recording-audio_520eaaf8.txt b/fundamentals_recording-audio_520eaaf8.txt
new file mode 100644
index 0000000000000000000000000000000000000000..ac389757564cbd7bf4e7f1954096a2d2a398e7cb
--- /dev/null
+++ b/fundamentals_recording-audio_520eaaf8.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/guides/fundamentals/recording-audio#chunked-recording
+Title: Recording Conversation Audio - Pipecat
+==================================================
+
+Recording Conversation Audio - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Fundamentals Recording Conversation Audio Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Guides Fundamentals Context Management Custom FrameProcessor Detecting Idle Users Ending a Pipeline Function Calling Muting User Input Recording Audio Recording Transcripts Features Gemini Multimodal Live Metrics Noise cancellation with Krisp OpenAI Audio Models and APIs Pipecat Flows Telephony Overview Dial-in: WebRTC (Daily) Dial-in: WebRTC (Twilio + Daily) Dial-in: Twilio (Media Streams) Dialout: WebRTC (Daily) Deploying your bot Overview Deployment pattern Example: Pipecat Cloud Example: Fly.io Example: Cerebrium Example: Modal ​ Overview Recording audio from conversations provides valuable data for analysis, debugging, and quality control. You have two options for how to record with Pipecat: ​ Option 1: Record using your transport service provider Record without writing custom code by using your transport provider’s recording capabilities. In addition to saving you development time, some providers offer unique recording capabilities. Refer to your service provider’s documentation to learn more. ​ Option 2: Create your own recording pipeline Pipecat’s AudioBufferProcessor makes it easy to capture high-quality audio recordings of both the user and bot during interactions. Opt for this approach if you want more control over your recording. This guide focuses on how to recording using the AudioBufferProcessor , including high-level guidance for how to set up post-processing jobs for longer recordings. ​ How the AudioBufferProcessor Works The AudioBufferProcessor captures audio by: Collecting audio frames from both the user (input) and bot (output) Emitting events with recorded audio data Providing options for composite or separate track recordings Add the processor to your pipeline after the transport.output() to capture both the user audio and the bot audio as it’s spoken. ​ Audio Recording Options The AudioBufferProcessor offers several configuration options: Composite recording : Combined audio from both user and bot Track-level recording : Separate audio files for user and bot Turn-based recording : Individual audio clips for each speaking turn Mono or stereo output : Single channel mixing or two-channel separation ​ Basic Implementation ​ Step 1: Create an Audio Buffer Processor Initialize the audio buffer processor with your desired configuration: Copy Ask AI from pipecat.processors.audio.audio_buffer_processor import AudioBufferProcessor # Create audio buffer processor with default settings audiobuffer = AudioBufferProcessor( num_channels = 1 , # 1 for mono, 2 for stereo (user left, bot right) enable_turn_audio = False , # Enable per-turn audio recording user_continuous_stream = True , # User has continuous audio stream ) ​ Step 2: Add to Your Pipeline Place the processor in your pipeline after all audio-producing components: Copy Ask AI pipeline = Pipeline( [ transport.input(), stt, context_aggregator.user(), llm, tts, transport.output(), audiobuffer, # Add after all audio components context_aggregator.assistant(), ] ) ​ Step 3: Start Recording Explicitly start recording when needed, typically when a session begins: Copy Ask AI @transport.event_handler ( "on_client_connected" ) async def on_client_connected ( transport , client ): logger.info( f "Client connected" ) # Important: Start recording explicitly await audiobuffer.start_recording() # Continue with session initialization... You must call start_recording() explicitly to begin capturing audio. The processor won’t record automatically when initialized. ​ Step 4: Handle Audio Data Register an event handler to process audio data: Copy Ask AI @audiobuffer.event_handler ( "on_audio_data" ) async def on_audio_data ( buffer , audio , sample_rate , num_channels ): # Save or process the composite audio timestamp = datetime.datetime.now().strftime( "%Y%m %d _%H%M%S" ) filename = f "recordings/conversation_ { timestamp } .wav" # Create the WAV file with wave.open(filename, "wb" ) as wf: wf.setnchannels(num_channels) wf.setsampwidth( 2 ) # 16-bit audio wf.setframerate(sample_rate) wf.writeframes(audio) logger.info( f "Saved recording to { filename } " ) If recording separate tracks, you can use the on_track_audio_data event handler to save user and bot audio separately. ​ Recording Longer Conversations For conversations that last a few minutes, it may be sufficient to just buffer the audio in memory. However, for longer sessions, storing audio in memory poses two challenges: Memory Usage : Long recordings can consume significant memory, leading to potential crashes or performance issues. Conversation Loss : If the application crashes or the connection drops, you may lose all recorded audio. Instead, consider using a chunked approach to record audio in manageable segments. This allows you to periodically save audio data to disk or upload it to cloud storage, reducing memory usage and ensuring data persistence. ​ Chunked Recording Set a reasonable buffer_size to trigger periodic uploads: Copy Ask AI # 30-second chunks (recommended for most use cases) SAMPLE_RATE = 24000 CHUNK_DURATION = 30 # seconds audiobuffer = AudioBufferProcessor( sample_rate = SAMPLE_RATE , buffer_size = SAMPLE_RATE * 2 * CHUNK_DURATION # 2 bytes per sample (16-bit) ) chunk_counter = 0 @audiobuffer.event_handler ( "on_track_audio_data" ) async def on_chunk_ready ( buffer , user_audio , bot_audio , sample_rate , num_channels ): global chunk_counter # Upload or save individual chunks await upload_audio_chunk( f "user_chunk_ { chunk_counter :03d} .wav" , user_audio, sample_rate, 1 ) await upload_audio_chunk( f "bot_chunk_ { chunk_counter :03d} .wav" , bot_audio, sample_rate, 1 ) chunk_counter += 1 ​ Multipart Upload Strategy For cloud storage, consider using multipart uploads to stream audio chunks: Conceptual Approach: Initialize multipart upload when recording starts Upload chunks as parts when buffers fill (every ~30 seconds) Complete multipart upload when recording ends Post-process to create final WAV file(s) Benefits: Memory efficient for long sessions Fault tolerant (no data loss if connection drops) Enables real-time processing and analysis Parallel upload of multiple tracks ​ Post-Processing Pipeline After uploading chunks, create final audio files using tools like FFmpeg: Concatenating Audio Files: Copy Ask AI # Method 1: Simple concatenation (same format) ffmpeg -i "concat:chunk_001.wav|chunk_002.wav|chunk_003.wav" -acodec copy final.wav # Method 2: Using file list (recommended for many chunks) # Create filelist.txt with format: # file 'chunk_001.wav' # file 'chunk_002.wav' # ... ffmpeg -f concat -safe 0 -i filelist.txt -c copy final_recording.wav Automation Considerations: Use sequence numbers in chunk filenames for proper ordering Include metadata (sample rate, channels, duration) with each chunk Implement retry logic for failed uploads Consider using cloud functions/lambdas for automatic post-processing ​ Next Steps Try the Audio Recording Example Explore a complete working example that demonstrates how to record and save both composite and track-level audio with Pipecat. AudioBufferProcessor Reference Read the complete API reference documentation for advanced configuration options and event handlers. Consider implementing audio recording in your application for quality assurance, training data collection, or creating conversation archives. The recorded audio can be stored locally, uploaded to cloud storage, or processed in real-time for further analysis. Muting User Input Recording Transcripts On this page Overview Option 1: Record using your transport service provider Option 2: Create your own recording pipeline How the AudioBufferProcessor Works Audio Recording Options Basic Implementation Step 1: Create an Audio Buffer Processor Step 2: Add to Your Pipeline Step 3: Start Recording Step 4: Handle Audio Data Recording Longer Conversations Chunked Recording Multipart Upload Strategy Post-Processing Pipeline Next Steps Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/fundamentals_recording-audio_861c2d4a.txt b/fundamentals_recording-audio_861c2d4a.txt
new file mode 100644
index 0000000000000000000000000000000000000000..eada0327d50d3674f24ee50dd70af7e3f7c4c928
--- /dev/null
+++ b/fundamentals_recording-audio_861c2d4a.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/guides/fundamentals/recording-audio#how-the-audiobufferprocessor-works
+Title: Recording Conversation Audio - Pipecat
+==================================================
+
+Recording Conversation Audio - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Fundamentals Recording Conversation Audio Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Guides Fundamentals Context Management Custom FrameProcessor Detecting Idle Users Ending a Pipeline Function Calling Muting User Input Recording Audio Recording Transcripts Features Gemini Multimodal Live Metrics Noise cancellation with Krisp OpenAI Audio Models and APIs Pipecat Flows Telephony Overview Dial-in: WebRTC (Daily) Dial-in: WebRTC (Twilio + Daily) Dial-in: Twilio (Media Streams) Dialout: WebRTC (Daily) Deploying your bot Overview Deployment pattern Example: Pipecat Cloud Example: Fly.io Example: Cerebrium Example: Modal ​ Overview Recording audio from conversations provides valuable data for analysis, debugging, and quality control. You have two options for how to record with Pipecat: ​ Option 1: Record using your transport service provider Record without writing custom code by using your transport provider’s recording capabilities. In addition to saving you development time, some providers offer unique recording capabilities. Refer to your service provider’s documentation to learn more. ​ Option 2: Create your own recording pipeline Pipecat’s AudioBufferProcessor makes it easy to capture high-quality audio recordings of both the user and bot during interactions. Opt for this approach if you want more control over your recording. This guide focuses on how to recording using the AudioBufferProcessor , including high-level guidance for how to set up post-processing jobs for longer recordings. ​ How the AudioBufferProcessor Works The AudioBufferProcessor captures audio by: Collecting audio frames from both the user (input) and bot (output) Emitting events with recorded audio data Providing options for composite or separate track recordings Add the processor to your pipeline after the transport.output() to capture both the user audio and the bot audio as it’s spoken. ​ Audio Recording Options The AudioBufferProcessor offers several configuration options: Composite recording : Combined audio from both user and bot Track-level recording : Separate audio files for user and bot Turn-based recording : Individual audio clips for each speaking turn Mono or stereo output : Single channel mixing or two-channel separation ​ Basic Implementation ​ Step 1: Create an Audio Buffer Processor Initialize the audio buffer processor with your desired configuration: Copy Ask AI from pipecat.processors.audio.audio_buffer_processor import AudioBufferProcessor # Create audio buffer processor with default settings audiobuffer = AudioBufferProcessor( num_channels = 1 , # 1 for mono, 2 for stereo (user left, bot right) enable_turn_audio = False , # Enable per-turn audio recording user_continuous_stream = True , # User has continuous audio stream ) ​ Step 2: Add to Your Pipeline Place the processor in your pipeline after all audio-producing components: Copy Ask AI pipeline = Pipeline( [ transport.input(), stt, context_aggregator.user(), llm, tts, transport.output(), audiobuffer, # Add after all audio components context_aggregator.assistant(), ] ) ​ Step 3: Start Recording Explicitly start recording when needed, typically when a session begins: Copy Ask AI @transport.event_handler ( "on_client_connected" ) async def on_client_connected ( transport , client ): logger.info( f "Client connected" ) # Important: Start recording explicitly await audiobuffer.start_recording() # Continue with session initialization... You must call start_recording() explicitly to begin capturing audio. The processor won’t record automatically when initialized. ​ Step 4: Handle Audio Data Register an event handler to process audio data: Copy Ask AI @audiobuffer.event_handler ( "on_audio_data" ) async def on_audio_data ( buffer , audio , sample_rate , num_channels ): # Save or process the composite audio timestamp = datetime.datetime.now().strftime( "%Y%m %d _%H%M%S" ) filename = f "recordings/conversation_ { timestamp } .wav" # Create the WAV file with wave.open(filename, "wb" ) as wf: wf.setnchannels(num_channels) wf.setsampwidth( 2 ) # 16-bit audio wf.setframerate(sample_rate) wf.writeframes(audio) logger.info( f "Saved recording to { filename } " ) If recording separate tracks, you can use the on_track_audio_data event handler to save user and bot audio separately. ​ Recording Longer Conversations For conversations that last a few minutes, it may be sufficient to just buffer the audio in memory. However, for longer sessions, storing audio in memory poses two challenges: Memory Usage : Long recordings can consume significant memory, leading to potential crashes or performance issues. Conversation Loss : If the application crashes or the connection drops, you may lose all recorded audio. Instead, consider using a chunked approach to record audio in manageable segments. This allows you to periodically save audio data to disk or upload it to cloud storage, reducing memory usage and ensuring data persistence. ​ Chunked Recording Set a reasonable buffer_size to trigger periodic uploads: Copy Ask AI # 30-second chunks (recommended for most use cases) SAMPLE_RATE = 24000 CHUNK_DURATION = 30 # seconds audiobuffer = AudioBufferProcessor( sample_rate = SAMPLE_RATE , buffer_size = SAMPLE_RATE * 2 * CHUNK_DURATION # 2 bytes per sample (16-bit) ) chunk_counter = 0 @audiobuffer.event_handler ( "on_track_audio_data" ) async def on_chunk_ready ( buffer , user_audio , bot_audio , sample_rate , num_channels ): global chunk_counter # Upload or save individual chunks await upload_audio_chunk( f "user_chunk_ { chunk_counter :03d} .wav" , user_audio, sample_rate, 1 ) await upload_audio_chunk( f "bot_chunk_ { chunk_counter :03d} .wav" , bot_audio, sample_rate, 1 ) chunk_counter += 1 ​ Multipart Upload Strategy For cloud storage, consider using multipart uploads to stream audio chunks: Conceptual Approach: Initialize multipart upload when recording starts Upload chunks as parts when buffers fill (every ~30 seconds) Complete multipart upload when recording ends Post-process to create final WAV file(s) Benefits: Memory efficient for long sessions Fault tolerant (no data loss if connection drops) Enables real-time processing and analysis Parallel upload of multiple tracks ​ Post-Processing Pipeline After uploading chunks, create final audio files using tools like FFmpeg: Concatenating Audio Files: Copy Ask AI # Method 1: Simple concatenation (same format) ffmpeg -i "concat:chunk_001.wav|chunk_002.wav|chunk_003.wav" -acodec copy final.wav # Method 2: Using file list (recommended for many chunks) # Create filelist.txt with format: # file 'chunk_001.wav' # file 'chunk_002.wav' # ... ffmpeg -f concat -safe 0 -i filelist.txt -c copy final_recording.wav Automation Considerations: Use sequence numbers in chunk filenames for proper ordering Include metadata (sample rate, channels, duration) with each chunk Implement retry logic for failed uploads Consider using cloud functions/lambdas for automatic post-processing ​ Next Steps Try the Audio Recording Example Explore a complete working example that demonstrates how to record and save both composite and track-level audio with Pipecat. AudioBufferProcessor Reference Read the complete API reference documentation for advanced configuration options and event handlers. Consider implementing audio recording in your application for quality assurance, training data collection, or creating conversation archives. The recorded audio can be stored locally, uploaded to cloud storage, or processed in real-time for further analysis. Muting User Input Recording Transcripts On this page Overview Option 1: Record using your transport service provider Option 2: Create your own recording pipeline How the AudioBufferProcessor Works Audio Recording Options Basic Implementation Step 1: Create an Audio Buffer Processor Step 2: Add to Your Pipeline Step 3: Start Recording Step 4: Handle Audio Data Recording Longer Conversations Chunked Recording Multipart Upload Strategy Post-Processing Pipeline Next Steps Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/fundamentals_recording-audio_aa4caeeb.txt b/fundamentals_recording-audio_aa4caeeb.txt
new file mode 100644
index 0000000000000000000000000000000000000000..d528011aed758fa03acb948d8e5105e76806473d
--- /dev/null
+++ b/fundamentals_recording-audio_aa4caeeb.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/guides/fundamentals/recording-audio
+Title: Recording Conversation Audio - Pipecat
+==================================================
+
+Recording Conversation Audio - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Fundamentals Recording Conversation Audio Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Guides Fundamentals Context Management Custom FrameProcessor Detecting Idle Users Ending a Pipeline Function Calling Muting User Input Recording Audio Recording Transcripts Features Gemini Multimodal Live Metrics Noise cancellation with Krisp OpenAI Audio Models and APIs Pipecat Flows Telephony Overview Dial-in: WebRTC (Daily) Dial-in: WebRTC (Twilio + Daily) Dial-in: Twilio (Media Streams) Dialout: WebRTC (Daily) Deploying your bot Overview Deployment pattern Example: Pipecat Cloud Example: Fly.io Example: Cerebrium Example: Modal ​ Overview Recording audio from conversations provides valuable data for analysis, debugging, and quality control. You have two options for how to record with Pipecat: ​ Option 1: Record using your transport service provider Record without writing custom code by using your transport provider’s recording capabilities. In addition to saving you development time, some providers offer unique recording capabilities. Refer to your service provider’s documentation to learn more. ​ Option 2: Create your own recording pipeline Pipecat’s AudioBufferProcessor makes it easy to capture high-quality audio recordings of both the user and bot during interactions. Opt for this approach if you want more control over your recording. This guide focuses on how to recording using the AudioBufferProcessor , including high-level guidance for how to set up post-processing jobs for longer recordings. ​ How the AudioBufferProcessor Works The AudioBufferProcessor captures audio by: Collecting audio frames from both the user (input) and bot (output) Emitting events with recorded audio data Providing options for composite or separate track recordings Add the processor to your pipeline after the transport.output() to capture both the user audio and the bot audio as it’s spoken. ​ Audio Recording Options The AudioBufferProcessor offers several configuration options: Composite recording : Combined audio from both user and bot Track-level recording : Separate audio files for user and bot Turn-based recording : Individual audio clips for each speaking turn Mono or stereo output : Single channel mixing or two-channel separation ​ Basic Implementation ​ Step 1: Create an Audio Buffer Processor Initialize the audio buffer processor with your desired configuration: Copy Ask AI from pipecat.processors.audio.audio_buffer_processor import AudioBufferProcessor # Create audio buffer processor with default settings audiobuffer = AudioBufferProcessor( num_channels = 1 , # 1 for mono, 2 for stereo (user left, bot right) enable_turn_audio = False , # Enable per-turn audio recording user_continuous_stream = True , # User has continuous audio stream ) ​ Step 2: Add to Your Pipeline Place the processor in your pipeline after all audio-producing components: Copy Ask AI pipeline = Pipeline( [ transport.input(), stt, context_aggregator.user(), llm, tts, transport.output(), audiobuffer, # Add after all audio components context_aggregator.assistant(), ] ) ​ Step 3: Start Recording Explicitly start recording when needed, typically when a session begins: Copy Ask AI @transport.event_handler ( "on_client_connected" ) async def on_client_connected ( transport , client ): logger.info( f "Client connected" ) # Important: Start recording explicitly await audiobuffer.start_recording() # Continue with session initialization... You must call start_recording() explicitly to begin capturing audio. The processor won’t record automatically when initialized. ​ Step 4: Handle Audio Data Register an event handler to process audio data: Copy Ask AI @audiobuffer.event_handler ( "on_audio_data" ) async def on_audio_data ( buffer , audio , sample_rate , num_channels ): # Save or process the composite audio timestamp = datetime.datetime.now().strftime( "%Y%m %d _%H%M%S" ) filename = f "recordings/conversation_ { timestamp } .wav" # Create the WAV file with wave.open(filename, "wb" ) as wf: wf.setnchannels(num_channels) wf.setsampwidth( 2 ) # 16-bit audio wf.setframerate(sample_rate) wf.writeframes(audio) logger.info( f "Saved recording to { filename } " ) If recording separate tracks, you can use the on_track_audio_data event handler to save user and bot audio separately. ​ Recording Longer Conversations For conversations that last a few minutes, it may be sufficient to just buffer the audio in memory. However, for longer sessions, storing audio in memory poses two challenges: Memory Usage : Long recordings can consume significant memory, leading to potential crashes or performance issues. Conversation Loss : If the application crashes or the connection drops, you may lose all recorded audio. Instead, consider using a chunked approach to record audio in manageable segments. This allows you to periodically save audio data to disk or upload it to cloud storage, reducing memory usage and ensuring data persistence. ​ Chunked Recording Set a reasonable buffer_size to trigger periodic uploads: Copy Ask AI # 30-second chunks (recommended for most use cases) SAMPLE_RATE = 24000 CHUNK_DURATION = 30 # seconds audiobuffer = AudioBufferProcessor( sample_rate = SAMPLE_RATE , buffer_size = SAMPLE_RATE * 2 * CHUNK_DURATION # 2 bytes per sample (16-bit) ) chunk_counter = 0 @audiobuffer.event_handler ( "on_track_audio_data" ) async def on_chunk_ready ( buffer , user_audio , bot_audio , sample_rate , num_channels ): global chunk_counter # Upload or save individual chunks await upload_audio_chunk( f "user_chunk_ { chunk_counter :03d} .wav" , user_audio, sample_rate, 1 ) await upload_audio_chunk( f "bot_chunk_ { chunk_counter :03d} .wav" , bot_audio, sample_rate, 1 ) chunk_counter += 1 ​ Multipart Upload Strategy For cloud storage, consider using multipart uploads to stream audio chunks: Conceptual Approach: Initialize multipart upload when recording starts Upload chunks as parts when buffers fill (every ~30 seconds) Complete multipart upload when recording ends Post-process to create final WAV file(s) Benefits: Memory efficient for long sessions Fault tolerant (no data loss if connection drops) Enables real-time processing and analysis Parallel upload of multiple tracks ​ Post-Processing Pipeline After uploading chunks, create final audio files using tools like FFmpeg: Concatenating Audio Files: Copy Ask AI # Method 1: Simple concatenation (same format) ffmpeg -i "concat:chunk_001.wav|chunk_002.wav|chunk_003.wav" -acodec copy final.wav # Method 2: Using file list (recommended for many chunks) # Create filelist.txt with format: # file 'chunk_001.wav' # file 'chunk_002.wav' # ... ffmpeg -f concat -safe 0 -i filelist.txt -c copy final_recording.wav Automation Considerations: Use sequence numbers in chunk filenames for proper ordering Include metadata (sample rate, channels, duration) with each chunk Implement retry logic for failed uploads Consider using cloud functions/lambdas for automatic post-processing ​ Next Steps Try the Audio Recording Example Explore a complete working example that demonstrates how to record and save both composite and track-level audio with Pipecat. AudioBufferProcessor Reference Read the complete API reference documentation for advanced configuration options and event handlers. Consider implementing audio recording in your application for quality assurance, training data collection, or creating conversation archives. The recorded audio can be stored locally, uploaded to cloud storage, or processed in real-time for further analysis. Muting User Input Recording Transcripts On this page Overview Option 1: Record using your transport service provider Option 2: Create your own recording pipeline How the AudioBufferProcessor Works Audio Recording Options Basic Implementation Step 1: Create an Audio Buffer Processor Step 2: Add to Your Pipeline Step 3: Start Recording Step 4: Handle Audio Data Recording Longer Conversations Chunked Recording Multipart Upload Strategy Post-Processing Pipeline Next Steps Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/fundamentals_recording-audio_c554c62e.txt b/fundamentals_recording-audio_c554c62e.txt
new file mode 100644
index 0000000000000000000000000000000000000000..f910845956c549f1dd4065403adafcbf03b1e231
--- /dev/null
+++ b/fundamentals_recording-audio_c554c62e.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/guides/fundamentals/recording-audio#recording-longer-conversations
+Title: Recording Conversation Audio - Pipecat
+==================================================
+
+Recording Conversation Audio - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Fundamentals Recording Conversation Audio Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Guides Fundamentals Context Management Custom FrameProcessor Detecting Idle Users Ending a Pipeline Function Calling Muting User Input Recording Audio Recording Transcripts Features Gemini Multimodal Live Metrics Noise cancellation with Krisp OpenAI Audio Models and APIs Pipecat Flows Telephony Overview Dial-in: WebRTC (Daily) Dial-in: WebRTC (Twilio + Daily) Dial-in: Twilio (Media Streams) Dialout: WebRTC (Daily) Deploying your bot Overview Deployment pattern Example: Pipecat Cloud Example: Fly.io Example: Cerebrium Example: Modal ​ Overview Recording audio from conversations provides valuable data for analysis, debugging, and quality control. You have two options for how to record with Pipecat: ​ Option 1: Record using your transport service provider Record without writing custom code by using your transport provider’s recording capabilities. In addition to saving you development time, some providers offer unique recording capabilities. Refer to your service provider’s documentation to learn more. ​ Option 2: Create your own recording pipeline Pipecat’s AudioBufferProcessor makes it easy to capture high-quality audio recordings of both the user and bot during interactions. Opt for this approach if you want more control over your recording. This guide focuses on how to recording using the AudioBufferProcessor , including high-level guidance for how to set up post-processing jobs for longer recordings. ​ How the AudioBufferProcessor Works The AudioBufferProcessor captures audio by: Collecting audio frames from both the user (input) and bot (output) Emitting events with recorded audio data Providing options for composite or separate track recordings Add the processor to your pipeline after the transport.output() to capture both the user audio and the bot audio as it’s spoken. ​ Audio Recording Options The AudioBufferProcessor offers several configuration options: Composite recording : Combined audio from both user and bot Track-level recording : Separate audio files for user and bot Turn-based recording : Individual audio clips for each speaking turn Mono or stereo output : Single channel mixing or two-channel separation ​ Basic Implementation ​ Step 1: Create an Audio Buffer Processor Initialize the audio buffer processor with your desired configuration: Copy Ask AI from pipecat.processors.audio.audio_buffer_processor import AudioBufferProcessor # Create audio buffer processor with default settings audiobuffer = AudioBufferProcessor( num_channels = 1 , # 1 for mono, 2 for stereo (user left, bot right) enable_turn_audio = False , # Enable per-turn audio recording user_continuous_stream = True , # User has continuous audio stream ) ​ Step 2: Add to Your Pipeline Place the processor in your pipeline after all audio-producing components: Copy Ask AI pipeline = Pipeline( [ transport.input(), stt, context_aggregator.user(), llm, tts, transport.output(), audiobuffer, # Add after all audio components context_aggregator.assistant(), ] ) ​ Step 3: Start Recording Explicitly start recording when needed, typically when a session begins: Copy Ask AI @transport.event_handler ( "on_client_connected" ) async def on_client_connected ( transport , client ): logger.info( f "Client connected" ) # Important: Start recording explicitly await audiobuffer.start_recording() # Continue with session initialization... You must call start_recording() explicitly to begin capturing audio. The processor won’t record automatically when initialized. ​ Step 4: Handle Audio Data Register an event handler to process audio data: Copy Ask AI @audiobuffer.event_handler ( "on_audio_data" ) async def on_audio_data ( buffer , audio , sample_rate , num_channels ): # Save or process the composite audio timestamp = datetime.datetime.now().strftime( "%Y%m %d _%H%M%S" ) filename = f "recordings/conversation_ { timestamp } .wav" # Create the WAV file with wave.open(filename, "wb" ) as wf: wf.setnchannels(num_channels) wf.setsampwidth( 2 ) # 16-bit audio wf.setframerate(sample_rate) wf.writeframes(audio) logger.info( f "Saved recording to { filename } " ) If recording separate tracks, you can use the on_track_audio_data event handler to save user and bot audio separately. ​ Recording Longer Conversations For conversations that last a few minutes, it may be sufficient to just buffer the audio in memory. However, for longer sessions, storing audio in memory poses two challenges: Memory Usage : Long recordings can consume significant memory, leading to potential crashes or performance issues. Conversation Loss : If the application crashes or the connection drops, you may lose all recorded audio. Instead, consider using a chunked approach to record audio in manageable segments. This allows you to periodically save audio data to disk or upload it to cloud storage, reducing memory usage and ensuring data persistence. ​ Chunked Recording Set a reasonable buffer_size to trigger periodic uploads: Copy Ask AI # 30-second chunks (recommended for most use cases) SAMPLE_RATE = 24000 CHUNK_DURATION = 30 # seconds audiobuffer = AudioBufferProcessor( sample_rate = SAMPLE_RATE , buffer_size = SAMPLE_RATE * 2 * CHUNK_DURATION # 2 bytes per sample (16-bit) ) chunk_counter = 0 @audiobuffer.event_handler ( "on_track_audio_data" ) async def on_chunk_ready ( buffer , user_audio , bot_audio , sample_rate , num_channels ): global chunk_counter # Upload or save individual chunks await upload_audio_chunk( f "user_chunk_ { chunk_counter :03d} .wav" , user_audio, sample_rate, 1 ) await upload_audio_chunk( f "bot_chunk_ { chunk_counter :03d} .wav" , bot_audio, sample_rate, 1 ) chunk_counter += 1 ​ Multipart Upload Strategy For cloud storage, consider using multipart uploads to stream audio chunks: Conceptual Approach: Initialize multipart upload when recording starts Upload chunks as parts when buffers fill (every ~30 seconds) Complete multipart upload when recording ends Post-process to create final WAV file(s) Benefits: Memory efficient for long sessions Fault tolerant (no data loss if connection drops) Enables real-time processing and analysis Parallel upload of multiple tracks ​ Post-Processing Pipeline After uploading chunks, create final audio files using tools like FFmpeg: Concatenating Audio Files: Copy Ask AI # Method 1: Simple concatenation (same format) ffmpeg -i "concat:chunk_001.wav|chunk_002.wav|chunk_003.wav" -acodec copy final.wav # Method 2: Using file list (recommended for many chunks) # Create filelist.txt with format: # file 'chunk_001.wav' # file 'chunk_002.wav' # ... ffmpeg -f concat -safe 0 -i filelist.txt -c copy final_recording.wav Automation Considerations: Use sequence numbers in chunk filenames for proper ordering Include metadata (sample rate, channels, duration) with each chunk Implement retry logic for failed uploads Consider using cloud functions/lambdas for automatic post-processing ​ Next Steps Try the Audio Recording Example Explore a complete working example that demonstrates how to record and save both composite and track-level audio with Pipecat. AudioBufferProcessor Reference Read the complete API reference documentation for advanced configuration options and event handlers. Consider implementing audio recording in your application for quality assurance, training data collection, or creating conversation archives. The recorded audio can be stored locally, uploaded to cloud storage, or processed in real-time for further analysis. Muting User Input Recording Transcripts On this page Overview Option 1: Record using your transport service provider Option 2: Create your own recording pipeline How the AudioBufferProcessor Works Audio Recording Options Basic Implementation Step 1: Create an Audio Buffer Processor Step 2: Add to Your Pipeline Step 3: Start Recording Step 4: Handle Audio Data Recording Longer Conversations Chunked Recording Multipart Upload Strategy Post-Processing Pipeline Next Steps Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/fundamentals_recording-transcripts_0b56b420.txt b/fundamentals_recording-transcripts_0b56b420.txt
new file mode 100644
index 0000000000000000000000000000000000000000..aecc21307579f9bd2094c5773dced3d65c5903ed
--- /dev/null
+++ b/fundamentals_recording-transcripts_0b56b420.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/guides/fundamentals/recording-transcripts#step-1%3A-create-a-transcript-processor
+Title: Recording Conversation Transcripts - Pipecat
+==================================================
+
+Recording Conversation Transcripts - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Fundamentals Recording Conversation Transcripts Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Guides Fundamentals Context Management Custom FrameProcessor Detecting Idle Users Ending a Pipeline Function Calling Muting User Input Recording Audio Recording Transcripts Features Gemini Multimodal Live Metrics Noise cancellation with Krisp OpenAI Audio Models and APIs Pipecat Flows Telephony Overview Dial-in: WebRTC (Daily) Dial-in: WebRTC (Twilio + Daily) Dial-in: Twilio (Media Streams) Dialout: WebRTC (Daily) Deploying your bot Overview Deployment pattern Example: Pipecat Cloud Example: Fly.io Example: Cerebrium Example: Modal ​ Overview Recording transcripts of conversations between users and your bot is useful for debugging, analysis, and creating a record of interactions. Pipecat’s TranscriptProcessor makes it easy to collect both user and bot messages as they occur. ​ How It Works The TranscriptProcessor collects transcripts by: Capturing what the user says (from TranscriptionFrame s) Capturing what the bot says (from TTSTextFrame s) Emitting events with transcript updates in real-time Allowing you to handle these events with custom logic The TranscriptProcessor provides two separate processors: one for user speech and one for assistant speech. Both emit the same event type when new transcript content is available. ​ Basic Implementation ​ Step 1: Create a Transcript Processor First, initialize the transcript processor: Copy Ask AI from pipecat.processors.transcript_processor import TranscriptProcessor # Create a single transcript processor instance transcript = TranscriptProcessor() ​ Step 2: Add to Your Pipeline Place the processors in your pipeline at the appropriate positions: Copy Ask AI pipeline = Pipeline( [ transport.input(), stt, # Speech-to-text transcript.user(), # Captures user transcripts context_aggregator.user(), llm, tts, # Text-to-speech transport.output(), transcript.assistant(), # Captures assistant transcripts context_aggregator.assistant(), ] ) Place transcript.user() after the STT processor and transcript.assistant() after transport.output() to ensure accurate transcript collection. ​ Step 3: Handle Transcript Updates Register an event handler to process transcript updates: Copy Ask AI @transcript.event_handler ( "on_transcript_update" ) async def handle_transcript_update ( processor , frame ): # Each message contains role (user/assistant), content, and timestamp for message in frame.messages: print ( f "[ { message.timestamp } ] { message.role } : { message.content } " ) In addition to console logging, you can save transcripts to a database or file for later analysis. ​ Next Steps Try the Transcript Example Explore a complete working example that demonstrates how to collect and save conversation transcripts with Pipecat. TranscriptProcessor Reference Read the complete API reference documentation for advanced configuration options and event handlers. Consider implementing transcript recording in your application for debugging during development and preserving important conversations in production. The transcript data can also be useful for analyzing conversation patterns and improving your bot’s responses over time. Recording Audio Gemini Multimodal Live On this page Overview How It Works Basic Implementation Step 1: Create a Transcript Processor Step 2: Add to Your Pipeline Step 3: Handle Transcript Updates Next Steps Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/fundamentals_recording-transcripts_3c26f260.txt b/fundamentals_recording-transcripts_3c26f260.txt
new file mode 100644
index 0000000000000000000000000000000000000000..138e3cc7ef55cb9865fd12d263cc09cf31754109
--- /dev/null
+++ b/fundamentals_recording-transcripts_3c26f260.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/guides/fundamentals/recording-transcripts#how-it-works
+Title: Recording Conversation Transcripts - Pipecat
+==================================================
+
+Recording Conversation Transcripts - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Fundamentals Recording Conversation Transcripts Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Guides Fundamentals Context Management Custom FrameProcessor Detecting Idle Users Ending a Pipeline Function Calling Muting User Input Recording Audio Recording Transcripts Features Gemini Multimodal Live Metrics Noise cancellation with Krisp OpenAI Audio Models and APIs Pipecat Flows Telephony Overview Dial-in: WebRTC (Daily) Dial-in: WebRTC (Twilio + Daily) Dial-in: Twilio (Media Streams) Dialout: WebRTC (Daily) Deploying your bot Overview Deployment pattern Example: Pipecat Cloud Example: Fly.io Example: Cerebrium Example: Modal ​ Overview Recording transcripts of conversations between users and your bot is useful for debugging, analysis, and creating a record of interactions. Pipecat’s TranscriptProcessor makes it easy to collect both user and bot messages as they occur. ​ How It Works The TranscriptProcessor collects transcripts by: Capturing what the user says (from TranscriptionFrame s) Capturing what the bot says (from TTSTextFrame s) Emitting events with transcript updates in real-time Allowing you to handle these events with custom logic The TranscriptProcessor provides two separate processors: one for user speech and one for assistant speech. Both emit the same event type when new transcript content is available. ​ Basic Implementation ​ Step 1: Create a Transcript Processor First, initialize the transcript processor: Copy Ask AI from pipecat.processors.transcript_processor import TranscriptProcessor # Create a single transcript processor instance transcript = TranscriptProcessor() ​ Step 2: Add to Your Pipeline Place the processors in your pipeline at the appropriate positions: Copy Ask AI pipeline = Pipeline( [ transport.input(), stt, # Speech-to-text transcript.user(), # Captures user transcripts context_aggregator.user(), llm, tts, # Text-to-speech transport.output(), transcript.assistant(), # Captures assistant transcripts context_aggregator.assistant(), ] ) Place transcript.user() after the STT processor and transcript.assistant() after transport.output() to ensure accurate transcript collection. ​ Step 3: Handle Transcript Updates Register an event handler to process transcript updates: Copy Ask AI @transcript.event_handler ( "on_transcript_update" ) async def handle_transcript_update ( processor , frame ): # Each message contains role (user/assistant), content, and timestamp for message in frame.messages: print ( f "[ { message.timestamp } ] { message.role } : { message.content } " ) In addition to console logging, you can save transcripts to a database or file for later analysis. ​ Next Steps Try the Transcript Example Explore a complete working example that demonstrates how to collect and save conversation transcripts with Pipecat. TranscriptProcessor Reference Read the complete API reference documentation for advanced configuration options and event handlers. Consider implementing transcript recording in your application for debugging during development and preserving important conversations in production. The transcript data can also be useful for analyzing conversation patterns and improving your bot’s responses over time. Recording Audio Gemini Multimodal Live On this page Overview How It Works Basic Implementation Step 1: Create a Transcript Processor Step 2: Add to Your Pipeline Step 3: Handle Transcript Updates Next Steps Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/fundamentals_recording-transcripts_aa938e7c.txt b/fundamentals_recording-transcripts_aa938e7c.txt
new file mode 100644
index 0000000000000000000000000000000000000000..6ddbb9ea0a522d5baf9777929d270ab413aa3911
--- /dev/null
+++ b/fundamentals_recording-transcripts_aa938e7c.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/guides/fundamentals/recording-transcripts#step-3%3A-handle-transcript-updates
+Title: Recording Conversation Transcripts - Pipecat
+==================================================
+
+Recording Conversation Transcripts - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Fundamentals Recording Conversation Transcripts Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Guides Fundamentals Context Management Custom FrameProcessor Detecting Idle Users Ending a Pipeline Function Calling Muting User Input Recording Audio Recording Transcripts Features Gemini Multimodal Live Metrics Noise cancellation with Krisp OpenAI Audio Models and APIs Pipecat Flows Telephony Overview Dial-in: WebRTC (Daily) Dial-in: WebRTC (Twilio + Daily) Dial-in: Twilio (Media Streams) Dialout: WebRTC (Daily) Deploying your bot Overview Deployment pattern Example: Pipecat Cloud Example: Fly.io Example: Cerebrium Example: Modal ​ Overview Recording transcripts of conversations between users and your bot is useful for debugging, analysis, and creating a record of interactions. Pipecat’s TranscriptProcessor makes it easy to collect both user and bot messages as they occur. ​ How It Works The TranscriptProcessor collects transcripts by: Capturing what the user says (from TranscriptionFrame s) Capturing what the bot says (from TTSTextFrame s) Emitting events with transcript updates in real-time Allowing you to handle these events with custom logic The TranscriptProcessor provides two separate processors: one for user speech and one for assistant speech. Both emit the same event type when new transcript content is available. ​ Basic Implementation ​ Step 1: Create a Transcript Processor First, initialize the transcript processor: Copy Ask AI from pipecat.processors.transcript_processor import TranscriptProcessor # Create a single transcript processor instance transcript = TranscriptProcessor() ​ Step 2: Add to Your Pipeline Place the processors in your pipeline at the appropriate positions: Copy Ask AI pipeline = Pipeline( [ transport.input(), stt, # Speech-to-text transcript.user(), # Captures user transcripts context_aggregator.user(), llm, tts, # Text-to-speech transport.output(), transcript.assistant(), # Captures assistant transcripts context_aggregator.assistant(), ] ) Place transcript.user() after the STT processor and transcript.assistant() after transport.output() to ensure accurate transcript collection. ​ Step 3: Handle Transcript Updates Register an event handler to process transcript updates: Copy Ask AI @transcript.event_handler ( "on_transcript_update" ) async def handle_transcript_update ( processor , frame ): # Each message contains role (user/assistant), content, and timestamp for message in frame.messages: print ( f "[ { message.timestamp } ] { message.role } : { message.content } " ) In addition to console logging, you can save transcripts to a database or file for later analysis. ​ Next Steps Try the Transcript Example Explore a complete working example that demonstrates how to collect and save conversation transcripts with Pipecat. TranscriptProcessor Reference Read the complete API reference documentation for advanced configuration options and event handlers. Consider implementing transcript recording in your application for debugging during development and preserving important conversations in production. The transcript data can also be useful for analyzing conversation patterns and improving your bot’s responses over time. Recording Audio Gemini Multimodal Live On this page Overview How It Works Basic Implementation Step 1: Create a Transcript Processor Step 2: Add to Your Pipeline Step 3: Handle Transcript Updates Next Steps Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/fundamentals_recording-transcripts_d6fc581e.txt b/fundamentals_recording-transcripts_d6fc581e.txt
new file mode 100644
index 0000000000000000000000000000000000000000..929e80a3757e9dd99102e8fa1a6ebc7845f3ceb4
--- /dev/null
+++ b/fundamentals_recording-transcripts_d6fc581e.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/guides/fundamentals/recording-transcripts
+Title: Recording Conversation Transcripts - Pipecat
+==================================================
+
+Recording Conversation Transcripts - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Fundamentals Recording Conversation Transcripts Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Guides Fundamentals Context Management Custom FrameProcessor Detecting Idle Users Ending a Pipeline Function Calling Muting User Input Recording Audio Recording Transcripts Features Gemini Multimodal Live Metrics Noise cancellation with Krisp OpenAI Audio Models and APIs Pipecat Flows Telephony Overview Dial-in: WebRTC (Daily) Dial-in: WebRTC (Twilio + Daily) Dial-in: Twilio (Media Streams) Dialout: WebRTC (Daily) Deploying your bot Overview Deployment pattern Example: Pipecat Cloud Example: Fly.io Example: Cerebrium Example: Modal ​ Overview Recording transcripts of conversations between users and your bot is useful for debugging, analysis, and creating a record of interactions. Pipecat’s TranscriptProcessor makes it easy to collect both user and bot messages as they occur. ​ How It Works The TranscriptProcessor collects transcripts by: Capturing what the user says (from TranscriptionFrame s) Capturing what the bot says (from TTSTextFrame s) Emitting events with transcript updates in real-time Allowing you to handle these events with custom logic The TranscriptProcessor provides two separate processors: one for user speech and one for assistant speech. Both emit the same event type when new transcript content is available. ​ Basic Implementation ​ Step 1: Create a Transcript Processor First, initialize the transcript processor: Copy Ask AI from pipecat.processors.transcript_processor import TranscriptProcessor # Create a single transcript processor instance transcript = TranscriptProcessor() ​ Step 2: Add to Your Pipeline Place the processors in your pipeline at the appropriate positions: Copy Ask AI pipeline = Pipeline( [ transport.input(), stt, # Speech-to-text transcript.user(), # Captures user transcripts context_aggregator.user(), llm, tts, # Text-to-speech transport.output(), transcript.assistant(), # Captures assistant transcripts context_aggregator.assistant(), ] ) Place transcript.user() after the STT processor and transcript.assistant() after transport.output() to ensure accurate transcript collection. ​ Step 3: Handle Transcript Updates Register an event handler to process transcript updates: Copy Ask AI @transcript.event_handler ( "on_transcript_update" ) async def handle_transcript_update ( processor , frame ): # Each message contains role (user/assistant), content, and timestamp for message in frame.messages: print ( f "[ { message.timestamp } ] { message.role } : { message.content } " ) In addition to console logging, you can save transcripts to a database or file for later analysis. ​ Next Steps Try the Transcript Example Explore a complete working example that demonstrates how to collect and save conversation transcripts with Pipecat. TranscriptProcessor Reference Read the complete API reference documentation for advanced configuration options and event handlers. Consider implementing transcript recording in your application for debugging during development and preserving important conversations in production. The transcript data can also be useful for analyzing conversation patterns and improving your bot’s responses over time. Recording Audio Gemini Multimodal Live On this page Overview How It Works Basic Implementation Step 1: Create a Transcript Processor Step 2: Add to Your Pipeline Step 3: Handle Transcript Updates Next Steps Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/fundamentals_user-input-muting_53db6cd0.txt b/fundamentals_user-input-muting_53db6cd0.txt
new file mode 100644
index 0000000000000000000000000000000000000000..407e3cfb0f5b9fdc704fc4c81e84a902e1f5570f
--- /dev/null
+++ b/fundamentals_user-input-muting_53db6cd0.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/guides/fundamentals/user-input-muting#mute-strategies
+Title: User Input Muting with STTMuteFilter - Pipecat
+==================================================
+
+User Input Muting with STTMuteFilter - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Fundamentals User Input Muting with STTMuteFilter Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Guides Fundamentals Context Management Custom FrameProcessor Detecting Idle Users Ending a Pipeline Function Calling Muting User Input Recording Audio Recording Transcripts Features Gemini Multimodal Live Metrics Noise cancellation with Krisp OpenAI Audio Models and APIs Pipecat Flows Telephony Overview Dial-in: WebRTC (Daily) Dial-in: WebRTC (Twilio + Daily) Dial-in: Twilio (Media Streams) Dialout: WebRTC (Daily) Deploying your bot Overview Deployment pattern Example: Pipecat Cloud Example: Fly.io Example: Cerebrium Example: Modal ​ Overview In conversational applications, there are moments when you don’t want to process user speech, such as during bot introductions or while executing function calls. Pipecat’s STTMuteFilter lets you selectively “mute” user input based on different conversation states. ​ When to Use STTMuteFilter Common scenarios for muting user input include: During introductions : Prevent the bot from being interrupted during its initial greeting While processing functions : Block input while the bot is retrieving external data During bot speech : Reduce false transcriptions while the bot is speaking For guided conversations : Create more structured interactions with clear turn-taking ​ How It Works The STTMuteFilter works by blocking specific user-related frames from flowing through your pipeline. When muted, it filters: Voice activity detection (VAD) events Interruption signals Raw audio input frames This prevents the Speech-to-Text service from receiving and processing the user’s speech during muted periods. The filter must be placed between your Transport and STT service in the pipeline to work correctly. ​ Mute Strategies The STTMuteFilter supports several strategies for determining when to mute user input: FIRST_SPEECH Mute only during the bot’s first speech utterance. Useful for introductions when you want the bot to complete its greeting before the user can speak. MUTE_UNTIL_FIRST_BOT_COMPLETE Start muted and remain muted until the first bot utterance completes. Ensures the bot’s initial instructions are fully delivered. FUNCTION_CALL Mute during function calls. Prevents users from speaking while the bot is processing external data requests. ALWAYS Mute whenever the bot is speaking. Creates a strict turn-taking conversation pattern. CUSTOM Use custom logic via callback to determine when to mute. Provides maximum flexibility for complex muting rules. The FIRST_SPEECH and MUTE_UNTIL_FIRST_BOT_COMPLETE strategies should not be used together as they handle the first bot speech differently. ​ Basic Implementation ​ Step 1: Configure the Filter First, create a configuration for the STTMuteFilter : Copy Ask AI from pipecat.processors.filters.stt_mute_filter import STTMuteConfig, STTMuteFilter, STTMuteStrategy # Configure with one or more strategies stt_mute_processor = STTMuteFilter( config = STTMuteConfig( strategies = { STTMuteStrategy. MUTE_UNTIL_FIRST_BOT_COMPLETE , STTMuteStrategy. FUNCTION_CALL , } ), ) ​ Step 2: Add to Your Pipeline Place the filter between your transport input and STT service: Copy Ask AI pipeline = Pipeline( [ transport.input(), # Transport user input stt_mute_processor, # Add the mute processor before STT stt, # Speech-to-text service context_aggregator.user(), # User responses llm, # LLM tts, # Text-to-speech transport.output(), # Transport bot output context_aggregator.assistant(), # Assistant spoken responses ] ) ​ Best Practices Place the filter correctly : Always position STTMuteFilter between transport input and STT Choose strategies wisely : Select the minimal set of strategies needed for your use case Test user experience : Excessive muting can frustrate users; balance control with usability Consider feedback : Provide visual cues when the user is muted to improve the experience ​ Next Steps Try the STTMuteFilter Example Explore a complete working example that demonstrates how to use STTMuteFilter to control user input during bot speech and function calls. STTMuteFilter Reference Read the complete API reference documentation for advanced configuration options and muting strategies. Experiment with different muting strategies to find the right balance for your application. For advanced scenarios, try implementing custom muting logic based on specific conversation states or content. Function Calling Recording Audio On this page Overview When to Use STTMuteFilter How It Works Mute Strategies Basic Implementation Step 1: Configure the Filter Step 2: Add to Your Pipeline Best Practices Next Steps Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/fundamentals_user-input-muting_56407bd7.txt b/fundamentals_user-input-muting_56407bd7.txt
new file mode 100644
index 0000000000000000000000000000000000000000..b54d84020c2a5df48b265f1711bba53630f226c3
--- /dev/null
+++ b/fundamentals_user-input-muting_56407bd7.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/guides/fundamentals/user-input-muting#step-1%3A-configure-the-filter
+Title: User Input Muting with STTMuteFilter - Pipecat
+==================================================
+
+User Input Muting with STTMuteFilter - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Fundamentals User Input Muting with STTMuteFilter Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Guides Fundamentals Context Management Custom FrameProcessor Detecting Idle Users Ending a Pipeline Function Calling Muting User Input Recording Audio Recording Transcripts Features Gemini Multimodal Live Metrics Noise cancellation with Krisp OpenAI Audio Models and APIs Pipecat Flows Telephony Overview Dial-in: WebRTC (Daily) Dial-in: WebRTC (Twilio + Daily) Dial-in: Twilio (Media Streams) Dialout: WebRTC (Daily) Deploying your bot Overview Deployment pattern Example: Pipecat Cloud Example: Fly.io Example: Cerebrium Example: Modal ​ Overview In conversational applications, there are moments when you don’t want to process user speech, such as during bot introductions or while executing function calls. Pipecat’s STTMuteFilter lets you selectively “mute” user input based on different conversation states. ​ When to Use STTMuteFilter Common scenarios for muting user input include: During introductions : Prevent the bot from being interrupted during its initial greeting While processing functions : Block input while the bot is retrieving external data During bot speech : Reduce false transcriptions while the bot is speaking For guided conversations : Create more structured interactions with clear turn-taking ​ How It Works The STTMuteFilter works by blocking specific user-related frames from flowing through your pipeline. When muted, it filters: Voice activity detection (VAD) events Interruption signals Raw audio input frames This prevents the Speech-to-Text service from receiving and processing the user’s speech during muted periods. The filter must be placed between your Transport and STT service in the pipeline to work correctly. ​ Mute Strategies The STTMuteFilter supports several strategies for determining when to mute user input: FIRST_SPEECH Mute only during the bot’s first speech utterance. Useful for introductions when you want the bot to complete its greeting before the user can speak. MUTE_UNTIL_FIRST_BOT_COMPLETE Start muted and remain muted until the first bot utterance completes. Ensures the bot’s initial instructions are fully delivered. FUNCTION_CALL Mute during function calls. Prevents users from speaking while the bot is processing external data requests. ALWAYS Mute whenever the bot is speaking. Creates a strict turn-taking conversation pattern. CUSTOM Use custom logic via callback to determine when to mute. Provides maximum flexibility for complex muting rules. The FIRST_SPEECH and MUTE_UNTIL_FIRST_BOT_COMPLETE strategies should not be used together as they handle the first bot speech differently. ​ Basic Implementation ​ Step 1: Configure the Filter First, create a configuration for the STTMuteFilter : Copy Ask AI from pipecat.processors.filters.stt_mute_filter import STTMuteConfig, STTMuteFilter, STTMuteStrategy # Configure with one or more strategies stt_mute_processor = STTMuteFilter( config = STTMuteConfig( strategies = { STTMuteStrategy. MUTE_UNTIL_FIRST_BOT_COMPLETE , STTMuteStrategy. FUNCTION_CALL , } ), ) ​ Step 2: Add to Your Pipeline Place the filter between your transport input and STT service: Copy Ask AI pipeline = Pipeline( [ transport.input(), # Transport user input stt_mute_processor, # Add the mute processor before STT stt, # Speech-to-text service context_aggregator.user(), # User responses llm, # LLM tts, # Text-to-speech transport.output(), # Transport bot output context_aggregator.assistant(), # Assistant spoken responses ] ) ​ Best Practices Place the filter correctly : Always position STTMuteFilter between transport input and STT Choose strategies wisely : Select the minimal set of strategies needed for your use case Test user experience : Excessive muting can frustrate users; balance control with usability Consider feedback : Provide visual cues when the user is muted to improve the experience ​ Next Steps Try the STTMuteFilter Example Explore a complete working example that demonstrates how to use STTMuteFilter to control user input during bot speech and function calls. STTMuteFilter Reference Read the complete API reference documentation for advanced configuration options and muting strategies. Experiment with different muting strategies to find the right balance for your application. For advanced scenarios, try implementing custom muting logic based on specific conversation states or content. Function Calling Recording Audio On this page Overview When to Use STTMuteFilter How It Works Mute Strategies Basic Implementation Step 1: Configure the Filter Step 2: Add to Your Pipeline Best Practices Next Steps Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/fundamentals_user-input-muting_ac4ef827.txt b/fundamentals_user-input-muting_ac4ef827.txt
new file mode 100644
index 0000000000000000000000000000000000000000..812b36cf2b124a2f61916b08b04db311430531ea
--- /dev/null
+++ b/fundamentals_user-input-muting_ac4ef827.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/guides/fundamentals/user-input-muting#how-it-works
+Title: User Input Muting with STTMuteFilter - Pipecat
+==================================================
+
+User Input Muting with STTMuteFilter - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Fundamentals User Input Muting with STTMuteFilter Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Guides Fundamentals Context Management Custom FrameProcessor Detecting Idle Users Ending a Pipeline Function Calling Muting User Input Recording Audio Recording Transcripts Features Gemini Multimodal Live Metrics Noise cancellation with Krisp OpenAI Audio Models and APIs Pipecat Flows Telephony Overview Dial-in: WebRTC (Daily) Dial-in: WebRTC (Twilio + Daily) Dial-in: Twilio (Media Streams) Dialout: WebRTC (Daily) Deploying your bot Overview Deployment pattern Example: Pipecat Cloud Example: Fly.io Example: Cerebrium Example: Modal ​ Overview In conversational applications, there are moments when you don’t want to process user speech, such as during bot introductions or while executing function calls. Pipecat’s STTMuteFilter lets you selectively “mute” user input based on different conversation states. ​ When to Use STTMuteFilter Common scenarios for muting user input include: During introductions : Prevent the bot from being interrupted during its initial greeting While processing functions : Block input while the bot is retrieving external data During bot speech : Reduce false transcriptions while the bot is speaking For guided conversations : Create more structured interactions with clear turn-taking ​ How It Works The STTMuteFilter works by blocking specific user-related frames from flowing through your pipeline. When muted, it filters: Voice activity detection (VAD) events Interruption signals Raw audio input frames This prevents the Speech-to-Text service from receiving and processing the user’s speech during muted periods. The filter must be placed between your Transport and STT service in the pipeline to work correctly. ​ Mute Strategies The STTMuteFilter supports several strategies for determining when to mute user input: FIRST_SPEECH Mute only during the bot’s first speech utterance. Useful for introductions when you want the bot to complete its greeting before the user can speak. MUTE_UNTIL_FIRST_BOT_COMPLETE Start muted and remain muted until the first bot utterance completes. Ensures the bot’s initial instructions are fully delivered. FUNCTION_CALL Mute during function calls. Prevents users from speaking while the bot is processing external data requests. ALWAYS Mute whenever the bot is speaking. Creates a strict turn-taking conversation pattern. CUSTOM Use custom logic via callback to determine when to mute. Provides maximum flexibility for complex muting rules. The FIRST_SPEECH and MUTE_UNTIL_FIRST_BOT_COMPLETE strategies should not be used together as they handle the first bot speech differently. ​ Basic Implementation ​ Step 1: Configure the Filter First, create a configuration for the STTMuteFilter : Copy Ask AI from pipecat.processors.filters.stt_mute_filter import STTMuteConfig, STTMuteFilter, STTMuteStrategy # Configure with one or more strategies stt_mute_processor = STTMuteFilter( config = STTMuteConfig( strategies = { STTMuteStrategy. MUTE_UNTIL_FIRST_BOT_COMPLETE , STTMuteStrategy. FUNCTION_CALL , } ), ) ​ Step 2: Add to Your Pipeline Place the filter between your transport input and STT service: Copy Ask AI pipeline = Pipeline( [ transport.input(), # Transport user input stt_mute_processor, # Add the mute processor before STT stt, # Speech-to-text service context_aggregator.user(), # User responses llm, # LLM tts, # Text-to-speech transport.output(), # Transport bot output context_aggregator.assistant(), # Assistant spoken responses ] ) ​ Best Practices Place the filter correctly : Always position STTMuteFilter between transport input and STT Choose strategies wisely : Select the minimal set of strategies needed for your use case Test user experience : Excessive muting can frustrate users; balance control with usability Consider feedback : Provide visual cues when the user is muted to improve the experience ​ Next Steps Try the STTMuteFilter Example Explore a complete working example that demonstrates how to use STTMuteFilter to control user input during bot speech and function calls. STTMuteFilter Reference Read the complete API reference documentation for advanced configuration options and muting strategies. Experiment with different muting strategies to find the right balance for your application. For advanced scenarios, try implementing custom muting logic based on specific conversation states or content. Function Calling Recording Audio On this page Overview When to Use STTMuteFilter How It Works Mute Strategies Basic Implementation Step 1: Configure the Filter Step 2: Add to Your Pipeline Best Practices Next Steps Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/getting-started_core-concepts_2ece347e.txt b/getting-started_core-concepts_2ece347e.txt
new file mode 100644
index 0000000000000000000000000000000000000000..33b623c2b755a920b034c13ce6da469eb53055a3
--- /dev/null
+++ b/getting-started_core-concepts_2ece347e.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/getting-started/core-concepts
+Title: Core Concepts - Pipecat
+==================================================
+
+Core Concepts - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Get Started Core Concepts Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Get Started Overview Installation & Setup Quickstart Core Concepts Next Steps & Examples Pipecat uses a pipeline-based architecture to handle real-time AI processing. Let’s look at how this works in practice, then break down the key components. ​ Real-time Processing in Action Consider how a voice assistant processes a user’s question and generates a response: Instead of waiting for complete responses at each step, Pipecat processes data in small units called frames that flow through the pipeline: Speech is transcribed in real-time as the user speaks Transcription is sent to the LLM as it becomes available LLM responses are processed as they stream in Text-to-speech begins generating audio for early sentences while later ones are still being generated Audio playback starts as soon as the first sentence is ready LLM context is aggregated and updated continuously and in real-time This streaming approach creates natural, responsive interactions. ​ Architecture Overview Here’s how Pipecat organizes these processes: The architecture consists of three key components: ​ 1. Frames Frames are containers for data moving through your application. Think of them like packages on a conveyor belt - each contains a specific type of cargo. For example: Audio data from a microphone Text from transcription LLM responses Generated speech audio Images or video Control signals and system messages Frames can flow in two directions: Downstream (normal processing flow) Upstream (for errors and control signals) ​ 2. Processors (Services) Processors are workers along our conveyor belt. Each one: Receives frames as inputs Processes specific frame types Passes through frames it doesn’t handle Generates new frames as output Frame processors can do anything, but for real-time, multimodal AI applications, they commonly include: A speech-to-text processor that receives raw audio input frames and outputs transcription frames An LLM processor takes context frames and produces text frames A text-to-speech processor that receives text frames and generates raw audio output frames An image generation processor that takes in text frames and outputs an image URL frame A logging processor might watch all frames but not modify them ​ 3. Pipelines Pipelines connect processors together, creating a path for frames to flow through your application. They can be: Copy Ask AI # Simple linear pipeline pipeline = Pipeline([ transport.input() # Speech -> Audio stt, # Audio -> Text llm, # Text -> Response tts, # Response -> Audio transport.output() # Audio -> Playback ]) # Complex parallel pipeline pipeline = Pipeline([ input_source, ParallelPipeline([ [image_processor, image_output], # Handle images [audio_processor, audio_output] # Handle audio ]) ]) The pipeline also contains the transport, which is the connection to the real world (e.g., microphone, speakers). ​ How It All Works Together Let’s see how these components handle a simple voice interaction: Input User speaks into their microphone Transport converts audio into frames Frames enter the pipeline Processing Transcription processor converts speech to text frames LLM processor takes text frames, generates response frames TTS processor converts response frames to audio frames Error frames flow upstream if issues occur System frames can bypass normal processing for immediate handling Output Audio frames reach the transport Transport plays the audio for the user This happens continuously and in parallel, creating smooth, real-time interactions. ​ Next Steps Now that you understand the big picture, the next step is to install and run your first Pipecat application. Check out the Installation & Setup guide to get started. Need help? Join our Discord community for support and discussions. Quickstart Next Steps & Examples On this page Real-time Processing in Action Architecture Overview 1. Frames 2. Processors (Services) 3. Pipelines How It All Works Together Next Steps Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/getting-started_core-concepts_31dbcea0.txt b/getting-started_core-concepts_31dbcea0.txt
new file mode 100644
index 0000000000000000000000000000000000000000..c26e7cacadc75e1ae6322c781c6f479cd5e832a6
--- /dev/null
+++ b/getting-started_core-concepts_31dbcea0.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/getting-started/core-concepts#how-it-all-works-together
+Title: Core Concepts - Pipecat
+==================================================
+
+Core Concepts - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Get Started Core Concepts Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Get Started Overview Installation & Setup Quickstart Core Concepts Next Steps & Examples Pipecat uses a pipeline-based architecture to handle real-time AI processing. Let’s look at how this works in practice, then break down the key components. ​ Real-time Processing in Action Consider how a voice assistant processes a user’s question and generates a response: Instead of waiting for complete responses at each step, Pipecat processes data in small units called frames that flow through the pipeline: Speech is transcribed in real-time as the user speaks Transcription is sent to the LLM as it becomes available LLM responses are processed as they stream in Text-to-speech begins generating audio for early sentences while later ones are still being generated Audio playback starts as soon as the first sentence is ready LLM context is aggregated and updated continuously and in real-time This streaming approach creates natural, responsive interactions. ​ Architecture Overview Here’s how Pipecat organizes these processes: The architecture consists of three key components: ​ 1. Frames Frames are containers for data moving through your application. Think of them like packages on a conveyor belt - each contains a specific type of cargo. For example: Audio data from a microphone Text from transcription LLM responses Generated speech audio Images or video Control signals and system messages Frames can flow in two directions: Downstream (normal processing flow) Upstream (for errors and control signals) ​ 2. Processors (Services) Processors are workers along our conveyor belt. Each one: Receives frames as inputs Processes specific frame types Passes through frames it doesn’t handle Generates new frames as output Frame processors can do anything, but for real-time, multimodal AI applications, they commonly include: A speech-to-text processor that receives raw audio input frames and outputs transcription frames An LLM processor takes context frames and produces text frames A text-to-speech processor that receives text frames and generates raw audio output frames An image generation processor that takes in text frames and outputs an image URL frame A logging processor might watch all frames but not modify them ​ 3. Pipelines Pipelines connect processors together, creating a path for frames to flow through your application. They can be: Copy Ask AI # Simple linear pipeline pipeline = Pipeline([ transport.input() # Speech -> Audio stt, # Audio -> Text llm, # Text -> Response tts, # Response -> Audio transport.output() # Audio -> Playback ]) # Complex parallel pipeline pipeline = Pipeline([ input_source, ParallelPipeline([ [image_processor, image_output], # Handle images [audio_processor, audio_output] # Handle audio ]) ]) The pipeline also contains the transport, which is the connection to the real world (e.g., microphone, speakers). ​ How It All Works Together Let’s see how these components handle a simple voice interaction: Input User speaks into their microphone Transport converts audio into frames Frames enter the pipeline Processing Transcription processor converts speech to text frames LLM processor takes text frames, generates response frames TTS processor converts response frames to audio frames Error frames flow upstream if issues occur System frames can bypass normal processing for immediate handling Output Audio frames reach the transport Transport plays the audio for the user This happens continuously and in parallel, creating smooth, real-time interactions. ​ Next Steps Now that you understand the big picture, the next step is to install and run your first Pipecat application. Check out the Installation & Setup guide to get started. Need help? Join our Discord community for support and discussions. Quickstart Next Steps & Examples On this page Real-time Processing in Action Architecture Overview 1. Frames 2. Processors (Services) 3. Pipelines How It All Works Together Next Steps Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/getting-started_core-concepts_4bd2bd13.txt b/getting-started_core-concepts_4bd2bd13.txt
new file mode 100644
index 0000000000000000000000000000000000000000..e1186305aae00379bd49e27ec630d3007d65ecc5
--- /dev/null
+++ b/getting-started_core-concepts_4bd2bd13.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/getting-started/core-concepts#next-steps
+Title: Core Concepts - Pipecat
+==================================================
+
+Core Concepts - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Get Started Core Concepts Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Get Started Overview Installation & Setup Quickstart Core Concepts Next Steps & Examples Pipecat uses a pipeline-based architecture to handle real-time AI processing. Let’s look at how this works in practice, then break down the key components. ​ Real-time Processing in Action Consider how a voice assistant processes a user’s question and generates a response: Instead of waiting for complete responses at each step, Pipecat processes data in small units called frames that flow through the pipeline: Speech is transcribed in real-time as the user speaks Transcription is sent to the LLM as it becomes available LLM responses are processed as they stream in Text-to-speech begins generating audio for early sentences while later ones are still being generated Audio playback starts as soon as the first sentence is ready LLM context is aggregated and updated continuously and in real-time This streaming approach creates natural, responsive interactions. ​ Architecture Overview Here’s how Pipecat organizes these processes: The architecture consists of three key components: ​ 1. Frames Frames are containers for data moving through your application. Think of them like packages on a conveyor belt - each contains a specific type of cargo. For example: Audio data from a microphone Text from transcription LLM responses Generated speech audio Images or video Control signals and system messages Frames can flow in two directions: Downstream (normal processing flow) Upstream (for errors and control signals) ​ 2. Processors (Services) Processors are workers along our conveyor belt. Each one: Receives frames as inputs Processes specific frame types Passes through frames it doesn’t handle Generates new frames as output Frame processors can do anything, but for real-time, multimodal AI applications, they commonly include: A speech-to-text processor that receives raw audio input frames and outputs transcription frames An LLM processor takes context frames and produces text frames A text-to-speech processor that receives text frames and generates raw audio output frames An image generation processor that takes in text frames and outputs an image URL frame A logging processor might watch all frames but not modify them ​ 3. Pipelines Pipelines connect processors together, creating a path for frames to flow through your application. They can be: Copy Ask AI # Simple linear pipeline pipeline = Pipeline([ transport.input() # Speech -> Audio stt, # Audio -> Text llm, # Text -> Response tts, # Response -> Audio transport.output() # Audio -> Playback ]) # Complex parallel pipeline pipeline = Pipeline([ input_source, ParallelPipeline([ [image_processor, image_output], # Handle images [audio_processor, audio_output] # Handle audio ]) ]) The pipeline also contains the transport, which is the connection to the real world (e.g., microphone, speakers). ​ How It All Works Together Let’s see how these components handle a simple voice interaction: Input User speaks into their microphone Transport converts audio into frames Frames enter the pipeline Processing Transcription processor converts speech to text frames LLM processor takes text frames, generates response frames TTS processor converts response frames to audio frames Error frames flow upstream if issues occur System frames can bypass normal processing for immediate handling Output Audio frames reach the transport Transport plays the audio for the user This happens continuously and in parallel, creating smooth, real-time interactions. ​ Next Steps Now that you understand the big picture, the next step is to install and run your first Pipecat application. Check out the Installation & Setup guide to get started. Need help? Join our Discord community for support and discussions. Quickstart Next Steps & Examples On this page Real-time Processing in Action Architecture Overview 1. Frames 2. Processors (Services) 3. Pipelines How It All Works Together Next Steps Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/getting-started_installation_22513523.txt b/getting-started_installation_22513523.txt
new file mode 100644
index 0000000000000000000000000000000000000000..a2de44486cb106e15056409c41298d1c77bc3f14
--- /dev/null
+++ b/getting-started_installation_22513523.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/getting-started/installation#accounts-you%E2%80%99ll-need
+Title: Installation & Setup - Pipecat
+==================================================
+
+Installation & Setup - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Get Started Installation & Setup Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Get Started Overview Installation & Setup Quickstart Core Concepts Next Steps & Examples ​ Prerequisites Pipecat requires Python 3.10 or higher. To check your Python version: Copy Ask AI python --version ​ Accounts you’ll need While you can use Pipecat without third-party services, integrating with the following providers will enhance your bot’s capabilities. Here’s what we recommend for getting started: ​ Essential services for the quickstart Cartesia Text-to-Speech : Required for the quickstart. Converts text to natural-sounding speech. ​ Recommended services for production Daily Transport : Handles real-time audio, video, and data exchange between bot and user. Recommended for production (our quickstart uses P2P WebRTC). Deepgram Speech-to-Text : Converts audio to text in realtime. OpenAI / Anthropic / Gemini LLM Inference : Generates streaming text responses based on user input. Cartesia / ElevenLabs / Rime Text-to-Speech : Additional TTS options for production use. Explore our full list of supported services for more integration options. ​ Setting up your project ​ Create a virtual environment We recommend using a virtual environment to manage your dependencies: Copy Ask AI mkdir pipecat-project cd pipecat-project python3 -m venv env Activate the virtual environment based on your operating system: macOS/Linux Windows Copy Ask AI source env/bin/activate Copy Ask AI source env/bin/activate Copy Ask AI source env/Scripts/activate # If using Git Bash # OR . \ env \ Scripts \ activate # If using Command Prompt # OR . \ env \ Scripts \ Activate.ps1 # If using PowerShell ​ Install Pipecat The pipecat-ai Python module uses optional dependencies to keep your installation lightweight. This approach lets you include only the specific AI libraries you need for your project. To install Pipecat with support for the recommended services above, use this command: Copy Ask AI pip install "pipecat-ai[webrtc,deepgram,openai,cartesia]" You can add this to your requirements.txt file or include any combination of supported integrations based on your needs. ​ Configure your services ​ Cartesia Text-to-Speech Setup Visit Cartesia’s Voice library Find a voice you like and click the three dots at the end of the row Select Copy ID to get your voice ID Go to Cartesia API keys and create an API key ​ Environment Setup Create a .env file in your project directory with your service credentials: Copy Ask AI CARTESIA_API_KEY=your_cartesia_api_key CARTESIA_VOICE_ID=your_voice_id Make sure to add .env to your .gitignore file to avoid accidentally committing sensitive API keys. ​ Next Steps Build Your First Bot Now that you have everything set up, proceed to the Quickstart guide to build your first Pipecat application. Overview Quickstart On this page Prerequisites Accounts you’ll need Essential services for the quickstart Recommended services for production Setting up your project Create a virtual environment Install Pipecat Configure your services Cartesia Text-to-Speech Setup Environment Setup Next Steps Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/getting-started_installation_2ac29b17.txt b/getting-started_installation_2ac29b17.txt
new file mode 100644
index 0000000000000000000000000000000000000000..6f1c724d68c08f73f4dcb74c8f99c254675e672d
--- /dev/null
+++ b/getting-started_installation_2ac29b17.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/getting-started/installation#recommended-services-for-production
+Title: Installation & Setup - Pipecat
+==================================================
+
+Installation & Setup - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Get Started Installation & Setup Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Get Started Overview Installation & Setup Quickstart Core Concepts Next Steps & Examples ​ Prerequisites Pipecat requires Python 3.10 or higher. To check your Python version: Copy Ask AI python --version ​ Accounts you’ll need While you can use Pipecat without third-party services, integrating with the following providers will enhance your bot’s capabilities. Here’s what we recommend for getting started: ​ Essential services for the quickstart Cartesia Text-to-Speech : Required for the quickstart. Converts text to natural-sounding speech. ​ Recommended services for production Daily Transport : Handles real-time audio, video, and data exchange between bot and user. Recommended for production (our quickstart uses P2P WebRTC). Deepgram Speech-to-Text : Converts audio to text in realtime. OpenAI / Anthropic / Gemini LLM Inference : Generates streaming text responses based on user input. Cartesia / ElevenLabs / Rime Text-to-Speech : Additional TTS options for production use. Explore our full list of supported services for more integration options. ​ Setting up your project ​ Create a virtual environment We recommend using a virtual environment to manage your dependencies: Copy Ask AI mkdir pipecat-project cd pipecat-project python3 -m venv env Activate the virtual environment based on your operating system: macOS/Linux Windows Copy Ask AI source env/bin/activate Copy Ask AI source env/bin/activate Copy Ask AI source env/Scripts/activate # If using Git Bash # OR . \ env \ Scripts \ activate # If using Command Prompt # OR . \ env \ Scripts \ Activate.ps1 # If using PowerShell ​ Install Pipecat The pipecat-ai Python module uses optional dependencies to keep your installation lightweight. This approach lets you include only the specific AI libraries you need for your project. To install Pipecat with support for the recommended services above, use this command: Copy Ask AI pip install "pipecat-ai[webrtc,deepgram,openai,cartesia]" You can add this to your requirements.txt file or include any combination of supported integrations based on your needs. ​ Configure your services ​ Cartesia Text-to-Speech Setup Visit Cartesia’s Voice library Find a voice you like and click the three dots at the end of the row Select Copy ID to get your voice ID Go to Cartesia API keys and create an API key ​ Environment Setup Create a .env file in your project directory with your service credentials: Copy Ask AI CARTESIA_API_KEY=your_cartesia_api_key CARTESIA_VOICE_ID=your_voice_id Make sure to add .env to your .gitignore file to avoid accidentally committing sensitive API keys. ​ Next Steps Build Your First Bot Now that you have everything set up, proceed to the Quickstart guide to build your first Pipecat application. Overview Quickstart On this page Prerequisites Accounts you’ll need Essential services for the quickstart Recommended services for production Setting up your project Create a virtual environment Install Pipecat Configure your services Cartesia Text-to-Speech Setup Environment Setup Next Steps Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/getting-started_installation_3c5a4d3e.txt b/getting-started_installation_3c5a4d3e.txt
new file mode 100644
index 0000000000000000000000000000000000000000..22a6a2d9a672b67f99b4f0d85d4b106312c40606
--- /dev/null
+++ b/getting-started_installation_3c5a4d3e.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/getting-started/installation#environment-setup
+Title: Installation & Setup - Pipecat
+==================================================
+
+Installation & Setup - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Get Started Installation & Setup Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Get Started Overview Installation & Setup Quickstart Core Concepts Next Steps & Examples ​ Prerequisites Pipecat requires Python 3.10 or higher. To check your Python version: Copy Ask AI python --version ​ Accounts you’ll need While you can use Pipecat without third-party services, integrating with the following providers will enhance your bot’s capabilities. Here’s what we recommend for getting started: ​ Essential services for the quickstart Cartesia Text-to-Speech : Required for the quickstart. Converts text to natural-sounding speech. ​ Recommended services for production Daily Transport : Handles real-time audio, video, and data exchange between bot and user. Recommended for production (our quickstart uses P2P WebRTC). Deepgram Speech-to-Text : Converts audio to text in realtime. OpenAI / Anthropic / Gemini LLM Inference : Generates streaming text responses based on user input. Cartesia / ElevenLabs / Rime Text-to-Speech : Additional TTS options for production use. Explore our full list of supported services for more integration options. ​ Setting up your project ​ Create a virtual environment We recommend using a virtual environment to manage your dependencies: Copy Ask AI mkdir pipecat-project cd pipecat-project python3 -m venv env Activate the virtual environment based on your operating system: macOS/Linux Windows Copy Ask AI source env/bin/activate Copy Ask AI source env/bin/activate Copy Ask AI source env/Scripts/activate # If using Git Bash # OR . \ env \ Scripts \ activate # If using Command Prompt # OR . \ env \ Scripts \ Activate.ps1 # If using PowerShell ​ Install Pipecat The pipecat-ai Python module uses optional dependencies to keep your installation lightweight. This approach lets you include only the specific AI libraries you need for your project. To install Pipecat with support for the recommended services above, use this command: Copy Ask AI pip install "pipecat-ai[webrtc,deepgram,openai,cartesia]" You can add this to your requirements.txt file or include any combination of supported integrations based on your needs. ​ Configure your services ​ Cartesia Text-to-Speech Setup Visit Cartesia’s Voice library Find a voice you like and click the three dots at the end of the row Select Copy ID to get your voice ID Go to Cartesia API keys and create an API key ​ Environment Setup Create a .env file in your project directory with your service credentials: Copy Ask AI CARTESIA_API_KEY=your_cartesia_api_key CARTESIA_VOICE_ID=your_voice_id Make sure to add .env to your .gitignore file to avoid accidentally committing sensitive API keys. ​ Next Steps Build Your First Bot Now that you have everything set up, proceed to the Quickstart guide to build your first Pipecat application. Overview Quickstart On this page Prerequisites Accounts you’ll need Essential services for the quickstart Recommended services for production Setting up your project Create a virtual environment Install Pipecat Configure your services Cartesia Text-to-Speech Setup Environment Setup Next Steps Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/getting-started_installation_85bea115.txt b/getting-started_installation_85bea115.txt
new file mode 100644
index 0000000000000000000000000000000000000000..b0024854591b0971b584cbe6580c19cc129cb17a
--- /dev/null
+++ b/getting-started_installation_85bea115.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/getting-started/installation#next-steps
+Title: Installation & Setup - Pipecat
+==================================================
+
+Installation & Setup - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Get Started Installation & Setup Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Get Started Overview Installation & Setup Quickstart Core Concepts Next Steps & Examples ​ Prerequisites Pipecat requires Python 3.10 or higher. To check your Python version: Copy Ask AI python --version ​ Accounts you’ll need While you can use Pipecat without third-party services, integrating with the following providers will enhance your bot’s capabilities. Here’s what we recommend for getting started: ​ Essential services for the quickstart Cartesia Text-to-Speech : Required for the quickstart. Converts text to natural-sounding speech. ​ Recommended services for production Daily Transport : Handles real-time audio, video, and data exchange between bot and user. Recommended for production (our quickstart uses P2P WebRTC). Deepgram Speech-to-Text : Converts audio to text in realtime. OpenAI / Anthropic / Gemini LLM Inference : Generates streaming text responses based on user input. Cartesia / ElevenLabs / Rime Text-to-Speech : Additional TTS options for production use. Explore our full list of supported services for more integration options. ​ Setting up your project ​ Create a virtual environment We recommend using a virtual environment to manage your dependencies: Copy Ask AI mkdir pipecat-project cd pipecat-project python3 -m venv env Activate the virtual environment based on your operating system: macOS/Linux Windows Copy Ask AI source env/bin/activate Copy Ask AI source env/bin/activate Copy Ask AI source env/Scripts/activate # If using Git Bash # OR . \ env \ Scripts \ activate # If using Command Prompt # OR . \ env \ Scripts \ Activate.ps1 # If using PowerShell ​ Install Pipecat The pipecat-ai Python module uses optional dependencies to keep your installation lightweight. This approach lets you include only the specific AI libraries you need for your project. To install Pipecat with support for the recommended services above, use this command: Copy Ask AI pip install "pipecat-ai[webrtc,deepgram,openai,cartesia]" You can add this to your requirements.txt file or include any combination of supported integrations based on your needs. ​ Configure your services ​ Cartesia Text-to-Speech Setup Visit Cartesia’s Voice library Find a voice you like and click the three dots at the end of the row Select Copy ID to get your voice ID Go to Cartesia API keys and create an API key ​ Environment Setup Create a .env file in your project directory with your service credentials: Copy Ask AI CARTESIA_API_KEY=your_cartesia_api_key CARTESIA_VOICE_ID=your_voice_id Make sure to add .env to your .gitignore file to avoid accidentally committing sensitive API keys. ​ Next Steps Build Your First Bot Now that you have everything set up, proceed to the Quickstart guide to build your first Pipecat application. Overview Quickstart On this page Prerequisites Accounts you’ll need Essential services for the quickstart Recommended services for production Setting up your project Create a virtual environment Install Pipecat Configure your services Cartesia Text-to-Speech Setup Environment Setup Next Steps Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/getting-started_installation_dfb5052c.txt b/getting-started_installation_dfb5052c.txt
new file mode 100644
index 0000000000000000000000000000000000000000..2cfae5b9d55cd8b81ad27a896bc942ce206ed8e9
--- /dev/null
+++ b/getting-started_installation_dfb5052c.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/getting-started/installation#essential-services-for-the-quickstart
+Title: Installation & Setup - Pipecat
+==================================================
+
+Installation & Setup - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Get Started Installation & Setup Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Get Started Overview Installation & Setup Quickstart Core Concepts Next Steps & Examples ​ Prerequisites Pipecat requires Python 3.10 or higher. To check your Python version: Copy Ask AI python --version ​ Accounts you’ll need While you can use Pipecat without third-party services, integrating with the following providers will enhance your bot’s capabilities. Here’s what we recommend for getting started: ​ Essential services for the quickstart Cartesia Text-to-Speech : Required for the quickstart. Converts text to natural-sounding speech. ​ Recommended services for production Daily Transport : Handles real-time audio, video, and data exchange between bot and user. Recommended for production (our quickstart uses P2P WebRTC). Deepgram Speech-to-Text : Converts audio to text in realtime. OpenAI / Anthropic / Gemini LLM Inference : Generates streaming text responses based on user input. Cartesia / ElevenLabs / Rime Text-to-Speech : Additional TTS options for production use. Explore our full list of supported services for more integration options. ​ Setting up your project ​ Create a virtual environment We recommend using a virtual environment to manage your dependencies: Copy Ask AI mkdir pipecat-project cd pipecat-project python3 -m venv env Activate the virtual environment based on your operating system: macOS/Linux Windows Copy Ask AI source env/bin/activate Copy Ask AI source env/bin/activate Copy Ask AI source env/Scripts/activate # If using Git Bash # OR . \ env \ Scripts \ activate # If using Command Prompt # OR . \ env \ Scripts \ Activate.ps1 # If using PowerShell ​ Install Pipecat The pipecat-ai Python module uses optional dependencies to keep your installation lightweight. This approach lets you include only the specific AI libraries you need for your project. To install Pipecat with support for the recommended services above, use this command: Copy Ask AI pip install "pipecat-ai[webrtc,deepgram,openai,cartesia]" You can add this to your requirements.txt file or include any combination of supported integrations based on your needs. ​ Configure your services ​ Cartesia Text-to-Speech Setup Visit Cartesia’s Voice library Find a voice you like and click the three dots at the end of the row Select Copy ID to get your voice ID Go to Cartesia API keys and create an API key ​ Environment Setup Create a .env file in your project directory with your service credentials: Copy Ask AI CARTESIA_API_KEY=your_cartesia_api_key CARTESIA_VOICE_ID=your_voice_id Make sure to add .env to your .gitignore file to avoid accidentally committing sensitive API keys. ​ Next Steps Build Your First Bot Now that you have everything set up, proceed to the Quickstart guide to build your first Pipecat application. Overview Quickstart On this page Prerequisites Accounts you’ll need Essential services for the quickstart Recommended services for production Setting up your project Create a virtual environment Install Pipecat Configure your services Cartesia Text-to-Speech Setup Environment Setup Next Steps Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/getting-started_next-steps_1372a187.txt b/getting-started_next-steps_1372a187.txt
new file mode 100644
index 0000000000000000000000000000000000000000..c89cd7a8b8b6194955bb94e460a210f41bbf1189
--- /dev/null
+++ b/getting-started_next-steps_1372a187.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/getting-started/next-steps#community-resources
+Title: Next Steps & Examples - Pipecat
+==================================================
+
+Next Steps & Examples - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Get Started Next Steps & Examples Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Get Started Overview Installation & Setup Quickstart Core Concepts Next Steps & Examples After completing the quickstart, you’re ready to explore more advanced Pipecat features. This guide will help you choose the right pathway based on your project goals. ​ Choose Your Development Path Learn the Fundamentals Master Pipecat’s core concepts through progressive examples Build Web Applications Create client/server voice applications with web interfaces Develop Phone Bots Build voice bots accessible through regular phone calls Structure Conversations Design complex conversational flows with Pipecat Flows Explore Use Cases See how others are using Pipecat in production Join the Community Connect with other developers building with Pipecat ​ Foundational Examples If you’re new to Pipecat, we recommend exploring our foundational examples that demonstrate core concepts in a progressive manner: Foundational Examples Repository A series of step-by-step examples that build upon each other to teach Pipecat fundamentals. ​ Web & Client/Server Applications For building web-based conversational applications with separate client and server components: Simple Chatbot Example A complete client/server example demonstrating how to connect your Pipecat bot to JS, React, iOS, and Android clients. ​ Telephony Integrations To make your bots accessible via regular phone calls: Twilio Chatbot Build phone-accessible bots using Twilio’s telephony platform Telnyx Chatbot Create voice bots using Telnyx’s communication API ​ Structured Conversations with Pipecat Flows For complex applications requiring structured dialogs: Pipecat Flows A powerful extension to Pipecat that allows you to build structured conversations and manage the conversation context. Try the Flow Editor to visually design your conversation flows. ​ Real-world Examples Get inspired by applications others have built with Pipecat: Storytelling Bot AI-powered storyteller that creates and narrates interactive stories Patient Intake System Automated form-filling system for healthcare applications StudyPal Interactive study assistant that helps with learning Vision-enabled Bot Multimodal assistant that can see and discuss images ​ Community Resources Join the Pipecat Discord Connect with other Pipecat developers to get help, share your projects, and stay updated on new features and releases. Core Concepts On this page Choose Your Development Path Foundational Examples Web & Client/Server Applications Telephony Integrations Structured Conversations with Pipecat Flows Real-world Examples Community Resources Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/getting-started_next-steps_2f3aaa60.txt b/getting-started_next-steps_2f3aaa60.txt
new file mode 100644
index 0000000000000000000000000000000000000000..10bd34518b20d9772174c9887b374920c502059f
--- /dev/null
+++ b/getting-started_next-steps_2f3aaa60.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/getting-started/next-steps#telephony-integrations
+Title: Next Steps & Examples - Pipecat
+==================================================
+
+Next Steps & Examples - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Get Started Next Steps & Examples Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Get Started Overview Installation & Setup Quickstart Core Concepts Next Steps & Examples After completing the quickstart, you’re ready to explore more advanced Pipecat features. This guide will help you choose the right pathway based on your project goals. ​ Choose Your Development Path Learn the Fundamentals Master Pipecat’s core concepts through progressive examples Build Web Applications Create client/server voice applications with web interfaces Develop Phone Bots Build voice bots accessible through regular phone calls Structure Conversations Design complex conversational flows with Pipecat Flows Explore Use Cases See how others are using Pipecat in production Join the Community Connect with other developers building with Pipecat ​ Foundational Examples If you’re new to Pipecat, we recommend exploring our foundational examples that demonstrate core concepts in a progressive manner: Foundational Examples Repository A series of step-by-step examples that build upon each other to teach Pipecat fundamentals. ​ Web & Client/Server Applications For building web-based conversational applications with separate client and server components: Simple Chatbot Example A complete client/server example demonstrating how to connect your Pipecat bot to JS, React, iOS, and Android clients. ​ Telephony Integrations To make your bots accessible via regular phone calls: Twilio Chatbot Build phone-accessible bots using Twilio’s telephony platform Telnyx Chatbot Create voice bots using Telnyx’s communication API ​ Structured Conversations with Pipecat Flows For complex applications requiring structured dialogs: Pipecat Flows A powerful extension to Pipecat that allows you to build structured conversations and manage the conversation context. Try the Flow Editor to visually design your conversation flows. ​ Real-world Examples Get inspired by applications others have built with Pipecat: Storytelling Bot AI-powered storyteller that creates and narrates interactive stories Patient Intake System Automated form-filling system for healthcare applications StudyPal Interactive study assistant that helps with learning Vision-enabled Bot Multimodal assistant that can see and discuss images ​ Community Resources Join the Pipecat Discord Connect with other Pipecat developers to get help, share your projects, and stay updated on new features and releases. Core Concepts On this page Choose Your Development Path Foundational Examples Web & Client/Server Applications Telephony Integrations Structured Conversations with Pipecat Flows Real-world Examples Community Resources Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/getting-started_next-steps_79de19e1.txt b/getting-started_next-steps_79de19e1.txt
new file mode 100644
index 0000000000000000000000000000000000000000..2207349fb39e4dd81e43ff7663353d31b2e20119
--- /dev/null
+++ b/getting-started_next-steps_79de19e1.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/_sites/docs.pipecat.ai/getting-started/next-steps#real-world-examples
+Title: Overview - Pipecat
+==================================================
+
+Overview - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Get Started Overview Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Get Started Overview Installation & Setup Quickstart Core Concepts Next Steps & Examples Pipecat is an open source Python framework that handles the complex orchestration of AI services, network transport, audio processing, and multimodal interactions. “Multimodal” means you can use any combination of audio, video, images, and/or text in your interactions. And “real-time” means that things are happening quickly enough that it feels conversational—a “back-and-forth” with a bot, not submitting a query and waiting for results. ​ What You Can Build Voice Assistants Natural, real-time conversations with AI using speech recognition and synthesis Interactive Agents Personal coaches and meeting assistants that can understand context and provide guidance Multimodal Apps Applications that combine voice, video, images, and text for rich interactions Creative Tools Storytelling experiences and social companions that engage users Business Solutions Customer intake flows and support bots for automated business processes Complex Flows Structured conversations using Pipecat Flows for managing complex interactions ​ How It Works The flow of interactions in a Pipecat application is typically straightforward: The bot says something The user says something The bot says something The user says something This continues until the conversation naturally ends. While this flow seems simple, making it feel natural requires sophisticated real-time processing. ​ Real-time Processing Pipecat’s pipeline architecture handles both simple voice interactions and complex multimodal processing. Let’s look at how data flows through the system: Voice app Multimodal app 1 Send Audio Transmit and capture streamed audio from the user 2 Transcribe Speech Convert speech to text as the user is talking 3 Process with LLM Generate responses using a large language model 4 Convert to Speech Transform text responses into natural speech 5 Play Audio Stream the audio response back to the user 1 Send Audio Transmit and capture streamed audio from the user 2 Transcribe Speech Convert speech to text as the user is talking 3 Process with LLM Generate responses using a large language model 4 Convert to Speech Transform text responses into natural speech 5 Play Audio Stream the audio response back to the user 1 Send Audio and Video Transmit and capture audio, video, and image inputs simultaneously 2 Process Streams Handle multiple input streams in parallel 3 Model Processing Send combined inputs to multimodal models (like GPT-4V) 4 Generate Outputs Create various outputs (text, images, audio, etc.) 5 Coordinate Presentation Synchronize and present multiple output types In both cases, Pipecat: Processes responses as they stream in Handles multiple input/output modalities concurrently Manages resource allocation and synchronization Coordinates parallel processing tasks This architecture creates fluid, natural interactions without noticeable delays, whether you’re building a simple voice assistant or a complex multimodal application. Pipecat’s pipeline architecture is particularly valuable for managing the complexity of real-time, multimodal interactions, ensuring smooth data flow and proper synchronization regardless of the input/output types involved. Pipecat handles all this complexity for you, letting you focus on building your application rather than managing the underlying infrastructure. ​ Next Steps Ready to build your first Pipecat application? Installation & Setup Prepare your environment and install required dependencies Quickstart Build and run your first Pipecat application Core Concepts Learn about pipelines, frames, and real-time processing Use Cases Explore example implementations and patterns ​ Join Our Community Discord Community Connect with other developers, share your projects, and get support from the Pipecat team. Installation & Setup On this page What You Can Build How It Works Real-time Processing Next Steps Join Our Community Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/getting-started_next-steps_a0efdd45.txt b/getting-started_next-steps_a0efdd45.txt
new file mode 100644
index 0000000000000000000000000000000000000000..4cb0759aaf7e86d8209d7bc92639f221a45b8124
--- /dev/null
+++ b/getting-started_next-steps_a0efdd45.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/_sites/docs.pipecat.ai/getting-started/next-steps#join-our-community
+Title: Overview - Pipecat
+==================================================
+
+Overview - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Get Started Overview Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Get Started Overview Installation & Setup Quickstart Core Concepts Next Steps & Examples Pipecat is an open source Python framework that handles the complex orchestration of AI services, network transport, audio processing, and multimodal interactions. “Multimodal” means you can use any combination of audio, video, images, and/or text in your interactions. And “real-time” means that things are happening quickly enough that it feels conversational—a “back-and-forth” with a bot, not submitting a query and waiting for results. ​ What You Can Build Voice Assistants Natural, real-time conversations with AI using speech recognition and synthesis Interactive Agents Personal coaches and meeting assistants that can understand context and provide guidance Multimodal Apps Applications that combine voice, video, images, and text for rich interactions Creative Tools Storytelling experiences and social companions that engage users Business Solutions Customer intake flows and support bots for automated business processes Complex Flows Structured conversations using Pipecat Flows for managing complex interactions ​ How It Works The flow of interactions in a Pipecat application is typically straightforward: The bot says something The user says something The bot says something The user says something This continues until the conversation naturally ends. While this flow seems simple, making it feel natural requires sophisticated real-time processing. ​ Real-time Processing Pipecat’s pipeline architecture handles both simple voice interactions and complex multimodal processing. Let’s look at how data flows through the system: Voice app Multimodal app 1 Send Audio Transmit and capture streamed audio from the user 2 Transcribe Speech Convert speech to text as the user is talking 3 Process with LLM Generate responses using a large language model 4 Convert to Speech Transform text responses into natural speech 5 Play Audio Stream the audio response back to the user 1 Send Audio Transmit and capture streamed audio from the user 2 Transcribe Speech Convert speech to text as the user is talking 3 Process with LLM Generate responses using a large language model 4 Convert to Speech Transform text responses into natural speech 5 Play Audio Stream the audio response back to the user 1 Send Audio and Video Transmit and capture audio, video, and image inputs simultaneously 2 Process Streams Handle multiple input streams in parallel 3 Model Processing Send combined inputs to multimodal models (like GPT-4V) 4 Generate Outputs Create various outputs (text, images, audio, etc.) 5 Coordinate Presentation Synchronize and present multiple output types In both cases, Pipecat: Processes responses as they stream in Handles multiple input/output modalities concurrently Manages resource allocation and synchronization Coordinates parallel processing tasks This architecture creates fluid, natural interactions without noticeable delays, whether you’re building a simple voice assistant or a complex multimodal application. Pipecat’s pipeline architecture is particularly valuable for managing the complexity of real-time, multimodal interactions, ensuring smooth data flow and proper synchronization regardless of the input/output types involved. Pipecat handles all this complexity for you, letting you focus on building your application rather than managing the underlying infrastructure. ​ Next Steps Ready to build your first Pipecat application? Installation & Setup Prepare your environment and install required dependencies Quickstart Build and run your first Pipecat application Core Concepts Learn about pipelines, frames, and real-time processing Use Cases Explore example implementations and patterns ​ Join Our Community Discord Community Connect with other developers, share your projects, and get support from the Pipecat team. Installation & Setup On this page What You Can Build How It Works Real-time Processing Next Steps Join Our Community Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/getting-started_next-steps_eff665fb.txt b/getting-started_next-steps_eff665fb.txt
new file mode 100644
index 0000000000000000000000000000000000000000..53ee2570ff85a5f974be5bd496f7d70dddd49436
--- /dev/null
+++ b/getting-started_next-steps_eff665fb.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/_sites/docs.pipecat.ai/getting-started/next-steps#next-steps
+Title: Overview - Pipecat
+==================================================
+
+Overview - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Get Started Overview Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Get Started Overview Installation & Setup Quickstart Core Concepts Next Steps & Examples Pipecat is an open source Python framework that handles the complex orchestration of AI services, network transport, audio processing, and multimodal interactions. “Multimodal” means you can use any combination of audio, video, images, and/or text in your interactions. And “real-time” means that things are happening quickly enough that it feels conversational—a “back-and-forth” with a bot, not submitting a query and waiting for results. ​ What You Can Build Voice Assistants Natural, real-time conversations with AI using speech recognition and synthesis Interactive Agents Personal coaches and meeting assistants that can understand context and provide guidance Multimodal Apps Applications that combine voice, video, images, and text for rich interactions Creative Tools Storytelling experiences and social companions that engage users Business Solutions Customer intake flows and support bots for automated business processes Complex Flows Structured conversations using Pipecat Flows for managing complex interactions ​ How It Works The flow of interactions in a Pipecat application is typically straightforward: The bot says something The user says something The bot says something The user says something This continues until the conversation naturally ends. While this flow seems simple, making it feel natural requires sophisticated real-time processing. ​ Real-time Processing Pipecat’s pipeline architecture handles both simple voice interactions and complex multimodal processing. Let’s look at how data flows through the system: Voice app Multimodal app 1 Send Audio Transmit and capture streamed audio from the user 2 Transcribe Speech Convert speech to text as the user is talking 3 Process with LLM Generate responses using a large language model 4 Convert to Speech Transform text responses into natural speech 5 Play Audio Stream the audio response back to the user 1 Send Audio Transmit and capture streamed audio from the user 2 Transcribe Speech Convert speech to text as the user is talking 3 Process with LLM Generate responses using a large language model 4 Convert to Speech Transform text responses into natural speech 5 Play Audio Stream the audio response back to the user 1 Send Audio and Video Transmit and capture audio, video, and image inputs simultaneously 2 Process Streams Handle multiple input streams in parallel 3 Model Processing Send combined inputs to multimodal models (like GPT-4V) 4 Generate Outputs Create various outputs (text, images, audio, etc.) 5 Coordinate Presentation Synchronize and present multiple output types In both cases, Pipecat: Processes responses as they stream in Handles multiple input/output modalities concurrently Manages resource allocation and synchronization Coordinates parallel processing tasks This architecture creates fluid, natural interactions without noticeable delays, whether you’re building a simple voice assistant or a complex multimodal application. Pipecat’s pipeline architecture is particularly valuable for managing the complexity of real-time, multimodal interactions, ensuring smooth data flow and proper synchronization regardless of the input/output types involved. Pipecat handles all this complexity for you, letting you focus on building your application rather than managing the underlying infrastructure. ​ Next Steps Ready to build your first Pipecat application? Installation & Setup Prepare your environment and install required dependencies Quickstart Build and run your first Pipecat application Core Concepts Learn about pipelines, frames, and real-time processing Use Cases Explore example implementations and patterns ​ Join Our Community Discord Community Connect with other developers, share your projects, and get support from the Pipecat team. Installation & Setup On this page What You Can Build How It Works Real-time Processing Next Steps Join Our Community Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/getting-started_overview_75170956.txt b/getting-started_overview_75170956.txt
new file mode 100644
index 0000000000000000000000000000000000000000..baeaf84456562d9dc1c205d81a71e496dcddf909
--- /dev/null
+++ b/getting-started_overview_75170956.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/getting-started/overview#next-steps
+Title: Overview - Pipecat
+==================================================
+
+Overview - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Get Started Overview Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Get Started Overview Installation & Setup Quickstart Core Concepts Next Steps & Examples Pipecat is an open source Python framework that handles the complex orchestration of AI services, network transport, audio processing, and multimodal interactions. “Multimodal” means you can use any combination of audio, video, images, and/or text in your interactions. And “real-time” means that things are happening quickly enough that it feels conversational—a “back-and-forth” with a bot, not submitting a query and waiting for results. ​ What You Can Build Voice Assistants Natural, real-time conversations with AI using speech recognition and synthesis Interactive Agents Personal coaches and meeting assistants that can understand context and provide guidance Multimodal Apps Applications that combine voice, video, images, and text for rich interactions Creative Tools Storytelling experiences and social companions that engage users Business Solutions Customer intake flows and support bots for automated business processes Complex Flows Structured conversations using Pipecat Flows for managing complex interactions ​ How It Works The flow of interactions in a Pipecat application is typically straightforward: The bot says something The user says something The bot says something The user says something This continues until the conversation naturally ends. While this flow seems simple, making it feel natural requires sophisticated real-time processing. ​ Real-time Processing Pipecat’s pipeline architecture handles both simple voice interactions and complex multimodal processing. Let’s look at how data flows through the system: Voice app Multimodal app 1 Send Audio Transmit and capture streamed audio from the user 2 Transcribe Speech Convert speech to text as the user is talking 3 Process with LLM Generate responses using a large language model 4 Convert to Speech Transform text responses into natural speech 5 Play Audio Stream the audio response back to the user 1 Send Audio Transmit and capture streamed audio from the user 2 Transcribe Speech Convert speech to text as the user is talking 3 Process with LLM Generate responses using a large language model 4 Convert to Speech Transform text responses into natural speech 5 Play Audio Stream the audio response back to the user 1 Send Audio and Video Transmit and capture audio, video, and image inputs simultaneously 2 Process Streams Handle multiple input streams in parallel 3 Model Processing Send combined inputs to multimodal models (like GPT-4V) 4 Generate Outputs Create various outputs (text, images, audio, etc.) 5 Coordinate Presentation Synchronize and present multiple output types In both cases, Pipecat: Processes responses as they stream in Handles multiple input/output modalities concurrently Manages resource allocation and synchronization Coordinates parallel processing tasks This architecture creates fluid, natural interactions without noticeable delays, whether you’re building a simple voice assistant or a complex multimodal application. Pipecat’s pipeline architecture is particularly valuable for managing the complexity of real-time, multimodal interactions, ensuring smooth data flow and proper synchronization regardless of the input/output types involved. Pipecat handles all this complexity for you, letting you focus on building your application rather than managing the underlying infrastructure. ​ Next Steps Ready to build your first Pipecat application? Installation & Setup Prepare your environment and install required dependencies Quickstart Build and run your first Pipecat application Core Concepts Learn about pipelines, frames, and real-time processing Use Cases Explore example implementations and patterns ​ Join Our Community Discord Community Connect with other developers, share your projects, and get support from the Pipecat team. Installation & Setup On this page What You Can Build How It Works Real-time Processing Next Steps Join Our Community Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/getting-started_overview_e54297b5.txt b/getting-started_overview_e54297b5.txt
new file mode 100644
index 0000000000000000000000000000000000000000..f484d2b8d00e82ca8a6b7987a3981b0e936acda7
--- /dev/null
+++ b/getting-started_overview_e54297b5.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/getting-started/overview#real-time-processing
+Title: Overview - Pipecat
+==================================================
+
+Overview - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Get Started Overview Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Get Started Overview Installation & Setup Quickstart Core Concepts Next Steps & Examples Pipecat is an open source Python framework that handles the complex orchestration of AI services, network transport, audio processing, and multimodal interactions. “Multimodal” means you can use any combination of audio, video, images, and/or text in your interactions. And “real-time” means that things are happening quickly enough that it feels conversational—a “back-and-forth” with a bot, not submitting a query and waiting for results. ​ What You Can Build Voice Assistants Natural, real-time conversations with AI using speech recognition and synthesis Interactive Agents Personal coaches and meeting assistants that can understand context and provide guidance Multimodal Apps Applications that combine voice, video, images, and text for rich interactions Creative Tools Storytelling experiences and social companions that engage users Business Solutions Customer intake flows and support bots for automated business processes Complex Flows Structured conversations using Pipecat Flows for managing complex interactions ​ How It Works The flow of interactions in a Pipecat application is typically straightforward: The bot says something The user says something The bot says something The user says something This continues until the conversation naturally ends. While this flow seems simple, making it feel natural requires sophisticated real-time processing. ​ Real-time Processing Pipecat’s pipeline architecture handles both simple voice interactions and complex multimodal processing. Let’s look at how data flows through the system: Voice app Multimodal app 1 Send Audio Transmit and capture streamed audio from the user 2 Transcribe Speech Convert speech to text as the user is talking 3 Process with LLM Generate responses using a large language model 4 Convert to Speech Transform text responses into natural speech 5 Play Audio Stream the audio response back to the user 1 Send Audio Transmit and capture streamed audio from the user 2 Transcribe Speech Convert speech to text as the user is talking 3 Process with LLM Generate responses using a large language model 4 Convert to Speech Transform text responses into natural speech 5 Play Audio Stream the audio response back to the user 1 Send Audio and Video Transmit and capture audio, video, and image inputs simultaneously 2 Process Streams Handle multiple input streams in parallel 3 Model Processing Send combined inputs to multimodal models (like GPT-4V) 4 Generate Outputs Create various outputs (text, images, audio, etc.) 5 Coordinate Presentation Synchronize and present multiple output types In both cases, Pipecat: Processes responses as they stream in Handles multiple input/output modalities concurrently Manages resource allocation and synchronization Coordinates parallel processing tasks This architecture creates fluid, natural interactions without noticeable delays, whether you’re building a simple voice assistant or a complex multimodal application. Pipecat’s pipeline architecture is particularly valuable for managing the complexity of real-time, multimodal interactions, ensuring smooth data flow and proper synchronization regardless of the input/output types involved. Pipecat handles all this complexity for you, letting you focus on building your application rather than managing the underlying infrastructure. ​ Next Steps Ready to build your first Pipecat application? Installation & Setup Prepare your environment and install required dependencies Quickstart Build and run your first Pipecat application Core Concepts Learn about pipelines, frames, and real-time processing Use Cases Explore example implementations and patterns ​ Join Our Community Discord Community Connect with other developers, share your projects, and get support from the Pipecat team. Installation & Setup On this page What You Can Build How It Works Real-time Processing Next Steps Join Our Community Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/getting-started_quickstart_24277365.txt b/getting-started_quickstart_24277365.txt
new file mode 100644
index 0000000000000000000000000000000000000000..4f0ed9faa75c9f76995f3a38e67bd6a0331424f4
--- /dev/null
+++ b/getting-started_quickstart_24277365.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/getting-started/quickstart#understanding-the-code
+Title: Quickstart - Pipecat
+==================================================
+
+Quickstart - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Get Started Quickstart Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Get Started Overview Installation & Setup Quickstart Core Concepts Next Steps & Examples In this quickstart, we’ll create a simple conversational bot that greets users when they join and exits when they leave. This example demonstrates the core components of a Pipecat application with a streamlined setup. ​ Set up your project ​ Create a project directory Copy Ask AI mkdir pipecat-quickstart && cd pipecat-quickstart ​ Set up a virtual environment macOS/Linux Windows Copy Ask AI python3 -m venv env source env/bin/activate Copy Ask AI python3 -m venv env source env/bin/activate Copy Ask AI python -m venv env source env/Scripts/activate # If using Git Bash # OR . \ env \ Scripts \ activate # If using Command Prompt # OR . \ env \ Scripts \ Activate.ps1 # If using PowerShell ​ Download the example files Linux/macOS Windows (PowerShell) Manual Download Copy Ask AI curl -O https://raw.githubusercontent.com/pipecat-ai/pipecat/main/examples/foundational/01-say-one-thing.py curl -O https://raw.githubusercontent.com/pipecat-ai/pipecat/main/examples/foundational/requirements.txt Copy Ask AI curl -O https://raw.githubusercontent.com/pipecat-ai/pipecat/main/examples/foundational/01-say-one-thing.py curl -O https://raw.githubusercontent.com/pipecat-ai/pipecat/main/examples/foundational/requirements.txt Copy Ask AI curl.exe - O https: // raw.githubusercontent.com / pipecat - ai / pipecat / main / examples / foundational / 01 - say - one - thing.py curl.exe - O https: // raw.githubusercontent.com / pipecat - ai / pipecat / main / examples / foundational / requirements.txt Download these files and save them to your project directory: 01-say-one-thing.py requirements.txt ​ Install dependencies Copy Ask AI pip install -r requirements.txt ​ Configure the environment Create a .env file with your Cartesia API key: macOS/Linux Windows (Command Prompt) Manual Method Copy Ask AI echo "CARTESIA_API_KEY=your_cartesia_api_key" > .env Copy Ask AI echo "CARTESIA_API_KEY=your_cartesia_api_key" > .env Copy Ask AI echo CARTESIA_API_KEY=your_cartesia_api_key > .env Create a file named .env in your project directory and add: Copy Ask AI CARTESIA_API_KEY=your_cartesia_api_key Replace your_cartesia_api_key with the actual API key you created during the installation step . ​ Run the example Start the bot with this command: Copy Ask AI python 01-say-one-thing.py You’ll see a URL (typically http://localhost:7860 ) in the console output. Open this URL in your browser to join the session. The bot will automatically join, greet you, and then disconnect. ​ Understanding the code Let’s examine the key components of 01-say-one-thing.py : Copy Ask AI # The `run_example()` function receives a transport which can be a # SmallWebRTCTransport, a DailyTransport or a FastAPIWebsocketTransport. The # global variable `transport_params` defines the transport settings, in this case # `audio_out_enabled=True` which means the example will send audio out (and you # will hear it). # Initialize Cartesia's text-to-speech service # Using a pre-selected British female voice # You can find other voices at https://play.cartesia.ai/voices tts = CartesiaTTSService( api_key = os.getenv( "CARTESIA_API_KEY" ), voice_id = "79a125e8-cd45-4c13-8a67-188112f4dd22" , # British Reading Lady ) # Create PipelineTask containing the Pipeline # The Pipeline processes text into speech and play it # 1. tts: Converts text messages into audio # 2. transport.output(): Sends audio into the WebRTC session task = PipelineTask(Pipeline([tts, transport.output()])) # When the client connects: # - Generate a greeting message # - Queue it for text-to-speech conversion # - Queue an EndFrame to shut down the pipeline when done @transport.event_handler ( "on_client_connected" ) async def on_client_connected ( transport , client ): await task.queue_frames([TTSSpeakFrame( f "Hello there!" ), EndFrame()]) # Create a pipeline runner and run the pipeline task runner = PipelineRunner( handle_sigint = False ) await runner.run(task) ​ Customize the example Try these simple modifications to enhance your bot: Change the voice Visit Cartesia’s voice library to find a different voice. Then update the voice_id parameter: Copy Ask AI tts = CartesiaTTSService( api_key = os.getenv( "CARTESIA_API_KEY" ), voice_id = "your_new_voice_id" , # Replace with a new voice ID ) Modify the greeting Change what the bot says when someone connects: Copy Ask AI await task.queue_frames([ TTSSpeakFrame( f "Welcome to Pipecat! I'm your virtual assistant." ), EndFrame() ]) Add a farewell message Add an additional message before ending: Copy Ask AI await task.queue_frames([ TTSSpeakFrame( f "Hello there!" ), TTSSpeakFrame( f "I hope you enjoy using Pipecat. Goodbye!" ), EndFrame() ]) ​ Next steps Try more examples Explore our GitHub repository with more sophisticated examples Core concepts Deepen your understanding of Pipecat’s architecture and components Supported services Integrate additional AI services like OpenAI, Anthropic, or Deepgram Join our community Get help, share projects, and connect with other Pipecat developers Installation & Setup Core Concepts On this page Set up your project Create a project directory Set up a virtual environment Download the example files Install dependencies Configure the environment Run the example Understanding the code Customize the example Next steps Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/getting-started_quickstart_bf21ab2d.txt b/getting-started_quickstart_bf21ab2d.txt
new file mode 100644
index 0000000000000000000000000000000000000000..77064bcee35eb325bab8d9aade370eb096faaa79
--- /dev/null
+++ b/getting-started_quickstart_bf21ab2d.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/getting-started/quickstart#next-steps
+Title: Quickstart - Pipecat
+==================================================
+
+Quickstart - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Get Started Quickstart Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Get Started Overview Installation & Setup Quickstart Core Concepts Next Steps & Examples In this quickstart, we’ll create a simple conversational bot that greets users when they join and exits when they leave. This example demonstrates the core components of a Pipecat application with a streamlined setup. ​ Set up your project ​ Create a project directory Copy Ask AI mkdir pipecat-quickstart && cd pipecat-quickstart ​ Set up a virtual environment macOS/Linux Windows Copy Ask AI python3 -m venv env source env/bin/activate Copy Ask AI python3 -m venv env source env/bin/activate Copy Ask AI python -m venv env source env/Scripts/activate # If using Git Bash # OR . \ env \ Scripts \ activate # If using Command Prompt # OR . \ env \ Scripts \ Activate.ps1 # If using PowerShell ​ Download the example files Linux/macOS Windows (PowerShell) Manual Download Copy Ask AI curl -O https://raw.githubusercontent.com/pipecat-ai/pipecat/main/examples/foundational/01-say-one-thing.py curl -O https://raw.githubusercontent.com/pipecat-ai/pipecat/main/examples/foundational/requirements.txt Copy Ask AI curl -O https://raw.githubusercontent.com/pipecat-ai/pipecat/main/examples/foundational/01-say-one-thing.py curl -O https://raw.githubusercontent.com/pipecat-ai/pipecat/main/examples/foundational/requirements.txt Copy Ask AI curl.exe - O https: // raw.githubusercontent.com / pipecat - ai / pipecat / main / examples / foundational / 01 - say - one - thing.py curl.exe - O https: // raw.githubusercontent.com / pipecat - ai / pipecat / main / examples / foundational / requirements.txt Download these files and save them to your project directory: 01-say-one-thing.py requirements.txt ​ Install dependencies Copy Ask AI pip install -r requirements.txt ​ Configure the environment Create a .env file with your Cartesia API key: macOS/Linux Windows (Command Prompt) Manual Method Copy Ask AI echo "CARTESIA_API_KEY=your_cartesia_api_key" > .env Copy Ask AI echo "CARTESIA_API_KEY=your_cartesia_api_key" > .env Copy Ask AI echo CARTESIA_API_KEY=your_cartesia_api_key > .env Create a file named .env in your project directory and add: Copy Ask AI CARTESIA_API_KEY=your_cartesia_api_key Replace your_cartesia_api_key with the actual API key you created during the installation step . ​ Run the example Start the bot with this command: Copy Ask AI python 01-say-one-thing.py You’ll see a URL (typically http://localhost:7860 ) in the console output. Open this URL in your browser to join the session. The bot will automatically join, greet you, and then disconnect. ​ Understanding the code Let’s examine the key components of 01-say-one-thing.py : Copy Ask AI # The `run_example()` function receives a transport which can be a # SmallWebRTCTransport, a DailyTransport or a FastAPIWebsocketTransport. The # global variable `transport_params` defines the transport settings, in this case # `audio_out_enabled=True` which means the example will send audio out (and you # will hear it). # Initialize Cartesia's text-to-speech service # Using a pre-selected British female voice # You can find other voices at https://play.cartesia.ai/voices tts = CartesiaTTSService( api_key = os.getenv( "CARTESIA_API_KEY" ), voice_id = "79a125e8-cd45-4c13-8a67-188112f4dd22" , # British Reading Lady ) # Create PipelineTask containing the Pipeline # The Pipeline processes text into speech and play it # 1. tts: Converts text messages into audio # 2. transport.output(): Sends audio into the WebRTC session task = PipelineTask(Pipeline([tts, transport.output()])) # When the client connects: # - Generate a greeting message # - Queue it for text-to-speech conversion # - Queue an EndFrame to shut down the pipeline when done @transport.event_handler ( "on_client_connected" ) async def on_client_connected ( transport , client ): await task.queue_frames([TTSSpeakFrame( f "Hello there!" ), EndFrame()]) # Create a pipeline runner and run the pipeline task runner = PipelineRunner( handle_sigint = False ) await runner.run(task) ​ Customize the example Try these simple modifications to enhance your bot: Change the voice Visit Cartesia’s voice library to find a different voice. Then update the voice_id parameter: Copy Ask AI tts = CartesiaTTSService( api_key = os.getenv( "CARTESIA_API_KEY" ), voice_id = "your_new_voice_id" , # Replace with a new voice ID ) Modify the greeting Change what the bot says when someone connects: Copy Ask AI await task.queue_frames([ TTSSpeakFrame( f "Welcome to Pipecat! I'm your virtual assistant." ), EndFrame() ]) Add a farewell message Add an additional message before ending: Copy Ask AI await task.queue_frames([ TTSSpeakFrame( f "Hello there!" ), TTSSpeakFrame( f "I hope you enjoy using Pipecat. Goodbye!" ), EndFrame() ]) ​ Next steps Try more examples Explore our GitHub repository with more sophisticated examples Core concepts Deepen your understanding of Pipecat’s architecture and components Supported services Integrate additional AI services like OpenAI, Anthropic, or Deepgram Join our community Get help, share projects, and connect with other Pipecat developers Installation & Setup Core Concepts On this page Set up your project Create a project directory Set up a virtual environment Download the example files Install dependencies Configure the environment Run the example Understanding the code Customize the example Next steps Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/getting-started_use-cases_b67ce2e4.txt b/getting-started_use-cases_b67ce2e4.txt
new file mode 100644
index 0000000000000000000000000000000000000000..95b8979deb28f19b4cffb90941be15a4425f7c7b
--- /dev/null
+++ b/getting-started_use-cases_b67ce2e4.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/getting-started/use-cases#how-it-works
+Title: Overview - Pipecat
+==================================================
+
+Overview - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Get Started Overview Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Get Started Overview Installation & Setup Quickstart Core Concepts Next Steps & Examples Pipecat is an open source Python framework that handles the complex orchestration of AI services, network transport, audio processing, and multimodal interactions. “Multimodal” means you can use any combination of audio, video, images, and/or text in your interactions. And “real-time” means that things are happening quickly enough that it feels conversational—a “back-and-forth” with a bot, not submitting a query and waiting for results. ​ What You Can Build Voice Assistants Natural, real-time conversations with AI using speech recognition and synthesis Interactive Agents Personal coaches and meeting assistants that can understand context and provide guidance Multimodal Apps Applications that combine voice, video, images, and text for rich interactions Creative Tools Storytelling experiences and social companions that engage users Business Solutions Customer intake flows and support bots for automated business processes Complex Flows Structured conversations using Pipecat Flows for managing complex interactions ​ How It Works The flow of interactions in a Pipecat application is typically straightforward: The bot says something The user says something The bot says something The user says something This continues until the conversation naturally ends. While this flow seems simple, making it feel natural requires sophisticated real-time processing. ​ Real-time Processing Pipecat’s pipeline architecture handles both simple voice interactions and complex multimodal processing. Let’s look at how data flows through the system: Voice app Multimodal app 1 Send Audio Transmit and capture streamed audio from the user 2 Transcribe Speech Convert speech to text as the user is talking 3 Process with LLM Generate responses using a large language model 4 Convert to Speech Transform text responses into natural speech 5 Play Audio Stream the audio response back to the user 1 Send Audio Transmit and capture streamed audio from the user 2 Transcribe Speech Convert speech to text as the user is talking 3 Process with LLM Generate responses using a large language model 4 Convert to Speech Transform text responses into natural speech 5 Play Audio Stream the audio response back to the user 1 Send Audio and Video Transmit and capture audio, video, and image inputs simultaneously 2 Process Streams Handle multiple input streams in parallel 3 Model Processing Send combined inputs to multimodal models (like GPT-4V) 4 Generate Outputs Create various outputs (text, images, audio, etc.) 5 Coordinate Presentation Synchronize and present multiple output types In both cases, Pipecat: Processes responses as they stream in Handles multiple input/output modalities concurrently Manages resource allocation and synchronization Coordinates parallel processing tasks This architecture creates fluid, natural interactions without noticeable delays, whether you’re building a simple voice assistant or a complex multimodal application. Pipecat’s pipeline architecture is particularly valuable for managing the complexity of real-time, multimodal interactions, ensuring smooth data flow and proper synchronization regardless of the input/output types involved. Pipecat handles all this complexity for you, letting you focus on building your application rather than managing the underlying infrastructure. ​ Next Steps Ready to build your first Pipecat application? Installation & Setup Prepare your environment and install required dependencies Quickstart Build and run your first Pipecat application Core Concepts Learn about pipelines, frames, and real-time processing Use Cases Explore example implementations and patterns ​ Join Our Community Discord Community Connect with other developers, share your projects, and get support from the Pipecat team. Installation & Setup On this page What You Can Build How It Works Real-time Processing Next Steps Join Our Community Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/getting-started_use-cases_b901ec26.txt b/getting-started_use-cases_b901ec26.txt
new file mode 100644
index 0000000000000000000000000000000000000000..641b50333bbce7ee3ffe78da1bd46f672243ddbb
--- /dev/null
+++ b/getting-started_use-cases_b901ec26.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/getting-started/use-cases#next-steps
+Title: Overview - Pipecat
+==================================================
+
+Overview - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Get Started Overview Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Get Started Overview Installation & Setup Quickstart Core Concepts Next Steps & Examples Pipecat is an open source Python framework that handles the complex orchestration of AI services, network transport, audio processing, and multimodal interactions. “Multimodal” means you can use any combination of audio, video, images, and/or text in your interactions. And “real-time” means that things are happening quickly enough that it feels conversational—a “back-and-forth” with a bot, not submitting a query and waiting for results. ​ What You Can Build Voice Assistants Natural, real-time conversations with AI using speech recognition and synthesis Interactive Agents Personal coaches and meeting assistants that can understand context and provide guidance Multimodal Apps Applications that combine voice, video, images, and text for rich interactions Creative Tools Storytelling experiences and social companions that engage users Business Solutions Customer intake flows and support bots for automated business processes Complex Flows Structured conversations using Pipecat Flows for managing complex interactions ​ How It Works The flow of interactions in a Pipecat application is typically straightforward: The bot says something The user says something The bot says something The user says something This continues until the conversation naturally ends. While this flow seems simple, making it feel natural requires sophisticated real-time processing. ​ Real-time Processing Pipecat’s pipeline architecture handles both simple voice interactions and complex multimodal processing. Let’s look at how data flows through the system: Voice app Multimodal app 1 Send Audio Transmit and capture streamed audio from the user 2 Transcribe Speech Convert speech to text as the user is talking 3 Process with LLM Generate responses using a large language model 4 Convert to Speech Transform text responses into natural speech 5 Play Audio Stream the audio response back to the user 1 Send Audio Transmit and capture streamed audio from the user 2 Transcribe Speech Convert speech to text as the user is talking 3 Process with LLM Generate responses using a large language model 4 Convert to Speech Transform text responses into natural speech 5 Play Audio Stream the audio response back to the user 1 Send Audio and Video Transmit and capture audio, video, and image inputs simultaneously 2 Process Streams Handle multiple input streams in parallel 3 Model Processing Send combined inputs to multimodal models (like GPT-4V) 4 Generate Outputs Create various outputs (text, images, audio, etc.) 5 Coordinate Presentation Synchronize and present multiple output types In both cases, Pipecat: Processes responses as they stream in Handles multiple input/output modalities concurrently Manages resource allocation and synchronization Coordinates parallel processing tasks This architecture creates fluid, natural interactions without noticeable delays, whether you’re building a simple voice assistant or a complex multimodal application. Pipecat’s pipeline architecture is particularly valuable for managing the complexity of real-time, multimodal interactions, ensuring smooth data flow and proper synchronization regardless of the input/output types involved. Pipecat handles all this complexity for you, letting you focus on building your application rather than managing the underlying infrastructure. ​ Next Steps Ready to build your first Pipecat application? Installation & Setup Prepare your environment and install required dependencies Quickstart Build and run your first Pipecat application Core Concepts Learn about pipelines, frames, and real-time processing Use Cases Explore example implementations and patterns ​ Join Our Community Discord Community Connect with other developers, share your projects, and get support from the Pipecat team. Installation & Setup On this page What You Can Build How It Works Real-time Processing Next Steps Join Our Community Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/homepage_267660fa.txt b/homepage_267660fa.txt
new file mode 100644
index 0000000000000000000000000000000000000000..233f01f6daa98e70df403f3e0999ad097994f6c1
--- /dev/null
+++ b/homepage_267660fa.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai#what-you-can-build
+Title: Overview - Pipecat
+==================================================
+
+Overview - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Get Started Overview Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Get Started Overview Installation & Setup Quickstart Core Concepts Next Steps & Examples Pipecat is an open source Python framework that handles the complex orchestration of AI services, network transport, audio processing, and multimodal interactions. “Multimodal” means you can use any combination of audio, video, images, and/or text in your interactions. And “real-time” means that things are happening quickly enough that it feels conversational—a “back-and-forth” with a bot, not submitting a query and waiting for results. ​ What You Can Build Voice Assistants Natural, real-time conversations with AI using speech recognition and synthesis Interactive Agents Personal coaches and meeting assistants that can understand context and provide guidance Multimodal Apps Applications that combine voice, video, images, and text for rich interactions Creative Tools Storytelling experiences and social companions that engage users Business Solutions Customer intake flows and support bots for automated business processes Complex Flows Structured conversations using Pipecat Flows for managing complex interactions ​ How It Works The flow of interactions in a Pipecat application is typically straightforward: The bot says something The user says something The bot says something The user says something This continues until the conversation naturally ends. While this flow seems simple, making it feel natural requires sophisticated real-time processing. ​ Real-time Processing Pipecat’s pipeline architecture handles both simple voice interactions and complex multimodal processing. Let’s look at how data flows through the system: Voice app Multimodal app 1 Send Audio Transmit and capture streamed audio from the user 2 Transcribe Speech Convert speech to text as the user is talking 3 Process with LLM Generate responses using a large language model 4 Convert to Speech Transform text responses into natural speech 5 Play Audio Stream the audio response back to the user 1 Send Audio Transmit and capture streamed audio from the user 2 Transcribe Speech Convert speech to text as the user is talking 3 Process with LLM Generate responses using a large language model 4 Convert to Speech Transform text responses into natural speech 5 Play Audio Stream the audio response back to the user 1 Send Audio and Video Transmit and capture audio, video, and image inputs simultaneously 2 Process Streams Handle multiple input streams in parallel 3 Model Processing Send combined inputs to multimodal models (like GPT-4V) 4 Generate Outputs Create various outputs (text, images, audio, etc.) 5 Coordinate Presentation Synchronize and present multiple output types In both cases, Pipecat: Processes responses as they stream in Handles multiple input/output modalities concurrently Manages resource allocation and synchronization Coordinates parallel processing tasks This architecture creates fluid, natural interactions without noticeable delays, whether you’re building a simple voice assistant or a complex multimodal application. Pipecat’s pipeline architecture is particularly valuable for managing the complexity of real-time, multimodal interactions, ensuring smooth data flow and proper synchronization regardless of the input/output types involved. Pipecat handles all this complexity for you, letting you focus on building your application rather than managing the underlying infrastructure. ​ Next Steps Ready to build your first Pipecat application? Installation & Setup Prepare your environment and install required dependencies Quickstart Build and run your first Pipecat application Core Concepts Learn about pipelines, frames, and real-time processing Use Cases Explore example implementations and patterns ​ Join Our Community Discord Community Connect with other developers, share your projects, and get support from the Pipecat team. Installation & Setup On this page What You Can Build How It Works Real-time Processing Next Steps Join Our Community Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/homepage_8e88c8ce.txt b/homepage_8e88c8ce.txt
new file mode 100644
index 0000000000000000000000000000000000000000..6dda4ab50f410a8ab5848143b84febfb8066459e
--- /dev/null
+++ b/homepage_8e88c8ce.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/#join-our-community
+Title: Overview - Pipecat
+==================================================
+
+Overview - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Get Started Overview Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Get Started Overview Installation & Setup Quickstart Core Concepts Next Steps & Examples Pipecat is an open source Python framework that handles the complex orchestration of AI services, network transport, audio processing, and multimodal interactions. “Multimodal” means you can use any combination of audio, video, images, and/or text in your interactions. And “real-time” means that things are happening quickly enough that it feels conversational—a “back-and-forth” with a bot, not submitting a query and waiting for results. ​ What You Can Build Voice Assistants Natural, real-time conversations with AI using speech recognition and synthesis Interactive Agents Personal coaches and meeting assistants that can understand context and provide guidance Multimodal Apps Applications that combine voice, video, images, and text for rich interactions Creative Tools Storytelling experiences and social companions that engage users Business Solutions Customer intake flows and support bots for automated business processes Complex Flows Structured conversations using Pipecat Flows for managing complex interactions ​ How It Works The flow of interactions in a Pipecat application is typically straightforward: The bot says something The user says something The bot says something The user says something This continues until the conversation naturally ends. While this flow seems simple, making it feel natural requires sophisticated real-time processing. ​ Real-time Processing Pipecat’s pipeline architecture handles both simple voice interactions and complex multimodal processing. Let’s look at how data flows through the system: Voice app Multimodal app 1 Send Audio Transmit and capture streamed audio from the user 2 Transcribe Speech Convert speech to text as the user is talking 3 Process with LLM Generate responses using a large language model 4 Convert to Speech Transform text responses into natural speech 5 Play Audio Stream the audio response back to the user 1 Send Audio Transmit and capture streamed audio from the user 2 Transcribe Speech Convert speech to text as the user is talking 3 Process with LLM Generate responses using a large language model 4 Convert to Speech Transform text responses into natural speech 5 Play Audio Stream the audio response back to the user 1 Send Audio and Video Transmit and capture audio, video, and image inputs simultaneously 2 Process Streams Handle multiple input streams in parallel 3 Model Processing Send combined inputs to multimodal models (like GPT-4V) 4 Generate Outputs Create various outputs (text, images, audio, etc.) 5 Coordinate Presentation Synchronize and present multiple output types In both cases, Pipecat: Processes responses as they stream in Handles multiple input/output modalities concurrently Manages resource allocation and synchronization Coordinates parallel processing tasks This architecture creates fluid, natural interactions without noticeable delays, whether you’re building a simple voice assistant or a complex multimodal application. Pipecat’s pipeline architecture is particularly valuable for managing the complexity of real-time, multimodal interactions, ensuring smooth data flow and proper synchronization regardless of the input/output types involved. Pipecat handles all this complexity for you, letting you focus on building your application rather than managing the underlying infrastructure. ​ Next Steps Ready to build your first Pipecat application? Installation & Setup Prepare your environment and install required dependencies Quickstart Build and run your first Pipecat application Core Concepts Learn about pipelines, frames, and real-time processing Use Cases Explore example implementations and patterns ​ Join Our Community Discord Community Connect with other developers, share your projects, and get support from the Pipecat team. Installation & Setup On this page What You Can Build How It Works Real-time Processing Next Steps Join Our Community Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/homepage_e1303335.txt b/homepage_e1303335.txt
new file mode 100644
index 0000000000000000000000000000000000000000..26e3b5749259bcaab265a501ff033fa2f4f07d3b
--- /dev/null
+++ b/homepage_e1303335.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai#how-it-works
+Title: Overview - Pipecat
+==================================================
+
+Overview - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Get Started Overview Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Get Started Overview Installation & Setup Quickstart Core Concepts Next Steps & Examples Pipecat is an open source Python framework that handles the complex orchestration of AI services, network transport, audio processing, and multimodal interactions. “Multimodal” means you can use any combination of audio, video, images, and/or text in your interactions. And “real-time” means that things are happening quickly enough that it feels conversational—a “back-and-forth” with a bot, not submitting a query and waiting for results. ​ What You Can Build Voice Assistants Natural, real-time conversations with AI using speech recognition and synthesis Interactive Agents Personal coaches and meeting assistants that can understand context and provide guidance Multimodal Apps Applications that combine voice, video, images, and text for rich interactions Creative Tools Storytelling experiences and social companions that engage users Business Solutions Customer intake flows and support bots for automated business processes Complex Flows Structured conversations using Pipecat Flows for managing complex interactions ​ How It Works The flow of interactions in a Pipecat application is typically straightforward: The bot says something The user says something The bot says something The user says something This continues until the conversation naturally ends. While this flow seems simple, making it feel natural requires sophisticated real-time processing. ​ Real-time Processing Pipecat’s pipeline architecture handles both simple voice interactions and complex multimodal processing. Let’s look at how data flows through the system: Voice app Multimodal app 1 Send Audio Transmit and capture streamed audio from the user 2 Transcribe Speech Convert speech to text as the user is talking 3 Process with LLM Generate responses using a large language model 4 Convert to Speech Transform text responses into natural speech 5 Play Audio Stream the audio response back to the user 1 Send Audio Transmit and capture streamed audio from the user 2 Transcribe Speech Convert speech to text as the user is talking 3 Process with LLM Generate responses using a large language model 4 Convert to Speech Transform text responses into natural speech 5 Play Audio Stream the audio response back to the user 1 Send Audio and Video Transmit and capture audio, video, and image inputs simultaneously 2 Process Streams Handle multiple input streams in parallel 3 Model Processing Send combined inputs to multimodal models (like GPT-4V) 4 Generate Outputs Create various outputs (text, images, audio, etc.) 5 Coordinate Presentation Synchronize and present multiple output types In both cases, Pipecat: Processes responses as they stream in Handles multiple input/output modalities concurrently Manages resource allocation and synchronization Coordinates parallel processing tasks This architecture creates fluid, natural interactions without noticeable delays, whether you’re building a simple voice assistant or a complex multimodal application. Pipecat’s pipeline architecture is particularly valuable for managing the complexity of real-time, multimodal interactions, ensuring smooth data flow and proper synchronization regardless of the input/output types involved. Pipecat handles all this complexity for you, letting you focus on building your application rather than managing the underlying infrastructure. ​ Next Steps Ready to build your first Pipecat application? Installation & Setup Prepare your environment and install required dependencies Quickstart Build and run your first Pipecat application Core Concepts Learn about pipelines, frames, and real-time processing Use Cases Explore example implementations and patterns ​ Join Our Community Discord Community Connect with other developers, share your projects, and get support from the Pipecat team. Installation & Setup On this page What You Can Build How It Works Real-time Processing Next Steps Join Our Community Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/image-generation_fal_302b3fbb.txt b/image-generation_fal_302b3fbb.txt
new file mode 100644
index 0000000000000000000000000000000000000000..bebe19f80fb63114c6c726802bdf1a0104c29573
--- /dev/null
+++ b/image-generation_fal_302b3fbb.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/image-generation/fal#param-size
+Title: fal - Pipecat
+==================================================
+
+fal - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Image Generation fal Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation fal Google Imagen OpenAI Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview FalImageGenService provides high-speed image generation capabilities using fal’s optimized Stable Diffusion XL models. It supports various image sizes, formats, and generation parameters with a focus on fast inference. ​ Installation To use FalImageGenService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[fal]" You’ll also need to set up your Fal API key as an environment variable: FAL_KEY You can obtain a fal API key by signing up at fal . ​ Configuration ​ Constructor Parameters ​ params InputParams required Generation parameters configuration ​ aiohttp_session aiohttp.ClientSession required HTTP session for image downloading ​ model str default: "fal-ai/fast-sdxl" Model identifier ​ key str Fal API key (alternative to environment variable) ​ Input Parameters Copy Ask AI class InputParams ( BaseModel ): seed: Optional[ int ] = None # Random seed for reproducibility num_inference_steps: int = 8 # Number of denoising steps num_images: int = 1 # Number of images to generate image_size: Union[ str , Dict[ str , int ]] = "square_hd" # Image dimensions expand_prompt: bool = False # Enhance prompt automatically enable_safety_checker: bool = True # Filter unsafe content format : str = "png" # Output image format ​ Supported Image Sizes Possible enum values: square_hd , square , portrait_4_3 , portrait_16_9 , landscape_4_3 , landscape_16_9 Note: For custom image sizes, you can pass the width and height as an object: Copy Ask AI { "image_size" : { "width" : 1280 , "height" : 720 } } See the fal docs for more information. ​ Output Frames ​ URLImageRawFrame ​ url string Generated image URL ​ image bytes Raw image data ​ size tuple Image dimensions (width, height) ​ format string Image format (e.g., ‘PNG’) ​ ErrorFrame ​ error string Error information if generation fails ​ Methods See the Image Generation base class methods for additional functionality. ​ Usage Example Copy Ask AI import aiohttp from pipecat.services.fal.image import FalImageGenService # Configure service async with aiohttp.ClientSession() as session: service = FalImageGenService( model = "fal-ai/fast-sdxl" , aiohttp_session = session, params = FalImageGenService.InputParams( num_inference_steps = 8 , image_size = "portrait_hd" , expand_prompt = True ) ) # Use in pipeline pipeline = Pipeline([ prompt_input, # Produces text prompts service, # Generates images image_handler # Handles generated images ]) ​ Frame Flow ​ Metrics Support The service collects processing metrics: Generation time Download time API response time Total processing duration ​ Notes Fast inference times with optimized models Supports various image sizes and formats Automatic prompt enhancement option Built-in safety filtering Asynchronous operation Efficient HTTP session management Comprehensive error handling OpenAI Realtime Beta Google Imagen On this page Overview Installation Configuration Constructor Parameters Input Parameters Supported Image Sizes Output Frames URLImageRawFrame ErrorFrame Methods Usage Example Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/image-generation_fal_3d0c1e98.txt b/image-generation_fal_3d0c1e98.txt
new file mode 100644
index 0000000000000000000000000000000000000000..7ea58b0e85d7bf73680ff06086a52a7778c01f3c
--- /dev/null
+++ b/image-generation_fal_3d0c1e98.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/image-generation/fal#input-parameters
+Title: fal - Pipecat
+==================================================
+
+fal - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Image Generation fal Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation fal Google Imagen OpenAI Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview FalImageGenService provides high-speed image generation capabilities using fal’s optimized Stable Diffusion XL models. It supports various image sizes, formats, and generation parameters with a focus on fast inference. ​ Installation To use FalImageGenService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[fal]" You’ll also need to set up your Fal API key as an environment variable: FAL_KEY You can obtain a fal API key by signing up at fal . ​ Configuration ​ Constructor Parameters ​ params InputParams required Generation parameters configuration ​ aiohttp_session aiohttp.ClientSession required HTTP session for image downloading ​ model str default: "fal-ai/fast-sdxl" Model identifier ​ key str Fal API key (alternative to environment variable) ​ Input Parameters Copy Ask AI class InputParams ( BaseModel ): seed: Optional[ int ] = None # Random seed for reproducibility num_inference_steps: int = 8 # Number of denoising steps num_images: int = 1 # Number of images to generate image_size: Union[ str , Dict[ str , int ]] = "square_hd" # Image dimensions expand_prompt: bool = False # Enhance prompt automatically enable_safety_checker: bool = True # Filter unsafe content format : str = "png" # Output image format ​ Supported Image Sizes Possible enum values: square_hd , square , portrait_4_3 , portrait_16_9 , landscape_4_3 , landscape_16_9 Note: For custom image sizes, you can pass the width and height as an object: Copy Ask AI { "image_size" : { "width" : 1280 , "height" : 720 } } See the fal docs for more information. ​ Output Frames ​ URLImageRawFrame ​ url string Generated image URL ​ image bytes Raw image data ​ size tuple Image dimensions (width, height) ​ format string Image format (e.g., ‘PNG’) ​ ErrorFrame ​ error string Error information if generation fails ​ Methods See the Image Generation base class methods for additional functionality. ​ Usage Example Copy Ask AI import aiohttp from pipecat.services.fal.image import FalImageGenService # Configure service async with aiohttp.ClientSession() as session: service = FalImageGenService( model = "fal-ai/fast-sdxl" , aiohttp_session = session, params = FalImageGenService.InputParams( num_inference_steps = 8 , image_size = "portrait_hd" , expand_prompt = True ) ) # Use in pipeline pipeline = Pipeline([ prompt_input, # Produces text prompts service, # Generates images image_handler # Handles generated images ]) ​ Frame Flow ​ Metrics Support The service collects processing metrics: Generation time Download time API response time Total processing duration ​ Notes Fast inference times with optimized models Supports various image sizes and formats Automatic prompt enhancement option Built-in safety filtering Asynchronous operation Efficient HTTP session management Comprehensive error handling OpenAI Realtime Beta Google Imagen On this page Overview Installation Configuration Constructor Parameters Input Parameters Supported Image Sizes Output Frames URLImageRawFrame ErrorFrame Methods Usage Example Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/image-generation_fal_6f22eb83.txt b/image-generation_fal_6f22eb83.txt
new file mode 100644
index 0000000000000000000000000000000000000000..5ca0a05a46daddd287b896013d1397502babfd61
--- /dev/null
+++ b/image-generation_fal_6f22eb83.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/image-generation/fal#param-model
+Title: fal - Pipecat
+==================================================
+
+fal - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Image Generation fal Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation fal Google Imagen OpenAI Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview FalImageGenService provides high-speed image generation capabilities using fal’s optimized Stable Diffusion XL models. It supports various image sizes, formats, and generation parameters with a focus on fast inference. ​ Installation To use FalImageGenService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[fal]" You’ll also need to set up your Fal API key as an environment variable: FAL_KEY You can obtain a fal API key by signing up at fal . ​ Configuration ​ Constructor Parameters ​ params InputParams required Generation parameters configuration ​ aiohttp_session aiohttp.ClientSession required HTTP session for image downloading ​ model str default: "fal-ai/fast-sdxl" Model identifier ​ key str Fal API key (alternative to environment variable) ​ Input Parameters Copy Ask AI class InputParams ( BaseModel ): seed: Optional[ int ] = None # Random seed for reproducibility num_inference_steps: int = 8 # Number of denoising steps num_images: int = 1 # Number of images to generate image_size: Union[ str , Dict[ str , int ]] = "square_hd" # Image dimensions expand_prompt: bool = False # Enhance prompt automatically enable_safety_checker: bool = True # Filter unsafe content format : str = "png" # Output image format ​ Supported Image Sizes Possible enum values: square_hd , square , portrait_4_3 , portrait_16_9 , landscape_4_3 , landscape_16_9 Note: For custom image sizes, you can pass the width and height as an object: Copy Ask AI { "image_size" : { "width" : 1280 , "height" : 720 } } See the fal docs for more information. ​ Output Frames ​ URLImageRawFrame ​ url string Generated image URL ​ image bytes Raw image data ​ size tuple Image dimensions (width, height) ​ format string Image format (e.g., ‘PNG’) ​ ErrorFrame ​ error string Error information if generation fails ​ Methods See the Image Generation base class methods for additional functionality. ​ Usage Example Copy Ask AI import aiohttp from pipecat.services.fal.image import FalImageGenService # Configure service async with aiohttp.ClientSession() as session: service = FalImageGenService( model = "fal-ai/fast-sdxl" , aiohttp_session = session, params = FalImageGenService.InputParams( num_inference_steps = 8 , image_size = "portrait_hd" , expand_prompt = True ) ) # Use in pipeline pipeline = Pipeline([ prompt_input, # Produces text prompts service, # Generates images image_handler # Handles generated images ]) ​ Frame Flow ​ Metrics Support The service collects processing metrics: Generation time Download time API response time Total processing duration ​ Notes Fast inference times with optimized models Supports various image sizes and formats Automatic prompt enhancement option Built-in safety filtering Asynchronous operation Efficient HTTP session management Comprehensive error handling OpenAI Realtime Beta Google Imagen On this page Overview Installation Configuration Constructor Parameters Input Parameters Supported Image Sizes Output Frames URLImageRawFrame ErrorFrame Methods Usage Example Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/image-generation_fal_706423b0.txt b/image-generation_fal_706423b0.txt
new file mode 100644
index 0000000000000000000000000000000000000000..a8d45289f7a342f5e82c80a975feaaabb302a44e
--- /dev/null
+++ b/image-generation_fal_706423b0.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/image-generation/fal#configuration
+Title: fal - Pipecat
+==================================================
+
+fal - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Image Generation fal Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation fal Google Imagen OpenAI Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview FalImageGenService provides high-speed image generation capabilities using fal’s optimized Stable Diffusion XL models. It supports various image sizes, formats, and generation parameters with a focus on fast inference. ​ Installation To use FalImageGenService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[fal]" You’ll also need to set up your Fal API key as an environment variable: FAL_KEY You can obtain a fal API key by signing up at fal . ​ Configuration ​ Constructor Parameters ​ params InputParams required Generation parameters configuration ​ aiohttp_session aiohttp.ClientSession required HTTP session for image downloading ​ model str default: "fal-ai/fast-sdxl" Model identifier ​ key str Fal API key (alternative to environment variable) ​ Input Parameters Copy Ask AI class InputParams ( BaseModel ): seed: Optional[ int ] = None # Random seed for reproducibility num_inference_steps: int = 8 # Number of denoising steps num_images: int = 1 # Number of images to generate image_size: Union[ str , Dict[ str , int ]] = "square_hd" # Image dimensions expand_prompt: bool = False # Enhance prompt automatically enable_safety_checker: bool = True # Filter unsafe content format : str = "png" # Output image format ​ Supported Image Sizes Possible enum values: square_hd , square , portrait_4_3 , portrait_16_9 , landscape_4_3 , landscape_16_9 Note: For custom image sizes, you can pass the width and height as an object: Copy Ask AI { "image_size" : { "width" : 1280 , "height" : 720 } } See the fal docs for more information. ​ Output Frames ​ URLImageRawFrame ​ url string Generated image URL ​ image bytes Raw image data ​ size tuple Image dimensions (width, height) ​ format string Image format (e.g., ‘PNG’) ​ ErrorFrame ​ error string Error information if generation fails ​ Methods See the Image Generation base class methods for additional functionality. ​ Usage Example Copy Ask AI import aiohttp from pipecat.services.fal.image import FalImageGenService # Configure service async with aiohttp.ClientSession() as session: service = FalImageGenService( model = "fal-ai/fast-sdxl" , aiohttp_session = session, params = FalImageGenService.InputParams( num_inference_steps = 8 , image_size = "portrait_hd" , expand_prompt = True ) ) # Use in pipeline pipeline = Pipeline([ prompt_input, # Produces text prompts service, # Generates images image_handler # Handles generated images ]) ​ Frame Flow ​ Metrics Support The service collects processing metrics: Generation time Download time API response time Total processing duration ​ Notes Fast inference times with optimized models Supports various image sizes and formats Automatic prompt enhancement option Built-in safety filtering Asynchronous operation Efficient HTTP session management Comprehensive error handling OpenAI Realtime Beta Google Imagen On this page Overview Installation Configuration Constructor Parameters Input Parameters Supported Image Sizes Output Frames URLImageRawFrame ErrorFrame Methods Usage Example Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/image-generation_fal_9fd62e31.txt b/image-generation_fal_9fd62e31.txt
new file mode 100644
index 0000000000000000000000000000000000000000..42ec739115caa911170adfdc13bd2c488676c202
--- /dev/null
+++ b/image-generation_fal_9fd62e31.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/image-generation/fal#param-key
+Title: fal - Pipecat
+==================================================
+
+fal - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Image Generation fal Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation fal Google Imagen OpenAI Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview FalImageGenService provides high-speed image generation capabilities using fal’s optimized Stable Diffusion XL models. It supports various image sizes, formats, and generation parameters with a focus on fast inference. ​ Installation To use FalImageGenService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[fal]" You’ll also need to set up your Fal API key as an environment variable: FAL_KEY You can obtain a fal API key by signing up at fal . ​ Configuration ​ Constructor Parameters ​ params InputParams required Generation parameters configuration ​ aiohttp_session aiohttp.ClientSession required HTTP session for image downloading ​ model str default: "fal-ai/fast-sdxl" Model identifier ​ key str Fal API key (alternative to environment variable) ​ Input Parameters Copy Ask AI class InputParams ( BaseModel ): seed: Optional[ int ] = None # Random seed for reproducibility num_inference_steps: int = 8 # Number of denoising steps num_images: int = 1 # Number of images to generate image_size: Union[ str , Dict[ str , int ]] = "square_hd" # Image dimensions expand_prompt: bool = False # Enhance prompt automatically enable_safety_checker: bool = True # Filter unsafe content format : str = "png" # Output image format ​ Supported Image Sizes Possible enum values: square_hd , square , portrait_4_3 , portrait_16_9 , landscape_4_3 , landscape_16_9 Note: For custom image sizes, you can pass the width and height as an object: Copy Ask AI { "image_size" : { "width" : 1280 , "height" : 720 } } See the fal docs for more information. ​ Output Frames ​ URLImageRawFrame ​ url string Generated image URL ​ image bytes Raw image data ​ size tuple Image dimensions (width, height) ​ format string Image format (e.g., ‘PNG’) ​ ErrorFrame ​ error string Error information if generation fails ​ Methods See the Image Generation base class methods for additional functionality. ​ Usage Example Copy Ask AI import aiohttp from pipecat.services.fal.image import FalImageGenService # Configure service async with aiohttp.ClientSession() as session: service = FalImageGenService( model = "fal-ai/fast-sdxl" , aiohttp_session = session, params = FalImageGenService.InputParams( num_inference_steps = 8 , image_size = "portrait_hd" , expand_prompt = True ) ) # Use in pipeline pipeline = Pipeline([ prompt_input, # Produces text prompts service, # Generates images image_handler # Handles generated images ]) ​ Frame Flow ​ Metrics Support The service collects processing metrics: Generation time Download time API response time Total processing duration ​ Notes Fast inference times with optimized models Supports various image sizes and formats Automatic prompt enhancement option Built-in safety filtering Asynchronous operation Efficient HTTP session management Comprehensive error handling OpenAI Realtime Beta Google Imagen On this page Overview Installation Configuration Constructor Parameters Input Parameters Supported Image Sizes Output Frames URLImageRawFrame ErrorFrame Methods Usage Example Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/image-generation_fal_c0e8d6d4.txt b/image-generation_fal_c0e8d6d4.txt
new file mode 100644
index 0000000000000000000000000000000000000000..5ccab616e50112226b82b0533ce36fed27e91b4f
--- /dev/null
+++ b/image-generation_fal_c0e8d6d4.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/image-generation/fal#output-frames
+Title: fal - Pipecat
+==================================================
+
+fal - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Image Generation fal Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation fal Google Imagen OpenAI Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview FalImageGenService provides high-speed image generation capabilities using fal’s optimized Stable Diffusion XL models. It supports various image sizes, formats, and generation parameters with a focus on fast inference. ​ Installation To use FalImageGenService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[fal]" You’ll also need to set up your Fal API key as an environment variable: FAL_KEY You can obtain a fal API key by signing up at fal . ​ Configuration ​ Constructor Parameters ​ params InputParams required Generation parameters configuration ​ aiohttp_session aiohttp.ClientSession required HTTP session for image downloading ​ model str default: "fal-ai/fast-sdxl" Model identifier ​ key str Fal API key (alternative to environment variable) ​ Input Parameters Copy Ask AI class InputParams ( BaseModel ): seed: Optional[ int ] = None # Random seed for reproducibility num_inference_steps: int = 8 # Number of denoising steps num_images: int = 1 # Number of images to generate image_size: Union[ str , Dict[ str , int ]] = "square_hd" # Image dimensions expand_prompt: bool = False # Enhance prompt automatically enable_safety_checker: bool = True # Filter unsafe content format : str = "png" # Output image format ​ Supported Image Sizes Possible enum values: square_hd , square , portrait_4_3 , portrait_16_9 , landscape_4_3 , landscape_16_9 Note: For custom image sizes, you can pass the width and height as an object: Copy Ask AI { "image_size" : { "width" : 1280 , "height" : 720 } } See the fal docs for more information. ​ Output Frames ​ URLImageRawFrame ​ url string Generated image URL ​ image bytes Raw image data ​ size tuple Image dimensions (width, height) ​ format string Image format (e.g., ‘PNG’) ​ ErrorFrame ​ error string Error information if generation fails ​ Methods See the Image Generation base class methods for additional functionality. ​ Usage Example Copy Ask AI import aiohttp from pipecat.services.fal.image import FalImageGenService # Configure service async with aiohttp.ClientSession() as session: service = FalImageGenService( model = "fal-ai/fast-sdxl" , aiohttp_session = session, params = FalImageGenService.InputParams( num_inference_steps = 8 , image_size = "portrait_hd" , expand_prompt = True ) ) # Use in pipeline pipeline = Pipeline([ prompt_input, # Produces text prompts service, # Generates images image_handler # Handles generated images ]) ​ Frame Flow ​ Metrics Support The service collects processing metrics: Generation time Download time API response time Total processing duration ​ Notes Fast inference times with optimized models Supports various image sizes and formats Automatic prompt enhancement option Built-in safety filtering Asynchronous operation Efficient HTTP session management Comprehensive error handling OpenAI Realtime Beta Google Imagen On this page Overview Installation Configuration Constructor Parameters Input Parameters Supported Image Sizes Output Frames URLImageRawFrame ErrorFrame Methods Usage Example Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/image-generation_fal_d4f42a4a.txt b/image-generation_fal_d4f42a4a.txt
new file mode 100644
index 0000000000000000000000000000000000000000..c77b6d8033599dac38333d2b6d7996f495edc0bc
--- /dev/null
+++ b/image-generation_fal_d4f42a4a.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/image-generation/fal
+Title: fal - Pipecat
+==================================================
+
+fal - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Image Generation fal Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation fal Google Imagen OpenAI Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview FalImageGenService provides high-speed image generation capabilities using fal’s optimized Stable Diffusion XL models. It supports various image sizes, formats, and generation parameters with a focus on fast inference. ​ Installation To use FalImageGenService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[fal]" You’ll also need to set up your Fal API key as an environment variable: FAL_KEY You can obtain a fal API key by signing up at fal . ​ Configuration ​ Constructor Parameters ​ params InputParams required Generation parameters configuration ​ aiohttp_session aiohttp.ClientSession required HTTP session for image downloading ​ model str default: "fal-ai/fast-sdxl" Model identifier ​ key str Fal API key (alternative to environment variable) ​ Input Parameters Copy Ask AI class InputParams ( BaseModel ): seed: Optional[ int ] = None # Random seed for reproducibility num_inference_steps: int = 8 # Number of denoising steps num_images: int = 1 # Number of images to generate image_size: Union[ str , Dict[ str , int ]] = "square_hd" # Image dimensions expand_prompt: bool = False # Enhance prompt automatically enable_safety_checker: bool = True # Filter unsafe content format : str = "png" # Output image format ​ Supported Image Sizes Possible enum values: square_hd , square , portrait_4_3 , portrait_16_9 , landscape_4_3 , landscape_16_9 Note: For custom image sizes, you can pass the width and height as an object: Copy Ask AI { "image_size" : { "width" : 1280 , "height" : 720 } } See the fal docs for more information. ​ Output Frames ​ URLImageRawFrame ​ url string Generated image URL ​ image bytes Raw image data ​ size tuple Image dimensions (width, height) ​ format string Image format (e.g., ‘PNG’) ​ ErrorFrame ​ error string Error information if generation fails ​ Methods See the Image Generation base class methods for additional functionality. ​ Usage Example Copy Ask AI import aiohttp from pipecat.services.fal.image import FalImageGenService # Configure service async with aiohttp.ClientSession() as session: service = FalImageGenService( model = "fal-ai/fast-sdxl" , aiohttp_session = session, params = FalImageGenService.InputParams( num_inference_steps = 8 , image_size = "portrait_hd" , expand_prompt = True ) ) # Use in pipeline pipeline = Pipeline([ prompt_input, # Produces text prompts service, # Generates images image_handler # Handles generated images ]) ​ Frame Flow ​ Metrics Support The service collects processing metrics: Generation time Download time API response time Total processing duration ​ Notes Fast inference times with optimized models Supports various image sizes and formats Automatic prompt enhancement option Built-in safety filtering Asynchronous operation Efficient HTTP session management Comprehensive error handling OpenAI Realtime Beta Google Imagen On this page Overview Installation Configuration Constructor Parameters Input Parameters Supported Image Sizes Output Frames URLImageRawFrame ErrorFrame Methods Usage Example Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/image-generation_fal_d929b087.txt b/image-generation_fal_d929b087.txt
new file mode 100644
index 0000000000000000000000000000000000000000..7b2b7d59f58e0d89fe62baf7fb7c1e4f23077618
--- /dev/null
+++ b/image-generation_fal_d929b087.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/image-generation/fal#usage-example
+Title: fal - Pipecat
+==================================================
+
+fal - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Image Generation fal Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation fal Google Imagen OpenAI Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview FalImageGenService provides high-speed image generation capabilities using fal’s optimized Stable Diffusion XL models. It supports various image sizes, formats, and generation parameters with a focus on fast inference. ​ Installation To use FalImageGenService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[fal]" You’ll also need to set up your Fal API key as an environment variable: FAL_KEY You can obtain a fal API key by signing up at fal . ​ Configuration ​ Constructor Parameters ​ params InputParams required Generation parameters configuration ​ aiohttp_session aiohttp.ClientSession required HTTP session for image downloading ​ model str default: "fal-ai/fast-sdxl" Model identifier ​ key str Fal API key (alternative to environment variable) ​ Input Parameters Copy Ask AI class InputParams ( BaseModel ): seed: Optional[ int ] = None # Random seed for reproducibility num_inference_steps: int = 8 # Number of denoising steps num_images: int = 1 # Number of images to generate image_size: Union[ str , Dict[ str , int ]] = "square_hd" # Image dimensions expand_prompt: bool = False # Enhance prompt automatically enable_safety_checker: bool = True # Filter unsafe content format : str = "png" # Output image format ​ Supported Image Sizes Possible enum values: square_hd , square , portrait_4_3 , portrait_16_9 , landscape_4_3 , landscape_16_9 Note: For custom image sizes, you can pass the width and height as an object: Copy Ask AI { "image_size" : { "width" : 1280 , "height" : 720 } } See the fal docs for more information. ​ Output Frames ​ URLImageRawFrame ​ url string Generated image URL ​ image bytes Raw image data ​ size tuple Image dimensions (width, height) ​ format string Image format (e.g., ‘PNG’) ​ ErrorFrame ​ error string Error information if generation fails ​ Methods See the Image Generation base class methods for additional functionality. ​ Usage Example Copy Ask AI import aiohttp from pipecat.services.fal.image import FalImageGenService # Configure service async with aiohttp.ClientSession() as session: service = FalImageGenService( model = "fal-ai/fast-sdxl" , aiohttp_session = session, params = FalImageGenService.InputParams( num_inference_steps = 8 , image_size = "portrait_hd" , expand_prompt = True ) ) # Use in pipeline pipeline = Pipeline([ prompt_input, # Produces text prompts service, # Generates images image_handler # Handles generated images ]) ​ Frame Flow ​ Metrics Support The service collects processing metrics: Generation time Download time API response time Total processing duration ​ Notes Fast inference times with optimized models Supports various image sizes and formats Automatic prompt enhancement option Built-in safety filtering Asynchronous operation Efficient HTTP session management Comprehensive error handling OpenAI Realtime Beta Google Imagen On this page Overview Installation Configuration Constructor Parameters Input Parameters Supported Image Sizes Output Frames URLImageRawFrame ErrorFrame Methods Usage Example Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/image-generation_fal_fabba98a.txt b/image-generation_fal_fabba98a.txt
new file mode 100644
index 0000000000000000000000000000000000000000..aea3b7a8b40052b53063c1381423ac650841bbc7
--- /dev/null
+++ b/image-generation_fal_fabba98a.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/image-generation/fal#param-image
+Title: fal - Pipecat
+==================================================
+
+fal - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Image Generation fal Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation fal Google Imagen OpenAI Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview FalImageGenService provides high-speed image generation capabilities using fal’s optimized Stable Diffusion XL models. It supports various image sizes, formats, and generation parameters with a focus on fast inference. ​ Installation To use FalImageGenService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[fal]" You’ll also need to set up your Fal API key as an environment variable: FAL_KEY You can obtain a fal API key by signing up at fal . ​ Configuration ​ Constructor Parameters ​ params InputParams required Generation parameters configuration ​ aiohttp_session aiohttp.ClientSession required HTTP session for image downloading ​ model str default: "fal-ai/fast-sdxl" Model identifier ​ key str Fal API key (alternative to environment variable) ​ Input Parameters Copy Ask AI class InputParams ( BaseModel ): seed: Optional[ int ] = None # Random seed for reproducibility num_inference_steps: int = 8 # Number of denoising steps num_images: int = 1 # Number of images to generate image_size: Union[ str , Dict[ str , int ]] = "square_hd" # Image dimensions expand_prompt: bool = False # Enhance prompt automatically enable_safety_checker: bool = True # Filter unsafe content format : str = "png" # Output image format ​ Supported Image Sizes Possible enum values: square_hd , square , portrait_4_3 , portrait_16_9 , landscape_4_3 , landscape_16_9 Note: For custom image sizes, you can pass the width and height as an object: Copy Ask AI { "image_size" : { "width" : 1280 , "height" : 720 } } See the fal docs for more information. ​ Output Frames ​ URLImageRawFrame ​ url string Generated image URL ​ image bytes Raw image data ​ size tuple Image dimensions (width, height) ​ format string Image format (e.g., ‘PNG’) ​ ErrorFrame ​ error string Error information if generation fails ​ Methods See the Image Generation base class methods for additional functionality. ​ Usage Example Copy Ask AI import aiohttp from pipecat.services.fal.image import FalImageGenService # Configure service async with aiohttp.ClientSession() as session: service = FalImageGenService( model = "fal-ai/fast-sdxl" , aiohttp_session = session, params = FalImageGenService.InputParams( num_inference_steps = 8 , image_size = "portrait_hd" , expand_prompt = True ) ) # Use in pipeline pipeline = Pipeline([ prompt_input, # Produces text prompts service, # Generates images image_handler # Handles generated images ]) ​ Frame Flow ​ Metrics Support The service collects processing metrics: Generation time Download time API response time Total processing duration ​ Notes Fast inference times with optimized models Supports various image sizes and formats Automatic prompt enhancement option Built-in safety filtering Asynchronous operation Efficient HTTP session management Comprehensive error handling OpenAI Realtime Beta Google Imagen On this page Overview Installation Configuration Constructor Parameters Input Parameters Supported Image Sizes Output Frames URLImageRawFrame ErrorFrame Methods Usage Example Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/image-generation_google-imagen_13b4b831.txt b/image-generation_google-imagen_13b4b831.txt
new file mode 100644
index 0000000000000000000000000000000000000000..0678d3289b41eb88554c3c81474a5892bf6da20d
--- /dev/null
+++ b/image-generation_google-imagen_13b4b831.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/image-generation/google-imagen#errorframe
+Title: Google Imagen - Pipecat
+==================================================
+
+Google Imagen - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Image Generation Google Imagen Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation fal Google Imagen OpenAI Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview GoogleImageGenService provides high-quality image generation capabilities using Google’s Imagen models. It supports generating multiple images from text prompts with various customization options. ​ Installation To use GoogleImageGenService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[google]" You’ll also need to set up your Google API key as an environment variable: GOOGLE_API_KEY ​ Configuration ​ Constructor Parameters ​ params InputParams default: "InputParams()" Generation parameters configuration ​ api_key str required Google API key for authentication ​ Input Parameters ​ number_of_images int default: "1" Number of images to generate (1-8) ​ model str default: "imagen-3.0-generate-002" Model identifier ​ negative_prompt str default: "None" Elements to exclude from generation ​ Input The service accepts text prompts through its image generation pipeline. ​ Output Frames ​ URLImageRawFrame ​ url string Generated image URL (null for Google implementation as it returns raw bytes) ​ image bytes Raw image data ​ size tuple Image dimensions (width, height) ​ format string Image format (e.g., ‘JPEG’) ​ ErrorFrame ​ error string Error information if generation fails ​ Usage Example Copy Ask AI from pipecat.services.google.image import GoogleImageGenService # Configure service image_gen = GoogleImageGenService( api_key = "your-google-api-key" , params = GoogleImageGenService.InputParams( number_of_images = 2 , model = "imagen-3.0-generate-002" , negative_prompt = "blurry, distorted, low quality" ) ) # Use in pipeline main_pipeline = Pipeline( [ transport.input(), context_aggregator.user(), llm_service, image_gen, tts_service, transport.output(), context_aggregator.assistant(), ] ) ​ Frame Flow ​ Metrics Support The service supports metrics collection: Time to First Byte (TTFB) Processing duration API response metrics ​ Model Support Google’s Imagen service offers different model variants: Model ID Description imagen-3.0-generate-002 Latest Imagen model with high-quality outputs See other available models in Google’s Imagen documentation . ​ Error Handling Copy Ask AI try : async for frame in service.run_image_gen(prompt): if isinstance (frame, ErrorFrame): handle_error(frame.error) except Exception as e: logger.error( f "Image generation error: { e } " ) fal OpenAI On this page Overview Installation Configuration Constructor Parameters Input Parameters Input Output Frames URLImageRawFrame ErrorFrame Usage Example Frame Flow Metrics Support Model Support Error Handling Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/image-generation_google-imagen_256c34ea.txt b/image-generation_google-imagen_256c34ea.txt
new file mode 100644
index 0000000000000000000000000000000000000000..62a10784a0df0e3ff33d5b41d915c86f9f9a935b
--- /dev/null
+++ b/image-generation_google-imagen_256c34ea.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/image-generation/google-imagen#configuration
+Title: Google Imagen - Pipecat
+==================================================
+
+Google Imagen - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Image Generation Google Imagen Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation fal Google Imagen OpenAI Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview GoogleImageGenService provides high-quality image generation capabilities using Google’s Imagen models. It supports generating multiple images from text prompts with various customization options. ​ Installation To use GoogleImageGenService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[google]" You’ll also need to set up your Google API key as an environment variable: GOOGLE_API_KEY ​ Configuration ​ Constructor Parameters ​ params InputParams default: "InputParams()" Generation parameters configuration ​ api_key str required Google API key for authentication ​ Input Parameters ​ number_of_images int default: "1" Number of images to generate (1-8) ​ model str default: "imagen-3.0-generate-002" Model identifier ​ negative_prompt str default: "None" Elements to exclude from generation ​ Input The service accepts text prompts through its image generation pipeline. ​ Output Frames ​ URLImageRawFrame ​ url string Generated image URL (null for Google implementation as it returns raw bytes) ​ image bytes Raw image data ​ size tuple Image dimensions (width, height) ​ format string Image format (e.g., ‘JPEG’) ​ ErrorFrame ​ error string Error information if generation fails ​ Usage Example Copy Ask AI from pipecat.services.google.image import GoogleImageGenService # Configure service image_gen = GoogleImageGenService( api_key = "your-google-api-key" , params = GoogleImageGenService.InputParams( number_of_images = 2 , model = "imagen-3.0-generate-002" , negative_prompt = "blurry, distorted, low quality" ) ) # Use in pipeline main_pipeline = Pipeline( [ transport.input(), context_aggregator.user(), llm_service, image_gen, tts_service, transport.output(), context_aggregator.assistant(), ] ) ​ Frame Flow ​ Metrics Support The service supports metrics collection: Time to First Byte (TTFB) Processing duration API response metrics ​ Model Support Google’s Imagen service offers different model variants: Model ID Description imagen-3.0-generate-002 Latest Imagen model with high-quality outputs See other available models in Google’s Imagen documentation . ​ Error Handling Copy Ask AI try : async for frame in service.run_image_gen(prompt): if isinstance (frame, ErrorFrame): handle_error(frame.error) except Exception as e: logger.error( f "Image generation error: { e } " ) fal OpenAI On this page Overview Installation Configuration Constructor Parameters Input Parameters Input Output Frames URLImageRawFrame ErrorFrame Usage Example Frame Flow Metrics Support Model Support Error Handling Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/image-generation_google-imagen_468ba298.txt b/image-generation_google-imagen_468ba298.txt
new file mode 100644
index 0000000000000000000000000000000000000000..d3e0f75b708ec49bd719ff9ec11afafb0956ef5e
--- /dev/null
+++ b/image-generation_google-imagen_468ba298.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/image-generation/google-imagen#param-number-of-images
+Title: Google Imagen - Pipecat
+==================================================
+
+Google Imagen - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Image Generation Google Imagen Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation fal Google Imagen OpenAI Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview GoogleImageGenService provides high-quality image generation capabilities using Google’s Imagen models. It supports generating multiple images from text prompts with various customization options. ​ Installation To use GoogleImageGenService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[google]" You’ll also need to set up your Google API key as an environment variable: GOOGLE_API_KEY ​ Configuration ​ Constructor Parameters ​ params InputParams default: "InputParams()" Generation parameters configuration ​ api_key str required Google API key for authentication ​ Input Parameters ​ number_of_images int default: "1" Number of images to generate (1-8) ​ model str default: "imagen-3.0-generate-002" Model identifier ​ negative_prompt str default: "None" Elements to exclude from generation ​ Input The service accepts text prompts through its image generation pipeline. ​ Output Frames ​ URLImageRawFrame ​ url string Generated image URL (null for Google implementation as it returns raw bytes) ​ image bytes Raw image data ​ size tuple Image dimensions (width, height) ​ format string Image format (e.g., ‘JPEG’) ​ ErrorFrame ​ error string Error information if generation fails ​ Usage Example Copy Ask AI from pipecat.services.google.image import GoogleImageGenService # Configure service image_gen = GoogleImageGenService( api_key = "your-google-api-key" , params = GoogleImageGenService.InputParams( number_of_images = 2 , model = "imagen-3.0-generate-002" , negative_prompt = "blurry, distorted, low quality" ) ) # Use in pipeline main_pipeline = Pipeline( [ transport.input(), context_aggregator.user(), llm_service, image_gen, tts_service, transport.output(), context_aggregator.assistant(), ] ) ​ Frame Flow ​ Metrics Support The service supports metrics collection: Time to First Byte (TTFB) Processing duration API response metrics ​ Model Support Google’s Imagen service offers different model variants: Model ID Description imagen-3.0-generate-002 Latest Imagen model with high-quality outputs See other available models in Google’s Imagen documentation . ​ Error Handling Copy Ask AI try : async for frame in service.run_image_gen(prompt): if isinstance (frame, ErrorFrame): handle_error(frame.error) except Exception as e: logger.error( f "Image generation error: { e } " ) fal OpenAI On this page Overview Installation Configuration Constructor Parameters Input Parameters Input Output Frames URLImageRawFrame ErrorFrame Usage Example Frame Flow Metrics Support Model Support Error Handling Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/image-generation_google-imagen_5b375cd0.txt b/image-generation_google-imagen_5b375cd0.txt
new file mode 100644
index 0000000000000000000000000000000000000000..ab8cb01b4f4fa5d397950da1de3e6e9592507e01
--- /dev/null
+++ b/image-generation_google-imagen_5b375cd0.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/image-generation/google-imagen#overview
+Title: Google Imagen - Pipecat
+==================================================
+
+Google Imagen - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Image Generation Google Imagen Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation fal Google Imagen OpenAI Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview GoogleImageGenService provides high-quality image generation capabilities using Google’s Imagen models. It supports generating multiple images from text prompts with various customization options. ​ Installation To use GoogleImageGenService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[google]" You’ll also need to set up your Google API key as an environment variable: GOOGLE_API_KEY ​ Configuration ​ Constructor Parameters ​ params InputParams default: "InputParams()" Generation parameters configuration ​ api_key str required Google API key for authentication ​ Input Parameters ​ number_of_images int default: "1" Number of images to generate (1-8) ​ model str default: "imagen-3.0-generate-002" Model identifier ​ negative_prompt str default: "None" Elements to exclude from generation ​ Input The service accepts text prompts through its image generation pipeline. ​ Output Frames ​ URLImageRawFrame ​ url string Generated image URL (null for Google implementation as it returns raw bytes) ​ image bytes Raw image data ​ size tuple Image dimensions (width, height) ​ format string Image format (e.g., ‘JPEG’) ​ ErrorFrame ​ error string Error information if generation fails ​ Usage Example Copy Ask AI from pipecat.services.google.image import GoogleImageGenService # Configure service image_gen = GoogleImageGenService( api_key = "your-google-api-key" , params = GoogleImageGenService.InputParams( number_of_images = 2 , model = "imagen-3.0-generate-002" , negative_prompt = "blurry, distorted, low quality" ) ) # Use in pipeline main_pipeline = Pipeline( [ transport.input(), context_aggregator.user(), llm_service, image_gen, tts_service, transport.output(), context_aggregator.assistant(), ] ) ​ Frame Flow ​ Metrics Support The service supports metrics collection: Time to First Byte (TTFB) Processing duration API response metrics ​ Model Support Google’s Imagen service offers different model variants: Model ID Description imagen-3.0-generate-002 Latest Imagen model with high-quality outputs See other available models in Google’s Imagen documentation . ​ Error Handling Copy Ask AI try : async for frame in service.run_image_gen(prompt): if isinstance (frame, ErrorFrame): handle_error(frame.error) except Exception as e: logger.error( f "Image generation error: { e } " ) fal OpenAI On this page Overview Installation Configuration Constructor Parameters Input Parameters Input Output Frames URLImageRawFrame ErrorFrame Usage Example Frame Flow Metrics Support Model Support Error Handling Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/image-generation_google-imagen_6937399d.txt b/image-generation_google-imagen_6937399d.txt
new file mode 100644
index 0000000000000000000000000000000000000000..e4720d8d784477a973fb263b2eed33ad45ed622f
--- /dev/null
+++ b/image-generation_google-imagen_6937399d.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/image-generation/google-imagen#metrics-support
+Title: Google Imagen - Pipecat
+==================================================
+
+Google Imagen - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Image Generation Google Imagen Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation fal Google Imagen OpenAI Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview GoogleImageGenService provides high-quality image generation capabilities using Google’s Imagen models. It supports generating multiple images from text prompts with various customization options. ​ Installation To use GoogleImageGenService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[google]" You’ll also need to set up your Google API key as an environment variable: GOOGLE_API_KEY ​ Configuration ​ Constructor Parameters ​ params InputParams default: "InputParams()" Generation parameters configuration ​ api_key str required Google API key for authentication ​ Input Parameters ​ number_of_images int default: "1" Number of images to generate (1-8) ​ model str default: "imagen-3.0-generate-002" Model identifier ​ negative_prompt str default: "None" Elements to exclude from generation ​ Input The service accepts text prompts through its image generation pipeline. ​ Output Frames ​ URLImageRawFrame ​ url string Generated image URL (null for Google implementation as it returns raw bytes) ​ image bytes Raw image data ​ size tuple Image dimensions (width, height) ​ format string Image format (e.g., ‘JPEG’) ​ ErrorFrame ​ error string Error information if generation fails ​ Usage Example Copy Ask AI from pipecat.services.google.image import GoogleImageGenService # Configure service image_gen = GoogleImageGenService( api_key = "your-google-api-key" , params = GoogleImageGenService.InputParams( number_of_images = 2 , model = "imagen-3.0-generate-002" , negative_prompt = "blurry, distorted, low quality" ) ) # Use in pipeline main_pipeline = Pipeline( [ transport.input(), context_aggregator.user(), llm_service, image_gen, tts_service, transport.output(), context_aggregator.assistant(), ] ) ​ Frame Flow ​ Metrics Support The service supports metrics collection: Time to First Byte (TTFB) Processing duration API response metrics ​ Model Support Google’s Imagen service offers different model variants: Model ID Description imagen-3.0-generate-002 Latest Imagen model with high-quality outputs See other available models in Google’s Imagen documentation . ​ Error Handling Copy Ask AI try : async for frame in service.run_image_gen(prompt): if isinstance (frame, ErrorFrame): handle_error(frame.error) except Exception as e: logger.error( f "Image generation error: { e } " ) fal OpenAI On this page Overview Installation Configuration Constructor Parameters Input Parameters Input Output Frames URLImageRawFrame ErrorFrame Usage Example Frame Flow Metrics Support Model Support Error Handling Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/image-generation_google-imagen_a1849ba6.txt b/image-generation_google-imagen_a1849ba6.txt
new file mode 100644
index 0000000000000000000000000000000000000000..580841086dd87d22b34b0620944bafc732d20029
--- /dev/null
+++ b/image-generation_google-imagen_a1849ba6.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/image-generation/google-imagen#param-error
+Title: Google Imagen - Pipecat
+==================================================
+
+Google Imagen - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Image Generation Google Imagen Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation fal Google Imagen OpenAI Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview GoogleImageGenService provides high-quality image generation capabilities using Google’s Imagen models. It supports generating multiple images from text prompts with various customization options. ​ Installation To use GoogleImageGenService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[google]" You’ll also need to set up your Google API key as an environment variable: GOOGLE_API_KEY ​ Configuration ​ Constructor Parameters ​ params InputParams default: "InputParams()" Generation parameters configuration ​ api_key str required Google API key for authentication ​ Input Parameters ​ number_of_images int default: "1" Number of images to generate (1-8) ​ model str default: "imagen-3.0-generate-002" Model identifier ​ negative_prompt str default: "None" Elements to exclude from generation ​ Input The service accepts text prompts through its image generation pipeline. ​ Output Frames ​ URLImageRawFrame ​ url string Generated image URL (null for Google implementation as it returns raw bytes) ​ image bytes Raw image data ​ size tuple Image dimensions (width, height) ​ format string Image format (e.g., ‘JPEG’) ​ ErrorFrame ​ error string Error information if generation fails ​ Usage Example Copy Ask AI from pipecat.services.google.image import GoogleImageGenService # Configure service image_gen = GoogleImageGenService( api_key = "your-google-api-key" , params = GoogleImageGenService.InputParams( number_of_images = 2 , model = "imagen-3.0-generate-002" , negative_prompt = "blurry, distorted, low quality" ) ) # Use in pipeline main_pipeline = Pipeline( [ transport.input(), context_aggregator.user(), llm_service, image_gen, tts_service, transport.output(), context_aggregator.assistant(), ] ) ​ Frame Flow ​ Metrics Support The service supports metrics collection: Time to First Byte (TTFB) Processing duration API response metrics ​ Model Support Google’s Imagen service offers different model variants: Model ID Description imagen-3.0-generate-002 Latest Imagen model with high-quality outputs See other available models in Google’s Imagen documentation . ​ Error Handling Copy Ask AI try : async for frame in service.run_image_gen(prompt): if isinstance (frame, ErrorFrame): handle_error(frame.error) except Exception as e: logger.error( f "Image generation error: { e } " ) fal OpenAI On this page Overview Installation Configuration Constructor Parameters Input Parameters Input Output Frames URLImageRawFrame ErrorFrame Usage Example Frame Flow Metrics Support Model Support Error Handling Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/image-generation_openai_018a82fc.txt b/image-generation_openai_018a82fc.txt
new file mode 100644
index 0000000000000000000000000000000000000000..72c65f5ef1841e9bbeefa9de73608cc3d68fc776
--- /dev/null
+++ b/image-generation_openai_018a82fc.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/image-generation/openai#param-error
+Title: OpenAI Image Generation - Pipecat
+==================================================
+
+OpenAI Image Generation - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Image Generation OpenAI Image Generation Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation fal Google Imagen OpenAI Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview OpenAIImageGenService provides high-quality image generation capabilities using OpenAI’s DALL-E models. It transforms text prompts into images with various size options and model configurations. ​ Installation No additional installation is required for the OpenAIImageGenService as it is part of the Pipecat AI package. You’ll also need an OpenAI API key for authentication. ​ Configuration ​ Constructor Parameters ​ api_key str required OpenAI API key for authentication ​ base_url str default: "None" Optional base URL for OpenAI API requests ​ aiohttp_session aiohttp.ClientSession required HTTP session for making requests ​ image_size str required Image dimensions - one of “256x256”, “512x512”, “1024x1024”, “1792x1024”, “1024x1792” ​ model str default: "dall-e-3" OpenAI model identifier for image generation ​ Input The service accepts text prompts through its image generation pipeline. ​ Output Frames ​ URLImageRawFrame ​ url string Generated image URL from OpenAI ​ image bytes Raw image data ​ size tuple Image dimensions (width, height) ​ format string Image format (e.g., ‘JPEG’) ​ ErrorFrame ​ error string Error information if generation fails ​ Usage Example Copy Ask AI import aiohttp from pipecat.pipeline.pipeline import Pipeline from pipecat.services.openai.image import OpenAIImageGenService # Create an aiohttp session aiohttp_session = aiohttp.ClientSession() # Configure service image_gen = OpenAIImageGenService( api_key = "your-openai-api-key" , aiohttp_session = aiohttp_session, image_size = "1024x1024" , model = "dall-e-3" ) # Use in pipeline main_pipeline = Pipeline( [ transport.input(), context_aggregator.user(), llm_service, image_gen, tts_service, transport.output(), context_aggregator.assistant(), ] ) ​ Frame Flow ​ Metrics Support The service supports metrics collection: Time to First Byte (TTFB) Processing duration API response metrics ​ Model Support OpenAI’s image generation service offers different model variants: Model ID Description dall-e-3 Latest DALL-E model with higher quality and better prompt following dall-e-2 Previous generation model with good quality and lower cost ​ Image Size Options Size Option Aspect Ratio Description 256x256 1:1 Small square image 512x512 1:1 Medium square image 1024x1024 1:1 Large square image 1792x1024 16:9 Horizontal/landscape orientation 1024x1792 9:16 Vertical/portrait orientation ​ Error Handling Copy Ask AI try : async for frame in image_gen.run_image_gen(prompt): if isinstance (frame, ErrorFrame): logger.error( f "Image generation error: { frame.error } " ) else : # Process successful image generation pass except Exception as e: logger.error( f "Unexpected error during image generation: { e } " ) Google Imagen Simli On this page Overview Installation Configuration Constructor Parameters Input Output Frames URLImageRawFrame ErrorFrame Usage Example Frame Flow Metrics Support Model Support Image Size Options Error Handling Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/image-generation_openai_0f5c2c9a.txt b/image-generation_openai_0f5c2c9a.txt
new file mode 100644
index 0000000000000000000000000000000000000000..1715b9091645f498e6ab844a782236634965f451
--- /dev/null
+++ b/image-generation_openai_0f5c2c9a.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/image-generation/openai#param-format
+Title: OpenAI Image Generation - Pipecat
+==================================================
+
+OpenAI Image Generation - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Image Generation OpenAI Image Generation Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation fal Google Imagen OpenAI Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview OpenAIImageGenService provides high-quality image generation capabilities using OpenAI’s DALL-E models. It transforms text prompts into images with various size options and model configurations. ​ Installation No additional installation is required for the OpenAIImageGenService as it is part of the Pipecat AI package. You’ll also need an OpenAI API key for authentication. ​ Configuration ​ Constructor Parameters ​ api_key str required OpenAI API key for authentication ​ base_url str default: "None" Optional base URL for OpenAI API requests ​ aiohttp_session aiohttp.ClientSession required HTTP session for making requests ​ image_size str required Image dimensions - one of “256x256”, “512x512”, “1024x1024”, “1792x1024”, “1024x1792” ​ model str default: "dall-e-3" OpenAI model identifier for image generation ​ Input The service accepts text prompts through its image generation pipeline. ​ Output Frames ​ URLImageRawFrame ​ url string Generated image URL from OpenAI ​ image bytes Raw image data ​ size tuple Image dimensions (width, height) ​ format string Image format (e.g., ‘JPEG’) ​ ErrorFrame ​ error string Error information if generation fails ​ Usage Example Copy Ask AI import aiohttp from pipecat.pipeline.pipeline import Pipeline from pipecat.services.openai.image import OpenAIImageGenService # Create an aiohttp session aiohttp_session = aiohttp.ClientSession() # Configure service image_gen = OpenAIImageGenService( api_key = "your-openai-api-key" , aiohttp_session = aiohttp_session, image_size = "1024x1024" , model = "dall-e-3" ) # Use in pipeline main_pipeline = Pipeline( [ transport.input(), context_aggregator.user(), llm_service, image_gen, tts_service, transport.output(), context_aggregator.assistant(), ] ) ​ Frame Flow ​ Metrics Support The service supports metrics collection: Time to First Byte (TTFB) Processing duration API response metrics ​ Model Support OpenAI’s image generation service offers different model variants: Model ID Description dall-e-3 Latest DALL-E model with higher quality and better prompt following dall-e-2 Previous generation model with good quality and lower cost ​ Image Size Options Size Option Aspect Ratio Description 256x256 1:1 Small square image 512x512 1:1 Medium square image 1024x1024 1:1 Large square image 1792x1024 16:9 Horizontal/landscape orientation 1024x1792 9:16 Vertical/portrait orientation ​ Error Handling Copy Ask AI try : async for frame in image_gen.run_image_gen(prompt): if isinstance (frame, ErrorFrame): logger.error( f "Image generation error: { frame.error } " ) else : # Process successful image generation pass except Exception as e: logger.error( f "Unexpected error during image generation: { e } " ) Google Imagen Simli On this page Overview Installation Configuration Constructor Parameters Input Output Frames URLImageRawFrame ErrorFrame Usage Example Frame Flow Metrics Support Model Support Image Size Options Error Handling Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/image-generation_openai_6a23b147.txt b/image-generation_openai_6a23b147.txt
new file mode 100644
index 0000000000000000000000000000000000000000..c8c155d2994be76996e66e3a34bb963d904de8c0
--- /dev/null
+++ b/image-generation_openai_6a23b147.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/image-generation/openai#input
+Title: OpenAI Image Generation - Pipecat
+==================================================
+
+OpenAI Image Generation - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Image Generation OpenAI Image Generation Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation fal Google Imagen OpenAI Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview OpenAIImageGenService provides high-quality image generation capabilities using OpenAI’s DALL-E models. It transforms text prompts into images with various size options and model configurations. ​ Installation No additional installation is required for the OpenAIImageGenService as it is part of the Pipecat AI package. You’ll also need an OpenAI API key for authentication. ​ Configuration ​ Constructor Parameters ​ api_key str required OpenAI API key for authentication ​ base_url str default: "None" Optional base URL for OpenAI API requests ​ aiohttp_session aiohttp.ClientSession required HTTP session for making requests ​ image_size str required Image dimensions - one of “256x256”, “512x512”, “1024x1024”, “1792x1024”, “1024x1792” ​ model str default: "dall-e-3" OpenAI model identifier for image generation ​ Input The service accepts text prompts through its image generation pipeline. ​ Output Frames ​ URLImageRawFrame ​ url string Generated image URL from OpenAI ​ image bytes Raw image data ​ size tuple Image dimensions (width, height) ​ format string Image format (e.g., ‘JPEG’) ​ ErrorFrame ​ error string Error information if generation fails ​ Usage Example Copy Ask AI import aiohttp from pipecat.pipeline.pipeline import Pipeline from pipecat.services.openai.image import OpenAIImageGenService # Create an aiohttp session aiohttp_session = aiohttp.ClientSession() # Configure service image_gen = OpenAIImageGenService( api_key = "your-openai-api-key" , aiohttp_session = aiohttp_session, image_size = "1024x1024" , model = "dall-e-3" ) # Use in pipeline main_pipeline = Pipeline( [ transport.input(), context_aggregator.user(), llm_service, image_gen, tts_service, transport.output(), context_aggregator.assistant(), ] ) ​ Frame Flow ​ Metrics Support The service supports metrics collection: Time to First Byte (TTFB) Processing duration API response metrics ​ Model Support OpenAI’s image generation service offers different model variants: Model ID Description dall-e-3 Latest DALL-E model with higher quality and better prompt following dall-e-2 Previous generation model with good quality and lower cost ​ Image Size Options Size Option Aspect Ratio Description 256x256 1:1 Small square image 512x512 1:1 Medium square image 1024x1024 1:1 Large square image 1792x1024 16:9 Horizontal/landscape orientation 1024x1792 9:16 Vertical/portrait orientation ​ Error Handling Copy Ask AI try : async for frame in image_gen.run_image_gen(prompt): if isinstance (frame, ErrorFrame): logger.error( f "Image generation error: { frame.error } " ) else : # Process successful image generation pass except Exception as e: logger.error( f "Unexpected error during image generation: { e } " ) Google Imagen Simli On this page Overview Installation Configuration Constructor Parameters Input Output Frames URLImageRawFrame ErrorFrame Usage Example Frame Flow Metrics Support Model Support Image Size Options Error Handling Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/image-generation_openai_9607138a.txt b/image-generation_openai_9607138a.txt
new file mode 100644
index 0000000000000000000000000000000000000000..49c3d261384b700202bad13fce285fd24a45ed15
--- /dev/null
+++ b/image-generation_openai_9607138a.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/image-generation/openai#urlimagerawframe
+Title: OpenAI Image Generation - Pipecat
+==================================================
+
+OpenAI Image Generation - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Image Generation OpenAI Image Generation Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation fal Google Imagen OpenAI Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview OpenAIImageGenService provides high-quality image generation capabilities using OpenAI’s DALL-E models. It transforms text prompts into images with various size options and model configurations. ​ Installation No additional installation is required for the OpenAIImageGenService as it is part of the Pipecat AI package. You’ll also need an OpenAI API key for authentication. ​ Configuration ​ Constructor Parameters ​ api_key str required OpenAI API key for authentication ​ base_url str default: "None" Optional base URL for OpenAI API requests ​ aiohttp_session aiohttp.ClientSession required HTTP session for making requests ​ image_size str required Image dimensions - one of “256x256”, “512x512”, “1024x1024”, “1792x1024”, “1024x1792” ​ model str default: "dall-e-3" OpenAI model identifier for image generation ​ Input The service accepts text prompts through its image generation pipeline. ​ Output Frames ​ URLImageRawFrame ​ url string Generated image URL from OpenAI ​ image bytes Raw image data ​ size tuple Image dimensions (width, height) ​ format string Image format (e.g., ‘JPEG’) ​ ErrorFrame ​ error string Error information if generation fails ​ Usage Example Copy Ask AI import aiohttp from pipecat.pipeline.pipeline import Pipeline from pipecat.services.openai.image import OpenAIImageGenService # Create an aiohttp session aiohttp_session = aiohttp.ClientSession() # Configure service image_gen = OpenAIImageGenService( api_key = "your-openai-api-key" , aiohttp_session = aiohttp_session, image_size = "1024x1024" , model = "dall-e-3" ) # Use in pipeline main_pipeline = Pipeline( [ transport.input(), context_aggregator.user(), llm_service, image_gen, tts_service, transport.output(), context_aggregator.assistant(), ] ) ​ Frame Flow ​ Metrics Support The service supports metrics collection: Time to First Byte (TTFB) Processing duration API response metrics ​ Model Support OpenAI’s image generation service offers different model variants: Model ID Description dall-e-3 Latest DALL-E model with higher quality and better prompt following dall-e-2 Previous generation model with good quality and lower cost ​ Image Size Options Size Option Aspect Ratio Description 256x256 1:1 Small square image 512x512 1:1 Medium square image 1024x1024 1:1 Large square image 1792x1024 16:9 Horizontal/landscape orientation 1024x1792 9:16 Vertical/portrait orientation ​ Error Handling Copy Ask AI try : async for frame in image_gen.run_image_gen(prompt): if isinstance (frame, ErrorFrame): logger.error( f "Image generation error: { frame.error } " ) else : # Process successful image generation pass except Exception as e: logger.error( f "Unexpected error during image generation: { e } " ) Google Imagen Simli On this page Overview Installation Configuration Constructor Parameters Input Output Frames URLImageRawFrame ErrorFrame Usage Example Frame Flow Metrics Support Model Support Image Size Options Error Handling Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/image-generation_openai_b8889faf.txt b/image-generation_openai_b8889faf.txt
new file mode 100644
index 0000000000000000000000000000000000000000..e523654e956399e7bdcd93b0151de4cee669669f
--- /dev/null
+++ b/image-generation_openai_b8889faf.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/image-generation/openai#error-handling
+Title: OpenAI Image Generation - Pipecat
+==================================================
+
+OpenAI Image Generation - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Image Generation OpenAI Image Generation Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation fal Google Imagen OpenAI Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview OpenAIImageGenService provides high-quality image generation capabilities using OpenAI’s DALL-E models. It transforms text prompts into images with various size options and model configurations. ​ Installation No additional installation is required for the OpenAIImageGenService as it is part of the Pipecat AI package. You’ll also need an OpenAI API key for authentication. ​ Configuration ​ Constructor Parameters ​ api_key str required OpenAI API key for authentication ​ base_url str default: "None" Optional base URL for OpenAI API requests ​ aiohttp_session aiohttp.ClientSession required HTTP session for making requests ​ image_size str required Image dimensions - one of “256x256”, “512x512”, “1024x1024”, “1792x1024”, “1024x1792” ​ model str default: "dall-e-3" OpenAI model identifier for image generation ​ Input The service accepts text prompts through its image generation pipeline. ​ Output Frames ​ URLImageRawFrame ​ url string Generated image URL from OpenAI ​ image bytes Raw image data ​ size tuple Image dimensions (width, height) ​ format string Image format (e.g., ‘JPEG’) ​ ErrorFrame ​ error string Error information if generation fails ​ Usage Example Copy Ask AI import aiohttp from pipecat.pipeline.pipeline import Pipeline from pipecat.services.openai.image import OpenAIImageGenService # Create an aiohttp session aiohttp_session = aiohttp.ClientSession() # Configure service image_gen = OpenAIImageGenService( api_key = "your-openai-api-key" , aiohttp_session = aiohttp_session, image_size = "1024x1024" , model = "dall-e-3" ) # Use in pipeline main_pipeline = Pipeline( [ transport.input(), context_aggregator.user(), llm_service, image_gen, tts_service, transport.output(), context_aggregator.assistant(), ] ) ​ Frame Flow ​ Metrics Support The service supports metrics collection: Time to First Byte (TTFB) Processing duration API response metrics ​ Model Support OpenAI’s image generation service offers different model variants: Model ID Description dall-e-3 Latest DALL-E model with higher quality and better prompt following dall-e-2 Previous generation model with good quality and lower cost ​ Image Size Options Size Option Aspect Ratio Description 256x256 1:1 Small square image 512x512 1:1 Medium square image 1024x1024 1:1 Large square image 1792x1024 16:9 Horizontal/landscape orientation 1024x1792 9:16 Vertical/portrait orientation ​ Error Handling Copy Ask AI try : async for frame in image_gen.run_image_gen(prompt): if isinstance (frame, ErrorFrame): logger.error( f "Image generation error: { frame.error } " ) else : # Process successful image generation pass except Exception as e: logger.error( f "Unexpected error during image generation: { e } " ) Google Imagen Simli On this page Overview Installation Configuration Constructor Parameters Input Output Frames URLImageRawFrame ErrorFrame Usage Example Frame Flow Metrics Support Model Support Image Size Options Error Handling Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/image-generation_openai_d01dd1cf.txt b/image-generation_openai_d01dd1cf.txt
new file mode 100644
index 0000000000000000000000000000000000000000..c7293df76157c4c37b7d5d1b4feab74ef49125dd
--- /dev/null
+++ b/image-generation_openai_d01dd1cf.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/image-generation/openai#param-image
+Title: OpenAI Image Generation - Pipecat
+==================================================
+
+OpenAI Image Generation - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Image Generation OpenAI Image Generation Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation fal Google Imagen OpenAI Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview OpenAIImageGenService provides high-quality image generation capabilities using OpenAI’s DALL-E models. It transforms text prompts into images with various size options and model configurations. ​ Installation No additional installation is required for the OpenAIImageGenService as it is part of the Pipecat AI package. You’ll also need an OpenAI API key for authentication. ​ Configuration ​ Constructor Parameters ​ api_key str required OpenAI API key for authentication ​ base_url str default: "None" Optional base URL for OpenAI API requests ​ aiohttp_session aiohttp.ClientSession required HTTP session for making requests ​ image_size str required Image dimensions - one of “256x256”, “512x512”, “1024x1024”, “1792x1024”, “1024x1792” ​ model str default: "dall-e-3" OpenAI model identifier for image generation ​ Input The service accepts text prompts through its image generation pipeline. ​ Output Frames ​ URLImageRawFrame ​ url string Generated image URL from OpenAI ​ image bytes Raw image data ​ size tuple Image dimensions (width, height) ​ format string Image format (e.g., ‘JPEG’) ​ ErrorFrame ​ error string Error information if generation fails ​ Usage Example Copy Ask AI import aiohttp from pipecat.pipeline.pipeline import Pipeline from pipecat.services.openai.image import OpenAIImageGenService # Create an aiohttp session aiohttp_session = aiohttp.ClientSession() # Configure service image_gen = OpenAIImageGenService( api_key = "your-openai-api-key" , aiohttp_session = aiohttp_session, image_size = "1024x1024" , model = "dall-e-3" ) # Use in pipeline main_pipeline = Pipeline( [ transport.input(), context_aggregator.user(), llm_service, image_gen, tts_service, transport.output(), context_aggregator.assistant(), ] ) ​ Frame Flow ​ Metrics Support The service supports metrics collection: Time to First Byte (TTFB) Processing duration API response metrics ​ Model Support OpenAI’s image generation service offers different model variants: Model ID Description dall-e-3 Latest DALL-E model with higher quality and better prompt following dall-e-2 Previous generation model with good quality and lower cost ​ Image Size Options Size Option Aspect Ratio Description 256x256 1:1 Small square image 512x512 1:1 Medium square image 1024x1024 1:1 Large square image 1792x1024 16:9 Horizontal/landscape orientation 1024x1792 9:16 Vertical/portrait orientation ​ Error Handling Copy Ask AI try : async for frame in image_gen.run_image_gen(prompt): if isinstance (frame, ErrorFrame): logger.error( f "Image generation error: { frame.error } " ) else : # Process successful image generation pass except Exception as e: logger.error( f "Unexpected error during image generation: { e } " ) Google Imagen Simli On this page Overview Installation Configuration Constructor Parameters Input Output Frames URLImageRawFrame ErrorFrame Usage Example Frame Flow Metrics Support Model Support Image Size Options Error Handling Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/image-generation_openai_d1c0ebc7.txt b/image-generation_openai_d1c0ebc7.txt
new file mode 100644
index 0000000000000000000000000000000000000000..c0e92939739ed273801750baba2449b36592a97e
--- /dev/null
+++ b/image-generation_openai_d1c0ebc7.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/image-generation/openai#param-size
+Title: OpenAI Image Generation - Pipecat
+==================================================
+
+OpenAI Image Generation - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Image Generation OpenAI Image Generation Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation fal Google Imagen OpenAI Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview OpenAIImageGenService provides high-quality image generation capabilities using OpenAI’s DALL-E models. It transforms text prompts into images with various size options and model configurations. ​ Installation No additional installation is required for the OpenAIImageGenService as it is part of the Pipecat AI package. You’ll also need an OpenAI API key for authentication. ​ Configuration ​ Constructor Parameters ​ api_key str required OpenAI API key for authentication ​ base_url str default: "None" Optional base URL for OpenAI API requests ​ aiohttp_session aiohttp.ClientSession required HTTP session for making requests ​ image_size str required Image dimensions - one of “256x256”, “512x512”, “1024x1024”, “1792x1024”, “1024x1792” ​ model str default: "dall-e-3" OpenAI model identifier for image generation ​ Input The service accepts text prompts through its image generation pipeline. ​ Output Frames ​ URLImageRawFrame ​ url string Generated image URL from OpenAI ​ image bytes Raw image data ​ size tuple Image dimensions (width, height) ​ format string Image format (e.g., ‘JPEG’) ​ ErrorFrame ​ error string Error information if generation fails ​ Usage Example Copy Ask AI import aiohttp from pipecat.pipeline.pipeline import Pipeline from pipecat.services.openai.image import OpenAIImageGenService # Create an aiohttp session aiohttp_session = aiohttp.ClientSession() # Configure service image_gen = OpenAIImageGenService( api_key = "your-openai-api-key" , aiohttp_session = aiohttp_session, image_size = "1024x1024" , model = "dall-e-3" ) # Use in pipeline main_pipeline = Pipeline( [ transport.input(), context_aggregator.user(), llm_service, image_gen, tts_service, transport.output(), context_aggregator.assistant(), ] ) ​ Frame Flow ​ Metrics Support The service supports metrics collection: Time to First Byte (TTFB) Processing duration API response metrics ​ Model Support OpenAI’s image generation service offers different model variants: Model ID Description dall-e-3 Latest DALL-E model with higher quality and better prompt following dall-e-2 Previous generation model with good quality and lower cost ​ Image Size Options Size Option Aspect Ratio Description 256x256 1:1 Small square image 512x512 1:1 Medium square image 1024x1024 1:1 Large square image 1792x1024 16:9 Horizontal/landscape orientation 1024x1792 9:16 Vertical/portrait orientation ​ Error Handling Copy Ask AI try : async for frame in image_gen.run_image_gen(prompt): if isinstance (frame, ErrorFrame): logger.error( f "Image generation error: { frame.error } " ) else : # Process successful image generation pass except Exception as e: logger.error( f "Unexpected error during image generation: { e } " ) Google Imagen Simli On this page Overview Installation Configuration Constructor Parameters Input Output Frames URLImageRawFrame ErrorFrame Usage Example Frame Flow Metrics Support Model Support Image Size Options Error Handling Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/index.rst.txt b/index.rst.txt
new file mode 100644
index 0000000000000000000000000000000000000000..6ffe2ed4466484ac2cde2523964a369b7e1c8d54
--- /dev/null
+++ b/index.rst.txt
@@ -0,0 +1,70 @@
+<!DOCTYPE html><html lang="en" class="dark"><head><meta charSet="utf-8" data-next-head=""/><meta name="viewport" content="width=device-width" data-next-head=""/><link rel="apple-touch-icon" type="image/png" sizes="180x180" href="https://mintlify.s3-us-west-1.amazonaws.com/daily/_generated/favicon/apple-touch-icon.png?v=3" data-next-head=""/><link rel="icon" type="image/png" sizes="32x32" href="https://mintlify.s3-us-west-1.amazonaws.com/daily/_generated/favicon/favicon-32x32.png?v=3" data-next-head=""/><link rel="icon" type="image/png" sizes="16x16" href="https://mintlify.s3-us-west-1.amazonaws.com/daily/_generated/favicon/favicon-16x16.png?v=3" data-next-head=""/><link rel="shortcut icon" type="image/x-icon" href="https://mintlify.s3-us-west-1.amazonaws.com/daily/_generated/favicon/favicon.ico?v=3" data-next-head=""/><meta name="msapplication-config" content="https://mintlify.s3-us-west-1.amazonaws.com/daily/_generated/favicon/browserconfig.xml?v=3" data-next-head=""/><meta name="apple-mobile-web-app-title" content="Pipecat" data-next-head=""/><meta name="application-name" content="Pipecat" data-next-head=""/><meta name="msapplication-TileColor" content="#4f46e5" data-next-head=""/><meta name="theme-color" content="#ffffff" data-next-head=""/><meta name="generator" content="Mintlify" data-next-head=""/><link rel="sitemap" type="application/xml" href="/sitemap.xml" data-next-head=""/><meta name="charset" content="utf-8" data-next-head=""/><meta property="og:type" content="website" data-next-head=""/><meta property="og:site_name" content="Pipecat" data-next-head=""/><meta property="twitter:card" content="summary_large_image" data-next-head=""/><meta property="og:title" content="Overview - Pipecat" data-next-head=""/><meta property="twitter:title" content="Overview - Pipecat" data-next-head=""/><meta property="og:description" content="Learn about Pipecat, the open source framework for building voice and multimodal AI agents." data-next-head=""/><meta name="description" content="Pipecat is a framework for building voice-enabled, real-time, multimodal AI applications" data-next-head=""/><meta property="og:locale" content="en_US" data-next-head=""/><meta property="og:logo" content="/logo/pipecat-logo.svg" data-next-head=""/><meta name="article:publisher" content="Pipecat AI" data-next-head=""/><meta property="twitter:description" content="Learn about Pipecat, the open source framework for building voice and multimodal AI agents." data-next-head=""/><meta property="twitter:url" content="https://docs.pipecat.ai/" data-next-head=""/><meta property="twitter:site" content="@pipecat_ai" data-next-head=""/><meta property="og:image:width" content="1200" data-next-head=""/><meta property="og:image:height" content="675" data-next-head=""/><meta property="og:image" content="https://daily.mintlify.app/mintlify-assets/_next/image?url=%2Fapi%2Fog%3Fdivision%3DDocumentation%26title%3DOverview%26description%3DPipecat%2Bis%2Ba%2Bframework%2Bfor%2Bbuilding%2Bvoice-enabled%252C%2Breal-time%252C%2Bmultimodal%2BAI%2Bapplications%26logoLight%3Dhttps%253A%252F%252Fmintlify.s3.us-west-1.amazonaws.com%252Fdaily%252Flogo%252Flight.svg%26logoDark%3Dhttps%253A%252F%252Fmintlify.s3.us-west-1.amazonaws.com%252Fdaily%252Flogo%252Fdark.svg%26primaryColor%3D%25234f46e5%26lightColor%3D%2523a5b4fc%26darkColor%3D%2523312e81%26backgroundLight%3D%2523ffffff%26backgroundDark%3D%25230a0d0d&amp;w=1200&amp;q=100" data-next-head=""/><meta property="twitter:image" content="https://daily.mintlify.app/mintlify-assets/_next/image?url=%2Fapi%2Fog%3Fdivision%3DDocumentation%26title%3DOverview%26description%3DPipecat%2Bis%2Ba%2Bframework%2Bfor%2Bbuilding%2Bvoice-enabled%252C%2Breal-time%252C%2Bmultimodal%2BAI%2Bapplications%26logoLight%3Dhttps%253A%252F%252Fmintlify.s3.us-west-1.amazonaws.com%252Fdaily%252Flogo%252Flight.svg%26logoDark%3Dhttps%253A%252F%252Fmintlify.s3.us-west-1.amazonaws.com%252Fdaily%252Flogo%252Fdark.svg%26primaryColor%3D%25234f46e5%26lightColor%3D%2523a5b4fc%26darkColor%3D%2523312e81%26backgroundLight%3D%2523ffffff%26backgroundDark%3D%25230a0d0d&amp;w=1200&amp;q=100" data-next-head=""/><meta property="twitter:image:width" content="1200" data-next-head=""/><meta property="twitter:image:height" content="630" data-next-head=""/><title data-next-head="">Overview - Pipecat</title><meta name="og:url" content="/getting-started/overview" data-next-head=""/><link rel="canonical" href="/getting-started/overview" data-next-head=""/><link rel="preload" href="/mintlify-assets/_next/static/media/e4af272ccee01ff0-s.p.woff2" as="font" type="font/woff2" crossorigin="anonymous" data-next-font="size-adjust"/><link rel="preload" href="/mintlify-assets/_next/static/media/bb3ef058b751a6ad-s.p.woff2" as="font" type="font/woff2" crossorigin="anonymous" data-next-font="size-adjust"/><link rel="preload" href="/mintlify-assets/_next/static/css/92ec57a7864f1f14.css" as="style"/><link rel="preload" href="/mintlify-assets/_next/static/css/19e66b131dc509b0.css" as="style"/><meta name="sentry-trace" content="d8366b01cb2b24225d22369cc16a98ae-5ae1d60702163eb5-1"/><meta name="baggage" content="sentry-environment=production,sentry-release=client%400.0.1433,sentry-trace_id=d8366b01cb2b24225d22369cc16a98ae,sentry-sample_rate=1,sentry-transaction=GET%20%2F_sites%2F%5Bsubdomain%5D%2F%5B%5B...slug%5D%5D,sentry-sampled=true"/><link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/katex@0.16.0/dist/katex.min.css" integrity="sha384-Xi8rHCmBmhbuyyhbI88391ZKP2dmfnOl4rT9ZfRI7mLTdk1wblIUnrIq35nqwEvC" crossorigin="anonymous"/><script id="_mintlify-banner-script" data-nscript="beforeInteractive">
+    (function() {
+      try {
+        var bannerKey = "docs.pipecat.ai-bannerDismissed";
+        var bannerContent = undefined;
+        
+        if (!bannerContent) {
+          document.documentElement.setAttribute('data-banner-state', 'hidden');
+          return;
+        }
+        
+        var dismissedValue = localStorage.getItem(bannerKey);
+        var shouldShowBanner = !dismissedValue || dismissedValue !== bannerContent;
+        
+        document.documentElement.setAttribute('data-banner-state', shouldShowBanner ? 'visible' : 'hidden');
+      } catch (e) {
+        document.documentElement.setAttribute('data-banner-state', 'hidden');
+      }
+    })();
+  </script><link rel="stylesheet" href="/mintlify-assets/_next/static/css/92ec57a7864f1f14.css" data-n-g=""/><link rel="stylesheet" href="/mintlify-assets/_next/static/css/19e66b131dc509b0.css" data-n-p=""/><noscript data-n-css=""></noscript><script defer="" noModule="" src="/mintlify-assets/_next/static/chunks/polyfills-42372ed130431b0a.js"></script><script src="/mintlify-assets/_next/static/chunks/webpack-ce4414b0857d4858.js" defer=""></script><script src="/mintlify-assets/_next/static/chunks/framework-1e86de504ff95d8a.js" defer=""></script><script src="/mintlify-assets/_next/static/chunks/main-88d78dd0df0c739c.js" defer=""></script><script src="/mintlify-assets/_next/static/chunks/pages/_app-6343a8b65e8b17b2.js" defer=""></script><script src="/mintlify-assets/_next/static/chunks/4e973673-a3489f3e596caabe.js" defer=""></script><script src="/mintlify-assets/_next/static/chunks/69ea16bc-4a9ae1feae908d3f.js" defer=""></script><script src="/mintlify-assets/_next/static/chunks/ce3ddbd1-72a6499036d28123.js" defer=""></script><script src="/mintlify-assets/_next/static/chunks/61521074-9905299d1fe6a0f3.js" defer=""></script><script src="/mintlify-assets/_next/static/chunks/46309-0f3545da763a3990.js" defer=""></script><script src="/mintlify-assets/_next/static/chunks/88533-515a65d7efc2923f.js" defer=""></script><script src="/mintlify-assets/_next/static/chunks/24586-b492aa58637b5c59.js" defer=""></script><script src="/mintlify-assets/_next/static/chunks/57163-ed441cfeaf702e50.js" defer=""></script><script src="/mintlify-assets/_next/static/chunks/74709-82c4596b32f96b8d.js" defer=""></script><script src="/mintlify-assets/_next/static/chunks/12777-d82dc6dba178a221.js" defer=""></script><script src="/mintlify-assets/_next/static/chunks/201-692997ccb7e52d7c.js" defer=""></script><script src="/mintlify-assets/_next/static/chunks/23446-0b76a818785e1908.js" defer=""></script><script src="/mintlify-assets/_next/static/chunks/74668-730e2daa3dcdc2b0.js" defer=""></script><script src="/mintlify-assets/_next/static/chunks/pages/_sites/%5Bsubdomain%5D/%5B%5B...slug%5D%5D-6f2dd744ec157273.js" defer=""></script><script src="/mintlify-assets/_next/static/RPzUnhb57G2BlnT2NDDZW/_buildManifest.js" defer=""></script><script src="/mintlify-assets/_next/static/RPzUnhb57G2BlnT2NDDZW/_ssgManifest.js" defer=""></script><style id="__jsx-4097608641">:root{--font-inter:'Inter', 'Inter Fallback';--font-jetbrains-mono:'JetBrains Mono', 'JetBrains Mono Fallback'}</style></head><link rel="preload" as="image" href="https://mintlify.s3.us-west-1.amazonaws.com/daily/logo/light.svg"/><link rel="preload" as="image" href="https://mintlify.s3.us-west-1.amazonaws.com/daily/logo/dark.svg"/><div id="__next"><main class="jsx-4097608641"><script>((e,i,s,u,m,a,l,h)=>{let d=document.documentElement,w=["light","dark"];function p(n){(Array.isArray(e)?e:[e]).forEach(y=>{let k=y==="class",S=k&&a?m.map(f=>a[f]||f):m;k?(d.classList.remove(...S),d.classList.add(a&&a[n]?a[n]:n)):d.setAttribute(y,n)}),R(n)}function R(n){h&&w.includes(n)&&(d.style.colorScheme=n)}function c(){return window.matchMedia("(prefers-color-scheme: dark)").matches?"dark":"light"}if(u)p(u);else try{let n=localStorage.getItem(i)||s,y=l&&n==="system"?c():n;p(y)}catch(n){}})("class","isDarkMode","system",null,["dark","light","true","false","system"],{"true":"dark","false":"light","dark":"dark","light":"light"},true,true)</script><style>:root {
+    --primary: 79 70 229;
+    --primary-light: 165 180 252;
+    --primary-dark: 49 46 129;
+    --background-light: 255 255 255;
+    --background-dark: 12 12 16;
+    --gray-50: 245 244 249;
+    --gray-100: 240 240 245;
+    --gray-200: 224 224 229;
+    --gray-300: 208 208 213;
+    --gray-400: 160 160 165;
+    --gray-500: 114 114 118;
+    --gray-600: 82 82 86;
+    --gray-700: 64 64 69;
+    --gray-800: 39 39 44;
+    --gray-900: 25 24 29;
+    --gray-950: 12 12 17;
+  }</style><span class="fixed inset-0 bg-background-light dark:bg-background-dark -z-10"></span><div class="relative antialiased text-gray-500 dark:text-gray-400"><div id="navbar" class="z-30 fixed lg:sticky top-0 w-full"><div id="navbar-transition" class="absolute w-full h-full backdrop-blur flex-none transition-colors duration-500 border-b border-gray-500/5 dark:border-gray-300/[0.06] supports-backdrop-blur:bg-background-light/60 dark:bg-transparent"></div><div id="banner" class="px-2 w-full [&amp;&gt;*]:text-white/90 md:h-10 relative text-center items-center justify-center text-sm prose prose-dark dark:prose-invert line-clamp-2 md:truncate md:[&amp;&gt;*]:truncate bg-primary-dark font-medium z-[45]"><div class="my-2 md:[&amp;&gt;p]:m-0"></div></div><div class="max-w-8xl mx-auto relative"><div><div class="relative"><div class="flex items-center lg:px-12 h-16 min-w-0 mx-4 lg:mx-0"><div class="h-full relative flex-1 flex items-center gap-x-4 min-w-0 border-b border-gray-500/5 dark:border-gray-300/[0.06]"><div class="flex-1 flex items-center gap-x-4"><a href="/"><span class="sr-only">Pipecat<!-- --> home page</span><img class="nav-logo w-auto h-7 relative object-contain block dark:hidden" src="https://mintlify.s3.us-west-1.amazonaws.com/daily/logo/light.svg" alt="light logo"/><img class="nav-logo w-auto h-7 relative object-contain hidden dark:block" src="https://mintlify.s3.us-west-1.amazonaws.com/daily/logo/dark.svg" alt="dark logo"/></a><div class="flex items-center gap-x-2"></div></div><div class="relative hidden lg:flex items-center gap-2.5 flex-1"><button type="button" class="flex pointer-events-auto rounded-xl w-full items-center text-sm leading-6 h-9 pl-3.5 pr-3 shadow-sm text-gray-500 dark:text-white/50 bg-background-light dark:bg-background-dark dark:brightness-[1.1] dark:ring-1 dark:hover:brightness-[1.25] ring-1 ring-gray-400/20 hover:ring-gray-600/25 dark:ring-gray-600/30 dark:hover:ring-gray-500/30 focus:outline-primary justify-between truncate gap-2 min-w-[43px]" id="search-bar-entry"><div class="flex items-center gap-2 min-w-[42px]"><svg xmlns="http://www.w3.org/2000/svg" width="16" height="16" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-search min-w-4 flex-none text-gray-700 hover:text-gray-800 dark:text-gray-400 hover:dark:text-gray-200"><circle cx="11" cy="11" r="8"></circle><path d="m21 21-4.3-4.3"></path></svg><div class="truncate min-w-0">Search...</div></div><span class="flex-none text-xs font-semibold">⌘<!-- -->K</span></button><button type="button" class="flex-none hidden lg:flex items-center justify-center gap-1.5 pl-3 pr-3.5 h-9 rounded-xl shadow-sm bg-background-light dark:bg-background-dark dark:brightness-[1.1] dark:ring-1 dark:hover:brightness-[1.25] ring-1 ring-gray-400/20 hover:ring-gray-600/25 dark:ring-gray-600/30 dark:hover:ring-gray-500/30 focus:outline-primary" id="assistant-entry"><svg xmlns="http://www.w3.org/2000/svg" width="18" height="18" viewBox="0 0 18 18" class="w-4 h-4 shrink-0 text-gray-700 hover:text-gray-800 dark:text-gray-400 hover:dark:text-gray-200"><g fill="currentColor"><path d="M5.658,2.99l-1.263-.421-.421-1.263c-.137-.408-.812-.408-.949,0l-.421,1.263-1.263,.421c-.204,.068-.342,.259-.342,.474s.138,.406,.342,.474l1.263,.421,.421,1.263c.068,.204,.26,.342,.475,.342s.406-.138,.475-.342l.421-1.263,1.263-.421c.204-.068,.342-.259,.342-.474s-.138-.406-.342-.474Z" fill="currentColor" data-stroke="none" stroke="none"></path><polygon points="9.5 2.75 11.412 7.587 16.25 9.5 11.412 11.413 9.5 16.25 7.587 11.413 2.75 9.5 7.587 7.587 9.5 2.75" fill="none" stroke="currentColor" stroke-linecap="round" stroke-linejoin="round" stroke-width="1.5"></polygon></g></svg><span class="text-sm text-gray-500 dark:text-white/50 whitespace-nowrap">Ask AI</span></button></div><div class="flex-1 relative hidden lg:flex items-center ml-auto justify-end space-x-4"><nav class="text-sm"><ul class="flex space-x-6 items-center"><li class="navbar-link"><a href="https://discord.gg/pipecat" class="flex items-center gap-1.5 whitespace-nowrap font-medium text-gray-600 hover:text-gray-900 dark:text-gray-400 dark:hover:text-gray-300" target="_blank">Join the Discord</a></li></ul></nav><div class="flex items-center"><button class="group p-2 flex items-center justify-center" aria-label="Toggle dark mode"><svg width="16" height="16" viewBox="0 0 16 16" fill="none" stroke="currentColor" xmlns="http://www.w3.org/2000/svg" class="h-4 w-4 block text-gray-400 dark:hidden group-hover:text-gray-600"><g clip-path="url(#clip0_2880_7340)"><path d="M8 1.11133V2.00022" stroke-width="1.5" stroke-linecap="round" stroke-linejoin="round"></path><path d="M12.8711 3.12891L12.2427 3.75735" stroke-width="1.5" stroke-linecap="round" stroke-linejoin="round"></path><path d="M14.8889 8H14" stroke-width="1.5" stroke-linecap="round" stroke-linejoin="round"></path><path d="M12.8711 12.8711L12.2427 12.2427" stroke-width="1.5" stroke-linecap="round" stroke-linejoin="round"></path><path d="M8 14.8889V14" stroke-width="1.5" stroke-linecap="round" stroke-linejoin="round"></path><path d="M3.12891 12.8711L3.75735 12.2427" stroke-width="1.5" stroke-linecap="round" stroke-linejoin="round"></path><path d="M1.11133 8H2.00022" stroke-width="1.5" stroke-linecap="round" stroke-linejoin="round"></path><path d="M3.12891 3.12891L3.75735 3.75735" stroke-width="1.5" stroke-linecap="round" stroke-linejoin="round"></path><path d="M8.00043 11.7782C10.0868 11.7782 11.7782 10.0868 11.7782 8.00043C11.7782 5.91402 10.0868 4.22266 8.00043 4.22266C5.91402 4.22266 4.22266 5.91402 4.22266 8.00043C4.22266 10.0868 5.91402 11.7782 8.00043 11.7782Z" stroke-width="1.5" stroke-linecap="round" stroke-linejoin="round"></path></g><defs><clipPath id="clip0_2880_7340"><rect width="16" height="16" fill="white"></rect></clipPath></defs></svg><svg xmlns="http://www.w3.org/2000/svg" width="24" height="24" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-moon h-4 w-4 hidden dark:block text-gray-500 dark:group-hover:text-gray-300"><path d="M12 3a6 6 0 0 0 9 9 9 9 0 1 1-9-9Z"></path></svg></button></div></div><div class="flex lg:hidden items-center gap-3"><button type="button" class="text-gray-500 w-8 h-8 flex items-center justify-center hover:text-gray-600 dark:text-gray-400 dark:hover:text-gray-300" id="search-bar-entry-mobile"><span class="sr-only">Search...</span><svg class="h-4 w-4 bg-gray-500 dark:bg-gray-400 hover:bg-gray-600 dark:hover:bg-gray-300" style="-webkit-mask-image:url(https://mintlify.b-cdn.net/v6.6.0/solid/magnifying-glass.svg);-webkit-mask-repeat:no-repeat;-webkit-mask-position:center;mask-image:url(https://mintlify.b-cdn.net/v6.6.0/solid/magnifying-glass.svg);mask-repeat:no-repeat;mask-position:center"></svg></button><button id="assistant-entry-mobile"><svg xmlns="http://www.w3.org/2000/svg" width="18" height="18" viewBox="0 0 18 18" class="size-4.5 text-gray-500 dark:text-gray-400 hover:text-gray-600 dark:hover:text-gray-300"><g fill="currentColor"><path d="M5.658,2.99l-1.263-.421-.421-1.263c-.137-.408-.812-.408-.949,0l-.421,1.263-1.263,.421c-.204,.068-.342,.259-.342,.474s.138,.406,.342,.474l1.263,.421,.421,1.263c.068,.204,.26,.342,.475,.342s.406-.138,.475-.342l.421-1.263,1.263-.421c.204-.068,.342-.259,.342-.474s-.138-.406-.342-.474Z" fill="currentColor" data-stroke="none" stroke="none"></path><polygon points="9.5 2.75 11.412 7.587 16.25 9.5 11.412 11.413 9.5 16.25 7.587 11.413 2.75 9.5 7.587 7.587 9.5 2.75" fill="none" stroke="currentColor" stroke-linecap="round" stroke-linejoin="round" stroke-width="1.5"></polygon></g></svg></button><button aria-label="More actions" class="h-7 w-5 flex items-center justify-end"><svg class="h-4 w-4 bg-gray-500 dark:bg-gray-400 hover:bg-gray-600 dark:hover:bg-gray-300" style="-webkit-mask-image:url(https://mintlify.b-cdn.net/v6.6.0/solid/ellipsis-vertical.svg);-webkit-mask-repeat:no-repeat;-webkit-mask-position:center;mask-image:url(https://mintlify.b-cdn.net/v6.6.0/solid/ellipsis-vertical.svg);mask-repeat:no-repeat;mask-position:center"></svg></button></div></div></div><button type="button" class="flex items-center h-14 py-4 px-5 lg:hidden focus:outline-none"><div class="text-gray-500 hover:text-gray-600 dark:text-gray-400 dark:hover:text-gray-300"><span class="sr-only">Navigation</span><svg class="h-4" fill="currentColor" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 448 512"><path d="M0 96C0 78.3 14.3 64 32 64H416c17.7 0 32 14.3 32 32s-14.3 32-32 32H32C14.3 128 0 113.7 0 96zM0 256c0-17.7 14.3-32 32-32H416c17.7 0 32 14.3 32 32s-14.3 32-32 32H32c-17.7 0-32-14.3-32-32zM448 416c0 17.7-14.3 32-32 32H32c-17.7 0-32-14.3-32-32s14.3-32 32-32H416c17.7 0 32 14.3 32 32z"></path></svg></div><div class="ml-4 flex text-sm leading-6 whitespace-nowrap min-w-0 space-x-3"><div class="flex items-center space-x-3"><span>Get Started</span><svg width="3" height="24" viewBox="0 -9 3 24" class="h-5 rotate-0 overflow-visible fill-gray-400"><path d="M0 0L3 3L0 6" fill="none" stroke="currentColor" stroke-width="1.5" stroke-linecap="round"></path></svg></div><div class="font-semibold text-gray-900 truncate dark:text-gray-200">Overview</div></div></button></div><div class="hidden lg:flex px-12 h-12"><div class="nav-tabs h-full flex text-sm space-x-6"><a class="link nav-tabs-item group relative h-full flex items-center hover:text-gray-800 dark:hover:text-gray-300 text-gray-800 dark:text-gray-200 font-semibold" href="/getting-started/overview">Getting Started<div class="absolute bottom-0 h-[1.5px] w-full bg-primary dark:bg-primary-light"></div></a><a class="link nav-tabs-item group relative h-full flex items-center font-medium text-gray-600 dark:text-gray-400 hover:text-gray-800 dark:hover:text-gray-300" href="/guides/introduction">Guides<div class="absolute bottom-0 h-[1.5px] w-full group-hover:bg-gray-200 dark:group-hover:bg-gray-700"></div></a><a class="link nav-tabs-item group relative h-full flex items-center font-medium text-gray-600 dark:text-gray-400 hover:text-gray-800 dark:hover:text-gray-300" href="/server/introduction">Server APIs<div class="absolute bottom-0 h-[1.5px] w-full group-hover:bg-gray-200 dark:group-hover:bg-gray-700"></div></a><a class="link nav-tabs-item group relative h-full flex items-center font-medium text-gray-600 dark:text-gray-400 hover:text-gray-800 dark:hover:text-gray-300" href="/client/introduction">Client SDKs<div class="absolute bottom-0 h-[1.5px] w-full group-hover:bg-gray-200 dark:group-hover:bg-gray-700"></div></a></div></div></div></div><span hidden="" style="position:fixed;top:1px;left:1px;width:1px;height:0;padding:0;margin:-1px;overflow:hidden;clip:rect(0, 0, 0, 0);white-space:nowrap;border-width:0;display:none"></span></div><div class="max-w-8xl px-4 mx-auto lg:px-8"><div class="z-20 hidden lg:block fixed bottom-0 right-auto w-[18rem]" id="sidebar" style="top:7rem"><div class="absolute inset-0 z-10 stable-scrollbar-gutter overflow-auto pr-8 pb-10" id="sidebar-content"><div class="relative lg:text-sm lg:leading-6"><div class="sticky top-0 h-8 z-10 bg-gradient-to-b from-background-light dark:from-background-dark"></div><div id="navigation-items"><li class="list-none"><a href="https://discord.gg/pipecat" target="_blank" rel="noreferrer" class="link nav-anchor pl-4 group flex items-center lg:text-sm lg:leading-6 mb-5 sm:mb-4 font-medium text-gray-600 hover:text-gray-900 dark:text-gray-400 dark:hover:text-gray-300"><div class="mr-4 rounded-md p-1 shadow-sm text-gray-400 dark:text-white/50 dark:bg-background-dark dark:brightness-[1.35] dark:ring-1 dark:hover:brightness-150 group-hover:brightness-100 group-hover:ring-0 ring-1 ring-gray-950/5 dark:ring-gray-700/40"><svg class="h-4 w-4 secondary-opacity group-hover:fill-primary-dark group-hover:bg-white bg-gray-400 dark:bg-gray-500" style="-webkit-mask-image:url(https://mintlify.b-cdn.net/v6.6.0/brands/discord.svg);-webkit-mask-repeat:no-repeat;-webkit-mask-position:center;mask-image:url(https://mintlify.b-cdn.net/v6.6.0/brands/discord.svg);mask-repeat:no-repeat;mask-position:center"></svg></div>Community</a></li><li class="list-none"><a href="https://github.com/pipecat-ai/pipecat" target="_blank" rel="noreferrer" class="link nav-anchor pl-4 group flex items-center lg:text-sm lg:leading-6 mb-5 sm:mb-4 font-medium text-gray-600 hover:text-gray-900 dark:text-gray-400 dark:hover:text-gray-300"><div class="mr-4 rounded-md p-1 shadow-sm text-gray-400 dark:text-white/50 dark:bg-background-dark dark:brightness-[1.35] dark:ring-1 dark:hover:brightness-150 group-hover:brightness-100 group-hover:ring-0 ring-1 ring-gray-950/5 dark:ring-gray-700/40"><svg class="h-4 w-4 secondary-opacity group-hover:fill-primary-dark group-hover:bg-white bg-gray-400 dark:bg-gray-500" style="-webkit-mask-image:url(https://mintlify.b-cdn.net/v6.6.0/brands/github.svg);-webkit-mask-repeat:no-repeat;-webkit-mask-position:center;mask-image:url(https://mintlify.b-cdn.net/v6.6.0/brands/github.svg);mask-repeat:no-repeat;mask-position:center"></svg></div>GitHub</a></li><li class="list-none"><a href="https://github.com/pipecat-ai/pipecat/tree/main/examples" target="_blank" rel="noreferrer" class="link nav-anchor pl-4 group flex items-center lg:text-sm lg:leading-6 mb-5 sm:mb-4 font-medium text-gray-600 hover:text-gray-900 dark:text-gray-400 dark:hover:text-gray-300"><div class="mr-4 rounded-md p-1 shadow-sm text-gray-400 dark:text-white/50 dark:bg-background-dark dark:brightness-[1.35] dark:ring-1 dark:hover:brightness-150 group-hover:brightness-100 group-hover:ring-0 ring-1 ring-gray-950/5 dark:ring-gray-700/40"><svg class="h-4 w-4 secondary-opacity group-hover:fill-primary-dark group-hover:bg-white bg-gray-400 dark:bg-gray-500" style="-webkit-mask-image:url(https://mintlify.b-cdn.net/v6.6.0/duotone/code.svg);-webkit-mask-repeat:no-repeat;-webkit-mask-position:center;mask-image:url(https://mintlify.b-cdn.net/v6.6.0/duotone/code.svg);mask-repeat:no-repeat;mask-position:center"></svg></div>Examples</a></li><li class="list-none"><a href="https://github.com/pipecat-ai/pipecat/blob/main/CHANGELOG.md" target="_blank" rel="noreferrer" class="link nav-anchor pl-4 group flex items-center lg:text-sm lg:leading-6 mb-5 sm:mb-4 font-medium text-gray-600 hover:text-gray-900 dark:text-gray-400 dark:hover:text-gray-300"><div class="mr-4 rounded-md p-1 shadow-sm text-gray-400 dark:text-white/50 dark:bg-background-dark dark:brightness-[1.35] dark:ring-1 dark:hover:brightness-150 group-hover:brightness-100 group-hover:ring-0 ring-1 ring-gray-950/5 dark:ring-gray-700/40"><svg class="h-4 w-4 secondary-opacity group-hover:fill-primary-dark group-hover:bg-white bg-gray-400 dark:bg-gray-500" style="-webkit-mask-image:url(https://mintlify.b-cdn.net/v6.6.0/duotone/file.svg);-webkit-mask-repeat:no-repeat;-webkit-mask-position:center;mask-image:url(https://mintlify.b-cdn.net/v6.6.0/duotone/file.svg);mask-repeat:no-repeat;mask-position:center"></svg></div>Changelog</a></li><div class="mt-12 lg:mt-8"><div class="sidebar-group-header flex items-center gap-2.5 pl-4 mb-3.5 lg:mb-2.5 font-semibold text-gray-900 dark:text-gray-200"><h5 id="sidebar-title">Get Started</h5></div><ul id="sidebar-group"><li id="/getting-started/overview" class="relative scroll-m-4 first:scroll-m-20" data-title="Overview"><a class="group mt-2 lg:mt-0 flex items-center pr-3 py-1.5 cursor-pointer focus:outline-primary dark:focus:outline-primary-light gap-x-3 rounded-xl bg-primary/10 text-primary font-semibold dark:text-primary-light dark:bg-primary-light/10" style="padding-left:1rem" href="/getting-started/overview"><div class="flex-1 flex items-center space-x-2.5"><div class="">Overview</div></div></a></li><li id="/getting-started/installation" class="relative scroll-m-4 first:scroll-m-20" data-title="Installation &amp; Setup"><a class="group mt-2 lg:mt-0 flex items-center pr-3 py-1.5 cursor-pointer focus:outline-primary dark:focus:outline-primary-light gap-x-3 rounded-xl hover:bg-gray-600/5 dark:hover:bg-gray-200/5 text-gray-700 hover:text-gray-900 dark:text-gray-400 dark:hover:text-gray-300" style="padding-left:1rem" href="/getting-started/installation"><div class="flex-1 flex items-center space-x-2.5"><div class="">Installation &amp; Setup</div></div></a></li><li id="/getting-started/quickstart" class="relative scroll-m-4 first:scroll-m-20" data-title="Quickstart"><a class="group mt-2 lg:mt-0 flex items-center pr-3 py-1.5 cursor-pointer focus:outline-primary dark:focus:outline-primary-light gap-x-3 rounded-xl hover:bg-gray-600/5 dark:hover:bg-gray-200/5 text-gray-700 hover:text-gray-900 dark:text-gray-400 dark:hover:text-gray-300" style="padding-left:1rem" href="/getting-started/quickstart"><div class="flex-1 flex items-center space-x-2.5"><div class="">Quickstart</div></div></a></li><li id="/getting-started/core-concepts" class="relative scroll-m-4 first:scroll-m-20" data-title="Core Concepts"><a class="group mt-2 lg:mt-0 flex items-center pr-3 py-1.5 cursor-pointer focus:outline-primary dark:focus:outline-primary-light gap-x-3 rounded-xl hover:bg-gray-600/5 dark:hover:bg-gray-200/5 text-gray-700 hover:text-gray-900 dark:text-gray-400 dark:hover:text-gray-300" style="padding-left:1rem" href="/getting-started/core-concepts"><div class="flex-1 flex items-center space-x-2.5"><div class="">Core Concepts</div></div></a></li><li id="/getting-started/next-steps" class="relative scroll-m-4 first:scroll-m-20" data-title="Next Steps &amp; Examples"><a class="group mt-2 lg:mt-0 flex items-center pr-3 py-1.5 cursor-pointer focus:outline-primary dark:focus:outline-primary-light gap-x-3 rounded-xl hover:bg-gray-600/5 dark:hover:bg-gray-200/5 text-gray-700 hover:text-gray-900 dark:text-gray-400 dark:hover:text-gray-300" style="padding-left:1rem" href="/getting-started/next-steps"><div class="flex-1 flex items-center space-x-2.5"><div class="">Next Steps &amp; Examples</div></div></a></li></ul></div></div></div></div></div><div class="" id="content-container"><div class="flex flex-row gap-12 box-border w-full pt-40 lg:pt-10"><div class="relative grow box-border flex-col w-full mx-auto px-1 lg:pl-[23.7rem] lg:-ml-12 xl:w-[calc(100%-28rem)]" id="content-area"><header id="header" class="relative"><div class="mt-0.5 space-y-2.5"><div class="eyebrow h-5 text-primary dark:text-primary-light text-sm font-semibold">Get Started</div><div class="flex items-center relative gap-2"><h1 id="page-title" class="inline-block text-2xl sm:text-3xl font-bold text-gray-900 tracking-tight dark:text-gray-200">Overview</h1></div></div><div class="mt-2 text-lg prose prose-gray dark:prose-invert"><p>Pipecat is a framework for building voice-enabled, real-time, multimodal AI applications</p></div></header><div class="flex flex-col gap-8"></div><div class="mdx-content relative mt-8 prose prose-gray dark:prose-invert" data-page-title="Overview" data-page-href="/getting-started/overview"><p>Pipecat is an open source Python framework that handles the complex orchestration of AI services, network transport, audio processing, and multimodal interactions. “Multimodal” means you can use any combination of audio, video, images, and/or text in your interactions. And “real-time” means that things are happening quickly enough that it feels conversational—a “back-and-forth” with a bot, not submitting a query and waiting for results.</p>
+<h2 class="flex whitespace-pre-wrap group font-semibold" id="what-you-can-build"><div class="absolute"><a href="#what-you-can-build" class="-ml-10 flex items-center opacity-0 border-0 group-hover:opacity-100" aria-label="Navigate to header">​<div class="w-6 h-6 rounded-md flex items-center justify-center shadow-sm text-gray-400 dark:text-white/50 dark:bg-background-dark dark:brightness-[1.35] dark:ring-1 dark:hover:brightness-150 bg-white ring-1 ring-gray-400/30 dark:ring-gray-700/25 hover:ring-gray-400/60 dark:hover:ring-white/20"><svg xmlns="http://www.w3.org/2000/svg" fill="gray" height="12px" viewBox="0 0 576 512"><path d="M0 256C0 167.6 71.6 96 160 96h72c13.3 0 24 10.7 24 24s-10.7 24-24 24H160C98.1 144 48 194.1 48 256s50.1 112 112 112h72c13.3 0 24 10.7 24 24s-10.7 24-24 24H160C71.6 416 0 344.4 0 256zm576 0c0 88.4-71.6 160-160 160H344c-13.3 0-24-10.7-24-24s10.7-24 24-24h72c61.9 0 112-50.1 112-112s-50.1-112-112-112H344c-13.3 0-24-10.7-24-24s10.7-24 24-24h72c88.4 0 160 71.6 160 160zM184 232H392c13.3 0 24 10.7 24 24s-10.7 24-24 24H184c-13.3 0-24-10.7-24-24s10.7-24 24-24z"></path></svg></div></a></div><span class="cursor-pointer">What You Can Build</span></h2>
+<div class="card-group not-prose grid gap-x-4 sm:grid-cols-3"><a class="card block font-normal group relative my-2 ring-2 ring-transparent rounded-2xl bg-white dark:bg-background-dark border border-gray-950/10 dark:border-white/10 overflow-hidden w-full cursor-pointer hover:!border-primary dark:hover:!border-primary-light" target="_blank" rel="noreferrer" href="https://github.com/pipecat-ai/pipecat/tree/main/examples/simple-chatbot"><div class="px-6 py-5 relative" data-component-part="card-content-container"><div id="card-link-arrow-icon" class="absolute text-gray-400 dark:text-gray-500 group-hover:text-primary dark:group-hover:text-primary-light top-5 right-5"><svg xmlns="http://www.w3.org/2000/svg" width="24" height="24" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-arrow-up-right w-4 h-4"><path d="M7 7h10v10"></path><path d="M7 17 17 7"></path></svg></div><div class="h-6 w-6 fill-gray-800 dark:fill-gray-100 text-gray-800 dark:text-gray-100" data-component-part="card-icon"><svg class="h-6 w-6 bg-primary dark:bg-primary-light !m-0 shrink-0" style="-webkit-mask-image:url(https://mintlify.b-cdn.net/v6.6.0/regular/microphone.svg);-webkit-mask-repeat:no-repeat;-webkit-mask-position:center;mask-image:url(https://mintlify.b-cdn.net/v6.6.0/regular/microphone.svg);mask-repeat:no-repeat;mask-position:center"></svg></div><div><h2 class="not-prose font-semibold text-base text-gray-800 dark:text-white mt-4" contentEditable="false" data-component-part="card-title">Voice Assistants</h2><div class="prose mt-1 font-normal text-sm leading-6 text-gray-600 dark:text-gray-400" data-component-part="card-content"><p>Natural, real-time conversations with AI using speech recognition and
+synthesis</p></div></div></div></a><a class="card block font-normal group relative my-2 ring-2 ring-transparent rounded-2xl bg-white dark:bg-background-dark border border-gray-950/10 dark:border-white/10 overflow-hidden w-full cursor-pointer hover:!border-primary dark:hover:!border-primary-light" target="_blank" rel="noreferrer" href="https://github.com/pipecat-ai/pipecat/tree/main/examples/studypal"><div class="px-6 py-5 relative" data-component-part="card-content-container"><div id="card-link-arrow-icon" class="absolute text-gray-400 dark:text-gray-500 group-hover:text-primary dark:group-hover:text-primary-light top-5 right-5"><svg xmlns="http://www.w3.org/2000/svg" width="24" height="24" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-arrow-up-right w-4 h-4"><path d="M7 7h10v10"></path><path d="M7 17 17 7"></path></svg></div><div class="h-6 w-6 fill-gray-800 dark:fill-gray-100 text-gray-800 dark:text-gray-100" data-component-part="card-icon"><svg class="h-6 w-6 bg-primary dark:bg-primary-light !m-0 shrink-0" style="-webkit-mask-image:url(https://mintlify.b-cdn.net/v6.6.0/regular/robot.svg);-webkit-mask-repeat:no-repeat;-webkit-mask-position:center;mask-image:url(https://mintlify.b-cdn.net/v6.6.0/regular/robot.svg);mask-repeat:no-repeat;mask-position:center"></svg></div><div><h2 class="not-prose font-semibold text-base text-gray-800 dark:text-white mt-4" contentEditable="false" data-component-part="card-title">Interactive Agents</h2><div class="prose mt-1 font-normal text-sm leading-6 text-gray-600 dark:text-gray-400" data-component-part="card-content"><p>Personal coaches and meeting assistants that can understand context and
+provide guidance</p></div></div></div></a><a class="card block font-normal group relative my-2 ring-2 ring-transparent rounded-2xl bg-white dark:bg-background-dark border border-gray-950/10 dark:border-white/10 overflow-hidden w-full cursor-pointer hover:!border-primary dark:hover:!border-primary-light" target="_blank" rel="noreferrer" href="https://github.com/pipecat-ai/pipecat/tree/main/examples/moondream-chatbot"><div class="px-6 py-5 relative" data-component-part="card-content-container"><div id="card-link-arrow-icon" class="absolute text-gray-400 dark:text-gray-500 group-hover:text-primary dark:group-hover:text-primary-light top-5 right-5"><svg xmlns="http://www.w3.org/2000/svg" width="24" height="24" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-arrow-up-right w-4 h-4"><path d="M7 7h10v10"></path><path d="M7 17 17 7"></path></svg></div><div class="h-6 w-6 fill-gray-800 dark:fill-gray-100 text-gray-800 dark:text-gray-100" data-component-part="card-icon"><svg class="h-6 w-6 bg-primary dark:bg-primary-light !m-0 shrink-0" style="-webkit-mask-image:url(https://mintlify.b-cdn.net/v6.6.0/regular/layer-group.svg);-webkit-mask-repeat:no-repeat;-webkit-mask-position:center;mask-image:url(https://mintlify.b-cdn.net/v6.6.0/regular/layer-group.svg);mask-repeat:no-repeat;mask-position:center"></svg></div><div><h2 class="not-prose font-semibold text-base text-gray-800 dark:text-white mt-4" contentEditable="false" data-component-part="card-title">Multimodal Apps</h2><div class="prose mt-1 font-normal text-sm leading-6 text-gray-600 dark:text-gray-400" data-component-part="card-content"><p>Applications that combine voice, video, images, and text for rich interactions</p></div></div></div></a><a class="card block font-normal group relative my-2 ring-2 ring-transparent rounded-2xl bg-white dark:bg-background-dark border border-gray-950/10 dark:border-white/10 overflow-hidden w-full cursor-pointer hover:!border-primary dark:hover:!border-primary-light" target="_blank" rel="noreferrer" href="https://github.com/pipecat-ai/pipecat/tree/main/examples/storytelling-chatbot"><div class="px-6 py-5 relative" data-component-part="card-content-container"><div id="card-link-arrow-icon" class="absolute text-gray-400 dark:text-gray-500 group-hover:text-primary dark:group-hover:text-primary-light top-5 right-5"><svg xmlns="http://www.w3.org/2000/svg" width="24" height="24" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-arrow-up-right w-4 h-4"><path d="M7 7h10v10"></path><path d="M7 17 17 7"></path></svg></div><div class="h-6 w-6 fill-gray-800 dark:fill-gray-100 text-gray-800 dark:text-gray-100" data-component-part="card-icon"><svg class="h-6 w-6 bg-primary dark:bg-primary-light !m-0 shrink-0" style="-webkit-mask-image:url(https://mintlify.b-cdn.net/v6.6.0/regular/wand-magic-sparkles.svg);-webkit-mask-repeat:no-repeat;-webkit-mask-position:center;mask-image:url(https://mintlify.b-cdn.net/v6.6.0/regular/wand-magic-sparkles.svg);mask-repeat:no-repeat;mask-position:center"></svg></div><div><h2 class="not-prose font-semibold text-base text-gray-800 dark:text-white mt-4" contentEditable="false" data-component-part="card-title">Creative Tools</h2><div class="prose mt-1 font-normal text-sm leading-6 text-gray-600 dark:text-gray-400" data-component-part="card-content"><p>Storytelling experiences and social companions that engage users</p></div></div></div></a><a class="card block font-normal group relative my-2 ring-2 ring-transparent rounded-2xl bg-white dark:bg-background-dark border border-gray-950/10 dark:border-white/10 overflow-hidden w-full cursor-pointer hover:!border-primary dark:hover:!border-primary-light" target="_blank" rel="noreferrer" href="https://github.com/pipecat-ai/pipecat/tree/main/examples/patient-intake"><div class="px-6 py-5 relative" data-component-part="card-content-container"><div id="card-link-arrow-icon" class="absolute text-gray-400 dark:text-gray-500 group-hover:text-primary dark:group-hover:text-primary-light top-5 right-5"><svg xmlns="http://www.w3.org/2000/svg" width="24" height="24" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-arrow-up-right w-4 h-4"><path d="M7 7h10v10"></path><path d="M7 17 17 7"></path></svg></div><div class="h-6 w-6 fill-gray-800 dark:fill-gray-100 text-gray-800 dark:text-gray-100" data-component-part="card-icon"><svg class="h-6 w-6 bg-primary dark:bg-primary-light !m-0 shrink-0" style="-webkit-mask-image:url(https://mintlify.b-cdn.net/v6.6.0/regular/building.svg);-webkit-mask-repeat:no-repeat;-webkit-mask-position:center;mask-image:url(https://mintlify.b-cdn.net/v6.6.0/regular/building.svg);mask-repeat:no-repeat;mask-position:center"></svg></div><div><h2 class="not-prose font-semibold text-base text-gray-800 dark:text-white mt-4" contentEditable="false" data-component-part="card-title">Business Solutions</h2><div class="prose mt-1 font-normal text-sm leading-6 text-gray-600 dark:text-gray-400" data-component-part="card-content"><p>Customer intake flows and support bots for automated business processes</p></div></div></div></a><a class="card block font-normal group relative my-2 ring-2 ring-transparent rounded-2xl bg-white dark:bg-background-dark border border-gray-950/10 dark:border-white/10 overflow-hidden w-full cursor-pointer hover:!border-primary dark:hover:!border-primary-light" target="_blank" rel="noreferrer" href="https://github.com/pipecat-ai/pipecat-flows"><div class="px-6 py-5 relative" data-component-part="card-content-container"><div id="card-link-arrow-icon" class="absolute text-gray-400 dark:text-gray-500 group-hover:text-primary dark:group-hover:text-primary-light top-5 right-5"><svg xmlns="http://www.w3.org/2000/svg" width="24" height="24" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-arrow-up-right w-4 h-4"><path d="M7 7h10v10"></path><path d="M7 17 17 7"></path></svg></div><div class="h-6 w-6 fill-gray-800 dark:fill-gray-100 text-gray-800 dark:text-gray-100" data-component-part="card-icon"><svg class="h-6 w-6 bg-primary dark:bg-primary-light !m-0 shrink-0" style="-webkit-mask-image:url(https://mintlify.b-cdn.net/v6.6.0/regular/diagram-project.svg);-webkit-mask-repeat:no-repeat;-webkit-mask-position:center;mask-image:url(https://mintlify.b-cdn.net/v6.6.0/regular/diagram-project.svg);mask-repeat:no-repeat;mask-position:center"></svg></div><div><h2 class="not-prose font-semibold text-base text-gray-800 dark:text-white mt-4" contentEditable="false" data-component-part="card-title">Complex Flows</h2><div class="prose mt-1 font-normal text-sm leading-6 text-gray-600 dark:text-gray-400" data-component-part="card-content"><p>Structured conversations using Pipecat Flows for managing complex
+interactions</p></div></div></div></a></div>
+<h2 class="flex whitespace-pre-wrap group font-semibold" id="how-it-works"><div class="absolute"><a href="#how-it-works" class="-ml-10 flex items-center opacity-0 border-0 group-hover:opacity-100" aria-label="Navigate to header">​<div class="w-6 h-6 rounded-md flex items-center justify-center shadow-sm text-gray-400 dark:text-white/50 dark:bg-background-dark dark:brightness-[1.35] dark:ring-1 dark:hover:brightness-150 bg-white ring-1 ring-gray-400/30 dark:ring-gray-700/25 hover:ring-gray-400/60 dark:hover:ring-white/20"><svg xmlns="http://www.w3.org/2000/svg" fill="gray" height="12px" viewBox="0 0 576 512"><path d="M0 256C0 167.6 71.6 96 160 96h72c13.3 0 24 10.7 24 24s-10.7 24-24 24H160C98.1 144 48 194.1 48 256s50.1 112 112 112h72c13.3 0 24 10.7 24 24s-10.7 24-24 24H160C71.6 416 0 344.4 0 256zm576 0c0 88.4-71.6 160-160 160H344c-13.3 0-24-10.7-24-24s10.7-24 24-24h72c61.9 0 112-50.1 112-112s-50.1-112-112-112H344c-13.3 0-24-10.7-24-24s10.7-24 24-24h72c88.4 0 160 71.6 160 160zM184 232H392c13.3 0 24 10.7 24 24s-10.7 24-24 24H184c-13.3 0-24-10.7-24-24s10.7-24 24-24z"></path></svg></div></a></div><span class="cursor-pointer">How It Works</span></h2>
+<p>The flow of interactions in a Pipecat application is typically straightforward:</p>
+<ol>
+<li>The bot says something</li>
+<li>The user says something</li>
+<li>The bot says something</li>
+<li>The user says something</li>
+</ol>
+<p>This continues until the conversation naturally ends. While this flow seems simple, making it feel natural requires sophisticated real-time processing.</p>
+<h3 class="flex whitespace-pre-wrap group font-semibold" id="real-time-processing"><div class="absolute"><a href="#real-time-processing" class="-ml-10 flex items-center opacity-0 border-0 group-hover:opacity-100" aria-label="Navigate to header">​<div class="w-6 h-6 rounded-md flex items-center justify-center shadow-sm text-gray-400 dark:text-white/50 dark:bg-background-dark dark:brightness-[1.35] dark:ring-1 dark:hover:brightness-150 bg-white ring-1 ring-gray-400/30 dark:ring-gray-700/25 hover:ring-gray-400/60 dark:hover:ring-white/20"><svg xmlns="http://www.w3.org/2000/svg" fill="gray" height="12px" viewBox="0 0 576 512"><path d="M0 256C0 167.6 71.6 96 160 96h72c13.3 0 24 10.7 24 24s-10.7 24-24 24H160C98.1 144 48 194.1 48 256s50.1 112 112 112h72c13.3 0 24 10.7 24 24s-10.7 24-24 24H160C71.6 416 0 344.4 0 256zm576 0c0 88.4-71.6 160-160 160H344c-13.3 0-24-10.7-24-24s10.7-24 24-24h72c61.9 0 112-50.1 112-112s-50.1-112-112-112H344c-13.3 0-24-10.7-24-24s10.7-24 24-24h72c88.4 0 160 71.6 160 160zM184 232H392c13.3 0 24 10.7 24 24s-10.7 24-24 24H184c-13.3 0-24-10.7-24-24s10.7-24 24-24z"></path></svg></div></a></div><span class="cursor-pointer">Real-time Processing</span></h3>
+<p>Pipecat’s pipeline architecture handles both simple voice interactions and complex multimodal processing. Let’s look at how data flows through the system:</p>
+<div class="tabs tabs tab-container" id="voice-app"><ul class="not-prose mb-6 pb-[1px] flex-none min-w-full overflow-auto border-b border-gray-200 gap-x-6 flex dark:border-gray-200/10" data-component-part="tabs-list"><li class="cursor-pointer"><button class="flex text-sm leading-6 font-semibold whitespace-nowrap pt-3 pb-2.5 -mb-px max-w-max border-b text-primary dark:text-primary-light border-current" data-component-part="tab-button" data-active="true">Voice app</button></li><li class="cursor-pointer"><button class="flex text-sm leading-6 font-semibold whitespace-nowrap pt-3 pb-2.5 -mb-px max-w-max border-b text-gray-900 border-transparent hover:border-gray-300 dark:text-gray-200 dark:hover:border-gray-700" data-component-part="tab-button" data-active="false">Multimodal app</button></li></ul><div class="prose dark:prose-dark overflow-x-auto" data-component-part="tab-content"><div role="list" class="steps ml-3.5 mt-10 mb-6"><div role="listitem" class="step relative flex items-start pb-5"><div class="absolute w-px h-[calc(100%-2.75rem)] top-[2.75rem] bg-gray-200/70 dark:bg-white/10" contentEditable="false"></div><div class="absolute ml-[-13px] py-2" data-component-part="step-number" contentEditable="false"><div class="size-7 shrink-0 rounded-full bg-gray-50 dark:bg-white/10 text-xs text-gray-900 dark:text-gray-50 font-semibold flex items-center justify-center">1</div></div><div class="w-full overflow-hidden pl-8 pr-px"><p class="mt-2 font-semibold prose dark:prose-invert text-gray-900 dark:text-gray-200" contentEditable="false" data-component-part="step-title">Send Audio</p><div data-component-part="step-content"><p>Transmit and capture streamed audio from the user</p></div></div></div><div role="listitem" class="step relative flex items-start pb-5"><div class="absolute w-px h-[calc(100%-2.75rem)] top-[2.75rem] bg-gray-200/70 dark:bg-white/10" contentEditable="false"></div><div class="absolute ml-[-13px] py-2" data-component-part="step-number" contentEditable="false"><div class="size-7 shrink-0 rounded-full bg-gray-50 dark:bg-white/10 text-xs text-gray-900 dark:text-gray-50 font-semibold flex items-center justify-center">2</div></div><div class="w-full overflow-hidden pl-8 pr-px"><p class="mt-2 font-semibold prose dark:prose-invert text-gray-900 dark:text-gray-200" contentEditable="false" data-component-part="step-title">Transcribe Speech</p><div data-component-part="step-content"><p>Convert speech to text as the user is talking</p></div></div></div><div role="listitem" class="step relative flex items-start pb-5"><div class="absolute w-px h-[calc(100%-2.75rem)] top-[2.75rem] bg-gray-200/70 dark:bg-white/10" contentEditable="false"></div><div class="absolute ml-[-13px] py-2" data-component-part="step-number" contentEditable="false"><div class="size-7 shrink-0 rounded-full bg-gray-50 dark:bg-white/10 text-xs text-gray-900 dark:text-gray-50 font-semibold flex items-center justify-center">3</div></div><div class="w-full overflow-hidden pl-8 pr-px"><p class="mt-2 font-semibold prose dark:prose-invert text-gray-900 dark:text-gray-200" contentEditable="false" data-component-part="step-title">Process with LLM</p><div data-component-part="step-content"><p>Generate responses using a large language model</p></div></div></div><div role="listitem" class="step relative flex items-start pb-5"><div class="absolute w-px h-[calc(100%-2.75rem)] top-[2.75rem] bg-gray-200/70 dark:bg-white/10" contentEditable="false"></div><div class="absolute ml-[-13px] py-2" data-component-part="step-number" contentEditable="false"><div class="size-7 shrink-0 rounded-full bg-gray-50 dark:bg-white/10 text-xs text-gray-900 dark:text-gray-50 font-semibold flex items-center justify-center">4</div></div><div class="w-full overflow-hidden pl-8 pr-px"><p class="mt-2 font-semibold prose dark:prose-invert text-gray-900 dark:text-gray-200" contentEditable="false" data-component-part="step-title">Convert to Speech</p><div data-component-part="step-content"><p>Transform text responses into natural speech</p></div></div></div><div role="listitem" class="step relative flex items-start pb-5"><div class="absolute w-px h-[calc(100%-2.75rem)] top-[2.75rem] bg-transparent bg-gradient-to-b from-gray-200 dark:from-white/10 via-80% to-transparent" contentEditable="false"></div><div class="absolute ml-[-13px] py-2" data-component-part="step-number" contentEditable="false"><div class="size-7 shrink-0 rounded-full bg-gray-50 dark:bg-white/10 text-xs text-gray-900 dark:text-gray-50 font-semibold flex items-center justify-center">5</div></div><div class="w-full overflow-hidden pl-8 pr-px"><p class="mt-2 font-semibold prose dark:prose-invert text-gray-900 dark:text-gray-200" contentEditable="false" data-component-part="step-title">Play Audio</p><div data-component-part="step-content">Stream the audio response back to the user</div></div></div></div></div><div class="hidden"><div><div role="list" class="steps ml-3.5 mt-10 mb-6"><div role="listitem" class="step relative flex items-start pb-5"><div class="absolute w-px h-[calc(100%-2.75rem)] top-[2.75rem] bg-gray-200/70 dark:bg-white/10" contentEditable="false"></div><div class="absolute ml-[-13px] py-2" data-component-part="step-number" contentEditable="false"><div class="size-7 shrink-0 rounded-full bg-gray-50 dark:bg-white/10 text-xs text-gray-900 dark:text-gray-50 font-semibold flex items-center justify-center">1</div></div><div class="w-full overflow-hidden pl-8 pr-px"><p class="mt-2 font-semibold prose dark:prose-invert text-gray-900 dark:text-gray-200" contentEditable="false" data-component-part="step-title">Send Audio</p><div data-component-part="step-content"><p>Transmit and capture streamed audio from the user</p></div></div></div><div role="listitem" class="step relative flex items-start pb-5"><div class="absolute w-px h-[calc(100%-2.75rem)] top-[2.75rem] bg-gray-200/70 dark:bg-white/10" contentEditable="false"></div><div class="absolute ml-[-13px] py-2" data-component-part="step-number" contentEditable="false"><div class="size-7 shrink-0 rounded-full bg-gray-50 dark:bg-white/10 text-xs text-gray-900 dark:text-gray-50 font-semibold flex items-center justify-center">2</div></div><div class="w-full overflow-hidden pl-8 pr-px"><p class="mt-2 font-semibold prose dark:prose-invert text-gray-900 dark:text-gray-200" contentEditable="false" data-component-part="step-title">Transcribe Speech</p><div data-component-part="step-content"><p>Convert speech to text as the user is talking</p></div></div></div><div role="listitem" class="step relative flex items-start pb-5"><div class="absolute w-px h-[calc(100%-2.75rem)] top-[2.75rem] bg-gray-200/70 dark:bg-white/10" contentEditable="false"></div><div class="absolute ml-[-13px] py-2" data-component-part="step-number" contentEditable="false"><div class="size-7 shrink-0 rounded-full bg-gray-50 dark:bg-white/10 text-xs text-gray-900 dark:text-gray-50 font-semibold flex items-center justify-center">3</div></div><div class="w-full overflow-hidden pl-8 pr-px"><p class="mt-2 font-semibold prose dark:prose-invert text-gray-900 dark:text-gray-200" contentEditable="false" data-component-part="step-title">Process with LLM</p><div data-component-part="step-content"><p>Generate responses using a large language model</p></div></div></div><div role="listitem" class="step relative flex items-start pb-5"><div class="absolute w-px h-[calc(100%-2.75rem)] top-[2.75rem] bg-gray-200/70 dark:bg-white/10" contentEditable="false"></div><div class="absolute ml-[-13px] py-2" data-component-part="step-number" contentEditable="false"><div class="size-7 shrink-0 rounded-full bg-gray-50 dark:bg-white/10 text-xs text-gray-900 dark:text-gray-50 font-semibold flex items-center justify-center">4</div></div><div class="w-full overflow-hidden pl-8 pr-px"><p class="mt-2 font-semibold prose dark:prose-invert text-gray-900 dark:text-gray-200" contentEditable="false" data-component-part="step-title">Convert to Speech</p><div data-component-part="step-content"><p>Transform text responses into natural speech</p></div></div></div><div role="listitem" class="step relative flex items-start pb-5"><div class="absolute w-px h-[calc(100%-2.75rem)] top-[2.75rem] bg-transparent bg-gradient-to-b from-gray-200 dark:from-white/10 via-80% to-transparent" contentEditable="false"></div><div class="absolute ml-[-13px] py-2" data-component-part="step-number" contentEditable="false"><div class="size-7 shrink-0 rounded-full bg-gray-50 dark:bg-white/10 text-xs text-gray-900 dark:text-gray-50 font-semibold flex items-center justify-center">5</div></div><div class="w-full overflow-hidden pl-8 pr-px"><p class="mt-2 font-semibold prose dark:prose-invert text-gray-900 dark:text-gray-200" contentEditable="false" data-component-part="step-title">Play Audio</p><div data-component-part="step-content">Stream the audio response back to the user</div></div></div></div></div><div><div role="list" class="steps ml-3.5 mt-10 mb-6"><div role="listitem" class="step relative flex items-start pb-5"><div class="absolute w-px h-[calc(100%-2.75rem)] top-[2.75rem] bg-gray-200/70 dark:bg-white/10" contentEditable="false"></div><div class="absolute ml-[-13px] py-2" data-component-part="step-number" contentEditable="false"><div class="size-7 shrink-0 rounded-full bg-gray-50 dark:bg-white/10 text-xs text-gray-900 dark:text-gray-50 font-semibold flex items-center justify-center">1</div></div><div class="w-full overflow-hidden pl-8 pr-px"><p class="mt-2 font-semibold prose dark:prose-invert text-gray-900 dark:text-gray-200" contentEditable="false" data-component-part="step-title">Send Audio and Video</p><div data-component-part="step-content"><p>Transmit and capture audio, video, and image inputs simultaneously</p></div></div></div><div role="listitem" class="step relative flex items-start pb-5"><div class="absolute w-px h-[calc(100%-2.75rem)] top-[2.75rem] bg-gray-200/70 dark:bg-white/10" contentEditable="false"></div><div class="absolute ml-[-13px] py-2" data-component-part="step-number" contentEditable="false"><div class="size-7 shrink-0 rounded-full bg-gray-50 dark:bg-white/10 text-xs text-gray-900 dark:text-gray-50 font-semibold flex items-center justify-center">2</div></div><div class="w-full overflow-hidden pl-8 pr-px"><p class="mt-2 font-semibold prose dark:prose-invert text-gray-900 dark:text-gray-200" contentEditable="false" data-component-part="step-title">Process Streams</p><div data-component-part="step-content">Handle multiple input streams in parallel</div></div></div><div role="listitem" class="step relative flex items-start pb-5"><div class="absolute w-px h-[calc(100%-2.75rem)] top-[2.75rem] bg-gray-200/70 dark:bg-white/10" contentEditable="false"></div><div class="absolute ml-[-13px] py-2" data-component-part="step-number" contentEditable="false"><div class="size-7 shrink-0 rounded-full bg-gray-50 dark:bg-white/10 text-xs text-gray-900 dark:text-gray-50 font-semibold flex items-center justify-center">3</div></div><div class="w-full overflow-hidden pl-8 pr-px"><p class="mt-2 font-semibold prose dark:prose-invert text-gray-900 dark:text-gray-200" contentEditable="false" data-component-part="step-title">Model Processing</p><div data-component-part="step-content"><p>Send combined inputs to multimodal models (like GPT-4V)</p></div></div></div><div role="listitem" class="step relative flex items-start pb-5"><div class="absolute w-px h-[calc(100%-2.75rem)] top-[2.75rem] bg-gray-200/70 dark:bg-white/10" contentEditable="false"></div><div class="absolute ml-[-13px] py-2" data-component-part="step-number" contentEditable="false"><div class="size-7 shrink-0 rounded-full bg-gray-50 dark:bg-white/10 text-xs text-gray-900 dark:text-gray-50 font-semibold flex items-center justify-center">4</div></div><div class="w-full overflow-hidden pl-8 pr-px"><p class="mt-2 font-semibold prose dark:prose-invert text-gray-900 dark:text-gray-200" contentEditable="false" data-component-part="step-title">Generate Outputs</p><div data-component-part="step-content"><p>Create various outputs (text, images, audio, etc.)</p></div></div></div><div role="listitem" class="step relative flex items-start pb-5"><div class="absolute w-px h-[calc(100%-2.75rem)] top-[2.75rem] bg-transparent bg-gradient-to-b from-gray-200 dark:from-white/10 via-80% to-transparent" contentEditable="false"></div><div class="absolute ml-[-13px] py-2" data-component-part="step-number" contentEditable="false"><div class="size-7 shrink-0 rounded-full bg-gray-50 dark:bg-white/10 text-xs text-gray-900 dark:text-gray-50 font-semibold flex items-center justify-center">5</div></div><div class="w-full overflow-hidden pl-8 pr-px"><p class="mt-2 font-semibold prose dark:prose-invert text-gray-900 dark:text-gray-200" contentEditable="false" data-component-part="step-title">Coordinate Presentation</p><div data-component-part="step-content"><p>Synchronize and present multiple output types</p></div></div></div></div></div></div></div>
+<p>In both cases, Pipecat:</p>
+<ul>
+<li>Processes responses as they stream in</li>
+<li>Handles multiple input/output modalities concurrently</li>
+<li>Manages resource allocation and synchronization</li>
+<li>Coordinates parallel processing tasks</li>
+</ul>
+<p>This architecture creates fluid, natural interactions without noticeable delays, whether you’re building a simple voice assistant or a complex multimodal application. Pipecat’s pipeline architecture is particularly valuable for managing the complexity of real-time, multimodal interactions, ensuring smooth data flow and proper synchronization regardless of the input/output types involved.</p>
+<div class="callout my-4 px-5 py-4 overflow-hidden rounded-2xl flex gap-3 border border-sky-500/20 bg-sky-50/50 dark:border-sky-500/30 dark:bg-sky-500/10" data-callout-type="note"><div class="mt-0.5 w-4" data-component-part="callout-icon"><svg width="14" height="14" viewBox="0 0 14 14" fill="currentColor" xmlns="http://www.w3.org/2000/svg" class="w-4 h-4 text-sky-500" aria-label="Note"><path fill-rule="evenodd" clip-rule="evenodd" d="M7 1.3C10.14 1.3 12.7 3.86 12.7 7C12.7 10.14 10.14 12.7 7 12.7C5.48908 12.6974 4.0408 12.096 2.97241 11.0276C1.90403 9.9592 1.30264 8.51092 1.3 7C1.3 3.86 3.86 1.3 7 1.3ZM7 0C3.14 0 0 3.14 0 7C0 10.86 3.14 14 7 14C10.86 14 14 10.86 14 7C14 3.14 10.86 0 7 0ZM8 3H6V8H8V3ZM8 9H6V11H8V9Z"></path></svg></div><div class="text-sm prose min-w-0 w-full text-sky-900 dark:text-sky-200" data-component-part="callout-content"><p>Pipecat handles all this complexity for you, letting you focus on building
+your application rather than managing the underlying infrastructure.</p></div></div>
+<h2 class="flex whitespace-pre-wrap group font-semibold" id="next-steps"><div class="absolute"><a href="#next-steps" class="-ml-10 flex items-center opacity-0 border-0 group-hover:opacity-100" aria-label="Navigate to header">​<div class="w-6 h-6 rounded-md flex items-center justify-center shadow-sm text-gray-400 dark:text-white/50 dark:bg-background-dark dark:brightness-[1.35] dark:ring-1 dark:hover:brightness-150 bg-white ring-1 ring-gray-400/30 dark:ring-gray-700/25 hover:ring-gray-400/60 dark:hover:ring-white/20"><svg xmlns="http://www.w3.org/2000/svg" fill="gray" height="12px" viewBox="0 0 576 512"><path d="M0 256C0 167.6 71.6 96 160 96h72c13.3 0 24 10.7 24 24s-10.7 24-24 24H160C98.1 144 48 194.1 48 256s50.1 112 112 112h72c13.3 0 24 10.7 24 24s-10.7 24-24 24H160C71.6 416 0 344.4 0 256zm576 0c0 88.4-71.6 160-160 160H344c-13.3 0-24-10.7-24-24s10.7-24 24-24h72c61.9 0 112-50.1 112-112s-50.1-112-112-112H344c-13.3 0-24-10.7-24-24s10.7-24 24-24h72c88.4 0 160 71.6 160 160zM184 232H392c13.3 0 24 10.7 24 24s-10.7 24-24 24H184c-13.3 0-24-10.7-24-24s10.7-24 24-24z"></path></svg></div></a></div><span class="cursor-pointer">Next Steps</span></h2>
+<p>Ready to build your first Pipecat application?</p>
+<div class="card-group not-prose grid gap-x-4 sm:grid-cols-2"><a class="card block font-normal group relative my-2 ring-2 ring-transparent rounded-2xl bg-white dark:bg-background-dark border border-gray-950/10 dark:border-white/10 overflow-hidden w-full cursor-pointer hover:!border-primary dark:hover:!border-primary-light" href="/getting-started/installation"><div class="px-6 py-5 relative" data-component-part="card-content-container"><div id="card-link-arrow-icon" class="absolute text-gray-400 dark:text-gray-500 group-hover:text-primary dark:group-hover:text-primary-light top-5 right-5 hidden"><svg xmlns="http://www.w3.org/2000/svg" width="24" height="24" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-arrow-up-right w-4 h-4"><path d="M7 7h10v10"></path><path d="M7 17 17 7"></path></svg></div><div class="h-6 w-6 fill-gray-800 dark:fill-gray-100 text-gray-800 dark:text-gray-100" data-component-part="card-icon"><svg class="h-6 w-6 bg-primary dark:bg-primary-light !m-0 shrink-0" style="-webkit-mask-image:url(https://mintlify.b-cdn.net/v6.6.0/regular/wrench.svg);-webkit-mask-repeat:no-repeat;-webkit-mask-position:center;mask-image:url(https://mintlify.b-cdn.net/v6.6.0/regular/wrench.svg);mask-repeat:no-repeat;mask-position:center"></svg></div><div><h2 class="not-prose font-semibold text-base text-gray-800 dark:text-white mt-4" contentEditable="false" data-component-part="card-title">Installation &amp; Setup</h2><div class="prose mt-1 font-normal text-sm leading-6 text-gray-600 dark:text-gray-400" data-component-part="card-content"><p>Prepare your environment and install required dependencies</p></div></div></div></a><a class="card block font-normal group relative my-2 ring-2 ring-transparent rounded-2xl bg-white dark:bg-background-dark border border-gray-950/10 dark:border-white/10 overflow-hidden w-full cursor-pointer hover:!border-primary dark:hover:!border-primary-light" href="/getting-started/quickstart"><div class="px-6 py-5 relative" data-component-part="card-content-container"><div id="card-link-arrow-icon" class="absolute text-gray-400 dark:text-gray-500 group-hover:text-primary dark:group-hover:text-primary-light top-5 right-5 hidden"><svg xmlns="http://www.w3.org/2000/svg" width="24" height="24" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-arrow-up-right w-4 h-4"><path d="M7 7h10v10"></path><path d="M7 17 17 7"></path></svg></div><div class="h-6 w-6 fill-gray-800 dark:fill-gray-100 text-gray-800 dark:text-gray-100" data-component-part="card-icon"><svg class="h-6 w-6 bg-primary dark:bg-primary-light !m-0 shrink-0" style="-webkit-mask-image:url(https://mintlify.b-cdn.net/v6.6.0/regular/rocket.svg);-webkit-mask-repeat:no-repeat;-webkit-mask-position:center;mask-image:url(https://mintlify.b-cdn.net/v6.6.0/regular/rocket.svg);mask-repeat:no-repeat;mask-position:center"></svg></div><div><h2 class="not-prose font-semibold text-base text-gray-800 dark:text-white mt-4" contentEditable="false" data-component-part="card-title">Quickstart</h2><div class="prose mt-1 font-normal text-sm leading-6 text-gray-600 dark:text-gray-400" data-component-part="card-content"><p>Build and run your first Pipecat application</p></div></div></div></a><a class="card block font-normal group relative my-2 ring-2 ring-transparent rounded-2xl bg-white dark:bg-background-dark border border-gray-950/10 dark:border-white/10 overflow-hidden w-full cursor-pointer hover:!border-primary dark:hover:!border-primary-light" href="/getting-started/core-concepts"><div class="px-6 py-5 relative" data-component-part="card-content-container"><div id="card-link-arrow-icon" class="absolute text-gray-400 dark:text-gray-500 group-hover:text-primary dark:group-hover:text-primary-light top-5 right-5 hidden"><svg xmlns="http://www.w3.org/2000/svg" width="24" height="24" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-arrow-up-right w-4 h-4"><path d="M7 7h10v10"></path><path d="M7 17 17 7"></path></svg></div><div class="h-6 w-6 fill-gray-800 dark:fill-gray-100 text-gray-800 dark:text-gray-100" data-component-part="card-icon"><svg class="h-6 w-6 bg-primary dark:bg-primary-light !m-0 shrink-0" style="-webkit-mask-image:url(https://mintlify.b-cdn.net/v6.6.0/regular/book.svg);-webkit-mask-repeat:no-repeat;-webkit-mask-position:center;mask-image:url(https://mintlify.b-cdn.net/v6.6.0/regular/book.svg);mask-repeat:no-repeat;mask-position:center"></svg></div><div><h2 class="not-prose font-semibold text-base text-gray-800 dark:text-white mt-4" contentEditable="false" data-component-part="card-title">Core Concepts</h2><div class="prose mt-1 font-normal text-sm leading-6 text-gray-600 dark:text-gray-400" data-component-part="card-content"><p>Learn about pipelines, frames, and real-time processing</p></div></div></div></a><a class="card block font-normal group relative my-2 ring-2 ring-transparent rounded-2xl bg-white dark:bg-background-dark border border-gray-950/10 dark:border-white/10 overflow-hidden w-full cursor-pointer hover:!border-primary dark:hover:!border-primary-light" href="/getting-started/use-cases"><div class="px-6 py-5 relative" data-component-part="card-content-container"><div id="card-link-arrow-icon" class="absolute text-gray-400 dark:text-gray-500 group-hover:text-primary dark:group-hover:text-primary-light top-5 right-5 hidden"><svg xmlns="http://www.w3.org/2000/svg" width="24" height="24" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-arrow-up-right w-4 h-4"><path d="M7 7h10v10"></path><path d="M7 17 17 7"></path></svg></div><div class="h-6 w-6 fill-gray-800 dark:fill-gray-100 text-gray-800 dark:text-gray-100" data-component-part="card-icon"><svg class="h-6 w-6 bg-primary dark:bg-primary-light !m-0 shrink-0" style="-webkit-mask-image:url(https://mintlify.b-cdn.net/v6.6.0/regular/lightbulb.svg);-webkit-mask-repeat:no-repeat;-webkit-mask-position:center;mask-image:url(https://mintlify.b-cdn.net/v6.6.0/regular/lightbulb.svg);mask-repeat:no-repeat;mask-position:center"></svg></div><div><h2 class="not-prose font-semibold text-base text-gray-800 dark:text-white mt-4" contentEditable="false" data-component-part="card-title">Use Cases</h2><div class="prose mt-1 font-normal text-sm leading-6 text-gray-600 dark:text-gray-400" data-component-part="card-content"><p>Explore example implementations and patterns</p></div></div></div></a></div>
+<h2 class="flex whitespace-pre-wrap group font-semibold" id="join-our-community"><div class="absolute"><a href="#join-our-community" class="-ml-10 flex items-center opacity-0 border-0 group-hover:opacity-100" aria-label="Navigate to header">​<div class="w-6 h-6 rounded-md flex items-center justify-center shadow-sm text-gray-400 dark:text-white/50 dark:bg-background-dark dark:brightness-[1.35] dark:ring-1 dark:hover:brightness-150 bg-white ring-1 ring-gray-400/30 dark:ring-gray-700/25 hover:ring-gray-400/60 dark:hover:ring-white/20"><svg xmlns="http://www.w3.org/2000/svg" fill="gray" height="12px" viewBox="0 0 576 512"><path d="M0 256C0 167.6 71.6 96 160 96h72c13.3 0 24 10.7 24 24s-10.7 24-24 24H160C98.1 144 48 194.1 48 256s50.1 112 112 112h72c13.3 0 24 10.7 24 24s-10.7 24-24 24H160C71.6 416 0 344.4 0 256zm576 0c0 88.4-71.6 160-160 160H344c-13.3 0-24-10.7-24-24s10.7-24 24-24h72c61.9 0 112-50.1 112-112s-50.1-112-112-112H344c-13.3 0-24-10.7-24-24s10.7-24 24-24h72c88.4 0 160 71.6 160 160zM184 232H392c13.3 0 24 10.7 24 24s-10.7 24-24 24H184c-13.3 0-24-10.7-24-24s10.7-24 24-24z"></path></svg></div></a></div><span class="cursor-pointer">Join Our Community</span></h2>
+<a class="card block font-normal group relative my-2 ring-2 ring-transparent rounded-2xl bg-white dark:bg-background-dark border border-gray-950/10 dark:border-white/10 overflow-hidden w-full cursor-pointer hover:!border-primary dark:hover:!border-primary-light" target="_blank" rel="noreferrer" href="https://discord.gg/pipecat"><div class="px-6 py-5 relative" data-component-part="card-content-container"><div id="card-link-arrow-icon" class="absolute text-gray-400 dark:text-gray-500 group-hover:text-primary dark:group-hover:text-primary-light top-5 right-5"><svg xmlns="http://www.w3.org/2000/svg" width="24" height="24" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-arrow-up-right w-4 h-4"><path d="M7 7h10v10"></path><path d="M7 17 17 7"></path></svg></div><div class="h-6 w-6 fill-gray-800 dark:fill-gray-100 text-gray-800 dark:text-gray-100" data-component-part="card-icon"><svg class="h-6 w-6 bg-primary dark:bg-primary-light !m-0 shrink-0" style="-webkit-mask-image:url(https://mintlify.b-cdn.net/v6.6.0/brands/discord.svg);-webkit-mask-repeat:no-repeat;-webkit-mask-position:center;mask-image:url(https://mintlify.b-cdn.net/v6.6.0/brands/discord.svg);mask-repeat:no-repeat;mask-position:center"></svg></div><div><h2 class="not-prose font-semibold text-base text-gray-800 dark:text-white mt-4" contentEditable="false" data-component-part="card-title">Discord Community</h2><div class="prose mt-1 font-normal text-sm leading-6 text-gray-600 dark:text-gray-400" data-component-part="card-content"><p>Connect with other developers, share your projects, and get support from the
+Pipecat team.</p></div></div></div></a></div><div class="leading-6 mt-14"><div id="pagination" class="mb-12 px-0.5 flex items-center text-sm font-semibold text-gray-700 dark:text-gray-200"><a class="flex items-center ml-auto space-x-3 group" href="/getting-started/installation"><span class="group-hover:text-gray-900 dark:group-hover:text-white">Installation &amp; Setup</span><svg viewBox="0 0 3 6" class="rotate-180 h-1.5 stroke-gray-400 overflow-visible group-hover:stroke-gray-600 dark:group-hover:stroke-gray-300"><path d="M3 0L0 3L3 6" fill="none" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"></path></svg></a></div><footer id="footer" class="flex gap-12 justify-between pt-10 border-t border-gray-100 sm:flex dark:border-gray-800/50 pb-28"><div class="flex gap-6 flex-wrap"><a href="https://x.com/pipecat_ai" target="_blank"><span class="sr-only">x</span><svg class="w-5 h-5 bg-gray-400 dark:bg-gray-500 hover:bg-gray-500 dark:hover:bg-gray-400" style="-webkit-mask-image:url(https://mintlify.b-cdn.net/v6.6.0/brands/x-twitter.svg);-webkit-mask-repeat:no-repeat;-webkit-mask-position:center;mask-image:url(https://mintlify.b-cdn.net/v6.6.0/brands/x-twitter.svg);mask-repeat:no-repeat;mask-position:center"></svg></a><a href="https://github.com/pipecat-ai/pipecat" target="_blank"><span class="sr-only">github</span><svg class="w-5 h-5 bg-gray-400 dark:bg-gray-500 hover:bg-gray-500 dark:hover:bg-gray-400" style="-webkit-mask-image:url(https://mintlify.b-cdn.net/v6.6.0/brands/github.svg);-webkit-mask-repeat:no-repeat;-webkit-mask-position:center;mask-image:url(https://mintlify.b-cdn.net/v6.6.0/brands/github.svg);mask-repeat:no-repeat;mask-position:center"></svg></a><a href="https://discord.gg/pipecat" target="_blank"><span class="sr-only">discord</span><svg class="w-5 h-5 bg-gray-400 dark:bg-gray-500 hover:bg-gray-500 dark:hover:bg-gray-400" style="-webkit-mask-image:url(https://mintlify.b-cdn.net/v6.6.0/brands/discord.svg);-webkit-mask-repeat:no-repeat;-webkit-mask-position:center;mask-image:url(https://mintlify.b-cdn.net/v6.6.0/brands/discord.svg);mask-repeat:no-repeat;mask-position:center"></svg></a></div><div class="flex items-center justify-between"><div class="sm:flex"><a href="https://mintlify.com/preview-request?utm_campaign=poweredBy&amp;utm_medium=referral&amp;utm_source=docs.pipecat.ai" target="_blank" rel="noreferrer" class="text-sm text-gray-500 dark:text-gray-400 hover:text-gray-700 dark:hover:text-gray-300 text-nowrap">Powered by Mintlify</a></div></div></footer></div></div><div class="hidden xl:flex self-start sticky xl:flex-col max-w-[28rem] h-[calc(100vh-9.5rem)] top-[9.5rem]" id="content-side-layout"><div class="z-10 hidden xl:flex pl-10 box-border w-[19rem] max-h-full" id="table-of-contents-layout"><div class="text-gray-600 text-sm leading-6 w-[16.5rem] overflow-y-auto space-y-2 pb-4 -mt-10 pt-10" id="table-of-contents"><div class="text-gray-700 dark:text-gray-300 font-medium flex items-center space-x-2"><svg width="16" height="16" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="2" xmlns="http://www.w3.org/2000/svg" class="h-3 w-3"><path d="M2.44434 12.6665H13.5554" stroke-linecap="round" stroke-linejoin="round"></path><path d="M2.44434 3.3335H13.5554" stroke-linecap="round" stroke-linejoin="round"></path><path d="M2.44434 8H7.33323" stroke-linecap="round" stroke-linejoin="round"></path></svg><span>On this page</span></div><ul id="table-of-contents-content" class="toc"><li class="toc-item relative" data-depth="0"><a href="#what-you-can-build" class="py-1 block font-medium hover:text-gray-900 dark:text-gray-400 dark:hover:text-gray-300">What You Can Build</a></li><li class="toc-item relative" data-depth="0"><a href="#how-it-works" class="py-1 block font-medium hover:text-gray-900 dark:text-gray-400 dark:hover:text-gray-300">How It Works</a></li><li class="toc-item relative" data-depth="1"><a style="margin-left:1rem" href="#real-time-processing" class="group flex items-start py-1 whitespace-pre-wrap text-gray-500 hover:text-gray-900 dark:text-gray-400 dark:hover:text-gray-300">Real-time Processing</a></li><li class="toc-item relative" data-depth="0"><a href="#next-steps" class="py-1 block font-medium hover:text-gray-900 dark:text-gray-400 dark:hover:text-gray-300">Next Steps</a></li><li class="toc-item relative" data-depth="0"><a href="#join-our-community" class="py-1 block font-medium hover:text-gray-900 dark:text-gray-400 dark:hover:text-gray-300">Join Our Community</a></li></ul></div></div></div></div></div></div><div class="z-10 fixed right-0 w-[23rem] border-l border-gray-500/5 dark:border-gray-300/[0.06] bg-background-light dark:bg-background-dark h-[calc(100vh-7rem)] top-[7rem] invisible" style="width:368px;min-width:368px;max-width:576px"><div class="absolute -left-1 top-0 bottom-0 w-1 cursor-col-resize hover:bg-gray-200/70 dark:hover:bg-white/[0.07] z-10" style="cursor:col-resize"></div><div id="chat-assistant-sheet" class="absolute inset-0 -top-px min-h-full flex flex-col overflow-hidden shrink-0 chat-assistant-sheet" aria-hidden="true"><div class="w-full flex flex-col pb-4 h-full lg:pt-3"><div class="chat-assistant-sheet-header flex items-center justify-between pb-3 px-4"><div class="flex items-center gap-2"><svg xmlns="http://www.w3.org/2000/svg" width="18" height="18" viewBox="0 0 18 18" class="size-5 text-primary dark:text-primary-light"><g fill="currentColor"><path d="M5.658,2.99l-1.263-.421-.421-1.263c-.137-.408-.812-.408-.949,0l-.421,1.263-1.263,.421c-.204,.068-.342,.259-.342,.474s.138,.406,.342,.474l1.263,.421,.421,1.263c.068,.204,.26,.342,.475,.342s.406-.138,.475-.342l.421-1.263,1.263-.421c.204-.068,.342-.259,.342-.474s-.138-.406-.342-.474Z" fill="currentColor" data-stroke="none" stroke="none"></path><polygon points="9.5 2.75 11.412 7.587 16.25 9.5 11.412 11.413 9.5 16.25 7.587 11.413 2.75 9.5 7.587 7.587 9.5 2.75" fill="none" stroke="currentColor" stroke-linecap="round" stroke-linejoin="round" stroke-width="1.5"></polygon></g></svg><span class="font-semibold text-gray-900 dark:text-gray-100">Assistant</span></div><div class="flex items-center gap-1"><button class="group hover:bg-gray-100 dark:hover:bg-white/10 p-1.5 rounded-lg"><svg width="16" height="16" viewBox="0 0 16 16" fill="none" xmlns="http://www.w3.org/2000/svg" class="size-3.5 text-gray-500 group-hover:text-gray-700 dark:group-hover:text-gray-300"><path d="M12.4444 3.55566L3.55554 12.4446" stroke="currentColor" stroke-width="1.5" stroke-linecap="round" stroke-linejoin="round"></path><path d="M3.55554 3.55566L12.4444 12.4446" stroke="currentColor" stroke-width="1.5" stroke-linecap="round" stroke-linejoin="round"></path></svg></button></div></div><div class="chat-assistant-sheet-content flex-1 overflow-y-auto relative px-4"><div class="mt-4 flex flex-col items-center text-sm justify-between"><div class="mx-8 text-center text-gray-400 dark:text-gray-600 text-xs chat-assistant-disclaimer-text">Responses are generated using AI and may contain mistakes.</div><div class="flex flex-col gap-4 text-gray-800 dark:text-gray-200"></div></div></div><div class="px-4"><div class=""><div class="flex items-end gap-2 relative"><textarea id="chat-assistant-textarea" autoComplete="off" placeholder="Ask a question..." class="grow w-full px-3.5 pr-10 py-2.5 bg-background-light dark:bg-background-dark border border-gray-200 dark:border-gray-600/30 rounded-2xl shadow-sm focus:outline-none focus:border-primary dark:focus:border-primary-light text-gray-900 dark:text-gray-100 text-sm chat-assistant-input" style="resize:none"></textarea><button class="absolute right-2.5 bottom-[9px] flex justify-center items-center p-1 w-6 h-6 rounded-lg bg-primary/30 dark:bg-primary-dark/30 chat-assistant-send-button" aria-label="Send message" disabled=""><svg xmlns="http://www.w3.org/2000/svg" width="24" height="24" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-arrow-up w-3.5 h-3.5 text-white dark:text-white"><path d="m5 12 7-7 7 7"></path><path d="M12 19V5"></path></svg></button></div></div></div></div></div></div></div></main></div><script id="__NEXT_DATA__" type="application/json">{"props":{"pageProps":{"mdxSource":{"compiledSource":"\"use strict\";\nconst {Fragment: _Fragment, jsx: _jsx, jsxs: _jsxs} = arguments[0];\nconst {useMDXComponents: _provideComponents} = arguments[0];\nfunction _createMdxContent(props) {\n  const _components = {\n    li: \"li\",\n    ol: \"ol\",\n    p: \"p\",\n    ul: \"ul\",\n    ..._provideComponents(),\n    ...props.components\n  }, {Card, CardGroup, Heading, Note, Step, Steps, Tab, Tabs} = _components;\n  if (!Card) _missingMdxReference(\"Card\", true);\n  if (!CardGroup) _missingMdxReference(\"CardGroup\", true);\n  if (!Heading) _missingMdxReference(\"Heading\", true);\n  if (!Note) _missingMdxReference(\"Note\", true);\n  if (!Step) _missingMdxReference(\"Step\", true);\n  if (!Steps) _missingMdxReference(\"Steps\", true);\n  if (!Tab) _missingMdxReference(\"Tab\", true);\n  if (!Tabs) _missingMdxReference(\"Tabs\", true);\n  return _jsxs(_Fragment, {\n    children: [_jsx(_components.p, {\n      children: \"Pipecat is an open source Python framework that handles the complex orchestration of AI services, network transport, audio processing, and multimodal interactions. “Multimodal” means you can use any combination of audio, video, images, and/or text in your interactions. And “real-time” means that things are happening quickly enough that it feels conversational—a “back-and-forth” with a bot, not submitting a query and waiting for results.\"\n    }), \"\\n\", _jsx(Heading, {\n      level: \"2\",\n      id: \"what-you-can-build\",\n      isAtRootLevel: \"true\",\n      children: \"What You Can Build\"\n    }), \"\\n\", _jsxs(CardGroup, {\n      cols: 3,\n      children: [_jsx(Card, {\n        title: \"Voice Assistants\",\n        icon: \"microphone\",\n        href: \"https://github.com/pipecat-ai/pipecat/tree/main/examples/simple-chatbot\",\n        children: _jsx(_components.p, {\n          children: \"Natural, real-time conversations with AI using speech recognition and\\nsynthesis\"\n        })\n      }), _jsx(Card, {\n        title: \"Interactive Agents\",\n        icon: \"robot\",\n        href: \"https://github.com/pipecat-ai/pipecat/tree/main/examples/studypal\",\n        children: _jsx(_components.p, {\n          children: \"Personal coaches and meeting assistants that can understand context and\\nprovide guidance\"\n        })\n      }), _jsx(Card, {\n        title: \"Multimodal Apps\",\n        icon: \"layer-group\",\n        href: \"https://github.com/pipecat-ai/pipecat/tree/main/examples/moondream-chatbot\",\n        children: _jsx(_components.p, {\n          children: \"Applications that combine voice, video, images, and text for rich interactions\"\n        })\n      }), _jsx(Card, {\n        title: \"Creative Tools\",\n        icon: \"wand-magic-sparkles\",\n        href: \"https://github.com/pipecat-ai/pipecat/tree/main/examples/storytelling-chatbot\",\n        children: _jsx(_components.p, {\n          children: \"Storytelling experiences and social companions that engage users\"\n        })\n      }), _jsx(Card, {\n        title: \"Business Solutions\",\n        icon: \"building\",\n        href: \"https://github.com/pipecat-ai/pipecat/tree/main/examples/patient-intake\",\n        children: _jsx(_components.p, {\n          children: \"Customer intake flows and support bots for automated business processes\"\n        })\n      }), _jsx(Card, {\n        title: \"Complex Flows\",\n        icon: \"diagram-project\",\n        href: \"https://github.com/pipecat-ai/pipecat-flows\",\n        children: _jsx(_components.p, {\n          children: \"Structured conversations using Pipecat Flows for managing complex\\ninteractions\"\n        })\n      })]\n    }), \"\\n\", _jsx(Heading, {\n      level: \"2\",\n      id: \"how-it-works\",\n      isAtRootLevel: \"true\",\n      children: \"How It Works\"\n    }), \"\\n\", _jsx(_components.p, {\n      children: \"The flow of interactions in a Pipecat application is typically straightforward:\"\n    }), \"\\n\", _jsxs(_components.ol, {\n      children: [\"\\n\", _jsx(_components.li, {\n        children: \"The bot says something\"\n      }), \"\\n\", _jsx(_components.li, {\n        children: \"The user says something\"\n      }), \"\\n\", _jsx(_components.li, {\n        children: \"The bot says something\"\n      }), \"\\n\", _jsx(_components.li, {\n        children: \"The user says something\"\n      }), \"\\n\"]\n    }), \"\\n\", _jsx(_components.p, {\n      children: \"This continues until the conversation naturally ends. While this flow seems simple, making it feel natural requires sophisticated real-time processing.\"\n    }), \"\\n\", _jsx(Heading, {\n      level: \"3\",\n      id: \"real-time-processing\",\n      isAtRootLevel: \"true\",\n      children: \"Real-time Processing\"\n    }), \"\\n\", _jsx(_components.p, {\n      children: \"Pipecat’s pipeline architecture handles both simple voice interactions and complex multimodal processing. Let’s look at how data flows through the system:\"\n    }), \"\\n\", _jsxs(Tabs, {\n      children: [_jsx(Tab, {\n        title: \"Voice app\",\n        children: _jsxs(Steps, {\n          children: [_jsx(Step, {\n            title: \"Send Audio\",\n            children: _jsx(_components.p, {\n              children: \"Transmit and capture streamed audio from the user\"\n            })\n          }), _jsx(Step, {\n            title: \"Transcribe Speech\",\n            children: _jsx(_components.p, {\n              children: \"Convert speech to text as the user is talking\"\n            })\n          }), _jsx(Step, {\n            title: \"Process with LLM\",\n            children: _jsx(_components.p, {\n              children: \"Generate responses using a large language model\"\n            })\n          }), _jsx(Step, {\n            title: \"Convert to Speech\",\n            children: _jsx(_components.p, {\n              children: \"Transform text responses into natural speech\"\n            })\n          }), _jsx(Step, {\n            title: \"Play Audio\",\n            children: \"Stream the audio response back to the user\"\n          })]\n        })\n      }), _jsx(Tab, {\n        title: \"Multimodal app\",\n        children: _jsxs(Steps, {\n          children: [_jsx(Step, {\n            title: \"Send Audio and Video\",\n            children: _jsx(_components.p, {\n              children: \"Transmit and capture audio, video, and image inputs simultaneously\"\n            })\n          }), _jsx(Step, {\n            title: \"Process Streams\",\n            children: \"Handle multiple input streams in parallel\"\n          }), _jsx(Step, {\n            title: \"Model Processing\",\n            children: _jsx(_components.p, {\n              children: \"Send combined inputs to multimodal models (like GPT-4V)\"\n            })\n          }), _jsx(Step, {\n            title: \"Generate Outputs\",\n            children: _jsx(_components.p, {\n              children: \"Create various outputs (text, images, audio, etc.)\"\n            })\n          }), _jsx(Step, {\n            title: \"Coordinate Presentation\",\n            children: _jsx(_components.p, {\n              children: \"Synchronize and present multiple output types\"\n            })\n          })]\n        })\n      })]\n    }), \"\\n\", _jsx(_components.p, {\n      children: \"In both cases, Pipecat:\"\n    }), \"\\n\", _jsxs(_components.ul, {\n      children: [\"\\n\", _jsx(_components.li, {\n        children: \"Processes responses as they stream in\"\n      }), \"\\n\", _jsx(_components.li, {\n        children: \"Handles multiple input/output modalities concurrently\"\n      }), \"\\n\", _jsx(_components.li, {\n        children: \"Manages resource allocation and synchronization\"\n      }), \"\\n\", _jsx(_components.li, {\n        children: \"Coordinates parallel processing tasks\"\n      }), \"\\n\"]\n    }), \"\\n\", _jsx(_components.p, {\n      children: \"This architecture creates fluid, natural interactions without noticeable delays, whether you’re building a simple voice assistant or a complex multimodal application. Pipecat’s pipeline architecture is particularly valuable for managing the complexity of real-time, multimodal interactions, ensuring smooth data flow and proper synchronization regardless of the input/output types involved.\"\n    }), \"\\n\", _jsx(Note, {\n      children: _jsx(_components.p, {\n        children: \"Pipecat handles all this complexity for you, letting you focus on building\\nyour application rather than managing the underlying infrastructure.\"\n      })\n    }), \"\\n\", _jsx(Heading, {\n      level: \"2\",\n      id: \"next-steps\",\n      isAtRootLevel: \"true\",\n      children: \"Next Steps\"\n    }), \"\\n\", _jsx(_components.p, {\n      children: \"Ready to build your first Pipecat application?\"\n    }), \"\\n\", _jsxs(CardGroup, {\n      cols: 2,\n      children: [_jsx(Card, {\n        title: \"Installation \u0026 Setup\",\n        icon: \"wrench\",\n        href: \"/getting-started/installation\",\n        children: _jsx(_components.p, {\n          children: \"Prepare your environment and install required dependencies\"\n        })\n      }), _jsx(Card, {\n        title: \"Quickstart\",\n        icon: \"rocket\",\n        href: \"/getting-started/quickstart\",\n        children: _jsx(_components.p, {\n          children: \"Build and run your first Pipecat application\"\n        })\n      }), _jsx(Card, {\n        title: \"Core Concepts\",\n        icon: \"book\",\n        href: \"/getting-started/core-concepts\",\n        children: _jsx(_components.p, {\n          children: \"Learn about pipelines, frames, and real-time processing\"\n        })\n      }), _jsx(Card, {\n        title: \"Use Cases\",\n        icon: \"lightbulb\",\n        href: \"/getting-started/use-cases\",\n        children: _jsx(_components.p, {\n          children: \"Explore example implementations and patterns\"\n        })\n      })]\n    }), \"\\n\", _jsx(Heading, {\n      level: \"2\",\n      id: \"join-our-community\",\n      isAtRootLevel: \"true\",\n      children: \"Join Our Community\"\n    }), \"\\n\", _jsx(Card, {\n      title: \"Discord Community\",\n      icon: \"discord\",\n      iconType: \"duotone\",\n      href: \"https://discord.gg/pipecat\",\n      children: _jsx(_components.p, {\n        children: \"Connect with other developers, share your projects, and get support from the\\nPipecat team.\"\n      })\n    })]\n  });\n}\nfunction MDXContent(props = {}) {\n  const {wrapper: MDXLayout} = {\n    ..._provideComponents(),\n    ...props.components\n  };\n  return MDXLayout ? _jsx(MDXLayout, {\n    ...props,\n    children: _jsx(_createMdxContent, {\n      ...props\n    })\n  }) : _createMdxContent(props);\n}\nreturn {\n  default: MDXContent\n};\nfunction _missingMdxReference(id, component) {\n  throw new Error(\"Expected \" + (component ? \"component\" : \"object\") + \" `\" + id + \"` to be defined: you likely forgot to import, pass, or provide it.\");\n}\n","frontmatter":{},"scope":{"config":{"$schema":"https://mintlify.com/docs.json","theme":"mint","name":"Pipecat","colors":{"primary":"#4f46e5","light":"#a5b4fc","dark":"#312e81"},"favicon":"/favicon.svg","navigation":{"tabs":[{"tab":"Getting Started","groups":[{"group":"Get Started","pages":["getting-started/overview","getting-started/installation","getting-started/quickstart","getting-started/core-concepts","getting-started/next-steps"]}]},{"tab":"Guides","groups":[{"group":"","pages":["guides/introduction"]},{"group":"Fundamentals","pages":["guides/fundamentals/context-management","guides/fundamentals/custom-frame-processor","guides/fundamentals/detecting-user-idle","guides/fundamentals/end-pipeline","guides/fundamentals/function-calling","guides/fundamentals/user-input-muting","guides/fundamentals/recording-audio","guides/fundamentals/recording-transcripts"]},{"group":"Features","pages":["guides/features/gemini-multimodal-live","guides/features/metrics","guides/features/krisp","guides/features/openai-audio-models-and-apis","guides/features/pipecat-flows"]},{"group":"Telephony","pages":["guides/telephony/overview","guides/telephony/daily-webrtc","guides/telephony/twilio-daily-webrtc","guides/telephony/twilio-websockets","guides/telephony/dialout"]},{"group":"Deploying your bot","pages":["guides/deployment/overview","guides/deployment/pattern","guides/deployment/pipecat-cloud","guides/deployment/fly","guides/deployment/cerebrium","guides/deployment/modal"]}]},{"tab":"Server APIs","groups":[{"group":"","pages":["server/introduction"]},{"group":"API Reference","pages":["server/links/server-reference"]},{"group":"Services","pages":["server/services/supported-services",{"group":"Transport","pages":["server/services/transport/daily","server/services/transport/small-webrtc","server/services/transport/fastapi-websocket","server/services/transport/websocket-server"]},{"group":"Serializers","pages":["server/services/serializers/introduction","server/services/serializers/exotel","server/services/serializers/plivo","server/services/serializers/twilio","server/services/serializers/telnyx"]},{"group":"Speech-to-Text","pages":["server/services/stt/assemblyai","server/services/stt/aws","server/services/stt/azure","server/services/stt/cartesia","server/services/stt/deepgram","server/services/stt/fal","server/services/stt/gladia","server/services/stt/google","server/services/stt/groq","server/services/stt/riva","server/services/stt/openai","server/services/stt/sambanova","server/services/stt/speechmatics","server/services/stt/ultravox","server/services/stt/whisper"]},{"group":"LLM","pages":["server/services/llm/anthropic","server/services/llm/aws","server/services/llm/azure","server/services/llm/cerebras","server/services/llm/deepseek","server/services/llm/fireworks","server/services/llm/gemini","server/services/llm/google-vertex","server/services/llm/grok","server/services/llm/groq","server/services/llm/nim","server/services/llm/ollama","server/services/llm/openai","server/services/llm/openpipe","server/services/llm/openrouter","server/services/llm/perplexity","server/services/llm/qwen","server/services/llm/sambanova","server/services/llm/together"]},{"group":"Text-to-Speech","pages":["server/services/tts/aws","server/services/tts/azure","server/services/tts/cartesia","server/services/tts/deepgram","server/services/tts/elevenlabs","server/services/tts/fish","server/services/tts/google","server/services/tts/groq","server/services/tts/lmnt","server/services/tts/minimax","server/services/tts/neuphonic","server/services/tts/riva","server/services/tts/openai","server/services/tts/piper","server/services/tts/playht","server/services/tts/rime","server/services/tts/sarvam","server/services/tts/xtts"]},{"group":"Speech-to-Speech","pages":["server/services/s2s/aws","server/services/s2s/gemini","server/services/s2s/openai"]},{"group":"Image Generation","pages":["server/services/image-generation/fal","server/services/image-generation/google-imagen","server/services/image-generation/openai"]},{"group":"Video","pages":["server/services/video/simli","server/services/video/tavus"]},{"group":"Memory","pages":["server/services/memory/mem0"]},{"group":"Vision","pages":["server/services/vision/moondream"]},{"group":"Analytics \u0026 Monitoring","pages":["server/services/analytics/sentry"]}]},{"group":"Utilities","pages":[{"group":"Advanced Frame Processors","pages":["server/utilities/frame/producer-consumer","server/utilities/user-idle-processor"]},{"group":"Audio Processing","pages":["server/utilities/audio/audio-buffer-processor","server/utilities/audio/koala-filter","server/utilities/audio/krisp-filter","server/utilities/audio/noisereduce-filter","server/utilities/audio/silero-vad-analyzer","server/utilities/audio/soundfile-mixer"]},{"group":"Frame Filters","pages":["server/utilities/filters/frame-filter","server/utilities/filters/function-filter","server/utilities/filters/identify-filter","server/utilities/filters/null-filter","server/utilities/filters/stt-mute","server/utilities/filters/wake-check-filter","server/utilities/filters/wake-notifier-filter"]},{"group":"Metrics and Telemetry","pages":["server/utilities/opentelemetry"]},{"group":"MCP","pages":["server/utilities/mcp/mcp"]},{"group":"Observers","pages":["server/utilities/observers/observer-pattern","server/utilities/observers/debug-observer","server/utilities/observers/llm-observer","server/utilities/observers/transcription-observer","server/utilities/observers/turn-tracking-observer"]},{"group":"Service Utilities","pages":["server/utilities/daily/rest-helpers"]},{"group":"Smart Turn Detection","pages":["server/utilities/smart-turn/smart-turn-overview","server/utilities/smart-turn/fal-smart-turn","server/utilities/smart-turn/local-coreml-smart-turn"]},{"group":"Task Handling and Monitoring","pages":["server/utilities/watchdog-timers"]},{"group":"Telephony","pages":["server/utilities/dtmf-aggregator"]},{"group":"Text Aggregators and Filters","pages":["server/utilities/text/markdown-text-filter","server/utilities/text/pattern-pair-aggregator"]},{"group":"User and Bot Transcriptions","pages":["server/utilities/transcript-processor"]},{"group":"User Interruptions","pages":["server/utilities/interruption-strategies"]}]},{"group":"Frameworks","pages":[{"group":"RTVI","pages":["server/frameworks/rtvi/introduction","server/frameworks/rtvi/rtvi-processor","server/frameworks/rtvi/rtvi-observer"]},"server/frameworks/flows/pipecat-flows"]},{"group":"Pipeline","pages":["server/pipeline/pipeline-params","server/pipeline/pipeline-task","server/pipeline/pipeline-idle-detection","server/pipeline/heartbeats","server/pipeline/parallel-pipeline"]}]},{"tab":"Client SDKs","groups":[{"group":"","pages":["client/introduction","client/rtvi-standard","client/migration-guide"]},{"group":"Javascript SDK","pages":["client/js/introduction",{"group":"API Reference","pages":["client/js/api-reference/client-constructor","client/js/api-reference/client-methods","client/js/api-reference/callbacks","client/js/api-reference/messages","client/js/api-reference/errors"]},{"group":"Transport packages","pages":["client/js/transports/transport","client/js/transports/daily","client/js/transports/small-webrtc","client/js/transports/websocket","client/js/transports/gemini","client/js/transports/openai-webrtc"]}]},{"group":"React SDK","pages":["client/react/introduction",{"group":"API Reference","pages":["client/react/components","client/react/hooks"]}]},{"group":"React Native SDK","pages":["client/react-native/introduction","client/react-native/api-reference"]},{"group":"iOS SDK","pages":["client/ios/introduction","client/ios/api-reference",{"group":"Transport packages","pages":["client/ios/transports/daily","client/ios/transports/gemini-websocket","client/ios/transports/openai-webrtc"]}]},{"group":"Android SDK","pages":["client/android/introduction","client/android/api-reference",{"group":"Transport packages","pages":["client/android/transports/daily","client/android/transports/gemini-websocket","client/android/transports/openai-webrtc","client/android/transports/small-webrtc"]}]},{"group":"C++ SDK","pages":["client/c++/introduction","client/c++/transport"]}]}],"global":{"anchors":[{"anchor":"Community","href":"https://discord.gg/pipecat","icon":"discord"},{"anchor":"GitHub","href":"https://github.com/pipecat-ai/pipecat","icon":"github"},{"anchor":"Examples","href":"https://github.com/pipecat-ai/pipecat/tree/main/examples","icon":"code"},{"anchor":"Changelog","href":"https://github.com/pipecat-ai/pipecat/blob/main/CHANGELOG.md","icon":"file"}]}},"logo":{"light":"https://mintlify.s3.us-west-1.amazonaws.com/daily/logo/light.svg","dark":"https://mintlify.s3.us-west-1.amazonaws.com/daily/logo/dark.svg"},"navbar":{"links":[{"label":"Join the Discord","href":"https://discord.gg/pipecat"}]},"seo":{"metatags":{"og:site_name":"Pipecat","og:description":"Learn about Pipecat, the open source framework for building voice and multimodal AI agents.","og:url":"https://docs.pipecat.ai/","og:locale":"en_US","og:logo":"/logo/pipecat-logo.svg","article:publisher":"Pipecat AI","twitter:description":"Learn about Pipecat, the open source framework for building voice and multimodal AI agents.","twitter:url":"https://docs.pipecat.ai/","twitter:site":"@pipecat_ai","og:image:width":"1200","og:image:height":"675"},"indexing":"navigable"},"footer":{"socials":{"x":"https://x.com/pipecat_ai","github":"https://github.com/pipecat-ai/pipecat","discord":"https://discord.gg/pipecat"}}},"pageMetadata":{"title":"Overview","description":"Pipecat is a framework for building voice-enabled, real-time, multimodal AI applications","href":"/getting-started/overview"}}},"mdxExtracts":{"tableOfContents":[{"title":"What You Can Build","slug":"what-you-can-build","depth":2,"children":[]},{"title":"How It Works","slug":"how-it-works","depth":2,"children":[{"title":"Real-time Processing","slug":"real-time-processing","depth":3,"children":[]}]},{"title":"Next Steps","slug":"next-steps","depth":2,"children":[]},{"title":"Join Our Community","slug":"join-our-community","depth":2,"children":[]}],"codeExamples":{}},"description":{"compiledSource":"\"use strict\";\nconst {jsx: _jsx} = arguments[0];\nconst {useMDXComponents: _provideComponents} = arguments[0];\nfunction _createMdxContent(props) {\n  const _components = {\n    p: \"p\",\n    ..._provideComponents(),\n    ...props.components\n  };\n  return _jsx(_components.p, {\n    children: \"Pipecat is a framework for building voice-enabled, real-time, multimodal AI applications\"\n  });\n}\nfunction MDXContent(props = {}) {\n  const {wrapper: MDXLayout} = {\n    ..._provideComponents(),\n    ...props.components\n  };\n  return MDXLayout ? _jsx(MDXLayout, {\n    ...props,\n    children: _jsx(_createMdxContent, {\n      ...props\n    })\n  }) : _createMdxContent(props);\n}\nreturn {\n  default: MDXContent\n};\n","frontmatter":{},"scope":{}},"pageData":{"navWithMetadata":[{"group":"Get Started","pages":[{"title":"Overview","description":"Pipecat is a framework for building voice-enabled, real-time, multimodal AI applications","href":"/getting-started/overview"},{"title":"Installation \u0026 Setup","description":"Get Pipecat and its required services installed on your machine","href":"/getting-started/installation"},{"title":"Quickstart","description":"Build and run your first Pipecat application","href":"/getting-started/quickstart"},{"title":"Core Concepts","description":"Understanding how Pipecat works: frames, processors, and pipelines.","href":"/getting-started/core-concepts"},{"title":"Next Steps \u0026 Examples","description":"Where to go next with Pipecat based on your project needs","href":"/getting-started/next-steps"}]},{"group":"","pages":[{"title":"Guides","description":"Learn how to deploy, scale, and extend your Pipecat applications","href":"/guides/introduction"}]},{"group":"Fundamentals","pages":[{"title":"Context Management","description":"A guide to working with Pipecat's Context and Context Aggregators","href":"/guides/fundamentals/context-management"},{"title":"Custom FrameProcessor","description":"Learn how to write your own custom FrameProcessor","href":"/guides/fundamentals/custom-frame-processor"},{"title":"Detecting Idle Users","description":"Learn how to detect and respond when users are inactive in conversations","href":"/guides/fundamentals/detecting-user-idle"},{"title":"Ending a Pipeline","description":"Best practices for properly terminating Pipecat pipelines","href":"/guides/fundamentals/end-pipeline"},{"title":"Function Calling","description":"Enable LLMs to interact with external services and APIs","href":"/guides/fundamentals/function-calling"},{"title":"User Input Muting with STTMuteFilter","sidebarTitle":"Muting User Input","description":"Learn how to control when user speech is processed in your conversational bot","href":"/guides/fundamentals/user-input-muting"},{"title":"Recording Conversation Audio","sidebarTitle":"Recording Audio","description":"Learn how to record and save audio from conversations between users and your bot","href":"/guides/fundamentals/recording-audio"},{"title":"Recording Conversation Transcripts","sidebarTitle":"Recording Transcripts","description":"Learn how to collect and save conversation transcripts between users and your bot","href":"/guides/fundamentals/recording-transcripts"}]},{"group":"Features","pages":[{"title":"Building with Gemini Multimodal Live","sidebarTitle":"Gemini Multimodal Live","description":"Create a real-time AI chatbot using Gemini Multimodal Live and Pipecat","href":"/guides/features/gemini-multimodal-live"},{"title":"Metrics","description":"Learn how to monitor performance and LLM/TTS usage with Pipecat.","href":"/guides/features/metrics"},{"title":"Noise cancellation with Krisp","description":"Learn how to integrate Krisp noise cancellation into your Pipecat application","href":"/guides/features/krisp"},{"title":"Building With OpenAI Audio Models and APIs","sidebarTitle":"OpenAI Audio Models and APIs","description":"Create voice agents with OpenAI audio models and Pipecat","href":"/guides/features/openai-audio-models-and-apis"},{"title":"Pipecat Flows","description":"Learn how to create structured conversations using Pipecat’s flow system","href":"/guides/features/pipecat-flows"}]},{"group":"Telephony","pages":[{"title":"Overview","description":"How to call your bots (or have them call you)","href":"/guides/telephony/overview"},{"title":"Dial-in: WebRTC (Daily)","description":"Call your Pipecat bot using Daily WebRTC","href":"/guides/telephony/daily-webrtc"},{"title":"Dial-in: WebRTC (Twilio + Daily)","description":"Connect phone calls to your Pipecat bot using Twilio and Daily's SIP integration","href":"/guides/telephony/twilio-daily-webrtc"},{"title":"Dial-in: Twilio (Media Streams)","description":"Call your Pipecat bot over websockets using Twilio","href":"/guides/telephony/twilio-websockets"},{"title":"Dialout: WebRTC (Daily)","description":"Learn how to preview changes locally","href":"/guides/telephony/dialout"}]},{"group":"Deploying your bot","pages":[{"title":"Overview","description":"How to deploy your Pipecat bot online","href":"/guides/deployment/overview"},{"title":"Deployment pattern","description":"Basic deployment pattern for Pipecat bots","href":"/guides/deployment/pattern"},{"title":"Example: Pipecat Cloud","description":"Deploy Pipecat agents with managed infrastructure","href":"/guides/deployment/pipecat-cloud"},{"title":"Example: Fly.io","description":"Deploy Pipecat bots to Fly.io machines","href":"/guides/deployment/fly"},{"title":"Example: Cerebrium","description":"Deploy Pipecat applications to Cerebrium","href":"/guides/deployment/cerebrium"},{"title":"Example: Modal","description":"Deploy Pipecat applications to Modal","href":"/guides/deployment/modal"}]},{"group":"","pages":[{"title":"Server API Reference","description":"Complete reference for Pipecat’s Python server APIs and services","href":"/server/introduction"}]},{"group":"API Reference","pages":[{"title":"Reference docs","url":"https://pipecat-docs.readthedocs.io/","icon":"book","description":null,"href":"/server/links/server-reference"}]},{"group":"Services","pages":[{"title":"Supported Services","description":"AI services integrated with Pipecat and their setup requirements","href":"/server/services/supported-services"},{"group":"Transport","pages":[{"title":"Daily WebRTC","description":"WebRTC transport implementation using Daily for real-time audio/video communication","href":"/server/services/transport/daily"},{"title":"SmallWebRTCTransport","description":"A lightweight WebRTC transport for peer-to-peer audio and video communication in Pipecat","href":"/server/services/transport/small-webrtc"},{"title":"FastAPI WebSocket","description":"WebSocket transport implementation for FastAPI web applications","href":"/server/services/transport/fastapi-websocket"},{"title":"WebSocket Server","description":"WebSocket server transport implementation for real-time audio communication","href":"/server/services/transport/websocket-server"}]},{"group":"Serializers","pages":[{"title":"Frame Serializer Overview","description":"Overview of frame serializers for converting between Pipecat frames and external protocols","href":"/server/services/serializers/introduction"},{"title":"ExotelFrameSerializer","description":"Serializer for Exotel WebSocket media streaming protocol","href":"/server/services/serializers/exotel"},{"title":"PlivoFrameSerializer","description":"Serializer for Plivo Audio Streaming WebSocket protocol","href":"/server/services/serializers/plivo"},{"title":"TwilioFrameSerializer","description":"Serializer for Twilio Media Streams WebSocket protocol","href":"/server/services/serializers/twilio"},{"title":"TelnyxFrameSerializer","description":"Serializer for Telnyx WebSocket media streaming protocol","href":"/server/services/serializers/telnyx"}]},{"group":"Speech-to-Text","pages":[{"title":"AssemblyAI","description":"Speech-to-text service implementation using AssemblyAI’s real-time transcription API","href":"/server/services/stt/assemblyai"},{"title":"AWS Transcribe","description":"Speech-to-text service implementation using Amazon Transcribe’s real-time transcription API","href":"/server/services/stt/aws"},{"title":"Azure","description":"Speech-to-text service using Azure Cognitive Services Speech SDK","href":"/server/services/stt/azure"},{"title":"Cartesia","description":"Speech-to-text service implementation using Cartesia’s real-time transcription API","href":"/server/services/stt/cartesia"},{"title":"Deepgram","description":"Speech-to-text service implementation using Deepgram’s real-time transcription API","href":"/server/services/stt/deepgram"},{"title":"Fal (Wizper)","description":"Speech-to-text service implementation using Fal’s Wizper API","href":"/server/services/stt/fal"},{"title":"Gladia","description":"Speech-to-text service implementation using Gladia’s API","href":"/server/services/stt/gladia"},{"title":"Google","description":"Speech-to-text service implementation using Google Cloud’s Speech-to-Text V2 API","href":"/server/services/stt/google"},{"title":"Groq (Whisper)","description":"Speech-to-text service implementation using Groq’s Whisper API","href":"/server/services/stt/groq"},{"title":"NVIDIA Riva","description":"Speech-to-text service implementation using NVIDIA Riva","href":"/server/services/stt/riva"},{"title":"OpenAI","description":"Speech-to-text service implementation using OpenAI’s Speech-to-Text APIs","href":"/server/services/stt/openai"},{"title":"SambaNova (Whisper)","description":"Speech-to-text service implementation using SambaNova’s Whisper API","href":"/server/services/stt/sambanova"},{"title":"Speechmatics","description":"Speech-to-text service implementation using Speechmatics’ real-time transcription STT API","href":"/server/services/stt/speechmatics"},{"title":"Ultravox","description":"Speech-to-text service implementation using a locally-loaded Ultravox multimodal model","href":"/server/services/stt/ultravox"},{"title":"Whisper","description":"Speech-to-text service implementation using locally-downloaded Whisper models","href":"/server/services/stt/whisper"}]},{"group":"LLM","pages":[{"title":"Anthropic","description":"Large Language Model service implementation using Anthropic’s Claude API","href":"/server/services/llm/anthropic"},{"title":"AWS Bedrock","description":"Large Language Model service implementation using Amazon Bedrock API","href":"/server/services/llm/aws"},{"title":"Azure","description":"Large Language Model service implementation using Azure OpenAI API","href":"/server/services/llm/azure"},{"title":"Cerebras","description":"LLM service implementation using Cerebras’s API with OpenAI-compatible interface","href":"/server/services/llm/cerebras"},{"title":"DeepSeek","description":"LLM service implementation using DeepSeek’s API with OpenAI-compatible interface","href":"/server/services/llm/deepseek"},{"title":"Fireworks AI","description":"LLM service implementation using Fireworks AI’s API with OpenAI-compatible interface","href":"/server/services/llm/fireworks"},{"title":"Google Gemini","description":"Large Language Model service implementation using Google’s Gemini API","href":"/server/services/llm/gemini"},{"title":"Google Vertex AI","description":"LLM service implementation using Google’s Vertex AI with OpenAI-compatible interface","href":"/server/services/llm/google-vertex"},{"title":"Grok","description":"LLM service implementation using Grok’s API with OpenAI-compatible interface","href":"/server/services/llm/grok"},{"title":"Groq","description":"LLM service implementation using Groq’s API with OpenAI-compatible interface","href":"/server/services/llm/groq"},{"title":"NVIDIA NIM","description":"LLM service implementation using NVIDIA’s NIM (NVIDIA Inference Microservice) API with OpenAI-compatible interface","href":"/server/services/llm/nim"},{"title":"Ollama","description":"LLM service implementation using Ollama with OpenAI-compatible interface","href":"/server/services/llm/ollama"},{"title":"OpenAI","description":"Large Language Model services using OpenAI’s chat completion API","href":"/server/services/llm/openai"},{"title":"OpenPipe","description":"LLM service implementation using OpenPipe for LLM request logging and fine-tuning","href":"/server/services/llm/openpipe"},{"title":"OpenRouter","description":"LLM service implementation using OpenRouter’s API with OpenAI-compatible interface","href":"/server/services/llm/openrouter"},{"title":"Perplexity","description":"LLM service implementation using Perplexity’s API with OpenAI-compatible interface","href":"/server/services/llm/perplexity"},{"title":"Qwen","description":"LLM service implementation using Alibaba Cloud’s Qwen models through an OpenAI-compatible interface","href":"/server/services/llm/qwen"},{"title":"SambaNova","description":"LLM service implementation using SambaNova's API with OpenAI-compatible interface","href":"/server/services/llm/sambanova"},{"title":"Together AI","description":"LLM service implementation using Together AI’s API with OpenAI-compatible interface","href":"/server/services/llm/together"}]},{"group":"Text-to-Speech","pages":[{"title":"AWS Polly","description":"Text-to-speech service implementation using AWS Polly","href":"/server/services/tts/aws"},{"title":"Azure","description":"Text-to-speech service using Azure Cognitive Services Speech SDK","href":"/server/services/tts/azure"},{"title":"Cartesia","description":"Text-to-speech services using Cartesia’s WebSocket and HTTP APIs","href":"/server/services/tts/cartesia"},{"title":"Deepgram","description":"Text-to-speech service implementation using Deepgram’s Aura API","href":"/server/services/tts/deepgram"},{"title":"ElevenLabs","description":"Text-to-speech service using ElevenLab’s streaming API with word-level timing","href":"/server/services/tts/elevenlabs"},{"title":"Fish Audio","description":"Real-time text-to-speech service using Fish Audio's WebSocket API","href":"/server/services/tts/fish"},{"title":"Google","description":"Text-to-speech service using Google’s Cloud Text-to-Speech API","href":"/server/services/tts/google"},{"title":"Groq","description":"Text-to-speech service implementation using Groq’s TTS API","href":"/server/services/tts/groq"},{"title":"LMNT","description":"Text-to-speech service implementation using LMNT’s streaming API","href":"/server/services/tts/lmnt"},{"title":"MiniMax","description":"Text-to-speech service implementation using MiniMax T2A API","href":"/server/services/tts/minimax"},{"title":"Neuphonic","description":"Text-to-speech service implementation using Neuphonic’s API","href":"/server/services/tts/neuphonic"},{"title":"NVIDIA Riva","description":"Text-to-speech service implementation using NVIDIA Riva","href":"/server/services/tts/riva"},{"title":"OpenAI","description":"Text-to-speech service using OpenAI’s TTS API","href":"/server/services/tts/openai"},{"title":"Piper","description":"Text-to-speech service implementation using the Piper TTS server","href":"/server/services/tts/piper"},{"title":"PlayHT","description":"Text-to-speech services using PlayHT’s WebSocket and HTTP APIs","href":"/server/services/tts/playht"},{"title":"Rime","description":"Text-to-speech service implementations using Rime AI","href":"/server/services/tts/rime"},{"title":"Sarvam AI","description":"Text-to-speech service implementation using Sarvam AI’s TTS API","href":"/server/services/tts/sarvam"},{"title":"XTTS","description":"Text-to-speech service implementation using Coqui’s XTTS streaming server","href":"/server/services/tts/xtts"}]},{"group":"Speech-to-Speech","pages":[{"title":"AWS Nova Sonic","description":"Real-time speech-to-speech service implementation using AWS Nova Sonic","href":"/server/services/s2s/aws"},{"title":"Gemini Multimodal Live","description":"A real-time, multimodal conversational AI service powered by Google’s Gemini","href":"/server/services/s2s/gemini"},{"title":"OpenAI Realtime Beta","description":"Real-time speech-to-speech service implementation using OpenAI’s Realtime Beta API","href":"/server/services/s2s/openai"}]},{"group":"Image Generation","pages":[{"title":"fal","description":"Image generation service implementation using fal’s fast SDXL models","href":"/server/services/image-generation/fal"},{"title":"Google Imagen","description":"Image generation service implementation using Google’s Imagen models","href":"/server/services/image-generation/google-imagen"},{"title":"OpenAI Image Generation","sidebarTitle":"OpenAI","description":"Image generation service implementation using OpenAI’s DALL-E models","href":"/server/services/image-generation/openai"}]},{"group":"Video","pages":[{"title":"Simli","description":"Video service for generating AI avatar responses using Simli's WebRTC API","href":"/server/services/video/simli"},{"title":"Tavus","description":"Video service implementation for generating AI avatar responses using Tavus","href":"/server/services/video/tavus"}]},{"group":"Memory","pages":[{"title":"Mem0","description":"Long-term conversation memory service powered by Mem0","href":"/server/services/memory/mem0"}]},{"group":"Vision","pages":[{"title":"Moondream","description":"Vision service implementation using Moondream for local image analysis and question answering","href":"/server/services/vision/moondream"}]},{"group":"Analytics \u0026 Monitoring","pages":[{"title":"Sentry Metrics","description":"Performance monitoring integration with Sentry for Pipecat frame processors","href":"/server/services/analytics/sentry"}]}]},{"group":"Utilities","pages":[{"group":"Advanced Frame Processors","pages":[{"title":"Producer \u0026 Consumer Processors","description":"Route frames between different parts of a pipeline, allowing selective frame sharing across parallel branches or within complex pipelines","href":"/server/utilities/frame/producer-consumer"},{"title":"UserIdleProcessor","description":"A processor that monitors user inactivity and triggers callbacks after specified timeout periods","href":"/server/utilities/user-idle-processor"}]},{"group":"Audio Processing","pages":[{"title":"AudioBufferProcessor","description":"Process and buffer audio frames from conversations with flexible event handling","href":"/server/utilities/audio/audio-buffer-processor"},{"title":"KoalaFilter","description":"Audio noise reduction filter using Koala AI technology from Picovoice","href":"/server/utilities/audio/koala-filter"},{"title":"KrispFilter","description":"Audio noise reduction filter using Krisp AI technology","href":"/server/utilities/audio/krisp-filter"},{"title":"NoisereduceFilter","description":"Audio noise reduction filter using the noisereduce library","href":"/server/utilities/audio/noisereduce-filter"},{"title":"SileroVADAnalyzer","description":"Voice Activity Detection analyzer using the Silero VAD ONNX model","href":"/server/utilities/audio/silero-vad-analyzer"},{"title":"SoundfileMixer","description":"Audio mixer for combining real-time audio with sound files","href":"/server/utilities/audio/soundfile-mixer"}]},{"group":"Frame Filters","pages":[{"title":"FrameFilter","description":"Processor that selectively passes through only specified frame types","href":"/server/utilities/filters/frame-filter"},{"title":"FunctionFilter","description":"Processor that filters frames using a custom filter function","href":"/server/utilities/filters/function-filter"},{"title":"IdentityFilter","description":"Processor that passes all frames through without modification","href":"/server/utilities/filters/identify-filter"},{"title":"NullFilter","description":"Processor that blocks all frames except system frames","href":"/server/utilities/filters/null-filter"},{"title":"STTMuteFilter","description":"Processor for controlling STT muting and interruption handling during bot speech and function calls","href":"/server/utilities/filters/stt-mute"},{"title":"WakeCheckFilter","description":"Processor that passes frames only after detecting wake phrases in transcriptions","href":"/server/utilities/filters/wake-check-filter"},{"title":"WakeNotifierFilter","description":"Processor that triggers a notifier when specified frame types pass a custom filter","href":"/server/utilities/filters/wake-notifier-filter"}]},{"group":"Metrics and Telemetry","pages":[{"title":"OpenTelemetry Tracing","sidebarTitle":"OpenTelemetry","description":"Monitor and analyze your Pipecat conversational pipelines using OpenTelemetry","href":"/server/utilities/opentelemetry"}]},{"group":"MCP","pages":[{"title":"MCPClient","description":"Service to connect to MCP (Model Context Protocol) servers","href":"/server/utilities/mcp/mcp"}]},{"group":"Observers","pages":[{"title":"Observer Pattern","description":"Understanding and implementing observers in Pipecat","href":"/server/utilities/observers/observer-pattern"},{"title":"Debug Log Observer","sidebarTitle":"Debug Observer","description":"Comprehensive frame logging with configurable filtering in Pipecat","href":"/server/utilities/observers/debug-observer"},{"title":"LLM Log Observer","sidebarTitle":"LLM Observer","description":"Logging LLM activity in Pipecat","href":"/server/utilities/observers/llm-observer"},{"title":"Transcription Log Observer","sidebarTitle":"Transcription Observer","description":"Logging speech-to-text transcription activity in Pipecat","href":"/server/utilities/observers/transcription-observer"},{"title":"Turn Tracking Observer","description":"Track conversation turns and events in your Pipecat pipeline","href":"/server/utilities/observers/turn-tracking-observer"}]},{"group":"Service Utilities","pages":[{"title":"Daily REST Helper","description":"Classes and methods for interacting with the Daily API to manage rooms and tokens","href":"/server/utilities/daily/rest-helpers"}]},{"group":"Smart Turn Detection","pages":[{"title":"Smart Turn Overview","description":"Advanced conversational turn detection powered by the smart-turn model","href":"/server/utilities/smart-turn/smart-turn-overview"},{"title":"Fal Smart Turn","description":"Cloud-hosted Smart Turn detection using Fal.ai","href":"/server/utilities/smart-turn/fal-smart-turn"},{"title":"Local CoreML Smart Turn","description":"Local Smart Turn detection on Apple Silicon using CoreML","href":"/server/utilities/smart-turn/local-coreml-smart-turn"}]},{"group":"Task Handling and Monitoring","pages":[{"title":"Watchdog Timers","description":"Monitor task freezes and processing times","href":"/server/utilities/watchdog-timers"}]},{"group":"Telephony","pages":[{"title":"DTMFAggregator","description":"Aggregates DTMF (phone keypad) input into meaningful sequences for LLM processing","href":"/server/utilities/dtmf-aggregator"}]},{"group":"Text Aggregators and Filters","pages":[{"title":"MarkdownTextFilter","description":"Converts Markdown-formatted text to TTS-friendly plain text while preserving structure","href":"/server/utilities/text/markdown-text-filter"},{"title":"PatternPairAggregator","description":"Text aggregator that identifies and processes content between pattern pairs in streaming text","href":"/server/utilities/text/pattern-pair-aggregator"}]},{"group":"User and Bot Transcriptions","pages":[{"title":"TranscriptProcessor","description":"Factory for creating and managing conversation transcript processors with shared event handling","href":"/server/utilities/transcript-processor"}]},{"group":"User Interruptions","pages":[{"title":"Interruption Strategies","description":"Configure when users can interrupt the bot to prevent unwanted interruptions from brief affirmations","href":"/server/utilities/interruption-strategies"}]}]},{"group":"Frameworks","pages":[{"group":"RTVI","pages":[{"title":"RTVI (Real-Time Voice Interaction)","sidebarTitle":"Overview","description":"Build real-time voice and multimodal applications with Pipecat’s RTVI protocol","href":"/server/frameworks/rtvi/introduction"},{"title":"RTVIProcessor","description":"Core coordinator for RTVI protocol communication","href":"/server/frameworks/rtvi/rtvi-processor"},{"title":"RTVI Observer","description":"Converting pipeline frames to RTVI protocol messages","href":"/server/frameworks/rtvi/rtvi-observer"}]},{"title":"Pipecat Flows","description":"Technical reference for Pipecat’s conversation flow system","href":"/server/frameworks/flows/pipecat-flows"}]},{"group":"Pipeline","pages":[{"title":"PipelineParams","description":"Configure pipeline execution with PipelineParams","href":"/server/pipeline/pipeline-params"},{"title":"PipelineTask","description":"Manage pipeline execution and lifecycle with PipelineTask","href":"/server/pipeline/pipeline-task"},{"title":"Pipeline Idle Detection","description":"Automatically detect and handle idle pipelines with no bot activity","href":"/server/pipeline/pipeline-idle-detection"},{"title":"Pipeline Heartbeats","description":"Monitor pipeline health with heartbeat frames","href":"/server/pipeline/heartbeats"},{"title":"ParallelPipeline","description":"Run multiple pipeline branches in parallel, with synchronized inputs and outputs for complex flows","href":"/server/pipeline/parallel-pipeline"}]},{"group":"","pages":[{"title":"Client SDKs","description":"Client libraries for building real-time AI applications with Pipecat","href":"/client/introduction"},{"title":"The RTVI Standard","description":"An overview of the RTVI standard and its key features","href":"/client/rtvi-standard"},{"title":"RTVIClient Migration Guide","description":"A Guide to migrating from RTVIClient to PipecatClient","href":"/client/migration-guide"}]},{"group":"Javascript SDK","pages":[{"title":"SDK Introduction","description":"Build web applications with Pipecat’s JavaScript client library","href":"/client/js/introduction"},{"group":"API Reference","pages":[{"title":"PipecatClient Constructor","description":"Setting up the PipecatClient","href":"/client/js/api-reference/client-constructor"},{"title":"Client Methods","description":null,"href":"/client/js/api-reference/client-methods"},{"title":"Callbacks and events","description":null,"href":"/client/js/api-reference/callbacks"},{"title":"Custom Messaging","description":null,"href":"/client/js/api-reference/messages"},{"title":"Errors","description":null,"href":"/client/js/api-reference/errors"}]},{"group":"Transport packages","pages":[{"title":"Transport Overview","sidebarTitle":"Overview","description":null,"href":"/client/js/transports/transport"},{"title":"Daily WebRTC Transport","description":null,"href":"/client/js/transports/daily"},{"title":"SmallWebRTCTransport","description":"A lightweight WebRTC transport for peer-to-peer connections with Pipecat","href":"/client/js/transports/small-webrtc"},{"title":"WebSocketTransport","description":"A lightweight transport for WebSocket based connections with Pipecat","href":"/client/js/transports/websocket"},{"title":"GeminiLiveWebSocketTransport","description":null,"href":"/client/js/transports/gemini"},{"title":"OpenAIRealTimeWebRTCTransport","description":null,"href":"/client/js/transports/openai-webrtc"}]}]},{"group":"React SDK","pages":[{"title":"SDK Introduction","description":"Build React applications with Pipecat's React client library","href":"/client/react/introduction"},{"group":"API Reference","pages":[{"title":"Components","description":"Ready-to-use React components for Pipecat applications","href":"/client/react/components"},{"title":"Hooks","description":"React hooks for accessing Pipecat client functionality","href":"/client/react/hooks"}]}]},{"group":"React Native SDK","pages":[{"title":"SDK Introduction","description":"Build React Native applications with Pipecat's React Native client library","href":"/client/react-native/introduction"},{"title":"API Reference","description":"API reference for the Pipecat React Native SDK","href":"/client/react-native/api-reference"}]},{"group":"iOS SDK","pages":[{"title":"SDK Introduction","description":"Build iOS applications with Pipecat’s Swift client library","href":"/client/ios/introduction"},{"title":"API Reference","url":"https://docs-ios.pipecat.ai/","description":null,"href":"/client/ios/api-reference"},{"group":"Transport packages","pages":[{"title":"Daily WebRTC Transport","description":"WebRTC implementation for iOS using Daily","href":"/client/ios/transports/daily"},{"title":"Gemini Live Websocket Transport","description":"Websocket implementation for iOS using Gemini","href":"/client/ios/transports/gemini-websocket"},{"title":"OpenAIRealTimeWebRTCTransport","description":null,"href":"/client/ios/transports/openai-webrtc"}]}]},{"group":"Android SDK","pages":[{"title":"SDK Introduction","description":"Build Android applications with Pipecat's Kotlin client library","href":"/client/android/introduction"},{"title":"API Reference","url":"https://docs-android.rtvi.ai/","description":null,"href":"/client/android/api-reference"},{"group":"Transport packages","pages":[{"title":"Daily WebRTC Transport","description":"WebRTC implementation for Android using Daily","href":"/client/android/transports/daily"},{"title":"Gemini Live Websocket Transport","description":"Websocket implementation for Android using Gemini","href":"/client/android/transports/gemini-websocket"},{"title":"OpenAI Realtime WebRTC Transport","description":"WebRTC implementation for Android using OpenAI","href":"/client/android/transports/openai-webrtc"},{"title":"Small WebRTC Transport","description":"WebRTC implementation for Android","href":"/client/android/transports/small-webrtc"}]}]},{"group":"C++ SDK","pages":[{"title":"SDK Introduction","description":"Build native applications with Pipecat’s C++ client library","href":"/client/c++/introduction"},{"title":"Daily WebRTC Transport","description":"WebRTC implementation for C++ using Daily","href":"/client/c++/transport"}]}],"docsNavWithMetadata":{"global":{"anchors":[{"anchor":"Community","href":"https://discord.gg/pipecat","icon":"discord"},{"anchor":"GitHub","href":"https://github.com/pipecat-ai/pipecat","icon":"github"},{"anchor":"Examples","href":"https://github.com/pipecat-ai/pipecat/tree/main/examples","icon":"code"},{"anchor":"Changelog","href":"https://github.com/pipecat-ai/pipecat/blob/main/CHANGELOG.md","icon":"file"}]},"tabs":[{"tab":"Getting Started","groups":[{"group":"Get Started","pages":[{"title":"Overview","description":"Pipecat is a framework for building voice-enabled, real-time, multimodal AI applications","href":"/getting-started/overview"},{"title":"Installation \u0026 Setup","description":"Get Pipecat and its required services installed on your machine","href":"/getting-started/installation"},{"title":"Quickstart","description":"Build and run your first Pipecat application","href":"/getting-started/quickstart"},{"title":"Core Concepts","description":"Understanding how Pipecat works: frames, processors, and pipelines.","href":"/getting-started/core-concepts"},{"title":"Next Steps \u0026 Examples","description":"Where to go next with Pipecat based on your project needs","href":"/getting-started/next-steps"}]}]},{"tab":"Guides","groups":[{"group":"","pages":[{"title":"Guides","description":"Learn how to deploy, scale, and extend your Pipecat applications","href":"/guides/introduction"}]},{"group":"Fundamentals","pages":[{"title":"Context Management","description":"A guide to working with Pipecat's Context and Context Aggregators","href":"/guides/fundamentals/context-management"},{"title":"Custom FrameProcessor","description":"Learn how to write your own custom FrameProcessor","href":"/guides/fundamentals/custom-frame-processor"},{"title":"Detecting Idle Users","description":"Learn how to detect and respond when users are inactive in conversations","href":"/guides/fundamentals/detecting-user-idle"},{"title":"Ending a Pipeline","description":"Best practices for properly terminating Pipecat pipelines","href":"/guides/fundamentals/end-pipeline"},{"title":"Function Calling","description":"Enable LLMs to interact with external services and APIs","href":"/guides/fundamentals/function-calling"},{"title":"User Input Muting with STTMuteFilter","sidebarTitle":"Muting User Input","description":"Learn how to control when user speech is processed in your conversational bot","href":"/guides/fundamentals/user-input-muting"},{"title":"Recording Conversation Audio","sidebarTitle":"Recording Audio","description":"Learn how to record and save audio from conversations between users and your bot","href":"/guides/fundamentals/recording-audio"},{"title":"Recording Conversation Transcripts","sidebarTitle":"Recording Transcripts","description":"Learn how to collect and save conversation transcripts between users and your bot","href":"/guides/fundamentals/recording-transcripts"}]},{"group":"Features","pages":[{"title":"Building with Gemini Multimodal Live","sidebarTitle":"Gemini Multimodal Live","description":"Create a real-time AI chatbot using Gemini Multimodal Live and Pipecat","href":"/guides/features/gemini-multimodal-live"},{"title":"Metrics","description":"Learn how to monitor performance and LLM/TTS usage with Pipecat.","href":"/guides/features/metrics"},{"title":"Noise cancellation with Krisp","description":"Learn how to integrate Krisp noise cancellation into your Pipecat application","href":"/guides/features/krisp"},{"title":"Building With OpenAI Audio Models and APIs","sidebarTitle":"OpenAI Audio Models and APIs","description":"Create voice agents with OpenAI audio models and Pipecat","href":"/guides/features/openai-audio-models-and-apis"},{"title":"Pipecat Flows","description":"Learn how to create structured conversations using Pipecat’s flow system","href":"/guides/features/pipecat-flows"}]},{"group":"Telephony","pages":[{"title":"Overview","description":"How to call your bots (or have them call you)","href":"/guides/telephony/overview"},{"title":"Dial-in: WebRTC (Daily)","description":"Call your Pipecat bot using Daily WebRTC","href":"/guides/telephony/daily-webrtc"},{"title":"Dial-in: WebRTC (Twilio + Daily)","description":"Connect phone calls to your Pipecat bot using Twilio and Daily's SIP integration","href":"/guides/telephony/twilio-daily-webrtc"},{"title":"Dial-in: Twilio (Media Streams)","description":"Call your Pipecat bot over websockets using Twilio","href":"/guides/telephony/twilio-websockets"},{"title":"Dialout: WebRTC (Daily)","description":"Learn how to preview changes locally","href":"/guides/telephony/dialout"}]},{"group":"Deploying your bot","pages":[{"title":"Overview","description":"How to deploy your Pipecat bot online","href":"/guides/deployment/overview"},{"title":"Deployment pattern","description":"Basic deployment pattern for Pipecat bots","href":"/guides/deployment/pattern"},{"title":"Example: Pipecat Cloud","description":"Deploy Pipecat agents with managed infrastructure","href":"/guides/deployment/pipecat-cloud"},{"title":"Example: Fly.io","description":"Deploy Pipecat bots to Fly.io machines","href":"/guides/deployment/fly"},{"title":"Example: Cerebrium","description":"Deploy Pipecat applications to Cerebrium","href":"/guides/deployment/cerebrium"},{"title":"Example: Modal","description":"Deploy Pipecat applications to Modal","href":"/guides/deployment/modal"}]}]},{"tab":"Server APIs","groups":[{"group":"","pages":[{"title":"Server API Reference","description":"Complete reference for Pipecat’s Python server APIs and services","href":"/server/introduction"}]},{"group":"API Reference","pages":[{"title":"Reference docs","url":"https://pipecat-docs.readthedocs.io/","icon":"book","description":null,"href":"/server/links/server-reference"}]},{"group":"Services","pages":[{"title":"Supported Services","description":"AI services integrated with Pipecat and their setup requirements","href":"/server/services/supported-services"},{"group":"Transport","pages":[{"title":"Daily WebRTC","description":"WebRTC transport implementation using Daily for real-time audio/video communication","href":"/server/services/transport/daily"},{"title":"SmallWebRTCTransport","description":"A lightweight WebRTC transport for peer-to-peer audio and video communication in Pipecat","href":"/server/services/transport/small-webrtc"},{"title":"FastAPI WebSocket","description":"WebSocket transport implementation for FastAPI web applications","href":"/server/services/transport/fastapi-websocket"},{"title":"WebSocket Server","description":"WebSocket server transport implementation for real-time audio communication","href":"/server/services/transport/websocket-server"}]},{"group":"Serializers","pages":[{"title":"Frame Serializer Overview","description":"Overview of frame serializers for converting between Pipecat frames and external protocols","href":"/server/services/serializers/introduction"},{"title":"ExotelFrameSerializer","description":"Serializer for Exotel WebSocket media streaming protocol","href":"/server/services/serializers/exotel"},{"title":"PlivoFrameSerializer","description":"Serializer for Plivo Audio Streaming WebSocket protocol","href":"/server/services/serializers/plivo"},{"title":"TwilioFrameSerializer","description":"Serializer for Twilio Media Streams WebSocket protocol","href":"/server/services/serializers/twilio"},{"title":"TelnyxFrameSerializer","description":"Serializer for Telnyx WebSocket media streaming protocol","href":"/server/services/serializers/telnyx"}]},{"group":"Speech-to-Text","pages":[{"title":"AssemblyAI","description":"Speech-to-text service implementation using AssemblyAI’s real-time transcription API","href":"/server/services/stt/assemblyai"},{"title":"AWS Transcribe","description":"Speech-to-text service implementation using Amazon Transcribe’s real-time transcription API","href":"/server/services/stt/aws"},{"title":"Azure","description":"Speech-to-text service using Azure Cognitive Services Speech SDK","href":"/server/services/stt/azure"},{"title":"Cartesia","description":"Speech-to-text service implementation using Cartesia’s real-time transcription API","href":"/server/services/stt/cartesia"},{"title":"Deepgram","description":"Speech-to-text service implementation using Deepgram’s real-time transcription API","href":"/server/services/stt/deepgram"},{"title":"Fal (Wizper)","description":"Speech-to-text service implementation using Fal’s Wizper API","href":"/server/services/stt/fal"},{"title":"Gladia","description":"Speech-to-text service implementation using Gladia’s API","href":"/server/services/stt/gladia"},{"title":"Google","description":"Speech-to-text service implementation using Google Cloud’s Speech-to-Text V2 API","href":"/server/services/stt/google"},{"title":"Groq (Whisper)","description":"Speech-to-text service implementation using Groq’s Whisper API","href":"/server/services/stt/groq"},{"title":"NVIDIA Riva","description":"Speech-to-text service implementation using NVIDIA Riva","href":"/server/services/stt/riva"},{"title":"OpenAI","description":"Speech-to-text service implementation using OpenAI’s Speech-to-Text APIs","href":"/server/services/stt/openai"},{"title":"SambaNova (Whisper)","description":"Speech-to-text service implementation using SambaNova’s Whisper API","href":"/server/services/stt/sambanova"},{"title":"Speechmatics","description":"Speech-to-text service implementation using Speechmatics’ real-time transcription STT API","href":"/server/services/stt/speechmatics"},{"title":"Ultravox","description":"Speech-to-text service implementation using a locally-loaded Ultravox multimodal model","href":"/server/services/stt/ultravox"},{"title":"Whisper","description":"Speech-to-text service implementation using locally-downloaded Whisper models","href":"/server/services/stt/whisper"}]},{"group":"LLM","pages":[{"title":"Anthropic","description":"Large Language Model service implementation using Anthropic’s Claude API","href":"/server/services/llm/anthropic"},{"title":"AWS Bedrock","description":"Large Language Model service implementation using Amazon Bedrock API","href":"/server/services/llm/aws"},{"title":"Azure","description":"Large Language Model service implementation using Azure OpenAI API","href":"/server/services/llm/azure"},{"title":"Cerebras","description":"LLM service implementation using Cerebras’s API with OpenAI-compatible interface","href":"/server/services/llm/cerebras"},{"title":"DeepSeek","description":"LLM service implementation using DeepSeek’s API with OpenAI-compatible interface","href":"/server/services/llm/deepseek"},{"title":"Fireworks AI","description":"LLM service implementation using Fireworks AI’s API with OpenAI-compatible interface","href":"/server/services/llm/fireworks"},{"title":"Google Gemini","description":"Large Language Model service implementation using Google’s Gemini API","href":"/server/services/llm/gemini"},{"title":"Google Vertex AI","description":"LLM service implementation using Google’s Vertex AI with OpenAI-compatible interface","href":"/server/services/llm/google-vertex"},{"title":"Grok","description":"LLM service implementation using Grok’s API with OpenAI-compatible interface","href":"/server/services/llm/grok"},{"title":"Groq","description":"LLM service implementation using Groq’s API with OpenAI-compatible interface","href":"/server/services/llm/groq"},{"title":"NVIDIA NIM","description":"LLM service implementation using NVIDIA’s NIM (NVIDIA Inference Microservice) API with OpenAI-compatible interface","href":"/server/services/llm/nim"},{"title":"Ollama","description":"LLM service implementation using Ollama with OpenAI-compatible interface","href":"/server/services/llm/ollama"},{"title":"OpenAI","description":"Large Language Model services using OpenAI’s chat completion API","href":"/server/services/llm/openai"},{"title":"OpenPipe","description":"LLM service implementation using OpenPipe for LLM request logging and fine-tuning","href":"/server/services/llm/openpipe"},{"title":"OpenRouter","description":"LLM service implementation using OpenRouter’s API with OpenAI-compatible interface","href":"/server/services/llm/openrouter"},{"title":"Perplexity","description":"LLM service implementation using Perplexity’s API with OpenAI-compatible interface","href":"/server/services/llm/perplexity"},{"title":"Qwen","description":"LLM service implementation using Alibaba Cloud’s Qwen models through an OpenAI-compatible interface","href":"/server/services/llm/qwen"},{"title":"SambaNova","description":"LLM service implementation using SambaNova's API with OpenAI-compatible interface","href":"/server/services/llm/sambanova"},{"title":"Together AI","description":"LLM service implementation using Together AI’s API with OpenAI-compatible interface","href":"/server/services/llm/together"}]},{"group":"Text-to-Speech","pages":[{"title":"AWS Polly","description":"Text-to-speech service implementation using AWS Polly","href":"/server/services/tts/aws"},{"title":"Azure","description":"Text-to-speech service using Azure Cognitive Services Speech SDK","href":"/server/services/tts/azure"},{"title":"Cartesia","description":"Text-to-speech services using Cartesia’s WebSocket and HTTP APIs","href":"/server/services/tts/cartesia"},{"title":"Deepgram","description":"Text-to-speech service implementation using Deepgram’s Aura API","href":"/server/services/tts/deepgram"},{"title":"ElevenLabs","description":"Text-to-speech service using ElevenLab’s streaming API with word-level timing","href":"/server/services/tts/elevenlabs"},{"title":"Fish Audio","description":"Real-time text-to-speech service using Fish Audio's WebSocket API","href":"/server/services/tts/fish"},{"title":"Google","description":"Text-to-speech service using Google’s Cloud Text-to-Speech API","href":"/server/services/tts/google"},{"title":"Groq","description":"Text-to-speech service implementation using Groq’s TTS API","href":"/server/services/tts/groq"},{"title":"LMNT","description":"Text-to-speech service implementation using LMNT’s streaming API","href":"/server/services/tts/lmnt"},{"title":"MiniMax","description":"Text-to-speech service implementation using MiniMax T2A API","href":"/server/services/tts/minimax"},{"title":"Neuphonic","description":"Text-to-speech service implementation using Neuphonic’s API","href":"/server/services/tts/neuphonic"},{"title":"NVIDIA Riva","description":"Text-to-speech service implementation using NVIDIA Riva","href":"/server/services/tts/riva"},{"title":"OpenAI","description":"Text-to-speech service using OpenAI’s TTS API","href":"/server/services/tts/openai"},{"title":"Piper","description":"Text-to-speech service implementation using the Piper TTS server","href":"/server/services/tts/piper"},{"title":"PlayHT","description":"Text-to-speech services using PlayHT’s WebSocket and HTTP APIs","href":"/server/services/tts/playht"},{"title":"Rime","description":"Text-to-speech service implementations using Rime AI","href":"/server/services/tts/rime"},{"title":"Sarvam AI","description":"Text-to-speech service implementation using Sarvam AI’s TTS API","href":"/server/services/tts/sarvam"},{"title":"XTTS","description":"Text-to-speech service implementation using Coqui’s XTTS streaming server","href":"/server/services/tts/xtts"}]},{"group":"Speech-to-Speech","pages":[{"title":"AWS Nova Sonic","description":"Real-time speech-to-speech service implementation using AWS Nova Sonic","href":"/server/services/s2s/aws"},{"title":"Gemini Multimodal Live","description":"A real-time, multimodal conversational AI service powered by Google’s Gemini","href":"/server/services/s2s/gemini"},{"title":"OpenAI Realtime Beta","description":"Real-time speech-to-speech service implementation using OpenAI’s Realtime Beta API","href":"/server/services/s2s/openai"}]},{"group":"Image Generation","pages":[{"title":"fal","description":"Image generation service implementation using fal’s fast SDXL models","href":"/server/services/image-generation/fal"},{"title":"Google Imagen","description":"Image generation service implementation using Google’s Imagen models","href":"/server/services/image-generation/google-imagen"},{"title":"OpenAI Image Generation","sidebarTitle":"OpenAI","description":"Image generation service implementation using OpenAI’s DALL-E models","href":"/server/services/image-generation/openai"}]},{"group":"Video","pages":[{"title":"Simli","description":"Video service for generating AI avatar responses using Simli's WebRTC API","href":"/server/services/video/simli"},{"title":"Tavus","description":"Video service implementation for generating AI avatar responses using Tavus","href":"/server/services/video/tavus"}]},{"group":"Memory","pages":[{"title":"Mem0","description":"Long-term conversation memory service powered by Mem0","href":"/server/services/memory/mem0"}]},{"group":"Vision","pages":[{"title":"Moondream","description":"Vision service implementation using Moondream for local image analysis and question answering","href":"/server/services/vision/moondream"}]},{"group":"Analytics \u0026 Monitoring","pages":[{"title":"Sentry Metrics","description":"Performance monitoring integration with Sentry for Pipecat frame processors","href":"/server/services/analytics/sentry"}]}]},{"group":"Utilities","pages":[{"group":"Advanced Frame Processors","pages":[{"title":"Producer \u0026 Consumer Processors","description":"Route frames between different parts of a pipeline, allowing selective frame sharing across parallel branches or within complex pipelines","href":"/server/utilities/frame/producer-consumer"},{"title":"UserIdleProcessor","description":"A processor that monitors user inactivity and triggers callbacks after specified timeout periods","href":"/server/utilities/user-idle-processor"}]},{"group":"Audio Processing","pages":[{"title":"AudioBufferProcessor","description":"Process and buffer audio frames from conversations with flexible event handling","href":"/server/utilities/audio/audio-buffer-processor"},{"title":"KoalaFilter","description":"Audio noise reduction filter using Koala AI technology from Picovoice","href":"/server/utilities/audio/koala-filter"},{"title":"KrispFilter","description":"Audio noise reduction filter using Krisp AI technology","href":"/server/utilities/audio/krisp-filter"},{"title":"NoisereduceFilter","description":"Audio noise reduction filter using the noisereduce library","href":"/server/utilities/audio/noisereduce-filter"},{"title":"SileroVADAnalyzer","description":"Voice Activity Detection analyzer using the Silero VAD ONNX model","href":"/server/utilities/audio/silero-vad-analyzer"},{"title":"SoundfileMixer","description":"Audio mixer for combining real-time audio with sound files","href":"/server/utilities/audio/soundfile-mixer"}]},{"group":"Frame Filters","pages":[{"title":"FrameFilter","description":"Processor that selectively passes through only specified frame types","href":"/server/utilities/filters/frame-filter"},{"title":"FunctionFilter","description":"Processor that filters frames using a custom filter function","href":"/server/utilities/filters/function-filter"},{"title":"IdentityFilter","description":"Processor that passes all frames through without modification","href":"/server/utilities/filters/identify-filter"},{"title":"NullFilter","description":"Processor that blocks all frames except system frames","href":"/server/utilities/filters/null-filter"},{"title":"STTMuteFilter","description":"Processor for controlling STT muting and interruption handling during bot speech and function calls","href":"/server/utilities/filters/stt-mute"},{"title":"WakeCheckFilter","description":"Processor that passes frames only after detecting wake phrases in transcriptions","href":"/server/utilities/filters/wake-check-filter"},{"title":"WakeNotifierFilter","description":"Processor that triggers a notifier when specified frame types pass a custom filter","href":"/server/utilities/filters/wake-notifier-filter"}]},{"group":"Metrics and Telemetry","pages":[{"title":"OpenTelemetry Tracing","sidebarTitle":"OpenTelemetry","description":"Monitor and analyze your Pipecat conversational pipelines using OpenTelemetry","href":"/server/utilities/opentelemetry"}]},{"group":"MCP","pages":[{"title":"MCPClient","description":"Service to connect to MCP (Model Context Protocol) servers","href":"/server/utilities/mcp/mcp"}]},{"group":"Observers","pages":[{"title":"Observer Pattern","description":"Understanding and implementing observers in Pipecat","href":"/server/utilities/observers/observer-pattern"},{"title":"Debug Log Observer","sidebarTitle":"Debug Observer","description":"Comprehensive frame logging with configurable filtering in Pipecat","href":"/server/utilities/observers/debug-observer"},{"title":"LLM Log Observer","sidebarTitle":"LLM Observer","description":"Logging LLM activity in Pipecat","href":"/server/utilities/observers/llm-observer"},{"title":"Transcription Log Observer","sidebarTitle":"Transcription Observer","description":"Logging speech-to-text transcription activity in Pipecat","href":"/server/utilities/observers/transcription-observer"},{"title":"Turn Tracking Observer","description":"Track conversation turns and events in your Pipecat pipeline","href":"/server/utilities/observers/turn-tracking-observer"}]},{"group":"Service Utilities","pages":[{"title":"Daily REST Helper","description":"Classes and methods for interacting with the Daily API to manage rooms and tokens","href":"/server/utilities/daily/rest-helpers"}]},{"group":"Smart Turn Detection","pages":[{"title":"Smart Turn Overview","description":"Advanced conversational turn detection powered by the smart-turn model","href":"/server/utilities/smart-turn/smart-turn-overview"},{"title":"Fal Smart Turn","description":"Cloud-hosted Smart Turn detection using Fal.ai","href":"/server/utilities/smart-turn/fal-smart-turn"},{"title":"Local CoreML Smart Turn","description":"Local Smart Turn detection on Apple Silicon using CoreML","href":"/server/utilities/smart-turn/local-coreml-smart-turn"}]},{"group":"Task Handling and Monitoring","pages":[{"title":"Watchdog Timers","description":"Monitor task freezes and processing times","href":"/server/utilities/watchdog-timers"}]},{"group":"Telephony","pages":[{"title":"DTMFAggregator","description":"Aggregates DTMF (phone keypad) input into meaningful sequences for LLM processing","href":"/server/utilities/dtmf-aggregator"}]},{"group":"Text Aggregators and Filters","pages":[{"title":"MarkdownTextFilter","description":"Converts Markdown-formatted text to TTS-friendly plain text while preserving structure","href":"/server/utilities/text/markdown-text-filter"},{"title":"PatternPairAggregator","description":"Text aggregator that identifies and processes content between pattern pairs in streaming text","href":"/server/utilities/text/pattern-pair-aggregator"}]},{"group":"User and Bot Transcriptions","pages":[{"title":"TranscriptProcessor","description":"Factory for creating and managing conversation transcript processors with shared event handling","href":"/server/utilities/transcript-processor"}]},{"group":"User Interruptions","pages":[{"title":"Interruption Strategies","description":"Configure when users can interrupt the bot to prevent unwanted interruptions from brief affirmations","href":"/server/utilities/interruption-strategies"}]}]},{"group":"Frameworks","pages":[{"group":"RTVI","pages":[{"title":"RTVI (Real-Time Voice Interaction)","sidebarTitle":"Overview","description":"Build real-time voice and multimodal applications with Pipecat’s RTVI protocol","href":"/server/frameworks/rtvi/introduction"},{"title":"RTVIProcessor","description":"Core coordinator for RTVI protocol communication","href":"/server/frameworks/rtvi/rtvi-processor"},{"title":"RTVI Observer","description":"Converting pipeline frames to RTVI protocol messages","href":"/server/frameworks/rtvi/rtvi-observer"}]},{"title":"Pipecat Flows","description":"Technical reference for Pipecat’s conversation flow system","href":"/server/frameworks/flows/pipecat-flows"}]},{"group":"Pipeline","pages":[{"title":"PipelineParams","description":"Configure pipeline execution with PipelineParams","href":"/server/pipeline/pipeline-params"},{"title":"PipelineTask","description":"Manage pipeline execution and lifecycle with PipelineTask","href":"/server/pipeline/pipeline-task"},{"title":"Pipeline Idle Detection","description":"Automatically detect and handle idle pipelines with no bot activity","href":"/server/pipeline/pipeline-idle-detection"},{"title":"Pipeline Heartbeats","description":"Monitor pipeline health with heartbeat frames","href":"/server/pipeline/heartbeats"},{"title":"ParallelPipeline","description":"Run multiple pipeline branches in parallel, with synchronized inputs and outputs for complex flows","href":"/server/pipeline/parallel-pipeline"}]}]},{"tab":"Client SDKs","groups":[{"group":"","pages":[{"title":"Client SDKs","description":"Client libraries for building real-time AI applications with Pipecat","href":"/client/introduction"},{"title":"The RTVI Standard","description":"An overview of the RTVI standard and its key features","href":"/client/rtvi-standard"},{"title":"RTVIClient Migration Guide","description":"A Guide to migrating from RTVIClient to PipecatClient","href":"/client/migration-guide"}]},{"group":"Javascript SDK","pages":[{"title":"SDK Introduction","description":"Build web applications with Pipecat’s JavaScript client library","href":"/client/js/introduction"},{"group":"API Reference","pages":[{"title":"PipecatClient Constructor","description":"Setting up the PipecatClient","href":"/client/js/api-reference/client-constructor"},{"title":"Client Methods","description":null,"href":"/client/js/api-reference/client-methods"},{"title":"Callbacks and events","description":null,"href":"/client/js/api-reference/callbacks"},{"title":"Custom Messaging","description":null,"href":"/client/js/api-reference/messages"},{"title":"Errors","description":null,"href":"/client/js/api-reference/errors"}]},{"group":"Transport packages","pages":[{"title":"Transport Overview","sidebarTitle":"Overview","description":null,"href":"/client/js/transports/transport"},{"title":"Daily WebRTC Transport","description":null,"href":"/client/js/transports/daily"},{"title":"SmallWebRTCTransport","description":"A lightweight WebRTC transport for peer-to-peer connections with Pipecat","href":"/client/js/transports/small-webrtc"},{"title":"WebSocketTransport","description":"A lightweight transport for WebSocket based connections with Pipecat","href":"/client/js/transports/websocket"},{"title":"GeminiLiveWebSocketTransport","description":null,"href":"/client/js/transports/gemini"},{"title":"OpenAIRealTimeWebRTCTransport","description":null,"href":"/client/js/transports/openai-webrtc"}]}]},{"group":"React SDK","pages":[{"title":"SDK Introduction","description":"Build React applications with Pipecat's React client library","href":"/client/react/introduction"},{"group":"API Reference","pages":[{"title":"Components","description":"Ready-to-use React components for Pipecat applications","href":"/client/react/components"},{"title":"Hooks","description":"React hooks for accessing Pipecat client functionality","href":"/client/react/hooks"}]}]},{"group":"React Native SDK","pages":[{"title":"SDK Introduction","description":"Build React Native applications with Pipecat's React Native client library","href":"/client/react-native/introduction"},{"title":"API Reference","description":"API reference for the Pipecat React Native SDK","href":"/client/react-native/api-reference"}]},{"group":"iOS SDK","pages":[{"title":"SDK Introduction","description":"Build iOS applications with Pipecat’s Swift client library","href":"/client/ios/introduction"},{"title":"API Reference","url":"https://docs-ios.pipecat.ai/","description":null,"href":"/client/ios/api-reference"},{"group":"Transport packages","pages":[{"title":"Daily WebRTC Transport","description":"WebRTC implementation for iOS using Daily","href":"/client/ios/transports/daily"},{"title":"Gemini Live Websocket Transport","description":"Websocket implementation for iOS using Gemini","href":"/client/ios/transports/gemini-websocket"},{"title":"OpenAIRealTimeWebRTCTransport","description":null,"href":"/client/ios/transports/openai-webrtc"}]}]},{"group":"Android SDK","pages":[{"title":"SDK Introduction","description":"Build Android applications with Pipecat's Kotlin client library","href":"/client/android/introduction"},{"title":"API Reference","url":"https://docs-android.rtvi.ai/","description":null,"href":"/client/android/api-reference"},{"group":"Transport packages","pages":[{"title":"Daily WebRTC Transport","description":"WebRTC implementation for Android using Daily","href":"/client/android/transports/daily"},{"title":"Gemini Live Websocket Transport","description":"Websocket implementation for Android using Gemini","href":"/client/android/transports/gemini-websocket"},{"title":"OpenAI Realtime WebRTC Transport","description":"WebRTC implementation for Android using OpenAI","href":"/client/android/transports/openai-webrtc"},{"title":"Small WebRTC Transport","description":"WebRTC implementation for Android","href":"/client/android/transports/small-webrtc"}]}]},{"group":"C++ SDK","pages":[{"title":"SDK Introduction","description":"Build native applications with Pipecat’s C++ client library","href":"/client/c++/introduction"},{"title":"Daily WebRTC Transport","description":"WebRTC implementation for C++ using Daily","href":"/client/c++/transport"}]}]}]},"pageMetadata":{"title":"Overview","description":"Pipecat is a framework for building voice-enabled, real-time, multimodal AI applications","href":"/getting-started/overview"},"mintConfig":{"layout":"topnav","sidebar":{"items":"container"},"topbar":{"style":"default"},"search":{"location":"top"},"rounded":"default","codeBlock":{"mode":"dark"},"topbarCtaButton":{"name":"","url":"","style":"pill","arrow":true},"$schema":"https://mintlify.com/schema.json","name":"Pipecat","logo":{"light":"https://mintlify.s3.us-west-1.amazonaws.com/daily/logo/light.svg","dark":"https://mintlify.s3.us-west-1.amazonaws.com/daily/logo/dark.svg"},"favicon":"/favicon.svg","metadata":{"og:site_name":"Pipecat","og:description":"Learn about Pipecat, the open source framework for building voice and multimodal AI agents.","og:url":"https://docs.pipecat.ai/","og:locale":"en_US","og:logo":"/logo/pipecat-logo.svg","article:publisher":"Pipecat AI","twitter:description":"Learn about Pipecat, the open source framework for building voice and multimodal AI agents.","twitter:url":"https://docs.pipecat.ai/","twitter:site":"@pipecat_ai","og:image:width":"1200","og:image:height":"675"},"colors":{"primary":"#4f46e5","light":"#a5b4fc","dark":"#312e81","anchors":{"from":"#312e81","to":"#6366f1"}},"topbarLinks":[{"url":"https://discord.gg/pipecat","name":"Join the Discord","_id":"686c39bbc459dfd6f1155b1b"}],"navigation":[{"group":"Get Started","pages":["getting-started/overview","getting-started/installation","getting-started/quickstart","getting-started/core-concepts","getting-started/next-steps"]},{"group":"","pages":["guides/introduction"]},{"group":"Fundamentals","pages":["guides/fundamentals/context-management","guides/fundamentals/custom-frame-processor","guides/fundamentals/detecting-user-idle","guides/fundamentals/end-pipeline","guides/fundamentals/function-calling","guides/fundamentals/user-input-muting","guides/fundamentals/recording-audio","guides/fundamentals/recording-transcripts"]},{"group":"Features","pages":["guides/features/gemini-multimodal-live","guides/features/metrics","guides/features/krisp","guides/features/openai-audio-models-and-apis","guides/features/pipecat-flows"]},{"group":"Telephony","pages":["guides/telephony/overview","guides/telephony/daily-webrtc","guides/telephony/twilio-daily-webrtc","guides/telephony/twilio-websockets","guides/telephony/dialout"]},{"group":"Deploying your bot","pages":["guides/deployment/overview","guides/deployment/pattern","guides/deployment/pipecat-cloud","guides/deployment/fly","guides/deployment/cerebrium","guides/deployment/modal"]},{"group":"","pages":["server/introduction"]},{"group":"API Reference","pages":["server/links/server-reference"]},{"group":"Services","pages":["server/services/supported-services",{"group":"Transport","pages":["server/services/transport/daily","server/services/transport/small-webrtc","server/services/transport/fastapi-websocket","server/services/transport/websocket-server"]},{"group":"Serializers","pages":["server/services/serializers/introduction","server/services/serializers/exotel","server/services/serializers/plivo","server/services/serializers/twilio","server/services/serializers/telnyx"]},{"group":"Speech-to-Text","pages":["server/services/stt/assemblyai","server/services/stt/aws","server/services/stt/azure","server/services/stt/cartesia","server/services/stt/deepgram","server/services/stt/fal","server/services/stt/gladia","server/services/stt/google","server/services/stt/groq","server/services/stt/riva","server/services/stt/openai","server/services/stt/sambanova","server/services/stt/speechmatics","server/services/stt/ultravox","server/services/stt/whisper"]},{"group":"LLM","pages":["server/services/llm/anthropic","server/services/llm/aws","server/services/llm/azure","server/services/llm/cerebras","server/services/llm/deepseek","server/services/llm/fireworks","server/services/llm/gemini","server/services/llm/google-vertex","server/services/llm/grok","server/services/llm/groq","server/services/llm/nim","server/services/llm/ollama","server/services/llm/openai","server/services/llm/openpipe","server/services/llm/openrouter","server/services/llm/perplexity","server/services/llm/qwen","server/services/llm/sambanova","server/services/llm/together"]},{"group":"Text-to-Speech","pages":["server/services/tts/aws","server/services/tts/azure","server/services/tts/cartesia","server/services/tts/deepgram","server/services/tts/elevenlabs","server/services/tts/fish","server/services/tts/google","server/services/tts/groq","server/services/tts/lmnt","server/services/tts/minimax","server/services/tts/neuphonic","server/services/tts/riva","server/services/tts/openai","server/services/tts/piper","server/services/tts/playht","server/services/tts/rime","server/services/tts/sarvam","server/services/tts/xtts"]},{"group":"Speech-to-Speech","pages":["server/services/s2s/aws","server/services/s2s/gemini","server/services/s2s/openai"]},{"group":"Image Generation","pages":["server/services/image-generation/fal","server/services/image-generation/google-imagen","server/services/image-generation/openai"]},{"group":"Video","pages":["server/services/video/simli","server/services/video/tavus"]},{"group":"Memory","pages":["server/services/memory/mem0"]},{"group":"Vision","pages":["server/services/vision/moondream"]},{"group":"Analytics \u0026 Monitoring","pages":["server/services/analytics/sentry"]}]},{"group":"Utilities","pages":[{"group":"Advanced Frame Processors","pages":["server/utilities/frame/producer-consumer","server/utilities/user-idle-processor"]},{"group":"Audio Processing","pages":["server/utilities/audio/audio-buffer-processor","server/utilities/audio/koala-filter","server/utilities/audio/krisp-filter","server/utilities/audio/noisereduce-filter","server/utilities/audio/silero-vad-analyzer","server/utilities/audio/soundfile-mixer"]},{"group":"Frame Filters","pages":["server/utilities/filters/frame-filter","server/utilities/filters/function-filter","server/utilities/filters/identify-filter","server/utilities/filters/null-filter","server/utilities/filters/stt-mute","server/utilities/filters/wake-check-filter","server/utilities/filters/wake-notifier-filter"]},{"group":"Metrics and Telemetry","pages":["server/utilities/opentelemetry"]},{"group":"MCP","pages":["server/utilities/mcp/mcp"]},{"group":"Observers","pages":["server/utilities/observers/observer-pattern","server/utilities/observers/debug-observer","server/utilities/observers/llm-observer","server/utilities/observers/transcription-observer","server/utilities/observers/turn-tracking-observer"]},{"group":"Service Utilities","pages":["server/utilities/daily/rest-helpers"]},{"group":"Smart Turn Detection","pages":["server/utilities/smart-turn/smart-turn-overview","server/utilities/smart-turn/fal-smart-turn","server/utilities/smart-turn/local-coreml-smart-turn"]},{"group":"Task Handling and Monitoring","pages":["server/utilities/watchdog-timers"]},{"group":"Telephony","pages":["server/utilities/dtmf-aggregator"]},{"group":"Text Aggregators and Filters","pages":["server/utilities/text/markdown-text-filter","server/utilities/text/pattern-pair-aggregator"]},{"group":"User and Bot Transcriptions","pages":["server/utilities/transcript-processor"]},{"group":"User Interruptions","pages":["server/utilities/interruption-strategies"]}]},{"group":"Frameworks","pages":[{"group":"RTVI","pages":["server/frameworks/rtvi/introduction","server/frameworks/rtvi/rtvi-processor","server/frameworks/rtvi/rtvi-observer"]},"server/frameworks/flows/pipecat-flows"]},{"group":"Pipeline","pages":["server/pipeline/pipeline-params","server/pipeline/pipeline-task","server/pipeline/pipeline-idle-detection","server/pipeline/heartbeats","server/pipeline/parallel-pipeline"]},{"group":"","pages":["client/introduction","client/rtvi-standard","client/migration-guide"]},{"group":"Javascript SDK","pages":["client/js/introduction",{"group":"API Reference","pages":["client/js/api-reference/client-constructor","client/js/api-reference/client-methods","client/js/api-reference/callbacks","client/js/api-reference/messages","client/js/api-reference/errors"]},{"group":"Transport packages","pages":["client/js/transports/transport","client/js/transports/daily","client/js/transports/small-webrtc","client/js/transports/websocket","client/js/transports/gemini","client/js/transports/openai-webrtc"]}]},{"group":"React SDK","pages":["client/react/introduction",{"group":"API Reference","pages":["client/react/components","client/react/hooks"]}]},{"group":"React Native SDK","pages":["client/react-native/introduction","client/react-native/api-reference"]},{"group":"iOS SDK","pages":["client/ios/introduction","client/ios/api-reference",{"group":"Transport packages","pages":["client/ios/transports/daily","client/ios/transports/gemini-websocket","client/ios/transports/openai-webrtc"]}]},{"group":"Android SDK","pages":["client/android/introduction","client/android/api-reference",{"group":"Transport packages","pages":["client/android/transports/daily","client/android/transports/gemini-websocket","client/android/transports/openai-webrtc","client/android/transports/small-webrtc"]}]},{"group":"C++ SDK","pages":["client/c++/introduction","client/c++/transport"]}],"anchors":[{"name":"Community","url":"https://discord.gg/pipecat","icon":"discord","_id":"686c39bbc459dfd6f1155b17"},{"name":"GitHub","url":"https://github.com/pipecat-ai/pipecat","icon":"github","_id":"686c39bbc459dfd6f1155b18"},{"name":"Examples","url":"https://github.com/pipecat-ai/pipecat/tree/main/examples","icon":"code","_id":"686c39bbc459dfd6f1155b19"},{"name":"Changelog","url":"https://github.com/pipecat-ai/pipecat/blob/main/CHANGELOG.md","icon":"file","_id":"686c39bbc459dfd6f1155b1a"}],"tabs":[{"name":"Getting Started","url":"getting-started","_id":"686c39bbc459dfd6f1155b13"},{"name":"Guides","url":"guides","_id":"686c39bbc459dfd6f1155b14"},{"name":"Server APIs","url":"server","_id":"686c39bbc459dfd6f1155b15"},{"name":"Client SDKs","url":"client","_id":"686c39bbc459dfd6f1155b16"}],"footerSocials":{"x":"https://x.com/pipecat_ai","github":"https://github.com/pipecat-ai/pipecat","discord":"https://discord.gg/pipecat"}},"docsConfig":{"$schema":"https://mintlify.com/docs.json","theme":"mint","name":"Pipecat","colors":{"primary":"#4f46e5","light":"#a5b4fc","dark":"#312e81"},"favicon":"/favicon.svg","navigation":{"tabs":[{"tab":"Getting Started","groups":[{"group":"Get Started","pages":["getting-started/overview","getting-started/installation","getting-started/quickstart","getting-started/core-concepts","getting-started/next-steps"]}]},{"tab":"Guides","groups":[{"group":"","pages":["guides/introduction"]},{"group":"Fundamentals","pages":["guides/fundamentals/context-management","guides/fundamentals/custom-frame-processor","guides/fundamentals/detecting-user-idle","guides/fundamentals/end-pipeline","guides/fundamentals/function-calling","guides/fundamentals/user-input-muting","guides/fundamentals/recording-audio","guides/fundamentals/recording-transcripts"]},{"group":"Features","pages":["guides/features/gemini-multimodal-live","guides/features/metrics","guides/features/krisp","guides/features/openai-audio-models-and-apis","guides/features/pipecat-flows"]},{"group":"Telephony","pages":["guides/telephony/overview","guides/telephony/daily-webrtc","guides/telephony/twilio-daily-webrtc","guides/telephony/twilio-websockets","guides/telephony/dialout"]},{"group":"Deploying your bot","pages":["guides/deployment/overview","guides/deployment/pattern","guides/deployment/pipecat-cloud","guides/deployment/fly","guides/deployment/cerebrium","guides/deployment/modal"]}]},{"tab":"Server APIs","groups":[{"group":"","pages":["server/introduction"]},{"group":"API Reference","pages":["server/links/server-reference"]},{"group":"Services","pages":["server/services/supported-services",{"group":"Transport","pages":["server/services/transport/daily","server/services/transport/small-webrtc","server/services/transport/fastapi-websocket","server/services/transport/websocket-server"]},{"group":"Serializers","pages":["server/services/serializers/introduction","server/services/serializers/exotel","server/services/serializers/plivo","server/services/serializers/twilio","server/services/serializers/telnyx"]},{"group":"Speech-to-Text","pages":["server/services/stt/assemblyai","server/services/stt/aws","server/services/stt/azure","server/services/stt/cartesia","server/services/stt/deepgram","server/services/stt/fal","server/services/stt/gladia","server/services/stt/google","server/services/stt/groq","server/services/stt/riva","server/services/stt/openai","server/services/stt/sambanova","server/services/stt/speechmatics","server/services/stt/ultravox","server/services/stt/whisper"]},{"group":"LLM","pages":["server/services/llm/anthropic","server/services/llm/aws","server/services/llm/azure","server/services/llm/cerebras","server/services/llm/deepseek","server/services/llm/fireworks","server/services/llm/gemini","server/services/llm/google-vertex","server/services/llm/grok","server/services/llm/groq","server/services/llm/nim","server/services/llm/ollama","server/services/llm/openai","server/services/llm/openpipe","server/services/llm/openrouter","server/services/llm/perplexity","server/services/llm/qwen","server/services/llm/sambanova","server/services/llm/together"]},{"group":"Text-to-Speech","pages":["server/services/tts/aws","server/services/tts/azure","server/services/tts/cartesia","server/services/tts/deepgram","server/services/tts/elevenlabs","server/services/tts/fish","server/services/tts/google","server/services/tts/groq","server/services/tts/lmnt","server/services/tts/minimax","server/services/tts/neuphonic","server/services/tts/riva","server/services/tts/openai","server/services/tts/piper","server/services/tts/playht","server/services/tts/rime","server/services/tts/sarvam","server/services/tts/xtts"]},{"group":"Speech-to-Speech","pages":["server/services/s2s/aws","server/services/s2s/gemini","server/services/s2s/openai"]},{"group":"Image Generation","pages":["server/services/image-generation/fal","server/services/image-generation/google-imagen","server/services/image-generation/openai"]},{"group":"Video","pages":["server/services/video/simli","server/services/video/tavus"]},{"group":"Memory","pages":["server/services/memory/mem0"]},{"group":"Vision","pages":["server/services/vision/moondream"]},{"group":"Analytics \u0026 Monitoring","pages":["server/services/analytics/sentry"]}]},{"group":"Utilities","pages":[{"group":"Advanced Frame Processors","pages":["server/utilities/frame/producer-consumer","server/utilities/user-idle-processor"]},{"group":"Audio Processing","pages":["server/utilities/audio/audio-buffer-processor","server/utilities/audio/koala-filter","server/utilities/audio/krisp-filter","server/utilities/audio/noisereduce-filter","server/utilities/audio/silero-vad-analyzer","server/utilities/audio/soundfile-mixer"]},{"group":"Frame Filters","pages":["server/utilities/filters/frame-filter","server/utilities/filters/function-filter","server/utilities/filters/identify-filter","server/utilities/filters/null-filter","server/utilities/filters/stt-mute","server/utilities/filters/wake-check-filter","server/utilities/filters/wake-notifier-filter"]},{"group":"Metrics and Telemetry","pages":["server/utilities/opentelemetry"]},{"group":"MCP","pages":["server/utilities/mcp/mcp"]},{"group":"Observers","pages":["server/utilities/observers/observer-pattern","server/utilities/observers/debug-observer","server/utilities/observers/llm-observer","server/utilities/observers/transcription-observer","server/utilities/observers/turn-tracking-observer"]},{"group":"Service Utilities","pages":["server/utilities/daily/rest-helpers"]},{"group":"Smart Turn Detection","pages":["server/utilities/smart-turn/smart-turn-overview","server/utilities/smart-turn/fal-smart-turn","server/utilities/smart-turn/local-coreml-smart-turn"]},{"group":"Task Handling and Monitoring","pages":["server/utilities/watchdog-timers"]},{"group":"Telephony","pages":["server/utilities/dtmf-aggregator"]},{"group":"Text Aggregators and Filters","pages":["server/utilities/text/markdown-text-filter","server/utilities/text/pattern-pair-aggregator"]},{"group":"User and Bot Transcriptions","pages":["server/utilities/transcript-processor"]},{"group":"User Interruptions","pages":["server/utilities/interruption-strategies"]}]},{"group":"Frameworks","pages":[{"group":"RTVI","pages":["server/frameworks/rtvi/introduction","server/frameworks/rtvi/rtvi-processor","server/frameworks/rtvi/rtvi-observer"]},"server/frameworks/flows/pipecat-flows"]},{"group":"Pipeline","pages":["server/pipeline/pipeline-params","server/pipeline/pipeline-task","server/pipeline/pipeline-idle-detection","server/pipeline/heartbeats","server/pipeline/parallel-pipeline"]}]},{"tab":"Client SDKs","groups":[{"group":"","pages":["client/introduction","client/rtvi-standard","client/migration-guide"]},{"group":"Javascript SDK","pages":["client/js/introduction",{"group":"API Reference","pages":["client/js/api-reference/client-constructor","client/js/api-reference/client-methods","client/js/api-reference/callbacks","client/js/api-reference/messages","client/js/api-reference/errors"]},{"group":"Transport packages","pages":["client/js/transports/transport","client/js/transports/daily","client/js/transports/small-webrtc","client/js/transports/websocket","client/js/transports/gemini","client/js/transports/openai-webrtc"]}]},{"group":"React SDK","pages":["client/react/introduction",{"group":"API Reference","pages":["client/react/components","client/react/hooks"]}]},{"group":"React Native SDK","pages":["client/react-native/introduction","client/react-native/api-reference"]},{"group":"iOS SDK","pages":["client/ios/introduction","client/ios/api-reference",{"group":"Transport packages","pages":["client/ios/transports/daily","client/ios/transports/gemini-websocket","client/ios/transports/openai-webrtc"]}]},{"group":"Android SDK","pages":["client/android/introduction","client/android/api-reference",{"group":"Transport packages","pages":["client/android/transports/daily","client/android/transports/gemini-websocket","client/android/transports/openai-webrtc","client/android/transports/small-webrtc"]}]},{"group":"C++ SDK","pages":["client/c++/introduction","client/c++/transport"]}]}],"global":{"anchors":[{"anchor":"Community","href":"https://discord.gg/pipecat","icon":"discord"},{"anchor":"GitHub","href":"https://github.com/pipecat-ai/pipecat","icon":"github"},{"anchor":"Examples","href":"https://github.com/pipecat-ai/pipecat/tree/main/examples","icon":"code"},{"anchor":"Changelog","href":"https://github.com/pipecat-ai/pipecat/blob/main/CHANGELOG.md","icon":"file"}]}},"logo":{"light":"https://mintlify.s3.us-west-1.amazonaws.com/daily/logo/light.svg","dark":"https://mintlify.s3.us-west-1.amazonaws.com/daily/logo/dark.svg"},"navbar":{"links":[{"label":"Join the Discord","href":"https://discord.gg/pipecat"}]},"seo":{"metatags":{"og:site_name":"Pipecat","og:description":"Learn about Pipecat, the open source framework for building voice and multimodal AI agents.","og:url":"https://docs.pipecat.ai/","og:locale":"en_US","og:logo":"/logo/pipecat-logo.svg","article:publisher":"Pipecat AI","twitter:description":"Learn about Pipecat, the open source framework for building voice and multimodal AI agents.","twitter:url":"https://docs.pipecat.ai/","twitter:site":"@pipecat_ai","og:image:width":"1200","og:image:height":"675"},"indexing":"navigable"},"footer":{"socials":{"x":"https://x.com/pipecat_ai","github":"https://github.com/pipecat-ai/pipecat","discord":"https://discord.gg/pipecat"}},"styling":{"codeblocks":"system"}},"apiReferenceData":{}},"favicons":{"icons":[{"rel":"apple-touch-icon","sizes":"180x180","href":"https://mintlify.s3-us-west-1.amazonaws.com/daily/_generated/favicon/apple-touch-icon.png?v=3","type":"image/png"},{"rel":"icon","sizes":"32x32","href":"https://mintlify.s3-us-west-1.amazonaws.com/daily/_generated/favicon/favicon-32x32.png?v=3","type":"image/png"},{"rel":"icon","sizes":"16x16","href":"https://mintlify.s3-us-west-1.amazonaws.com/daily/_generated/favicon/favicon-16x16.png?v=3","type":"image/png"},{"rel":"shortcut icon","href":"https://mintlify.s3-us-west-1.amazonaws.com/daily/_generated/favicon/favicon.ico?v=3","type":"image/x-icon"}],"browserconfig":"https://mintlify.s3-us-west-1.amazonaws.com/daily/_generated/favicon/browserconfig.xml?v=3"},"subdomain":"docs.pipecat.ai","actualSubdomain":"daily","internalAnalyticsWriteKey":"phc_TXdpocbGVeZVm5VJmAsHTMrCofBQu3e0kN8HGMNGTVW","trieve":{"datasetId":"9d34b74e-d3be-4364-9339-1e5ced1d88bc","chatEnabled":true},"shouldIndex":true,"org":{"plan":"startup","createdAt":"2024-05-30T15:35:56.346Z"},"cssFiles":[],"jsFiles":[],"mdxSourceWithNoJs":{"compiledSource":"\"use strict\";\nconst {Fragment: _Fragment, jsx: _jsx, jsxs: _jsxs} = arguments[0];\nconst {useMDXComponents: _provideComponents} = arguments[0];\nfunction _createMdxContent(props) {\n  const _components = {\n    li: \"li\",\n    ol: \"ol\",\n    p: \"p\",\n    ul: \"ul\",\n    ..._provideComponents(),\n    ...props.components\n  }, {Card, CardGroup, Heading, Note, Step, Steps, Tab, Tabs} = _components;\n  if (!Card) _missingMdxReference(\"Card\", true);\n  if (!CardGroup) _missingMdxReference(\"CardGroup\", true);\n  if (!Heading) _missingMdxReference(\"Heading\", true);\n  if (!Note) _missingMdxReference(\"Note\", true);\n  if (!Step) _missingMdxReference(\"Step\", true);\n  if (!Steps) _missingMdxReference(\"Steps\", true);\n  if (!Tab) _missingMdxReference(\"Tab\", true);\n  if (!Tabs) _missingMdxReference(\"Tabs\", true);\n  return _jsxs(_Fragment, {\n    children: [_jsx(_components.p, {\n      children: \"Pipecat is an open source Python framework that handles the complex orchestration of AI services, network transport, audio processing, and multimodal interactions. “Multimodal” means you can use any combination of audio, video, images, and/or text in your interactions. And “real-time” means that things are happening quickly enough that it feels conversational—a “back-and-forth” with a bot, not submitting a query and waiting for results.\"\n    }), \"\\n\", _jsx(Heading, {\n      level: \"2\",\n      id: \"what-you-can-build\",\n      isAtRootLevel: \"true\",\n      children: \"What You Can Build\"\n    }), \"\\n\", _jsxs(CardGroup, {\n      cols: 3,\n      children: [_jsx(Card, {\n        title: \"Voice Assistants\",\n        icon: \"microphone\",\n        href: \"https://github.com/pipecat-ai/pipecat/tree/main/examples/simple-chatbot\",\n        children: _jsx(_components.p, {\n          children: \"Natural, real-time conversations with AI using speech recognition and\\nsynthesis\"\n        })\n      }), _jsx(Card, {\n        title: \"Interactive Agents\",\n        icon: \"robot\",\n        href: \"https://github.com/pipecat-ai/pipecat/tree/main/examples/studypal\",\n        children: _jsx(_components.p, {\n          children: \"Personal coaches and meeting assistants that can understand context and\\nprovide guidance\"\n        })\n      }), _jsx(Card, {\n        title: \"Multimodal Apps\",\n        icon: \"layer-group\",\n        href: \"https://github.com/pipecat-ai/pipecat/tree/main/examples/moondream-chatbot\",\n        children: _jsx(_components.p, {\n          children: \"Applications that combine voice, video, images, and text for rich interactions\"\n        })\n      }), _jsx(Card, {\n        title: \"Creative Tools\",\n        icon: \"wand-magic-sparkles\",\n        href: \"https://github.com/pipecat-ai/pipecat/tree/main/examples/storytelling-chatbot\",\n        children: _jsx(_components.p, {\n          children: \"Storytelling experiences and social companions that engage users\"\n        })\n      }), _jsx(Card, {\n        title: \"Business Solutions\",\n        icon: \"building\",\n        href: \"https://github.com/pipecat-ai/pipecat/tree/main/examples/patient-intake\",\n        children: _jsx(_components.p, {\n          children: \"Customer intake flows and support bots for automated business processes\"\n        })\n      }), _jsx(Card, {\n        title: \"Complex Flows\",\n        icon: \"diagram-project\",\n        href: \"https://github.com/pipecat-ai/pipecat-flows\",\n        children: _jsx(_components.p, {\n          children: \"Structured conversations using Pipecat Flows for managing complex\\ninteractions\"\n        })\n      })]\n    }), \"\\n\", _jsx(Heading, {\n      level: \"2\",\n      id: \"how-it-works\",\n      isAtRootLevel: \"true\",\n      children: \"How It Works\"\n    }), \"\\n\", _jsx(_components.p, {\n      children: \"The flow of interactions in a Pipecat application is typically straightforward:\"\n    }), \"\\n\", _jsxs(_components.ol, {\n      children: [\"\\n\", _jsx(_components.li, {\n        children: \"The bot says something\"\n      }), \"\\n\", _jsx(_components.li, {\n        children: \"The user says something\"\n      }), \"\\n\", _jsx(_components.li, {\n        children: \"The bot says something\"\n      }), \"\\n\", _jsx(_components.li, {\n        children: \"The user says something\"\n      }), \"\\n\"]\n    }), \"\\n\", _jsx(_components.p, {\n      children: \"This continues until the conversation naturally ends. While this flow seems simple, making it feel natural requires sophisticated real-time processing.\"\n    }), \"\\n\", _jsx(Heading, {\n      level: \"3\",\n      id: \"real-time-processing\",\n      isAtRootLevel: \"true\",\n      children: \"Real-time Processing\"\n    }), \"\\n\", _jsx(_components.p, {\n      children: \"Pipecat’s pipeline architecture handles both simple voice interactions and complex multimodal processing. Let’s look at how data flows through the system:\"\n    }), \"\\n\", _jsxs(Tabs, {\n      children: [_jsx(Tab, {\n        title: \"Voice app\",\n        children: _jsxs(Steps, {\n          children: [_jsx(Step, {\n            title: \"Send Audio\",\n            children: _jsx(_components.p, {\n              children: \"Transmit and capture streamed audio from the user\"\n            })\n          }), _jsx(Step, {\n            title: \"Transcribe Speech\",\n            children: _jsx(_components.p, {\n              children: \"Convert speech to text as the user is talking\"\n            })\n          }), _jsx(Step, {\n            title: \"Process with LLM\",\n            children: _jsx(_components.p, {\n              children: \"Generate responses using a large language model\"\n            })\n          }), _jsx(Step, {\n            title: \"Convert to Speech\",\n            children: _jsx(_components.p, {\n              children: \"Transform text responses into natural speech\"\n            })\n          }), _jsx(Step, {\n            title: \"Play Audio\",\n            children: \"Stream the audio response back to the user\"\n          })]\n        })\n      }), _jsx(Tab, {\n        title: \"Multimodal app\",\n        children: _jsxs(Steps, {\n          children: [_jsx(Step, {\n            title: \"Send Audio and Video\",\n            children: _jsx(_components.p, {\n              children: \"Transmit and capture audio, video, and image inputs simultaneously\"\n            })\n          }), _jsx(Step, {\n            title: \"Process Streams\",\n            children: \"Handle multiple input streams in parallel\"\n          }), _jsx(Step, {\n            title: \"Model Processing\",\n            children: _jsx(_components.p, {\n              children: \"Send combined inputs to multimodal models (like GPT-4V)\"\n            })\n          }), _jsx(Step, {\n            title: \"Generate Outputs\",\n            children: _jsx(_components.p, {\n              children: \"Create various outputs (text, images, audio, etc.)\"\n            })\n          }), _jsx(Step, {\n            title: \"Coordinate Presentation\",\n            children: _jsx(_components.p, {\n              children: \"Synchronize and present multiple output types\"\n            })\n          })]\n        })\n      })]\n    }), \"\\n\", _jsx(_components.p, {\n      children: \"In both cases, Pipecat:\"\n    }), \"\\n\", _jsxs(_components.ul, {\n      children: [\"\\n\", _jsx(_components.li, {\n        children: \"Processes responses as they stream in\"\n      }), \"\\n\", _jsx(_components.li, {\n        children: \"Handles multiple input/output modalities concurrently\"\n      }), \"\\n\", _jsx(_components.li, {\n        children: \"Manages resource allocation and synchronization\"\n      }), \"\\n\", _jsx(_components.li, {\n        children: \"Coordinates parallel processing tasks\"\n      }), \"\\n\"]\n    }), \"\\n\", _jsx(_components.p, {\n      children: \"This architecture creates fluid, natural interactions without noticeable delays, whether you’re building a simple voice assistant or a complex multimodal application. Pipecat’s pipeline architecture is particularly valuable for managing the complexity of real-time, multimodal interactions, ensuring smooth data flow and proper synchronization regardless of the input/output types involved.\"\n    }), \"\\n\", _jsx(Note, {\n      children: _jsx(_components.p, {\n        children: \"Pipecat handles all this complexity for you, letting you focus on building\\nyour application rather than managing the underlying infrastructure.\"\n      })\n    }), \"\\n\", _jsx(Heading, {\n      level: \"2\",\n      id: \"next-steps\",\n      isAtRootLevel: \"true\",\n      children: \"Next Steps\"\n    }), \"\\n\", _jsx(_components.p, {\n      children: \"Ready to build your first Pipecat application?\"\n    }), \"\\n\", _jsxs(CardGroup, {\n      cols: 2,\n      children: [_jsx(Card, {\n        title: \"Installation \u0026 Setup\",\n        icon: \"wrench\",\n        href: \"/getting-started/installation\",\n        children: _jsx(_components.p, {\n          children: \"Prepare your environment and install required dependencies\"\n        })\n      }), _jsx(Card, {\n        title: \"Quickstart\",\n        icon: \"rocket\",\n        href: \"/getting-started/quickstart\",\n        children: _jsx(_components.p, {\n          children: \"Build and run your first Pipecat application\"\n        })\n      }), _jsx(Card, {\n        title: \"Core Concepts\",\n        icon: \"book\",\n        href: \"/getting-started/core-concepts\",\n        children: _jsx(_components.p, {\n          children: \"Learn about pipelines, frames, and real-time processing\"\n        })\n      }), _jsx(Card, {\n        title: \"Use Cases\",\n        icon: \"lightbulb\",\n        href: \"/getting-started/use-cases\",\n        children: _jsx(_components.p, {\n          children: \"Explore example implementations and patterns\"\n        })\n      })]\n    }), \"\\n\", _jsx(Heading, {\n      level: \"2\",\n      id: \"join-our-community\",\n      isAtRootLevel: \"true\",\n      children: \"Join Our Community\"\n    }), \"\\n\", _jsx(Card, {\n      title: \"Discord Community\",\n      icon: \"discord\",\n      iconType: \"duotone\",\n      href: \"https://discord.gg/pipecat\",\n      children: _jsx(_components.p, {\n        children: \"Connect with other developers, share your projects, and get support from the\\nPipecat team.\"\n      })\n    })]\n  });\n}\nfunction MDXContent(props = {}) {\n  const {wrapper: MDXLayout} = {\n    ..._provideComponents(),\n    ...props.components\n  };\n  return MDXLayout ? _jsx(MDXLayout, {\n    ...props,\n    children: _jsx(_createMdxContent, {\n      ...props\n    })\n  }) : _createMdxContent(props);\n}\nreturn {\n  default: MDXContent\n};\nfunction _missingMdxReference(id, component) {\n  throw new Error(\"Expected \" + (component ? \"component\" : \"object\") + \" `\" + id + \"` to be defined: you likely forgot to import, pass, or provide it.\");\n}\n","frontmatter":{},"scope":{"config":{"$schema":"https://mintlify.com/docs.json","theme":"mint","name":"Pipecat","colors":{"primary":"#4f46e5","light":"#a5b4fc","dark":"#312e81"},"favicon":"/favicon.svg","navigation":{"tabs":[{"tab":"Getting Started","groups":[{"group":"Get Started","pages":["getting-started/overview","getting-started/installation","getting-started/quickstart","getting-started/core-concepts","getting-started/next-steps"]}]},{"tab":"Guides","groups":[{"group":"","pages":["guides/introduction"]},{"group":"Fundamentals","pages":["guides/fundamentals/context-management","guides/fundamentals/custom-frame-processor","guides/fundamentals/detecting-user-idle","guides/fundamentals/end-pipeline","guides/fundamentals/function-calling","guides/fundamentals/user-input-muting","guides/fundamentals/recording-audio","guides/fundamentals/recording-transcripts"]},{"group":"Features","pages":["guides/features/gemini-multimodal-live","guides/features/metrics","guides/features/krisp","guides/features/openai-audio-models-and-apis","guides/features/pipecat-flows"]},{"group":"Telephony","pages":["guides/telephony/overview","guides/telephony/daily-webrtc","guides/telephony/twilio-daily-webrtc","guides/telephony/twilio-websockets","guides/telephony/dialout"]},{"group":"Deploying your bot","pages":["guides/deployment/overview","guides/deployment/pattern","guides/deployment/pipecat-cloud","guides/deployment/fly","guides/deployment/cerebrium","guides/deployment/modal"]}]},{"tab":"Server APIs","groups":[{"group":"","pages":["server/introduction"]},{"group":"API Reference","pages":["server/links/server-reference"]},{"group":"Services","pages":["server/services/supported-services",{"group":"Transport","pages":["server/services/transport/daily","server/services/transport/small-webrtc","server/services/transport/fastapi-websocket","server/services/transport/websocket-server"]},{"group":"Serializers","pages":["server/services/serializers/introduction","server/services/serializers/exotel","server/services/serializers/plivo","server/services/serializers/twilio","server/services/serializers/telnyx"]},{"group":"Speech-to-Text","pages":["server/services/stt/assemblyai","server/services/stt/aws","server/services/stt/azure","server/services/stt/cartesia","server/services/stt/deepgram","server/services/stt/fal","server/services/stt/gladia","server/services/stt/google","server/services/stt/groq","server/services/stt/riva","server/services/stt/openai","server/services/stt/sambanova","server/services/stt/speechmatics","server/services/stt/ultravox","server/services/stt/whisper"]},{"group":"LLM","pages":["server/services/llm/anthropic","server/services/llm/aws","server/services/llm/azure","server/services/llm/cerebras","server/services/llm/deepseek","server/services/llm/fireworks","server/services/llm/gemini","server/services/llm/google-vertex","server/services/llm/grok","server/services/llm/groq","server/services/llm/nim","server/services/llm/ollama","server/services/llm/openai","server/services/llm/openpipe","server/services/llm/openrouter","server/services/llm/perplexity","server/services/llm/qwen","server/services/llm/sambanova","server/services/llm/together"]},{"group":"Text-to-Speech","pages":["server/services/tts/aws","server/services/tts/azure","server/services/tts/cartesia","server/services/tts/deepgram","server/services/tts/elevenlabs","server/services/tts/fish","server/services/tts/google","server/services/tts/groq","server/services/tts/lmnt","server/services/tts/minimax","server/services/tts/neuphonic","server/services/tts/riva","server/services/tts/openai","server/services/tts/piper","server/services/tts/playht","server/services/tts/rime","server/services/tts/sarvam","server/services/tts/xtts"]},{"group":"Speech-to-Speech","pages":["server/services/s2s/aws","server/services/s2s/gemini","server/services/s2s/openai"]},{"group":"Image Generation","pages":["server/services/image-generation/fal","server/services/image-generation/google-imagen","server/services/image-generation/openai"]},{"group":"Video","pages":["server/services/video/simli","server/services/video/tavus"]},{"group":"Memory","pages":["server/services/memory/mem0"]},{"group":"Vision","pages":["server/services/vision/moondream"]},{"group":"Analytics \u0026 Monitoring","pages":["server/services/analytics/sentry"]}]},{"group":"Utilities","pages":[{"group":"Advanced Frame Processors","pages":["server/utilities/frame/producer-consumer","server/utilities/user-idle-processor"]},{"group":"Audio Processing","pages":["server/utilities/audio/audio-buffer-processor","server/utilities/audio/koala-filter","server/utilities/audio/krisp-filter","server/utilities/audio/noisereduce-filter","server/utilities/audio/silero-vad-analyzer","server/utilities/audio/soundfile-mixer"]},{"group":"Frame Filters","pages":["server/utilities/filters/frame-filter","server/utilities/filters/function-filter","server/utilities/filters/identify-filter","server/utilities/filters/null-filter","server/utilities/filters/stt-mute","server/utilities/filters/wake-check-filter","server/utilities/filters/wake-notifier-filter"]},{"group":"Metrics and Telemetry","pages":["server/utilities/opentelemetry"]},{"group":"MCP","pages":["server/utilities/mcp/mcp"]},{"group":"Observers","pages":["server/utilities/observers/observer-pattern","server/utilities/observers/debug-observer","server/utilities/observers/llm-observer","server/utilities/observers/transcription-observer","server/utilities/observers/turn-tracking-observer"]},{"group":"Service Utilities","pages":["server/utilities/daily/rest-helpers"]},{"group":"Smart Turn Detection","pages":["server/utilities/smart-turn/smart-turn-overview","server/utilities/smart-turn/fal-smart-turn","server/utilities/smart-turn/local-coreml-smart-turn"]},{"group":"Task Handling and Monitoring","pages":["server/utilities/watchdog-timers"]},{"group":"Telephony","pages":["server/utilities/dtmf-aggregator"]},{"group":"Text Aggregators and Filters","pages":["server/utilities/text/markdown-text-filter","server/utilities/text/pattern-pair-aggregator"]},{"group":"User and Bot Transcriptions","pages":["server/utilities/transcript-processor"]},{"group":"User Interruptions","pages":["server/utilities/interruption-strategies"]}]},{"group":"Frameworks","pages":[{"group":"RTVI","pages":["server/frameworks/rtvi/introduction","server/frameworks/rtvi/rtvi-processor","server/frameworks/rtvi/rtvi-observer"]},"server/frameworks/flows/pipecat-flows"]},{"group":"Pipeline","pages":["server/pipeline/pipeline-params","server/pipeline/pipeline-task","server/pipeline/pipeline-idle-detection","server/pipeline/heartbeats","server/pipeline/parallel-pipeline"]}]},{"tab":"Client SDKs","groups":[{"group":"","pages":["client/introduction","client/rtvi-standard","client/migration-guide"]},{"group":"Javascript SDK","pages":["client/js/introduction",{"group":"API Reference","pages":["client/js/api-reference/client-constructor","client/js/api-reference/client-methods","client/js/api-reference/callbacks","client/js/api-reference/messages","client/js/api-reference/errors"]},{"group":"Transport packages","pages":["client/js/transports/transport","client/js/transports/daily","client/js/transports/small-webrtc","client/js/transports/websocket","client/js/transports/gemini","client/js/transports/openai-webrtc"]}]},{"group":"React SDK","pages":["client/react/introduction",{"group":"API Reference","pages":["client/react/components","client/react/hooks"]}]},{"group":"React Native SDK","pages":["client/react-native/introduction","client/react-native/api-reference"]},{"group":"iOS SDK","pages":["client/ios/introduction","client/ios/api-reference",{"group":"Transport packages","pages":["client/ios/transports/daily","client/ios/transports/gemini-websocket","client/ios/transports/openai-webrtc"]}]},{"group":"Android SDK","pages":["client/android/introduction","client/android/api-reference",{"group":"Transport packages","pages":["client/android/transports/daily","client/android/transports/gemini-websocket","client/android/transports/openai-webrtc","client/android/transports/small-webrtc"]}]},{"group":"C++ SDK","pages":["client/c++/introduction","client/c++/transport"]}]}],"global":{"anchors":[{"anchor":"Community","href":"https://discord.gg/pipecat","icon":"discord"},{"anchor":"GitHub","href":"https://github.com/pipecat-ai/pipecat","icon":"github"},{"anchor":"Examples","href":"https://github.com/pipecat-ai/pipecat/tree/main/examples","icon":"code"},{"anchor":"Changelog","href":"https://github.com/pipecat-ai/pipecat/blob/main/CHANGELOG.md","icon":"file"}]}},"logo":{"light":"https://mintlify.s3.us-west-1.amazonaws.com/daily/logo/light.svg","dark":"https://mintlify.s3.us-west-1.amazonaws.com/daily/logo/dark.svg"},"navbar":{"links":[{"label":"Join the Discord","href":"https://discord.gg/pipecat"}]},"seo":{"metatags":{"og:site_name":"Pipecat","og:description":"Learn about Pipecat, the open source framework for building voice and multimodal AI agents.","og:url":"https://docs.pipecat.ai/","og:locale":"en_US","og:logo":"/logo/pipecat-logo.svg","article:publisher":"Pipecat AI","twitter:description":"Learn about Pipecat, the open source framework for building voice and multimodal AI agents.","twitter:url":"https://docs.pipecat.ai/","twitter:site":"@pipecat_ai","og:image:width":"1200","og:image:height":"675"},"indexing":"navigable"},"footer":{"socials":{"x":"https://x.com/pipecat_ai","github":"https://github.com/pipecat-ai/pipecat","discord":"https://discord.gg/pipecat"}}},"pageMetadata":{"title":"Overview","description":"Pipecat is a framework for building voice-enabled, real-time, multimodal AI applications","href":"/getting-started/overview"}}},"entitlements":{"AI_CHAT":{"status":"ENABLED"}},"gitSource":{"type":"github","owner":"pipecat-ai","repo":"docs","deployBranch":"main","contentDirectory":"","isPrivate":false},"banner":null,"buildId":"686c39af4aa8731b39db955a:success","slug":"getting-started/overview"},"__N_SSG":true},"page":"/_sites/[subdomain]/[[...slug]]","query":{"subdomain":"docs.pipecat.ai","slug":["getting-started","overview"]},"buildId":"RPzUnhb57G2BlnT2NDDZW","assetPrefix":"/mintlify-assets","isFallback":false,"isExperimentalCompile":false,"gsp":true,"scriptLoader":[]}</script></html>
\ No newline at end of file
diff --git a/ios_introduction_70040911.txt b/ios_introduction_70040911.txt
new file mode 100644
index 0000000000000000000000000000000000000000..c75ce75f0fd99393c8c3c0c72d074a121a9c8075
--- /dev/null
+++ b/ios_introduction_70040911.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/client/ios/introduction
+Title: SDK Introduction - Pipecat
+==================================================
+
+SDK Introduction - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation iOS SDK SDK Introduction Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Client SDKs The RTVI Standard RTVIClient Migration Guide Javascript SDK SDK Introduction API Reference Transport packages React SDK SDK Introduction API Reference React Native SDK SDK Introduction API Reference iOS SDK SDK Introduction API Reference Transport packages Android SDK SDK Introduction API Reference Transport packages C++ SDK SDK Introduction Daily WebRTC Transport The Pipecat iOS SDK provides a Swift implementation for building voice and multimodal AI applications on iOS. It handles: Real-time audio streaming Bot communication and state management Media device handling Configuration management Event handling ​ Installation Add the SDK to your project using Swift Package Manager: Copy Ask AI // Core SDK . package ( url : "https://github.com/pipecat-ai/pipecat-client-ios.git" , from : "0.3.0" ), // Daily transport implementation . package ( url : "https://github.com/pipecat-ai/pipecat-client-ios-daily.git" , from : "0.3.0" ), Then add the dependencies to your target: Copy Ask AI . target ( name : "YourApp" , dependencies : [ . product ( name : "PipecatClientIOS" , package : "pipecat-client-ios" ) . product ( name : "PipecatClientIOSDaily" , package : "pipecat-client-ios-daily" ) ]), ​ Example Here’s a simple example using Daily as the transport layer: Copy Ask AI import PipecatClientIOS import PipecatClientIOSDaily let clientConfig = [ ServiceConfig ( service : "llm" , options : [ Option ( name : "model" , value : . string ( "meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo" )), Option ( name : "messages" , value : . array ([ . object ([ "role" : . string ( "system" ), "content" : . string ( "You are a helpful assistant." ) ]) ])) ] ), ServiceConfig ( service : "tts" , options : [ Option ( name : "voice" , value : . string ( "79a125e8-cd45-4c13-8a67-188112f4dd22" )) ] ) ] let options = RTVIClientOptions. init ( enableMic : true , params : RTVIClientParams ( baseUrl : $PIPECAT_API_URL, config : clientConfig ) ) let client = RTVIClient. init ( transport : DailyTransport. init ( options : configOptions), options : configOptions ) try await client. start () ​ Documentation API Reference Complete SDK API documentation Source Pipecat Client iOS Demo Simple Chatbot Demo Daily Transport WebRTC implementation using Daily API Reference API Reference On this page Installation Example Documentation Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/llm_anthropic_11639834.txt b/llm_anthropic_11639834.txt
new file mode 100644
index 0000000000000000000000000000000000000000..990351c9c49e359f3f71d7db8e5b31f107ece3c9
--- /dev/null
+++ b/llm_anthropic_11639834.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/llm/anthropic#metrics
+Title: Anthropic - Pipecat
+==================================================
+
+Anthropic - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation LLM Anthropic Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Anthropic AWS Bedrock Azure Cerebras DeepSeek Fireworks AI Google Gemini Google Vertex AI Grok Groq NVIDIA NIM Ollama OpenAI OpenPipe OpenRouter Perplexity Qwen SambaNova Together AI Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview AnthropicLLMService provides integration with Anthropic’s Claude models, supporting streaming responses, function calling, and prompt caching with specialized context handling for Anthropic’s message format. API Reference Complete API documentation and method details Anthropic Docs Official Anthropic API documentation and features Example Code Working example with function calling ​ Installation To use Anthropic services, install the required dependency: Copy Ask AI pip install "pipecat-ai[anthropic]" You’ll also need to set up your Anthropic API key as an environment variable: ANTHROPIC_API_KEY . Get your API key from Anthropic Console . ​ Frames ​ Input OpenAILLMContextFrame - Conversation context and history LLMMessagesFrame - Direct message list VisionImageRawFrame - Images for vision processing LLMUpdateSettingsFrame - Runtime parameter updates LLMEnablePromptCachingFrame - Toggle prompt caching ​ Output LLMFullResponseStartFrame / LLMFullResponseEndFrame - Response boundaries LLMTextFrame - Streamed completion chunks FunctionCallInProgressFrame / FunctionCallResultFrame - Function call lifecycle ErrorFrame - API or processing errors ​ Function Calling Function Calling Guide Learn how to implement function calling with standardized schemas, register handlers, manage context properly, and control execution flow in your conversational AI applications. ​ Context Management Context Management Guide Learn how to manage conversation context, handle message history, and integrate context aggregators for consistent conversational experiences. ​ Usage Example Copy Ask AI import os from pipecat.services.anthropic.llm import AnthropicLLMService from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext from pipecat.adapters.schemas.function_schema import FunctionSchema from pipecat.adapters.schemas.tools_schema import ToolsSchema # Configure the service llm = AnthropicLLMService( api_key = os.getenv( "ANTHROPIC_API_KEY" ), model = "claude-sonnet-4-20250514" , params = AnthropicLLMService.InputParams( temperature = 0.7 , enable_prompt_caching_beta = True ) ) # Define function for tool calling weather_function = FunctionSchema( name = "get_weather" , description = "Get current weather information" , properties = { "location" : { "type" : "string" , "description" : "City and state, e.g. San Francisco, CA" } }, required = [ "location" ] ) tools = ToolsSchema( standard_tools = [weather_function]) # Create context with system message context = OpenAILLMContext( messages = [{ "role" : "user" , "content" : "What's the weather like?" }], tools = tools ) # Create context aggregators context_aggregator = llm.create_context_aggregator(context) # Register function handler async def get_weather ( params ): location = params.arguments[ "location" ] await params.result_callback( f "Weather in { location } : 72°F and sunny" ) llm.register_function( "get_weather" , get_weather) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, context_aggregator.user(), # Handles user messages llm, # Processes with Anthropic tts, transport.output(), context_aggregator.assistant() # Captures responses ]) ​ Metrics The service provides: Time to First Byte (TTFB) - Latency from request to first response token Processing Duration - Total request processing time Token Usage - Prompt tokens, completion tokens, and total usage Cache Metrics - Cache creation and read token usage Enable with: Copy Ask AI task = PipelineTask( pipeline, params = PipelineParams( enable_metrics = True , enable_usage_metrics = True ) ) ​ Additional Notes Streaming Responses : All responses are streamed for low latency Context Persistence : Use context aggregators to maintain conversation history Error Handling : Automatic retry logic for rate limits and transient errors Message Format : Automatically converts between OpenAI and Anthropic message formats Prompt Caching : Reduces costs and latency for repeated context patterns Whisper AWS Bedrock On this page Overview Installation Frames Input Output Function Calling Context Management Usage Example Metrics Additional Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/llm_anthropic_23126ef3.txt b/llm_anthropic_23126ef3.txt
new file mode 100644
index 0000000000000000000000000000000000000000..404cbe125d352799d63e9b8b8b2bbd02637c1cb4
--- /dev/null
+++ b/llm_anthropic_23126ef3.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/llm/anthropic#frames
+Title: Anthropic - Pipecat
+==================================================
+
+Anthropic - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation LLM Anthropic Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Anthropic AWS Bedrock Azure Cerebras DeepSeek Fireworks AI Google Gemini Google Vertex AI Grok Groq NVIDIA NIM Ollama OpenAI OpenPipe OpenRouter Perplexity Qwen SambaNova Together AI Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview AnthropicLLMService provides integration with Anthropic’s Claude models, supporting streaming responses, function calling, and prompt caching with specialized context handling for Anthropic’s message format. API Reference Complete API documentation and method details Anthropic Docs Official Anthropic API documentation and features Example Code Working example with function calling ​ Installation To use Anthropic services, install the required dependency: Copy Ask AI pip install "pipecat-ai[anthropic]" You’ll also need to set up your Anthropic API key as an environment variable: ANTHROPIC_API_KEY . Get your API key from Anthropic Console . ​ Frames ​ Input OpenAILLMContextFrame - Conversation context and history LLMMessagesFrame - Direct message list VisionImageRawFrame - Images for vision processing LLMUpdateSettingsFrame - Runtime parameter updates LLMEnablePromptCachingFrame - Toggle prompt caching ​ Output LLMFullResponseStartFrame / LLMFullResponseEndFrame - Response boundaries LLMTextFrame - Streamed completion chunks FunctionCallInProgressFrame / FunctionCallResultFrame - Function call lifecycle ErrorFrame - API or processing errors ​ Function Calling Function Calling Guide Learn how to implement function calling with standardized schemas, register handlers, manage context properly, and control execution flow in your conversational AI applications. ​ Context Management Context Management Guide Learn how to manage conversation context, handle message history, and integrate context aggregators for consistent conversational experiences. ​ Usage Example Copy Ask AI import os from pipecat.services.anthropic.llm import AnthropicLLMService from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext from pipecat.adapters.schemas.function_schema import FunctionSchema from pipecat.adapters.schemas.tools_schema import ToolsSchema # Configure the service llm = AnthropicLLMService( api_key = os.getenv( "ANTHROPIC_API_KEY" ), model = "claude-sonnet-4-20250514" , params = AnthropicLLMService.InputParams( temperature = 0.7 , enable_prompt_caching_beta = True ) ) # Define function for tool calling weather_function = FunctionSchema( name = "get_weather" , description = "Get current weather information" , properties = { "location" : { "type" : "string" , "description" : "City and state, e.g. San Francisco, CA" } }, required = [ "location" ] ) tools = ToolsSchema( standard_tools = [weather_function]) # Create context with system message context = OpenAILLMContext( messages = [{ "role" : "user" , "content" : "What's the weather like?" }], tools = tools ) # Create context aggregators context_aggregator = llm.create_context_aggregator(context) # Register function handler async def get_weather ( params ): location = params.arguments[ "location" ] await params.result_callback( f "Weather in { location } : 72°F and sunny" ) llm.register_function( "get_weather" , get_weather) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, context_aggregator.user(), # Handles user messages llm, # Processes with Anthropic tts, transport.output(), context_aggregator.assistant() # Captures responses ]) ​ Metrics The service provides: Time to First Byte (TTFB) - Latency from request to first response token Processing Duration - Total request processing time Token Usage - Prompt tokens, completion tokens, and total usage Cache Metrics - Cache creation and read token usage Enable with: Copy Ask AI task = PipelineTask( pipeline, params = PipelineParams( enable_metrics = True , enable_usage_metrics = True ) ) ​ Additional Notes Streaming Responses : All responses are streamed for low latency Context Persistence : Use context aggregators to maintain conversation history Error Handling : Automatic retry logic for rate limits and transient errors Message Format : Automatically converts between OpenAI and Anthropic message formats Prompt Caching : Reduces costs and latency for repeated context patterns Whisper AWS Bedrock On this page Overview Installation Frames Input Output Function Calling Context Management Usage Example Metrics Additional Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/llm_anthropic_60356b63.txt b/llm_anthropic_60356b63.txt
new file mode 100644
index 0000000000000000000000000000000000000000..747ff417c81f593cf3c59d994ad3b3892a9ad9b7
--- /dev/null
+++ b/llm_anthropic_60356b63.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/llm/anthropic#output
+Title: Anthropic - Pipecat
+==================================================
+
+Anthropic - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation LLM Anthropic Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Anthropic AWS Bedrock Azure Cerebras DeepSeek Fireworks AI Google Gemini Google Vertex AI Grok Groq NVIDIA NIM Ollama OpenAI OpenPipe OpenRouter Perplexity Qwen SambaNova Together AI Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview AnthropicLLMService provides integration with Anthropic’s Claude models, supporting streaming responses, function calling, and prompt caching with specialized context handling for Anthropic’s message format. API Reference Complete API documentation and method details Anthropic Docs Official Anthropic API documentation and features Example Code Working example with function calling ​ Installation To use Anthropic services, install the required dependency: Copy Ask AI pip install "pipecat-ai[anthropic]" You’ll also need to set up your Anthropic API key as an environment variable: ANTHROPIC_API_KEY . Get your API key from Anthropic Console . ​ Frames ​ Input OpenAILLMContextFrame - Conversation context and history LLMMessagesFrame - Direct message list VisionImageRawFrame - Images for vision processing LLMUpdateSettingsFrame - Runtime parameter updates LLMEnablePromptCachingFrame - Toggle prompt caching ​ Output LLMFullResponseStartFrame / LLMFullResponseEndFrame - Response boundaries LLMTextFrame - Streamed completion chunks FunctionCallInProgressFrame / FunctionCallResultFrame - Function call lifecycle ErrorFrame - API or processing errors ​ Function Calling Function Calling Guide Learn how to implement function calling with standardized schemas, register handlers, manage context properly, and control execution flow in your conversational AI applications. ​ Context Management Context Management Guide Learn how to manage conversation context, handle message history, and integrate context aggregators for consistent conversational experiences. ​ Usage Example Copy Ask AI import os from pipecat.services.anthropic.llm import AnthropicLLMService from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext from pipecat.adapters.schemas.function_schema import FunctionSchema from pipecat.adapters.schemas.tools_schema import ToolsSchema # Configure the service llm = AnthropicLLMService( api_key = os.getenv( "ANTHROPIC_API_KEY" ), model = "claude-sonnet-4-20250514" , params = AnthropicLLMService.InputParams( temperature = 0.7 , enable_prompt_caching_beta = True ) ) # Define function for tool calling weather_function = FunctionSchema( name = "get_weather" , description = "Get current weather information" , properties = { "location" : { "type" : "string" , "description" : "City and state, e.g. San Francisco, CA" } }, required = [ "location" ] ) tools = ToolsSchema( standard_tools = [weather_function]) # Create context with system message context = OpenAILLMContext( messages = [{ "role" : "user" , "content" : "What's the weather like?" }], tools = tools ) # Create context aggregators context_aggregator = llm.create_context_aggregator(context) # Register function handler async def get_weather ( params ): location = params.arguments[ "location" ] await params.result_callback( f "Weather in { location } : 72°F and sunny" ) llm.register_function( "get_weather" , get_weather) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, context_aggregator.user(), # Handles user messages llm, # Processes with Anthropic tts, transport.output(), context_aggregator.assistant() # Captures responses ]) ​ Metrics The service provides: Time to First Byte (TTFB) - Latency from request to first response token Processing Duration - Total request processing time Token Usage - Prompt tokens, completion tokens, and total usage Cache Metrics - Cache creation and read token usage Enable with: Copy Ask AI task = PipelineTask( pipeline, params = PipelineParams( enable_metrics = True , enable_usage_metrics = True ) ) ​ Additional Notes Streaming Responses : All responses are streamed for low latency Context Persistence : Use context aggregators to maintain conversation history Error Handling : Automatic retry logic for rate limits and transient errors Message Format : Automatically converts between OpenAI and Anthropic message formats Prompt Caching : Reduces costs and latency for repeated context patterns Whisper AWS Bedrock On this page Overview Installation Frames Input Output Function Calling Context Management Usage Example Metrics Additional Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/llm_anthropic_d00e3ad0.txt b/llm_anthropic_d00e3ad0.txt
new file mode 100644
index 0000000000000000000000000000000000000000..4d03b8fcdf2bc4df1a297582411bd03b8aa1b7ad
--- /dev/null
+++ b/llm_anthropic_d00e3ad0.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/llm/anthropic
+Title: Anthropic - Pipecat
+==================================================
+
+Anthropic - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation LLM Anthropic Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Anthropic AWS Bedrock Azure Cerebras DeepSeek Fireworks AI Google Gemini Google Vertex AI Grok Groq NVIDIA NIM Ollama OpenAI OpenPipe OpenRouter Perplexity Qwen SambaNova Together AI Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview AnthropicLLMService provides integration with Anthropic’s Claude models, supporting streaming responses, function calling, and prompt caching with specialized context handling for Anthropic’s message format. API Reference Complete API documentation and method details Anthropic Docs Official Anthropic API documentation and features Example Code Working example with function calling ​ Installation To use Anthropic services, install the required dependency: Copy Ask AI pip install "pipecat-ai[anthropic]" You’ll also need to set up your Anthropic API key as an environment variable: ANTHROPIC_API_KEY . Get your API key from Anthropic Console . ​ Frames ​ Input OpenAILLMContextFrame - Conversation context and history LLMMessagesFrame - Direct message list VisionImageRawFrame - Images for vision processing LLMUpdateSettingsFrame - Runtime parameter updates LLMEnablePromptCachingFrame - Toggle prompt caching ​ Output LLMFullResponseStartFrame / LLMFullResponseEndFrame - Response boundaries LLMTextFrame - Streamed completion chunks FunctionCallInProgressFrame / FunctionCallResultFrame - Function call lifecycle ErrorFrame - API or processing errors ​ Function Calling Function Calling Guide Learn how to implement function calling with standardized schemas, register handlers, manage context properly, and control execution flow in your conversational AI applications. ​ Context Management Context Management Guide Learn how to manage conversation context, handle message history, and integrate context aggregators for consistent conversational experiences. ​ Usage Example Copy Ask AI import os from pipecat.services.anthropic.llm import AnthropicLLMService from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext from pipecat.adapters.schemas.function_schema import FunctionSchema from pipecat.adapters.schemas.tools_schema import ToolsSchema # Configure the service llm = AnthropicLLMService( api_key = os.getenv( "ANTHROPIC_API_KEY" ), model = "claude-sonnet-4-20250514" , params = AnthropicLLMService.InputParams( temperature = 0.7 , enable_prompt_caching_beta = True ) ) # Define function for tool calling weather_function = FunctionSchema( name = "get_weather" , description = "Get current weather information" , properties = { "location" : { "type" : "string" , "description" : "City and state, e.g. San Francisco, CA" } }, required = [ "location" ] ) tools = ToolsSchema( standard_tools = [weather_function]) # Create context with system message context = OpenAILLMContext( messages = [{ "role" : "user" , "content" : "What's the weather like?" }], tools = tools ) # Create context aggregators context_aggregator = llm.create_context_aggregator(context) # Register function handler async def get_weather ( params ): location = params.arguments[ "location" ] await params.result_callback( f "Weather in { location } : 72°F and sunny" ) llm.register_function( "get_weather" , get_weather) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, context_aggregator.user(), # Handles user messages llm, # Processes with Anthropic tts, transport.output(), context_aggregator.assistant() # Captures responses ]) ​ Metrics The service provides: Time to First Byte (TTFB) - Latency from request to first response token Processing Duration - Total request processing time Token Usage - Prompt tokens, completion tokens, and total usage Cache Metrics - Cache creation and read token usage Enable with: Copy Ask AI task = PipelineTask( pipeline, params = PipelineParams( enable_metrics = True , enable_usage_metrics = True ) ) ​ Additional Notes Streaming Responses : All responses are streamed for low latency Context Persistence : Use context aggregators to maintain conversation history Error Handling : Automatic retry logic for rate limits and transient errors Message Format : Automatically converts between OpenAI and Anthropic message formats Prompt Caching : Reduces costs and latency for repeated context patterns Whisper AWS Bedrock On this page Overview Installation Frames Input Output Function Calling Context Management Usage Example Metrics Additional Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/llm_azure_12f2bcc6.txt b/llm_azure_12f2bcc6.txt
new file mode 100644
index 0000000000000000000000000000000000000000..a6f1476e3c109b6ce43423e1f868d08a9c6b611c
--- /dev/null
+++ b/llm_azure_12f2bcc6.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/llm/azure#azure-vs-openai-differences
+Title: Azure - Pipecat
+==================================================
+
+Azure - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation LLM Azure Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Anthropic AWS Bedrock Azure Cerebras DeepSeek Fireworks AI Google Gemini Google Vertex AI Grok Groq NVIDIA NIM Ollama OpenAI OpenPipe OpenRouter Perplexity Qwen SambaNova Together AI Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview AzureLLMService provides access to Azure OpenAI’s language models through an OpenAI-compatible interface. It inherits from OpenAILLMService and supports streaming responses, function calling, and context management. API Reference Complete API documentation and method details Azure OpenAI Docs Official Azure OpenAI documentation and setup Example Code Working example with function calling ​ Installation To use Azure OpenAI services, install the required dependency: Copy Ask AI pip install "pipecat-ai[azure]" You’ll need to set up your Azure OpenAI credentials: AZURE_CHATGPT_API_KEY - Your Azure OpenAI API key AZURE_CHATGPT_ENDPOINT - Your Azure OpenAI endpoint URL AZURE_CHATGPT_MODEL - Your model deployment name Get your credentials from the Azure Portal under your Azure OpenAI resource. ​ Frames ​ Input OpenAILLMContextFrame - Conversation context and history LLMMessagesFrame - Direct message list VisionImageRawFrame - Images for vision processing LLMUpdateSettingsFrame - Runtime parameter updates ​ Output LLMFullResponseStartFrame / LLMFullResponseEndFrame - Response boundaries LLMTextFrame - Streamed completion chunks FunctionCallInProgressFrame / FunctionCallResultFrame - Function call lifecycle ErrorFrame - API or processing errors ​ Azure vs OpenAI Differences Feature Azure OpenAI Standard OpenAI Authentication API key + endpoint API key only Deployment Custom deployment names Model names directly Compliance Enterprise SOC, HIPAA Standard compliance Regional Multiple Azure regions OpenAI regions only Pricing Azure billing integration OpenAI billing ​ Function Calling Function Calling Guide Learn how to implement function calling with standardized schemas, register handlers, manage context properly, and control execution flow in your conversational AI applications. ​ Context Management Context Management Guide Learn how to manage conversation context, handle message history, and integrate context aggregators for consistent conversational experiences. ​ Usage Example Copy Ask AI import os from pipecat.services.azure.llm import AzureLLMService from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext from pipecat.adapters.schemas.function_schema import FunctionSchema from pipecat.adapters.schemas.tools_schema import ToolsSchema # Configure Azure OpenAI service llm = AzureLLMService( api_key = os.getenv( "AZURE_CHATGPT_API_KEY" ), endpoint = os.getenv( "AZURE_CHATGPT_ENDPOINT" ), model = os.getenv( "AZURE_CHATGPT_MODEL" ), # Your deployment name params = AzureLLMService.InputParams( temperature = 0.7 , max_completion_tokens = 1000 ) ) # Define function for tool calling weather_function = FunctionSchema( name = "get_current_weather" , description = "Get current weather information" , properties = { "location" : { "type" : "string" , "description" : "City and state, e.g. San Francisco, CA" }, "format" : { "type" : "string" , "enum" : [ "celsius" , "fahrenheit" ], "description" : "Temperature unit to use" } }, required = [ "location" , "format" ] ) tools = ToolsSchema( standard_tools = [weather_function]) # Create context context = OpenAILLMContext( messages = [ { "role" : "system" , "content" : "You are a helpful assistant. Keep responses concise for voice output." } ], tools = tools ) # Create context aggregators context_aggregator = llm.create_context_aggregator(context) # Register function handler with event callback async def fetch_weather ( params ): location = params.arguments[ "location" ] await params.result_callback({ "conditions" : "sunny" , "temperature" : "75°F" }) llm.register_function( "get_current_weather" , fetch_weather) # Optional: Add function call feedback @llm.event_handler ( "on_function_calls_started" ) async def on_function_calls_started ( service , function_calls ): await tts.queue_frame(TTSSpeakFrame( "Let me check on that." )) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, context_aggregator.user(), llm, tts, transport.output(), context_aggregator.assistant() ]) ​ Metrics Inherits all OpenAI metrics capabilities: Time to First Byte (TTFB) - Response latency measurement Processing Duration - Total request processing time Token Usage - Prompt tokens, completion tokens, and totals Enable with: Copy Ask AI task = PipelineTask( pipeline, params = PipelineParams( enable_metrics = True , enable_usage_metrics = True ) ) ​ Additional Notes OpenAI Compatibility : Full compatibility with OpenAI API features and parameters Regional Deployment : Deploy in your preferred Azure region for compliance and latency Deployment Names : Use your Azure deployment name as the model parameter, not OpenAI model names Automatic Retries : Built-in retry logic handles transient Azure service issues AWS Bedrock Cerebras On this page Overview Installation Frames Input Output Azure vs OpenAI Differences Function Calling Context Management Usage Example Metrics Additional Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/llm_azure_249074f2.txt b/llm_azure_249074f2.txt
new file mode 100644
index 0000000000000000000000000000000000000000..d47358eb0f2b70fad4aafe8d8afe2d956b86d3b7
--- /dev/null
+++ b/llm_azure_249074f2.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/llm/azure#additional-notes
+Title: Azure - Pipecat
+==================================================
+
+Azure - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation LLM Azure Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Anthropic AWS Bedrock Azure Cerebras DeepSeek Fireworks AI Google Gemini Google Vertex AI Grok Groq NVIDIA NIM Ollama OpenAI OpenPipe OpenRouter Perplexity Qwen SambaNova Together AI Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview AzureLLMService provides access to Azure OpenAI’s language models through an OpenAI-compatible interface. It inherits from OpenAILLMService and supports streaming responses, function calling, and context management. API Reference Complete API documentation and method details Azure OpenAI Docs Official Azure OpenAI documentation and setup Example Code Working example with function calling ​ Installation To use Azure OpenAI services, install the required dependency: Copy Ask AI pip install "pipecat-ai[azure]" You’ll need to set up your Azure OpenAI credentials: AZURE_CHATGPT_API_KEY - Your Azure OpenAI API key AZURE_CHATGPT_ENDPOINT - Your Azure OpenAI endpoint URL AZURE_CHATGPT_MODEL - Your model deployment name Get your credentials from the Azure Portal under your Azure OpenAI resource. ​ Frames ​ Input OpenAILLMContextFrame - Conversation context and history LLMMessagesFrame - Direct message list VisionImageRawFrame - Images for vision processing LLMUpdateSettingsFrame - Runtime parameter updates ​ Output LLMFullResponseStartFrame / LLMFullResponseEndFrame - Response boundaries LLMTextFrame - Streamed completion chunks FunctionCallInProgressFrame / FunctionCallResultFrame - Function call lifecycle ErrorFrame - API or processing errors ​ Azure vs OpenAI Differences Feature Azure OpenAI Standard OpenAI Authentication API key + endpoint API key only Deployment Custom deployment names Model names directly Compliance Enterprise SOC, HIPAA Standard compliance Regional Multiple Azure regions OpenAI regions only Pricing Azure billing integration OpenAI billing ​ Function Calling Function Calling Guide Learn how to implement function calling with standardized schemas, register handlers, manage context properly, and control execution flow in your conversational AI applications. ​ Context Management Context Management Guide Learn how to manage conversation context, handle message history, and integrate context aggregators for consistent conversational experiences. ​ Usage Example Copy Ask AI import os from pipecat.services.azure.llm import AzureLLMService from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext from pipecat.adapters.schemas.function_schema import FunctionSchema from pipecat.adapters.schemas.tools_schema import ToolsSchema # Configure Azure OpenAI service llm = AzureLLMService( api_key = os.getenv( "AZURE_CHATGPT_API_KEY" ), endpoint = os.getenv( "AZURE_CHATGPT_ENDPOINT" ), model = os.getenv( "AZURE_CHATGPT_MODEL" ), # Your deployment name params = AzureLLMService.InputParams( temperature = 0.7 , max_completion_tokens = 1000 ) ) # Define function for tool calling weather_function = FunctionSchema( name = "get_current_weather" , description = "Get current weather information" , properties = { "location" : { "type" : "string" , "description" : "City and state, e.g. San Francisco, CA" }, "format" : { "type" : "string" , "enum" : [ "celsius" , "fahrenheit" ], "description" : "Temperature unit to use" } }, required = [ "location" , "format" ] ) tools = ToolsSchema( standard_tools = [weather_function]) # Create context context = OpenAILLMContext( messages = [ { "role" : "system" , "content" : "You are a helpful assistant. Keep responses concise for voice output." } ], tools = tools ) # Create context aggregators context_aggregator = llm.create_context_aggregator(context) # Register function handler with event callback async def fetch_weather ( params ): location = params.arguments[ "location" ] await params.result_callback({ "conditions" : "sunny" , "temperature" : "75°F" }) llm.register_function( "get_current_weather" , fetch_weather) # Optional: Add function call feedback @llm.event_handler ( "on_function_calls_started" ) async def on_function_calls_started ( service , function_calls ): await tts.queue_frame(TTSSpeakFrame( "Let me check on that." )) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, context_aggregator.user(), llm, tts, transport.output(), context_aggregator.assistant() ]) ​ Metrics Inherits all OpenAI metrics capabilities: Time to First Byte (TTFB) - Response latency measurement Processing Duration - Total request processing time Token Usage - Prompt tokens, completion tokens, and totals Enable with: Copy Ask AI task = PipelineTask( pipeline, params = PipelineParams( enable_metrics = True , enable_usage_metrics = True ) ) ​ Additional Notes OpenAI Compatibility : Full compatibility with OpenAI API features and parameters Regional Deployment : Deploy in your preferred Azure region for compliance and latency Deployment Names : Use your Azure deployment name as the model parameter, not OpenAI model names Automatic Retries : Built-in retry logic handles transient Azure service issues AWS Bedrock Cerebras On this page Overview Installation Frames Input Output Azure vs OpenAI Differences Function Calling Context Management Usage Example Metrics Additional Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/llm_azure_52726914.txt b/llm_azure_52726914.txt
new file mode 100644
index 0000000000000000000000000000000000000000..a4d4e84e4f6ccf83b890dccceb69e283d59107d7
--- /dev/null
+++ b/llm_azure_52726914.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/llm/azure#output
+Title: Azure - Pipecat
+==================================================
+
+Azure - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation LLM Azure Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Anthropic AWS Bedrock Azure Cerebras DeepSeek Fireworks AI Google Gemini Google Vertex AI Grok Groq NVIDIA NIM Ollama OpenAI OpenPipe OpenRouter Perplexity Qwen SambaNova Together AI Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview AzureLLMService provides access to Azure OpenAI’s language models through an OpenAI-compatible interface. It inherits from OpenAILLMService and supports streaming responses, function calling, and context management. API Reference Complete API documentation and method details Azure OpenAI Docs Official Azure OpenAI documentation and setup Example Code Working example with function calling ​ Installation To use Azure OpenAI services, install the required dependency: Copy Ask AI pip install "pipecat-ai[azure]" You’ll need to set up your Azure OpenAI credentials: AZURE_CHATGPT_API_KEY - Your Azure OpenAI API key AZURE_CHATGPT_ENDPOINT - Your Azure OpenAI endpoint URL AZURE_CHATGPT_MODEL - Your model deployment name Get your credentials from the Azure Portal under your Azure OpenAI resource. ​ Frames ​ Input OpenAILLMContextFrame - Conversation context and history LLMMessagesFrame - Direct message list VisionImageRawFrame - Images for vision processing LLMUpdateSettingsFrame - Runtime parameter updates ​ Output LLMFullResponseStartFrame / LLMFullResponseEndFrame - Response boundaries LLMTextFrame - Streamed completion chunks FunctionCallInProgressFrame / FunctionCallResultFrame - Function call lifecycle ErrorFrame - API or processing errors ​ Azure vs OpenAI Differences Feature Azure OpenAI Standard OpenAI Authentication API key + endpoint API key only Deployment Custom deployment names Model names directly Compliance Enterprise SOC, HIPAA Standard compliance Regional Multiple Azure regions OpenAI regions only Pricing Azure billing integration OpenAI billing ​ Function Calling Function Calling Guide Learn how to implement function calling with standardized schemas, register handlers, manage context properly, and control execution flow in your conversational AI applications. ​ Context Management Context Management Guide Learn how to manage conversation context, handle message history, and integrate context aggregators for consistent conversational experiences. ​ Usage Example Copy Ask AI import os from pipecat.services.azure.llm import AzureLLMService from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext from pipecat.adapters.schemas.function_schema import FunctionSchema from pipecat.adapters.schemas.tools_schema import ToolsSchema # Configure Azure OpenAI service llm = AzureLLMService( api_key = os.getenv( "AZURE_CHATGPT_API_KEY" ), endpoint = os.getenv( "AZURE_CHATGPT_ENDPOINT" ), model = os.getenv( "AZURE_CHATGPT_MODEL" ), # Your deployment name params = AzureLLMService.InputParams( temperature = 0.7 , max_completion_tokens = 1000 ) ) # Define function for tool calling weather_function = FunctionSchema( name = "get_current_weather" , description = "Get current weather information" , properties = { "location" : { "type" : "string" , "description" : "City and state, e.g. San Francisco, CA" }, "format" : { "type" : "string" , "enum" : [ "celsius" , "fahrenheit" ], "description" : "Temperature unit to use" } }, required = [ "location" , "format" ] ) tools = ToolsSchema( standard_tools = [weather_function]) # Create context context = OpenAILLMContext( messages = [ { "role" : "system" , "content" : "You are a helpful assistant. Keep responses concise for voice output." } ], tools = tools ) # Create context aggregators context_aggregator = llm.create_context_aggregator(context) # Register function handler with event callback async def fetch_weather ( params ): location = params.arguments[ "location" ] await params.result_callback({ "conditions" : "sunny" , "temperature" : "75°F" }) llm.register_function( "get_current_weather" , fetch_weather) # Optional: Add function call feedback @llm.event_handler ( "on_function_calls_started" ) async def on_function_calls_started ( service , function_calls ): await tts.queue_frame(TTSSpeakFrame( "Let me check on that." )) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, context_aggregator.user(), llm, tts, transport.output(), context_aggregator.assistant() ]) ​ Metrics Inherits all OpenAI metrics capabilities: Time to First Byte (TTFB) - Response latency measurement Processing Duration - Total request processing time Token Usage - Prompt tokens, completion tokens, and totals Enable with: Copy Ask AI task = PipelineTask( pipeline, params = PipelineParams( enable_metrics = True , enable_usage_metrics = True ) ) ​ Additional Notes OpenAI Compatibility : Full compatibility with OpenAI API features and parameters Regional Deployment : Deploy in your preferred Azure region for compliance and latency Deployment Names : Use your Azure deployment name as the model parameter, not OpenAI model names Automatic Retries : Built-in retry logic handles transient Azure service issues AWS Bedrock Cerebras On this page Overview Installation Frames Input Output Azure vs OpenAI Differences Function Calling Context Management Usage Example Metrics Additional Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/llm_azure_56823eb0.txt b/llm_azure_56823eb0.txt
new file mode 100644
index 0000000000000000000000000000000000000000..67ca9ca7f7e13d79fbb47455074600ae8928e287
--- /dev/null
+++ b/llm_azure_56823eb0.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/llm/azure#frames
+Title: Azure - Pipecat
+==================================================
+
+Azure - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation LLM Azure Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Anthropic AWS Bedrock Azure Cerebras DeepSeek Fireworks AI Google Gemini Google Vertex AI Grok Groq NVIDIA NIM Ollama OpenAI OpenPipe OpenRouter Perplexity Qwen SambaNova Together AI Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview AzureLLMService provides access to Azure OpenAI’s language models through an OpenAI-compatible interface. It inherits from OpenAILLMService and supports streaming responses, function calling, and context management. API Reference Complete API documentation and method details Azure OpenAI Docs Official Azure OpenAI documentation and setup Example Code Working example with function calling ​ Installation To use Azure OpenAI services, install the required dependency: Copy Ask AI pip install "pipecat-ai[azure]" You’ll need to set up your Azure OpenAI credentials: AZURE_CHATGPT_API_KEY - Your Azure OpenAI API key AZURE_CHATGPT_ENDPOINT - Your Azure OpenAI endpoint URL AZURE_CHATGPT_MODEL - Your model deployment name Get your credentials from the Azure Portal under your Azure OpenAI resource. ​ Frames ​ Input OpenAILLMContextFrame - Conversation context and history LLMMessagesFrame - Direct message list VisionImageRawFrame - Images for vision processing LLMUpdateSettingsFrame - Runtime parameter updates ​ Output LLMFullResponseStartFrame / LLMFullResponseEndFrame - Response boundaries LLMTextFrame - Streamed completion chunks FunctionCallInProgressFrame / FunctionCallResultFrame - Function call lifecycle ErrorFrame - API or processing errors ​ Azure vs OpenAI Differences Feature Azure OpenAI Standard OpenAI Authentication API key + endpoint API key only Deployment Custom deployment names Model names directly Compliance Enterprise SOC, HIPAA Standard compliance Regional Multiple Azure regions OpenAI regions only Pricing Azure billing integration OpenAI billing ​ Function Calling Function Calling Guide Learn how to implement function calling with standardized schemas, register handlers, manage context properly, and control execution flow in your conversational AI applications. ​ Context Management Context Management Guide Learn how to manage conversation context, handle message history, and integrate context aggregators for consistent conversational experiences. ​ Usage Example Copy Ask AI import os from pipecat.services.azure.llm import AzureLLMService from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext from pipecat.adapters.schemas.function_schema import FunctionSchema from pipecat.adapters.schemas.tools_schema import ToolsSchema # Configure Azure OpenAI service llm = AzureLLMService( api_key = os.getenv( "AZURE_CHATGPT_API_KEY" ), endpoint = os.getenv( "AZURE_CHATGPT_ENDPOINT" ), model = os.getenv( "AZURE_CHATGPT_MODEL" ), # Your deployment name params = AzureLLMService.InputParams( temperature = 0.7 , max_completion_tokens = 1000 ) ) # Define function for tool calling weather_function = FunctionSchema( name = "get_current_weather" , description = "Get current weather information" , properties = { "location" : { "type" : "string" , "description" : "City and state, e.g. San Francisco, CA" }, "format" : { "type" : "string" , "enum" : [ "celsius" , "fahrenheit" ], "description" : "Temperature unit to use" } }, required = [ "location" , "format" ] ) tools = ToolsSchema( standard_tools = [weather_function]) # Create context context = OpenAILLMContext( messages = [ { "role" : "system" , "content" : "You are a helpful assistant. Keep responses concise for voice output." } ], tools = tools ) # Create context aggregators context_aggregator = llm.create_context_aggregator(context) # Register function handler with event callback async def fetch_weather ( params ): location = params.arguments[ "location" ] await params.result_callback({ "conditions" : "sunny" , "temperature" : "75°F" }) llm.register_function( "get_current_weather" , fetch_weather) # Optional: Add function call feedback @llm.event_handler ( "on_function_calls_started" ) async def on_function_calls_started ( service , function_calls ): await tts.queue_frame(TTSSpeakFrame( "Let me check on that." )) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, context_aggregator.user(), llm, tts, transport.output(), context_aggregator.assistant() ]) ​ Metrics Inherits all OpenAI metrics capabilities: Time to First Byte (TTFB) - Response latency measurement Processing Duration - Total request processing time Token Usage - Prompt tokens, completion tokens, and totals Enable with: Copy Ask AI task = PipelineTask( pipeline, params = PipelineParams( enable_metrics = True , enable_usage_metrics = True ) ) ​ Additional Notes OpenAI Compatibility : Full compatibility with OpenAI API features and parameters Regional Deployment : Deploy in your preferred Azure region for compliance and latency Deployment Names : Use your Azure deployment name as the model parameter, not OpenAI model names Automatic Retries : Built-in retry logic handles transient Azure service issues AWS Bedrock Cerebras On this page Overview Installation Frames Input Output Azure vs OpenAI Differences Function Calling Context Management Usage Example Metrics Additional Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/llm_azure_6658be14.txt b/llm_azure_6658be14.txt
new file mode 100644
index 0000000000000000000000000000000000000000..dd54f8f2b39651220ab5d29d6649c983dc49c0b9
--- /dev/null
+++ b/llm_azure_6658be14.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/llm/azure
+Title: Azure - Pipecat
+==================================================
+
+Azure - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation LLM Azure Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Anthropic AWS Bedrock Azure Cerebras DeepSeek Fireworks AI Google Gemini Google Vertex AI Grok Groq NVIDIA NIM Ollama OpenAI OpenPipe OpenRouter Perplexity Qwen SambaNova Together AI Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview AzureLLMService provides access to Azure OpenAI’s language models through an OpenAI-compatible interface. It inherits from OpenAILLMService and supports streaming responses, function calling, and context management. API Reference Complete API documentation and method details Azure OpenAI Docs Official Azure OpenAI documentation and setup Example Code Working example with function calling ​ Installation To use Azure OpenAI services, install the required dependency: Copy Ask AI pip install "pipecat-ai[azure]" You’ll need to set up your Azure OpenAI credentials: AZURE_CHATGPT_API_KEY - Your Azure OpenAI API key AZURE_CHATGPT_ENDPOINT - Your Azure OpenAI endpoint URL AZURE_CHATGPT_MODEL - Your model deployment name Get your credentials from the Azure Portal under your Azure OpenAI resource. ​ Frames ​ Input OpenAILLMContextFrame - Conversation context and history LLMMessagesFrame - Direct message list VisionImageRawFrame - Images for vision processing LLMUpdateSettingsFrame - Runtime parameter updates ​ Output LLMFullResponseStartFrame / LLMFullResponseEndFrame - Response boundaries LLMTextFrame - Streamed completion chunks FunctionCallInProgressFrame / FunctionCallResultFrame - Function call lifecycle ErrorFrame - API or processing errors ​ Azure vs OpenAI Differences Feature Azure OpenAI Standard OpenAI Authentication API key + endpoint API key only Deployment Custom deployment names Model names directly Compliance Enterprise SOC, HIPAA Standard compliance Regional Multiple Azure regions OpenAI regions only Pricing Azure billing integration OpenAI billing ​ Function Calling Function Calling Guide Learn how to implement function calling with standardized schemas, register handlers, manage context properly, and control execution flow in your conversational AI applications. ​ Context Management Context Management Guide Learn how to manage conversation context, handle message history, and integrate context aggregators for consistent conversational experiences. ​ Usage Example Copy Ask AI import os from pipecat.services.azure.llm import AzureLLMService from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext from pipecat.adapters.schemas.function_schema import FunctionSchema from pipecat.adapters.schemas.tools_schema import ToolsSchema # Configure Azure OpenAI service llm = AzureLLMService( api_key = os.getenv( "AZURE_CHATGPT_API_KEY" ), endpoint = os.getenv( "AZURE_CHATGPT_ENDPOINT" ), model = os.getenv( "AZURE_CHATGPT_MODEL" ), # Your deployment name params = AzureLLMService.InputParams( temperature = 0.7 , max_completion_tokens = 1000 ) ) # Define function for tool calling weather_function = FunctionSchema( name = "get_current_weather" , description = "Get current weather information" , properties = { "location" : { "type" : "string" , "description" : "City and state, e.g. San Francisco, CA" }, "format" : { "type" : "string" , "enum" : [ "celsius" , "fahrenheit" ], "description" : "Temperature unit to use" } }, required = [ "location" , "format" ] ) tools = ToolsSchema( standard_tools = [weather_function]) # Create context context = OpenAILLMContext( messages = [ { "role" : "system" , "content" : "You are a helpful assistant. Keep responses concise for voice output." } ], tools = tools ) # Create context aggregators context_aggregator = llm.create_context_aggregator(context) # Register function handler with event callback async def fetch_weather ( params ): location = params.arguments[ "location" ] await params.result_callback({ "conditions" : "sunny" , "temperature" : "75°F" }) llm.register_function( "get_current_weather" , fetch_weather) # Optional: Add function call feedback @llm.event_handler ( "on_function_calls_started" ) async def on_function_calls_started ( service , function_calls ): await tts.queue_frame(TTSSpeakFrame( "Let me check on that." )) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, context_aggregator.user(), llm, tts, transport.output(), context_aggregator.assistant() ]) ​ Metrics Inherits all OpenAI metrics capabilities: Time to First Byte (TTFB) - Response latency measurement Processing Duration - Total request processing time Token Usage - Prompt tokens, completion tokens, and totals Enable with: Copy Ask AI task = PipelineTask( pipeline, params = PipelineParams( enable_metrics = True , enable_usage_metrics = True ) ) ​ Additional Notes OpenAI Compatibility : Full compatibility with OpenAI API features and parameters Regional Deployment : Deploy in your preferred Azure region for compliance and latency Deployment Names : Use your Azure deployment name as the model parameter, not OpenAI model names Automatic Retries : Built-in retry logic handles transient Azure service issues AWS Bedrock Cerebras On this page Overview Installation Frames Input Output Azure vs OpenAI Differences Function Calling Context Management Usage Example Metrics Additional Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/llm_azure_95f603e8.txt b/llm_azure_95f603e8.txt
new file mode 100644
index 0000000000000000000000000000000000000000..64a77c0d1bf84dc2b5204a078773e9020cc1a926
--- /dev/null
+++ b/llm_azure_95f603e8.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/llm/azure#installation
+Title: Azure - Pipecat
+==================================================
+
+Azure - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation LLM Azure Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Anthropic AWS Bedrock Azure Cerebras DeepSeek Fireworks AI Google Gemini Google Vertex AI Grok Groq NVIDIA NIM Ollama OpenAI OpenPipe OpenRouter Perplexity Qwen SambaNova Together AI Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview AzureLLMService provides access to Azure OpenAI’s language models through an OpenAI-compatible interface. It inherits from OpenAILLMService and supports streaming responses, function calling, and context management. API Reference Complete API documentation and method details Azure OpenAI Docs Official Azure OpenAI documentation and setup Example Code Working example with function calling ​ Installation To use Azure OpenAI services, install the required dependency: Copy Ask AI pip install "pipecat-ai[azure]" You’ll need to set up your Azure OpenAI credentials: AZURE_CHATGPT_API_KEY - Your Azure OpenAI API key AZURE_CHATGPT_ENDPOINT - Your Azure OpenAI endpoint URL AZURE_CHATGPT_MODEL - Your model deployment name Get your credentials from the Azure Portal under your Azure OpenAI resource. ​ Frames ​ Input OpenAILLMContextFrame - Conversation context and history LLMMessagesFrame - Direct message list VisionImageRawFrame - Images for vision processing LLMUpdateSettingsFrame - Runtime parameter updates ​ Output LLMFullResponseStartFrame / LLMFullResponseEndFrame - Response boundaries LLMTextFrame - Streamed completion chunks FunctionCallInProgressFrame / FunctionCallResultFrame - Function call lifecycle ErrorFrame - API or processing errors ​ Azure vs OpenAI Differences Feature Azure OpenAI Standard OpenAI Authentication API key + endpoint API key only Deployment Custom deployment names Model names directly Compliance Enterprise SOC, HIPAA Standard compliance Regional Multiple Azure regions OpenAI regions only Pricing Azure billing integration OpenAI billing ​ Function Calling Function Calling Guide Learn how to implement function calling with standardized schemas, register handlers, manage context properly, and control execution flow in your conversational AI applications. ​ Context Management Context Management Guide Learn how to manage conversation context, handle message history, and integrate context aggregators for consistent conversational experiences. ​ Usage Example Copy Ask AI import os from pipecat.services.azure.llm import AzureLLMService from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext from pipecat.adapters.schemas.function_schema import FunctionSchema from pipecat.adapters.schemas.tools_schema import ToolsSchema # Configure Azure OpenAI service llm = AzureLLMService( api_key = os.getenv( "AZURE_CHATGPT_API_KEY" ), endpoint = os.getenv( "AZURE_CHATGPT_ENDPOINT" ), model = os.getenv( "AZURE_CHATGPT_MODEL" ), # Your deployment name params = AzureLLMService.InputParams( temperature = 0.7 , max_completion_tokens = 1000 ) ) # Define function for tool calling weather_function = FunctionSchema( name = "get_current_weather" , description = "Get current weather information" , properties = { "location" : { "type" : "string" , "description" : "City and state, e.g. San Francisco, CA" }, "format" : { "type" : "string" , "enum" : [ "celsius" , "fahrenheit" ], "description" : "Temperature unit to use" } }, required = [ "location" , "format" ] ) tools = ToolsSchema( standard_tools = [weather_function]) # Create context context = OpenAILLMContext( messages = [ { "role" : "system" , "content" : "You are a helpful assistant. Keep responses concise for voice output." } ], tools = tools ) # Create context aggregators context_aggregator = llm.create_context_aggregator(context) # Register function handler with event callback async def fetch_weather ( params ): location = params.arguments[ "location" ] await params.result_callback({ "conditions" : "sunny" , "temperature" : "75°F" }) llm.register_function( "get_current_weather" , fetch_weather) # Optional: Add function call feedback @llm.event_handler ( "on_function_calls_started" ) async def on_function_calls_started ( service , function_calls ): await tts.queue_frame(TTSSpeakFrame( "Let me check on that." )) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, context_aggregator.user(), llm, tts, transport.output(), context_aggregator.assistant() ]) ​ Metrics Inherits all OpenAI metrics capabilities: Time to First Byte (TTFB) - Response latency measurement Processing Duration - Total request processing time Token Usage - Prompt tokens, completion tokens, and totals Enable with: Copy Ask AI task = PipelineTask( pipeline, params = PipelineParams( enable_metrics = True , enable_usage_metrics = True ) ) ​ Additional Notes OpenAI Compatibility : Full compatibility with OpenAI API features and parameters Regional Deployment : Deploy in your preferred Azure region for compliance and latency Deployment Names : Use your Azure deployment name as the model parameter, not OpenAI model names Automatic Retries : Built-in retry logic handles transient Azure service issues AWS Bedrock Cerebras On this page Overview Installation Frames Input Output Azure vs OpenAI Differences Function Calling Context Management Usage Example Metrics Additional Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/llm_azure_bf8067b5.txt b/llm_azure_bf8067b5.txt
new file mode 100644
index 0000000000000000000000000000000000000000..a608876cfdf9cea24bab0b5a62d6fa4c47d04f75
--- /dev/null
+++ b/llm_azure_bf8067b5.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/llm/azure#overview
+Title: Azure - Pipecat
+==================================================
+
+Azure - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation LLM Azure Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Anthropic AWS Bedrock Azure Cerebras DeepSeek Fireworks AI Google Gemini Google Vertex AI Grok Groq NVIDIA NIM Ollama OpenAI OpenPipe OpenRouter Perplexity Qwen SambaNova Together AI Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview AzureLLMService provides access to Azure OpenAI’s language models through an OpenAI-compatible interface. It inherits from OpenAILLMService and supports streaming responses, function calling, and context management. API Reference Complete API documentation and method details Azure OpenAI Docs Official Azure OpenAI documentation and setup Example Code Working example with function calling ​ Installation To use Azure OpenAI services, install the required dependency: Copy Ask AI pip install "pipecat-ai[azure]" You’ll need to set up your Azure OpenAI credentials: AZURE_CHATGPT_API_KEY - Your Azure OpenAI API key AZURE_CHATGPT_ENDPOINT - Your Azure OpenAI endpoint URL AZURE_CHATGPT_MODEL - Your model deployment name Get your credentials from the Azure Portal under your Azure OpenAI resource. ​ Frames ​ Input OpenAILLMContextFrame - Conversation context and history LLMMessagesFrame - Direct message list VisionImageRawFrame - Images for vision processing LLMUpdateSettingsFrame - Runtime parameter updates ​ Output LLMFullResponseStartFrame / LLMFullResponseEndFrame - Response boundaries LLMTextFrame - Streamed completion chunks FunctionCallInProgressFrame / FunctionCallResultFrame - Function call lifecycle ErrorFrame - API or processing errors ​ Azure vs OpenAI Differences Feature Azure OpenAI Standard OpenAI Authentication API key + endpoint API key only Deployment Custom deployment names Model names directly Compliance Enterprise SOC, HIPAA Standard compliance Regional Multiple Azure regions OpenAI regions only Pricing Azure billing integration OpenAI billing ​ Function Calling Function Calling Guide Learn how to implement function calling with standardized schemas, register handlers, manage context properly, and control execution flow in your conversational AI applications. ​ Context Management Context Management Guide Learn how to manage conversation context, handle message history, and integrate context aggregators for consistent conversational experiences. ​ Usage Example Copy Ask AI import os from pipecat.services.azure.llm import AzureLLMService from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext from pipecat.adapters.schemas.function_schema import FunctionSchema from pipecat.adapters.schemas.tools_schema import ToolsSchema # Configure Azure OpenAI service llm = AzureLLMService( api_key = os.getenv( "AZURE_CHATGPT_API_KEY" ), endpoint = os.getenv( "AZURE_CHATGPT_ENDPOINT" ), model = os.getenv( "AZURE_CHATGPT_MODEL" ), # Your deployment name params = AzureLLMService.InputParams( temperature = 0.7 , max_completion_tokens = 1000 ) ) # Define function for tool calling weather_function = FunctionSchema( name = "get_current_weather" , description = "Get current weather information" , properties = { "location" : { "type" : "string" , "description" : "City and state, e.g. San Francisco, CA" }, "format" : { "type" : "string" , "enum" : [ "celsius" , "fahrenheit" ], "description" : "Temperature unit to use" } }, required = [ "location" , "format" ] ) tools = ToolsSchema( standard_tools = [weather_function]) # Create context context = OpenAILLMContext( messages = [ { "role" : "system" , "content" : "You are a helpful assistant. Keep responses concise for voice output." } ], tools = tools ) # Create context aggregators context_aggregator = llm.create_context_aggregator(context) # Register function handler with event callback async def fetch_weather ( params ): location = params.arguments[ "location" ] await params.result_callback({ "conditions" : "sunny" , "temperature" : "75°F" }) llm.register_function( "get_current_weather" , fetch_weather) # Optional: Add function call feedback @llm.event_handler ( "on_function_calls_started" ) async def on_function_calls_started ( service , function_calls ): await tts.queue_frame(TTSSpeakFrame( "Let me check on that." )) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, context_aggregator.user(), llm, tts, transport.output(), context_aggregator.assistant() ]) ​ Metrics Inherits all OpenAI metrics capabilities: Time to First Byte (TTFB) - Response latency measurement Processing Duration - Total request processing time Token Usage - Prompt tokens, completion tokens, and totals Enable with: Copy Ask AI task = PipelineTask( pipeline, params = PipelineParams( enable_metrics = True , enable_usage_metrics = True ) ) ​ Additional Notes OpenAI Compatibility : Full compatibility with OpenAI API features and parameters Regional Deployment : Deploy in your preferred Azure region for compliance and latency Deployment Names : Use your Azure deployment name as the model parameter, not OpenAI model names Automatic Retries : Built-in retry logic handles transient Azure service issues AWS Bedrock Cerebras On this page Overview Installation Frames Input Output Azure vs OpenAI Differences Function Calling Context Management Usage Example Metrics Additional Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/llm_azure_d1a83f87.txt b/llm_azure_d1a83f87.txt
new file mode 100644
index 0000000000000000000000000000000000000000..72301866d9bc883279d984a426cb7b9afcc7665a
--- /dev/null
+++ b/llm_azure_d1a83f87.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/llm/azure#input
+Title: Azure - Pipecat
+==================================================
+
+Azure - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation LLM Azure Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Anthropic AWS Bedrock Azure Cerebras DeepSeek Fireworks AI Google Gemini Google Vertex AI Grok Groq NVIDIA NIM Ollama OpenAI OpenPipe OpenRouter Perplexity Qwen SambaNova Together AI Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview AzureLLMService provides access to Azure OpenAI’s language models through an OpenAI-compatible interface. It inherits from OpenAILLMService and supports streaming responses, function calling, and context management. API Reference Complete API documentation and method details Azure OpenAI Docs Official Azure OpenAI documentation and setup Example Code Working example with function calling ​ Installation To use Azure OpenAI services, install the required dependency: Copy Ask AI pip install "pipecat-ai[azure]" You’ll need to set up your Azure OpenAI credentials: AZURE_CHATGPT_API_KEY - Your Azure OpenAI API key AZURE_CHATGPT_ENDPOINT - Your Azure OpenAI endpoint URL AZURE_CHATGPT_MODEL - Your model deployment name Get your credentials from the Azure Portal under your Azure OpenAI resource. ​ Frames ​ Input OpenAILLMContextFrame - Conversation context and history LLMMessagesFrame - Direct message list VisionImageRawFrame - Images for vision processing LLMUpdateSettingsFrame - Runtime parameter updates ​ Output LLMFullResponseStartFrame / LLMFullResponseEndFrame - Response boundaries LLMTextFrame - Streamed completion chunks FunctionCallInProgressFrame / FunctionCallResultFrame - Function call lifecycle ErrorFrame - API or processing errors ​ Azure vs OpenAI Differences Feature Azure OpenAI Standard OpenAI Authentication API key + endpoint API key only Deployment Custom deployment names Model names directly Compliance Enterprise SOC, HIPAA Standard compliance Regional Multiple Azure regions OpenAI regions only Pricing Azure billing integration OpenAI billing ​ Function Calling Function Calling Guide Learn how to implement function calling with standardized schemas, register handlers, manage context properly, and control execution flow in your conversational AI applications. ​ Context Management Context Management Guide Learn how to manage conversation context, handle message history, and integrate context aggregators for consistent conversational experiences. ​ Usage Example Copy Ask AI import os from pipecat.services.azure.llm import AzureLLMService from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext from pipecat.adapters.schemas.function_schema import FunctionSchema from pipecat.adapters.schemas.tools_schema import ToolsSchema # Configure Azure OpenAI service llm = AzureLLMService( api_key = os.getenv( "AZURE_CHATGPT_API_KEY" ), endpoint = os.getenv( "AZURE_CHATGPT_ENDPOINT" ), model = os.getenv( "AZURE_CHATGPT_MODEL" ), # Your deployment name params = AzureLLMService.InputParams( temperature = 0.7 , max_completion_tokens = 1000 ) ) # Define function for tool calling weather_function = FunctionSchema( name = "get_current_weather" , description = "Get current weather information" , properties = { "location" : { "type" : "string" , "description" : "City and state, e.g. San Francisco, CA" }, "format" : { "type" : "string" , "enum" : [ "celsius" , "fahrenheit" ], "description" : "Temperature unit to use" } }, required = [ "location" , "format" ] ) tools = ToolsSchema( standard_tools = [weather_function]) # Create context context = OpenAILLMContext( messages = [ { "role" : "system" , "content" : "You are a helpful assistant. Keep responses concise for voice output." } ], tools = tools ) # Create context aggregators context_aggregator = llm.create_context_aggregator(context) # Register function handler with event callback async def fetch_weather ( params ): location = params.arguments[ "location" ] await params.result_callback({ "conditions" : "sunny" , "temperature" : "75°F" }) llm.register_function( "get_current_weather" , fetch_weather) # Optional: Add function call feedback @llm.event_handler ( "on_function_calls_started" ) async def on_function_calls_started ( service , function_calls ): await tts.queue_frame(TTSSpeakFrame( "Let me check on that." )) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, context_aggregator.user(), llm, tts, transport.output(), context_aggregator.assistant() ]) ​ Metrics Inherits all OpenAI metrics capabilities: Time to First Byte (TTFB) - Response latency measurement Processing Duration - Total request processing time Token Usage - Prompt tokens, completion tokens, and totals Enable with: Copy Ask AI task = PipelineTask( pipeline, params = PipelineParams( enable_metrics = True , enable_usage_metrics = True ) ) ​ Additional Notes OpenAI Compatibility : Full compatibility with OpenAI API features and parameters Regional Deployment : Deploy in your preferred Azure region for compliance and latency Deployment Names : Use your Azure deployment name as the model parameter, not OpenAI model names Automatic Retries : Built-in retry logic handles transient Azure service issues AWS Bedrock Cerebras On this page Overview Installation Frames Input Output Azure vs OpenAI Differences Function Calling Context Management Usage Example Metrics Additional Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/llm_azure_f64aa812.txt b/llm_azure_f64aa812.txt
new file mode 100644
index 0000000000000000000000000000000000000000..e3dd9a383ae25d76972a75cd08b2a7c2ba071e49
--- /dev/null
+++ b/llm_azure_f64aa812.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/llm/azure#metrics
+Title: Azure - Pipecat
+==================================================
+
+Azure - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation LLM Azure Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Anthropic AWS Bedrock Azure Cerebras DeepSeek Fireworks AI Google Gemini Google Vertex AI Grok Groq NVIDIA NIM Ollama OpenAI OpenPipe OpenRouter Perplexity Qwen SambaNova Together AI Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview AzureLLMService provides access to Azure OpenAI’s language models through an OpenAI-compatible interface. It inherits from OpenAILLMService and supports streaming responses, function calling, and context management. API Reference Complete API documentation and method details Azure OpenAI Docs Official Azure OpenAI documentation and setup Example Code Working example with function calling ​ Installation To use Azure OpenAI services, install the required dependency: Copy Ask AI pip install "pipecat-ai[azure]" You’ll need to set up your Azure OpenAI credentials: AZURE_CHATGPT_API_KEY - Your Azure OpenAI API key AZURE_CHATGPT_ENDPOINT - Your Azure OpenAI endpoint URL AZURE_CHATGPT_MODEL - Your model deployment name Get your credentials from the Azure Portal under your Azure OpenAI resource. ​ Frames ​ Input OpenAILLMContextFrame - Conversation context and history LLMMessagesFrame - Direct message list VisionImageRawFrame - Images for vision processing LLMUpdateSettingsFrame - Runtime parameter updates ​ Output LLMFullResponseStartFrame / LLMFullResponseEndFrame - Response boundaries LLMTextFrame - Streamed completion chunks FunctionCallInProgressFrame / FunctionCallResultFrame - Function call lifecycle ErrorFrame - API or processing errors ​ Azure vs OpenAI Differences Feature Azure OpenAI Standard OpenAI Authentication API key + endpoint API key only Deployment Custom deployment names Model names directly Compliance Enterprise SOC, HIPAA Standard compliance Regional Multiple Azure regions OpenAI regions only Pricing Azure billing integration OpenAI billing ​ Function Calling Function Calling Guide Learn how to implement function calling with standardized schemas, register handlers, manage context properly, and control execution flow in your conversational AI applications. ​ Context Management Context Management Guide Learn how to manage conversation context, handle message history, and integrate context aggregators for consistent conversational experiences. ​ Usage Example Copy Ask AI import os from pipecat.services.azure.llm import AzureLLMService from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext from pipecat.adapters.schemas.function_schema import FunctionSchema from pipecat.adapters.schemas.tools_schema import ToolsSchema # Configure Azure OpenAI service llm = AzureLLMService( api_key = os.getenv( "AZURE_CHATGPT_API_KEY" ), endpoint = os.getenv( "AZURE_CHATGPT_ENDPOINT" ), model = os.getenv( "AZURE_CHATGPT_MODEL" ), # Your deployment name params = AzureLLMService.InputParams( temperature = 0.7 , max_completion_tokens = 1000 ) ) # Define function for tool calling weather_function = FunctionSchema( name = "get_current_weather" , description = "Get current weather information" , properties = { "location" : { "type" : "string" , "description" : "City and state, e.g. San Francisco, CA" }, "format" : { "type" : "string" , "enum" : [ "celsius" , "fahrenheit" ], "description" : "Temperature unit to use" } }, required = [ "location" , "format" ] ) tools = ToolsSchema( standard_tools = [weather_function]) # Create context context = OpenAILLMContext( messages = [ { "role" : "system" , "content" : "You are a helpful assistant. Keep responses concise for voice output." } ], tools = tools ) # Create context aggregators context_aggregator = llm.create_context_aggregator(context) # Register function handler with event callback async def fetch_weather ( params ): location = params.arguments[ "location" ] await params.result_callback({ "conditions" : "sunny" , "temperature" : "75°F" }) llm.register_function( "get_current_weather" , fetch_weather) # Optional: Add function call feedback @llm.event_handler ( "on_function_calls_started" ) async def on_function_calls_started ( service , function_calls ): await tts.queue_frame(TTSSpeakFrame( "Let me check on that." )) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, context_aggregator.user(), llm, tts, transport.output(), context_aggregator.assistant() ]) ​ Metrics Inherits all OpenAI metrics capabilities: Time to First Byte (TTFB) - Response latency measurement Processing Duration - Total request processing time Token Usage - Prompt tokens, completion tokens, and totals Enable with: Copy Ask AI task = PipelineTask( pipeline, params = PipelineParams( enable_metrics = True , enable_usage_metrics = True ) ) ​ Additional Notes OpenAI Compatibility : Full compatibility with OpenAI API features and parameters Regional Deployment : Deploy in your preferred Azure region for compliance and latency Deployment Names : Use your Azure deployment name as the model parameter, not OpenAI model names Automatic Retries : Built-in retry logic handles transient Azure service issues AWS Bedrock Cerebras On this page Overview Installation Frames Input Output Azure vs OpenAI Differences Function Calling Context Management Usage Example Metrics Additional Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/llm_cerebras_37a3d337.txt b/llm_cerebras_37a3d337.txt
new file mode 100644
index 0000000000000000000000000000000000000000..3636bf51e96ab17996e144a19c795ea8e5645c92
--- /dev/null
+++ b/llm_cerebras_37a3d337.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/llm/cerebras#usage-example
+Title: Cerebras - Pipecat
+==================================================
+
+Cerebras - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation LLM Cerebras Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Anthropic AWS Bedrock Azure Cerebras DeepSeek Fireworks AI Google Gemini Google Vertex AI Grok Groq NVIDIA NIM Ollama OpenAI OpenPipe OpenRouter Perplexity Qwen SambaNova Together AI Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview CerebrasLLMService provides access to Cerebras’s language models through an OpenAI-compatible interface. It inherits from OpenAILLMService and supports streaming responses, function calling, and context management. API Reference Complete API documentation and method details Cerebras Docs Official Cerebras inference API documentation Example Code Working example with function calling ​ Installation To use Cerebras services, install the required dependency: Copy Ask AI pip install "pipecat-ai[cerebras]" You’ll also need to set up your Cerebras API key as an environment variable: CEREBRAS_API_KEY . Get your API key from Cerebras Cloud . ​ Frames ​ Input OpenAILLMContextFrame - Conversation context and history LLMMessagesFrame - Direct message list VisionImageRawFrame - Images for vision processing LLMUpdateSettingsFrame - Runtime parameter updates ​ Output LLMFullResponseStartFrame / LLMFullResponseEndFrame - Response boundaries LLMTextFrame - Streamed completion chunks FunctionCallInProgressFrame / FunctionCallResultFrame - Function call lifecycle ErrorFrame - API or processing errors ​ Function Calling Function Calling Guide Learn how to implement function calling with standardized schemas, register handlers, manage context properly, and control execution flow in your conversational AI applications. ​ Context Management Context Management Guide Learn how to manage conversation context, handle message history, and integrate context aggregators for consistent conversational experiences. ​ Usage Example Copy Ask AI import os from pipecat.services.cerebras.llm import CerebrasLLMService from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext from pipecat.adapters.schemas.function_schema import FunctionSchema from pipecat.adapters.schemas.tools_schema import ToolsSchema # Configure the service llm = CerebrasLLMService( api_key = os.getenv( "CEREBRAS_API_KEY" ), model = "llama-3.3-70b" , params = CerebrasLLMService.InputParams( temperature = 0.7 , max_completion_tokens = 1000 ) ) # Define function for tool calling weather_function = FunctionSchema( name = "get_current_weather" , description = "Get current weather information" , properties = { "location" : { "type" : "string" , "description" : "City and state, e.g. San Francisco, CA" }, "format" : { "type" : "string" , "enum" : [ "celsius" , "fahrenheit" ], "description" : "Temperature unit to use" } }, required = [ "location" , "format" ] ) tools = ToolsSchema( standard_tools = [weather_function]) # Create context context = OpenAILLMContext( messages = [ { "role" : "system" , "content" : "You are a helpful assistant for weather information. Keep responses concise for voice output." } ], tools = tools ) # Create context aggregators context_aggregator = llm.create_context_aggregator(context) # Register function handler async def fetch_weather ( params ): location = params.arguments[ "location" ] await params.result_callback({ "conditions" : "sunny" , "temperature" : "75°F" }) llm.register_function( "get_current_weather" , fetch_weather) # Optional: Add function call feedback @llm.event_handler ( "on_function_calls_started" ) async def on_function_calls_started ( service , function_calls ): await tts.queue_frame(TTSSpeakFrame( "Let me check on that." )) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, context_aggregator.user(), llm, tts, transport.output(), context_aggregator.assistant() ]) ​ Metrics Inherits all OpenAI-compatible metrics: Time to First Byte (TTFB) - Ultra-low latency measurement Processing Duration - Total request processing time Token Usage - Prompt tokens, completion tokens, and totals Enable with: Copy Ask AI task = PipelineTask( pipeline, params = PipelineParams( enable_metrics = True , enable_usage_metrics = True ) ) ​ Additional Notes OpenAI Compatibility : Full compatibility with OpenAI API parameters and responses Streaming Responses : All responses are streamed for minimal latency Function Calling : Full support for OpenAI-style tool calling Open Source Models : Access to latest Llama models with commercial licensing Azure DeepSeek On this page Overview Installation Frames Input Output Function Calling Context Management Usage Example Metrics Additional Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/llm_cerebras_72c0ccd8.txt b/llm_cerebras_72c0ccd8.txt
new file mode 100644
index 0000000000000000000000000000000000000000..5a96b15d5b66f240129e810fd5a619367d776045
--- /dev/null
+++ b/llm_cerebras_72c0ccd8.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/llm/cerebras#frames
+Title: Cerebras - Pipecat
+==================================================
+
+Cerebras - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation LLM Cerebras Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Anthropic AWS Bedrock Azure Cerebras DeepSeek Fireworks AI Google Gemini Google Vertex AI Grok Groq NVIDIA NIM Ollama OpenAI OpenPipe OpenRouter Perplexity Qwen SambaNova Together AI Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview CerebrasLLMService provides access to Cerebras’s language models through an OpenAI-compatible interface. It inherits from OpenAILLMService and supports streaming responses, function calling, and context management. API Reference Complete API documentation and method details Cerebras Docs Official Cerebras inference API documentation Example Code Working example with function calling ​ Installation To use Cerebras services, install the required dependency: Copy Ask AI pip install "pipecat-ai[cerebras]" You’ll also need to set up your Cerebras API key as an environment variable: CEREBRAS_API_KEY . Get your API key from Cerebras Cloud . ​ Frames ​ Input OpenAILLMContextFrame - Conversation context and history LLMMessagesFrame - Direct message list VisionImageRawFrame - Images for vision processing LLMUpdateSettingsFrame - Runtime parameter updates ​ Output LLMFullResponseStartFrame / LLMFullResponseEndFrame - Response boundaries LLMTextFrame - Streamed completion chunks FunctionCallInProgressFrame / FunctionCallResultFrame - Function call lifecycle ErrorFrame - API or processing errors ​ Function Calling Function Calling Guide Learn how to implement function calling with standardized schemas, register handlers, manage context properly, and control execution flow in your conversational AI applications. ​ Context Management Context Management Guide Learn how to manage conversation context, handle message history, and integrate context aggregators for consistent conversational experiences. ​ Usage Example Copy Ask AI import os from pipecat.services.cerebras.llm import CerebrasLLMService from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext from pipecat.adapters.schemas.function_schema import FunctionSchema from pipecat.adapters.schemas.tools_schema import ToolsSchema # Configure the service llm = CerebrasLLMService( api_key = os.getenv( "CEREBRAS_API_KEY" ), model = "llama-3.3-70b" , params = CerebrasLLMService.InputParams( temperature = 0.7 , max_completion_tokens = 1000 ) ) # Define function for tool calling weather_function = FunctionSchema( name = "get_current_weather" , description = "Get current weather information" , properties = { "location" : { "type" : "string" , "description" : "City and state, e.g. San Francisco, CA" }, "format" : { "type" : "string" , "enum" : [ "celsius" , "fahrenheit" ], "description" : "Temperature unit to use" } }, required = [ "location" , "format" ] ) tools = ToolsSchema( standard_tools = [weather_function]) # Create context context = OpenAILLMContext( messages = [ { "role" : "system" , "content" : "You are a helpful assistant for weather information. Keep responses concise for voice output." } ], tools = tools ) # Create context aggregators context_aggregator = llm.create_context_aggregator(context) # Register function handler async def fetch_weather ( params ): location = params.arguments[ "location" ] await params.result_callback({ "conditions" : "sunny" , "temperature" : "75°F" }) llm.register_function( "get_current_weather" , fetch_weather) # Optional: Add function call feedback @llm.event_handler ( "on_function_calls_started" ) async def on_function_calls_started ( service , function_calls ): await tts.queue_frame(TTSSpeakFrame( "Let me check on that." )) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, context_aggregator.user(), llm, tts, transport.output(), context_aggregator.assistant() ]) ​ Metrics Inherits all OpenAI-compatible metrics: Time to First Byte (TTFB) - Ultra-low latency measurement Processing Duration - Total request processing time Token Usage - Prompt tokens, completion tokens, and totals Enable with: Copy Ask AI task = PipelineTask( pipeline, params = PipelineParams( enable_metrics = True , enable_usage_metrics = True ) ) ​ Additional Notes OpenAI Compatibility : Full compatibility with OpenAI API parameters and responses Streaming Responses : All responses are streamed for minimal latency Function Calling : Full support for OpenAI-style tool calling Open Source Models : Access to latest Llama models with commercial licensing Azure DeepSeek On this page Overview Installation Frames Input Output Function Calling Context Management Usage Example Metrics Additional Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/llm_deepseek_5997947f.txt b/llm_deepseek_5997947f.txt
new file mode 100644
index 0000000000000000000000000000000000000000..8f7b0d2e769ad8ac1d1b797c1481fbcf22e15142
--- /dev/null
+++ b/llm_deepseek_5997947f.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/llm/deepseek#input
+Title: DeepSeek - Pipecat
+==================================================
+
+DeepSeek - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation LLM DeepSeek Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Anthropic AWS Bedrock Azure Cerebras DeepSeek Fireworks AI Google Gemini Google Vertex AI Grok Groq NVIDIA NIM Ollama OpenAI OpenPipe OpenRouter Perplexity Qwen SambaNova Together AI Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview DeepSeekLLMService provides access to DeepSeek’s language models through an OpenAI-compatible interface. It inherits from OpenAILLMService and supports streaming responses, function calling, and context management. API Reference Complete API documentation and method details DeepSeek Docs Official DeepSeek API documentation and features Example Code Working example with function calling ​ Installation To use DeepSeek services, install the required dependency: Copy Ask AI pip install "pipecat-ai[deepseek]" You’ll also need to set up your DeepSeek API key as an environment variable: DEEPSEEK_API_KEY . Get your API key from DeepSeek Platform . ​ Frames ​ Input OpenAILLMContextFrame - Conversation context and history LLMMessagesFrame - Direct message list VisionImageRawFrame - Images for vision processing LLMUpdateSettingsFrame - Runtime parameter updates ​ Output LLMFullResponseStartFrame / LLMFullResponseEndFrame - Response boundaries LLMTextFrame - Streamed completion chunks FunctionCallInProgressFrame / FunctionCallResultFrame - Function call lifecycle ErrorFrame - API or processing errors ​ Function Calling Function Calling Guide Learn how to implement function calling with standardized schemas, register handlers, manage context properly, and control execution flow in your conversational AI applications. ​ Context Management Context Management Guide Learn how to manage conversation context, handle message history, and integrate context aggregators for consistent conversational experiences. ​ Usage Example Copy Ask AI import os from pipecat.services.deepseek.llm import DeepSeekLLMService from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext from pipecat.adapters.schemas.function_schema import FunctionSchema from pipecat.adapters.schemas.tools_schema import ToolsSchema # Configure DeepSeek service llm = DeepSeekLLMService( api_key = os.getenv( "DEEPSEEK_API_KEY" ), model = "deepseek-chat" , params = DeepSeekLLMService.InputParams( temperature = 0.7 , max_tokens = 1000 ) ) # Define function for tool calling weather_function = FunctionSchema( name = "get_current_weather" , description = "Get current weather information" , properties = { "location" : { "type" : "string" , "description" : "City and state, e.g. San Francisco, CA" }, "format" : { "type" : "string" , "enum" : [ "celsius" , "fahrenheit" ], "description" : "Temperature unit to use" } }, required = [ "location" , "format" ] ) tools = ToolsSchema( standard_tools = [weather_function]) # Create context with reasoning-focused system message context = OpenAILLMContext( messages = [ { "role" : "system" , "content" : """You are a helpful assistant with strong reasoning capabilities. Infer temperature units based on location unless specified. Provide logical, step-by-step responses.""" } ], tools = tools ) # Create context aggregators context_aggregator = llm.create_context_aggregator(context) # Register function handler with feedback async def fetch_weather ( params ): location = params.arguments[ "location" ] await params.result_callback({ "conditions" : "sunny" , "temperature" : "75°F" }) llm.register_function( "get_current_weather" , fetch_weather) # Optional: Add function call feedback @llm.event_handler ( "on_function_calls_started" ) async def on_function_calls_started ( service , function_calls ): await tts.queue_frame(TTSSpeakFrame( "Let me check on that." )) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, context_aggregator.user(), llm, tts, transport.output(), context_aggregator.assistant() ]) ​ Metrics Inherits all OpenAI metrics capabilities: Time to First Byte (TTFB) - Response latency measurement Processing Duration - Total request processing time Token Usage - Prompt tokens, completion tokens, and totals Enable with: Copy Ask AI task = PipelineTask( pipeline, params = PipelineParams( enable_metrics = True , enable_usage_metrics = True ) ) ​ Additional Notes OpenAI Compatibility : Full compatibility with OpenAI API features and parameters Cost Efficiency : Competitive pricing compared to other high-capability models Streaming Support : Real-time response streaming for low-latency applications Cerebras Fireworks AI On this page Overview Installation Frames Input Output Function Calling Context Management Usage Example Metrics Additional Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/llm_fireworks_491acc32.txt b/llm_fireworks_491acc32.txt
new file mode 100644
index 0000000000000000000000000000000000000000..f6b609b1f5f0d662f1c99ad1ce333a803ae4ddd4
--- /dev/null
+++ b/llm_fireworks_491acc32.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/llm/fireworks#usage-example
+Title: Fireworks AI - Pipecat
+==================================================
+
+Fireworks AI - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation LLM Fireworks AI Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Anthropic AWS Bedrock Azure Cerebras DeepSeek Fireworks AI Google Gemini Google Vertex AI Grok Groq NVIDIA NIM Ollama OpenAI OpenPipe OpenRouter Perplexity Qwen SambaNova Together AI Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview FireworksLLMService provides access to Fireworks AI’s language models through an OpenAI-compatible interface. It inherits from OpenAILLMService and supports streaming responses, function calling, and context management. API Reference Complete API documentation and method details Fireworks Docs Official Fireworks AI API documentation and features Example Code Working example with function calling ​ Installation To use Fireworks AI services, install the required dependency: Copy Ask AI pip install "pipecat-ai[fireworks]" You’ll also need to set up your Fireworks API key as an environment variable: FIREWORKS_API_KEY . Get your API key from Fireworks AI Console . ​ Frames ​ Input OpenAILLMContextFrame - Conversation context and history LLMMessagesFrame - Direct message list VisionImageRawFrame - Images for vision processing LLMUpdateSettingsFrame - Runtime parameter updates ​ Output LLMFullResponseStartFrame / LLMFullResponseEndFrame - Response boundaries LLMTextFrame - Streamed completion chunks FunctionCallInProgressFrame / FunctionCallResultFrame - Function call lifecycle ErrorFrame - API or processing errors ​ Function Calling Function Calling Guide Learn how to implement function calling with standardized schemas, register handlers, manage context properly, and control execution flow in your conversational AI applications. ​ Context Management Context Management Guide Learn how to manage conversation context, handle message history, and integrate context aggregators for consistent conversational experiences. ​ Usage Example Copy Ask AI import os from pipecat.services.fireworks.llm import FireworksLLMService from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext from pipecat.adapters.schemas.function_schema import FunctionSchema from pipecat.adapters.schemas.tools_schema import ToolsSchema # Configure Fireworks service llm = FireworksLLMService( api_key = os.getenv( "FIREWORKS_API_KEY" ), model = "accounts/fireworks/models/firefunction-v2" , # Optimized for function calling params = FireworksLLMService.InputParams( temperature = 0.7 , max_tokens = 1000 ) ) # Define function for tool calling weather_function = FunctionSchema( name = "get_current_weather" , description = "Get current weather information" , properties = { "location" : { "type" : "string" , "description" : "City and state, e.g. San Francisco, CA" }, "format" : { "type" : "string" , "enum" : [ "celsius" , "fahrenheit" ], "description" : "Temperature unit to use" } }, required = [ "location" , "format" ] ) tools = ToolsSchema( standard_tools = [weather_function]) # Create context context = OpenAILLMContext( messages = [ { "role" : "system" , "content" : """You are a helpful assistant optimized for voice interactions. Keep responses concise and avoid special characters for audio output.""" } ], tools = tools ) # Create context aggregators context_aggregator = llm.create_context_aggregator(context) # Register function handler with feedback async def fetch_weather ( params ): location = params.arguments[ "location" ] await params.result_callback({ "conditions" : "sunny" , "temperature" : "75°F" }) llm.register_function( "get_current_weather" , fetch_weather) # Optional: Add function call feedback @llm.event_handler ( "on_function_calls_started" ) async def on_function_calls_started ( service , function_calls ): await tts.queue_frame(TTSSpeakFrame( "Let me check on that." )) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, context_aggregator.user(), llm, tts, transport.output(), context_aggregator.assistant() ]) ​ Metrics Inherits all OpenAI metrics capabilities: Time to First Byte (TTFB) - Response latency measurement Processing Duration - Total request processing time Token Usage - Prompt tokens, completion tokens, and totals Enable with: Copy Ask AI task = PipelineTask( pipeline, params = PipelineParams( enable_metrics = True , enable_usage_metrics = True ) ) ​ Additional Notes OpenAI Compatibility : Full compatibility with OpenAI API features and parameters Function Calling : Specialized firefunction models optimized for tool use Cost Effective : Competitive pricing for open-source model inference DeepSeek Google Gemini On this page Overview Installation Frames Input Output Function Calling Context Management Usage Example Metrics Additional Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/llm_fireworks_a67b9d2e.txt b/llm_fireworks_a67b9d2e.txt
new file mode 100644
index 0000000000000000000000000000000000000000..95dda92bfe247cbf71aca4b4eee63da809649b46
--- /dev/null
+++ b/llm_fireworks_a67b9d2e.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/llm/fireworks#installation
+Title: Fireworks AI - Pipecat
+==================================================
+
+Fireworks AI - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation LLM Fireworks AI Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Anthropic AWS Bedrock Azure Cerebras DeepSeek Fireworks AI Google Gemini Google Vertex AI Grok Groq NVIDIA NIM Ollama OpenAI OpenPipe OpenRouter Perplexity Qwen SambaNova Together AI Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview FireworksLLMService provides access to Fireworks AI’s language models through an OpenAI-compatible interface. It inherits from OpenAILLMService and supports streaming responses, function calling, and context management. API Reference Complete API documentation and method details Fireworks Docs Official Fireworks AI API documentation and features Example Code Working example with function calling ​ Installation To use Fireworks AI services, install the required dependency: Copy Ask AI pip install "pipecat-ai[fireworks]" You’ll also need to set up your Fireworks API key as an environment variable: FIREWORKS_API_KEY . Get your API key from Fireworks AI Console . ​ Frames ​ Input OpenAILLMContextFrame - Conversation context and history LLMMessagesFrame - Direct message list VisionImageRawFrame - Images for vision processing LLMUpdateSettingsFrame - Runtime parameter updates ​ Output LLMFullResponseStartFrame / LLMFullResponseEndFrame - Response boundaries LLMTextFrame - Streamed completion chunks FunctionCallInProgressFrame / FunctionCallResultFrame - Function call lifecycle ErrorFrame - API or processing errors ​ Function Calling Function Calling Guide Learn how to implement function calling with standardized schemas, register handlers, manage context properly, and control execution flow in your conversational AI applications. ​ Context Management Context Management Guide Learn how to manage conversation context, handle message history, and integrate context aggregators for consistent conversational experiences. ​ Usage Example Copy Ask AI import os from pipecat.services.fireworks.llm import FireworksLLMService from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext from pipecat.adapters.schemas.function_schema import FunctionSchema from pipecat.adapters.schemas.tools_schema import ToolsSchema # Configure Fireworks service llm = FireworksLLMService( api_key = os.getenv( "FIREWORKS_API_KEY" ), model = "accounts/fireworks/models/firefunction-v2" , # Optimized for function calling params = FireworksLLMService.InputParams( temperature = 0.7 , max_tokens = 1000 ) ) # Define function for tool calling weather_function = FunctionSchema( name = "get_current_weather" , description = "Get current weather information" , properties = { "location" : { "type" : "string" , "description" : "City and state, e.g. San Francisco, CA" }, "format" : { "type" : "string" , "enum" : [ "celsius" , "fahrenheit" ], "description" : "Temperature unit to use" } }, required = [ "location" , "format" ] ) tools = ToolsSchema( standard_tools = [weather_function]) # Create context context = OpenAILLMContext( messages = [ { "role" : "system" , "content" : """You are a helpful assistant optimized for voice interactions. Keep responses concise and avoid special characters for audio output.""" } ], tools = tools ) # Create context aggregators context_aggregator = llm.create_context_aggregator(context) # Register function handler with feedback async def fetch_weather ( params ): location = params.arguments[ "location" ] await params.result_callback({ "conditions" : "sunny" , "temperature" : "75°F" }) llm.register_function( "get_current_weather" , fetch_weather) # Optional: Add function call feedback @llm.event_handler ( "on_function_calls_started" ) async def on_function_calls_started ( service , function_calls ): await tts.queue_frame(TTSSpeakFrame( "Let me check on that." )) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, context_aggregator.user(), llm, tts, transport.output(), context_aggregator.assistant() ]) ​ Metrics Inherits all OpenAI metrics capabilities: Time to First Byte (TTFB) - Response latency measurement Processing Duration - Total request processing time Token Usage - Prompt tokens, completion tokens, and totals Enable with: Copy Ask AI task = PipelineTask( pipeline, params = PipelineParams( enable_metrics = True , enable_usage_metrics = True ) ) ​ Additional Notes OpenAI Compatibility : Full compatibility with OpenAI API features and parameters Function Calling : Specialized firefunction models optimized for tool use Cost Effective : Competitive pricing for open-source model inference DeepSeek Google Gemini On this page Overview Installation Frames Input Output Function Calling Context Management Usage Example Metrics Additional Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/llm_fireworks_ed73fb74.txt b/llm_fireworks_ed73fb74.txt
new file mode 100644
index 0000000000000000000000000000000000000000..f610bf25d22b3dd736bce0ab38463478c19b84ed
--- /dev/null
+++ b/llm_fireworks_ed73fb74.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/llm/fireworks#context-management
+Title: Fireworks AI - Pipecat
+==================================================
+
+Fireworks AI - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation LLM Fireworks AI Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Anthropic AWS Bedrock Azure Cerebras DeepSeek Fireworks AI Google Gemini Google Vertex AI Grok Groq NVIDIA NIM Ollama OpenAI OpenPipe OpenRouter Perplexity Qwen SambaNova Together AI Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview FireworksLLMService provides access to Fireworks AI’s language models through an OpenAI-compatible interface. It inherits from OpenAILLMService and supports streaming responses, function calling, and context management. API Reference Complete API documentation and method details Fireworks Docs Official Fireworks AI API documentation and features Example Code Working example with function calling ​ Installation To use Fireworks AI services, install the required dependency: Copy Ask AI pip install "pipecat-ai[fireworks]" You’ll also need to set up your Fireworks API key as an environment variable: FIREWORKS_API_KEY . Get your API key from Fireworks AI Console . ​ Frames ​ Input OpenAILLMContextFrame - Conversation context and history LLMMessagesFrame - Direct message list VisionImageRawFrame - Images for vision processing LLMUpdateSettingsFrame - Runtime parameter updates ​ Output LLMFullResponseStartFrame / LLMFullResponseEndFrame - Response boundaries LLMTextFrame - Streamed completion chunks FunctionCallInProgressFrame / FunctionCallResultFrame - Function call lifecycle ErrorFrame - API or processing errors ​ Function Calling Function Calling Guide Learn how to implement function calling with standardized schemas, register handlers, manage context properly, and control execution flow in your conversational AI applications. ​ Context Management Context Management Guide Learn how to manage conversation context, handle message history, and integrate context aggregators for consistent conversational experiences. ​ Usage Example Copy Ask AI import os from pipecat.services.fireworks.llm import FireworksLLMService from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext from pipecat.adapters.schemas.function_schema import FunctionSchema from pipecat.adapters.schemas.tools_schema import ToolsSchema # Configure Fireworks service llm = FireworksLLMService( api_key = os.getenv( "FIREWORKS_API_KEY" ), model = "accounts/fireworks/models/firefunction-v2" , # Optimized for function calling params = FireworksLLMService.InputParams( temperature = 0.7 , max_tokens = 1000 ) ) # Define function for tool calling weather_function = FunctionSchema( name = "get_current_weather" , description = "Get current weather information" , properties = { "location" : { "type" : "string" , "description" : "City and state, e.g. San Francisco, CA" }, "format" : { "type" : "string" , "enum" : [ "celsius" , "fahrenheit" ], "description" : "Temperature unit to use" } }, required = [ "location" , "format" ] ) tools = ToolsSchema( standard_tools = [weather_function]) # Create context context = OpenAILLMContext( messages = [ { "role" : "system" , "content" : """You are a helpful assistant optimized for voice interactions. Keep responses concise and avoid special characters for audio output.""" } ], tools = tools ) # Create context aggregators context_aggregator = llm.create_context_aggregator(context) # Register function handler with feedback async def fetch_weather ( params ): location = params.arguments[ "location" ] await params.result_callback({ "conditions" : "sunny" , "temperature" : "75°F" }) llm.register_function( "get_current_weather" , fetch_weather) # Optional: Add function call feedback @llm.event_handler ( "on_function_calls_started" ) async def on_function_calls_started ( service , function_calls ): await tts.queue_frame(TTSSpeakFrame( "Let me check on that." )) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, context_aggregator.user(), llm, tts, transport.output(), context_aggregator.assistant() ]) ​ Metrics Inherits all OpenAI metrics capabilities: Time to First Byte (TTFB) - Response latency measurement Processing Duration - Total request processing time Token Usage - Prompt tokens, completion tokens, and totals Enable with: Copy Ask AI task = PipelineTask( pipeline, params = PipelineParams( enable_metrics = True , enable_usage_metrics = True ) ) ​ Additional Notes OpenAI Compatibility : Full compatibility with OpenAI API features and parameters Function Calling : Specialized firefunction models optimized for tool use Cost Effective : Competitive pricing for open-source model inference DeepSeek Google Gemini On this page Overview Installation Frames Input Output Function Calling Context Management Usage Example Metrics Additional Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/llm_fireworks_fb18e3a6.txt b/llm_fireworks_fb18e3a6.txt
new file mode 100644
index 0000000000000000000000000000000000000000..fd6e80c4cc6248ef560b52aceec2ad7f64d490c2
--- /dev/null
+++ b/llm_fireworks_fb18e3a6.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/llm/fireworks#output
+Title: Fireworks AI - Pipecat
+==================================================
+
+Fireworks AI - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation LLM Fireworks AI Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Anthropic AWS Bedrock Azure Cerebras DeepSeek Fireworks AI Google Gemini Google Vertex AI Grok Groq NVIDIA NIM Ollama OpenAI OpenPipe OpenRouter Perplexity Qwen SambaNova Together AI Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview FireworksLLMService provides access to Fireworks AI’s language models through an OpenAI-compatible interface. It inherits from OpenAILLMService and supports streaming responses, function calling, and context management. API Reference Complete API documentation and method details Fireworks Docs Official Fireworks AI API documentation and features Example Code Working example with function calling ​ Installation To use Fireworks AI services, install the required dependency: Copy Ask AI pip install "pipecat-ai[fireworks]" You’ll also need to set up your Fireworks API key as an environment variable: FIREWORKS_API_KEY . Get your API key from Fireworks AI Console . ​ Frames ​ Input OpenAILLMContextFrame - Conversation context and history LLMMessagesFrame - Direct message list VisionImageRawFrame - Images for vision processing LLMUpdateSettingsFrame - Runtime parameter updates ​ Output LLMFullResponseStartFrame / LLMFullResponseEndFrame - Response boundaries LLMTextFrame - Streamed completion chunks FunctionCallInProgressFrame / FunctionCallResultFrame - Function call lifecycle ErrorFrame - API or processing errors ​ Function Calling Function Calling Guide Learn how to implement function calling with standardized schemas, register handlers, manage context properly, and control execution flow in your conversational AI applications. ​ Context Management Context Management Guide Learn how to manage conversation context, handle message history, and integrate context aggregators for consistent conversational experiences. ​ Usage Example Copy Ask AI import os from pipecat.services.fireworks.llm import FireworksLLMService from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext from pipecat.adapters.schemas.function_schema import FunctionSchema from pipecat.adapters.schemas.tools_schema import ToolsSchema # Configure Fireworks service llm = FireworksLLMService( api_key = os.getenv( "FIREWORKS_API_KEY" ), model = "accounts/fireworks/models/firefunction-v2" , # Optimized for function calling params = FireworksLLMService.InputParams( temperature = 0.7 , max_tokens = 1000 ) ) # Define function for tool calling weather_function = FunctionSchema( name = "get_current_weather" , description = "Get current weather information" , properties = { "location" : { "type" : "string" , "description" : "City and state, e.g. San Francisco, CA" }, "format" : { "type" : "string" , "enum" : [ "celsius" , "fahrenheit" ], "description" : "Temperature unit to use" } }, required = [ "location" , "format" ] ) tools = ToolsSchema( standard_tools = [weather_function]) # Create context context = OpenAILLMContext( messages = [ { "role" : "system" , "content" : """You are a helpful assistant optimized for voice interactions. Keep responses concise and avoid special characters for audio output.""" } ], tools = tools ) # Create context aggregators context_aggregator = llm.create_context_aggregator(context) # Register function handler with feedback async def fetch_weather ( params ): location = params.arguments[ "location" ] await params.result_callback({ "conditions" : "sunny" , "temperature" : "75°F" }) llm.register_function( "get_current_weather" , fetch_weather) # Optional: Add function call feedback @llm.event_handler ( "on_function_calls_started" ) async def on_function_calls_started ( service , function_calls ): await tts.queue_frame(TTSSpeakFrame( "Let me check on that." )) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, context_aggregator.user(), llm, tts, transport.output(), context_aggregator.assistant() ]) ​ Metrics Inherits all OpenAI metrics capabilities: Time to First Byte (TTFB) - Response latency measurement Processing Duration - Total request processing time Token Usage - Prompt tokens, completion tokens, and totals Enable with: Copy Ask AI task = PipelineTask( pipeline, params = PipelineParams( enable_metrics = True , enable_usage_metrics = True ) ) ​ Additional Notes OpenAI Compatibility : Full compatibility with OpenAI API features and parameters Function Calling : Specialized firefunction models optimized for tool use Cost Effective : Competitive pricing for open-source model inference DeepSeek Google Gemini On this page Overview Installation Frames Input Output Function Calling Context Management Usage Example Metrics Additional Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/llm_gemini_4c644b1d.txt b/llm_gemini_4c644b1d.txt
new file mode 100644
index 0000000000000000000000000000000000000000..d381113a0d418762c2e9c9fe10b434a4e0cbee9d
--- /dev/null
+++ b/llm_gemini_4c644b1d.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/llm/gemini#usage-example
+Title: Google Gemini - Pipecat
+==================================================
+
+Google Gemini - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation LLM Google Gemini Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Anthropic AWS Bedrock Azure Cerebras DeepSeek Fireworks AI Google Gemini Google Vertex AI Grok Groq NVIDIA NIM Ollama OpenAI OpenPipe OpenRouter Perplexity Qwen SambaNova Together AI Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview GoogleLLMService provides integration with Google’s Gemini models, supporting streaming responses, function calling, and multimodal inputs. It includes specialized context handling for Google’s message format while maintaining compatibility with OpenAI-style contexts. API Reference Complete API documentation and method details Gemini Docs Official Google Gemini API documentation and features Example Code Working example with function calling ​ Installation To use GoogleLLMService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[google]" You’ll also need to set up your Google API key as an environment variable: GOOGLE_API_KEY . Get your API key from Google AI Studio . ​ Frames ​ Input OpenAILLMContextFrame - Conversation context and history LLMMessagesFrame - Direct message list VisionImageRawFrame - Images for vision processing LLMUpdateSettingsFrame - Runtime parameter updates ​ Output LLMFullResponseStartFrame / LLMFullResponseEndFrame - Response boundaries LLMTextFrame - Streamed completion chunks LLMSearchResponseFrame - Search grounding results with citations FunctionCallInProgressFrame / FunctionCallResultFrame - Function call lifecycle ErrorFrame - API or processing errors ​ Search Grounding Google Gemini’s search grounding feature enables real-time web search integration, allowing the model to access current information and provide citations. This is particularly valuable for applications requiring up-to-date information. ​ Enabling Search Grounding Copy Ask AI # Configure search grounding tool search_tool = { "google_search_retrieval" : { "dynamic_retrieval_config" : { "mode" : "MODE_DYNAMIC" , "dynamic_threshold" : 0.3 , # Lower = more frequent grounding } } } # Initialize with search grounding llm = GoogleLLMService( api_key = os.getenv( "GOOGLE_API_KEY" ), model = "gemini-1.5-flash-002" , system_instruction = "You are a helpful assistant with access to current information." , tools = [search_tool] ) ​ Handling Search Results Search grounding produces LLMSearchResponseFrame with detailed citation information: Copy Ask AI @pipeline.event_handler ( "llm_search_response" ) async def handle_search_response ( frame ): print ( f "Search result: { frame.search_result } " ) print ( f "Sources: { len (frame.origins) } citations" ) for origin in frame.origins: print ( f "- { origin[ 'site_title' ] } : { origin[ 'site_uri' ] } " ) ​ Function Calling Function Calling Guide Learn how to implement function calling with standardized schemas, register handlers, manage context properly, and control execution flow in your conversational AI applications. ​ Context Management Context Management Guide Learn how to manage conversation context, handle message history, and integrate context aggregators for consistent conversational experiences. ​ Usage Example Copy Ask AI import os from pipecat.services.google.llm import GoogleLLMService from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext from pipecat.adapters.schemas.function_schema import FunctionSchema from pipecat.adapters.schemas.tools_schema import ToolsSchema # Configure Gemini service with search grounding search_tool = { "google_search_retrieval" : { "dynamic_retrieval_config" : { "mode" : "MODE_DYNAMIC" , "dynamic_threshold" : 0.3 } } } llm = GoogleLLMService( api_key = os.getenv( "GOOGLE_API_KEY" ), model = "gemini-2.0-flash" , system_instruction = """You are a helpful assistant with access to current information. When users ask about recent events, use search to provide accurate, up-to-date information.""" , tools = [search_tool], params = GoogleLLMService.InputParams( temperature = 0.7 , max_tokens = 1000 ) ) # Define function for tool calling weather_function = FunctionSchema( name = "get_weather" , description = "Get current weather information" , properties = { "location" : { "type" : "string" , "description" : "City and state, e.g. San Francisco, CA" } }, required = [ "location" ] ) # Define image capture function for multimodal capabilities image_function = FunctionSchema( name = "get_image" , description = "Capture and analyze an image from the video stream" , properties = { "question" : { "type" : "string" , "description" : "Question about what to analyze in the image" } }, required = [ "question" ] ) tools = ToolsSchema( standard_tools = [weather_function, image_function]) # Create context with multimodal system prompt context = OpenAILLMContext( messages = [ { "role" : "system" , "content" : """You are a helpful assistant with access to current information and vision capabilities. You can answer questions about weather, analyze images from video streams, and search for current information. Keep responses concise for voice output.""" }, { "role" : "user" , "content" : "Hello! What can you help me with?" } ], tools = tools ) # Create context aggregators context_aggregator = llm.create_context_aggregator(context) # Register function handlers async def get_weather ( params ): location = params.arguments[ "location" ] await params.result_callback( f "Weather in { location } : 72°F and sunny" ) async def get_image ( params ): question = params.arguments[ "question" ] # Request image from video stream await params.llm.request_image_frame( user_id = client_id, function_name = params.function_name, tool_call_id = params.tool_call_id, text_content = question ) await params.result_callback( f "Analyzing image for: { question } " ) llm.register_function( "get_weather" , get_weather) llm.register_function( "get_image" , get_image) # Optional: Add function call feedback @llm.event_handler ( "on_function_calls_started" ) async def on_function_calls_started ( service , function_calls ): await tts.queue_frame(TTSSpeakFrame( "Let me check on that." )) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, context_aggregator.user(), llm, tts, transport.output(), context_aggregator.assistant() ]) ​ Metrics Google Gemini provides comprehensive usage tracking: Time to First Byte (TTFB) - Response latency measurement Processing Duration - Total request processing time Token Usage - Prompt tokens, completion tokens, and totals Enable with: Copy Ask AI task = PipelineTask( pipeline, params = PipelineParams( enable_metrics = True , enable_usage_metrics = True ) ) ​ Additional Notes Multimodal Capabilities : Native support for text, images, audio, and video processing Search Grounding : Real-time web search with automatic citation and source attribution System Instructions : Handle system messages differently than OpenAI - set during initialization Vision Functions : Built-in support for image capture and analysis from video streams Fireworks AI Google Vertex AI On this page Overview Installation Frames Input Output Search Grounding Enabling Search Grounding Handling Search Results Function Calling Context Management Usage Example Metrics Additional Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/llm_gemini_5ca2f93b.txt b/llm_gemini_5ca2f93b.txt
new file mode 100644
index 0000000000000000000000000000000000000000..aaa7a0112f3c372a5f161dd3267b850c5f121c0a
--- /dev/null
+++ b/llm_gemini_5ca2f93b.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/llm/gemini#search-grounding
+Title: Google Gemini - Pipecat
+==================================================
+
+Google Gemini - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation LLM Google Gemini Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Anthropic AWS Bedrock Azure Cerebras DeepSeek Fireworks AI Google Gemini Google Vertex AI Grok Groq NVIDIA NIM Ollama OpenAI OpenPipe OpenRouter Perplexity Qwen SambaNova Together AI Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview GoogleLLMService provides integration with Google’s Gemini models, supporting streaming responses, function calling, and multimodal inputs. It includes specialized context handling for Google’s message format while maintaining compatibility with OpenAI-style contexts. API Reference Complete API documentation and method details Gemini Docs Official Google Gemini API documentation and features Example Code Working example with function calling ​ Installation To use GoogleLLMService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[google]" You’ll also need to set up your Google API key as an environment variable: GOOGLE_API_KEY . Get your API key from Google AI Studio . ​ Frames ​ Input OpenAILLMContextFrame - Conversation context and history LLMMessagesFrame - Direct message list VisionImageRawFrame - Images for vision processing LLMUpdateSettingsFrame - Runtime parameter updates ​ Output LLMFullResponseStartFrame / LLMFullResponseEndFrame - Response boundaries LLMTextFrame - Streamed completion chunks LLMSearchResponseFrame - Search grounding results with citations FunctionCallInProgressFrame / FunctionCallResultFrame - Function call lifecycle ErrorFrame - API or processing errors ​ Search Grounding Google Gemini’s search grounding feature enables real-time web search integration, allowing the model to access current information and provide citations. This is particularly valuable for applications requiring up-to-date information. ​ Enabling Search Grounding Copy Ask AI # Configure search grounding tool search_tool = { "google_search_retrieval" : { "dynamic_retrieval_config" : { "mode" : "MODE_DYNAMIC" , "dynamic_threshold" : 0.3 , # Lower = more frequent grounding } } } # Initialize with search grounding llm = GoogleLLMService( api_key = os.getenv( "GOOGLE_API_KEY" ), model = "gemini-1.5-flash-002" , system_instruction = "You are a helpful assistant with access to current information." , tools = [search_tool] ) ​ Handling Search Results Search grounding produces LLMSearchResponseFrame with detailed citation information: Copy Ask AI @pipeline.event_handler ( "llm_search_response" ) async def handle_search_response ( frame ): print ( f "Search result: { frame.search_result } " ) print ( f "Sources: { len (frame.origins) } citations" ) for origin in frame.origins: print ( f "- { origin[ 'site_title' ] } : { origin[ 'site_uri' ] } " ) ​ Function Calling Function Calling Guide Learn how to implement function calling with standardized schemas, register handlers, manage context properly, and control execution flow in your conversational AI applications. ​ Context Management Context Management Guide Learn how to manage conversation context, handle message history, and integrate context aggregators for consistent conversational experiences. ​ Usage Example Copy Ask AI import os from pipecat.services.google.llm import GoogleLLMService from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext from pipecat.adapters.schemas.function_schema import FunctionSchema from pipecat.adapters.schemas.tools_schema import ToolsSchema # Configure Gemini service with search grounding search_tool = { "google_search_retrieval" : { "dynamic_retrieval_config" : { "mode" : "MODE_DYNAMIC" , "dynamic_threshold" : 0.3 } } } llm = GoogleLLMService( api_key = os.getenv( "GOOGLE_API_KEY" ), model = "gemini-2.0-flash" , system_instruction = """You are a helpful assistant with access to current information. When users ask about recent events, use search to provide accurate, up-to-date information.""" , tools = [search_tool], params = GoogleLLMService.InputParams( temperature = 0.7 , max_tokens = 1000 ) ) # Define function for tool calling weather_function = FunctionSchema( name = "get_weather" , description = "Get current weather information" , properties = { "location" : { "type" : "string" , "description" : "City and state, e.g. San Francisco, CA" } }, required = [ "location" ] ) # Define image capture function for multimodal capabilities image_function = FunctionSchema( name = "get_image" , description = "Capture and analyze an image from the video stream" , properties = { "question" : { "type" : "string" , "description" : "Question about what to analyze in the image" } }, required = [ "question" ] ) tools = ToolsSchema( standard_tools = [weather_function, image_function]) # Create context with multimodal system prompt context = OpenAILLMContext( messages = [ { "role" : "system" , "content" : """You are a helpful assistant with access to current information and vision capabilities. You can answer questions about weather, analyze images from video streams, and search for current information. Keep responses concise for voice output.""" }, { "role" : "user" , "content" : "Hello! What can you help me with?" } ], tools = tools ) # Create context aggregators context_aggregator = llm.create_context_aggregator(context) # Register function handlers async def get_weather ( params ): location = params.arguments[ "location" ] await params.result_callback( f "Weather in { location } : 72°F and sunny" ) async def get_image ( params ): question = params.arguments[ "question" ] # Request image from video stream await params.llm.request_image_frame( user_id = client_id, function_name = params.function_name, tool_call_id = params.tool_call_id, text_content = question ) await params.result_callback( f "Analyzing image for: { question } " ) llm.register_function( "get_weather" , get_weather) llm.register_function( "get_image" , get_image) # Optional: Add function call feedback @llm.event_handler ( "on_function_calls_started" ) async def on_function_calls_started ( service , function_calls ): await tts.queue_frame(TTSSpeakFrame( "Let me check on that." )) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, context_aggregator.user(), llm, tts, transport.output(), context_aggregator.assistant() ]) ​ Metrics Google Gemini provides comprehensive usage tracking: Time to First Byte (TTFB) - Response latency measurement Processing Duration - Total request processing time Token Usage - Prompt tokens, completion tokens, and totals Enable with: Copy Ask AI task = PipelineTask( pipeline, params = PipelineParams( enable_metrics = True , enable_usage_metrics = True ) ) ​ Additional Notes Multimodal Capabilities : Native support for text, images, audio, and video processing Search Grounding : Real-time web search with automatic citation and source attribution System Instructions : Handle system messages differently than OpenAI - set during initialization Vision Functions : Built-in support for image capture and analysis from video streams Fireworks AI Google Vertex AI On this page Overview Installation Frames Input Output Search Grounding Enabling Search Grounding Handling Search Results Function Calling Context Management Usage Example Metrics Additional Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/llm_gemini_866dc380.txt b/llm_gemini_866dc380.txt
new file mode 100644
index 0000000000000000000000000000000000000000..14181ad46d279e067161df0147b66808b3bf46c9
--- /dev/null
+++ b/llm_gemini_866dc380.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/llm/gemini#frames
+Title: Google Gemini - Pipecat
+==================================================
+
+Google Gemini - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation LLM Google Gemini Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Anthropic AWS Bedrock Azure Cerebras DeepSeek Fireworks AI Google Gemini Google Vertex AI Grok Groq NVIDIA NIM Ollama OpenAI OpenPipe OpenRouter Perplexity Qwen SambaNova Together AI Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview GoogleLLMService provides integration with Google’s Gemini models, supporting streaming responses, function calling, and multimodal inputs. It includes specialized context handling for Google’s message format while maintaining compatibility with OpenAI-style contexts. API Reference Complete API documentation and method details Gemini Docs Official Google Gemini API documentation and features Example Code Working example with function calling ​ Installation To use GoogleLLMService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[google]" You’ll also need to set up your Google API key as an environment variable: GOOGLE_API_KEY . Get your API key from Google AI Studio . ​ Frames ​ Input OpenAILLMContextFrame - Conversation context and history LLMMessagesFrame - Direct message list VisionImageRawFrame - Images for vision processing LLMUpdateSettingsFrame - Runtime parameter updates ​ Output LLMFullResponseStartFrame / LLMFullResponseEndFrame - Response boundaries LLMTextFrame - Streamed completion chunks LLMSearchResponseFrame - Search grounding results with citations FunctionCallInProgressFrame / FunctionCallResultFrame - Function call lifecycle ErrorFrame - API or processing errors ​ Search Grounding Google Gemini’s search grounding feature enables real-time web search integration, allowing the model to access current information and provide citations. This is particularly valuable for applications requiring up-to-date information. ​ Enabling Search Grounding Copy Ask AI # Configure search grounding tool search_tool = { "google_search_retrieval" : { "dynamic_retrieval_config" : { "mode" : "MODE_DYNAMIC" , "dynamic_threshold" : 0.3 , # Lower = more frequent grounding } } } # Initialize with search grounding llm = GoogleLLMService( api_key = os.getenv( "GOOGLE_API_KEY" ), model = "gemini-1.5-flash-002" , system_instruction = "You are a helpful assistant with access to current information." , tools = [search_tool] ) ​ Handling Search Results Search grounding produces LLMSearchResponseFrame with detailed citation information: Copy Ask AI @pipeline.event_handler ( "llm_search_response" ) async def handle_search_response ( frame ): print ( f "Search result: { frame.search_result } " ) print ( f "Sources: { len (frame.origins) } citations" ) for origin in frame.origins: print ( f "- { origin[ 'site_title' ] } : { origin[ 'site_uri' ] } " ) ​ Function Calling Function Calling Guide Learn how to implement function calling with standardized schemas, register handlers, manage context properly, and control execution flow in your conversational AI applications. ​ Context Management Context Management Guide Learn how to manage conversation context, handle message history, and integrate context aggregators for consistent conversational experiences. ​ Usage Example Copy Ask AI import os from pipecat.services.google.llm import GoogleLLMService from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext from pipecat.adapters.schemas.function_schema import FunctionSchema from pipecat.adapters.schemas.tools_schema import ToolsSchema # Configure Gemini service with search grounding search_tool = { "google_search_retrieval" : { "dynamic_retrieval_config" : { "mode" : "MODE_DYNAMIC" , "dynamic_threshold" : 0.3 } } } llm = GoogleLLMService( api_key = os.getenv( "GOOGLE_API_KEY" ), model = "gemini-2.0-flash" , system_instruction = """You are a helpful assistant with access to current information. When users ask about recent events, use search to provide accurate, up-to-date information.""" , tools = [search_tool], params = GoogleLLMService.InputParams( temperature = 0.7 , max_tokens = 1000 ) ) # Define function for tool calling weather_function = FunctionSchema( name = "get_weather" , description = "Get current weather information" , properties = { "location" : { "type" : "string" , "description" : "City and state, e.g. San Francisco, CA" } }, required = [ "location" ] ) # Define image capture function for multimodal capabilities image_function = FunctionSchema( name = "get_image" , description = "Capture and analyze an image from the video stream" , properties = { "question" : { "type" : "string" , "description" : "Question about what to analyze in the image" } }, required = [ "question" ] ) tools = ToolsSchema( standard_tools = [weather_function, image_function]) # Create context with multimodal system prompt context = OpenAILLMContext( messages = [ { "role" : "system" , "content" : """You are a helpful assistant with access to current information and vision capabilities. You can answer questions about weather, analyze images from video streams, and search for current information. Keep responses concise for voice output.""" }, { "role" : "user" , "content" : "Hello! What can you help me with?" } ], tools = tools ) # Create context aggregators context_aggregator = llm.create_context_aggregator(context) # Register function handlers async def get_weather ( params ): location = params.arguments[ "location" ] await params.result_callback( f "Weather in { location } : 72°F and sunny" ) async def get_image ( params ): question = params.arguments[ "question" ] # Request image from video stream await params.llm.request_image_frame( user_id = client_id, function_name = params.function_name, tool_call_id = params.tool_call_id, text_content = question ) await params.result_callback( f "Analyzing image for: { question } " ) llm.register_function( "get_weather" , get_weather) llm.register_function( "get_image" , get_image) # Optional: Add function call feedback @llm.event_handler ( "on_function_calls_started" ) async def on_function_calls_started ( service , function_calls ): await tts.queue_frame(TTSSpeakFrame( "Let me check on that." )) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, context_aggregator.user(), llm, tts, transport.output(), context_aggregator.assistant() ]) ​ Metrics Google Gemini provides comprehensive usage tracking: Time to First Byte (TTFB) - Response latency measurement Processing Duration - Total request processing time Token Usage - Prompt tokens, completion tokens, and totals Enable with: Copy Ask AI task = PipelineTask( pipeline, params = PipelineParams( enable_metrics = True , enable_usage_metrics = True ) ) ​ Additional Notes Multimodal Capabilities : Native support for text, images, audio, and video processing Search Grounding : Real-time web search with automatic citation and source attribution System Instructions : Handle system messages differently than OpenAI - set during initialization Vision Functions : Built-in support for image capture and analysis from video streams Fireworks AI Google Vertex AI On this page Overview Installation Frames Input Output Search Grounding Enabling Search Grounding Handling Search Results Function Calling Context Management Usage Example Metrics Additional Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/llm_gemini_e902aaab.txt b/llm_gemini_e902aaab.txt
new file mode 100644
index 0000000000000000000000000000000000000000..154f6f3e7a9cab64e1cdf09b54ee1a4f61079015
--- /dev/null
+++ b/llm_gemini_e902aaab.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/llm/gemini#context-management
+Title: Google Gemini - Pipecat
+==================================================
+
+Google Gemini - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation LLM Google Gemini Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Anthropic AWS Bedrock Azure Cerebras DeepSeek Fireworks AI Google Gemini Google Vertex AI Grok Groq NVIDIA NIM Ollama OpenAI OpenPipe OpenRouter Perplexity Qwen SambaNova Together AI Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview GoogleLLMService provides integration with Google’s Gemini models, supporting streaming responses, function calling, and multimodal inputs. It includes specialized context handling for Google’s message format while maintaining compatibility with OpenAI-style contexts. API Reference Complete API documentation and method details Gemini Docs Official Google Gemini API documentation and features Example Code Working example with function calling ​ Installation To use GoogleLLMService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[google]" You’ll also need to set up your Google API key as an environment variable: GOOGLE_API_KEY . Get your API key from Google AI Studio . ​ Frames ​ Input OpenAILLMContextFrame - Conversation context and history LLMMessagesFrame - Direct message list VisionImageRawFrame - Images for vision processing LLMUpdateSettingsFrame - Runtime parameter updates ​ Output LLMFullResponseStartFrame / LLMFullResponseEndFrame - Response boundaries LLMTextFrame - Streamed completion chunks LLMSearchResponseFrame - Search grounding results with citations FunctionCallInProgressFrame / FunctionCallResultFrame - Function call lifecycle ErrorFrame - API or processing errors ​ Search Grounding Google Gemini’s search grounding feature enables real-time web search integration, allowing the model to access current information and provide citations. This is particularly valuable for applications requiring up-to-date information. ​ Enabling Search Grounding Copy Ask AI # Configure search grounding tool search_tool = { "google_search_retrieval" : { "dynamic_retrieval_config" : { "mode" : "MODE_DYNAMIC" , "dynamic_threshold" : 0.3 , # Lower = more frequent grounding } } } # Initialize with search grounding llm = GoogleLLMService( api_key = os.getenv( "GOOGLE_API_KEY" ), model = "gemini-1.5-flash-002" , system_instruction = "You are a helpful assistant with access to current information." , tools = [search_tool] ) ​ Handling Search Results Search grounding produces LLMSearchResponseFrame with detailed citation information: Copy Ask AI @pipeline.event_handler ( "llm_search_response" ) async def handle_search_response ( frame ): print ( f "Search result: { frame.search_result } " ) print ( f "Sources: { len (frame.origins) } citations" ) for origin in frame.origins: print ( f "- { origin[ 'site_title' ] } : { origin[ 'site_uri' ] } " ) ​ Function Calling Function Calling Guide Learn how to implement function calling with standardized schemas, register handlers, manage context properly, and control execution flow in your conversational AI applications. ​ Context Management Context Management Guide Learn how to manage conversation context, handle message history, and integrate context aggregators for consistent conversational experiences. ​ Usage Example Copy Ask AI import os from pipecat.services.google.llm import GoogleLLMService from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext from pipecat.adapters.schemas.function_schema import FunctionSchema from pipecat.adapters.schemas.tools_schema import ToolsSchema # Configure Gemini service with search grounding search_tool = { "google_search_retrieval" : { "dynamic_retrieval_config" : { "mode" : "MODE_DYNAMIC" , "dynamic_threshold" : 0.3 } } } llm = GoogleLLMService( api_key = os.getenv( "GOOGLE_API_KEY" ), model = "gemini-2.0-flash" , system_instruction = """You are a helpful assistant with access to current information. When users ask about recent events, use search to provide accurate, up-to-date information.""" , tools = [search_tool], params = GoogleLLMService.InputParams( temperature = 0.7 , max_tokens = 1000 ) ) # Define function for tool calling weather_function = FunctionSchema( name = "get_weather" , description = "Get current weather information" , properties = { "location" : { "type" : "string" , "description" : "City and state, e.g. San Francisco, CA" } }, required = [ "location" ] ) # Define image capture function for multimodal capabilities image_function = FunctionSchema( name = "get_image" , description = "Capture and analyze an image from the video stream" , properties = { "question" : { "type" : "string" , "description" : "Question about what to analyze in the image" } }, required = [ "question" ] ) tools = ToolsSchema( standard_tools = [weather_function, image_function]) # Create context with multimodal system prompt context = OpenAILLMContext( messages = [ { "role" : "system" , "content" : """You are a helpful assistant with access to current information and vision capabilities. You can answer questions about weather, analyze images from video streams, and search for current information. Keep responses concise for voice output.""" }, { "role" : "user" , "content" : "Hello! What can you help me with?" } ], tools = tools ) # Create context aggregators context_aggregator = llm.create_context_aggregator(context) # Register function handlers async def get_weather ( params ): location = params.arguments[ "location" ] await params.result_callback( f "Weather in { location } : 72°F and sunny" ) async def get_image ( params ): question = params.arguments[ "question" ] # Request image from video stream await params.llm.request_image_frame( user_id = client_id, function_name = params.function_name, tool_call_id = params.tool_call_id, text_content = question ) await params.result_callback( f "Analyzing image for: { question } " ) llm.register_function( "get_weather" , get_weather) llm.register_function( "get_image" , get_image) # Optional: Add function call feedback @llm.event_handler ( "on_function_calls_started" ) async def on_function_calls_started ( service , function_calls ): await tts.queue_frame(TTSSpeakFrame( "Let me check on that." )) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, context_aggregator.user(), llm, tts, transport.output(), context_aggregator.assistant() ]) ​ Metrics Google Gemini provides comprehensive usage tracking: Time to First Byte (TTFB) - Response latency measurement Processing Duration - Total request processing time Token Usage - Prompt tokens, completion tokens, and totals Enable with: Copy Ask AI task = PipelineTask( pipeline, params = PipelineParams( enable_metrics = True , enable_usage_metrics = True ) ) ​ Additional Notes Multimodal Capabilities : Native support for text, images, audio, and video processing Search Grounding : Real-time web search with automatic citation and source attribution System Instructions : Handle system messages differently than OpenAI - set during initialization Vision Functions : Built-in support for image capture and analysis from video streams Fireworks AI Google Vertex AI On this page Overview Installation Frames Input Output Search Grounding Enabling Search Grounding Handling Search Results Function Calling Context Management Usage Example Metrics Additional Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/llm_google-vertex_0ec9db9d.txt b/llm_google-vertex_0ec9db9d.txt
new file mode 100644
index 0000000000000000000000000000000000000000..ca4676adfd3c9a490f8cb640a00f878f08519643
--- /dev/null
+++ b/llm_google-vertex_0ec9db9d.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/llm/google-vertex#param-project-id
+Title: Google Vertex AI - Pipecat
+==================================================
+
+Google Vertex AI - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation LLM Google Vertex AI Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Anthropic AWS Bedrock Azure Cerebras DeepSeek Fireworks AI Google Gemini Google Vertex AI Grok Groq NVIDIA NIM Ollama OpenAI OpenPipe OpenRouter Perplexity Qwen SambaNova Together AI Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview GoogleVertexLLMService provides access to Google’s language models through Vertex AI while maintaining an OpenAI-compatible interface. It inherits from OpenAILLMService and supports all the features of the OpenAI interface while connecting to Google’s AI services. ​ Installation To use GoogleVertexLLMService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[google]" You’ll also need to set up Google Cloud credentials. You can either: Set the GOOGLE_APPLICATION_CREDENTIALS environment variable pointing to your service account JSON file Provide credentials directly to the service constructor ​ Configuration ​ Constructor Parameters ​ credentials Optional[str] JSON string of Google service account credentials ​ credentials_path Optional[str] Path to the Google service account JSON file ​ model str default: "google/gemini-2.0-flash-001" Model identifier ​ params InputParams Vertex AI specific parameters ​ Input Parameters Extends the OpenAI input parameters with Vertex AI specific options: ​ location str default: "us-east4" Google Cloud region where the model is deployed ​ project_id str required Google Cloud project ID Also inherits all OpenAI-compatible parameters: ​ frequency_penalty Optional[float] Reduces likelihood of repeating tokens based on their frequency. Range: [-2.0, 2.0] ​ max_tokens Optional[int] Maximum number of tokens to generate. Must be greater than or equal to 1 ​ presence_penalty Optional[float] Reduces likelihood of repeating any tokens that have appeared. Range: [-2.0, 2.0] ​ temperature Optional[float] Controls randomness in the output. Range: [0.0, 2.0] ​ top_p Optional[float] Controls diversity via nucleus sampling. Range: [0.0, 1.0] ​ Usage Example Copy Ask AI from pipecat.services.google.llm_vertex import GoogleVertexLLMService from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext from pipecat.pipeline.pipeline import Pipeline from pipecat.pipeline.task import PipelineParams, PipelineTask # Configure service llm = GoogleVertexLLMService( credentials_path = "/path/to/service-account.json" , model = "google/gemini-2.0-flash-001" , params = GoogleVertexLLMService.InputParams( project_id = "your-google-cloud-project-id" , location = "us-east4" ) ) # Create context with system message context = OpenAILLMContext( messages = [ { "role" : "system" , "content" : "You are a helpful assistant in a voice conversation. Keep responses concise." } ] ) # Create context aggregator for message handling context_aggregator = llm.create_context_aggregator(context) # Set up pipeline pipeline = Pipeline([ transport.input(), context_aggregator.user(), llm, tts, transport.output(), context_aggregator.assistant() ]) # Create and configure task task = PipelineTask( pipeline, params = PipelineParams( allow_interruptions = True , enable_metrics = True , enable_usage_metrics = True , ), ) ​ Authentication The service supports multiple authentication methods: Direct credentials string - Pass the JSON credentials as a string to the constructor Credentials file path - Provide a path to the service account JSON file Environment variable - Set GOOGLE_APPLICATION_CREDENTIALS to the path of your service account file The service automatically handles token refresh, with tokens having a 1-hour lifetime. ​ Methods See the LLM base class methods for additional functionality. ​ Function Calling This service supports function calling (also known as tool calling) through the OpenAI-compatible interface, which allows the LLM to request information from external services and APIs. Function Calling Guide Learn how to implement function calling with standardized schemas, register handlers, manage context properly, and control execution flow in your conversational AI applications. ​ Available Models Model Name Description google/gemini-2.0-flash-001 Fast, efficient text generation model google/gemini-2.0-pro-001 Comprehensive, high-quality model google/gemini-1.5-pro-001 Versatile multimodal model google/gemini-1.5-flash-001 Fast, efficient multimodal model See Google Vertex AI documentation for a complete list of supported models and their capabilities. ​ Frame Flow Inherits the OpenAI LLM Service frame flow: ​ Metrics Support The service collects standard LLM metrics: Token usage (prompt and completion) Processing duration Time to First Byte (TTFB) Function call metrics ​ Notes Uses Google Cloud’s Vertex AI API Maintains OpenAI-compatible interface Supports streaming responses Handles function calling Manages conversation context Includes token usage tracking Thread-safe processing Automatic token refresh Requires Google Cloud project setup Google Gemini Grok On this page Overview Installation Configuration Constructor Parameters Input Parameters Usage Example Authentication Methods Function Calling Available Models Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/llm_google-vertex_77c37360.txt b/llm_google-vertex_77c37360.txt
new file mode 100644
index 0000000000000000000000000000000000000000..f5619dbf4075baedb4a98d364a318ccbf6275701
--- /dev/null
+++ b/llm_google-vertex_77c37360.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/llm/google-vertex#usage-example
+Title: Google Vertex AI - Pipecat
+==================================================
+
+Google Vertex AI - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation LLM Google Vertex AI Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Anthropic AWS Bedrock Azure Cerebras DeepSeek Fireworks AI Google Gemini Google Vertex AI Grok Groq NVIDIA NIM Ollama OpenAI OpenPipe OpenRouter Perplexity Qwen SambaNova Together AI Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview GoogleVertexLLMService provides access to Google’s language models through Vertex AI while maintaining an OpenAI-compatible interface. It inherits from OpenAILLMService and supports all the features of the OpenAI interface while connecting to Google’s AI services. ​ Installation To use GoogleVertexLLMService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[google]" You’ll also need to set up Google Cloud credentials. You can either: Set the GOOGLE_APPLICATION_CREDENTIALS environment variable pointing to your service account JSON file Provide credentials directly to the service constructor ​ Configuration ​ Constructor Parameters ​ credentials Optional[str] JSON string of Google service account credentials ​ credentials_path Optional[str] Path to the Google service account JSON file ​ model str default: "google/gemini-2.0-flash-001" Model identifier ​ params InputParams Vertex AI specific parameters ​ Input Parameters Extends the OpenAI input parameters with Vertex AI specific options: ​ location str default: "us-east4" Google Cloud region where the model is deployed ​ project_id str required Google Cloud project ID Also inherits all OpenAI-compatible parameters: ​ frequency_penalty Optional[float] Reduces likelihood of repeating tokens based on their frequency. Range: [-2.0, 2.0] ​ max_tokens Optional[int] Maximum number of tokens to generate. Must be greater than or equal to 1 ​ presence_penalty Optional[float] Reduces likelihood of repeating any tokens that have appeared. Range: [-2.0, 2.0] ​ temperature Optional[float] Controls randomness in the output. Range: [0.0, 2.0] ​ top_p Optional[float] Controls diversity via nucleus sampling. Range: [0.0, 1.0] ​ Usage Example Copy Ask AI from pipecat.services.google.llm_vertex import GoogleVertexLLMService from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext from pipecat.pipeline.pipeline import Pipeline from pipecat.pipeline.task import PipelineParams, PipelineTask # Configure service llm = GoogleVertexLLMService( credentials_path = "/path/to/service-account.json" , model = "google/gemini-2.0-flash-001" , params = GoogleVertexLLMService.InputParams( project_id = "your-google-cloud-project-id" , location = "us-east4" ) ) # Create context with system message context = OpenAILLMContext( messages = [ { "role" : "system" , "content" : "You are a helpful assistant in a voice conversation. Keep responses concise." } ] ) # Create context aggregator for message handling context_aggregator = llm.create_context_aggregator(context) # Set up pipeline pipeline = Pipeline([ transport.input(), context_aggregator.user(), llm, tts, transport.output(), context_aggregator.assistant() ]) # Create and configure task task = PipelineTask( pipeline, params = PipelineParams( allow_interruptions = True , enable_metrics = True , enable_usage_metrics = True , ), ) ​ Authentication The service supports multiple authentication methods: Direct credentials string - Pass the JSON credentials as a string to the constructor Credentials file path - Provide a path to the service account JSON file Environment variable - Set GOOGLE_APPLICATION_CREDENTIALS to the path of your service account file The service automatically handles token refresh, with tokens having a 1-hour lifetime. ​ Methods See the LLM base class methods for additional functionality. ​ Function Calling This service supports function calling (also known as tool calling) through the OpenAI-compatible interface, which allows the LLM to request information from external services and APIs. Function Calling Guide Learn how to implement function calling with standardized schemas, register handlers, manage context properly, and control execution flow in your conversational AI applications. ​ Available Models Model Name Description google/gemini-2.0-flash-001 Fast, efficient text generation model google/gemini-2.0-pro-001 Comprehensive, high-quality model google/gemini-1.5-pro-001 Versatile multimodal model google/gemini-1.5-flash-001 Fast, efficient multimodal model See Google Vertex AI documentation for a complete list of supported models and their capabilities. ​ Frame Flow Inherits the OpenAI LLM Service frame flow: ​ Metrics Support The service collects standard LLM metrics: Token usage (prompt and completion) Processing duration Time to First Byte (TTFB) Function call metrics ​ Notes Uses Google Cloud’s Vertex AI API Maintains OpenAI-compatible interface Supports streaming responses Handles function calling Manages conversation context Includes token usage tracking Thread-safe processing Automatic token refresh Requires Google Cloud project setup Google Gemini Grok On this page Overview Installation Configuration Constructor Parameters Input Parameters Usage Example Authentication Methods Function Calling Available Models Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/llm_google-vertex_e3ed737e.txt b/llm_google-vertex_e3ed737e.txt
new file mode 100644
index 0000000000000000000000000000000000000000..dbb95dc375a631bc1c1679908cca4cdea85c1e36
--- /dev/null
+++ b/llm_google-vertex_e3ed737e.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/llm/google-vertex#constructor-parameters
+Title: Google Vertex AI - Pipecat
+==================================================
+
+Google Vertex AI - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation LLM Google Vertex AI Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Anthropic AWS Bedrock Azure Cerebras DeepSeek Fireworks AI Google Gemini Google Vertex AI Grok Groq NVIDIA NIM Ollama OpenAI OpenPipe OpenRouter Perplexity Qwen SambaNova Together AI Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview GoogleVertexLLMService provides access to Google’s language models through Vertex AI while maintaining an OpenAI-compatible interface. It inherits from OpenAILLMService and supports all the features of the OpenAI interface while connecting to Google’s AI services. ​ Installation To use GoogleVertexLLMService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[google]" You’ll also need to set up Google Cloud credentials. You can either: Set the GOOGLE_APPLICATION_CREDENTIALS environment variable pointing to your service account JSON file Provide credentials directly to the service constructor ​ Configuration ​ Constructor Parameters ​ credentials Optional[str] JSON string of Google service account credentials ​ credentials_path Optional[str] Path to the Google service account JSON file ​ model str default: "google/gemini-2.0-flash-001" Model identifier ​ params InputParams Vertex AI specific parameters ​ Input Parameters Extends the OpenAI input parameters with Vertex AI specific options: ​ location str default: "us-east4" Google Cloud region where the model is deployed ​ project_id str required Google Cloud project ID Also inherits all OpenAI-compatible parameters: ​ frequency_penalty Optional[float] Reduces likelihood of repeating tokens based on their frequency. Range: [-2.0, 2.0] ​ max_tokens Optional[int] Maximum number of tokens to generate. Must be greater than or equal to 1 ​ presence_penalty Optional[float] Reduces likelihood of repeating any tokens that have appeared. Range: [-2.0, 2.0] ​ temperature Optional[float] Controls randomness in the output. Range: [0.0, 2.0] ​ top_p Optional[float] Controls diversity via nucleus sampling. Range: [0.0, 1.0] ​ Usage Example Copy Ask AI from pipecat.services.google.llm_vertex import GoogleVertexLLMService from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext from pipecat.pipeline.pipeline import Pipeline from pipecat.pipeline.task import PipelineParams, PipelineTask # Configure service llm = GoogleVertexLLMService( credentials_path = "/path/to/service-account.json" , model = "google/gemini-2.0-flash-001" , params = GoogleVertexLLMService.InputParams( project_id = "your-google-cloud-project-id" , location = "us-east4" ) ) # Create context with system message context = OpenAILLMContext( messages = [ { "role" : "system" , "content" : "You are a helpful assistant in a voice conversation. Keep responses concise." } ] ) # Create context aggregator for message handling context_aggregator = llm.create_context_aggregator(context) # Set up pipeline pipeline = Pipeline([ transport.input(), context_aggregator.user(), llm, tts, transport.output(), context_aggregator.assistant() ]) # Create and configure task task = PipelineTask( pipeline, params = PipelineParams( allow_interruptions = True , enable_metrics = True , enable_usage_metrics = True , ), ) ​ Authentication The service supports multiple authentication methods: Direct credentials string - Pass the JSON credentials as a string to the constructor Credentials file path - Provide a path to the service account JSON file Environment variable - Set GOOGLE_APPLICATION_CREDENTIALS to the path of your service account file The service automatically handles token refresh, with tokens having a 1-hour lifetime. ​ Methods See the LLM base class methods for additional functionality. ​ Function Calling This service supports function calling (also known as tool calling) through the OpenAI-compatible interface, which allows the LLM to request information from external services and APIs. Function Calling Guide Learn how to implement function calling with standardized schemas, register handlers, manage context properly, and control execution flow in your conversational AI applications. ​ Available Models Model Name Description google/gemini-2.0-flash-001 Fast, efficient text generation model google/gemini-2.0-pro-001 Comprehensive, high-quality model google/gemini-1.5-pro-001 Versatile multimodal model google/gemini-1.5-flash-001 Fast, efficient multimodal model See Google Vertex AI documentation for a complete list of supported models and their capabilities. ​ Frame Flow Inherits the OpenAI LLM Service frame flow: ​ Metrics Support The service collects standard LLM metrics: Token usage (prompt and completion) Processing duration Time to First Byte (TTFB) Function call metrics ​ Notes Uses Google Cloud’s Vertex AI API Maintains OpenAI-compatible interface Supports streaming responses Handles function calling Manages conversation context Includes token usage tracking Thread-safe processing Automatic token refresh Requires Google Cloud project setup Google Gemini Grok On this page Overview Installation Configuration Constructor Parameters Input Parameters Usage Example Authentication Methods Function Calling Available Models Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/llm_google-vertex_f469901d.txt b/llm_google-vertex_f469901d.txt
new file mode 100644
index 0000000000000000000000000000000000000000..7bc4659f5d6a920f3c3c315f176f7b22d43d72dc
--- /dev/null
+++ b/llm_google-vertex_f469901d.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/llm/google-vertex#param-params
+Title: Google Vertex AI - Pipecat
+==================================================
+
+Google Vertex AI - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation LLM Google Vertex AI Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Anthropic AWS Bedrock Azure Cerebras DeepSeek Fireworks AI Google Gemini Google Vertex AI Grok Groq NVIDIA NIM Ollama OpenAI OpenPipe OpenRouter Perplexity Qwen SambaNova Together AI Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview GoogleVertexLLMService provides access to Google’s language models through Vertex AI while maintaining an OpenAI-compatible interface. It inherits from OpenAILLMService and supports all the features of the OpenAI interface while connecting to Google’s AI services. ​ Installation To use GoogleVertexLLMService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[google]" You’ll also need to set up Google Cloud credentials. You can either: Set the GOOGLE_APPLICATION_CREDENTIALS environment variable pointing to your service account JSON file Provide credentials directly to the service constructor ​ Configuration ​ Constructor Parameters ​ credentials Optional[str] JSON string of Google service account credentials ​ credentials_path Optional[str] Path to the Google service account JSON file ​ model str default: "google/gemini-2.0-flash-001" Model identifier ​ params InputParams Vertex AI specific parameters ​ Input Parameters Extends the OpenAI input parameters with Vertex AI specific options: ​ location str default: "us-east4" Google Cloud region where the model is deployed ​ project_id str required Google Cloud project ID Also inherits all OpenAI-compatible parameters: ​ frequency_penalty Optional[float] Reduces likelihood of repeating tokens based on their frequency. Range: [-2.0, 2.0] ​ max_tokens Optional[int] Maximum number of tokens to generate. Must be greater than or equal to 1 ​ presence_penalty Optional[float] Reduces likelihood of repeating any tokens that have appeared. Range: [-2.0, 2.0] ​ temperature Optional[float] Controls randomness in the output. Range: [0.0, 2.0] ​ top_p Optional[float] Controls diversity via nucleus sampling. Range: [0.0, 1.0] ​ Usage Example Copy Ask AI from pipecat.services.google.llm_vertex import GoogleVertexLLMService from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext from pipecat.pipeline.pipeline import Pipeline from pipecat.pipeline.task import PipelineParams, PipelineTask # Configure service llm = GoogleVertexLLMService( credentials_path = "/path/to/service-account.json" , model = "google/gemini-2.0-flash-001" , params = GoogleVertexLLMService.InputParams( project_id = "your-google-cloud-project-id" , location = "us-east4" ) ) # Create context with system message context = OpenAILLMContext( messages = [ { "role" : "system" , "content" : "You are a helpful assistant in a voice conversation. Keep responses concise." } ] ) # Create context aggregator for message handling context_aggregator = llm.create_context_aggregator(context) # Set up pipeline pipeline = Pipeline([ transport.input(), context_aggregator.user(), llm, tts, transport.output(), context_aggregator.assistant() ]) # Create and configure task task = PipelineTask( pipeline, params = PipelineParams( allow_interruptions = True , enable_metrics = True , enable_usage_metrics = True , ), ) ​ Authentication The service supports multiple authentication methods: Direct credentials string - Pass the JSON credentials as a string to the constructor Credentials file path - Provide a path to the service account JSON file Environment variable - Set GOOGLE_APPLICATION_CREDENTIALS to the path of your service account file The service automatically handles token refresh, with tokens having a 1-hour lifetime. ​ Methods See the LLM base class methods for additional functionality. ​ Function Calling This service supports function calling (also known as tool calling) through the OpenAI-compatible interface, which allows the LLM to request information from external services and APIs. Function Calling Guide Learn how to implement function calling with standardized schemas, register handlers, manage context properly, and control execution flow in your conversational AI applications. ​ Available Models Model Name Description google/gemini-2.0-flash-001 Fast, efficient text generation model google/gemini-2.0-pro-001 Comprehensive, high-quality model google/gemini-1.5-pro-001 Versatile multimodal model google/gemini-1.5-flash-001 Fast, efficient multimodal model See Google Vertex AI documentation for a complete list of supported models and their capabilities. ​ Frame Flow Inherits the OpenAI LLM Service frame flow: ​ Metrics Support The service collects standard LLM metrics: Token usage (prompt and completion) Processing duration Time to First Byte (TTFB) Function call metrics ​ Notes Uses Google Cloud’s Vertex AI API Maintains OpenAI-compatible interface Supports streaming responses Handles function calling Manages conversation context Includes token usage tracking Thread-safe processing Automatic token refresh Requires Google Cloud project setup Google Gemini Grok On this page Overview Installation Configuration Constructor Parameters Input Parameters Usage Example Authentication Methods Function Calling Available Models Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/llm_grok_a3bc95ee.txt b/llm_grok_a3bc95ee.txt
new file mode 100644
index 0000000000000000000000000000000000000000..953575bdb5f5bb4aa12f9a8911205572ddbdc91c
--- /dev/null
+++ b/llm_grok_a3bc95ee.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/llm/grok#function-calling
+Title: Grok - Pipecat
+==================================================
+
+Grok - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation LLM Grok Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Anthropic AWS Bedrock Azure Cerebras DeepSeek Fireworks AI Google Gemini Google Vertex AI Grok Groq NVIDIA NIM Ollama OpenAI OpenPipe OpenRouter Perplexity Qwen SambaNova Together AI Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview GrokLLMService provides access to Grok’s language models through an OpenAI-compatible interface. It inherits from OpenAILLMService and supports streaming responses, function calling, and context management. API Reference Complete API documentation and method details Grok Docs Official Grok API documentation and features Example Code Working example with function calling ​ Installation To use GrokLLMService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[grok]" You’ll also need to set up your Grok API key as an environment variable: GROK_API_KEY . Get your API key from X.AI Console . ​ Frames ​ Input OpenAILLMContextFrame - Conversation context and history LLMMessagesFrame - Direct message list VisionImageRawFrame - Images for vision processing LLMUpdateSettingsFrame - Runtime parameter updates ​ Output LLMFullResponseStartFrame / LLMFullResponseEndFrame - Response boundaries LLMTextFrame - Streamed completion chunks FunctionCallInProgressFrame / FunctionCallResultFrame - Function call lifecycle ErrorFrame - API or processing errors ​ Function Calling Function Calling Guide Learn how to implement function calling with standardized schemas, register handlers, manage context properly, and control execution flow in your conversational AI applications. ​ Context Management Context Management Guide Learn how to manage conversation context, handle message history, and integrate context aggregators for consistent conversational experiences. ​ Usage Example Copy Ask AI import os from pipecat.services.grok.llm import GrokLLMService from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext from pipecat.adapters.schemas.function_schema import FunctionSchema from pipecat.adapters.schemas.tools_schema import ToolsSchema # Configure Grok service llm = GrokLLMService( api_key = os.getenv( "GROK_API_KEY" ), model = "grok-3-beta" , params = GrokLLMService.InputParams( temperature = 0.8 , # Higher for creative responses max_tokens = 1000 ) ) # Define function for tool calling weather_function = FunctionSchema( name = "get_current_weather" , description = "Get current weather information" , properties = { "location" : { "type" : "string" , "description" : "City and state, e.g. San Francisco, CA" }, "format" : { "type" : "string" , "enum" : [ "celsius" , "fahrenheit" ], "description" : "Temperature unit to use" } }, required = [ "location" , "format" ] ) tools = ToolsSchema( standard_tools = [weather_function]) # Create context optimized for voice interaction context = OpenAILLMContext( messages = [ { "role" : "system" , "content" : """You are a helpful and creative assistant in a voice conversation. Your output will be converted to audio, so avoid special characters. Respond in an engaging and helpful way while being succinct.""" } ], tools = tools ) # Create context aggregators context_aggregator = llm.create_context_aggregator(context) # Register function handler async def fetch_weather ( params ): location = params.arguments[ "location" ] await params.result_callback({ "conditions" : "sunny" , "temperature" : "75°F" }) llm.register_function( "get_current_weather" , fetch_weather) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, context_aggregator.user(), llm, tts, transport.output(), context_aggregator.assistant() ]) ​ Metrics Inherits all OpenAI metrics capabilities with specialized token tracking: Time to First Byte (TTFB) - Response latency measurement Processing Duration - Total request processing time Token Usage - Accumulated prompt tokens, completion tokens, and totals Grok uses incremental token reporting, so metrics are accumulated and reported at the end of each response. Enable with: Copy Ask AI task = PipelineTask( pipeline, params = PipelineParams( enable_metrics = True , enable_usage_metrics = True ) ) ​ Additional Notes OpenAI Compatibility : Full compatibility with OpenAI API features and parameters Real-time Information : Access to current events and up-to-date information Vision Capabilities : Image understanding and analysis with grok-2-vision model Google Vertex AI Groq On this page Overview Installation Frames Input Output Function Calling Context Management Usage Example Metrics Additional Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/llm_grok_b9559743.txt b/llm_grok_b9559743.txt
new file mode 100644
index 0000000000000000000000000000000000000000..8e2c2f24a48319fbf7cbfff6076d20cbf00bd114
--- /dev/null
+++ b/llm_grok_b9559743.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/llm/grok#input
+Title: Grok - Pipecat
+==================================================
+
+Grok - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation LLM Grok Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Anthropic AWS Bedrock Azure Cerebras DeepSeek Fireworks AI Google Gemini Google Vertex AI Grok Groq NVIDIA NIM Ollama OpenAI OpenPipe OpenRouter Perplexity Qwen SambaNova Together AI Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview GrokLLMService provides access to Grok’s language models through an OpenAI-compatible interface. It inherits from OpenAILLMService and supports streaming responses, function calling, and context management. API Reference Complete API documentation and method details Grok Docs Official Grok API documentation and features Example Code Working example with function calling ​ Installation To use GrokLLMService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[grok]" You’ll also need to set up your Grok API key as an environment variable: GROK_API_KEY . Get your API key from X.AI Console . ​ Frames ​ Input OpenAILLMContextFrame - Conversation context and history LLMMessagesFrame - Direct message list VisionImageRawFrame - Images for vision processing LLMUpdateSettingsFrame - Runtime parameter updates ​ Output LLMFullResponseStartFrame / LLMFullResponseEndFrame - Response boundaries LLMTextFrame - Streamed completion chunks FunctionCallInProgressFrame / FunctionCallResultFrame - Function call lifecycle ErrorFrame - API or processing errors ​ Function Calling Function Calling Guide Learn how to implement function calling with standardized schemas, register handlers, manage context properly, and control execution flow in your conversational AI applications. ​ Context Management Context Management Guide Learn how to manage conversation context, handle message history, and integrate context aggregators for consistent conversational experiences. ​ Usage Example Copy Ask AI import os from pipecat.services.grok.llm import GrokLLMService from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext from pipecat.adapters.schemas.function_schema import FunctionSchema from pipecat.adapters.schemas.tools_schema import ToolsSchema # Configure Grok service llm = GrokLLMService( api_key = os.getenv( "GROK_API_KEY" ), model = "grok-3-beta" , params = GrokLLMService.InputParams( temperature = 0.8 , # Higher for creative responses max_tokens = 1000 ) ) # Define function for tool calling weather_function = FunctionSchema( name = "get_current_weather" , description = "Get current weather information" , properties = { "location" : { "type" : "string" , "description" : "City and state, e.g. San Francisco, CA" }, "format" : { "type" : "string" , "enum" : [ "celsius" , "fahrenheit" ], "description" : "Temperature unit to use" } }, required = [ "location" , "format" ] ) tools = ToolsSchema( standard_tools = [weather_function]) # Create context optimized for voice interaction context = OpenAILLMContext( messages = [ { "role" : "system" , "content" : """You are a helpful and creative assistant in a voice conversation. Your output will be converted to audio, so avoid special characters. Respond in an engaging and helpful way while being succinct.""" } ], tools = tools ) # Create context aggregators context_aggregator = llm.create_context_aggregator(context) # Register function handler async def fetch_weather ( params ): location = params.arguments[ "location" ] await params.result_callback({ "conditions" : "sunny" , "temperature" : "75°F" }) llm.register_function( "get_current_weather" , fetch_weather) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, context_aggregator.user(), llm, tts, transport.output(), context_aggregator.assistant() ]) ​ Metrics Inherits all OpenAI metrics capabilities with specialized token tracking: Time to First Byte (TTFB) - Response latency measurement Processing Duration - Total request processing time Token Usage - Accumulated prompt tokens, completion tokens, and totals Grok uses incremental token reporting, so metrics are accumulated and reported at the end of each response. Enable with: Copy Ask AI task = PipelineTask( pipeline, params = PipelineParams( enable_metrics = True , enable_usage_metrics = True ) ) ​ Additional Notes OpenAI Compatibility : Full compatibility with OpenAI API features and parameters Real-time Information : Access to current events and up-to-date information Vision Capabilities : Image understanding and analysis with grok-2-vision model Google Vertex AI Groq On this page Overview Installation Frames Input Output Function Calling Context Management Usage Example Metrics Additional Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/llm_grok_d4bad360.txt b/llm_grok_d4bad360.txt
new file mode 100644
index 0000000000000000000000000000000000000000..9841d14399b20207274b973c627267ce8fdadd00
--- /dev/null
+++ b/llm_grok_d4bad360.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/llm/grok#output
+Title: Grok - Pipecat
+==================================================
+
+Grok - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation LLM Grok Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Anthropic AWS Bedrock Azure Cerebras DeepSeek Fireworks AI Google Gemini Google Vertex AI Grok Groq NVIDIA NIM Ollama OpenAI OpenPipe OpenRouter Perplexity Qwen SambaNova Together AI Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview GrokLLMService provides access to Grok’s language models through an OpenAI-compatible interface. It inherits from OpenAILLMService and supports streaming responses, function calling, and context management. API Reference Complete API documentation and method details Grok Docs Official Grok API documentation and features Example Code Working example with function calling ​ Installation To use GrokLLMService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[grok]" You’ll also need to set up your Grok API key as an environment variable: GROK_API_KEY . Get your API key from X.AI Console . ​ Frames ​ Input OpenAILLMContextFrame - Conversation context and history LLMMessagesFrame - Direct message list VisionImageRawFrame - Images for vision processing LLMUpdateSettingsFrame - Runtime parameter updates ​ Output LLMFullResponseStartFrame / LLMFullResponseEndFrame - Response boundaries LLMTextFrame - Streamed completion chunks FunctionCallInProgressFrame / FunctionCallResultFrame - Function call lifecycle ErrorFrame - API or processing errors ​ Function Calling Function Calling Guide Learn how to implement function calling with standardized schemas, register handlers, manage context properly, and control execution flow in your conversational AI applications. ​ Context Management Context Management Guide Learn how to manage conversation context, handle message history, and integrate context aggregators for consistent conversational experiences. ​ Usage Example Copy Ask AI import os from pipecat.services.grok.llm import GrokLLMService from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext from pipecat.adapters.schemas.function_schema import FunctionSchema from pipecat.adapters.schemas.tools_schema import ToolsSchema # Configure Grok service llm = GrokLLMService( api_key = os.getenv( "GROK_API_KEY" ), model = "grok-3-beta" , params = GrokLLMService.InputParams( temperature = 0.8 , # Higher for creative responses max_tokens = 1000 ) ) # Define function for tool calling weather_function = FunctionSchema( name = "get_current_weather" , description = "Get current weather information" , properties = { "location" : { "type" : "string" , "description" : "City and state, e.g. San Francisco, CA" }, "format" : { "type" : "string" , "enum" : [ "celsius" , "fahrenheit" ], "description" : "Temperature unit to use" } }, required = [ "location" , "format" ] ) tools = ToolsSchema( standard_tools = [weather_function]) # Create context optimized for voice interaction context = OpenAILLMContext( messages = [ { "role" : "system" , "content" : """You are a helpful and creative assistant in a voice conversation. Your output will be converted to audio, so avoid special characters. Respond in an engaging and helpful way while being succinct.""" } ], tools = tools ) # Create context aggregators context_aggregator = llm.create_context_aggregator(context) # Register function handler async def fetch_weather ( params ): location = params.arguments[ "location" ] await params.result_callback({ "conditions" : "sunny" , "temperature" : "75°F" }) llm.register_function( "get_current_weather" , fetch_weather) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, context_aggregator.user(), llm, tts, transport.output(), context_aggregator.assistant() ]) ​ Metrics Inherits all OpenAI metrics capabilities with specialized token tracking: Time to First Byte (TTFB) - Response latency measurement Processing Duration - Total request processing time Token Usage - Accumulated prompt tokens, completion tokens, and totals Grok uses incremental token reporting, so metrics are accumulated and reported at the end of each response. Enable with: Copy Ask AI task = PipelineTask( pipeline, params = PipelineParams( enable_metrics = True , enable_usage_metrics = True ) ) ​ Additional Notes OpenAI Compatibility : Full compatibility with OpenAI API features and parameters Real-time Information : Access to current events and up-to-date information Vision Capabilities : Image understanding and analysis with grok-2-vision model Google Vertex AI Groq On this page Overview Installation Frames Input Output Function Calling Context Management Usage Example Metrics Additional Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/llm_groq_08d80ef2.txt b/llm_groq_08d80ef2.txt
new file mode 100644
index 0000000000000000000000000000000000000000..1704e3668bfc3d12f8eefd4dbca2cc386125a846
--- /dev/null
+++ b/llm_groq_08d80ef2.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/llm/groq#usage-example
+Title: Groq - Pipecat
+==================================================
+
+Groq - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation LLM Groq Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Anthropic AWS Bedrock Azure Cerebras DeepSeek Fireworks AI Google Gemini Google Vertex AI Grok Groq NVIDIA NIM Ollama OpenAI OpenPipe OpenRouter Perplexity Qwen SambaNova Together AI Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview GroqLLMService provides access to Groq’s language models through an OpenAI-compatible interface. It inherits from OpenAILLMService and supports streaming responses, function calling, and context management. API Reference Complete API documentation and method details Groq Docs Official Groq API documentation and features Example Code Working example with function calling ​ Installation To use Groq services, install the required dependency: Copy Ask AI pip install "pipecat-ai[groq]" You’ll also need to set up your Groq API key as an environment variable: GROQ_API_KEY . Get your API key for free from Groq Console . ​ Frames ​ Input OpenAILLMContextFrame - Conversation context and history LLMMessagesFrame - Direct message list VisionImageRawFrame - Images for vision processing (select models) LLMUpdateSettingsFrame - Runtime parameter updates ​ Output LLMFullResponseStartFrame / LLMFullResponseEndFrame - Response boundaries LLMTextFrame - Streamed completion chunks FunctionCallInProgressFrame / FunctionCallResultFrame - Function call lifecycle ErrorFrame - API or processing errors ​ Function Calling Function Calling Guide Learn how to implement function calling with standardized schemas, register handlers, manage context properly, and control execution flow in your conversational AI applications. ​ Context Management Context Management Guide Learn how to manage conversation context, handle message history, and integrate context aggregators for consistent conversational experiences. ​ Usage Example Copy Ask AI import os from pipecat.services.groq.llm import GroqLLMService from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext from pipecat.adapters.schemas.function_schema import FunctionSchema from pipecat.adapters.schemas.tools_schema import ToolsSchema # Configure Groq service for speed llm = GroqLLMService( api_key = os.getenv( "GROQ_API_KEY" ), model = "llama-3.3-70b-versatile" , # Fast, capable model params = GroqLLMService.InputParams( temperature = 0.7 , max_tokens = 1000 ) ) # Define function for tool calling weather_function = FunctionSchema( name = "get_current_weather" , description = "Get current weather information" , properties = { "location" : { "type" : "string" , "description" : "City and state, e.g. San Francisco, CA" }, "format" : { "type" : "string" , "enum" : [ "celsius" , "fahrenheit" ], "description" : "Temperature unit to use" } }, required = [ "location" , "format" ] ) tools = ToolsSchema( standard_tools = [weather_function]) # Create context optimized for voice interaction context = OpenAILLMContext( messages = [ { "role" : "system" , "content" : """You are a helpful assistant optimized for voice conversations. Keep responses concise and avoid special characters that don't work well in speech.""" } ], tools = tools ) # Create context aggregators with fast timeout for speed from pipecat.processors.aggregators.llm_response import LLMUserAggregatorParams context_aggregator = llm.create_context_aggregator( context, user_params = LLMUserAggregatorParams( aggregation_timeout = 0.05 ) # Fast aggregation ) # Register function handler with feedback async def fetch_weather ( params ): location = params.arguments[ "location" ] await params.result_callback({ "conditions" : "sunny" , "temperature" : "75°F" }) llm.register_function( "get_current_weather" , fetch_weather) # Optional: Add function call feedback for better UX @llm.event_handler ( "on_function_calls_started" ) async def on_function_calls_started ( service , function_calls ): await tts.queue_frame(TTSSpeakFrame( "Let me check on that." )) # Use in pipeline with Groq STT for full Groq stack pipeline = Pipeline([ transport.input(), groq_stt, # GroqSTTService for consistent ecosystem context_aggregator.user(), llm, tts, transport.output(), context_aggregator.assistant() ]) ​ Metrics Inherits all OpenAI metrics capabilities: Time to First Byte (TTFB) - Ultra-low latency measurements Processing Duration - Hardware-accelerated processing times Token Usage - Prompt tokens, completion tokens, and totals Function Call Metrics - Tool usage and execution tracking Enable with: Copy Ask AI task = PipelineTask( pipeline, params = PipelineParams( enable_metrics = True , enable_usage_metrics = True ) ) ​ Additional Notes OpenAI Compatibility : Full compatibility with OpenAI API features and parameters Real-time Optimized : Ideal for conversational AI and streaming applications Open Source Models : Access to Llama, Mixtral, and other open-source models Vision Support : Select models support image understanding capabilities Free Tier : Generous free tier available for development and testing Grok NVIDIA NIM On this page Overview Installation Frames Input Output Function Calling Context Management Usage Example Metrics Additional Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/llm_groq_195f068a.txt b/llm_groq_195f068a.txt
new file mode 100644
index 0000000000000000000000000000000000000000..e6c4e9b6093c7a34b06d4db23928df84b9889a7c
--- /dev/null
+++ b/llm_groq_195f068a.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/llm/groq#output
+Title: Groq - Pipecat
+==================================================
+
+Groq - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation LLM Groq Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Anthropic AWS Bedrock Azure Cerebras DeepSeek Fireworks AI Google Gemini Google Vertex AI Grok Groq NVIDIA NIM Ollama OpenAI OpenPipe OpenRouter Perplexity Qwen SambaNova Together AI Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview GroqLLMService provides access to Groq’s language models through an OpenAI-compatible interface. It inherits from OpenAILLMService and supports streaming responses, function calling, and context management. API Reference Complete API documentation and method details Groq Docs Official Groq API documentation and features Example Code Working example with function calling ​ Installation To use Groq services, install the required dependency: Copy Ask AI pip install "pipecat-ai[groq]" You’ll also need to set up your Groq API key as an environment variable: GROQ_API_KEY . Get your API key for free from Groq Console . ​ Frames ​ Input OpenAILLMContextFrame - Conversation context and history LLMMessagesFrame - Direct message list VisionImageRawFrame - Images for vision processing (select models) LLMUpdateSettingsFrame - Runtime parameter updates ​ Output LLMFullResponseStartFrame / LLMFullResponseEndFrame - Response boundaries LLMTextFrame - Streamed completion chunks FunctionCallInProgressFrame / FunctionCallResultFrame - Function call lifecycle ErrorFrame - API or processing errors ​ Function Calling Function Calling Guide Learn how to implement function calling with standardized schemas, register handlers, manage context properly, and control execution flow in your conversational AI applications. ​ Context Management Context Management Guide Learn how to manage conversation context, handle message history, and integrate context aggregators for consistent conversational experiences. ​ Usage Example Copy Ask AI import os from pipecat.services.groq.llm import GroqLLMService from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext from pipecat.adapters.schemas.function_schema import FunctionSchema from pipecat.adapters.schemas.tools_schema import ToolsSchema # Configure Groq service for speed llm = GroqLLMService( api_key = os.getenv( "GROQ_API_KEY" ), model = "llama-3.3-70b-versatile" , # Fast, capable model params = GroqLLMService.InputParams( temperature = 0.7 , max_tokens = 1000 ) ) # Define function for tool calling weather_function = FunctionSchema( name = "get_current_weather" , description = "Get current weather information" , properties = { "location" : { "type" : "string" , "description" : "City and state, e.g. San Francisco, CA" }, "format" : { "type" : "string" , "enum" : [ "celsius" , "fahrenheit" ], "description" : "Temperature unit to use" } }, required = [ "location" , "format" ] ) tools = ToolsSchema( standard_tools = [weather_function]) # Create context optimized for voice interaction context = OpenAILLMContext( messages = [ { "role" : "system" , "content" : """You are a helpful assistant optimized for voice conversations. Keep responses concise and avoid special characters that don't work well in speech.""" } ], tools = tools ) # Create context aggregators with fast timeout for speed from pipecat.processors.aggregators.llm_response import LLMUserAggregatorParams context_aggregator = llm.create_context_aggregator( context, user_params = LLMUserAggregatorParams( aggregation_timeout = 0.05 ) # Fast aggregation ) # Register function handler with feedback async def fetch_weather ( params ): location = params.arguments[ "location" ] await params.result_callback({ "conditions" : "sunny" , "temperature" : "75°F" }) llm.register_function( "get_current_weather" , fetch_weather) # Optional: Add function call feedback for better UX @llm.event_handler ( "on_function_calls_started" ) async def on_function_calls_started ( service , function_calls ): await tts.queue_frame(TTSSpeakFrame( "Let me check on that." )) # Use in pipeline with Groq STT for full Groq stack pipeline = Pipeline([ transport.input(), groq_stt, # GroqSTTService for consistent ecosystem context_aggregator.user(), llm, tts, transport.output(), context_aggregator.assistant() ]) ​ Metrics Inherits all OpenAI metrics capabilities: Time to First Byte (TTFB) - Ultra-low latency measurements Processing Duration - Hardware-accelerated processing times Token Usage - Prompt tokens, completion tokens, and totals Function Call Metrics - Tool usage and execution tracking Enable with: Copy Ask AI task = PipelineTask( pipeline, params = PipelineParams( enable_metrics = True , enable_usage_metrics = True ) ) ​ Additional Notes OpenAI Compatibility : Full compatibility with OpenAI API features and parameters Real-time Optimized : Ideal for conversational AI and streaming applications Open Source Models : Access to Llama, Mixtral, and other open-source models Vision Support : Select models support image understanding capabilities Free Tier : Generous free tier available for development and testing Grok NVIDIA NIM On this page Overview Installation Frames Input Output Function Calling Context Management Usage Example Metrics Additional Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/llm_groq_29386055.txt b/llm_groq_29386055.txt
new file mode 100644
index 0000000000000000000000000000000000000000..29cb39166b1ed8a24c081cc3435c06cb3de64cab
--- /dev/null
+++ b/llm_groq_29386055.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/llm/groq#overview
+Title: Groq - Pipecat
+==================================================
+
+Groq - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation LLM Groq Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Anthropic AWS Bedrock Azure Cerebras DeepSeek Fireworks AI Google Gemini Google Vertex AI Grok Groq NVIDIA NIM Ollama OpenAI OpenPipe OpenRouter Perplexity Qwen SambaNova Together AI Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview GroqLLMService provides access to Groq’s language models through an OpenAI-compatible interface. It inherits from OpenAILLMService and supports streaming responses, function calling, and context management. API Reference Complete API documentation and method details Groq Docs Official Groq API documentation and features Example Code Working example with function calling ​ Installation To use Groq services, install the required dependency: Copy Ask AI pip install "pipecat-ai[groq]" You’ll also need to set up your Groq API key as an environment variable: GROQ_API_KEY . Get your API key for free from Groq Console . ​ Frames ​ Input OpenAILLMContextFrame - Conversation context and history LLMMessagesFrame - Direct message list VisionImageRawFrame - Images for vision processing (select models) LLMUpdateSettingsFrame - Runtime parameter updates ​ Output LLMFullResponseStartFrame / LLMFullResponseEndFrame - Response boundaries LLMTextFrame - Streamed completion chunks FunctionCallInProgressFrame / FunctionCallResultFrame - Function call lifecycle ErrorFrame - API or processing errors ​ Function Calling Function Calling Guide Learn how to implement function calling with standardized schemas, register handlers, manage context properly, and control execution flow in your conversational AI applications. ​ Context Management Context Management Guide Learn how to manage conversation context, handle message history, and integrate context aggregators for consistent conversational experiences. ​ Usage Example Copy Ask AI import os from pipecat.services.groq.llm import GroqLLMService from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext from pipecat.adapters.schemas.function_schema import FunctionSchema from pipecat.adapters.schemas.tools_schema import ToolsSchema # Configure Groq service for speed llm = GroqLLMService( api_key = os.getenv( "GROQ_API_KEY" ), model = "llama-3.3-70b-versatile" , # Fast, capable model params = GroqLLMService.InputParams( temperature = 0.7 , max_tokens = 1000 ) ) # Define function for tool calling weather_function = FunctionSchema( name = "get_current_weather" , description = "Get current weather information" , properties = { "location" : { "type" : "string" , "description" : "City and state, e.g. San Francisco, CA" }, "format" : { "type" : "string" , "enum" : [ "celsius" , "fahrenheit" ], "description" : "Temperature unit to use" } }, required = [ "location" , "format" ] ) tools = ToolsSchema( standard_tools = [weather_function]) # Create context optimized for voice interaction context = OpenAILLMContext( messages = [ { "role" : "system" , "content" : """You are a helpful assistant optimized for voice conversations. Keep responses concise and avoid special characters that don't work well in speech.""" } ], tools = tools ) # Create context aggregators with fast timeout for speed from pipecat.processors.aggregators.llm_response import LLMUserAggregatorParams context_aggregator = llm.create_context_aggregator( context, user_params = LLMUserAggregatorParams( aggregation_timeout = 0.05 ) # Fast aggregation ) # Register function handler with feedback async def fetch_weather ( params ): location = params.arguments[ "location" ] await params.result_callback({ "conditions" : "sunny" , "temperature" : "75°F" }) llm.register_function( "get_current_weather" , fetch_weather) # Optional: Add function call feedback for better UX @llm.event_handler ( "on_function_calls_started" ) async def on_function_calls_started ( service , function_calls ): await tts.queue_frame(TTSSpeakFrame( "Let me check on that." )) # Use in pipeline with Groq STT for full Groq stack pipeline = Pipeline([ transport.input(), groq_stt, # GroqSTTService for consistent ecosystem context_aggregator.user(), llm, tts, transport.output(), context_aggregator.assistant() ]) ​ Metrics Inherits all OpenAI metrics capabilities: Time to First Byte (TTFB) - Ultra-low latency measurements Processing Duration - Hardware-accelerated processing times Token Usage - Prompt tokens, completion tokens, and totals Function Call Metrics - Tool usage and execution tracking Enable with: Copy Ask AI task = PipelineTask( pipeline, params = PipelineParams( enable_metrics = True , enable_usage_metrics = True ) ) ​ Additional Notes OpenAI Compatibility : Full compatibility with OpenAI API features and parameters Real-time Optimized : Ideal for conversational AI and streaming applications Open Source Models : Access to Llama, Mixtral, and other open-source models Vision Support : Select models support image understanding capabilities Free Tier : Generous free tier available for development and testing Grok NVIDIA NIM On this page Overview Installation Frames Input Output Function Calling Context Management Usage Example Metrics Additional Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/llm_groq_84ae5ca8.txt b/llm_groq_84ae5ca8.txt
new file mode 100644
index 0000000000000000000000000000000000000000..f90cc9b68f47506b1dc17c80628e3f4e4053be31
--- /dev/null
+++ b/llm_groq_84ae5ca8.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/llm/groq#installation
+Title: Groq - Pipecat
+==================================================
+
+Groq - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation LLM Groq Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Anthropic AWS Bedrock Azure Cerebras DeepSeek Fireworks AI Google Gemini Google Vertex AI Grok Groq NVIDIA NIM Ollama OpenAI OpenPipe OpenRouter Perplexity Qwen SambaNova Together AI Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview GroqLLMService provides access to Groq’s language models through an OpenAI-compatible interface. It inherits from OpenAILLMService and supports streaming responses, function calling, and context management. API Reference Complete API documentation and method details Groq Docs Official Groq API documentation and features Example Code Working example with function calling ​ Installation To use Groq services, install the required dependency: Copy Ask AI pip install "pipecat-ai[groq]" You’ll also need to set up your Groq API key as an environment variable: GROQ_API_KEY . Get your API key for free from Groq Console . ​ Frames ​ Input OpenAILLMContextFrame - Conversation context and history LLMMessagesFrame - Direct message list VisionImageRawFrame - Images for vision processing (select models) LLMUpdateSettingsFrame - Runtime parameter updates ​ Output LLMFullResponseStartFrame / LLMFullResponseEndFrame - Response boundaries LLMTextFrame - Streamed completion chunks FunctionCallInProgressFrame / FunctionCallResultFrame - Function call lifecycle ErrorFrame - API or processing errors ​ Function Calling Function Calling Guide Learn how to implement function calling with standardized schemas, register handlers, manage context properly, and control execution flow in your conversational AI applications. ​ Context Management Context Management Guide Learn how to manage conversation context, handle message history, and integrate context aggregators for consistent conversational experiences. ​ Usage Example Copy Ask AI import os from pipecat.services.groq.llm import GroqLLMService from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext from pipecat.adapters.schemas.function_schema import FunctionSchema from pipecat.adapters.schemas.tools_schema import ToolsSchema # Configure Groq service for speed llm = GroqLLMService( api_key = os.getenv( "GROQ_API_KEY" ), model = "llama-3.3-70b-versatile" , # Fast, capable model params = GroqLLMService.InputParams( temperature = 0.7 , max_tokens = 1000 ) ) # Define function for tool calling weather_function = FunctionSchema( name = "get_current_weather" , description = "Get current weather information" , properties = { "location" : { "type" : "string" , "description" : "City and state, e.g. San Francisco, CA" }, "format" : { "type" : "string" , "enum" : [ "celsius" , "fahrenheit" ], "description" : "Temperature unit to use" } }, required = [ "location" , "format" ] ) tools = ToolsSchema( standard_tools = [weather_function]) # Create context optimized for voice interaction context = OpenAILLMContext( messages = [ { "role" : "system" , "content" : """You are a helpful assistant optimized for voice conversations. Keep responses concise and avoid special characters that don't work well in speech.""" } ], tools = tools ) # Create context aggregators with fast timeout for speed from pipecat.processors.aggregators.llm_response import LLMUserAggregatorParams context_aggregator = llm.create_context_aggregator( context, user_params = LLMUserAggregatorParams( aggregation_timeout = 0.05 ) # Fast aggregation ) # Register function handler with feedback async def fetch_weather ( params ): location = params.arguments[ "location" ] await params.result_callback({ "conditions" : "sunny" , "temperature" : "75°F" }) llm.register_function( "get_current_weather" , fetch_weather) # Optional: Add function call feedback for better UX @llm.event_handler ( "on_function_calls_started" ) async def on_function_calls_started ( service , function_calls ): await tts.queue_frame(TTSSpeakFrame( "Let me check on that." )) # Use in pipeline with Groq STT for full Groq stack pipeline = Pipeline([ transport.input(), groq_stt, # GroqSTTService for consistent ecosystem context_aggregator.user(), llm, tts, transport.output(), context_aggregator.assistant() ]) ​ Metrics Inherits all OpenAI metrics capabilities: Time to First Byte (TTFB) - Ultra-low latency measurements Processing Duration - Hardware-accelerated processing times Token Usage - Prompt tokens, completion tokens, and totals Function Call Metrics - Tool usage and execution tracking Enable with: Copy Ask AI task = PipelineTask( pipeline, params = PipelineParams( enable_metrics = True , enable_usage_metrics = True ) ) ​ Additional Notes OpenAI Compatibility : Full compatibility with OpenAI API features and parameters Real-time Optimized : Ideal for conversational AI and streaming applications Open Source Models : Access to Llama, Mixtral, and other open-source models Vision Support : Select models support image understanding capabilities Free Tier : Generous free tier available for development and testing Grok NVIDIA NIM On this page Overview Installation Frames Input Output Function Calling Context Management Usage Example Metrics Additional Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/llm_groq_c3fd026b.txt b/llm_groq_c3fd026b.txt
new file mode 100644
index 0000000000000000000000000000000000000000..ef884c416349149ab97f1d95187cb6e70fc980eb
--- /dev/null
+++ b/llm_groq_c3fd026b.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/llm/groq#metrics
+Title: Groq - Pipecat
+==================================================
+
+Groq - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation LLM Groq Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Anthropic AWS Bedrock Azure Cerebras DeepSeek Fireworks AI Google Gemini Google Vertex AI Grok Groq NVIDIA NIM Ollama OpenAI OpenPipe OpenRouter Perplexity Qwen SambaNova Together AI Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview GroqLLMService provides access to Groq’s language models through an OpenAI-compatible interface. It inherits from OpenAILLMService and supports streaming responses, function calling, and context management. API Reference Complete API documentation and method details Groq Docs Official Groq API documentation and features Example Code Working example with function calling ​ Installation To use Groq services, install the required dependency: Copy Ask AI pip install "pipecat-ai[groq]" You’ll also need to set up your Groq API key as an environment variable: GROQ_API_KEY . Get your API key for free from Groq Console . ​ Frames ​ Input OpenAILLMContextFrame - Conversation context and history LLMMessagesFrame - Direct message list VisionImageRawFrame - Images for vision processing (select models) LLMUpdateSettingsFrame - Runtime parameter updates ​ Output LLMFullResponseStartFrame / LLMFullResponseEndFrame - Response boundaries LLMTextFrame - Streamed completion chunks FunctionCallInProgressFrame / FunctionCallResultFrame - Function call lifecycle ErrorFrame - API or processing errors ​ Function Calling Function Calling Guide Learn how to implement function calling with standardized schemas, register handlers, manage context properly, and control execution flow in your conversational AI applications. ​ Context Management Context Management Guide Learn how to manage conversation context, handle message history, and integrate context aggregators for consistent conversational experiences. ​ Usage Example Copy Ask AI import os from pipecat.services.groq.llm import GroqLLMService from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext from pipecat.adapters.schemas.function_schema import FunctionSchema from pipecat.adapters.schemas.tools_schema import ToolsSchema # Configure Groq service for speed llm = GroqLLMService( api_key = os.getenv( "GROQ_API_KEY" ), model = "llama-3.3-70b-versatile" , # Fast, capable model params = GroqLLMService.InputParams( temperature = 0.7 , max_tokens = 1000 ) ) # Define function for tool calling weather_function = FunctionSchema( name = "get_current_weather" , description = "Get current weather information" , properties = { "location" : { "type" : "string" , "description" : "City and state, e.g. San Francisco, CA" }, "format" : { "type" : "string" , "enum" : [ "celsius" , "fahrenheit" ], "description" : "Temperature unit to use" } }, required = [ "location" , "format" ] ) tools = ToolsSchema( standard_tools = [weather_function]) # Create context optimized for voice interaction context = OpenAILLMContext( messages = [ { "role" : "system" , "content" : """You are a helpful assistant optimized for voice conversations. Keep responses concise and avoid special characters that don't work well in speech.""" } ], tools = tools ) # Create context aggregators with fast timeout for speed from pipecat.processors.aggregators.llm_response import LLMUserAggregatorParams context_aggregator = llm.create_context_aggregator( context, user_params = LLMUserAggregatorParams( aggregation_timeout = 0.05 ) # Fast aggregation ) # Register function handler with feedback async def fetch_weather ( params ): location = params.arguments[ "location" ] await params.result_callback({ "conditions" : "sunny" , "temperature" : "75°F" }) llm.register_function( "get_current_weather" , fetch_weather) # Optional: Add function call feedback for better UX @llm.event_handler ( "on_function_calls_started" ) async def on_function_calls_started ( service , function_calls ): await tts.queue_frame(TTSSpeakFrame( "Let me check on that." )) # Use in pipeline with Groq STT for full Groq stack pipeline = Pipeline([ transport.input(), groq_stt, # GroqSTTService for consistent ecosystem context_aggregator.user(), llm, tts, transport.output(), context_aggregator.assistant() ]) ​ Metrics Inherits all OpenAI metrics capabilities: Time to First Byte (TTFB) - Ultra-low latency measurements Processing Duration - Hardware-accelerated processing times Token Usage - Prompt tokens, completion tokens, and totals Function Call Metrics - Tool usage and execution tracking Enable with: Copy Ask AI task = PipelineTask( pipeline, params = PipelineParams( enable_metrics = True , enable_usage_metrics = True ) ) ​ Additional Notes OpenAI Compatibility : Full compatibility with OpenAI API features and parameters Real-time Optimized : Ideal for conversational AI and streaming applications Open Source Models : Access to Llama, Mixtral, and other open-source models Vision Support : Select models support image understanding capabilities Free Tier : Generous free tier available for development and testing Grok NVIDIA NIM On this page Overview Installation Frames Input Output Function Calling Context Management Usage Example Metrics Additional Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/llm_groq_fefa0635.txt b/llm_groq_fefa0635.txt
new file mode 100644
index 0000000000000000000000000000000000000000..4fac14b0a934531884529cdf6d25a677e6327d49
--- /dev/null
+++ b/llm_groq_fefa0635.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/llm/groq#additional-notes
+Title: Groq - Pipecat
+==================================================
+
+Groq - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation LLM Groq Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Anthropic AWS Bedrock Azure Cerebras DeepSeek Fireworks AI Google Gemini Google Vertex AI Grok Groq NVIDIA NIM Ollama OpenAI OpenPipe OpenRouter Perplexity Qwen SambaNova Together AI Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview GroqLLMService provides access to Groq’s language models through an OpenAI-compatible interface. It inherits from OpenAILLMService and supports streaming responses, function calling, and context management. API Reference Complete API documentation and method details Groq Docs Official Groq API documentation and features Example Code Working example with function calling ​ Installation To use Groq services, install the required dependency: Copy Ask AI pip install "pipecat-ai[groq]" You’ll also need to set up your Groq API key as an environment variable: GROQ_API_KEY . Get your API key for free from Groq Console . ​ Frames ​ Input OpenAILLMContextFrame - Conversation context and history LLMMessagesFrame - Direct message list VisionImageRawFrame - Images for vision processing (select models) LLMUpdateSettingsFrame - Runtime parameter updates ​ Output LLMFullResponseStartFrame / LLMFullResponseEndFrame - Response boundaries LLMTextFrame - Streamed completion chunks FunctionCallInProgressFrame / FunctionCallResultFrame - Function call lifecycle ErrorFrame - API or processing errors ​ Function Calling Function Calling Guide Learn how to implement function calling with standardized schemas, register handlers, manage context properly, and control execution flow in your conversational AI applications. ​ Context Management Context Management Guide Learn how to manage conversation context, handle message history, and integrate context aggregators for consistent conversational experiences. ​ Usage Example Copy Ask AI import os from pipecat.services.groq.llm import GroqLLMService from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext from pipecat.adapters.schemas.function_schema import FunctionSchema from pipecat.adapters.schemas.tools_schema import ToolsSchema # Configure Groq service for speed llm = GroqLLMService( api_key = os.getenv( "GROQ_API_KEY" ), model = "llama-3.3-70b-versatile" , # Fast, capable model params = GroqLLMService.InputParams( temperature = 0.7 , max_tokens = 1000 ) ) # Define function for tool calling weather_function = FunctionSchema( name = "get_current_weather" , description = "Get current weather information" , properties = { "location" : { "type" : "string" , "description" : "City and state, e.g. San Francisco, CA" }, "format" : { "type" : "string" , "enum" : [ "celsius" , "fahrenheit" ], "description" : "Temperature unit to use" } }, required = [ "location" , "format" ] ) tools = ToolsSchema( standard_tools = [weather_function]) # Create context optimized for voice interaction context = OpenAILLMContext( messages = [ { "role" : "system" , "content" : """You are a helpful assistant optimized for voice conversations. Keep responses concise and avoid special characters that don't work well in speech.""" } ], tools = tools ) # Create context aggregators with fast timeout for speed from pipecat.processors.aggregators.llm_response import LLMUserAggregatorParams context_aggregator = llm.create_context_aggregator( context, user_params = LLMUserAggregatorParams( aggregation_timeout = 0.05 ) # Fast aggregation ) # Register function handler with feedback async def fetch_weather ( params ): location = params.arguments[ "location" ] await params.result_callback({ "conditions" : "sunny" , "temperature" : "75°F" }) llm.register_function( "get_current_weather" , fetch_weather) # Optional: Add function call feedback for better UX @llm.event_handler ( "on_function_calls_started" ) async def on_function_calls_started ( service , function_calls ): await tts.queue_frame(TTSSpeakFrame( "Let me check on that." )) # Use in pipeline with Groq STT for full Groq stack pipeline = Pipeline([ transport.input(), groq_stt, # GroqSTTService for consistent ecosystem context_aggregator.user(), llm, tts, transport.output(), context_aggregator.assistant() ]) ​ Metrics Inherits all OpenAI metrics capabilities: Time to First Byte (TTFB) - Ultra-low latency measurements Processing Duration - Hardware-accelerated processing times Token Usage - Prompt tokens, completion tokens, and totals Function Call Metrics - Tool usage and execution tracking Enable with: Copy Ask AI task = PipelineTask( pipeline, params = PipelineParams( enable_metrics = True , enable_usage_metrics = True ) ) ​ Additional Notes OpenAI Compatibility : Full compatibility with OpenAI API features and parameters Real-time Optimized : Ideal for conversational AI and streaming applications Open Source Models : Access to Llama, Mixtral, and other open-source models Vision Support : Select models support image understanding capabilities Free Tier : Generous free tier available for development and testing Grok NVIDIA NIM On this page Overview Installation Frames Input Output Function Calling Context Management Usage Example Metrics Additional Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/llm_nim_1b768057.txt b/llm_nim_1b768057.txt
new file mode 100644
index 0000000000000000000000000000000000000000..cec00e2d9f8ab34dab2695e6e391bfac905a8f30
--- /dev/null
+++ b/llm_nim_1b768057.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/llm/nim#additional-notes
+Title: NVIDIA NIM - Pipecat
+==================================================
+
+NVIDIA NIM - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation LLM NVIDIA NIM Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Anthropic AWS Bedrock Azure Cerebras DeepSeek Fireworks AI Google Gemini Google Vertex AI Grok Groq NVIDIA NIM Ollama OpenAI OpenPipe OpenRouter Perplexity Qwen SambaNova Together AI Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview NimLLMService provides access to NVIDIA’s NIM language models through an OpenAI-compatible interface. It inherits from OpenAILLMService and supports streaming responses, function calling, and context management, with special handling for NVIDIA’s incremental token reporting. API Reference Complete API documentation and method details NVIDIA NIM Docs Official NVIDIA NIM documentation and setup Example Code Working example with function calling ​ Installation To use NVIDIA NIM services, install the required dependencies: Copy Ask AI pip install "pipecat-ai[nim]" You’ll also need to set up your NVIDIA API key as an environment variable: NVIDIA_API_KEY . Get your API key from NVIDIA Build . ​ Frames ​ Input OpenAILLMContextFrame - Conversation context and history LLMMessagesFrame - Direct message list VisionImageRawFrame - Images for vision processing LLMUpdateSettingsFrame - Runtime parameter updates ​ Output LLMFullResponseStartFrame / LLMFullResponseEndFrame - Response boundaries LLMTextFrame - Streamed completion chunks FunctionCallInProgressFrame / FunctionCallResultFrame - Function call lifecycle ErrorFrame - API or processing errors ​ Function Calling Function Calling Guide Learn how to implement function calling with standardized schemas, register handlers, manage context properly, and control execution flow in your conversational AI applications. ​ Context Management Context Management Guide Learn how to manage conversation context, handle message history, and integrate context aggregators for consistent conversational experiences. ​ Usage Example Copy Ask AI import os from pipecat.services.nim.llm import NimLLMService from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext from pipecat.adapters.schemas.function_schema import FunctionSchema from pipecat.adapters.schemas.tools_schema import ToolsSchema # Configure NVIDIA NIM service llm = NimLLMService( api_key = os.getenv( "NVIDIA_API_KEY" ), model = "nvidia/llama-3.1-nemotron-70b-instruct" , params = NimLLMService.InputParams( temperature = 0.7 , max_tokens = 1000 ) ) # Define function for tool calling weather_function = FunctionSchema( name = "get_current_weather" , description = "Get current weather information" , properties = { "location" : { "type" : "string" , "description" : "City and state, e.g. San Francisco, CA" }, "format" : { "type" : "string" , "enum" : [ "celsius" , "fahrenheit" ], "description" : "Temperature unit to use" } }, required = [ "location" , "format" ] ) tools = ToolsSchema( standard_tools = [weather_function]) # Create context optimized for voice context = OpenAILLMContext( messages = [ { "role" : "system" , "content" : """You are a helpful assistant optimized for voice interactions. Keep responses concise and avoid special characters for better speech synthesis.""" } ], tools = tools ) # Create context aggregators context_aggregator = llm.create_context_aggregator(context) # Register function handler with feedback async def fetch_weather ( params ): location = params.arguments[ "location" ] await params.result_callback({ "conditions" : "sunny" , "temperature" : "75°F" }) llm.register_function( "get_current_weather" , fetch_weather) # Optional: Add function call feedback @llm.event_handler ( "on_function_calls_started" ) async def on_function_calls_started ( service , function_calls ): await tts.queue_frame(TTSSpeakFrame( "Let me check on that." )) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, context_aggregator.user(), llm, tts, transport.output(), context_aggregator.assistant() ]) ​ Metrics Includes specialized token usage tracking for NIM’s incremental reporting: Time to First Byte (TTFB) - Response latency measurement Processing Duration - Total request processing time Token Usage - Tracks tokens used per request, compatible with NIM’s incremental reporting Enable with: Copy Ask AI task = PipelineTask( pipeline, params = PipelineParams( enable_metrics = True , enable_usage_metrics = True ) ) ​ Additional Notes OpenAI Compatibility : Full compatibility with OpenAI API features and parameters NVIDIA Optimization : Hardware-accelerated inference on NVIDIA infrastructure Token Reporting : Custom handling for NIM’s incremental vs. OpenAI’s final token reporting Model Variety : Access to Nemotron and other NVIDIA-optimized model variants Groq Ollama On this page Overview Installation Frames Input Output Function Calling Context Management Usage Example Metrics Additional Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/llm_nim_5498dcf4.txt b/llm_nim_5498dcf4.txt
new file mode 100644
index 0000000000000000000000000000000000000000..14901daa7fbb1fdc38208b838b46055462ccdbcf
--- /dev/null
+++ b/llm_nim_5498dcf4.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/llm/nim#function-calling
+Title: NVIDIA NIM - Pipecat
+==================================================
+
+NVIDIA NIM - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation LLM NVIDIA NIM Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Anthropic AWS Bedrock Azure Cerebras DeepSeek Fireworks AI Google Gemini Google Vertex AI Grok Groq NVIDIA NIM Ollama OpenAI OpenPipe OpenRouter Perplexity Qwen SambaNova Together AI Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview NimLLMService provides access to NVIDIA’s NIM language models through an OpenAI-compatible interface. It inherits from OpenAILLMService and supports streaming responses, function calling, and context management, with special handling for NVIDIA’s incremental token reporting. API Reference Complete API documentation and method details NVIDIA NIM Docs Official NVIDIA NIM documentation and setup Example Code Working example with function calling ​ Installation To use NVIDIA NIM services, install the required dependencies: Copy Ask AI pip install "pipecat-ai[nim]" You’ll also need to set up your NVIDIA API key as an environment variable: NVIDIA_API_KEY . Get your API key from NVIDIA Build . ​ Frames ​ Input OpenAILLMContextFrame - Conversation context and history LLMMessagesFrame - Direct message list VisionImageRawFrame - Images for vision processing LLMUpdateSettingsFrame - Runtime parameter updates ​ Output LLMFullResponseStartFrame / LLMFullResponseEndFrame - Response boundaries LLMTextFrame - Streamed completion chunks FunctionCallInProgressFrame / FunctionCallResultFrame - Function call lifecycle ErrorFrame - API or processing errors ​ Function Calling Function Calling Guide Learn how to implement function calling with standardized schemas, register handlers, manage context properly, and control execution flow in your conversational AI applications. ​ Context Management Context Management Guide Learn how to manage conversation context, handle message history, and integrate context aggregators for consistent conversational experiences. ​ Usage Example Copy Ask AI import os from pipecat.services.nim.llm import NimLLMService from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext from pipecat.adapters.schemas.function_schema import FunctionSchema from pipecat.adapters.schemas.tools_schema import ToolsSchema # Configure NVIDIA NIM service llm = NimLLMService( api_key = os.getenv( "NVIDIA_API_KEY" ), model = "nvidia/llama-3.1-nemotron-70b-instruct" , params = NimLLMService.InputParams( temperature = 0.7 , max_tokens = 1000 ) ) # Define function for tool calling weather_function = FunctionSchema( name = "get_current_weather" , description = "Get current weather information" , properties = { "location" : { "type" : "string" , "description" : "City and state, e.g. San Francisco, CA" }, "format" : { "type" : "string" , "enum" : [ "celsius" , "fahrenheit" ], "description" : "Temperature unit to use" } }, required = [ "location" , "format" ] ) tools = ToolsSchema( standard_tools = [weather_function]) # Create context optimized for voice context = OpenAILLMContext( messages = [ { "role" : "system" , "content" : """You are a helpful assistant optimized for voice interactions. Keep responses concise and avoid special characters for better speech synthesis.""" } ], tools = tools ) # Create context aggregators context_aggregator = llm.create_context_aggregator(context) # Register function handler with feedback async def fetch_weather ( params ): location = params.arguments[ "location" ] await params.result_callback({ "conditions" : "sunny" , "temperature" : "75°F" }) llm.register_function( "get_current_weather" , fetch_weather) # Optional: Add function call feedback @llm.event_handler ( "on_function_calls_started" ) async def on_function_calls_started ( service , function_calls ): await tts.queue_frame(TTSSpeakFrame( "Let me check on that." )) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, context_aggregator.user(), llm, tts, transport.output(), context_aggregator.assistant() ]) ​ Metrics Includes specialized token usage tracking for NIM’s incremental reporting: Time to First Byte (TTFB) - Response latency measurement Processing Duration - Total request processing time Token Usage - Tracks tokens used per request, compatible with NIM’s incremental reporting Enable with: Copy Ask AI task = PipelineTask( pipeline, params = PipelineParams( enable_metrics = True , enable_usage_metrics = True ) ) ​ Additional Notes OpenAI Compatibility : Full compatibility with OpenAI API features and parameters NVIDIA Optimization : Hardware-accelerated inference on NVIDIA infrastructure Token Reporting : Custom handling for NIM’s incremental vs. OpenAI’s final token reporting Model Variety : Access to Nemotron and other NVIDIA-optimized model variants Groq Ollama On this page Overview Installation Frames Input Output Function Calling Context Management Usage Example Metrics Additional Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/llm_nim_efaa97ed.txt b/llm_nim_efaa97ed.txt
new file mode 100644
index 0000000000000000000000000000000000000000..85a7d83701cad21fe1b30238c52844e5d74cc594
--- /dev/null
+++ b/llm_nim_efaa97ed.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/llm/nim#output
+Title: NVIDIA NIM - Pipecat
+==================================================
+
+NVIDIA NIM - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation LLM NVIDIA NIM Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Anthropic AWS Bedrock Azure Cerebras DeepSeek Fireworks AI Google Gemini Google Vertex AI Grok Groq NVIDIA NIM Ollama OpenAI OpenPipe OpenRouter Perplexity Qwen SambaNova Together AI Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview NimLLMService provides access to NVIDIA’s NIM language models through an OpenAI-compatible interface. It inherits from OpenAILLMService and supports streaming responses, function calling, and context management, with special handling for NVIDIA’s incremental token reporting. API Reference Complete API documentation and method details NVIDIA NIM Docs Official NVIDIA NIM documentation and setup Example Code Working example with function calling ​ Installation To use NVIDIA NIM services, install the required dependencies: Copy Ask AI pip install "pipecat-ai[nim]" You’ll also need to set up your NVIDIA API key as an environment variable: NVIDIA_API_KEY . Get your API key from NVIDIA Build . ​ Frames ​ Input OpenAILLMContextFrame - Conversation context and history LLMMessagesFrame - Direct message list VisionImageRawFrame - Images for vision processing LLMUpdateSettingsFrame - Runtime parameter updates ​ Output LLMFullResponseStartFrame / LLMFullResponseEndFrame - Response boundaries LLMTextFrame - Streamed completion chunks FunctionCallInProgressFrame / FunctionCallResultFrame - Function call lifecycle ErrorFrame - API or processing errors ​ Function Calling Function Calling Guide Learn how to implement function calling with standardized schemas, register handlers, manage context properly, and control execution flow in your conversational AI applications. ​ Context Management Context Management Guide Learn how to manage conversation context, handle message history, and integrate context aggregators for consistent conversational experiences. ​ Usage Example Copy Ask AI import os from pipecat.services.nim.llm import NimLLMService from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext from pipecat.adapters.schemas.function_schema import FunctionSchema from pipecat.adapters.schemas.tools_schema import ToolsSchema # Configure NVIDIA NIM service llm = NimLLMService( api_key = os.getenv( "NVIDIA_API_KEY" ), model = "nvidia/llama-3.1-nemotron-70b-instruct" , params = NimLLMService.InputParams( temperature = 0.7 , max_tokens = 1000 ) ) # Define function for tool calling weather_function = FunctionSchema( name = "get_current_weather" , description = "Get current weather information" , properties = { "location" : { "type" : "string" , "description" : "City and state, e.g. San Francisco, CA" }, "format" : { "type" : "string" , "enum" : [ "celsius" , "fahrenheit" ], "description" : "Temperature unit to use" } }, required = [ "location" , "format" ] ) tools = ToolsSchema( standard_tools = [weather_function]) # Create context optimized for voice context = OpenAILLMContext( messages = [ { "role" : "system" , "content" : """You are a helpful assistant optimized for voice interactions. Keep responses concise and avoid special characters for better speech synthesis.""" } ], tools = tools ) # Create context aggregators context_aggregator = llm.create_context_aggregator(context) # Register function handler with feedback async def fetch_weather ( params ): location = params.arguments[ "location" ] await params.result_callback({ "conditions" : "sunny" , "temperature" : "75°F" }) llm.register_function( "get_current_weather" , fetch_weather) # Optional: Add function call feedback @llm.event_handler ( "on_function_calls_started" ) async def on_function_calls_started ( service , function_calls ): await tts.queue_frame(TTSSpeakFrame( "Let me check on that." )) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, context_aggregator.user(), llm, tts, transport.output(), context_aggregator.assistant() ]) ​ Metrics Includes specialized token usage tracking for NIM’s incremental reporting: Time to First Byte (TTFB) - Response latency measurement Processing Duration - Total request processing time Token Usage - Tracks tokens used per request, compatible with NIM’s incremental reporting Enable with: Copy Ask AI task = PipelineTask( pipeline, params = PipelineParams( enable_metrics = True , enable_usage_metrics = True ) ) ​ Additional Notes OpenAI Compatibility : Full compatibility with OpenAI API features and parameters NVIDIA Optimization : Hardware-accelerated inference on NVIDIA infrastructure Token Reporting : Custom handling for NIM’s incremental vs. OpenAI’s final token reporting Model Variety : Access to Nemotron and other NVIDIA-optimized model variants Groq Ollama On this page Overview Installation Frames Input Output Function Calling Context Management Usage Example Metrics Additional Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/llm_ollama_5178a1e5.txt b/llm_ollama_5178a1e5.txt
new file mode 100644
index 0000000000000000000000000000000000000000..60a45a4a558df827b6c74e0c219e938b185c8dbf
--- /dev/null
+++ b/llm_ollama_5178a1e5.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/llm/ollama#context-management
+Title: Ollama - Pipecat
+==================================================
+
+Ollama - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation LLM Ollama Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Anthropic AWS Bedrock Azure Cerebras DeepSeek Fireworks AI Google Gemini Google Vertex AI Grok Groq NVIDIA NIM Ollama OpenAI OpenPipe OpenRouter Perplexity Qwen SambaNova Together AI Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview OLLamaLLMService provides access to locally-run Ollama models through an OpenAI-compatible interface. It inherits from BaseOpenAILLMService and allows you to run various open-source models locally while maintaining compatibility with OpenAI’s API format. API Reference Complete API documentation and method details Ollama Docs Official Ollama documentation and model library Download Ollama Download and setup instructions for Ollama ​ Installation To use Ollama services, you need to install both Ollama and the Pipecat dependency: Install Ollama on your system from ollama.com/download Install Pipecat dependency : Copy Ask AI pip install "pipecat-ai[ollama]" Pull a model (first time only): Copy Ask AI ollama pull llama2 Ollama runs as a local service on port 11434. No API key required! ​ Frames ​ Input OpenAILLMContextFrame - Conversation context and history LLMMessagesFrame - Direct message list VisionImageRawFrame - Images for vision models LLMUpdateSettingsFrame - Runtime parameter updates ​ Output LLMFullResponseStartFrame / LLMFullResponseEndFrame - Response boundaries LLMTextFrame - Streamed completion chunks FunctionCallInProgressFrame / FunctionCallResultFrame - Function call lifecycle ErrorFrame - Connection or processing errors ​ Function Calling Function Calling Guide Learn how to implement function calling with standardized schemas, register handlers, manage context properly, and control execution flow in your conversational AI applications. ​ Context Management Context Management Guide Learn how to manage conversation context, handle message history, and integrate context aggregators for consistent conversational experiences. ​ Usage Example Copy Ask AI from pipecat.services.ollama.llm import OLLamaLLMService from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext from pipecat.adapters.schemas.function_schema import FunctionSchema from pipecat.adapters.schemas.tools_schema import ToolsSchema # Configure local Ollama service llm = OLLamaLLMService( model = "llama3.1" , # Must be pulled first: ollama pull llama3.1 base_url = "http://localhost:11434/v1" , # Default Ollama endpoint params = OLLamaLLMService.InputParams( temperature = 0.7 , max_tokens = 1000 ) ) # Define function for local processing weather_function = FunctionSchema( name = "get_current_weather" , description = "Get current weather information" , properties = { "location" : { "type" : "string" , "description" : "City and state, e.g. San Francisco, CA" }, "format" : { "type" : "string" , "enum" : [ "celsius" , "fahrenheit" ], "description" : "Temperature unit to use" } }, required = [ "location" , "format" ] ) tools = ToolsSchema( standard_tools = [weather_function]) # Create context optimized for local model context = OpenAILLMContext( messages = [ { "role" : "system" , "content" : """You are a helpful assistant running locally. Be concise and efficient in your responses while maintaining helpfulness.""" } ], tools = tools ) # Create context aggregators context_aggregator = llm.create_context_aggregator(context) # Register function handler - all processing stays local async def fetch_weather ( params ): location = params.arguments[ "location" ] # Local weather lookup or cached data await params.result_callback({ "conditions" : "sunny" , "temperature" : "22°C" }) llm.register_function( "get_current_weather" , fetch_weather) # Use in pipeline - completely offline capable pipeline = Pipeline([ transport.input(), stt, # Can use local STT too context_aggregator.user(), llm, # All inference happens locally tts, # Can use local TTS too transport.output(), context_aggregator.assistant() ]) ​ Metrics Inherits all OpenAI metrics capabilities for local monitoring: Time to First Byte (TTFB) - Local inference latency Processing Duration - Model execution time Token Usage - Local token counting (if supported by model) Enable with: Copy Ask AI task = PipelineTask( pipeline, params = PipelineParams( enable_metrics = True , enable_usage_metrics = True ) ) ​ Additional Notes Run models locally : Ollama allows you to run various open-source models on your own hardware, providing flexibility and control. OpenAI Compatibility : Full compatibility with OpenAI API features and parameters Privacy centric : All processing happens locally, ensuring data privacy and security. NVIDIA NIM OpenAI On this page Overview Installation Frames Input Output Function Calling Context Management Usage Example Metrics Additional Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/llm_ollama_64d3ee76.txt b/llm_ollama_64d3ee76.txt
new file mode 100644
index 0000000000000000000000000000000000000000..1a13f67d5cef9b8a513b46151eadc8c3e6677a58
--- /dev/null
+++ b/llm_ollama_64d3ee76.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/llm/ollama#function-calling
+Title: Ollama - Pipecat
+==================================================
+
+Ollama - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation LLM Ollama Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Anthropic AWS Bedrock Azure Cerebras DeepSeek Fireworks AI Google Gemini Google Vertex AI Grok Groq NVIDIA NIM Ollama OpenAI OpenPipe OpenRouter Perplexity Qwen SambaNova Together AI Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview OLLamaLLMService provides access to locally-run Ollama models through an OpenAI-compatible interface. It inherits from BaseOpenAILLMService and allows you to run various open-source models locally while maintaining compatibility with OpenAI’s API format. API Reference Complete API documentation and method details Ollama Docs Official Ollama documentation and model library Download Ollama Download and setup instructions for Ollama ​ Installation To use Ollama services, you need to install both Ollama and the Pipecat dependency: Install Ollama on your system from ollama.com/download Install Pipecat dependency : Copy Ask AI pip install "pipecat-ai[ollama]" Pull a model (first time only): Copy Ask AI ollama pull llama2 Ollama runs as a local service on port 11434. No API key required! ​ Frames ​ Input OpenAILLMContextFrame - Conversation context and history LLMMessagesFrame - Direct message list VisionImageRawFrame - Images for vision models LLMUpdateSettingsFrame - Runtime parameter updates ​ Output LLMFullResponseStartFrame / LLMFullResponseEndFrame - Response boundaries LLMTextFrame - Streamed completion chunks FunctionCallInProgressFrame / FunctionCallResultFrame - Function call lifecycle ErrorFrame - Connection or processing errors ​ Function Calling Function Calling Guide Learn how to implement function calling with standardized schemas, register handlers, manage context properly, and control execution flow in your conversational AI applications. ​ Context Management Context Management Guide Learn how to manage conversation context, handle message history, and integrate context aggregators for consistent conversational experiences. ​ Usage Example Copy Ask AI from pipecat.services.ollama.llm import OLLamaLLMService from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext from pipecat.adapters.schemas.function_schema import FunctionSchema from pipecat.adapters.schemas.tools_schema import ToolsSchema # Configure local Ollama service llm = OLLamaLLMService( model = "llama3.1" , # Must be pulled first: ollama pull llama3.1 base_url = "http://localhost:11434/v1" , # Default Ollama endpoint params = OLLamaLLMService.InputParams( temperature = 0.7 , max_tokens = 1000 ) ) # Define function for local processing weather_function = FunctionSchema( name = "get_current_weather" , description = "Get current weather information" , properties = { "location" : { "type" : "string" , "description" : "City and state, e.g. San Francisco, CA" }, "format" : { "type" : "string" , "enum" : [ "celsius" , "fahrenheit" ], "description" : "Temperature unit to use" } }, required = [ "location" , "format" ] ) tools = ToolsSchema( standard_tools = [weather_function]) # Create context optimized for local model context = OpenAILLMContext( messages = [ { "role" : "system" , "content" : """You are a helpful assistant running locally. Be concise and efficient in your responses while maintaining helpfulness.""" } ], tools = tools ) # Create context aggregators context_aggregator = llm.create_context_aggregator(context) # Register function handler - all processing stays local async def fetch_weather ( params ): location = params.arguments[ "location" ] # Local weather lookup or cached data await params.result_callback({ "conditions" : "sunny" , "temperature" : "22°C" }) llm.register_function( "get_current_weather" , fetch_weather) # Use in pipeline - completely offline capable pipeline = Pipeline([ transport.input(), stt, # Can use local STT too context_aggregator.user(), llm, # All inference happens locally tts, # Can use local TTS too transport.output(), context_aggregator.assistant() ]) ​ Metrics Inherits all OpenAI metrics capabilities for local monitoring: Time to First Byte (TTFB) - Local inference latency Processing Duration - Model execution time Token Usage - Local token counting (if supported by model) Enable with: Copy Ask AI task = PipelineTask( pipeline, params = PipelineParams( enable_metrics = True , enable_usage_metrics = True ) ) ​ Additional Notes Run models locally : Ollama allows you to run various open-source models on your own hardware, providing flexibility and control. OpenAI Compatibility : Full compatibility with OpenAI API features and parameters Privacy centric : All processing happens locally, ensuring data privacy and security. NVIDIA NIM OpenAI On this page Overview Installation Frames Input Output Function Calling Context Management Usage Example Metrics Additional Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/llm_ollama_9960d1f5.txt b/llm_ollama_9960d1f5.txt
new file mode 100644
index 0000000000000000000000000000000000000000..3c66eb9495ddeb36dcbfedd947a5fc445f65de6e
--- /dev/null
+++ b/llm_ollama_9960d1f5.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/llm/ollama#installation
+Title: Ollama - Pipecat
+==================================================
+
+Ollama - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation LLM Ollama Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Anthropic AWS Bedrock Azure Cerebras DeepSeek Fireworks AI Google Gemini Google Vertex AI Grok Groq NVIDIA NIM Ollama OpenAI OpenPipe OpenRouter Perplexity Qwen SambaNova Together AI Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview OLLamaLLMService provides access to locally-run Ollama models through an OpenAI-compatible interface. It inherits from BaseOpenAILLMService and allows you to run various open-source models locally while maintaining compatibility with OpenAI’s API format. API Reference Complete API documentation and method details Ollama Docs Official Ollama documentation and model library Download Ollama Download and setup instructions for Ollama ​ Installation To use Ollama services, you need to install both Ollama and the Pipecat dependency: Install Ollama on your system from ollama.com/download Install Pipecat dependency : Copy Ask AI pip install "pipecat-ai[ollama]" Pull a model (first time only): Copy Ask AI ollama pull llama2 Ollama runs as a local service on port 11434. No API key required! ​ Frames ​ Input OpenAILLMContextFrame - Conversation context and history LLMMessagesFrame - Direct message list VisionImageRawFrame - Images for vision models LLMUpdateSettingsFrame - Runtime parameter updates ​ Output LLMFullResponseStartFrame / LLMFullResponseEndFrame - Response boundaries LLMTextFrame - Streamed completion chunks FunctionCallInProgressFrame / FunctionCallResultFrame - Function call lifecycle ErrorFrame - Connection or processing errors ​ Function Calling Function Calling Guide Learn how to implement function calling with standardized schemas, register handlers, manage context properly, and control execution flow in your conversational AI applications. ​ Context Management Context Management Guide Learn how to manage conversation context, handle message history, and integrate context aggregators for consistent conversational experiences. ​ Usage Example Copy Ask AI from pipecat.services.ollama.llm import OLLamaLLMService from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext from pipecat.adapters.schemas.function_schema import FunctionSchema from pipecat.adapters.schemas.tools_schema import ToolsSchema # Configure local Ollama service llm = OLLamaLLMService( model = "llama3.1" , # Must be pulled first: ollama pull llama3.1 base_url = "http://localhost:11434/v1" , # Default Ollama endpoint params = OLLamaLLMService.InputParams( temperature = 0.7 , max_tokens = 1000 ) ) # Define function for local processing weather_function = FunctionSchema( name = "get_current_weather" , description = "Get current weather information" , properties = { "location" : { "type" : "string" , "description" : "City and state, e.g. San Francisco, CA" }, "format" : { "type" : "string" , "enum" : [ "celsius" , "fahrenheit" ], "description" : "Temperature unit to use" } }, required = [ "location" , "format" ] ) tools = ToolsSchema( standard_tools = [weather_function]) # Create context optimized for local model context = OpenAILLMContext( messages = [ { "role" : "system" , "content" : """You are a helpful assistant running locally. Be concise and efficient in your responses while maintaining helpfulness.""" } ], tools = tools ) # Create context aggregators context_aggregator = llm.create_context_aggregator(context) # Register function handler - all processing stays local async def fetch_weather ( params ): location = params.arguments[ "location" ] # Local weather lookup or cached data await params.result_callback({ "conditions" : "sunny" , "temperature" : "22°C" }) llm.register_function( "get_current_weather" , fetch_weather) # Use in pipeline - completely offline capable pipeline = Pipeline([ transport.input(), stt, # Can use local STT too context_aggregator.user(), llm, # All inference happens locally tts, # Can use local TTS too transport.output(), context_aggregator.assistant() ]) ​ Metrics Inherits all OpenAI metrics capabilities for local monitoring: Time to First Byte (TTFB) - Local inference latency Processing Duration - Model execution time Token Usage - Local token counting (if supported by model) Enable with: Copy Ask AI task = PipelineTask( pipeline, params = PipelineParams( enable_metrics = True , enable_usage_metrics = True ) ) ​ Additional Notes Run models locally : Ollama allows you to run various open-source models on your own hardware, providing flexibility and control. OpenAI Compatibility : Full compatibility with OpenAI API features and parameters Privacy centric : All processing happens locally, ensuring data privacy and security. NVIDIA NIM OpenAI On this page Overview Installation Frames Input Output Function Calling Context Management Usage Example Metrics Additional Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/llm_ollama_a251200b.txt b/llm_ollama_a251200b.txt
new file mode 100644
index 0000000000000000000000000000000000000000..78d1b317022f42069bd80abd0bfb19b5ed72722d
--- /dev/null
+++ b/llm_ollama_a251200b.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/llm/ollama#usage-example
+Title: Ollama - Pipecat
+==================================================
+
+Ollama - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation LLM Ollama Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Anthropic AWS Bedrock Azure Cerebras DeepSeek Fireworks AI Google Gemini Google Vertex AI Grok Groq NVIDIA NIM Ollama OpenAI OpenPipe OpenRouter Perplexity Qwen SambaNova Together AI Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview OLLamaLLMService provides access to locally-run Ollama models through an OpenAI-compatible interface. It inherits from BaseOpenAILLMService and allows you to run various open-source models locally while maintaining compatibility with OpenAI’s API format. API Reference Complete API documentation and method details Ollama Docs Official Ollama documentation and model library Download Ollama Download and setup instructions for Ollama ​ Installation To use Ollama services, you need to install both Ollama and the Pipecat dependency: Install Ollama on your system from ollama.com/download Install Pipecat dependency : Copy Ask AI pip install "pipecat-ai[ollama]" Pull a model (first time only): Copy Ask AI ollama pull llama2 Ollama runs as a local service on port 11434. No API key required! ​ Frames ​ Input OpenAILLMContextFrame - Conversation context and history LLMMessagesFrame - Direct message list VisionImageRawFrame - Images for vision models LLMUpdateSettingsFrame - Runtime parameter updates ​ Output LLMFullResponseStartFrame / LLMFullResponseEndFrame - Response boundaries LLMTextFrame - Streamed completion chunks FunctionCallInProgressFrame / FunctionCallResultFrame - Function call lifecycle ErrorFrame - Connection or processing errors ​ Function Calling Function Calling Guide Learn how to implement function calling with standardized schemas, register handlers, manage context properly, and control execution flow in your conversational AI applications. ​ Context Management Context Management Guide Learn how to manage conversation context, handle message history, and integrate context aggregators for consistent conversational experiences. ​ Usage Example Copy Ask AI from pipecat.services.ollama.llm import OLLamaLLMService from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext from pipecat.adapters.schemas.function_schema import FunctionSchema from pipecat.adapters.schemas.tools_schema import ToolsSchema # Configure local Ollama service llm = OLLamaLLMService( model = "llama3.1" , # Must be pulled first: ollama pull llama3.1 base_url = "http://localhost:11434/v1" , # Default Ollama endpoint params = OLLamaLLMService.InputParams( temperature = 0.7 , max_tokens = 1000 ) ) # Define function for local processing weather_function = FunctionSchema( name = "get_current_weather" , description = "Get current weather information" , properties = { "location" : { "type" : "string" , "description" : "City and state, e.g. San Francisco, CA" }, "format" : { "type" : "string" , "enum" : [ "celsius" , "fahrenheit" ], "description" : "Temperature unit to use" } }, required = [ "location" , "format" ] ) tools = ToolsSchema( standard_tools = [weather_function]) # Create context optimized for local model context = OpenAILLMContext( messages = [ { "role" : "system" , "content" : """You are a helpful assistant running locally. Be concise and efficient in your responses while maintaining helpfulness.""" } ], tools = tools ) # Create context aggregators context_aggregator = llm.create_context_aggregator(context) # Register function handler - all processing stays local async def fetch_weather ( params ): location = params.arguments[ "location" ] # Local weather lookup or cached data await params.result_callback({ "conditions" : "sunny" , "temperature" : "22°C" }) llm.register_function( "get_current_weather" , fetch_weather) # Use in pipeline - completely offline capable pipeline = Pipeline([ transport.input(), stt, # Can use local STT too context_aggregator.user(), llm, # All inference happens locally tts, # Can use local TTS too transport.output(), context_aggregator.assistant() ]) ​ Metrics Inherits all OpenAI metrics capabilities for local monitoring: Time to First Byte (TTFB) - Local inference latency Processing Duration - Model execution time Token Usage - Local token counting (if supported by model) Enable with: Copy Ask AI task = PipelineTask( pipeline, params = PipelineParams( enable_metrics = True , enable_usage_metrics = True ) ) ​ Additional Notes Run models locally : Ollama allows you to run various open-source models on your own hardware, providing flexibility and control. OpenAI Compatibility : Full compatibility with OpenAI API features and parameters Privacy centric : All processing happens locally, ensuring data privacy and security. NVIDIA NIM OpenAI On this page Overview Installation Frames Input Output Function Calling Context Management Usage Example Metrics Additional Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/llm_openai_15ec657b.txt b/llm_openai_15ec657b.txt
new file mode 100644
index 0000000000000000000000000000000000000000..5117e07fbd4d534d249f1b7d04daae643c895682
--- /dev/null
+++ b/llm_openai_15ec657b.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/llm/openai#output
+Title: OpenAI - Pipecat
+==================================================
+
+OpenAI - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation LLM OpenAI Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Anthropic AWS Bedrock Azure Cerebras DeepSeek Fireworks AI Google Gemini Google Vertex AI Grok Groq NVIDIA NIM Ollama OpenAI OpenPipe OpenRouter Perplexity Qwen SambaNova Together AI Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview OpenAILLMService provides chat completion capabilities using OpenAI’s API, supporting streaming responses, function calling, vision input, and advanced context management for conversational AI applications. API Reference Complete API documentation and method details OpenAI Docs Official OpenAI API documentation Example Code Function calling example with weather API ​ Installation To use OpenAI services, install the required dependencies: Copy Ask AI pip install "pipecat-ai[openai]" You’ll also need to set up your OpenAI API key as an environment variable: OPENAI_API_KEY . Get your API key from the OpenAI Platform . ​ Frames ​ Input OpenAILLMContextFrame - OpenAI-specific conversation context LLMMessagesFrame - Standard conversation messages VisionImageRawFrame - Images for vision model processing LLMUpdateSettingsFrame - Runtime model configuration updates ​ Output LLMFullResponseStartFrame / LLMFullResponseEndFrame - Response boundaries LLMTextFrame - Streamed completion chunks FunctionCallInProgressFrame / FunctionCallResultFrame - Function call lifecycle ErrorFrame - API or processing errors ​ Function Calling Function Calling Guide Learn how to implement function calling with standardized schemas, register handlers, manage context properly, and control execution flow in your conversational AI applications. ​ Context Management Context Management Guide Learn how to manage conversation context, handle message history, and integrate context aggregators for consistent conversational experiences. ​ Usage Example ​ Basic Conversation with Function Calling Copy Ask AI import os from pipecat.services.openai.llm import OpenAILLMService from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext from pipecat.adapters.schemas.function_schema import FunctionSchema from pipecat.adapters.schemas.tools_schema import ToolsSchema from pipecat.services.llm_service import FunctionCallParams # Configure the service llm = OpenAILLMService( model = "gpt-4o" , api_key = os.getenv( "OPENAI_API_KEY" ), params = OpenAILLMService.InputParams( temperature = 0.7 , ) ) # Define function schema weather_function = FunctionSchema( name = "get_weather" , description = "Get current weather information" , properties = { "location" : { "type" : "string" , "description" : "City name" } }, required = [ "location" ] ) # Create tools and context tools = ToolsSchema( standard_tools = [weather_function]) context = OpenAILLMContext( messages = [{ "role" : "system" , "content" : "You are a helpful assistant. Keep responses concise." }], tools = tools ) # Register function handler async def get_weather_handler ( params : FunctionCallParams): location = params.arguments.get( "location" ) # Call weather API here... weather_data = { "temperature" : "75°F" , "conditions" : "sunny" } await params.result_callback(weather_data) llm.register_function( "get_weather" , get_weather_handler) # Create context aggregators context_aggregator = llm.create_context_aggregator(context) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, context_aggregator.user(), # Handles user messages llm, # Processes with OpenAI tts, transport.output(), context_aggregator.assistant() # Captures responses ]) ​ Metrics The service provides: Time to First Byte (TTFB) - Latency from request to first response token Processing Duration - Total request processing time Token Usage - Prompt tokens, completion tokens, and total usage Enable with: Copy Ask AI task = PipelineTask( pipeline, params = PipelineParams( enable_metrics = True , enable_usage_metrics = True ) ) ​ Additional Notes Streaming Responses : All responses are streamed for low latency Context Persistence : Use context aggregators to maintain conversation history Error Handling : Automatic retry logic for rate limits and transient errors Compatible Services : Works with OpenAI-compatible APIs by setting base_url Ollama OpenPipe On this page Overview Installation Frames Input Output Function Calling Context Management Usage Example Basic Conversation with Function Calling Metrics Additional Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/llm_openai_2c6f2215.txt b/llm_openai_2c6f2215.txt
new file mode 100644
index 0000000000000000000000000000000000000000..7acdd660084f29531938935c589a1874e4e4a095
--- /dev/null
+++ b/llm_openai_2c6f2215.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/llm/openai#context-management
+Title: OpenAI - Pipecat
+==================================================
+
+OpenAI - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation LLM OpenAI Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Anthropic AWS Bedrock Azure Cerebras DeepSeek Fireworks AI Google Gemini Google Vertex AI Grok Groq NVIDIA NIM Ollama OpenAI OpenPipe OpenRouter Perplexity Qwen SambaNova Together AI Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview OpenAILLMService provides chat completion capabilities using OpenAI’s API, supporting streaming responses, function calling, vision input, and advanced context management for conversational AI applications. API Reference Complete API documentation and method details OpenAI Docs Official OpenAI API documentation Example Code Function calling example with weather API ​ Installation To use OpenAI services, install the required dependencies: Copy Ask AI pip install "pipecat-ai[openai]" You’ll also need to set up your OpenAI API key as an environment variable: OPENAI_API_KEY . Get your API key from the OpenAI Platform . ​ Frames ​ Input OpenAILLMContextFrame - OpenAI-specific conversation context LLMMessagesFrame - Standard conversation messages VisionImageRawFrame - Images for vision model processing LLMUpdateSettingsFrame - Runtime model configuration updates ​ Output LLMFullResponseStartFrame / LLMFullResponseEndFrame - Response boundaries LLMTextFrame - Streamed completion chunks FunctionCallInProgressFrame / FunctionCallResultFrame - Function call lifecycle ErrorFrame - API or processing errors ​ Function Calling Function Calling Guide Learn how to implement function calling with standardized schemas, register handlers, manage context properly, and control execution flow in your conversational AI applications. ​ Context Management Context Management Guide Learn how to manage conversation context, handle message history, and integrate context aggregators for consistent conversational experiences. ​ Usage Example ​ Basic Conversation with Function Calling Copy Ask AI import os from pipecat.services.openai.llm import OpenAILLMService from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext from pipecat.adapters.schemas.function_schema import FunctionSchema from pipecat.adapters.schemas.tools_schema import ToolsSchema from pipecat.services.llm_service import FunctionCallParams # Configure the service llm = OpenAILLMService( model = "gpt-4o" , api_key = os.getenv( "OPENAI_API_KEY" ), params = OpenAILLMService.InputParams( temperature = 0.7 , ) ) # Define function schema weather_function = FunctionSchema( name = "get_weather" , description = "Get current weather information" , properties = { "location" : { "type" : "string" , "description" : "City name" } }, required = [ "location" ] ) # Create tools and context tools = ToolsSchema( standard_tools = [weather_function]) context = OpenAILLMContext( messages = [{ "role" : "system" , "content" : "You are a helpful assistant. Keep responses concise." }], tools = tools ) # Register function handler async def get_weather_handler ( params : FunctionCallParams): location = params.arguments.get( "location" ) # Call weather API here... weather_data = { "temperature" : "75°F" , "conditions" : "sunny" } await params.result_callback(weather_data) llm.register_function( "get_weather" , get_weather_handler) # Create context aggregators context_aggregator = llm.create_context_aggregator(context) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, context_aggregator.user(), # Handles user messages llm, # Processes with OpenAI tts, transport.output(), context_aggregator.assistant() # Captures responses ]) ​ Metrics The service provides: Time to First Byte (TTFB) - Latency from request to first response token Processing Duration - Total request processing time Token Usage - Prompt tokens, completion tokens, and total usage Enable with: Copy Ask AI task = PipelineTask( pipeline, params = PipelineParams( enable_metrics = True , enable_usage_metrics = True ) ) ​ Additional Notes Streaming Responses : All responses are streamed for low latency Context Persistence : Use context aggregators to maintain conversation history Error Handling : Automatic retry logic for rate limits and transient errors Compatible Services : Works with OpenAI-compatible APIs by setting base_url Ollama OpenPipe On this page Overview Installation Frames Input Output Function Calling Context Management Usage Example Basic Conversation with Function Calling Metrics Additional Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/llm_openai_4fb2f175.txt b/llm_openai_4fb2f175.txt
new file mode 100644
index 0000000000000000000000000000000000000000..a2e563acc340951285aba453489a8b51cfc65334
--- /dev/null
+++ b/llm_openai_4fb2f175.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/llm/openai#metrics
+Title: OpenAI - Pipecat
+==================================================
+
+OpenAI - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation LLM OpenAI Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Anthropic AWS Bedrock Azure Cerebras DeepSeek Fireworks AI Google Gemini Google Vertex AI Grok Groq NVIDIA NIM Ollama OpenAI OpenPipe OpenRouter Perplexity Qwen SambaNova Together AI Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview OpenAILLMService provides chat completion capabilities using OpenAI’s API, supporting streaming responses, function calling, vision input, and advanced context management for conversational AI applications. API Reference Complete API documentation and method details OpenAI Docs Official OpenAI API documentation Example Code Function calling example with weather API ​ Installation To use OpenAI services, install the required dependencies: Copy Ask AI pip install "pipecat-ai[openai]" You’ll also need to set up your OpenAI API key as an environment variable: OPENAI_API_KEY . Get your API key from the OpenAI Platform . ​ Frames ​ Input OpenAILLMContextFrame - OpenAI-specific conversation context LLMMessagesFrame - Standard conversation messages VisionImageRawFrame - Images for vision model processing LLMUpdateSettingsFrame - Runtime model configuration updates ​ Output LLMFullResponseStartFrame / LLMFullResponseEndFrame - Response boundaries LLMTextFrame - Streamed completion chunks FunctionCallInProgressFrame / FunctionCallResultFrame - Function call lifecycle ErrorFrame - API or processing errors ​ Function Calling Function Calling Guide Learn how to implement function calling with standardized schemas, register handlers, manage context properly, and control execution flow in your conversational AI applications. ​ Context Management Context Management Guide Learn how to manage conversation context, handle message history, and integrate context aggregators for consistent conversational experiences. ​ Usage Example ​ Basic Conversation with Function Calling Copy Ask AI import os from pipecat.services.openai.llm import OpenAILLMService from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext from pipecat.adapters.schemas.function_schema import FunctionSchema from pipecat.adapters.schemas.tools_schema import ToolsSchema from pipecat.services.llm_service import FunctionCallParams # Configure the service llm = OpenAILLMService( model = "gpt-4o" , api_key = os.getenv( "OPENAI_API_KEY" ), params = OpenAILLMService.InputParams( temperature = 0.7 , ) ) # Define function schema weather_function = FunctionSchema( name = "get_weather" , description = "Get current weather information" , properties = { "location" : { "type" : "string" , "description" : "City name" } }, required = [ "location" ] ) # Create tools and context tools = ToolsSchema( standard_tools = [weather_function]) context = OpenAILLMContext( messages = [{ "role" : "system" , "content" : "You are a helpful assistant. Keep responses concise." }], tools = tools ) # Register function handler async def get_weather_handler ( params : FunctionCallParams): location = params.arguments.get( "location" ) # Call weather API here... weather_data = { "temperature" : "75°F" , "conditions" : "sunny" } await params.result_callback(weather_data) llm.register_function( "get_weather" , get_weather_handler) # Create context aggregators context_aggregator = llm.create_context_aggregator(context) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, context_aggregator.user(), # Handles user messages llm, # Processes with OpenAI tts, transport.output(), context_aggregator.assistant() # Captures responses ]) ​ Metrics The service provides: Time to First Byte (TTFB) - Latency from request to first response token Processing Duration - Total request processing time Token Usage - Prompt tokens, completion tokens, and total usage Enable with: Copy Ask AI task = PipelineTask( pipeline, params = PipelineParams( enable_metrics = True , enable_usage_metrics = True ) ) ​ Additional Notes Streaming Responses : All responses are streamed for low latency Context Persistence : Use context aggregators to maintain conversation history Error Handling : Automatic retry logic for rate limits and transient errors Compatible Services : Works with OpenAI-compatible APIs by setting base_url Ollama OpenPipe On this page Overview Installation Frames Input Output Function Calling Context Management Usage Example Basic Conversation with Function Calling Metrics Additional Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/llm_openai_e92d9b54.txt b/llm_openai_e92d9b54.txt
new file mode 100644
index 0000000000000000000000000000000000000000..cae78913f07827d77312a9eaa2f31cf6482f7752
--- /dev/null
+++ b/llm_openai_e92d9b54.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/llm/openai#usage-example
+Title: OpenAI - Pipecat
+==================================================
+
+OpenAI - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation LLM OpenAI Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Anthropic AWS Bedrock Azure Cerebras DeepSeek Fireworks AI Google Gemini Google Vertex AI Grok Groq NVIDIA NIM Ollama OpenAI OpenPipe OpenRouter Perplexity Qwen SambaNova Together AI Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview OpenAILLMService provides chat completion capabilities using OpenAI’s API, supporting streaming responses, function calling, vision input, and advanced context management for conversational AI applications. API Reference Complete API documentation and method details OpenAI Docs Official OpenAI API documentation Example Code Function calling example with weather API ​ Installation To use OpenAI services, install the required dependencies: Copy Ask AI pip install "pipecat-ai[openai]" You’ll also need to set up your OpenAI API key as an environment variable: OPENAI_API_KEY . Get your API key from the OpenAI Platform . ​ Frames ​ Input OpenAILLMContextFrame - OpenAI-specific conversation context LLMMessagesFrame - Standard conversation messages VisionImageRawFrame - Images for vision model processing LLMUpdateSettingsFrame - Runtime model configuration updates ​ Output LLMFullResponseStartFrame / LLMFullResponseEndFrame - Response boundaries LLMTextFrame - Streamed completion chunks FunctionCallInProgressFrame / FunctionCallResultFrame - Function call lifecycle ErrorFrame - API or processing errors ​ Function Calling Function Calling Guide Learn how to implement function calling with standardized schemas, register handlers, manage context properly, and control execution flow in your conversational AI applications. ​ Context Management Context Management Guide Learn how to manage conversation context, handle message history, and integrate context aggregators for consistent conversational experiences. ​ Usage Example ​ Basic Conversation with Function Calling Copy Ask AI import os from pipecat.services.openai.llm import OpenAILLMService from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext from pipecat.adapters.schemas.function_schema import FunctionSchema from pipecat.adapters.schemas.tools_schema import ToolsSchema from pipecat.services.llm_service import FunctionCallParams # Configure the service llm = OpenAILLMService( model = "gpt-4o" , api_key = os.getenv( "OPENAI_API_KEY" ), params = OpenAILLMService.InputParams( temperature = 0.7 , ) ) # Define function schema weather_function = FunctionSchema( name = "get_weather" , description = "Get current weather information" , properties = { "location" : { "type" : "string" , "description" : "City name" } }, required = [ "location" ] ) # Create tools and context tools = ToolsSchema( standard_tools = [weather_function]) context = OpenAILLMContext( messages = [{ "role" : "system" , "content" : "You are a helpful assistant. Keep responses concise." }], tools = tools ) # Register function handler async def get_weather_handler ( params : FunctionCallParams): location = params.arguments.get( "location" ) # Call weather API here... weather_data = { "temperature" : "75°F" , "conditions" : "sunny" } await params.result_callback(weather_data) llm.register_function( "get_weather" , get_weather_handler) # Create context aggregators context_aggregator = llm.create_context_aggregator(context) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, context_aggregator.user(), # Handles user messages llm, # Processes with OpenAI tts, transport.output(), context_aggregator.assistant() # Captures responses ]) ​ Metrics The service provides: Time to First Byte (TTFB) - Latency from request to first response token Processing Duration - Total request processing time Token Usage - Prompt tokens, completion tokens, and total usage Enable with: Copy Ask AI task = PipelineTask( pipeline, params = PipelineParams( enable_metrics = True , enable_usage_metrics = True ) ) ​ Additional Notes Streaming Responses : All responses are streamed for low latency Context Persistence : Use context aggregators to maintain conversation history Error Handling : Automatic retry logic for rate limits and transient errors Compatible Services : Works with OpenAI-compatible APIs by setting base_url Ollama OpenPipe On this page Overview Installation Frames Input Output Function Calling Context Management Usage Example Basic Conversation with Function Calling Metrics Additional Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/llm_openpipe_632aff35.txt b/llm_openpipe_632aff35.txt
new file mode 100644
index 0000000000000000000000000000000000000000..439a76eaf96f791d044d7d0dcfba71f30869d79c
--- /dev/null
+++ b/llm_openpipe_632aff35.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/llm/openpipe#metrics
+Title: OpenPipe - Pipecat
+==================================================
+
+OpenPipe - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation LLM OpenPipe Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Anthropic AWS Bedrock Azure Cerebras DeepSeek Fireworks AI Google Gemini Google Vertex AI Grok Groq NVIDIA NIM Ollama OpenAI OpenPipe OpenRouter Perplexity Qwen SambaNova Together AI Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview OpenPipeLLMService extends the BaseOpenAILLMService to provide integration with OpenPipe, enabling request logging, model fine-tuning, and performance monitoring. It maintains compatibility with OpenAI’s API while adding OpenPipe’s logging and optimization capabilities. API Reference Complete API documentation and method details OpenPipe Docs Official OpenPipe API documentation and features ​ Installation To use OpenPipeLLMService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[openpipe]" You’ll need to set up both API keys as environment variables: OPENPIPE_API_KEY - Your OpenPipe API key OPENAI_API_KEY - Your OpenAI API key Get your OpenPipe API key from OpenPipe Dashboard . ​ Frames ​ Input OpenAILLMContextFrame - Conversation context and history LLMMessagesFrame - Direct message list VisionImageRawFrame - Images for vision processing LLMUpdateSettingsFrame - Runtime parameter updates ​ Output LLMFullResponseStartFrame / LLMFullResponseEndFrame - Response boundaries LLMTextFrame - Streamed completion chunks FunctionCallInProgressFrame / FunctionCallResultFrame - Function call lifecycle ErrorFrame - API or processing errors ​ Function Calling Function Calling Guide Learn how to implement function calling with standardized schemas, register handlers, manage context properly, and control execution flow in your conversational AI applications. ​ Context Management Context Management Guide Learn how to manage conversation context, handle message history, and integrate context aggregators for consistent conversational experiences. ​ Usage Example Copy Ask AI import os from pipecat.services.openpipe.llm import OpenPipeLLMService from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext from pipecat.adapters.schemas.function_schema import FunctionSchema from pipecat.adapters.schemas.tools_schema import ToolsSchema # Configure OpenPipe service with comprehensive logging llm = OpenPipeLLMService( model = "gpt-4o" , api_key = os.getenv( "OPENAI_API_KEY" ), openpipe_api_key = os.getenv( "OPENPIPE_API_KEY" ), tags = { "environment" : "production" , "feature" : "conversational-ai" , "deployment" : "voice-assistant" , "version" : "v1.2" }, params = OpenPipeLLMService.InputParams( temperature = 0.7 , max_completion_tokens = 1000 ) ) # Define function for monitoring tool usage weather_function = FunctionSchema( name = "get_weather" , description = "Get current weather information" , properties = { "location" : { "type" : "string" , "description" : "City and state, e.g. San Francisco, CA" } }, required = [ "location" ] ) tools = ToolsSchema( standard_tools = [weather_function]) # Create context with system optimization context = OpenAILLMContext( messages = [ { "role" : "system" , "content" : """You are a helpful voice assistant. Keep responses concise and natural for speech synthesis. All conversations are logged for quality improvement.""" } ], tools = tools ) # Create context aggregators context_aggregator = llm.create_context_aggregator(context) # Register function with logging awareness async def get_weather ( params ): location = params.arguments[ "location" ] # Function calls are automatically logged by OpenPipe await params.result_callback( f "Weather in { location } : 72°F and sunny" ) llm.register_function( "get_weather" , get_weather) # Use in pipeline - all requests automatically logged pipeline = Pipeline([ transport.input(), stt, context_aggregator.user(), llm, # Automatic logging happens here tts, transport.output(), context_aggregator.assistant() ]) ​ Metrics Inherits all OpenAI metrics plus OpenPipe-specific logging: Time to First Byte (TTFB) - Response latency measurement Processing Duration - Total request processing time Token Usage - Detailed consumption tracking Enable with: Copy Ask AI task = PipelineTask( pipeline, params = PipelineParams( enable_metrics = True , enable_usage_metrics = True ) ) ​ Additional Notes OpenAI Compatibility : Full compatibility with OpenAI API features and parameters Privacy Aware : Configurable data retention and filtering policies Cost Optimization : Detailed analytics help optimize model usage and costs Fine-tuning Pipeline : Seamless transition from logging to custom model training OpenAI OpenRouter On this page Overview Installation Frames Input Output Function Calling Context Management Usage Example Metrics Additional Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/llm_openpipe_ce37cb2c.txt b/llm_openpipe_ce37cb2c.txt
new file mode 100644
index 0000000000000000000000000000000000000000..320e0c1106a4c509ba4fd55877edcf8905124fc3
--- /dev/null
+++ b/llm_openpipe_ce37cb2c.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/llm/openpipe#installation
+Title: OpenPipe - Pipecat
+==================================================
+
+OpenPipe - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation LLM OpenPipe Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Anthropic AWS Bedrock Azure Cerebras DeepSeek Fireworks AI Google Gemini Google Vertex AI Grok Groq NVIDIA NIM Ollama OpenAI OpenPipe OpenRouter Perplexity Qwen SambaNova Together AI Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview OpenPipeLLMService extends the BaseOpenAILLMService to provide integration with OpenPipe, enabling request logging, model fine-tuning, and performance monitoring. It maintains compatibility with OpenAI’s API while adding OpenPipe’s logging and optimization capabilities. API Reference Complete API documentation and method details OpenPipe Docs Official OpenPipe API documentation and features ​ Installation To use OpenPipeLLMService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[openpipe]" You’ll need to set up both API keys as environment variables: OPENPIPE_API_KEY - Your OpenPipe API key OPENAI_API_KEY - Your OpenAI API key Get your OpenPipe API key from OpenPipe Dashboard . ​ Frames ​ Input OpenAILLMContextFrame - Conversation context and history LLMMessagesFrame - Direct message list VisionImageRawFrame - Images for vision processing LLMUpdateSettingsFrame - Runtime parameter updates ​ Output LLMFullResponseStartFrame / LLMFullResponseEndFrame - Response boundaries LLMTextFrame - Streamed completion chunks FunctionCallInProgressFrame / FunctionCallResultFrame - Function call lifecycle ErrorFrame - API or processing errors ​ Function Calling Function Calling Guide Learn how to implement function calling with standardized schemas, register handlers, manage context properly, and control execution flow in your conversational AI applications. ​ Context Management Context Management Guide Learn how to manage conversation context, handle message history, and integrate context aggregators for consistent conversational experiences. ​ Usage Example Copy Ask AI import os from pipecat.services.openpipe.llm import OpenPipeLLMService from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext from pipecat.adapters.schemas.function_schema import FunctionSchema from pipecat.adapters.schemas.tools_schema import ToolsSchema # Configure OpenPipe service with comprehensive logging llm = OpenPipeLLMService( model = "gpt-4o" , api_key = os.getenv( "OPENAI_API_KEY" ), openpipe_api_key = os.getenv( "OPENPIPE_API_KEY" ), tags = { "environment" : "production" , "feature" : "conversational-ai" , "deployment" : "voice-assistant" , "version" : "v1.2" }, params = OpenPipeLLMService.InputParams( temperature = 0.7 , max_completion_tokens = 1000 ) ) # Define function for monitoring tool usage weather_function = FunctionSchema( name = "get_weather" , description = "Get current weather information" , properties = { "location" : { "type" : "string" , "description" : "City and state, e.g. San Francisco, CA" } }, required = [ "location" ] ) tools = ToolsSchema( standard_tools = [weather_function]) # Create context with system optimization context = OpenAILLMContext( messages = [ { "role" : "system" , "content" : """You are a helpful voice assistant. Keep responses concise and natural for speech synthesis. All conversations are logged for quality improvement.""" } ], tools = tools ) # Create context aggregators context_aggregator = llm.create_context_aggregator(context) # Register function with logging awareness async def get_weather ( params ): location = params.arguments[ "location" ] # Function calls are automatically logged by OpenPipe await params.result_callback( f "Weather in { location } : 72°F and sunny" ) llm.register_function( "get_weather" , get_weather) # Use in pipeline - all requests automatically logged pipeline = Pipeline([ transport.input(), stt, context_aggregator.user(), llm, # Automatic logging happens here tts, transport.output(), context_aggregator.assistant() ]) ​ Metrics Inherits all OpenAI metrics plus OpenPipe-specific logging: Time to First Byte (TTFB) - Response latency measurement Processing Duration - Total request processing time Token Usage - Detailed consumption tracking Enable with: Copy Ask AI task = PipelineTask( pipeline, params = PipelineParams( enable_metrics = True , enable_usage_metrics = True ) ) ​ Additional Notes OpenAI Compatibility : Full compatibility with OpenAI API features and parameters Privacy Aware : Configurable data retention and filtering policies Cost Optimization : Detailed analytics help optimize model usage and costs Fine-tuning Pipeline : Seamless transition from logging to custom model training OpenAI OpenRouter On this page Overview Installation Frames Input Output Function Calling Context Management Usage Example Metrics Additional Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/llm_openpipe_de11b5ce.txt b/llm_openpipe_de11b5ce.txt
new file mode 100644
index 0000000000000000000000000000000000000000..6c53fc977a61f5f580b783938a54b1b1066ae0ee
--- /dev/null
+++ b/llm_openpipe_de11b5ce.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/llm/openpipe#input
+Title: OpenPipe - Pipecat
+==================================================
+
+OpenPipe - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation LLM OpenPipe Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Anthropic AWS Bedrock Azure Cerebras DeepSeek Fireworks AI Google Gemini Google Vertex AI Grok Groq NVIDIA NIM Ollama OpenAI OpenPipe OpenRouter Perplexity Qwen SambaNova Together AI Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview OpenPipeLLMService extends the BaseOpenAILLMService to provide integration with OpenPipe, enabling request logging, model fine-tuning, and performance monitoring. It maintains compatibility with OpenAI’s API while adding OpenPipe’s logging and optimization capabilities. API Reference Complete API documentation and method details OpenPipe Docs Official OpenPipe API documentation and features ​ Installation To use OpenPipeLLMService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[openpipe]" You’ll need to set up both API keys as environment variables: OPENPIPE_API_KEY - Your OpenPipe API key OPENAI_API_KEY - Your OpenAI API key Get your OpenPipe API key from OpenPipe Dashboard . ​ Frames ​ Input OpenAILLMContextFrame - Conversation context and history LLMMessagesFrame - Direct message list VisionImageRawFrame - Images for vision processing LLMUpdateSettingsFrame - Runtime parameter updates ​ Output LLMFullResponseStartFrame / LLMFullResponseEndFrame - Response boundaries LLMTextFrame - Streamed completion chunks FunctionCallInProgressFrame / FunctionCallResultFrame - Function call lifecycle ErrorFrame - API or processing errors ​ Function Calling Function Calling Guide Learn how to implement function calling with standardized schemas, register handlers, manage context properly, and control execution flow in your conversational AI applications. ​ Context Management Context Management Guide Learn how to manage conversation context, handle message history, and integrate context aggregators for consistent conversational experiences. ​ Usage Example Copy Ask AI import os from pipecat.services.openpipe.llm import OpenPipeLLMService from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext from pipecat.adapters.schemas.function_schema import FunctionSchema from pipecat.adapters.schemas.tools_schema import ToolsSchema # Configure OpenPipe service with comprehensive logging llm = OpenPipeLLMService( model = "gpt-4o" , api_key = os.getenv( "OPENAI_API_KEY" ), openpipe_api_key = os.getenv( "OPENPIPE_API_KEY" ), tags = { "environment" : "production" , "feature" : "conversational-ai" , "deployment" : "voice-assistant" , "version" : "v1.2" }, params = OpenPipeLLMService.InputParams( temperature = 0.7 , max_completion_tokens = 1000 ) ) # Define function for monitoring tool usage weather_function = FunctionSchema( name = "get_weather" , description = "Get current weather information" , properties = { "location" : { "type" : "string" , "description" : "City and state, e.g. San Francisco, CA" } }, required = [ "location" ] ) tools = ToolsSchema( standard_tools = [weather_function]) # Create context with system optimization context = OpenAILLMContext( messages = [ { "role" : "system" , "content" : """You are a helpful voice assistant. Keep responses concise and natural for speech synthesis. All conversations are logged for quality improvement.""" } ], tools = tools ) # Create context aggregators context_aggregator = llm.create_context_aggregator(context) # Register function with logging awareness async def get_weather ( params ): location = params.arguments[ "location" ] # Function calls are automatically logged by OpenPipe await params.result_callback( f "Weather in { location } : 72°F and sunny" ) llm.register_function( "get_weather" , get_weather) # Use in pipeline - all requests automatically logged pipeline = Pipeline([ transport.input(), stt, context_aggregator.user(), llm, # Automatic logging happens here tts, transport.output(), context_aggregator.assistant() ]) ​ Metrics Inherits all OpenAI metrics plus OpenPipe-specific logging: Time to First Byte (TTFB) - Response latency measurement Processing Duration - Total request processing time Token Usage - Detailed consumption tracking Enable with: Copy Ask AI task = PipelineTask( pipeline, params = PipelineParams( enable_metrics = True , enable_usage_metrics = True ) ) ​ Additional Notes OpenAI Compatibility : Full compatibility with OpenAI API features and parameters Privacy Aware : Configurable data retention and filtering policies Cost Optimization : Detailed analytics help optimize model usage and costs Fine-tuning Pipeline : Seamless transition from logging to custom model training OpenAI OpenRouter On this page Overview Installation Frames Input Output Function Calling Context Management Usage Example Metrics Additional Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/llm_openpipe_e19b36ca.txt b/llm_openpipe_e19b36ca.txt
new file mode 100644
index 0000000000000000000000000000000000000000..79b950290485b7eb6a62393fdc3a99f0cba5465f
--- /dev/null
+++ b/llm_openpipe_e19b36ca.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/llm/openpipe#additional-notes
+Title: OpenPipe - Pipecat
+==================================================
+
+OpenPipe - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation LLM OpenPipe Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Anthropic AWS Bedrock Azure Cerebras DeepSeek Fireworks AI Google Gemini Google Vertex AI Grok Groq NVIDIA NIM Ollama OpenAI OpenPipe OpenRouter Perplexity Qwen SambaNova Together AI Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview OpenPipeLLMService extends the BaseOpenAILLMService to provide integration with OpenPipe, enabling request logging, model fine-tuning, and performance monitoring. It maintains compatibility with OpenAI’s API while adding OpenPipe’s logging and optimization capabilities. API Reference Complete API documentation and method details OpenPipe Docs Official OpenPipe API documentation and features ​ Installation To use OpenPipeLLMService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[openpipe]" You’ll need to set up both API keys as environment variables: OPENPIPE_API_KEY - Your OpenPipe API key OPENAI_API_KEY - Your OpenAI API key Get your OpenPipe API key from OpenPipe Dashboard . ​ Frames ​ Input OpenAILLMContextFrame - Conversation context and history LLMMessagesFrame - Direct message list VisionImageRawFrame - Images for vision processing LLMUpdateSettingsFrame - Runtime parameter updates ​ Output LLMFullResponseStartFrame / LLMFullResponseEndFrame - Response boundaries LLMTextFrame - Streamed completion chunks FunctionCallInProgressFrame / FunctionCallResultFrame - Function call lifecycle ErrorFrame - API or processing errors ​ Function Calling Function Calling Guide Learn how to implement function calling with standardized schemas, register handlers, manage context properly, and control execution flow in your conversational AI applications. ​ Context Management Context Management Guide Learn how to manage conversation context, handle message history, and integrate context aggregators for consistent conversational experiences. ​ Usage Example Copy Ask AI import os from pipecat.services.openpipe.llm import OpenPipeLLMService from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext from pipecat.adapters.schemas.function_schema import FunctionSchema from pipecat.adapters.schemas.tools_schema import ToolsSchema # Configure OpenPipe service with comprehensive logging llm = OpenPipeLLMService( model = "gpt-4o" , api_key = os.getenv( "OPENAI_API_KEY" ), openpipe_api_key = os.getenv( "OPENPIPE_API_KEY" ), tags = { "environment" : "production" , "feature" : "conversational-ai" , "deployment" : "voice-assistant" , "version" : "v1.2" }, params = OpenPipeLLMService.InputParams( temperature = 0.7 , max_completion_tokens = 1000 ) ) # Define function for monitoring tool usage weather_function = FunctionSchema( name = "get_weather" , description = "Get current weather information" , properties = { "location" : { "type" : "string" , "description" : "City and state, e.g. San Francisco, CA" } }, required = [ "location" ] ) tools = ToolsSchema( standard_tools = [weather_function]) # Create context with system optimization context = OpenAILLMContext( messages = [ { "role" : "system" , "content" : """You are a helpful voice assistant. Keep responses concise and natural for speech synthesis. All conversations are logged for quality improvement.""" } ], tools = tools ) # Create context aggregators context_aggregator = llm.create_context_aggregator(context) # Register function with logging awareness async def get_weather ( params ): location = params.arguments[ "location" ] # Function calls are automatically logged by OpenPipe await params.result_callback( f "Weather in { location } : 72°F and sunny" ) llm.register_function( "get_weather" , get_weather) # Use in pipeline - all requests automatically logged pipeline = Pipeline([ transport.input(), stt, context_aggregator.user(), llm, # Automatic logging happens here tts, transport.output(), context_aggregator.assistant() ]) ​ Metrics Inherits all OpenAI metrics plus OpenPipe-specific logging: Time to First Byte (TTFB) - Response latency measurement Processing Duration - Total request processing time Token Usage - Detailed consumption tracking Enable with: Copy Ask AI task = PipelineTask( pipeline, params = PipelineParams( enable_metrics = True , enable_usage_metrics = True ) ) ​ Additional Notes OpenAI Compatibility : Full compatibility with OpenAI API features and parameters Privacy Aware : Configurable data retention and filtering policies Cost Optimization : Detailed analytics help optimize model usage and costs Fine-tuning Pipeline : Seamless transition from logging to custom model training OpenAI OpenRouter On this page Overview Installation Frames Input Output Function Calling Context Management Usage Example Metrics Additional Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/llm_openpipe_f6b50edf.txt b/llm_openpipe_f6b50edf.txt
new file mode 100644
index 0000000000000000000000000000000000000000..347a5de3ad072bb5cee4fd5103e4816333dc81d6
--- /dev/null
+++ b/llm_openpipe_f6b50edf.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/llm/openpipe#context-management
+Title: OpenPipe - Pipecat
+==================================================
+
+OpenPipe - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation LLM OpenPipe Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Anthropic AWS Bedrock Azure Cerebras DeepSeek Fireworks AI Google Gemini Google Vertex AI Grok Groq NVIDIA NIM Ollama OpenAI OpenPipe OpenRouter Perplexity Qwen SambaNova Together AI Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview OpenPipeLLMService extends the BaseOpenAILLMService to provide integration with OpenPipe, enabling request logging, model fine-tuning, and performance monitoring. It maintains compatibility with OpenAI’s API while adding OpenPipe’s logging and optimization capabilities. API Reference Complete API documentation and method details OpenPipe Docs Official OpenPipe API documentation and features ​ Installation To use OpenPipeLLMService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[openpipe]" You’ll need to set up both API keys as environment variables: OPENPIPE_API_KEY - Your OpenPipe API key OPENAI_API_KEY - Your OpenAI API key Get your OpenPipe API key from OpenPipe Dashboard . ​ Frames ​ Input OpenAILLMContextFrame - Conversation context and history LLMMessagesFrame - Direct message list VisionImageRawFrame - Images for vision processing LLMUpdateSettingsFrame - Runtime parameter updates ​ Output LLMFullResponseStartFrame / LLMFullResponseEndFrame - Response boundaries LLMTextFrame - Streamed completion chunks FunctionCallInProgressFrame / FunctionCallResultFrame - Function call lifecycle ErrorFrame - API or processing errors ​ Function Calling Function Calling Guide Learn how to implement function calling with standardized schemas, register handlers, manage context properly, and control execution flow in your conversational AI applications. ​ Context Management Context Management Guide Learn how to manage conversation context, handle message history, and integrate context aggregators for consistent conversational experiences. ​ Usage Example Copy Ask AI import os from pipecat.services.openpipe.llm import OpenPipeLLMService from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext from pipecat.adapters.schemas.function_schema import FunctionSchema from pipecat.adapters.schemas.tools_schema import ToolsSchema # Configure OpenPipe service with comprehensive logging llm = OpenPipeLLMService( model = "gpt-4o" , api_key = os.getenv( "OPENAI_API_KEY" ), openpipe_api_key = os.getenv( "OPENPIPE_API_KEY" ), tags = { "environment" : "production" , "feature" : "conversational-ai" , "deployment" : "voice-assistant" , "version" : "v1.2" }, params = OpenPipeLLMService.InputParams( temperature = 0.7 , max_completion_tokens = 1000 ) ) # Define function for monitoring tool usage weather_function = FunctionSchema( name = "get_weather" , description = "Get current weather information" , properties = { "location" : { "type" : "string" , "description" : "City and state, e.g. San Francisco, CA" } }, required = [ "location" ] ) tools = ToolsSchema( standard_tools = [weather_function]) # Create context with system optimization context = OpenAILLMContext( messages = [ { "role" : "system" , "content" : """You are a helpful voice assistant. Keep responses concise and natural for speech synthesis. All conversations are logged for quality improvement.""" } ], tools = tools ) # Create context aggregators context_aggregator = llm.create_context_aggregator(context) # Register function with logging awareness async def get_weather ( params ): location = params.arguments[ "location" ] # Function calls are automatically logged by OpenPipe await params.result_callback( f "Weather in { location } : 72°F and sunny" ) llm.register_function( "get_weather" , get_weather) # Use in pipeline - all requests automatically logged pipeline = Pipeline([ transport.input(), stt, context_aggregator.user(), llm, # Automatic logging happens here tts, transport.output(), context_aggregator.assistant() ]) ​ Metrics Inherits all OpenAI metrics plus OpenPipe-specific logging: Time to First Byte (TTFB) - Response latency measurement Processing Duration - Total request processing time Token Usage - Detailed consumption tracking Enable with: Copy Ask AI task = PipelineTask( pipeline, params = PipelineParams( enable_metrics = True , enable_usage_metrics = True ) ) ​ Additional Notes OpenAI Compatibility : Full compatibility with OpenAI API features and parameters Privacy Aware : Configurable data retention and filtering policies Cost Optimization : Detailed analytics help optimize model usage and costs Fine-tuning Pipeline : Seamless transition from logging to custom model training OpenAI OpenRouter On this page Overview Installation Frames Input Output Function Calling Context Management Usage Example Metrics Additional Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/llm_openrouter_254e8278.txt b/llm_openrouter_254e8278.txt
new file mode 100644
index 0000000000000000000000000000000000000000..105948697c6a7329d7e564a9e32cf8b544b50a6f
--- /dev/null
+++ b/llm_openrouter_254e8278.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/llm/openrouter#function-calling
+Title: OpenRouter - Pipecat
+==================================================
+
+OpenRouter - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation LLM OpenRouter Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Anthropic AWS Bedrock Azure Cerebras DeepSeek Fireworks AI Google Gemini Google Vertex AI Grok Groq NVIDIA NIM Ollama OpenAI OpenPipe OpenRouter Perplexity Qwen SambaNova Together AI Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview OpenRouterLLMService provides access to OpenRouter’s language models through an OpenAI-compatible interface. It inherits from OpenAILLMService and supports streaming responses, function calling, and context management. API Reference Complete API documentation and method details OpenRouter Docs Official OpenRouter API documentation and features Example Code Working example with function calling ​ Installation To use OpenRouterLLMService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[openrouter]" You’ll also need to set up your OpenRouter API key as an environment variable: OPENROUTER_API_KEY . Get your API key from OpenRouter . Free tier includes $1 of credits. ​ Frames ​ Input OpenAILLMContextFrame - Conversation context and history LLMMessagesFrame - Direct message list VisionImageRawFrame - Images for vision processing (select models) LLMUpdateSettingsFrame - Runtime parameter updates ​ Output LLMFullResponseStartFrame / LLMFullResponseEndFrame - Response boundaries LLMTextFrame - Streamed completion chunks FunctionCallInProgressFrame / FunctionCallResultFrame - Function call lifecycle ErrorFrame - API or processing errors ​ Function Calling Function Calling Guide Learn how to implement function calling with standardized schemas, register handlers, manage context properly, and control execution flow in your conversational AI applications. ​ Context Management Context Management Guide Learn how to manage conversation context, handle message history, and integrate context aggregators for consistent conversational experiences. ​ Usage Example Copy Ask AI import os from pipecat.services.openrouter.llm import OpenRouterLLMService from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext from pipecat.adapters.schemas.function_schema import FunctionSchema from pipecat.adapters.schemas.tools_schema import ToolsSchema # Configure OpenRouter service llm = OpenRouterLLMService( api_key = os.getenv( "OPENROUTER_API_KEY" ), model = "openai/gpt-4o-2024-11-20" , # Easy model switching params = OpenRouterLLMService.InputParams( temperature = 0.7 , max_tokens = 1000 ) ) # Define function for tool calling weather_function = FunctionSchema( name = "get_current_weather" , description = "Get current weather information" , properties = { "location" : { "type" : "string" , "description" : "City and state, e.g. San Francisco, CA" }, "format" : { "type" : "string" , "enum" : [ "celsius" , "fahrenheit" ], "description" : "Temperature unit to use" } }, required = [ "location" , "format" ] ) tools = ToolsSchema( standard_tools = [weather_function]) # Create context context = OpenAILLMContext( messages = [ { "role" : "system" , "content" : """You are a helpful assistant optimized for voice conversations. Keep responses concise and avoid special characters for better speech synthesis.""" } ], tools = tools ) # Create context aggregators context_aggregator = llm.create_context_aggregator(context) # Register function handler with feedback async def fetch_weather ( params ): location = params.arguments[ "location" ] await params.result_callback({ "conditions" : "sunny" , "temperature" : "75°F" }) llm.register_function( "get_current_weather" , fetch_weather) # Optional: Add function call feedback @llm.event_handler ( "on_function_calls_started" ) async def on_function_calls_started ( service , function_calls ): await tts.queue_frame(TTSSpeakFrame( "Let me check on that." )) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, context_aggregator.user(), llm, tts, transport.output(), context_aggregator.assistant() ]) # Easy model switching for different use cases # llm.set_model_name("anthropic/claude-3.5-sonnet") # Switch to Claude # llm.set_model_name("meta-llama/llama-3.1-70b-instruct") # Switch to Llama ​ Metrics Inherits all OpenAI metrics capabilities: Time to First Byte (TTFB) - Response latency measurement Processing Duration - Total request processing time Token Usage - Prompt tokens, completion tokens, and totals Enable with: Copy Ask AI task = PipelineTask( pipeline, params = PipelineParams( enable_metrics = True , enable_usage_metrics = True ) ) ​ Additional Notes Model Variety : Access 70+ models from OpenAI, Anthropic, Meta, Google, and more OpenAI Compatibility : Full compatibility with existing OpenAI code Easy Switching : Change models with a single parameter update Fallback Support : Built-in model fallbacks for high availability OpenPipe Perplexity On this page Overview Installation Frames Input Output Function Calling Context Management Usage Example Metrics Additional Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/llm_openrouter_89889cb1.txt b/llm_openrouter_89889cb1.txt
new file mode 100644
index 0000000000000000000000000000000000000000..e2d39aad485507742cb833698136085626ab8b4f
--- /dev/null
+++ b/llm_openrouter_89889cb1.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/llm/openrouter#overview
+Title: OpenRouter - Pipecat
+==================================================
+
+OpenRouter - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation LLM OpenRouter Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Anthropic AWS Bedrock Azure Cerebras DeepSeek Fireworks AI Google Gemini Google Vertex AI Grok Groq NVIDIA NIM Ollama OpenAI OpenPipe OpenRouter Perplexity Qwen SambaNova Together AI Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview OpenRouterLLMService provides access to OpenRouter’s language models through an OpenAI-compatible interface. It inherits from OpenAILLMService and supports streaming responses, function calling, and context management. API Reference Complete API documentation and method details OpenRouter Docs Official OpenRouter API documentation and features Example Code Working example with function calling ​ Installation To use OpenRouterLLMService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[openrouter]" You’ll also need to set up your OpenRouter API key as an environment variable: OPENROUTER_API_KEY . Get your API key from OpenRouter . Free tier includes $1 of credits. ​ Frames ​ Input OpenAILLMContextFrame - Conversation context and history LLMMessagesFrame - Direct message list VisionImageRawFrame - Images for vision processing (select models) LLMUpdateSettingsFrame - Runtime parameter updates ​ Output LLMFullResponseStartFrame / LLMFullResponseEndFrame - Response boundaries LLMTextFrame - Streamed completion chunks FunctionCallInProgressFrame / FunctionCallResultFrame - Function call lifecycle ErrorFrame - API or processing errors ​ Function Calling Function Calling Guide Learn how to implement function calling with standardized schemas, register handlers, manage context properly, and control execution flow in your conversational AI applications. ​ Context Management Context Management Guide Learn how to manage conversation context, handle message history, and integrate context aggregators for consistent conversational experiences. ​ Usage Example Copy Ask AI import os from pipecat.services.openrouter.llm import OpenRouterLLMService from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext from pipecat.adapters.schemas.function_schema import FunctionSchema from pipecat.adapters.schemas.tools_schema import ToolsSchema # Configure OpenRouter service llm = OpenRouterLLMService( api_key = os.getenv( "OPENROUTER_API_KEY" ), model = "openai/gpt-4o-2024-11-20" , # Easy model switching params = OpenRouterLLMService.InputParams( temperature = 0.7 , max_tokens = 1000 ) ) # Define function for tool calling weather_function = FunctionSchema( name = "get_current_weather" , description = "Get current weather information" , properties = { "location" : { "type" : "string" , "description" : "City and state, e.g. San Francisco, CA" }, "format" : { "type" : "string" , "enum" : [ "celsius" , "fahrenheit" ], "description" : "Temperature unit to use" } }, required = [ "location" , "format" ] ) tools = ToolsSchema( standard_tools = [weather_function]) # Create context context = OpenAILLMContext( messages = [ { "role" : "system" , "content" : """You are a helpful assistant optimized for voice conversations. Keep responses concise and avoid special characters for better speech synthesis.""" } ], tools = tools ) # Create context aggregators context_aggregator = llm.create_context_aggregator(context) # Register function handler with feedback async def fetch_weather ( params ): location = params.arguments[ "location" ] await params.result_callback({ "conditions" : "sunny" , "temperature" : "75°F" }) llm.register_function( "get_current_weather" , fetch_weather) # Optional: Add function call feedback @llm.event_handler ( "on_function_calls_started" ) async def on_function_calls_started ( service , function_calls ): await tts.queue_frame(TTSSpeakFrame( "Let me check on that." )) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, context_aggregator.user(), llm, tts, transport.output(), context_aggregator.assistant() ]) # Easy model switching for different use cases # llm.set_model_name("anthropic/claude-3.5-sonnet") # Switch to Claude # llm.set_model_name("meta-llama/llama-3.1-70b-instruct") # Switch to Llama ​ Metrics Inherits all OpenAI metrics capabilities: Time to First Byte (TTFB) - Response latency measurement Processing Duration - Total request processing time Token Usage - Prompt tokens, completion tokens, and totals Enable with: Copy Ask AI task = PipelineTask( pipeline, params = PipelineParams( enable_metrics = True , enable_usage_metrics = True ) ) ​ Additional Notes Model Variety : Access 70+ models from OpenAI, Anthropic, Meta, Google, and more OpenAI Compatibility : Full compatibility with existing OpenAI code Easy Switching : Change models with a single parameter update Fallback Support : Built-in model fallbacks for high availability OpenPipe Perplexity On this page Overview Installation Frames Input Output Function Calling Context Management Usage Example Metrics Additional Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/llm_openrouter_ddbd932b.txt b/llm_openrouter_ddbd932b.txt
new file mode 100644
index 0000000000000000000000000000000000000000..e059672feae74a0e17dff4e4ff79208c56d8362a
--- /dev/null
+++ b/llm_openrouter_ddbd932b.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/llm/openrouter#usage-example
+Title: OpenRouter - Pipecat
+==================================================
+
+OpenRouter - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation LLM OpenRouter Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Anthropic AWS Bedrock Azure Cerebras DeepSeek Fireworks AI Google Gemini Google Vertex AI Grok Groq NVIDIA NIM Ollama OpenAI OpenPipe OpenRouter Perplexity Qwen SambaNova Together AI Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview OpenRouterLLMService provides access to OpenRouter’s language models through an OpenAI-compatible interface. It inherits from OpenAILLMService and supports streaming responses, function calling, and context management. API Reference Complete API documentation and method details OpenRouter Docs Official OpenRouter API documentation and features Example Code Working example with function calling ​ Installation To use OpenRouterLLMService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[openrouter]" You’ll also need to set up your OpenRouter API key as an environment variable: OPENROUTER_API_KEY . Get your API key from OpenRouter . Free tier includes $1 of credits. ​ Frames ​ Input OpenAILLMContextFrame - Conversation context and history LLMMessagesFrame - Direct message list VisionImageRawFrame - Images for vision processing (select models) LLMUpdateSettingsFrame - Runtime parameter updates ​ Output LLMFullResponseStartFrame / LLMFullResponseEndFrame - Response boundaries LLMTextFrame - Streamed completion chunks FunctionCallInProgressFrame / FunctionCallResultFrame - Function call lifecycle ErrorFrame - API or processing errors ​ Function Calling Function Calling Guide Learn how to implement function calling with standardized schemas, register handlers, manage context properly, and control execution flow in your conversational AI applications. ​ Context Management Context Management Guide Learn how to manage conversation context, handle message history, and integrate context aggregators for consistent conversational experiences. ​ Usage Example Copy Ask AI import os from pipecat.services.openrouter.llm import OpenRouterLLMService from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext from pipecat.adapters.schemas.function_schema import FunctionSchema from pipecat.adapters.schemas.tools_schema import ToolsSchema # Configure OpenRouter service llm = OpenRouterLLMService( api_key = os.getenv( "OPENROUTER_API_KEY" ), model = "openai/gpt-4o-2024-11-20" , # Easy model switching params = OpenRouterLLMService.InputParams( temperature = 0.7 , max_tokens = 1000 ) ) # Define function for tool calling weather_function = FunctionSchema( name = "get_current_weather" , description = "Get current weather information" , properties = { "location" : { "type" : "string" , "description" : "City and state, e.g. San Francisco, CA" }, "format" : { "type" : "string" , "enum" : [ "celsius" , "fahrenheit" ], "description" : "Temperature unit to use" } }, required = [ "location" , "format" ] ) tools = ToolsSchema( standard_tools = [weather_function]) # Create context context = OpenAILLMContext( messages = [ { "role" : "system" , "content" : """You are a helpful assistant optimized for voice conversations. Keep responses concise and avoid special characters for better speech synthesis.""" } ], tools = tools ) # Create context aggregators context_aggregator = llm.create_context_aggregator(context) # Register function handler with feedback async def fetch_weather ( params ): location = params.arguments[ "location" ] await params.result_callback({ "conditions" : "sunny" , "temperature" : "75°F" }) llm.register_function( "get_current_weather" , fetch_weather) # Optional: Add function call feedback @llm.event_handler ( "on_function_calls_started" ) async def on_function_calls_started ( service , function_calls ): await tts.queue_frame(TTSSpeakFrame( "Let me check on that." )) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, context_aggregator.user(), llm, tts, transport.output(), context_aggregator.assistant() ]) # Easy model switching for different use cases # llm.set_model_name("anthropic/claude-3.5-sonnet") # Switch to Claude # llm.set_model_name("meta-llama/llama-3.1-70b-instruct") # Switch to Llama ​ Metrics Inherits all OpenAI metrics capabilities: Time to First Byte (TTFB) - Response latency measurement Processing Duration - Total request processing time Token Usage - Prompt tokens, completion tokens, and totals Enable with: Copy Ask AI task = PipelineTask( pipeline, params = PipelineParams( enable_metrics = True , enable_usage_metrics = True ) ) ​ Additional Notes Model Variety : Access 70+ models from OpenAI, Anthropic, Meta, Google, and more OpenAI Compatibility : Full compatibility with existing OpenAI code Easy Switching : Change models with a single parameter update Fallback Support : Built-in model fallbacks for high availability OpenPipe Perplexity On this page Overview Installation Frames Input Output Function Calling Context Management Usage Example Metrics Additional Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/llm_openrouter_e5cd4522.txt b/llm_openrouter_e5cd4522.txt
new file mode 100644
index 0000000000000000000000000000000000000000..5bf1e8eb9451228c6480955378cf40d5361a27b6
--- /dev/null
+++ b/llm_openrouter_e5cd4522.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/llm/openrouter#installation
+Title: OpenRouter - Pipecat
+==================================================
+
+OpenRouter - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation LLM OpenRouter Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Anthropic AWS Bedrock Azure Cerebras DeepSeek Fireworks AI Google Gemini Google Vertex AI Grok Groq NVIDIA NIM Ollama OpenAI OpenPipe OpenRouter Perplexity Qwen SambaNova Together AI Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview OpenRouterLLMService provides access to OpenRouter’s language models through an OpenAI-compatible interface. It inherits from OpenAILLMService and supports streaming responses, function calling, and context management. API Reference Complete API documentation and method details OpenRouter Docs Official OpenRouter API documentation and features Example Code Working example with function calling ​ Installation To use OpenRouterLLMService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[openrouter]" You’ll also need to set up your OpenRouter API key as an environment variable: OPENROUTER_API_KEY . Get your API key from OpenRouter . Free tier includes $1 of credits. ​ Frames ​ Input OpenAILLMContextFrame - Conversation context and history LLMMessagesFrame - Direct message list VisionImageRawFrame - Images for vision processing (select models) LLMUpdateSettingsFrame - Runtime parameter updates ​ Output LLMFullResponseStartFrame / LLMFullResponseEndFrame - Response boundaries LLMTextFrame - Streamed completion chunks FunctionCallInProgressFrame / FunctionCallResultFrame - Function call lifecycle ErrorFrame - API or processing errors ​ Function Calling Function Calling Guide Learn how to implement function calling with standardized schemas, register handlers, manage context properly, and control execution flow in your conversational AI applications. ​ Context Management Context Management Guide Learn how to manage conversation context, handle message history, and integrate context aggregators for consistent conversational experiences. ​ Usage Example Copy Ask AI import os from pipecat.services.openrouter.llm import OpenRouterLLMService from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext from pipecat.adapters.schemas.function_schema import FunctionSchema from pipecat.adapters.schemas.tools_schema import ToolsSchema # Configure OpenRouter service llm = OpenRouterLLMService( api_key = os.getenv( "OPENROUTER_API_KEY" ), model = "openai/gpt-4o-2024-11-20" , # Easy model switching params = OpenRouterLLMService.InputParams( temperature = 0.7 , max_tokens = 1000 ) ) # Define function for tool calling weather_function = FunctionSchema( name = "get_current_weather" , description = "Get current weather information" , properties = { "location" : { "type" : "string" , "description" : "City and state, e.g. San Francisco, CA" }, "format" : { "type" : "string" , "enum" : [ "celsius" , "fahrenheit" ], "description" : "Temperature unit to use" } }, required = [ "location" , "format" ] ) tools = ToolsSchema( standard_tools = [weather_function]) # Create context context = OpenAILLMContext( messages = [ { "role" : "system" , "content" : """You are a helpful assistant optimized for voice conversations. Keep responses concise and avoid special characters for better speech synthesis.""" } ], tools = tools ) # Create context aggregators context_aggregator = llm.create_context_aggregator(context) # Register function handler with feedback async def fetch_weather ( params ): location = params.arguments[ "location" ] await params.result_callback({ "conditions" : "sunny" , "temperature" : "75°F" }) llm.register_function( "get_current_weather" , fetch_weather) # Optional: Add function call feedback @llm.event_handler ( "on_function_calls_started" ) async def on_function_calls_started ( service , function_calls ): await tts.queue_frame(TTSSpeakFrame( "Let me check on that." )) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, context_aggregator.user(), llm, tts, transport.output(), context_aggregator.assistant() ]) # Easy model switching for different use cases # llm.set_model_name("anthropic/claude-3.5-sonnet") # Switch to Claude # llm.set_model_name("meta-llama/llama-3.1-70b-instruct") # Switch to Llama ​ Metrics Inherits all OpenAI metrics capabilities: Time to First Byte (TTFB) - Response latency measurement Processing Duration - Total request processing time Token Usage - Prompt tokens, completion tokens, and totals Enable with: Copy Ask AI task = PipelineTask( pipeline, params = PipelineParams( enable_metrics = True , enable_usage_metrics = True ) ) ​ Additional Notes Model Variety : Access 70+ models from OpenAI, Anthropic, Meta, Google, and more OpenAI Compatibility : Full compatibility with existing OpenAI code Easy Switching : Change models with a single parameter update Fallback Support : Built-in model fallbacks for high availability OpenPipe Perplexity On this page Overview Installation Frames Input Output Function Calling Context Management Usage Example Metrics Additional Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/llm_perplexity_2794af2a.txt b/llm_perplexity_2794af2a.txt
new file mode 100644
index 0000000000000000000000000000000000000000..c7758fa261a6c5349d7d4efef1260d40da777f65
--- /dev/null
+++ b/llm_perplexity_2794af2a.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/llm/perplexity#frames
+Title: Perplexity - Pipecat
+==================================================
+
+Perplexity - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation LLM Perplexity Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Anthropic AWS Bedrock Azure Cerebras DeepSeek Fireworks AI Google Gemini Google Vertex AI Grok Groq NVIDIA NIM Ollama OpenAI OpenPipe OpenRouter Perplexity Qwen SambaNova Together AI Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview PerplexityLLMService provides access to Perplexity’s language models through an OpenAI-compatible interface. It inherits from OpenAILLMService and supports streaming responses and context management, with special handling for Perplexity’s incremental token reporting. API Reference Complete API documentation and method details Perplexity Docs Official Perplexity API documentation and features Example Code Working example with search capabilities Unlike other LLM services, Perplexity does not support function calling. Instead, they offer native internet search built in without requiring special function calls. ​ Installation To use PerplexityLLMService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[perplexity]" You’ll also need to set up your Perplexity API key as an environment variable: PERPLEXITY_API_KEY . Get your API key from Perplexity API . ​ Frames ​ Input OpenAILLMContextFrame - Conversation context and history LLMMessagesFrame - Direct message list LLMUpdateSettingsFrame - Runtime parameter updates ​ Output LLMFullResponseStartFrame / LLMFullResponseEndFrame - Response boundaries LLMTextFrame - Streamed completion chunks with citations ErrorFrame - API or processing errors ​ Context Management Context Management Guide Learn how to manage conversation context, handle message history, and integrate context aggregators for consistent conversational experiences. ​ Usage Example Copy Ask AI import os from pipecat.services.perplexity.llm import PerplexityLLMService from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext # Configure Perplexity service llm = PerplexityLLMService( api_key = os.getenv( "PERPLEXITY_API_KEY" ), model = "sonar-pro" , # Pro model for enhanced capabilities params = PerplexityLLMService.InputParams( temperature = 0.7 , max_tokens = 1000 ) ) # Create context optimized for search and current information context = OpenAILLMContext( messages = [ { "role" : "system" , "content" : """You are a knowledgeable assistant with access to real-time information. When answering questions, use your search capabilities to provide current, accurate information. Always cite your sources when possible. Keep responses concise for voice output.""" } ] ) # Create context aggregators context_aggregator = llm.create_context_aggregator(context) # Use in pipeline for information-rich conversations pipeline = Pipeline([ transport.input(), stt, context_aggregator.user(), llm, # Will automatically search and cite sources tts, transport.output(), context_aggregator.assistant() ]) # Enable metrics with special TTFB reporting for Perplexity task = PipelineTask( pipeline, params = PipelineParams( enable_metrics = True , enable_usage_metrics = True , report_only_initial_ttfb = True , # Optimized for Perplexity's response pattern ) ) ​ Metrics The service provides specialized token tracking for Perplexity’s incremental reporting: Time to First Byte (TTFB) - Response latency measurement Processing Duration - Total request processing time Token Usage - Accumulated prompt and completion tokens Enable with: Copy Ask AI task = PipelineTask( pipeline, params = PipelineParams( enable_metrics = True , enable_usage_metrics = True , ) ) ​ Additional Notes No Function Calling : Perplexity doesn’t support traditional function calling but provides superior built-in search Real-time Data : Access to current information without complex function orchestration Source Citations : Automatic citation of web sources in responses OpenAI Compatible : Uses familiar OpenAI-style interface and parameters OpenRouter Qwen On this page Overview Installation Frames Input Output Context Management Usage Example Metrics Additional Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/llm_perplexity_3e2c950b.txt b/llm_perplexity_3e2c950b.txt
new file mode 100644
index 0000000000000000000000000000000000000000..32265c17cc320a2dc44a47b35f947169c27e7d99
--- /dev/null
+++ b/llm_perplexity_3e2c950b.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/llm/perplexity#input
+Title: Perplexity - Pipecat
+==================================================
+
+Perplexity - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation LLM Perplexity Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Anthropic AWS Bedrock Azure Cerebras DeepSeek Fireworks AI Google Gemini Google Vertex AI Grok Groq NVIDIA NIM Ollama OpenAI OpenPipe OpenRouter Perplexity Qwen SambaNova Together AI Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview PerplexityLLMService provides access to Perplexity’s language models through an OpenAI-compatible interface. It inherits from OpenAILLMService and supports streaming responses and context management, with special handling for Perplexity’s incremental token reporting. API Reference Complete API documentation and method details Perplexity Docs Official Perplexity API documentation and features Example Code Working example with search capabilities Unlike other LLM services, Perplexity does not support function calling. Instead, they offer native internet search built in without requiring special function calls. ​ Installation To use PerplexityLLMService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[perplexity]" You’ll also need to set up your Perplexity API key as an environment variable: PERPLEXITY_API_KEY . Get your API key from Perplexity API . ​ Frames ​ Input OpenAILLMContextFrame - Conversation context and history LLMMessagesFrame - Direct message list LLMUpdateSettingsFrame - Runtime parameter updates ​ Output LLMFullResponseStartFrame / LLMFullResponseEndFrame - Response boundaries LLMTextFrame - Streamed completion chunks with citations ErrorFrame - API or processing errors ​ Context Management Context Management Guide Learn how to manage conversation context, handle message history, and integrate context aggregators for consistent conversational experiences. ​ Usage Example Copy Ask AI import os from pipecat.services.perplexity.llm import PerplexityLLMService from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext # Configure Perplexity service llm = PerplexityLLMService( api_key = os.getenv( "PERPLEXITY_API_KEY" ), model = "sonar-pro" , # Pro model for enhanced capabilities params = PerplexityLLMService.InputParams( temperature = 0.7 , max_tokens = 1000 ) ) # Create context optimized for search and current information context = OpenAILLMContext( messages = [ { "role" : "system" , "content" : """You are a knowledgeable assistant with access to real-time information. When answering questions, use your search capabilities to provide current, accurate information. Always cite your sources when possible. Keep responses concise for voice output.""" } ] ) # Create context aggregators context_aggregator = llm.create_context_aggregator(context) # Use in pipeline for information-rich conversations pipeline = Pipeline([ transport.input(), stt, context_aggregator.user(), llm, # Will automatically search and cite sources tts, transport.output(), context_aggregator.assistant() ]) # Enable metrics with special TTFB reporting for Perplexity task = PipelineTask( pipeline, params = PipelineParams( enable_metrics = True , enable_usage_metrics = True , report_only_initial_ttfb = True , # Optimized for Perplexity's response pattern ) ) ​ Metrics The service provides specialized token tracking for Perplexity’s incremental reporting: Time to First Byte (TTFB) - Response latency measurement Processing Duration - Total request processing time Token Usage - Accumulated prompt and completion tokens Enable with: Copy Ask AI task = PipelineTask( pipeline, params = PipelineParams( enable_metrics = True , enable_usage_metrics = True , ) ) ​ Additional Notes No Function Calling : Perplexity doesn’t support traditional function calling but provides superior built-in search Real-time Data : Access to current information without complex function orchestration Source Citations : Automatic citation of web sources in responses OpenAI Compatible : Uses familiar OpenAI-style interface and parameters OpenRouter Qwen On this page Overview Installation Frames Input Output Context Management Usage Example Metrics Additional Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/llm_perplexity_425feda1.txt b/llm_perplexity_425feda1.txt
new file mode 100644
index 0000000000000000000000000000000000000000..b7e206f54a87d4ee18c5341a8ab1d37dba36c998
--- /dev/null
+++ b/llm_perplexity_425feda1.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/llm/perplexity#installation
+Title: Perplexity - Pipecat
+==================================================
+
+Perplexity - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation LLM Perplexity Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Anthropic AWS Bedrock Azure Cerebras DeepSeek Fireworks AI Google Gemini Google Vertex AI Grok Groq NVIDIA NIM Ollama OpenAI OpenPipe OpenRouter Perplexity Qwen SambaNova Together AI Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview PerplexityLLMService provides access to Perplexity’s language models through an OpenAI-compatible interface. It inherits from OpenAILLMService and supports streaming responses and context management, with special handling for Perplexity’s incremental token reporting. API Reference Complete API documentation and method details Perplexity Docs Official Perplexity API documentation and features Example Code Working example with search capabilities Unlike other LLM services, Perplexity does not support function calling. Instead, they offer native internet search built in without requiring special function calls. ​ Installation To use PerplexityLLMService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[perplexity]" You’ll also need to set up your Perplexity API key as an environment variable: PERPLEXITY_API_KEY . Get your API key from Perplexity API . ​ Frames ​ Input OpenAILLMContextFrame - Conversation context and history LLMMessagesFrame - Direct message list LLMUpdateSettingsFrame - Runtime parameter updates ​ Output LLMFullResponseStartFrame / LLMFullResponseEndFrame - Response boundaries LLMTextFrame - Streamed completion chunks with citations ErrorFrame - API or processing errors ​ Context Management Context Management Guide Learn how to manage conversation context, handle message history, and integrate context aggregators for consistent conversational experiences. ​ Usage Example Copy Ask AI import os from pipecat.services.perplexity.llm import PerplexityLLMService from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext # Configure Perplexity service llm = PerplexityLLMService( api_key = os.getenv( "PERPLEXITY_API_KEY" ), model = "sonar-pro" , # Pro model for enhanced capabilities params = PerplexityLLMService.InputParams( temperature = 0.7 , max_tokens = 1000 ) ) # Create context optimized for search and current information context = OpenAILLMContext( messages = [ { "role" : "system" , "content" : """You are a knowledgeable assistant with access to real-time information. When answering questions, use your search capabilities to provide current, accurate information. Always cite your sources when possible. Keep responses concise for voice output.""" } ] ) # Create context aggregators context_aggregator = llm.create_context_aggregator(context) # Use in pipeline for information-rich conversations pipeline = Pipeline([ transport.input(), stt, context_aggregator.user(), llm, # Will automatically search and cite sources tts, transport.output(), context_aggregator.assistant() ]) # Enable metrics with special TTFB reporting for Perplexity task = PipelineTask( pipeline, params = PipelineParams( enable_metrics = True , enable_usage_metrics = True , report_only_initial_ttfb = True , # Optimized for Perplexity's response pattern ) ) ​ Metrics The service provides specialized token tracking for Perplexity’s incremental reporting: Time to First Byte (TTFB) - Response latency measurement Processing Duration - Total request processing time Token Usage - Accumulated prompt and completion tokens Enable with: Copy Ask AI task = PipelineTask( pipeline, params = PipelineParams( enable_metrics = True , enable_usage_metrics = True , ) ) ​ Additional Notes No Function Calling : Perplexity doesn’t support traditional function calling but provides superior built-in search Real-time Data : Access to current information without complex function orchestration Source Citations : Automatic citation of web sources in responses OpenAI Compatible : Uses familiar OpenAI-style interface and parameters OpenRouter Qwen On this page Overview Installation Frames Input Output Context Management Usage Example Metrics Additional Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/llm_perplexity_9702684b.txt b/llm_perplexity_9702684b.txt
new file mode 100644
index 0000000000000000000000000000000000000000..73ad5101e778e986d7c4c9523ba00c9ee0d6c9f7
--- /dev/null
+++ b/llm_perplexity_9702684b.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/llm/perplexity#metrics
+Title: Perplexity - Pipecat
+==================================================
+
+Perplexity - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation LLM Perplexity Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Anthropic AWS Bedrock Azure Cerebras DeepSeek Fireworks AI Google Gemini Google Vertex AI Grok Groq NVIDIA NIM Ollama OpenAI OpenPipe OpenRouter Perplexity Qwen SambaNova Together AI Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview PerplexityLLMService provides access to Perplexity’s language models through an OpenAI-compatible interface. It inherits from OpenAILLMService and supports streaming responses and context management, with special handling for Perplexity’s incremental token reporting. API Reference Complete API documentation and method details Perplexity Docs Official Perplexity API documentation and features Example Code Working example with search capabilities Unlike other LLM services, Perplexity does not support function calling. Instead, they offer native internet search built in without requiring special function calls. ​ Installation To use PerplexityLLMService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[perplexity]" You’ll also need to set up your Perplexity API key as an environment variable: PERPLEXITY_API_KEY . Get your API key from Perplexity API . ​ Frames ​ Input OpenAILLMContextFrame - Conversation context and history LLMMessagesFrame - Direct message list LLMUpdateSettingsFrame - Runtime parameter updates ​ Output LLMFullResponseStartFrame / LLMFullResponseEndFrame - Response boundaries LLMTextFrame - Streamed completion chunks with citations ErrorFrame - API or processing errors ​ Context Management Context Management Guide Learn how to manage conversation context, handle message history, and integrate context aggregators for consistent conversational experiences. ​ Usage Example Copy Ask AI import os from pipecat.services.perplexity.llm import PerplexityLLMService from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext # Configure Perplexity service llm = PerplexityLLMService( api_key = os.getenv( "PERPLEXITY_API_KEY" ), model = "sonar-pro" , # Pro model for enhanced capabilities params = PerplexityLLMService.InputParams( temperature = 0.7 , max_tokens = 1000 ) ) # Create context optimized for search and current information context = OpenAILLMContext( messages = [ { "role" : "system" , "content" : """You are a knowledgeable assistant with access to real-time information. When answering questions, use your search capabilities to provide current, accurate information. Always cite your sources when possible. Keep responses concise for voice output.""" } ] ) # Create context aggregators context_aggregator = llm.create_context_aggregator(context) # Use in pipeline for information-rich conversations pipeline = Pipeline([ transport.input(), stt, context_aggregator.user(), llm, # Will automatically search and cite sources tts, transport.output(), context_aggregator.assistant() ]) # Enable metrics with special TTFB reporting for Perplexity task = PipelineTask( pipeline, params = PipelineParams( enable_metrics = True , enable_usage_metrics = True , report_only_initial_ttfb = True , # Optimized for Perplexity's response pattern ) ) ​ Metrics The service provides specialized token tracking for Perplexity’s incremental reporting: Time to First Byte (TTFB) - Response latency measurement Processing Duration - Total request processing time Token Usage - Accumulated prompt and completion tokens Enable with: Copy Ask AI task = PipelineTask( pipeline, params = PipelineParams( enable_metrics = True , enable_usage_metrics = True , ) ) ​ Additional Notes No Function Calling : Perplexity doesn’t support traditional function calling but provides superior built-in search Real-time Data : Access to current information without complex function orchestration Source Citations : Automatic citation of web sources in responses OpenAI Compatible : Uses familiar OpenAI-style interface and parameters OpenRouter Qwen On this page Overview Installation Frames Input Output Context Management Usage Example Metrics Additional Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/llm_qwen_083591aa.txt b/llm_qwen_083591aa.txt
new file mode 100644
index 0000000000000000000000000000000000000000..51010f3edf35bfa320f8dd7693e5c44457f8d8e9
--- /dev/null
+++ b/llm_qwen_083591aa.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/llm/qwen#frames
+Title: Qwen - Pipecat
+==================================================
+
+Qwen - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation LLM Qwen Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Anthropic AWS Bedrock Azure Cerebras DeepSeek Fireworks AI Google Gemini Google Vertex AI Grok Groq NVIDIA NIM Ollama OpenAI OpenPipe OpenRouter Perplexity Qwen SambaNova Together AI Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview QwenLLMService provides access to Alibaba Cloud’s Qwen language models through an OpenAI-compatible interface. It inherits from OpenAILLMService and supports streaming responses, function calling, and context management, with particularly strong capabilities for Chinese language processing. API Reference Complete API documentation and method details Qwen Docs Official Qwen API documentation and features Example Code Working example with function calling ​ Installation To use QwenLLMService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[qwen]" You’ll also need to set up your DashScope API key as an environment variable: QWEN_API_KEY . Get your API key from Alibaba Cloud Model Studio . ​ Frames ​ Input OpenAILLMContextFrame - Conversation context and history LLMMessagesFrame - Direct message list VisionImageRawFrame - Images for vision processing LLMUpdateSettingsFrame - Runtime parameter updates ​ Output LLMFullResponseStartFrame / LLMFullResponseEndFrame - Response boundaries LLMTextFrame - Streamed completion chunks FunctionCallInProgressFrame / FunctionCallResultFrame - Function call lifecycle ErrorFrame - API or processing errors ​ Function Calling Function Calling Guide Learn how to implement function calling with standardized schemas, register handlers, manage context properly, and control execution flow in your conversational AI applications. ​ Context Management Context Management Guide Learn how to manage conversation context, handle message history, and integrate context aggregators for consistent conversational experiences. ​ Usage Example Copy Ask AI import os from pipecat.services.qwen.llm import QwenLLMService from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext from pipecat.adapters.schemas.function_schema import FunctionSchema from pipecat.adapters.schemas.tools_schema import ToolsSchema # Configure Qwen service llm = QwenLLMService( api_key = os.getenv( "QWEN_API_KEY" ), model = "qwen2.5-72b-instruct" , # High-quality open source model params = QwenLLMService.InputParams( temperature = 0.7 , max_tokens = 1000 ) ) # Define function for tool calling weather_function = FunctionSchema( name = "get_current_weather" , description = "Get current weather information" , properties = { "location" : { "type" : "string" , "description" : "City and country, e.g. Beijing, China or San Francisco, USA" }, "format" : { "type" : "string" , "enum" : [ "celsius" , "fahrenheit" ], "description" : "Temperature unit to use" } }, required = [ "location" , "format" ] ) tools = ToolsSchema( standard_tools = [weather_function]) # Create bilingual context for Chinese/English support context = OpenAILLMContext( messages = [ { "role" : "system" , "content" : """You are a helpful assistant in voice conversations. Keep responses concise for speech output. You can respond in Chinese when the user speaks Chinese, or English when they speak English. 你是一个语音对话助手。请保持简洁的回答以适合语音输出。 当用户用中文交流时用中文回答，用英文交流时用英文回答。""" } ], tools = tools ) # Create context aggregators context_aggregator = llm.create_context_aggregator(context) # Register function handler async def fetch_weather ( params ): location = params.arguments[ "location" ] # Return response that works well in both languages await params.result_callback({ "conditions" : "sunny" , "temperature" : "22°C" , "location" : location }) llm.register_function( "get_current_weather" , fetch_weather) # Optional: Add function call feedback @llm.event_handler ( "on_function_calls_started" ) async def on_function_calls_started ( service , function_calls ): await tts.queue_frame(TTSSpeakFrame( "Let me check on that." )) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, context_aggregator.user(), llm, tts, # Consider QwenTTSService for Chinese speech transport.output(), context_aggregator.assistant() ]) ​ Metrics Inherits all OpenAI metrics capabilities: Time to First Byte (TTFB) - Response latency measurement Processing Duration - Total request processing time Token Usage - Prompt tokens, completion tokens, and totals Enable with: Copy Ask AI task = PipelineTask( pipeline, params = PipelineParams( enable_metrics = True , enable_usage_metrics = True ) ) ​ Additional Notes OpenAI Compatibility : Full compatibility with OpenAI API features and parameters Long Context Support : Models support up to 1M token contexts for extensive conversations Multilingual Excellence : Superior performance in Chinese with strong English capabilities Code-Switching : Seamlessly handles mixed Chinese-English conversations Alibaba Cloud Integration : Native integration with Alibaba Cloud ecosystem Perplexity SambaNova On this page Overview Installation Frames Input Output Function Calling Context Management Usage Example Metrics Additional Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/llm_qwen_99ef9e24.txt b/llm_qwen_99ef9e24.txt
new file mode 100644
index 0000000000000000000000000000000000000000..7b474251403add7c8054c4cf98008fe13f9f996c
--- /dev/null
+++ b/llm_qwen_99ef9e24.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/llm/qwen#function-calling
+Title: Qwen - Pipecat
+==================================================
+
+Qwen - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation LLM Qwen Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Anthropic AWS Bedrock Azure Cerebras DeepSeek Fireworks AI Google Gemini Google Vertex AI Grok Groq NVIDIA NIM Ollama OpenAI OpenPipe OpenRouter Perplexity Qwen SambaNova Together AI Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview QwenLLMService provides access to Alibaba Cloud’s Qwen language models through an OpenAI-compatible interface. It inherits from OpenAILLMService and supports streaming responses, function calling, and context management, with particularly strong capabilities for Chinese language processing. API Reference Complete API documentation and method details Qwen Docs Official Qwen API documentation and features Example Code Working example with function calling ​ Installation To use QwenLLMService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[qwen]" You’ll also need to set up your DashScope API key as an environment variable: QWEN_API_KEY . Get your API key from Alibaba Cloud Model Studio . ​ Frames ​ Input OpenAILLMContextFrame - Conversation context and history LLMMessagesFrame - Direct message list VisionImageRawFrame - Images for vision processing LLMUpdateSettingsFrame - Runtime parameter updates ​ Output LLMFullResponseStartFrame / LLMFullResponseEndFrame - Response boundaries LLMTextFrame - Streamed completion chunks FunctionCallInProgressFrame / FunctionCallResultFrame - Function call lifecycle ErrorFrame - API or processing errors ​ Function Calling Function Calling Guide Learn how to implement function calling with standardized schemas, register handlers, manage context properly, and control execution flow in your conversational AI applications. ​ Context Management Context Management Guide Learn how to manage conversation context, handle message history, and integrate context aggregators for consistent conversational experiences. ​ Usage Example Copy Ask AI import os from pipecat.services.qwen.llm import QwenLLMService from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext from pipecat.adapters.schemas.function_schema import FunctionSchema from pipecat.adapters.schemas.tools_schema import ToolsSchema # Configure Qwen service llm = QwenLLMService( api_key = os.getenv( "QWEN_API_KEY" ), model = "qwen2.5-72b-instruct" , # High-quality open source model params = QwenLLMService.InputParams( temperature = 0.7 , max_tokens = 1000 ) ) # Define function for tool calling weather_function = FunctionSchema( name = "get_current_weather" , description = "Get current weather information" , properties = { "location" : { "type" : "string" , "description" : "City and country, e.g. Beijing, China or San Francisco, USA" }, "format" : { "type" : "string" , "enum" : [ "celsius" , "fahrenheit" ], "description" : "Temperature unit to use" } }, required = [ "location" , "format" ] ) tools = ToolsSchema( standard_tools = [weather_function]) # Create bilingual context for Chinese/English support context = OpenAILLMContext( messages = [ { "role" : "system" , "content" : """You are a helpful assistant in voice conversations. Keep responses concise for speech output. You can respond in Chinese when the user speaks Chinese, or English when they speak English. 你是一个语音对话助手。请保持简洁的回答以适合语音输出。 当用户用中文交流时用中文回答，用英文交流时用英文回答。""" } ], tools = tools ) # Create context aggregators context_aggregator = llm.create_context_aggregator(context) # Register function handler async def fetch_weather ( params ): location = params.arguments[ "location" ] # Return response that works well in both languages await params.result_callback({ "conditions" : "sunny" , "temperature" : "22°C" , "location" : location }) llm.register_function( "get_current_weather" , fetch_weather) # Optional: Add function call feedback @llm.event_handler ( "on_function_calls_started" ) async def on_function_calls_started ( service , function_calls ): await tts.queue_frame(TTSSpeakFrame( "Let me check on that." )) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, context_aggregator.user(), llm, tts, # Consider QwenTTSService for Chinese speech transport.output(), context_aggregator.assistant() ]) ​ Metrics Inherits all OpenAI metrics capabilities: Time to First Byte (TTFB) - Response latency measurement Processing Duration - Total request processing time Token Usage - Prompt tokens, completion tokens, and totals Enable with: Copy Ask AI task = PipelineTask( pipeline, params = PipelineParams( enable_metrics = True , enable_usage_metrics = True ) ) ​ Additional Notes OpenAI Compatibility : Full compatibility with OpenAI API features and parameters Long Context Support : Models support up to 1M token contexts for extensive conversations Multilingual Excellence : Superior performance in Chinese with strong English capabilities Code-Switching : Seamlessly handles mixed Chinese-English conversations Alibaba Cloud Integration : Native integration with Alibaba Cloud ecosystem Perplexity SambaNova On this page Overview Installation Frames Input Output Function Calling Context Management Usage Example Metrics Additional Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/llm_qwen_9e96ac50.txt b/llm_qwen_9e96ac50.txt
new file mode 100644
index 0000000000000000000000000000000000000000..dc6c32ec644dff02cd7c7a728f5400db61f1e8af
--- /dev/null
+++ b/llm_qwen_9e96ac50.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/llm/qwen#input
+Title: Qwen - Pipecat
+==================================================
+
+Qwen - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation LLM Qwen Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Anthropic AWS Bedrock Azure Cerebras DeepSeek Fireworks AI Google Gemini Google Vertex AI Grok Groq NVIDIA NIM Ollama OpenAI OpenPipe OpenRouter Perplexity Qwen SambaNova Together AI Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview QwenLLMService provides access to Alibaba Cloud’s Qwen language models through an OpenAI-compatible interface. It inherits from OpenAILLMService and supports streaming responses, function calling, and context management, with particularly strong capabilities for Chinese language processing. API Reference Complete API documentation and method details Qwen Docs Official Qwen API documentation and features Example Code Working example with function calling ​ Installation To use QwenLLMService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[qwen]" You’ll also need to set up your DashScope API key as an environment variable: QWEN_API_KEY . Get your API key from Alibaba Cloud Model Studio . ​ Frames ​ Input OpenAILLMContextFrame - Conversation context and history LLMMessagesFrame - Direct message list VisionImageRawFrame - Images for vision processing LLMUpdateSettingsFrame - Runtime parameter updates ​ Output LLMFullResponseStartFrame / LLMFullResponseEndFrame - Response boundaries LLMTextFrame - Streamed completion chunks FunctionCallInProgressFrame / FunctionCallResultFrame - Function call lifecycle ErrorFrame - API or processing errors ​ Function Calling Function Calling Guide Learn how to implement function calling with standardized schemas, register handlers, manage context properly, and control execution flow in your conversational AI applications. ​ Context Management Context Management Guide Learn how to manage conversation context, handle message history, and integrate context aggregators for consistent conversational experiences. ​ Usage Example Copy Ask AI import os from pipecat.services.qwen.llm import QwenLLMService from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext from pipecat.adapters.schemas.function_schema import FunctionSchema from pipecat.adapters.schemas.tools_schema import ToolsSchema # Configure Qwen service llm = QwenLLMService( api_key = os.getenv( "QWEN_API_KEY" ), model = "qwen2.5-72b-instruct" , # High-quality open source model params = QwenLLMService.InputParams( temperature = 0.7 , max_tokens = 1000 ) ) # Define function for tool calling weather_function = FunctionSchema( name = "get_current_weather" , description = "Get current weather information" , properties = { "location" : { "type" : "string" , "description" : "City and country, e.g. Beijing, China or San Francisco, USA" }, "format" : { "type" : "string" , "enum" : [ "celsius" , "fahrenheit" ], "description" : "Temperature unit to use" } }, required = [ "location" , "format" ] ) tools = ToolsSchema( standard_tools = [weather_function]) # Create bilingual context for Chinese/English support context = OpenAILLMContext( messages = [ { "role" : "system" , "content" : """You are a helpful assistant in voice conversations. Keep responses concise for speech output. You can respond in Chinese when the user speaks Chinese, or English when they speak English. 你是一个语音对话助手。请保持简洁的回答以适合语音输出。 当用户用中文交流时用中文回答，用英文交流时用英文回答。""" } ], tools = tools ) # Create context aggregators context_aggregator = llm.create_context_aggregator(context) # Register function handler async def fetch_weather ( params ): location = params.arguments[ "location" ] # Return response that works well in both languages await params.result_callback({ "conditions" : "sunny" , "temperature" : "22°C" , "location" : location }) llm.register_function( "get_current_weather" , fetch_weather) # Optional: Add function call feedback @llm.event_handler ( "on_function_calls_started" ) async def on_function_calls_started ( service , function_calls ): await tts.queue_frame(TTSSpeakFrame( "Let me check on that." )) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, context_aggregator.user(), llm, tts, # Consider QwenTTSService for Chinese speech transport.output(), context_aggregator.assistant() ]) ​ Metrics Inherits all OpenAI metrics capabilities: Time to First Byte (TTFB) - Response latency measurement Processing Duration - Total request processing time Token Usage - Prompt tokens, completion tokens, and totals Enable with: Copy Ask AI task = PipelineTask( pipeline, params = PipelineParams( enable_metrics = True , enable_usage_metrics = True ) ) ​ Additional Notes OpenAI Compatibility : Full compatibility with OpenAI API features and parameters Long Context Support : Models support up to 1M token contexts for extensive conversations Multilingual Excellence : Superior performance in Chinese with strong English capabilities Code-Switching : Seamlessly handles mixed Chinese-English conversations Alibaba Cloud Integration : Native integration with Alibaba Cloud ecosystem Perplexity SambaNova On this page Overview Installation Frames Input Output Function Calling Context Management Usage Example Metrics Additional Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/llm_qwen_bfd520b3.txt b/llm_qwen_bfd520b3.txt
new file mode 100644
index 0000000000000000000000000000000000000000..6f038284c63658c0f2ac72b79e2bd96b50349b9b
--- /dev/null
+++ b/llm_qwen_bfd520b3.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/llm/qwen#metrics
+Title: Qwen - Pipecat
+==================================================
+
+Qwen - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation LLM Qwen Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Anthropic AWS Bedrock Azure Cerebras DeepSeek Fireworks AI Google Gemini Google Vertex AI Grok Groq NVIDIA NIM Ollama OpenAI OpenPipe OpenRouter Perplexity Qwen SambaNova Together AI Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview QwenLLMService provides access to Alibaba Cloud’s Qwen language models through an OpenAI-compatible interface. It inherits from OpenAILLMService and supports streaming responses, function calling, and context management, with particularly strong capabilities for Chinese language processing. API Reference Complete API documentation and method details Qwen Docs Official Qwen API documentation and features Example Code Working example with function calling ​ Installation To use QwenLLMService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[qwen]" You’ll also need to set up your DashScope API key as an environment variable: QWEN_API_KEY . Get your API key from Alibaba Cloud Model Studio . ​ Frames ​ Input OpenAILLMContextFrame - Conversation context and history LLMMessagesFrame - Direct message list VisionImageRawFrame - Images for vision processing LLMUpdateSettingsFrame - Runtime parameter updates ​ Output LLMFullResponseStartFrame / LLMFullResponseEndFrame - Response boundaries LLMTextFrame - Streamed completion chunks FunctionCallInProgressFrame / FunctionCallResultFrame - Function call lifecycle ErrorFrame - API or processing errors ​ Function Calling Function Calling Guide Learn how to implement function calling with standardized schemas, register handlers, manage context properly, and control execution flow in your conversational AI applications. ​ Context Management Context Management Guide Learn how to manage conversation context, handle message history, and integrate context aggregators for consistent conversational experiences. ​ Usage Example Copy Ask AI import os from pipecat.services.qwen.llm import QwenLLMService from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext from pipecat.adapters.schemas.function_schema import FunctionSchema from pipecat.adapters.schemas.tools_schema import ToolsSchema # Configure Qwen service llm = QwenLLMService( api_key = os.getenv( "QWEN_API_KEY" ), model = "qwen2.5-72b-instruct" , # High-quality open source model params = QwenLLMService.InputParams( temperature = 0.7 , max_tokens = 1000 ) ) # Define function for tool calling weather_function = FunctionSchema( name = "get_current_weather" , description = "Get current weather information" , properties = { "location" : { "type" : "string" , "description" : "City and country, e.g. Beijing, China or San Francisco, USA" }, "format" : { "type" : "string" , "enum" : [ "celsius" , "fahrenheit" ], "description" : "Temperature unit to use" } }, required = [ "location" , "format" ] ) tools = ToolsSchema( standard_tools = [weather_function]) # Create bilingual context for Chinese/English support context = OpenAILLMContext( messages = [ { "role" : "system" , "content" : """You are a helpful assistant in voice conversations. Keep responses concise for speech output. You can respond in Chinese when the user speaks Chinese, or English when they speak English. 你是一个语音对话助手。请保持简洁的回答以适合语音输出。 当用户用中文交流时用中文回答，用英文交流时用英文回答。""" } ], tools = tools ) # Create context aggregators context_aggregator = llm.create_context_aggregator(context) # Register function handler async def fetch_weather ( params ): location = params.arguments[ "location" ] # Return response that works well in both languages await params.result_callback({ "conditions" : "sunny" , "temperature" : "22°C" , "location" : location }) llm.register_function( "get_current_weather" , fetch_weather) # Optional: Add function call feedback @llm.event_handler ( "on_function_calls_started" ) async def on_function_calls_started ( service , function_calls ): await tts.queue_frame(TTSSpeakFrame( "Let me check on that." )) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, context_aggregator.user(), llm, tts, # Consider QwenTTSService for Chinese speech transport.output(), context_aggregator.assistant() ]) ​ Metrics Inherits all OpenAI metrics capabilities: Time to First Byte (TTFB) - Response latency measurement Processing Duration - Total request processing time Token Usage - Prompt tokens, completion tokens, and totals Enable with: Copy Ask AI task = PipelineTask( pipeline, params = PipelineParams( enable_metrics = True , enable_usage_metrics = True ) ) ​ Additional Notes OpenAI Compatibility : Full compatibility with OpenAI API features and parameters Long Context Support : Models support up to 1M token contexts for extensive conversations Multilingual Excellence : Superior performance in Chinese with strong English capabilities Code-Switching : Seamlessly handles mixed Chinese-English conversations Alibaba Cloud Integration : Native integration with Alibaba Cloud ecosystem Perplexity SambaNova On this page Overview Installation Frames Input Output Function Calling Context Management Usage Example Metrics Additional Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/llm_qwen_e90faf04.txt b/llm_qwen_e90faf04.txt
new file mode 100644
index 0000000000000000000000000000000000000000..1d254806b4d62356bb7135f79a23d513eb44462c
--- /dev/null
+++ b/llm_qwen_e90faf04.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/llm/qwen#context-management
+Title: Qwen - Pipecat
+==================================================
+
+Qwen - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation LLM Qwen Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Anthropic AWS Bedrock Azure Cerebras DeepSeek Fireworks AI Google Gemini Google Vertex AI Grok Groq NVIDIA NIM Ollama OpenAI OpenPipe OpenRouter Perplexity Qwen SambaNova Together AI Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview QwenLLMService provides access to Alibaba Cloud’s Qwen language models through an OpenAI-compatible interface. It inherits from OpenAILLMService and supports streaming responses, function calling, and context management, with particularly strong capabilities for Chinese language processing. API Reference Complete API documentation and method details Qwen Docs Official Qwen API documentation and features Example Code Working example with function calling ​ Installation To use QwenLLMService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[qwen]" You’ll also need to set up your DashScope API key as an environment variable: QWEN_API_KEY . Get your API key from Alibaba Cloud Model Studio . ​ Frames ​ Input OpenAILLMContextFrame - Conversation context and history LLMMessagesFrame - Direct message list VisionImageRawFrame - Images for vision processing LLMUpdateSettingsFrame - Runtime parameter updates ​ Output LLMFullResponseStartFrame / LLMFullResponseEndFrame - Response boundaries LLMTextFrame - Streamed completion chunks FunctionCallInProgressFrame / FunctionCallResultFrame - Function call lifecycle ErrorFrame - API or processing errors ​ Function Calling Function Calling Guide Learn how to implement function calling with standardized schemas, register handlers, manage context properly, and control execution flow in your conversational AI applications. ​ Context Management Context Management Guide Learn how to manage conversation context, handle message history, and integrate context aggregators for consistent conversational experiences. ​ Usage Example Copy Ask AI import os from pipecat.services.qwen.llm import QwenLLMService from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext from pipecat.adapters.schemas.function_schema import FunctionSchema from pipecat.adapters.schemas.tools_schema import ToolsSchema # Configure Qwen service llm = QwenLLMService( api_key = os.getenv( "QWEN_API_KEY" ), model = "qwen2.5-72b-instruct" , # High-quality open source model params = QwenLLMService.InputParams( temperature = 0.7 , max_tokens = 1000 ) ) # Define function for tool calling weather_function = FunctionSchema( name = "get_current_weather" , description = "Get current weather information" , properties = { "location" : { "type" : "string" , "description" : "City and country, e.g. Beijing, China or San Francisco, USA" }, "format" : { "type" : "string" , "enum" : [ "celsius" , "fahrenheit" ], "description" : "Temperature unit to use" } }, required = [ "location" , "format" ] ) tools = ToolsSchema( standard_tools = [weather_function]) # Create bilingual context for Chinese/English support context = OpenAILLMContext( messages = [ { "role" : "system" , "content" : """You are a helpful assistant in voice conversations. Keep responses concise for speech output. You can respond in Chinese when the user speaks Chinese, or English when they speak English. 你是一个语音对话助手。请保持简洁的回答以适合语音输出。 当用户用中文交流时用中文回答，用英文交流时用英文回答。""" } ], tools = tools ) # Create context aggregators context_aggregator = llm.create_context_aggregator(context) # Register function handler async def fetch_weather ( params ): location = params.arguments[ "location" ] # Return response that works well in both languages await params.result_callback({ "conditions" : "sunny" , "temperature" : "22°C" , "location" : location }) llm.register_function( "get_current_weather" , fetch_weather) # Optional: Add function call feedback @llm.event_handler ( "on_function_calls_started" ) async def on_function_calls_started ( service , function_calls ): await tts.queue_frame(TTSSpeakFrame( "Let me check on that." )) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, context_aggregator.user(), llm, tts, # Consider QwenTTSService for Chinese speech transport.output(), context_aggregator.assistant() ]) ​ Metrics Inherits all OpenAI metrics capabilities: Time to First Byte (TTFB) - Response latency measurement Processing Duration - Total request processing time Token Usage - Prompt tokens, completion tokens, and totals Enable with: Copy Ask AI task = PipelineTask( pipeline, params = PipelineParams( enable_metrics = True , enable_usage_metrics = True ) ) ​ Additional Notes OpenAI Compatibility : Full compatibility with OpenAI API features and parameters Long Context Support : Models support up to 1M token contexts for extensive conversations Multilingual Excellence : Superior performance in Chinese with strong English capabilities Code-Switching : Seamlessly handles mixed Chinese-English conversations Alibaba Cloud Integration : Native integration with Alibaba Cloud ecosystem Perplexity SambaNova On this page Overview Installation Frames Input Output Function Calling Context Management Usage Example Metrics Additional Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/llm_sambanova_12ddac0a.txt b/llm_sambanova_12ddac0a.txt
new file mode 100644
index 0000000000000000000000000000000000000000..487f0fe744eccf0bf1a9876f89e52399cf4ac25f
--- /dev/null
+++ b/llm_sambanova_12ddac0a.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/llm/sambanova#param-max-tokens
+Title: SambaNova - Pipecat
+==================================================
+
+SambaNova - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation LLM SambaNova Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Anthropic AWS Bedrock Azure Cerebras DeepSeek Fireworks AI Google Gemini Google Vertex AI Grok Groq NVIDIA NIM Ollama OpenAI OpenPipe OpenRouter Perplexity Qwen SambaNova Together AI Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview SambaNovaLLMService provides access to SambaNova’s language models through an OpenAI-compatible interface. It inherits from OpenAILLMService and supports streaming responses, function calling, and context management. ​ Installation To use SambaNovaLLMService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[sambanova]" You also need to set up your SambaNova API key as an environment variable: SAMBANOVA_API_KEY . Get your SambaNova API key here . ​ Configuration ​ Constructor Parameters ​ api_key str required Your SambaNova API key ​ model str default: "Llama-4-Maverick-17B-128E-Instruct" Model identifier ​ base_url str default: "https://api.sambanova.ai/v1" SambaNova API endpoint ​ Input Parameters Inherits OpenAI-compatible parameters: ​ max_tokens Optional[int] Maximum number of tokens to generate. Must be greater than or equal to 1. ​ temperature Optional[float] Controls randomness in the output. Range: [0.0, 1.0]. ​ top_p Optional[float] Controls diversity via nucleus sampling. Range: [0.0, 1.0] ​ Usage Example Copy Ask AI from pipecat.adapters.schemas.function_schema import FunctionSchema from pipecat.adapters.schemas.tools_schema import ToolsSchema from pipecat.services.sambanova.llm import SambaNovaLLMService from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext from openai.types.chat import ChatCompletionToolParam from pipecat.pipeline.pipeline import Pipeline from pipecat.pipeline.task import PipelineParams, PipelineTask from pipecat.services.llm_service import FunctionCallParams # Configure service llm = SambaNovaLLMService( api_key 'your-sambanova-api-key' , model = 'Llama-4-Maverick-17B-128E-Instruct' , params = SambaNovaLLMService.InputParams( temperature = 0.7 , max_tokens = 1024 ), ) # Define function to call async def fetch_weather ( params : FunctionCallParams) -> Any: """Mock function that fetches the weather forcast from an API.""" await params.result_callback({ 'conditions' : 'nice' , 'temperature' : '20 Degrees Celsius' }) # Register function handlers llm.register_function( 'get_current_weather' , fetch_weather) # Define weather function using standardized schema weather_function = FunctionSchema( name = 'get_current_weather' , description = 'Get the current weather' , properties = { 'location' : { 'type' : 'string' , 'description' : 'The city and state.' , }, 'format' : { 'type' : 'string' , 'enum' : [ 'celsius' , 'fahrenheit' ], 'description' : "The temperature unit to use. Infer this from the user's location." , }, }, required = [ 'location' , 'format' ], ) # Create tools schema tools = ToolsSchema( standard_tools = [weather_function]) # Define system message messages = [ { 'role' : 'system' , 'content' : 'You are a helpful LLM in a WebRTC call. ' 'Your goal is to demonstrate your capabilities of weather forecasting in a succinct way. ' 'Introduce yourself to the user and then wait for their question. ' 'Elaborate your response into a conversational answer in a creative and helpful way. ' 'Your output will be converted to audio so do not include special characters in your answer. ' 'Once the final answer has been provided, please stop, unless the user asks another question. ' , }, ] # Create context with system message and tools context = OpenAILLMContext(messages, tools) # Context aggregator context_aggregator = llm.create_context_aggregator(context) # Create context aggregator for message handling context_aggregator = llm.create_context_aggregator(context) # Set up pipeline pipeline = Pipeline( [ transport.input(), stt, context_aggregator.user(), llm, tts, transport.output(), context_aggregator.assistant(), ] ) # Create and configure task task = PipelineTask( pipeline, params = PipelineParams( allow_interruptions = True , enable_metrics = True , enable_usage_metrics = True , ), ) ​ Methods See the LLM base class methods for additional functionality. ​ Function Calling This service supports function calling (also known as tool calling) which allows the LLM to request information from external services and APIs. For example, you can enable your bot to: Check current weather conditions. Query databases. Access external APIs. Perform custom actions. Function Calling Guide Learn how to implement function calling with standardized schemas, register handlers, manage context properly, and control execution flow in your conversational AI applications. ​ Available Models Model Name Description DeepSeek-R1 deepseek-ai/DeepSeek-R1 DeepSeek-R1-Distill-Llama-70B deepseek-ai/DeepSeek-R1-Distill-Llama-70B DeepSeek-V3-0324 deepseek-ai/DeepSeek-V3-0324 Llama-4-Maverick-17B-128E-Instruct meta-llama/Llama-4-Maverick-17B-128E-Instruct Llama-4-Scout-17B-16E-Instruct meta-llama/Llama-4-Scout-17B-16E-Instruct Meta-Llama-3.3-70B-Instruct meta-llama/Llama-3.3-70B-Instruct Meta-Llama-3.2-3B-Instruct meta-llama/Llama-3.2-3B-Instruct Meta-Llama-3.2-1B-Instruct meta-llama/Llama-3.2-1B-Instruct Meta-Llama-3.1-405B-Instruct meta-llama/Llama-3.1-405B-Instruct Meta-Llama-3.1-8B-Instruct meta-llama/Llama-3.1-8B-Instruct Meta-Llama-Guard-3-8B meta-llama/Llama-Guard-3-8B QwQ-32B Qwen/QwQ-32B Qwen3-32B Qwen/Qwen3-32B Llama-3.3-Swallow-70B-Instruct-v0.4 Tokyotech-llm/Llama-3.3-Swallow-70B-Instruct-v0.4 See SambaNova’s docs for a complete list of supported models. ​ Frame Flow Inherits the OpenAI LLM Service frame flow: ​ Metrics Support The service collects standard LLM metrics: Token usage (prompt and completion). Processing duration. Time to First Byte (TTFB). Function call metrics. ​ Notes OpenAI-compatible interface. Supports streaming responses. Handles function calling. Manages conversation context. Includes token usage tracking. Thread-safe processing. Automatic error handling. Qwen Together AI On this page Overview Installation Configuration Constructor Parameters Input Parameters Usage Example Methods Function Calling Available Models Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/llm_sambanova_8ae2ced0.txt b/llm_sambanova_8ae2ced0.txt
new file mode 100644
index 0000000000000000000000000000000000000000..f2e36924299d77eba9e20cfbe8c552b81de4b7ff
--- /dev/null
+++ b/llm_sambanova_8ae2ced0.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/llm/sambanova#methods
+Title: SambaNova - Pipecat
+==================================================
+
+SambaNova - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation LLM SambaNova Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Anthropic AWS Bedrock Azure Cerebras DeepSeek Fireworks AI Google Gemini Google Vertex AI Grok Groq NVIDIA NIM Ollama OpenAI OpenPipe OpenRouter Perplexity Qwen SambaNova Together AI Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview SambaNovaLLMService provides access to SambaNova’s language models through an OpenAI-compatible interface. It inherits from OpenAILLMService and supports streaming responses, function calling, and context management. ​ Installation To use SambaNovaLLMService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[sambanova]" You also need to set up your SambaNova API key as an environment variable: SAMBANOVA_API_KEY . Get your SambaNova API key here . ​ Configuration ​ Constructor Parameters ​ api_key str required Your SambaNova API key ​ model str default: "Llama-4-Maverick-17B-128E-Instruct" Model identifier ​ base_url str default: "https://api.sambanova.ai/v1" SambaNova API endpoint ​ Input Parameters Inherits OpenAI-compatible parameters: ​ max_tokens Optional[int] Maximum number of tokens to generate. Must be greater than or equal to 1. ​ temperature Optional[float] Controls randomness in the output. Range: [0.0, 1.0]. ​ top_p Optional[float] Controls diversity via nucleus sampling. Range: [0.0, 1.0] ​ Usage Example Copy Ask AI from pipecat.adapters.schemas.function_schema import FunctionSchema from pipecat.adapters.schemas.tools_schema import ToolsSchema from pipecat.services.sambanova.llm import SambaNovaLLMService from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext from openai.types.chat import ChatCompletionToolParam from pipecat.pipeline.pipeline import Pipeline from pipecat.pipeline.task import PipelineParams, PipelineTask from pipecat.services.llm_service import FunctionCallParams # Configure service llm = SambaNovaLLMService( api_key 'your-sambanova-api-key' , model = 'Llama-4-Maverick-17B-128E-Instruct' , params = SambaNovaLLMService.InputParams( temperature = 0.7 , max_tokens = 1024 ), ) # Define function to call async def fetch_weather ( params : FunctionCallParams) -> Any: """Mock function that fetches the weather forcast from an API.""" await params.result_callback({ 'conditions' : 'nice' , 'temperature' : '20 Degrees Celsius' }) # Register function handlers llm.register_function( 'get_current_weather' , fetch_weather) # Define weather function using standardized schema weather_function = FunctionSchema( name = 'get_current_weather' , description = 'Get the current weather' , properties = { 'location' : { 'type' : 'string' , 'description' : 'The city and state.' , }, 'format' : { 'type' : 'string' , 'enum' : [ 'celsius' , 'fahrenheit' ], 'description' : "The temperature unit to use. Infer this from the user's location." , }, }, required = [ 'location' , 'format' ], ) # Create tools schema tools = ToolsSchema( standard_tools = [weather_function]) # Define system message messages = [ { 'role' : 'system' , 'content' : 'You are a helpful LLM in a WebRTC call. ' 'Your goal is to demonstrate your capabilities of weather forecasting in a succinct way. ' 'Introduce yourself to the user and then wait for their question. ' 'Elaborate your response into a conversational answer in a creative and helpful way. ' 'Your output will be converted to audio so do not include special characters in your answer. ' 'Once the final answer has been provided, please stop, unless the user asks another question. ' , }, ] # Create context with system message and tools context = OpenAILLMContext(messages, tools) # Context aggregator context_aggregator = llm.create_context_aggregator(context) # Create context aggregator for message handling context_aggregator = llm.create_context_aggregator(context) # Set up pipeline pipeline = Pipeline( [ transport.input(), stt, context_aggregator.user(), llm, tts, transport.output(), context_aggregator.assistant(), ] ) # Create and configure task task = PipelineTask( pipeline, params = PipelineParams( allow_interruptions = True , enable_metrics = True , enable_usage_metrics = True , ), ) ​ Methods See the LLM base class methods for additional functionality. ​ Function Calling This service supports function calling (also known as tool calling) which allows the LLM to request information from external services and APIs. For example, you can enable your bot to: Check current weather conditions. Query databases. Access external APIs. Perform custom actions. Function Calling Guide Learn how to implement function calling with standardized schemas, register handlers, manage context properly, and control execution flow in your conversational AI applications. ​ Available Models Model Name Description DeepSeek-R1 deepseek-ai/DeepSeek-R1 DeepSeek-R1-Distill-Llama-70B deepseek-ai/DeepSeek-R1-Distill-Llama-70B DeepSeek-V3-0324 deepseek-ai/DeepSeek-V3-0324 Llama-4-Maverick-17B-128E-Instruct meta-llama/Llama-4-Maverick-17B-128E-Instruct Llama-4-Scout-17B-16E-Instruct meta-llama/Llama-4-Scout-17B-16E-Instruct Meta-Llama-3.3-70B-Instruct meta-llama/Llama-3.3-70B-Instruct Meta-Llama-3.2-3B-Instruct meta-llama/Llama-3.2-3B-Instruct Meta-Llama-3.2-1B-Instruct meta-llama/Llama-3.2-1B-Instruct Meta-Llama-3.1-405B-Instruct meta-llama/Llama-3.1-405B-Instruct Meta-Llama-3.1-8B-Instruct meta-llama/Llama-3.1-8B-Instruct Meta-Llama-Guard-3-8B meta-llama/Llama-Guard-3-8B QwQ-32B Qwen/QwQ-32B Qwen3-32B Qwen/Qwen3-32B Llama-3.3-Swallow-70B-Instruct-v0.4 Tokyotech-llm/Llama-3.3-Swallow-70B-Instruct-v0.4 See SambaNova’s docs for a complete list of supported models. ​ Frame Flow Inherits the OpenAI LLM Service frame flow: ​ Metrics Support The service collects standard LLM metrics: Token usage (prompt and completion). Processing duration. Time to First Byte (TTFB). Function call metrics. ​ Notes OpenAI-compatible interface. Supports streaming responses. Handles function calling. Manages conversation context. Includes token usage tracking. Thread-safe processing. Automatic error handling. Qwen Together AI On this page Overview Installation Configuration Constructor Parameters Input Parameters Usage Example Methods Function Calling Available Models Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/llm_sambanova_9312bf78.txt b/llm_sambanova_9312bf78.txt
new file mode 100644
index 0000000000000000000000000000000000000000..d1c00736de658d4a87c35a0ce4f0930c361a2d1d
--- /dev/null
+++ b/llm_sambanova_9312bf78.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/llm/sambanova#installation
+Title: SambaNova - Pipecat
+==================================================
+
+SambaNova - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation LLM SambaNova Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Anthropic AWS Bedrock Azure Cerebras DeepSeek Fireworks AI Google Gemini Google Vertex AI Grok Groq NVIDIA NIM Ollama OpenAI OpenPipe OpenRouter Perplexity Qwen SambaNova Together AI Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview SambaNovaLLMService provides access to SambaNova’s language models through an OpenAI-compatible interface. It inherits from OpenAILLMService and supports streaming responses, function calling, and context management. ​ Installation To use SambaNovaLLMService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[sambanova]" You also need to set up your SambaNova API key as an environment variable: SAMBANOVA_API_KEY . Get your SambaNova API key here . ​ Configuration ​ Constructor Parameters ​ api_key str required Your SambaNova API key ​ model str default: "Llama-4-Maverick-17B-128E-Instruct" Model identifier ​ base_url str default: "https://api.sambanova.ai/v1" SambaNova API endpoint ​ Input Parameters Inherits OpenAI-compatible parameters: ​ max_tokens Optional[int] Maximum number of tokens to generate. Must be greater than or equal to 1. ​ temperature Optional[float] Controls randomness in the output. Range: [0.0, 1.0]. ​ top_p Optional[float] Controls diversity via nucleus sampling. Range: [0.0, 1.0] ​ Usage Example Copy Ask AI from pipecat.adapters.schemas.function_schema import FunctionSchema from pipecat.adapters.schemas.tools_schema import ToolsSchema from pipecat.services.sambanova.llm import SambaNovaLLMService from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext from openai.types.chat import ChatCompletionToolParam from pipecat.pipeline.pipeline import Pipeline from pipecat.pipeline.task import PipelineParams, PipelineTask from pipecat.services.llm_service import FunctionCallParams # Configure service llm = SambaNovaLLMService( api_key 'your-sambanova-api-key' , model = 'Llama-4-Maverick-17B-128E-Instruct' , params = SambaNovaLLMService.InputParams( temperature = 0.7 , max_tokens = 1024 ), ) # Define function to call async def fetch_weather ( params : FunctionCallParams) -> Any: """Mock function that fetches the weather forcast from an API.""" await params.result_callback({ 'conditions' : 'nice' , 'temperature' : '20 Degrees Celsius' }) # Register function handlers llm.register_function( 'get_current_weather' , fetch_weather) # Define weather function using standardized schema weather_function = FunctionSchema( name = 'get_current_weather' , description = 'Get the current weather' , properties = { 'location' : { 'type' : 'string' , 'description' : 'The city and state.' , }, 'format' : { 'type' : 'string' , 'enum' : [ 'celsius' , 'fahrenheit' ], 'description' : "The temperature unit to use. Infer this from the user's location." , }, }, required = [ 'location' , 'format' ], ) # Create tools schema tools = ToolsSchema( standard_tools = [weather_function]) # Define system message messages = [ { 'role' : 'system' , 'content' : 'You are a helpful LLM in a WebRTC call. ' 'Your goal is to demonstrate your capabilities of weather forecasting in a succinct way. ' 'Introduce yourself to the user and then wait for their question. ' 'Elaborate your response into a conversational answer in a creative and helpful way. ' 'Your output will be converted to audio so do not include special characters in your answer. ' 'Once the final answer has been provided, please stop, unless the user asks another question. ' , }, ] # Create context with system message and tools context = OpenAILLMContext(messages, tools) # Context aggregator context_aggregator = llm.create_context_aggregator(context) # Create context aggregator for message handling context_aggregator = llm.create_context_aggregator(context) # Set up pipeline pipeline = Pipeline( [ transport.input(), stt, context_aggregator.user(), llm, tts, transport.output(), context_aggregator.assistant(), ] ) # Create and configure task task = PipelineTask( pipeline, params = PipelineParams( allow_interruptions = True , enable_metrics = True , enable_usage_metrics = True , ), ) ​ Methods See the LLM base class methods for additional functionality. ​ Function Calling This service supports function calling (also known as tool calling) which allows the LLM to request information from external services and APIs. For example, you can enable your bot to: Check current weather conditions. Query databases. Access external APIs. Perform custom actions. Function Calling Guide Learn how to implement function calling with standardized schemas, register handlers, manage context properly, and control execution flow in your conversational AI applications. ​ Available Models Model Name Description DeepSeek-R1 deepseek-ai/DeepSeek-R1 DeepSeek-R1-Distill-Llama-70B deepseek-ai/DeepSeek-R1-Distill-Llama-70B DeepSeek-V3-0324 deepseek-ai/DeepSeek-V3-0324 Llama-4-Maverick-17B-128E-Instruct meta-llama/Llama-4-Maverick-17B-128E-Instruct Llama-4-Scout-17B-16E-Instruct meta-llama/Llama-4-Scout-17B-16E-Instruct Meta-Llama-3.3-70B-Instruct meta-llama/Llama-3.3-70B-Instruct Meta-Llama-3.2-3B-Instruct meta-llama/Llama-3.2-3B-Instruct Meta-Llama-3.2-1B-Instruct meta-llama/Llama-3.2-1B-Instruct Meta-Llama-3.1-405B-Instruct meta-llama/Llama-3.1-405B-Instruct Meta-Llama-3.1-8B-Instruct meta-llama/Llama-3.1-8B-Instruct Meta-Llama-Guard-3-8B meta-llama/Llama-Guard-3-8B QwQ-32B Qwen/QwQ-32B Qwen3-32B Qwen/Qwen3-32B Llama-3.3-Swallow-70B-Instruct-v0.4 Tokyotech-llm/Llama-3.3-Swallow-70B-Instruct-v0.4 See SambaNova’s docs for a complete list of supported models. ​ Frame Flow Inherits the OpenAI LLM Service frame flow: ​ Metrics Support The service collects standard LLM metrics: Token usage (prompt and completion). Processing duration. Time to First Byte (TTFB). Function call metrics. ​ Notes OpenAI-compatible interface. Supports streaming responses. Handles function calling. Manages conversation context. Includes token usage tracking. Thread-safe processing. Automatic error handling. Qwen Together AI On this page Overview Installation Configuration Constructor Parameters Input Parameters Usage Example Methods Function Calling Available Models Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/llm_sambanova_9636e3f2.txt b/llm_sambanova_9636e3f2.txt
new file mode 100644
index 0000000000000000000000000000000000000000..d7655c2a96719efd73f4b32a6f0285af66deb286
--- /dev/null
+++ b/llm_sambanova_9636e3f2.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/llm/sambanova#metrics-support
+Title: SambaNova - Pipecat
+==================================================
+
+SambaNova - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation LLM SambaNova Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Anthropic AWS Bedrock Azure Cerebras DeepSeek Fireworks AI Google Gemini Google Vertex AI Grok Groq NVIDIA NIM Ollama OpenAI OpenPipe OpenRouter Perplexity Qwen SambaNova Together AI Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview SambaNovaLLMService provides access to SambaNova’s language models through an OpenAI-compatible interface. It inherits from OpenAILLMService and supports streaming responses, function calling, and context management. ​ Installation To use SambaNovaLLMService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[sambanova]" You also need to set up your SambaNova API key as an environment variable: SAMBANOVA_API_KEY . Get your SambaNova API key here . ​ Configuration ​ Constructor Parameters ​ api_key str required Your SambaNova API key ​ model str default: "Llama-4-Maverick-17B-128E-Instruct" Model identifier ​ base_url str default: "https://api.sambanova.ai/v1" SambaNova API endpoint ​ Input Parameters Inherits OpenAI-compatible parameters: ​ max_tokens Optional[int] Maximum number of tokens to generate. Must be greater than or equal to 1. ​ temperature Optional[float] Controls randomness in the output. Range: [0.0, 1.0]. ​ top_p Optional[float] Controls diversity via nucleus sampling. Range: [0.0, 1.0] ​ Usage Example Copy Ask AI from pipecat.adapters.schemas.function_schema import FunctionSchema from pipecat.adapters.schemas.tools_schema import ToolsSchema from pipecat.services.sambanova.llm import SambaNovaLLMService from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext from openai.types.chat import ChatCompletionToolParam from pipecat.pipeline.pipeline import Pipeline from pipecat.pipeline.task import PipelineParams, PipelineTask from pipecat.services.llm_service import FunctionCallParams # Configure service llm = SambaNovaLLMService( api_key 'your-sambanova-api-key' , model = 'Llama-4-Maverick-17B-128E-Instruct' , params = SambaNovaLLMService.InputParams( temperature = 0.7 , max_tokens = 1024 ), ) # Define function to call async def fetch_weather ( params : FunctionCallParams) -> Any: """Mock function that fetches the weather forcast from an API.""" await params.result_callback({ 'conditions' : 'nice' , 'temperature' : '20 Degrees Celsius' }) # Register function handlers llm.register_function( 'get_current_weather' , fetch_weather) # Define weather function using standardized schema weather_function = FunctionSchema( name = 'get_current_weather' , description = 'Get the current weather' , properties = { 'location' : { 'type' : 'string' , 'description' : 'The city and state.' , }, 'format' : { 'type' : 'string' , 'enum' : [ 'celsius' , 'fahrenheit' ], 'description' : "The temperature unit to use. Infer this from the user's location." , }, }, required = [ 'location' , 'format' ], ) # Create tools schema tools = ToolsSchema( standard_tools = [weather_function]) # Define system message messages = [ { 'role' : 'system' , 'content' : 'You are a helpful LLM in a WebRTC call. ' 'Your goal is to demonstrate your capabilities of weather forecasting in a succinct way. ' 'Introduce yourself to the user and then wait for their question. ' 'Elaborate your response into a conversational answer in a creative and helpful way. ' 'Your output will be converted to audio so do not include special characters in your answer. ' 'Once the final answer has been provided, please stop, unless the user asks another question. ' , }, ] # Create context with system message and tools context = OpenAILLMContext(messages, tools) # Context aggregator context_aggregator = llm.create_context_aggregator(context) # Create context aggregator for message handling context_aggregator = llm.create_context_aggregator(context) # Set up pipeline pipeline = Pipeline( [ transport.input(), stt, context_aggregator.user(), llm, tts, transport.output(), context_aggregator.assistant(), ] ) # Create and configure task task = PipelineTask( pipeline, params = PipelineParams( allow_interruptions = True , enable_metrics = True , enable_usage_metrics = True , ), ) ​ Methods See the LLM base class methods for additional functionality. ​ Function Calling This service supports function calling (also known as tool calling) which allows the LLM to request information from external services and APIs. For example, you can enable your bot to: Check current weather conditions. Query databases. Access external APIs. Perform custom actions. Function Calling Guide Learn how to implement function calling with standardized schemas, register handlers, manage context properly, and control execution flow in your conversational AI applications. ​ Available Models Model Name Description DeepSeek-R1 deepseek-ai/DeepSeek-R1 DeepSeek-R1-Distill-Llama-70B deepseek-ai/DeepSeek-R1-Distill-Llama-70B DeepSeek-V3-0324 deepseek-ai/DeepSeek-V3-0324 Llama-4-Maverick-17B-128E-Instruct meta-llama/Llama-4-Maverick-17B-128E-Instruct Llama-4-Scout-17B-16E-Instruct meta-llama/Llama-4-Scout-17B-16E-Instruct Meta-Llama-3.3-70B-Instruct meta-llama/Llama-3.3-70B-Instruct Meta-Llama-3.2-3B-Instruct meta-llama/Llama-3.2-3B-Instruct Meta-Llama-3.2-1B-Instruct meta-llama/Llama-3.2-1B-Instruct Meta-Llama-3.1-405B-Instruct meta-llama/Llama-3.1-405B-Instruct Meta-Llama-3.1-8B-Instruct meta-llama/Llama-3.1-8B-Instruct Meta-Llama-Guard-3-8B meta-llama/Llama-Guard-3-8B QwQ-32B Qwen/QwQ-32B Qwen3-32B Qwen/Qwen3-32B Llama-3.3-Swallow-70B-Instruct-v0.4 Tokyotech-llm/Llama-3.3-Swallow-70B-Instruct-v0.4 See SambaNova’s docs for a complete list of supported models. ​ Frame Flow Inherits the OpenAI LLM Service frame flow: ​ Metrics Support The service collects standard LLM metrics: Token usage (prompt and completion). Processing duration. Time to First Byte (TTFB). Function call metrics. ​ Notes OpenAI-compatible interface. Supports streaming responses. Handles function calling. Manages conversation context. Includes token usage tracking. Thread-safe processing. Automatic error handling. Qwen Together AI On this page Overview Installation Configuration Constructor Parameters Input Parameters Usage Example Methods Function Calling Available Models Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/llm_sambanova_cda446a7.txt b/llm_sambanova_cda446a7.txt
new file mode 100644
index 0000000000000000000000000000000000000000..5c8c025e62b477eb8894cf82f23eac3c62aafb93
--- /dev/null
+++ b/llm_sambanova_cda446a7.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/llm/sambanova#available-models
+Title: SambaNova - Pipecat
+==================================================
+
+SambaNova - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation LLM SambaNova Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Anthropic AWS Bedrock Azure Cerebras DeepSeek Fireworks AI Google Gemini Google Vertex AI Grok Groq NVIDIA NIM Ollama OpenAI OpenPipe OpenRouter Perplexity Qwen SambaNova Together AI Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview SambaNovaLLMService provides access to SambaNova’s language models through an OpenAI-compatible interface. It inherits from OpenAILLMService and supports streaming responses, function calling, and context management. ​ Installation To use SambaNovaLLMService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[sambanova]" You also need to set up your SambaNova API key as an environment variable: SAMBANOVA_API_KEY . Get your SambaNova API key here . ​ Configuration ​ Constructor Parameters ​ api_key str required Your SambaNova API key ​ model str default: "Llama-4-Maverick-17B-128E-Instruct" Model identifier ​ base_url str default: "https://api.sambanova.ai/v1" SambaNova API endpoint ​ Input Parameters Inherits OpenAI-compatible parameters: ​ max_tokens Optional[int] Maximum number of tokens to generate. Must be greater than or equal to 1. ​ temperature Optional[float] Controls randomness in the output. Range: [0.0, 1.0]. ​ top_p Optional[float] Controls diversity via nucleus sampling. Range: [0.0, 1.0] ​ Usage Example Copy Ask AI from pipecat.adapters.schemas.function_schema import FunctionSchema from pipecat.adapters.schemas.tools_schema import ToolsSchema from pipecat.services.sambanova.llm import SambaNovaLLMService from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext from openai.types.chat import ChatCompletionToolParam from pipecat.pipeline.pipeline import Pipeline from pipecat.pipeline.task import PipelineParams, PipelineTask from pipecat.services.llm_service import FunctionCallParams # Configure service llm = SambaNovaLLMService( api_key 'your-sambanova-api-key' , model = 'Llama-4-Maverick-17B-128E-Instruct' , params = SambaNovaLLMService.InputParams( temperature = 0.7 , max_tokens = 1024 ), ) # Define function to call async def fetch_weather ( params : FunctionCallParams) -> Any: """Mock function that fetches the weather forcast from an API.""" await params.result_callback({ 'conditions' : 'nice' , 'temperature' : '20 Degrees Celsius' }) # Register function handlers llm.register_function( 'get_current_weather' , fetch_weather) # Define weather function using standardized schema weather_function = FunctionSchema( name = 'get_current_weather' , description = 'Get the current weather' , properties = { 'location' : { 'type' : 'string' , 'description' : 'The city and state.' , }, 'format' : { 'type' : 'string' , 'enum' : [ 'celsius' , 'fahrenheit' ], 'description' : "The temperature unit to use. Infer this from the user's location." , }, }, required = [ 'location' , 'format' ], ) # Create tools schema tools = ToolsSchema( standard_tools = [weather_function]) # Define system message messages = [ { 'role' : 'system' , 'content' : 'You are a helpful LLM in a WebRTC call. ' 'Your goal is to demonstrate your capabilities of weather forecasting in a succinct way. ' 'Introduce yourself to the user and then wait for their question. ' 'Elaborate your response into a conversational answer in a creative and helpful way. ' 'Your output will be converted to audio so do not include special characters in your answer. ' 'Once the final answer has been provided, please stop, unless the user asks another question. ' , }, ] # Create context with system message and tools context = OpenAILLMContext(messages, tools) # Context aggregator context_aggregator = llm.create_context_aggregator(context) # Create context aggregator for message handling context_aggregator = llm.create_context_aggregator(context) # Set up pipeline pipeline = Pipeline( [ transport.input(), stt, context_aggregator.user(), llm, tts, transport.output(), context_aggregator.assistant(), ] ) # Create and configure task task = PipelineTask( pipeline, params = PipelineParams( allow_interruptions = True , enable_metrics = True , enable_usage_metrics = True , ), ) ​ Methods See the LLM base class methods for additional functionality. ​ Function Calling This service supports function calling (also known as tool calling) which allows the LLM to request information from external services and APIs. For example, you can enable your bot to: Check current weather conditions. Query databases. Access external APIs. Perform custom actions. Function Calling Guide Learn how to implement function calling with standardized schemas, register handlers, manage context properly, and control execution flow in your conversational AI applications. ​ Available Models Model Name Description DeepSeek-R1 deepseek-ai/DeepSeek-R1 DeepSeek-R1-Distill-Llama-70B deepseek-ai/DeepSeek-R1-Distill-Llama-70B DeepSeek-V3-0324 deepseek-ai/DeepSeek-V3-0324 Llama-4-Maverick-17B-128E-Instruct meta-llama/Llama-4-Maverick-17B-128E-Instruct Llama-4-Scout-17B-16E-Instruct meta-llama/Llama-4-Scout-17B-16E-Instruct Meta-Llama-3.3-70B-Instruct meta-llama/Llama-3.3-70B-Instruct Meta-Llama-3.2-3B-Instruct meta-llama/Llama-3.2-3B-Instruct Meta-Llama-3.2-1B-Instruct meta-llama/Llama-3.2-1B-Instruct Meta-Llama-3.1-405B-Instruct meta-llama/Llama-3.1-405B-Instruct Meta-Llama-3.1-8B-Instruct meta-llama/Llama-3.1-8B-Instruct Meta-Llama-Guard-3-8B meta-llama/Llama-Guard-3-8B QwQ-32B Qwen/QwQ-32B Qwen3-32B Qwen/Qwen3-32B Llama-3.3-Swallow-70B-Instruct-v0.4 Tokyotech-llm/Llama-3.3-Swallow-70B-Instruct-v0.4 See SambaNova’s docs for a complete list of supported models. ​ Frame Flow Inherits the OpenAI LLM Service frame flow: ​ Metrics Support The service collects standard LLM metrics: Token usage (prompt and completion). Processing duration. Time to First Byte (TTFB). Function call metrics. ​ Notes OpenAI-compatible interface. Supports streaming responses. Handles function calling. Manages conversation context. Includes token usage tracking. Thread-safe processing. Automatic error handling. Qwen Together AI On this page Overview Installation Configuration Constructor Parameters Input Parameters Usage Example Methods Function Calling Available Models Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/llm_sambanova_da476082.txt b/llm_sambanova_da476082.txt
new file mode 100644
index 0000000000000000000000000000000000000000..1313753d73998e0e91513b1dfcf2ceb895ba85f3
--- /dev/null
+++ b/llm_sambanova_da476082.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/llm/sambanova#frame-flow
+Title: SambaNova - Pipecat
+==================================================
+
+SambaNova - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation LLM SambaNova Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Anthropic AWS Bedrock Azure Cerebras DeepSeek Fireworks AI Google Gemini Google Vertex AI Grok Groq NVIDIA NIM Ollama OpenAI OpenPipe OpenRouter Perplexity Qwen SambaNova Together AI Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview SambaNovaLLMService provides access to SambaNova’s language models through an OpenAI-compatible interface. It inherits from OpenAILLMService and supports streaming responses, function calling, and context management. ​ Installation To use SambaNovaLLMService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[sambanova]" You also need to set up your SambaNova API key as an environment variable: SAMBANOVA_API_KEY . Get your SambaNova API key here . ​ Configuration ​ Constructor Parameters ​ api_key str required Your SambaNova API key ​ model str default: "Llama-4-Maverick-17B-128E-Instruct" Model identifier ​ base_url str default: "https://api.sambanova.ai/v1" SambaNova API endpoint ​ Input Parameters Inherits OpenAI-compatible parameters: ​ max_tokens Optional[int] Maximum number of tokens to generate. Must be greater than or equal to 1. ​ temperature Optional[float] Controls randomness in the output. Range: [0.0, 1.0]. ​ top_p Optional[float] Controls diversity via nucleus sampling. Range: [0.0, 1.0] ​ Usage Example Copy Ask AI from pipecat.adapters.schemas.function_schema import FunctionSchema from pipecat.adapters.schemas.tools_schema import ToolsSchema from pipecat.services.sambanova.llm import SambaNovaLLMService from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext from openai.types.chat import ChatCompletionToolParam from pipecat.pipeline.pipeline import Pipeline from pipecat.pipeline.task import PipelineParams, PipelineTask from pipecat.services.llm_service import FunctionCallParams # Configure service llm = SambaNovaLLMService( api_key 'your-sambanova-api-key' , model = 'Llama-4-Maverick-17B-128E-Instruct' , params = SambaNovaLLMService.InputParams( temperature = 0.7 , max_tokens = 1024 ), ) # Define function to call async def fetch_weather ( params : FunctionCallParams) -> Any: """Mock function that fetches the weather forcast from an API.""" await params.result_callback({ 'conditions' : 'nice' , 'temperature' : '20 Degrees Celsius' }) # Register function handlers llm.register_function( 'get_current_weather' , fetch_weather) # Define weather function using standardized schema weather_function = FunctionSchema( name = 'get_current_weather' , description = 'Get the current weather' , properties = { 'location' : { 'type' : 'string' , 'description' : 'The city and state.' , }, 'format' : { 'type' : 'string' , 'enum' : [ 'celsius' , 'fahrenheit' ], 'description' : "The temperature unit to use. Infer this from the user's location." , }, }, required = [ 'location' , 'format' ], ) # Create tools schema tools = ToolsSchema( standard_tools = [weather_function]) # Define system message messages = [ { 'role' : 'system' , 'content' : 'You are a helpful LLM in a WebRTC call. ' 'Your goal is to demonstrate your capabilities of weather forecasting in a succinct way. ' 'Introduce yourself to the user and then wait for their question. ' 'Elaborate your response into a conversational answer in a creative and helpful way. ' 'Your output will be converted to audio so do not include special characters in your answer. ' 'Once the final answer has been provided, please stop, unless the user asks another question. ' , }, ] # Create context with system message and tools context = OpenAILLMContext(messages, tools) # Context aggregator context_aggregator = llm.create_context_aggregator(context) # Create context aggregator for message handling context_aggregator = llm.create_context_aggregator(context) # Set up pipeline pipeline = Pipeline( [ transport.input(), stt, context_aggregator.user(), llm, tts, transport.output(), context_aggregator.assistant(), ] ) # Create and configure task task = PipelineTask( pipeline, params = PipelineParams( allow_interruptions = True , enable_metrics = True , enable_usage_metrics = True , ), ) ​ Methods See the LLM base class methods for additional functionality. ​ Function Calling This service supports function calling (also known as tool calling) which allows the LLM to request information from external services and APIs. For example, you can enable your bot to: Check current weather conditions. Query databases. Access external APIs. Perform custom actions. Function Calling Guide Learn how to implement function calling with standardized schemas, register handlers, manage context properly, and control execution flow in your conversational AI applications. ​ Available Models Model Name Description DeepSeek-R1 deepseek-ai/DeepSeek-R1 DeepSeek-R1-Distill-Llama-70B deepseek-ai/DeepSeek-R1-Distill-Llama-70B DeepSeek-V3-0324 deepseek-ai/DeepSeek-V3-0324 Llama-4-Maverick-17B-128E-Instruct meta-llama/Llama-4-Maverick-17B-128E-Instruct Llama-4-Scout-17B-16E-Instruct meta-llama/Llama-4-Scout-17B-16E-Instruct Meta-Llama-3.3-70B-Instruct meta-llama/Llama-3.3-70B-Instruct Meta-Llama-3.2-3B-Instruct meta-llama/Llama-3.2-3B-Instruct Meta-Llama-3.2-1B-Instruct meta-llama/Llama-3.2-1B-Instruct Meta-Llama-3.1-405B-Instruct meta-llama/Llama-3.1-405B-Instruct Meta-Llama-3.1-8B-Instruct meta-llama/Llama-3.1-8B-Instruct Meta-Llama-Guard-3-8B meta-llama/Llama-Guard-3-8B QwQ-32B Qwen/QwQ-32B Qwen3-32B Qwen/Qwen3-32B Llama-3.3-Swallow-70B-Instruct-v0.4 Tokyotech-llm/Llama-3.3-Swallow-70B-Instruct-v0.4 See SambaNova’s docs for a complete list of supported models. ​ Frame Flow Inherits the OpenAI LLM Service frame flow: ​ Metrics Support The service collects standard LLM metrics: Token usage (prompt and completion). Processing duration. Time to First Byte (TTFB). Function call metrics. ​ Notes OpenAI-compatible interface. Supports streaming responses. Handles function calling. Manages conversation context. Includes token usage tracking. Thread-safe processing. Automatic error handling. Qwen Together AI On this page Overview Installation Configuration Constructor Parameters Input Parameters Usage Example Methods Function Calling Available Models Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/llm_sambanova_f89434c4.txt b/llm_sambanova_f89434c4.txt
new file mode 100644
index 0000000000000000000000000000000000000000..4e3f6d254cdf938d5eb7e3edc19f7a1fe41b4cfc
--- /dev/null
+++ b/llm_sambanova_f89434c4.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/llm/sambanova#param-top-p
+Title: SambaNova - Pipecat
+==================================================
+
+SambaNova - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation LLM SambaNova Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Anthropic AWS Bedrock Azure Cerebras DeepSeek Fireworks AI Google Gemini Google Vertex AI Grok Groq NVIDIA NIM Ollama OpenAI OpenPipe OpenRouter Perplexity Qwen SambaNova Together AI Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview SambaNovaLLMService provides access to SambaNova’s language models through an OpenAI-compatible interface. It inherits from OpenAILLMService and supports streaming responses, function calling, and context management. ​ Installation To use SambaNovaLLMService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[sambanova]" You also need to set up your SambaNova API key as an environment variable: SAMBANOVA_API_KEY . Get your SambaNova API key here . ​ Configuration ​ Constructor Parameters ​ api_key str required Your SambaNova API key ​ model str default: "Llama-4-Maverick-17B-128E-Instruct" Model identifier ​ base_url str default: "https://api.sambanova.ai/v1" SambaNova API endpoint ​ Input Parameters Inherits OpenAI-compatible parameters: ​ max_tokens Optional[int] Maximum number of tokens to generate. Must be greater than or equal to 1. ​ temperature Optional[float] Controls randomness in the output. Range: [0.0, 1.0]. ​ top_p Optional[float] Controls diversity via nucleus sampling. Range: [0.0, 1.0] ​ Usage Example Copy Ask AI from pipecat.adapters.schemas.function_schema import FunctionSchema from pipecat.adapters.schemas.tools_schema import ToolsSchema from pipecat.services.sambanova.llm import SambaNovaLLMService from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext from openai.types.chat import ChatCompletionToolParam from pipecat.pipeline.pipeline import Pipeline from pipecat.pipeline.task import PipelineParams, PipelineTask from pipecat.services.llm_service import FunctionCallParams # Configure service llm = SambaNovaLLMService( api_key 'your-sambanova-api-key' , model = 'Llama-4-Maverick-17B-128E-Instruct' , params = SambaNovaLLMService.InputParams( temperature = 0.7 , max_tokens = 1024 ), ) # Define function to call async def fetch_weather ( params : FunctionCallParams) -> Any: """Mock function that fetches the weather forcast from an API.""" await params.result_callback({ 'conditions' : 'nice' , 'temperature' : '20 Degrees Celsius' }) # Register function handlers llm.register_function( 'get_current_weather' , fetch_weather) # Define weather function using standardized schema weather_function = FunctionSchema( name = 'get_current_weather' , description = 'Get the current weather' , properties = { 'location' : { 'type' : 'string' , 'description' : 'The city and state.' , }, 'format' : { 'type' : 'string' , 'enum' : [ 'celsius' , 'fahrenheit' ], 'description' : "The temperature unit to use. Infer this from the user's location." , }, }, required = [ 'location' , 'format' ], ) # Create tools schema tools = ToolsSchema( standard_tools = [weather_function]) # Define system message messages = [ { 'role' : 'system' , 'content' : 'You are a helpful LLM in a WebRTC call. ' 'Your goal is to demonstrate your capabilities of weather forecasting in a succinct way. ' 'Introduce yourself to the user and then wait for their question. ' 'Elaborate your response into a conversational answer in a creative and helpful way. ' 'Your output will be converted to audio so do not include special characters in your answer. ' 'Once the final answer has been provided, please stop, unless the user asks another question. ' , }, ] # Create context with system message and tools context = OpenAILLMContext(messages, tools) # Context aggregator context_aggregator = llm.create_context_aggregator(context) # Create context aggregator for message handling context_aggregator = llm.create_context_aggregator(context) # Set up pipeline pipeline = Pipeline( [ transport.input(), stt, context_aggregator.user(), llm, tts, transport.output(), context_aggregator.assistant(), ] ) # Create and configure task task = PipelineTask( pipeline, params = PipelineParams( allow_interruptions = True , enable_metrics = True , enable_usage_metrics = True , ), ) ​ Methods See the LLM base class methods for additional functionality. ​ Function Calling This service supports function calling (also known as tool calling) which allows the LLM to request information from external services and APIs. For example, you can enable your bot to: Check current weather conditions. Query databases. Access external APIs. Perform custom actions. Function Calling Guide Learn how to implement function calling with standardized schemas, register handlers, manage context properly, and control execution flow in your conversational AI applications. ​ Available Models Model Name Description DeepSeek-R1 deepseek-ai/DeepSeek-R1 DeepSeek-R1-Distill-Llama-70B deepseek-ai/DeepSeek-R1-Distill-Llama-70B DeepSeek-V3-0324 deepseek-ai/DeepSeek-V3-0324 Llama-4-Maverick-17B-128E-Instruct meta-llama/Llama-4-Maverick-17B-128E-Instruct Llama-4-Scout-17B-16E-Instruct meta-llama/Llama-4-Scout-17B-16E-Instruct Meta-Llama-3.3-70B-Instruct meta-llama/Llama-3.3-70B-Instruct Meta-Llama-3.2-3B-Instruct meta-llama/Llama-3.2-3B-Instruct Meta-Llama-3.2-1B-Instruct meta-llama/Llama-3.2-1B-Instruct Meta-Llama-3.1-405B-Instruct meta-llama/Llama-3.1-405B-Instruct Meta-Llama-3.1-8B-Instruct meta-llama/Llama-3.1-8B-Instruct Meta-Llama-Guard-3-8B meta-llama/Llama-Guard-3-8B QwQ-32B Qwen/QwQ-32B Qwen3-32B Qwen/Qwen3-32B Llama-3.3-Swallow-70B-Instruct-v0.4 Tokyotech-llm/Llama-3.3-Swallow-70B-Instruct-v0.4 See SambaNova’s docs for a complete list of supported models. ​ Frame Flow Inherits the OpenAI LLM Service frame flow: ​ Metrics Support The service collects standard LLM metrics: Token usage (prompt and completion). Processing duration. Time to First Byte (TTFB). Function call metrics. ​ Notes OpenAI-compatible interface. Supports streaming responses. Handles function calling. Manages conversation context. Includes token usage tracking. Thread-safe processing. Automatic error handling. Qwen Together AI On this page Overview Installation Configuration Constructor Parameters Input Parameters Usage Example Methods Function Calling Available Models Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/llm_together_4c92e3fb.txt b/llm_together_4c92e3fb.txt
new file mode 100644
index 0000000000000000000000000000000000000000..7c620a3e1771812670e3e43344a4953d402fee1e
--- /dev/null
+++ b/llm_together_4c92e3fb.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/llm/together#frames
+Title: Together AI - Pipecat
+==================================================
+
+Together AI - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation LLM Together AI Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Anthropic AWS Bedrock Azure Cerebras DeepSeek Fireworks AI Google Gemini Google Vertex AI Grok Groq NVIDIA NIM Ollama OpenAI OpenPipe OpenRouter Perplexity Qwen SambaNova Together AI Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview TogetherLLMService provides access to Together AI’s language models, including Meta’s Llama 3.1 and 3.2 models, through an OpenAI-compatible interface. It inherits from OpenAILLMService and supports streaming responses, function calling, and context management. API Reference Complete API documentation and method details Together AI Docs Official Together AI API documentation and features Example Code Working example with function calling ​ Installation To use TogetherLLMService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[together]" You’ll also need to set up your Together AI API key as an environment variable: TOGETHER_API_KEY . Get your API key from Together AI Console . ​ Frames ​ Input OpenAILLMContextFrame - Conversation context and history LLMMessagesFrame - Direct message list VisionImageRawFrame - Images for vision processing (select models) LLMUpdateSettingsFrame - Runtime parameter updates ​ Output LLMFullResponseStartFrame / LLMFullResponseEndFrame - Response boundaries LLMTextFrame - Streamed completion chunks FunctionCallInProgressFrame / FunctionCallResultFrame - Function call lifecycle ErrorFrame - API or processing errors ​ Function Calling Function Calling Guide Learn how to implement function calling with standardized schemas, register handlers, manage context properly, and control execution flow in your conversational AI applications. ​ Context Management Context Management Guide Learn how to manage conversation context, handle message history, and integrate context aggregators for consistent conversational experiences. ​ Usage Example Copy Ask AI import os from pipecat.services.together.llm import TogetherLLMService from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext from pipecat.adapters.schemas.function_schema import FunctionSchema from pipecat.adapters.schemas.tools_schema import ToolsSchema # Configure Together AI service llm = TogetherLLMService( api_key = os.getenv( "TOGETHER_API_KEY" ), model = "meta-llama/Meta-Llama-3.1-70B-Instruct-Turbo" , # Balanced performance params = TogetherLLMService.InputParams( temperature = 0.7 , max_tokens = 1000 ) ) # Define function for tool calling weather_function = FunctionSchema( name = "get_current_weather" , description = "Get current weather information" , properties = { "location" : { "type" : "string" , "description" : "City and state, e.g. San Francisco, CA" }, "format" : { "type" : "string" , "enum" : [ "celsius" , "fahrenheit" ], "description" : "Temperature unit to use" } }, required = [ "location" , "format" ] ) tools = ToolsSchema( standard_tools = [weather_function]) # Create context optimized for voice context = OpenAILLMContext( messages = [ { "role" : "system" , "content" : """You are a helpful assistant in a voice conversation. Keep responses concise and avoid special characters for better speech synthesis.""" } ], tools = tools ) # Create context aggregators context_aggregator = llm.create_context_aggregator(context) # Register function handler with feedback async def fetch_weather ( params ): location = params.arguments[ "location" ] await params.result_callback({ "conditions" : "sunny" , "temperature" : "75°F" }) llm.register_function( "get_current_weather" , fetch_weather) # Optional: Add function call feedback @llm.event_handler ( "on_function_calls_started" ) async def on_function_calls_started ( service , function_calls ): await tts.queue_frame(TTSSpeakFrame( "Let me check on that." )) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, context_aggregator.user(), llm, tts, transport.output(), context_aggregator.assistant() ]) ​ Metrics Inherits all OpenAI metrics capabilities: Time to First Byte (TTFB) - Response latency measurement Processing Duration - Total request processing time Token Usage - Prompt tokens, completion tokens, and totals Enable with: Copy Ask AI task = PipelineTask( pipeline, params = PipelineParams( enable_metrics = True , enable_usage_metrics = True ) ) ​ Additional Notes OpenAI Compatibility : Full compatibility with OpenAI API features and parameters Open Source Models : Access to cutting-edge open-source models like Llama Vision Support : Select models support multimodal image and text understanding Competitive Pricing : Cost-effective alternative to proprietary model APIs Flexible Scaling : Choose model size based on performance vs cost requirements SambaNova AWS Polly On this page Overview Installation Frames Input Output Function Calling Context Management Usage Example Metrics Additional Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/llm_together_5190055f.txt b/llm_together_5190055f.txt
new file mode 100644
index 0000000000000000000000000000000000000000..1f718a5a52907507277790a25454a53575f7e85f
--- /dev/null
+++ b/llm_together_5190055f.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/llm/together#context-management
+Title: Together AI - Pipecat
+==================================================
+
+Together AI - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation LLM Together AI Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Anthropic AWS Bedrock Azure Cerebras DeepSeek Fireworks AI Google Gemini Google Vertex AI Grok Groq NVIDIA NIM Ollama OpenAI OpenPipe OpenRouter Perplexity Qwen SambaNova Together AI Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview TogetherLLMService provides access to Together AI’s language models, including Meta’s Llama 3.1 and 3.2 models, through an OpenAI-compatible interface. It inherits from OpenAILLMService and supports streaming responses, function calling, and context management. API Reference Complete API documentation and method details Together AI Docs Official Together AI API documentation and features Example Code Working example with function calling ​ Installation To use TogetherLLMService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[together]" You’ll also need to set up your Together AI API key as an environment variable: TOGETHER_API_KEY . Get your API key from Together AI Console . ​ Frames ​ Input OpenAILLMContextFrame - Conversation context and history LLMMessagesFrame - Direct message list VisionImageRawFrame - Images for vision processing (select models) LLMUpdateSettingsFrame - Runtime parameter updates ​ Output LLMFullResponseStartFrame / LLMFullResponseEndFrame - Response boundaries LLMTextFrame - Streamed completion chunks FunctionCallInProgressFrame / FunctionCallResultFrame - Function call lifecycle ErrorFrame - API or processing errors ​ Function Calling Function Calling Guide Learn how to implement function calling with standardized schemas, register handlers, manage context properly, and control execution flow in your conversational AI applications. ​ Context Management Context Management Guide Learn how to manage conversation context, handle message history, and integrate context aggregators for consistent conversational experiences. ​ Usage Example Copy Ask AI import os from pipecat.services.together.llm import TogetherLLMService from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext from pipecat.adapters.schemas.function_schema import FunctionSchema from pipecat.adapters.schemas.tools_schema import ToolsSchema # Configure Together AI service llm = TogetherLLMService( api_key = os.getenv( "TOGETHER_API_KEY" ), model = "meta-llama/Meta-Llama-3.1-70B-Instruct-Turbo" , # Balanced performance params = TogetherLLMService.InputParams( temperature = 0.7 , max_tokens = 1000 ) ) # Define function for tool calling weather_function = FunctionSchema( name = "get_current_weather" , description = "Get current weather information" , properties = { "location" : { "type" : "string" , "description" : "City and state, e.g. San Francisco, CA" }, "format" : { "type" : "string" , "enum" : [ "celsius" , "fahrenheit" ], "description" : "Temperature unit to use" } }, required = [ "location" , "format" ] ) tools = ToolsSchema( standard_tools = [weather_function]) # Create context optimized for voice context = OpenAILLMContext( messages = [ { "role" : "system" , "content" : """You are a helpful assistant in a voice conversation. Keep responses concise and avoid special characters for better speech synthesis.""" } ], tools = tools ) # Create context aggregators context_aggregator = llm.create_context_aggregator(context) # Register function handler with feedback async def fetch_weather ( params ): location = params.arguments[ "location" ] await params.result_callback({ "conditions" : "sunny" , "temperature" : "75°F" }) llm.register_function( "get_current_weather" , fetch_weather) # Optional: Add function call feedback @llm.event_handler ( "on_function_calls_started" ) async def on_function_calls_started ( service , function_calls ): await tts.queue_frame(TTSSpeakFrame( "Let me check on that." )) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, context_aggregator.user(), llm, tts, transport.output(), context_aggregator.assistant() ]) ​ Metrics Inherits all OpenAI metrics capabilities: Time to First Byte (TTFB) - Response latency measurement Processing Duration - Total request processing time Token Usage - Prompt tokens, completion tokens, and totals Enable with: Copy Ask AI task = PipelineTask( pipeline, params = PipelineParams( enable_metrics = True , enable_usage_metrics = True ) ) ​ Additional Notes OpenAI Compatibility : Full compatibility with OpenAI API features and parameters Open Source Models : Access to cutting-edge open-source models like Llama Vision Support : Select models support multimodal image and text understanding Competitive Pricing : Cost-effective alternative to proprietary model APIs Flexible Scaling : Choose model size based on performance vs cost requirements SambaNova AWS Polly On this page Overview Installation Frames Input Output Function Calling Context Management Usage Example Metrics Additional Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/llm_together_ccde5679.txt b/llm_together_ccde5679.txt
new file mode 100644
index 0000000000000000000000000000000000000000..41a5177c1f4a87392f2e70fe197d49a197e21010
--- /dev/null
+++ b/llm_together_ccde5679.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/llm/together#usage-example
+Title: Together AI - Pipecat
+==================================================
+
+Together AI - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation LLM Together AI Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Anthropic AWS Bedrock Azure Cerebras DeepSeek Fireworks AI Google Gemini Google Vertex AI Grok Groq NVIDIA NIM Ollama OpenAI OpenPipe OpenRouter Perplexity Qwen SambaNova Together AI Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview TogetherLLMService provides access to Together AI’s language models, including Meta’s Llama 3.1 and 3.2 models, through an OpenAI-compatible interface. It inherits from OpenAILLMService and supports streaming responses, function calling, and context management. API Reference Complete API documentation and method details Together AI Docs Official Together AI API documentation and features Example Code Working example with function calling ​ Installation To use TogetherLLMService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[together]" You’ll also need to set up your Together AI API key as an environment variable: TOGETHER_API_KEY . Get your API key from Together AI Console . ​ Frames ​ Input OpenAILLMContextFrame - Conversation context and history LLMMessagesFrame - Direct message list VisionImageRawFrame - Images for vision processing (select models) LLMUpdateSettingsFrame - Runtime parameter updates ​ Output LLMFullResponseStartFrame / LLMFullResponseEndFrame - Response boundaries LLMTextFrame - Streamed completion chunks FunctionCallInProgressFrame / FunctionCallResultFrame - Function call lifecycle ErrorFrame - API or processing errors ​ Function Calling Function Calling Guide Learn how to implement function calling with standardized schemas, register handlers, manage context properly, and control execution flow in your conversational AI applications. ​ Context Management Context Management Guide Learn how to manage conversation context, handle message history, and integrate context aggregators for consistent conversational experiences. ​ Usage Example Copy Ask AI import os from pipecat.services.together.llm import TogetherLLMService from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext from pipecat.adapters.schemas.function_schema import FunctionSchema from pipecat.adapters.schemas.tools_schema import ToolsSchema # Configure Together AI service llm = TogetherLLMService( api_key = os.getenv( "TOGETHER_API_KEY" ), model = "meta-llama/Meta-Llama-3.1-70B-Instruct-Turbo" , # Balanced performance params = TogetherLLMService.InputParams( temperature = 0.7 , max_tokens = 1000 ) ) # Define function for tool calling weather_function = FunctionSchema( name = "get_current_weather" , description = "Get current weather information" , properties = { "location" : { "type" : "string" , "description" : "City and state, e.g. San Francisco, CA" }, "format" : { "type" : "string" , "enum" : [ "celsius" , "fahrenheit" ], "description" : "Temperature unit to use" } }, required = [ "location" , "format" ] ) tools = ToolsSchema( standard_tools = [weather_function]) # Create context optimized for voice context = OpenAILLMContext( messages = [ { "role" : "system" , "content" : """You are a helpful assistant in a voice conversation. Keep responses concise and avoid special characters for better speech synthesis.""" } ], tools = tools ) # Create context aggregators context_aggregator = llm.create_context_aggregator(context) # Register function handler with feedback async def fetch_weather ( params ): location = params.arguments[ "location" ] await params.result_callback({ "conditions" : "sunny" , "temperature" : "75°F" }) llm.register_function( "get_current_weather" , fetch_weather) # Optional: Add function call feedback @llm.event_handler ( "on_function_calls_started" ) async def on_function_calls_started ( service , function_calls ): await tts.queue_frame(TTSSpeakFrame( "Let me check on that." )) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, context_aggregator.user(), llm, tts, transport.output(), context_aggregator.assistant() ]) ​ Metrics Inherits all OpenAI metrics capabilities: Time to First Byte (TTFB) - Response latency measurement Processing Duration - Total request processing time Token Usage - Prompt tokens, completion tokens, and totals Enable with: Copy Ask AI task = PipelineTask( pipeline, params = PipelineParams( enable_metrics = True , enable_usage_metrics = True ) ) ​ Additional Notes OpenAI Compatibility : Full compatibility with OpenAI API features and parameters Open Source Models : Access to cutting-edge open-source models like Llama Vision Support : Select models support multimodal image and text understanding Competitive Pricing : Cost-effective alternative to proprietary model APIs Flexible Scaling : Choose model size based on performance vs cost requirements SambaNova AWS Polly On this page Overview Installation Frames Input Output Function Calling Context Management Usage Example Metrics Additional Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/llm_together_dd62fd16.txt b/llm_together_dd62fd16.txt
new file mode 100644
index 0000000000000000000000000000000000000000..1d9fa9a5ea2d815abd85270ee703c0bcf8f72ed5
--- /dev/null
+++ b/llm_together_dd62fd16.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/llm/together#additional-notes
+Title: Together AI - Pipecat
+==================================================
+
+Together AI - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation LLM Together AI Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Anthropic AWS Bedrock Azure Cerebras DeepSeek Fireworks AI Google Gemini Google Vertex AI Grok Groq NVIDIA NIM Ollama OpenAI OpenPipe OpenRouter Perplexity Qwen SambaNova Together AI Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview TogetherLLMService provides access to Together AI’s language models, including Meta’s Llama 3.1 and 3.2 models, through an OpenAI-compatible interface. It inherits from OpenAILLMService and supports streaming responses, function calling, and context management. API Reference Complete API documentation and method details Together AI Docs Official Together AI API documentation and features Example Code Working example with function calling ​ Installation To use TogetherLLMService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[together]" You’ll also need to set up your Together AI API key as an environment variable: TOGETHER_API_KEY . Get your API key from Together AI Console . ​ Frames ​ Input OpenAILLMContextFrame - Conversation context and history LLMMessagesFrame - Direct message list VisionImageRawFrame - Images for vision processing (select models) LLMUpdateSettingsFrame - Runtime parameter updates ​ Output LLMFullResponseStartFrame / LLMFullResponseEndFrame - Response boundaries LLMTextFrame - Streamed completion chunks FunctionCallInProgressFrame / FunctionCallResultFrame - Function call lifecycle ErrorFrame - API or processing errors ​ Function Calling Function Calling Guide Learn how to implement function calling with standardized schemas, register handlers, manage context properly, and control execution flow in your conversational AI applications. ​ Context Management Context Management Guide Learn how to manage conversation context, handle message history, and integrate context aggregators for consistent conversational experiences. ​ Usage Example Copy Ask AI import os from pipecat.services.together.llm import TogetherLLMService from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext from pipecat.adapters.schemas.function_schema import FunctionSchema from pipecat.adapters.schemas.tools_schema import ToolsSchema # Configure Together AI service llm = TogetherLLMService( api_key = os.getenv( "TOGETHER_API_KEY" ), model = "meta-llama/Meta-Llama-3.1-70B-Instruct-Turbo" , # Balanced performance params = TogetherLLMService.InputParams( temperature = 0.7 , max_tokens = 1000 ) ) # Define function for tool calling weather_function = FunctionSchema( name = "get_current_weather" , description = "Get current weather information" , properties = { "location" : { "type" : "string" , "description" : "City and state, e.g. San Francisco, CA" }, "format" : { "type" : "string" , "enum" : [ "celsius" , "fahrenheit" ], "description" : "Temperature unit to use" } }, required = [ "location" , "format" ] ) tools = ToolsSchema( standard_tools = [weather_function]) # Create context optimized for voice context = OpenAILLMContext( messages = [ { "role" : "system" , "content" : """You are a helpful assistant in a voice conversation. Keep responses concise and avoid special characters for better speech synthesis.""" } ], tools = tools ) # Create context aggregators context_aggregator = llm.create_context_aggregator(context) # Register function handler with feedback async def fetch_weather ( params ): location = params.arguments[ "location" ] await params.result_callback({ "conditions" : "sunny" , "temperature" : "75°F" }) llm.register_function( "get_current_weather" , fetch_weather) # Optional: Add function call feedback @llm.event_handler ( "on_function_calls_started" ) async def on_function_calls_started ( service , function_calls ): await tts.queue_frame(TTSSpeakFrame( "Let me check on that." )) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, context_aggregator.user(), llm, tts, transport.output(), context_aggregator.assistant() ]) ​ Metrics Inherits all OpenAI metrics capabilities: Time to First Byte (TTFB) - Response latency measurement Processing Duration - Total request processing time Token Usage - Prompt tokens, completion tokens, and totals Enable with: Copy Ask AI task = PipelineTask( pipeline, params = PipelineParams( enable_metrics = True , enable_usage_metrics = True ) ) ​ Additional Notes OpenAI Compatibility : Full compatibility with OpenAI API features and parameters Open Source Models : Access to cutting-edge open-source models like Llama Vision Support : Select models support multimodal image and text understanding Competitive Pricing : Cost-effective alternative to proprietary model APIs Flexible Scaling : Choose model size based on performance vs cost requirements SambaNova AWS Polly On this page Overview Installation Frames Input Output Function Calling Context Management Usage Example Metrics Additional Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/llm_together_e7ccd69c.txt b/llm_together_e7ccd69c.txt
new file mode 100644
index 0000000000000000000000000000000000000000..39a50fbd2fd934a2169c6c8e113c0b50aef37855
--- /dev/null
+++ b/llm_together_e7ccd69c.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/llm/together#function-calling
+Title: Together AI - Pipecat
+==================================================
+
+Together AI - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation LLM Together AI Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Anthropic AWS Bedrock Azure Cerebras DeepSeek Fireworks AI Google Gemini Google Vertex AI Grok Groq NVIDIA NIM Ollama OpenAI OpenPipe OpenRouter Perplexity Qwen SambaNova Together AI Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview TogetherLLMService provides access to Together AI’s language models, including Meta’s Llama 3.1 and 3.2 models, through an OpenAI-compatible interface. It inherits from OpenAILLMService and supports streaming responses, function calling, and context management. API Reference Complete API documentation and method details Together AI Docs Official Together AI API documentation and features Example Code Working example with function calling ​ Installation To use TogetherLLMService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[together]" You’ll also need to set up your Together AI API key as an environment variable: TOGETHER_API_KEY . Get your API key from Together AI Console . ​ Frames ​ Input OpenAILLMContextFrame - Conversation context and history LLMMessagesFrame - Direct message list VisionImageRawFrame - Images for vision processing (select models) LLMUpdateSettingsFrame - Runtime parameter updates ​ Output LLMFullResponseStartFrame / LLMFullResponseEndFrame - Response boundaries LLMTextFrame - Streamed completion chunks FunctionCallInProgressFrame / FunctionCallResultFrame - Function call lifecycle ErrorFrame - API or processing errors ​ Function Calling Function Calling Guide Learn how to implement function calling with standardized schemas, register handlers, manage context properly, and control execution flow in your conversational AI applications. ​ Context Management Context Management Guide Learn how to manage conversation context, handle message history, and integrate context aggregators for consistent conversational experiences. ​ Usage Example Copy Ask AI import os from pipecat.services.together.llm import TogetherLLMService from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext from pipecat.adapters.schemas.function_schema import FunctionSchema from pipecat.adapters.schemas.tools_schema import ToolsSchema # Configure Together AI service llm = TogetherLLMService( api_key = os.getenv( "TOGETHER_API_KEY" ), model = "meta-llama/Meta-Llama-3.1-70B-Instruct-Turbo" , # Balanced performance params = TogetherLLMService.InputParams( temperature = 0.7 , max_tokens = 1000 ) ) # Define function for tool calling weather_function = FunctionSchema( name = "get_current_weather" , description = "Get current weather information" , properties = { "location" : { "type" : "string" , "description" : "City and state, e.g. San Francisco, CA" }, "format" : { "type" : "string" , "enum" : [ "celsius" , "fahrenheit" ], "description" : "Temperature unit to use" } }, required = [ "location" , "format" ] ) tools = ToolsSchema( standard_tools = [weather_function]) # Create context optimized for voice context = OpenAILLMContext( messages = [ { "role" : "system" , "content" : """You are a helpful assistant in a voice conversation. Keep responses concise and avoid special characters for better speech synthesis.""" } ], tools = tools ) # Create context aggregators context_aggregator = llm.create_context_aggregator(context) # Register function handler with feedback async def fetch_weather ( params ): location = params.arguments[ "location" ] await params.result_callback({ "conditions" : "sunny" , "temperature" : "75°F" }) llm.register_function( "get_current_weather" , fetch_weather) # Optional: Add function call feedback @llm.event_handler ( "on_function_calls_started" ) async def on_function_calls_started ( service , function_calls ): await tts.queue_frame(TTSSpeakFrame( "Let me check on that." )) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, context_aggregator.user(), llm, tts, transport.output(), context_aggregator.assistant() ]) ​ Metrics Inherits all OpenAI metrics capabilities: Time to First Byte (TTFB) - Response latency measurement Processing Duration - Total request processing time Token Usage - Prompt tokens, completion tokens, and totals Enable with: Copy Ask AI task = PipelineTask( pipeline, params = PipelineParams( enable_metrics = True , enable_usage_metrics = True ) ) ​ Additional Notes OpenAI Compatibility : Full compatibility with OpenAI API features and parameters Open Source Models : Access to cutting-edge open-source models like Llama Vision Support : Select models support multimodal image and text understanding Competitive Pricing : Cost-effective alternative to proprietary model APIs Flexible Scaling : Choose model size based on performance vs cost requirements SambaNova AWS Polly On this page Overview Installation Frames Input Output Function Calling Context Management Usage Example Metrics Additional Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/llm_together_fe91267a.txt b/llm_together_fe91267a.txt
new file mode 100644
index 0000000000000000000000000000000000000000..9f140efd70fe623d994960a3edf16e83d43edf55
--- /dev/null
+++ b/llm_together_fe91267a.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/llm/together#overview
+Title: Together AI - Pipecat
+==================================================
+
+Together AI - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation LLM Together AI Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Anthropic AWS Bedrock Azure Cerebras DeepSeek Fireworks AI Google Gemini Google Vertex AI Grok Groq NVIDIA NIM Ollama OpenAI OpenPipe OpenRouter Perplexity Qwen SambaNova Together AI Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview TogetherLLMService provides access to Together AI’s language models, including Meta’s Llama 3.1 and 3.2 models, through an OpenAI-compatible interface. It inherits from OpenAILLMService and supports streaming responses, function calling, and context management. API Reference Complete API documentation and method details Together AI Docs Official Together AI API documentation and features Example Code Working example with function calling ​ Installation To use TogetherLLMService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[together]" You’ll also need to set up your Together AI API key as an environment variable: TOGETHER_API_KEY . Get your API key from Together AI Console . ​ Frames ​ Input OpenAILLMContextFrame - Conversation context and history LLMMessagesFrame - Direct message list VisionImageRawFrame - Images for vision processing (select models) LLMUpdateSettingsFrame - Runtime parameter updates ​ Output LLMFullResponseStartFrame / LLMFullResponseEndFrame - Response boundaries LLMTextFrame - Streamed completion chunks FunctionCallInProgressFrame / FunctionCallResultFrame - Function call lifecycle ErrorFrame - API or processing errors ​ Function Calling Function Calling Guide Learn how to implement function calling with standardized schemas, register handlers, manage context properly, and control execution flow in your conversational AI applications. ​ Context Management Context Management Guide Learn how to manage conversation context, handle message history, and integrate context aggregators for consistent conversational experiences. ​ Usage Example Copy Ask AI import os from pipecat.services.together.llm import TogetherLLMService from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext from pipecat.adapters.schemas.function_schema import FunctionSchema from pipecat.adapters.schemas.tools_schema import ToolsSchema # Configure Together AI service llm = TogetherLLMService( api_key = os.getenv( "TOGETHER_API_KEY" ), model = "meta-llama/Meta-Llama-3.1-70B-Instruct-Turbo" , # Balanced performance params = TogetherLLMService.InputParams( temperature = 0.7 , max_tokens = 1000 ) ) # Define function for tool calling weather_function = FunctionSchema( name = "get_current_weather" , description = "Get current weather information" , properties = { "location" : { "type" : "string" , "description" : "City and state, e.g. San Francisco, CA" }, "format" : { "type" : "string" , "enum" : [ "celsius" , "fahrenheit" ], "description" : "Temperature unit to use" } }, required = [ "location" , "format" ] ) tools = ToolsSchema( standard_tools = [weather_function]) # Create context optimized for voice context = OpenAILLMContext( messages = [ { "role" : "system" , "content" : """You are a helpful assistant in a voice conversation. Keep responses concise and avoid special characters for better speech synthesis.""" } ], tools = tools ) # Create context aggregators context_aggregator = llm.create_context_aggregator(context) # Register function handler with feedback async def fetch_weather ( params ): location = params.arguments[ "location" ] await params.result_callback({ "conditions" : "sunny" , "temperature" : "75°F" }) llm.register_function( "get_current_weather" , fetch_weather) # Optional: Add function call feedback @llm.event_handler ( "on_function_calls_started" ) async def on_function_calls_started ( service , function_calls ): await tts.queue_frame(TTSSpeakFrame( "Let me check on that." )) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, context_aggregator.user(), llm, tts, transport.output(), context_aggregator.assistant() ]) ​ Metrics Inherits all OpenAI metrics capabilities: Time to First Byte (TTFB) - Response latency measurement Processing Duration - Total request processing time Token Usage - Prompt tokens, completion tokens, and totals Enable with: Copy Ask AI task = PipelineTask( pipeline, params = PipelineParams( enable_metrics = True , enable_usage_metrics = True ) ) ​ Additional Notes OpenAI Compatibility : Full compatibility with OpenAI API features and parameters Open Source Models : Access to cutting-edge open-source models like Llama Vision Support : Select models support multimodal image and text understanding Competitive Pricing : Cost-effective alternative to proprietary model APIs Flexible Scaling : Choose model size based on performance vs cost requirements SambaNova AWS Polly On this page Overview Installation Frames Input Output Function Calling Context Management Usage Example Metrics Additional Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/mcp_mcp_51902d7b.txt b/mcp_mcp_51902d7b.txt
new file mode 100644
index 0000000000000000000000000000000000000000..3efc1b48abebc9edd97bea615a602dc55246f56c
--- /dev/null
+++ b/mcp_mcp_51902d7b.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/mcp/mcp#mcp-stdio-transport-implementation
+Title: MCPClient - Pipecat
+==================================================
+
+MCPClient - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation MCP MCPClient Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP MCPClient Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview MCP is an open standard for enabling AI agents to interact with external data and tools. MCPClient provides a way to access and call tools via MCP. For example, instead of writing bespoke function call implementations for an external API, you may use an MCP server that provides a bridge to the API. Be aware there may be security implications. See MCP documenation for more details. ​ Installation To use MCPClient , install the required dependencies: Copy Ask AI pip install "pipecat-ai[mcp]" You may also need to set environment variables as required by the specific MCP server to which you are connecting. ​ Configuration ​ Constructor Parameters You can connect to your MCP server via Stdio or SSE transport. See here for more documentation on MCP transports. ​ server_params str | StdioServerParameters required You can provide either: URL: “ https://your.mcp.server/sse ” StdioServerParameters, which are defined as: Copy Ask AI StdioServerParameters( command = "python" , # Executable args = [ "example_server.py" ], # Optional command line arguments env = None , # Optional environment variables ) ​ Input Parameters See more information regarding server params here . ​ Usage Example ​ MCP Stdio Transport Implementation Copy Ask AI # Import MCPClient and StdioServerParameters ... from mcp import StdioServerParameters from pipecat.services.mcp_service import MCPClient ... # Initialize an LLM llm = ... # Initialize and configure MCPClient with server parameters mcp = MCPClient( server_params = StdioServerParameters( command = shutil.which( "npx" ), args = [ "-y" , "@name/mcp-server-name@latest" ], env = { "ENV_API_KEY" : "<env_api_key>" }, ) ) # Create tools schema from the MCP server and register them with llm tools = await mcp.register_tools(llm) # Create context with system message and tools # Tip: Let the LLM know it has access to tools from an MCP server by including it in the system prompt. context = OpenAILLMContext( messages = [ { "role" : "system" , "content" : "You are a helpful assistant in a voice conversation. You have access to MCP tools. Keep responses concise." } ], tools = tools ) ​ MCP SSE Transport Implementation Copy Ask AI # Import MCPClient ... from pipecat.services.mcp_service import MCPClient ... # Initialize an LLM llm = ... # Initialize and configure MCPClient with MCP SSE server url mcp = MCPClient( server_params = "https://your.mcp.server/sse" ) # Create tools schema from the MCP server and register them with llm tools = await mcp.register_tools(llm) # Create context with system message and tools # Tip: Let the LLM know it has access to tools from an MCP server by including it in the system prompt. context = OpenAILLMContext( messages = [ { "role" : "system" , "content" : "You are a helpful assistant in a voice conversation. You have access to MCP tools. Keep responses concise." } ], tools = tools ) ​ Methods ​ register_tools async method Converts MCP tools to Pipecat-friendly function definitions and registers the functions with the llm. Copy Ask AI async def register_tools ( self , llm ) -> ToolsSchema: ​ Additional documentation See MCP’s docs for MCP related updates. OpenTelemetry Observer Pattern On this page Overview Installation Configuration Constructor Parameters Input Parameters Usage Example MCP Stdio Transport Implementation MCP SSE Transport Implementation Methods Additional documentation Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/mcp_mcp_923dda2b.txt b/mcp_mcp_923dda2b.txt
new file mode 100644
index 0000000000000000000000000000000000000000..0693e00e8516681d28c3702814efc8c62a8cc7e9
--- /dev/null
+++ b/mcp_mcp_923dda2b.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/mcp/mcp#input-parameters
+Title: MCPClient - Pipecat
+==================================================
+
+MCPClient - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation MCP MCPClient Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP MCPClient Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview MCP is an open standard for enabling AI agents to interact with external data and tools. MCPClient provides a way to access and call tools via MCP. For example, instead of writing bespoke function call implementations for an external API, you may use an MCP server that provides a bridge to the API. Be aware there may be security implications. See MCP documenation for more details. ​ Installation To use MCPClient , install the required dependencies: Copy Ask AI pip install "pipecat-ai[mcp]" You may also need to set environment variables as required by the specific MCP server to which you are connecting. ​ Configuration ​ Constructor Parameters You can connect to your MCP server via Stdio or SSE transport. See here for more documentation on MCP transports. ​ server_params str | StdioServerParameters required You can provide either: URL: “ https://your.mcp.server/sse ” StdioServerParameters, which are defined as: Copy Ask AI StdioServerParameters( command = "python" , # Executable args = [ "example_server.py" ], # Optional command line arguments env = None , # Optional environment variables ) ​ Input Parameters See more information regarding server params here . ​ Usage Example ​ MCP Stdio Transport Implementation Copy Ask AI # Import MCPClient and StdioServerParameters ... from mcp import StdioServerParameters from pipecat.services.mcp_service import MCPClient ... # Initialize an LLM llm = ... # Initialize and configure MCPClient with server parameters mcp = MCPClient( server_params = StdioServerParameters( command = shutil.which( "npx" ), args = [ "-y" , "@name/mcp-server-name@latest" ], env = { "ENV_API_KEY" : "<env_api_key>" }, ) ) # Create tools schema from the MCP server and register them with llm tools = await mcp.register_tools(llm) # Create context with system message and tools # Tip: Let the LLM know it has access to tools from an MCP server by including it in the system prompt. context = OpenAILLMContext( messages = [ { "role" : "system" , "content" : "You are a helpful assistant in a voice conversation. You have access to MCP tools. Keep responses concise." } ], tools = tools ) ​ MCP SSE Transport Implementation Copy Ask AI # Import MCPClient ... from pipecat.services.mcp_service import MCPClient ... # Initialize an LLM llm = ... # Initialize and configure MCPClient with MCP SSE server url mcp = MCPClient( server_params = "https://your.mcp.server/sse" ) # Create tools schema from the MCP server and register them with llm tools = await mcp.register_tools(llm) # Create context with system message and tools # Tip: Let the LLM know it has access to tools from an MCP server by including it in the system prompt. context = OpenAILLMContext( messages = [ { "role" : "system" , "content" : "You are a helpful assistant in a voice conversation. You have access to MCP tools. Keep responses concise." } ], tools = tools ) ​ Methods ​ register_tools async method Converts MCP tools to Pipecat-friendly function definitions and registers the functions with the llm. Copy Ask AI async def register_tools ( self , llm ) -> ToolsSchema: ​ Additional documentation See MCP’s docs for MCP related updates. OpenTelemetry Observer Pattern On this page Overview Installation Configuration Constructor Parameters Input Parameters Usage Example MCP Stdio Transport Implementation MCP SSE Transport Implementation Methods Additional documentation Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/mcp_mcp_99da9717.txt b/mcp_mcp_99da9717.txt
new file mode 100644
index 0000000000000000000000000000000000000000..928895842b14bd4b87f46d91fed92bee2a88fb97
--- /dev/null
+++ b/mcp_mcp_99da9717.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/mcp/mcp#installation
+Title: MCPClient - Pipecat
+==================================================
+
+MCPClient - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation MCP MCPClient Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP MCPClient Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview MCP is an open standard for enabling AI agents to interact with external data and tools. MCPClient provides a way to access and call tools via MCP. For example, instead of writing bespoke function call implementations for an external API, you may use an MCP server that provides a bridge to the API. Be aware there may be security implications. See MCP documenation for more details. ​ Installation To use MCPClient , install the required dependencies: Copy Ask AI pip install "pipecat-ai[mcp]" You may also need to set environment variables as required by the specific MCP server to which you are connecting. ​ Configuration ​ Constructor Parameters You can connect to your MCP server via Stdio or SSE transport. See here for more documentation on MCP transports. ​ server_params str | StdioServerParameters required You can provide either: URL: “ https://your.mcp.server/sse ” StdioServerParameters, which are defined as: Copy Ask AI StdioServerParameters( command = "python" , # Executable args = [ "example_server.py" ], # Optional command line arguments env = None , # Optional environment variables ) ​ Input Parameters See more information regarding server params here . ​ Usage Example ​ MCP Stdio Transport Implementation Copy Ask AI # Import MCPClient and StdioServerParameters ... from mcp import StdioServerParameters from pipecat.services.mcp_service import MCPClient ... # Initialize an LLM llm = ... # Initialize and configure MCPClient with server parameters mcp = MCPClient( server_params = StdioServerParameters( command = shutil.which( "npx" ), args = [ "-y" , "@name/mcp-server-name@latest" ], env = { "ENV_API_KEY" : "<env_api_key>" }, ) ) # Create tools schema from the MCP server and register them with llm tools = await mcp.register_tools(llm) # Create context with system message and tools # Tip: Let the LLM know it has access to tools from an MCP server by including it in the system prompt. context = OpenAILLMContext( messages = [ { "role" : "system" , "content" : "You are a helpful assistant in a voice conversation. You have access to MCP tools. Keep responses concise." } ], tools = tools ) ​ MCP SSE Transport Implementation Copy Ask AI # Import MCPClient ... from pipecat.services.mcp_service import MCPClient ... # Initialize an LLM llm = ... # Initialize and configure MCPClient with MCP SSE server url mcp = MCPClient( server_params = "https://your.mcp.server/sse" ) # Create tools schema from the MCP server and register them with llm tools = await mcp.register_tools(llm) # Create context with system message and tools # Tip: Let the LLM know it has access to tools from an MCP server by including it in the system prompt. context = OpenAILLMContext( messages = [ { "role" : "system" , "content" : "You are a helpful assistant in a voice conversation. You have access to MCP tools. Keep responses concise." } ], tools = tools ) ​ Methods ​ register_tools async method Converts MCP tools to Pipecat-friendly function definitions and registers the functions with the llm. Copy Ask AI async def register_tools ( self , llm ) -> ToolsSchema: ​ Additional documentation See MCP’s docs for MCP related updates. OpenTelemetry Observer Pattern On this page Overview Installation Configuration Constructor Parameters Input Parameters Usage Example MCP Stdio Transport Implementation MCP SSE Transport Implementation Methods Additional documentation Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/mcp_mcp_e4e56e4e.txt b/mcp_mcp_e4e56e4e.txt
new file mode 100644
index 0000000000000000000000000000000000000000..3c0eba0af73acde90d7cfdedf1fab772f7968651
--- /dev/null
+++ b/mcp_mcp_e4e56e4e.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/mcp/mcp#configuration
+Title: MCPClient - Pipecat
+==================================================
+
+MCPClient - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation MCP MCPClient Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP MCPClient Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview MCP is an open standard for enabling AI agents to interact with external data and tools. MCPClient provides a way to access and call tools via MCP. For example, instead of writing bespoke function call implementations for an external API, you may use an MCP server that provides a bridge to the API. Be aware there may be security implications. See MCP documenation for more details. ​ Installation To use MCPClient , install the required dependencies: Copy Ask AI pip install "pipecat-ai[mcp]" You may also need to set environment variables as required by the specific MCP server to which you are connecting. ​ Configuration ​ Constructor Parameters You can connect to your MCP server via Stdio or SSE transport. See here for more documentation on MCP transports. ​ server_params str | StdioServerParameters required You can provide either: URL: “ https://your.mcp.server/sse ” StdioServerParameters, which are defined as: Copy Ask AI StdioServerParameters( command = "python" , # Executable args = [ "example_server.py" ], # Optional command line arguments env = None , # Optional environment variables ) ​ Input Parameters See more information regarding server params here . ​ Usage Example ​ MCP Stdio Transport Implementation Copy Ask AI # Import MCPClient and StdioServerParameters ... from mcp import StdioServerParameters from pipecat.services.mcp_service import MCPClient ... # Initialize an LLM llm = ... # Initialize and configure MCPClient with server parameters mcp = MCPClient( server_params = StdioServerParameters( command = shutil.which( "npx" ), args = [ "-y" , "@name/mcp-server-name@latest" ], env = { "ENV_API_KEY" : "<env_api_key>" }, ) ) # Create tools schema from the MCP server and register them with llm tools = await mcp.register_tools(llm) # Create context with system message and tools # Tip: Let the LLM know it has access to tools from an MCP server by including it in the system prompt. context = OpenAILLMContext( messages = [ { "role" : "system" , "content" : "You are a helpful assistant in a voice conversation. You have access to MCP tools. Keep responses concise." } ], tools = tools ) ​ MCP SSE Transport Implementation Copy Ask AI # Import MCPClient ... from pipecat.services.mcp_service import MCPClient ... # Initialize an LLM llm = ... # Initialize and configure MCPClient with MCP SSE server url mcp = MCPClient( server_params = "https://your.mcp.server/sse" ) # Create tools schema from the MCP server and register them with llm tools = await mcp.register_tools(llm) # Create context with system message and tools # Tip: Let the LLM know it has access to tools from an MCP server by including it in the system prompt. context = OpenAILLMContext( messages = [ { "role" : "system" , "content" : "You are a helpful assistant in a voice conversation. You have access to MCP tools. Keep responses concise." } ], tools = tools ) ​ Methods ​ register_tools async method Converts MCP tools to Pipecat-friendly function definitions and registers the functions with the llm. Copy Ask AI async def register_tools ( self , llm ) -> ToolsSchema: ​ Additional documentation See MCP’s docs for MCP related updates. OpenTelemetry Observer Pattern On this page Overview Installation Configuration Constructor Parameters Input Parameters Usage Example MCP Stdio Transport Implementation MCP SSE Transport Implementation Methods Additional documentation Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/memory_mem0_03251ffb.txt b/memory_mem0_03251ffb.txt
new file mode 100644
index 0000000000000000000000000000000000000000..4d95e74c09f3a621baba08ef4289e3ee2d7efa5c
--- /dev/null
+++ b/memory_mem0_03251ffb.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/memory/mem0#constructor-parameters
+Title: Mem0 - Pipecat
+==================================================
+
+Mem0 - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Memory Mem0 Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Mem0 Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview Mem0MemoryService provides long-term memory capabilities for conversational agents by integrating with Mem0’s API. It automatically stores conversation history and retrieves relevant past context based on the current conversation, enhancing LLM responses with persistent memory across sessions. ​ Installation To use the Mem0 memory service, install the required dependencies: Copy Ask AI pip install "pipecat-ai[mem0]" You’ll also need to set up your Mem0 API key as an environment variable: MEM0_API_KEY . You can obtain a Mem0 API key by signing up at mem0.ai . ​ Mem0MemoryService ​ Constructor Parameters ​ api_key str required Mem0 API key for accessing the service ​ user_id str Unique identifier for the end user to associate with memories ​ agent_id str Identifier for the agent using the memory service ​ run_id str Identifier for the specific conversation session ​ params InputParams Configuration parameters for memory retrieval (see below) ​ local_config dict Configuration for using local LLMs and embedders instead of Mem0’s cloud API (see Local Configuration section) At least one of user_id , agent_id , or run_id must be provided to organize memories. ​ Input Parameters The params object accepts the following configuration settings: ​ search_limit int default: "10" Maximum number of relevant memories to retrieve per query ​ search_threshold float default: "0.1" Relevance threshold for memory retrieval (0.0 to 1.0) ​ api_version str default: "v2" Mem0 API version to use ​ system_prompt str Prefix text to add before retrieved memories ​ add_as_system_message bool default: "True" Whether to add memories as a system message (True) or user message (False) ​ position int default: "1" Position in the context where memories should be inserted ​ Input Frames The service processes the following input frames: ​ OpenAILLMContextFrame Frame Contains OpenAI-specific conversation context ​ LLMMessagesFrame Frame Contains conversation messages in standard format ​ Output Frames The service may produce the following output frames: ​ LLMMessagesFrame Frame Enhanced messages with relevant memories included ​ OpenAILLMContextFrame Frame Enhanced OpenAI context with memories included ​ ErrorFrame Frame Contains error information if memory operations fail ​ Memory Operations The service performs two main operations automatically: ​ Message Storage All conversation messages are stored in Mem0 for future reference. The service: Captures full message history from context frames Associates messages with the specified user/agent/run IDs Stores metadata to enable efficient retrieval ​ Memory Retrieval When a new user message is detected, the service: Uses the message as a search query Retrieves relevant past memories from Mem0 Formats memories with the configured system prompt Adds the formatted memories to the conversation context Passes the enhanced context downstream in the pipeline ​ Pipeline Positioning The memory service should be positioned after the user context aggregator but before the LLM service: Copy Ask AI context_aggregator.user() → memory_service → llm This ensures that: The user’s latest message is included in the context The memory service can enhance the context before the LLM processes it The LLM receives the enhanced context with relevant memories ​ Usage Examples ​ Basic Integration Copy Ask AI from pipecat.services.mem0.memory import Mem0MemoryService from pipecat.pipeline.pipeline import Pipeline # Create the memory service memory = Mem0MemoryService( api_key = os.getenv( "MEM0_API_KEY" ), user_id = "user123" , # Unique user identifier ) # Position the memory service between context aggregator and LLM pipeline = Pipeline([ transport.input(), context_aggregator.user(), memory, # <-- Memory service enhances context here llm, tts, transport.output(), context_aggregator.assistant() ]) ​ Using Local Configuration The local_config parameter allows you to use your own LLM and embedding providers instead of Mem0’s cloud API. This is useful for self-hosted deployments or when you want more control over the memory processing. Copy Ask AI local_config = { "llm" : { "provider" : str , # LLM provider name (e.g., "anthropic", "openai") "config" : { # Provider-specific configuration "model" : str , # Model name "api_key" : str , # API key for the provider # Other provider-specific parameters } }, "embedder" : { "provider" : str , # Embedding provider name (e.g., "openai") "config" : { # Provider-specific configuration "model" : str , # Model name # Other provider-specific parameters } } } # Initialize Mem0 memory service with local configuration memory = Mem0MemoryService( local_config = local_config, # Use local LLM for memory processing user_id = "user123" , # Unique identifier for the user ) When using local_config do not provide the api_key parameter. ​ Frame Flow ​ Error Handling The service includes basic error handling to ensure conversation flow continues even when memory operations fail: Exceptions during memory storage and retrieval are caught and logged If an error occurs during frame processing, an ErrorFrame is emitted with error details The original frame is still passed downstream to prevent the pipeline from stalling Connection and authentication errors from the Mem0 API will be logged but won’t interrupt the conversation While the service attempts to handle errors gracefully, memory operations that fail may result in missing context in conversations. Monitor your application logs for memory-related errors. Tavus Moondream On this page Overview Installation Mem0MemoryService Constructor Parameters Input Parameters Input Frames Output Frames Memory Operations Message Storage Memory Retrieval Pipeline Positioning Usage Examples Basic Integration Using Local Configuration Frame Flow Error Handling Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/memory_mem0_3cfa2d65.txt b/memory_mem0_3cfa2d65.txt
new file mode 100644
index 0000000000000000000000000000000000000000..09da974e8b20ba3560952460cc88da6ec0ebb025
--- /dev/null
+++ b/memory_mem0_3cfa2d65.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/memory/mem0#using-local-configuration
+Title: Mem0 - Pipecat
+==================================================
+
+Mem0 - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Memory Mem0 Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Mem0 Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview Mem0MemoryService provides long-term memory capabilities for conversational agents by integrating with Mem0’s API. It automatically stores conversation history and retrieves relevant past context based on the current conversation, enhancing LLM responses with persistent memory across sessions. ​ Installation To use the Mem0 memory service, install the required dependencies: Copy Ask AI pip install "pipecat-ai[mem0]" You’ll also need to set up your Mem0 API key as an environment variable: MEM0_API_KEY . You can obtain a Mem0 API key by signing up at mem0.ai . ​ Mem0MemoryService ​ Constructor Parameters ​ api_key str required Mem0 API key for accessing the service ​ user_id str Unique identifier for the end user to associate with memories ​ agent_id str Identifier for the agent using the memory service ​ run_id str Identifier for the specific conversation session ​ params InputParams Configuration parameters for memory retrieval (see below) ​ local_config dict Configuration for using local LLMs and embedders instead of Mem0’s cloud API (see Local Configuration section) At least one of user_id , agent_id , or run_id must be provided to organize memories. ​ Input Parameters The params object accepts the following configuration settings: ​ search_limit int default: "10" Maximum number of relevant memories to retrieve per query ​ search_threshold float default: "0.1" Relevance threshold for memory retrieval (0.0 to 1.0) ​ api_version str default: "v2" Mem0 API version to use ​ system_prompt str Prefix text to add before retrieved memories ​ add_as_system_message bool default: "True" Whether to add memories as a system message (True) or user message (False) ​ position int default: "1" Position in the context where memories should be inserted ​ Input Frames The service processes the following input frames: ​ OpenAILLMContextFrame Frame Contains OpenAI-specific conversation context ​ LLMMessagesFrame Frame Contains conversation messages in standard format ​ Output Frames The service may produce the following output frames: ​ LLMMessagesFrame Frame Enhanced messages with relevant memories included ​ OpenAILLMContextFrame Frame Enhanced OpenAI context with memories included ​ ErrorFrame Frame Contains error information if memory operations fail ​ Memory Operations The service performs two main operations automatically: ​ Message Storage All conversation messages are stored in Mem0 for future reference. The service: Captures full message history from context frames Associates messages with the specified user/agent/run IDs Stores metadata to enable efficient retrieval ​ Memory Retrieval When a new user message is detected, the service: Uses the message as a search query Retrieves relevant past memories from Mem0 Formats memories with the configured system prompt Adds the formatted memories to the conversation context Passes the enhanced context downstream in the pipeline ​ Pipeline Positioning The memory service should be positioned after the user context aggregator but before the LLM service: Copy Ask AI context_aggregator.user() → memory_service → llm This ensures that: The user’s latest message is included in the context The memory service can enhance the context before the LLM processes it The LLM receives the enhanced context with relevant memories ​ Usage Examples ​ Basic Integration Copy Ask AI from pipecat.services.mem0.memory import Mem0MemoryService from pipecat.pipeline.pipeline import Pipeline # Create the memory service memory = Mem0MemoryService( api_key = os.getenv( "MEM0_API_KEY" ), user_id = "user123" , # Unique user identifier ) # Position the memory service between context aggregator and LLM pipeline = Pipeline([ transport.input(), context_aggregator.user(), memory, # <-- Memory service enhances context here llm, tts, transport.output(), context_aggregator.assistant() ]) ​ Using Local Configuration The local_config parameter allows you to use your own LLM and embedding providers instead of Mem0’s cloud API. This is useful for self-hosted deployments or when you want more control over the memory processing. Copy Ask AI local_config = { "llm" : { "provider" : str , # LLM provider name (e.g., "anthropic", "openai") "config" : { # Provider-specific configuration "model" : str , # Model name "api_key" : str , # API key for the provider # Other provider-specific parameters } }, "embedder" : { "provider" : str , # Embedding provider name (e.g., "openai") "config" : { # Provider-specific configuration "model" : str , # Model name # Other provider-specific parameters } } } # Initialize Mem0 memory service with local configuration memory = Mem0MemoryService( local_config = local_config, # Use local LLM for memory processing user_id = "user123" , # Unique identifier for the user ) When using local_config do not provide the api_key parameter. ​ Frame Flow ​ Error Handling The service includes basic error handling to ensure conversation flow continues even when memory operations fail: Exceptions during memory storage and retrieval are caught and logged If an error occurs during frame processing, an ErrorFrame is emitted with error details The original frame is still passed downstream to prevent the pipeline from stalling Connection and authentication errors from the Mem0 API will be logged but won’t interrupt the conversation While the service attempts to handle errors gracefully, memory operations that fail may result in missing context in conversations. Monitor your application logs for memory-related errors. Tavus Moondream On this page Overview Installation Mem0MemoryService Constructor Parameters Input Parameters Input Frames Output Frames Memory Operations Message Storage Memory Retrieval Pipeline Positioning Usage Examples Basic Integration Using Local Configuration Frame Flow Error Handling Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/memory_mem0_3f19f368.txt b/memory_mem0_3f19f368.txt
new file mode 100644
index 0000000000000000000000000000000000000000..4dd13810772e308ff35b3de20cc37d87a40cd4e4
--- /dev/null
+++ b/memory_mem0_3f19f368.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/memory/mem0
+Title: Mem0 - Pipecat
+==================================================
+
+Mem0 - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Memory Mem0 Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Mem0 Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview Mem0MemoryService provides long-term memory capabilities for conversational agents by integrating with Mem0’s API. It automatically stores conversation history and retrieves relevant past context based on the current conversation, enhancing LLM responses with persistent memory across sessions. ​ Installation To use the Mem0 memory service, install the required dependencies: Copy Ask AI pip install "pipecat-ai[mem0]" You’ll also need to set up your Mem0 API key as an environment variable: MEM0_API_KEY . You can obtain a Mem0 API key by signing up at mem0.ai . ​ Mem0MemoryService ​ Constructor Parameters ​ api_key str required Mem0 API key for accessing the service ​ user_id str Unique identifier for the end user to associate with memories ​ agent_id str Identifier for the agent using the memory service ​ run_id str Identifier for the specific conversation session ​ params InputParams Configuration parameters for memory retrieval (see below) ​ local_config dict Configuration for using local LLMs and embedders instead of Mem0’s cloud API (see Local Configuration section) At least one of user_id , agent_id , or run_id must be provided to organize memories. ​ Input Parameters The params object accepts the following configuration settings: ​ search_limit int default: "10" Maximum number of relevant memories to retrieve per query ​ search_threshold float default: "0.1" Relevance threshold for memory retrieval (0.0 to 1.0) ​ api_version str default: "v2" Mem0 API version to use ​ system_prompt str Prefix text to add before retrieved memories ​ add_as_system_message bool default: "True" Whether to add memories as a system message (True) or user message (False) ​ position int default: "1" Position in the context where memories should be inserted ​ Input Frames The service processes the following input frames: ​ OpenAILLMContextFrame Frame Contains OpenAI-specific conversation context ​ LLMMessagesFrame Frame Contains conversation messages in standard format ​ Output Frames The service may produce the following output frames: ​ LLMMessagesFrame Frame Enhanced messages with relevant memories included ​ OpenAILLMContextFrame Frame Enhanced OpenAI context with memories included ​ ErrorFrame Frame Contains error information if memory operations fail ​ Memory Operations The service performs two main operations automatically: ​ Message Storage All conversation messages are stored in Mem0 for future reference. The service: Captures full message history from context frames Associates messages with the specified user/agent/run IDs Stores metadata to enable efficient retrieval ​ Memory Retrieval When a new user message is detected, the service: Uses the message as a search query Retrieves relevant past memories from Mem0 Formats memories with the configured system prompt Adds the formatted memories to the conversation context Passes the enhanced context downstream in the pipeline ​ Pipeline Positioning The memory service should be positioned after the user context aggregator but before the LLM service: Copy Ask AI context_aggregator.user() → memory_service → llm This ensures that: The user’s latest message is included in the context The memory service can enhance the context before the LLM processes it The LLM receives the enhanced context with relevant memories ​ Usage Examples ​ Basic Integration Copy Ask AI from pipecat.services.mem0.memory import Mem0MemoryService from pipecat.pipeline.pipeline import Pipeline # Create the memory service memory = Mem0MemoryService( api_key = os.getenv( "MEM0_API_KEY" ), user_id = "user123" , # Unique user identifier ) # Position the memory service between context aggregator and LLM pipeline = Pipeline([ transport.input(), context_aggregator.user(), memory, # <-- Memory service enhances context here llm, tts, transport.output(), context_aggregator.assistant() ]) ​ Using Local Configuration The local_config parameter allows you to use your own LLM and embedding providers instead of Mem0’s cloud API. This is useful for self-hosted deployments or when you want more control over the memory processing. Copy Ask AI local_config = { "llm" : { "provider" : str , # LLM provider name (e.g., "anthropic", "openai") "config" : { # Provider-specific configuration "model" : str , # Model name "api_key" : str , # API key for the provider # Other provider-specific parameters } }, "embedder" : { "provider" : str , # Embedding provider name (e.g., "openai") "config" : { # Provider-specific configuration "model" : str , # Model name # Other provider-specific parameters } } } # Initialize Mem0 memory service with local configuration memory = Mem0MemoryService( local_config = local_config, # Use local LLM for memory processing user_id = "user123" , # Unique identifier for the user ) When using local_config do not provide the api_key parameter. ​ Frame Flow ​ Error Handling The service includes basic error handling to ensure conversation flow continues even when memory operations fail: Exceptions during memory storage and retrieval are caught and logged If an error occurs during frame processing, an ErrorFrame is emitted with error details The original frame is still passed downstream to prevent the pipeline from stalling Connection and authentication errors from the Mem0 API will be logged but won’t interrupt the conversation While the service attempts to handle errors gracefully, memory operations that fail may result in missing context in conversations. Monitor your application logs for memory-related errors. Tavus Moondream On this page Overview Installation Mem0MemoryService Constructor Parameters Input Parameters Input Frames Output Frames Memory Operations Message Storage Memory Retrieval Pipeline Positioning Usage Examples Basic Integration Using Local Configuration Frame Flow Error Handling Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/memory_mem0_4f441906.txt b/memory_mem0_4f441906.txt
new file mode 100644
index 0000000000000000000000000000000000000000..47864af48696bc0b0d7346b77e7f99ff69c4bf5b
--- /dev/null
+++ b/memory_mem0_4f441906.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/memory/mem0#usage-examples
+Title: Mem0 - Pipecat
+==================================================
+
+Mem0 - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Memory Mem0 Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Mem0 Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview Mem0MemoryService provides long-term memory capabilities for conversational agents by integrating with Mem0’s API. It automatically stores conversation history and retrieves relevant past context based on the current conversation, enhancing LLM responses with persistent memory across sessions. ​ Installation To use the Mem0 memory service, install the required dependencies: Copy Ask AI pip install "pipecat-ai[mem0]" You’ll also need to set up your Mem0 API key as an environment variable: MEM0_API_KEY . You can obtain a Mem0 API key by signing up at mem0.ai . ​ Mem0MemoryService ​ Constructor Parameters ​ api_key str required Mem0 API key for accessing the service ​ user_id str Unique identifier for the end user to associate with memories ​ agent_id str Identifier for the agent using the memory service ​ run_id str Identifier for the specific conversation session ​ params InputParams Configuration parameters for memory retrieval (see below) ​ local_config dict Configuration for using local LLMs and embedders instead of Mem0’s cloud API (see Local Configuration section) At least one of user_id , agent_id , or run_id must be provided to organize memories. ​ Input Parameters The params object accepts the following configuration settings: ​ search_limit int default: "10" Maximum number of relevant memories to retrieve per query ​ search_threshold float default: "0.1" Relevance threshold for memory retrieval (0.0 to 1.0) ​ api_version str default: "v2" Mem0 API version to use ​ system_prompt str Prefix text to add before retrieved memories ​ add_as_system_message bool default: "True" Whether to add memories as a system message (True) or user message (False) ​ position int default: "1" Position in the context where memories should be inserted ​ Input Frames The service processes the following input frames: ​ OpenAILLMContextFrame Frame Contains OpenAI-specific conversation context ​ LLMMessagesFrame Frame Contains conversation messages in standard format ​ Output Frames The service may produce the following output frames: ​ LLMMessagesFrame Frame Enhanced messages with relevant memories included ​ OpenAILLMContextFrame Frame Enhanced OpenAI context with memories included ​ ErrorFrame Frame Contains error information if memory operations fail ​ Memory Operations The service performs two main operations automatically: ​ Message Storage All conversation messages are stored in Mem0 for future reference. The service: Captures full message history from context frames Associates messages with the specified user/agent/run IDs Stores metadata to enable efficient retrieval ​ Memory Retrieval When a new user message is detected, the service: Uses the message as a search query Retrieves relevant past memories from Mem0 Formats memories with the configured system prompt Adds the formatted memories to the conversation context Passes the enhanced context downstream in the pipeline ​ Pipeline Positioning The memory service should be positioned after the user context aggregator but before the LLM service: Copy Ask AI context_aggregator.user() → memory_service → llm This ensures that: The user’s latest message is included in the context The memory service can enhance the context before the LLM processes it The LLM receives the enhanced context with relevant memories ​ Usage Examples ​ Basic Integration Copy Ask AI from pipecat.services.mem0.memory import Mem0MemoryService from pipecat.pipeline.pipeline import Pipeline # Create the memory service memory = Mem0MemoryService( api_key = os.getenv( "MEM0_API_KEY" ), user_id = "user123" , # Unique user identifier ) # Position the memory service between context aggregator and LLM pipeline = Pipeline([ transport.input(), context_aggregator.user(), memory, # <-- Memory service enhances context here llm, tts, transport.output(), context_aggregator.assistant() ]) ​ Using Local Configuration The local_config parameter allows you to use your own LLM and embedding providers instead of Mem0’s cloud API. This is useful for self-hosted deployments or when you want more control over the memory processing. Copy Ask AI local_config = { "llm" : { "provider" : str , # LLM provider name (e.g., "anthropic", "openai") "config" : { # Provider-specific configuration "model" : str , # Model name "api_key" : str , # API key for the provider # Other provider-specific parameters } }, "embedder" : { "provider" : str , # Embedding provider name (e.g., "openai") "config" : { # Provider-specific configuration "model" : str , # Model name # Other provider-specific parameters } } } # Initialize Mem0 memory service with local configuration memory = Mem0MemoryService( local_config = local_config, # Use local LLM for memory processing user_id = "user123" , # Unique identifier for the user ) When using local_config do not provide the api_key parameter. ​ Frame Flow ​ Error Handling The service includes basic error handling to ensure conversation flow continues even when memory operations fail: Exceptions during memory storage and retrieval are caught and logged If an error occurs during frame processing, an ErrorFrame is emitted with error details The original frame is still passed downstream to prevent the pipeline from stalling Connection and authentication errors from the Mem0 API will be logged but won’t interrupt the conversation While the service attempts to handle errors gracefully, memory operations that fail may result in missing context in conversations. Monitor your application logs for memory-related errors. Tavus Moondream On this page Overview Installation Mem0MemoryService Constructor Parameters Input Parameters Input Frames Output Frames Memory Operations Message Storage Memory Retrieval Pipeline Positioning Usage Examples Basic Integration Using Local Configuration Frame Flow Error Handling Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/memory_mem0_594927f2.txt b/memory_mem0_594927f2.txt
new file mode 100644
index 0000000000000000000000000000000000000000..5c899c1a22415b1d4fbfa901210bc797365683ae
--- /dev/null
+++ b/memory_mem0_594927f2.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/memory/mem0#param-open-aillm-context-frame-1
+Title: Mem0 - Pipecat
+==================================================
+
+Mem0 - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Memory Mem0 Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Mem0 Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview Mem0MemoryService provides long-term memory capabilities for conversational agents by integrating with Mem0’s API. It automatically stores conversation history and retrieves relevant past context based on the current conversation, enhancing LLM responses with persistent memory across sessions. ​ Installation To use the Mem0 memory service, install the required dependencies: Copy Ask AI pip install "pipecat-ai[mem0]" You’ll also need to set up your Mem0 API key as an environment variable: MEM0_API_KEY . You can obtain a Mem0 API key by signing up at mem0.ai . ​ Mem0MemoryService ​ Constructor Parameters ​ api_key str required Mem0 API key for accessing the service ​ user_id str Unique identifier for the end user to associate with memories ​ agent_id str Identifier for the agent using the memory service ​ run_id str Identifier for the specific conversation session ​ params InputParams Configuration parameters for memory retrieval (see below) ​ local_config dict Configuration for using local LLMs and embedders instead of Mem0’s cloud API (see Local Configuration section) At least one of user_id , agent_id , or run_id must be provided to organize memories. ​ Input Parameters The params object accepts the following configuration settings: ​ search_limit int default: "10" Maximum number of relevant memories to retrieve per query ​ search_threshold float default: "0.1" Relevance threshold for memory retrieval (0.0 to 1.0) ​ api_version str default: "v2" Mem0 API version to use ​ system_prompt str Prefix text to add before retrieved memories ​ add_as_system_message bool default: "True" Whether to add memories as a system message (True) or user message (False) ​ position int default: "1" Position in the context where memories should be inserted ​ Input Frames The service processes the following input frames: ​ OpenAILLMContextFrame Frame Contains OpenAI-specific conversation context ​ LLMMessagesFrame Frame Contains conversation messages in standard format ​ Output Frames The service may produce the following output frames: ​ LLMMessagesFrame Frame Enhanced messages with relevant memories included ​ OpenAILLMContextFrame Frame Enhanced OpenAI context with memories included ​ ErrorFrame Frame Contains error information if memory operations fail ​ Memory Operations The service performs two main operations automatically: ​ Message Storage All conversation messages are stored in Mem0 for future reference. The service: Captures full message history from context frames Associates messages with the specified user/agent/run IDs Stores metadata to enable efficient retrieval ​ Memory Retrieval When a new user message is detected, the service: Uses the message as a search query Retrieves relevant past memories from Mem0 Formats memories with the configured system prompt Adds the formatted memories to the conversation context Passes the enhanced context downstream in the pipeline ​ Pipeline Positioning The memory service should be positioned after the user context aggregator but before the LLM service: Copy Ask AI context_aggregator.user() → memory_service → llm This ensures that: The user’s latest message is included in the context The memory service can enhance the context before the LLM processes it The LLM receives the enhanced context with relevant memories ​ Usage Examples ​ Basic Integration Copy Ask AI from pipecat.services.mem0.memory import Mem0MemoryService from pipecat.pipeline.pipeline import Pipeline # Create the memory service memory = Mem0MemoryService( api_key = os.getenv( "MEM0_API_KEY" ), user_id = "user123" , # Unique user identifier ) # Position the memory service between context aggregator and LLM pipeline = Pipeline([ transport.input(), context_aggregator.user(), memory, # <-- Memory service enhances context here llm, tts, transport.output(), context_aggregator.assistant() ]) ​ Using Local Configuration The local_config parameter allows you to use your own LLM and embedding providers instead of Mem0’s cloud API. This is useful for self-hosted deployments or when you want more control over the memory processing. Copy Ask AI local_config = { "llm" : { "provider" : str , # LLM provider name (e.g., "anthropic", "openai") "config" : { # Provider-specific configuration "model" : str , # Model name "api_key" : str , # API key for the provider # Other provider-specific parameters } }, "embedder" : { "provider" : str , # Embedding provider name (e.g., "openai") "config" : { # Provider-specific configuration "model" : str , # Model name # Other provider-specific parameters } } } # Initialize Mem0 memory service with local configuration memory = Mem0MemoryService( local_config = local_config, # Use local LLM for memory processing user_id = "user123" , # Unique identifier for the user ) When using local_config do not provide the api_key parameter. ​ Frame Flow ​ Error Handling The service includes basic error handling to ensure conversation flow continues even when memory operations fail: Exceptions during memory storage and retrieval are caught and logged If an error occurs during frame processing, an ErrorFrame is emitted with error details The original frame is still passed downstream to prevent the pipeline from stalling Connection and authentication errors from the Mem0 API will be logged but won’t interrupt the conversation While the service attempts to handle errors gracefully, memory operations that fail may result in missing context in conversations. Monitor your application logs for memory-related errors. Tavus Moondream On this page Overview Installation Mem0MemoryService Constructor Parameters Input Parameters Input Frames Output Frames Memory Operations Message Storage Memory Retrieval Pipeline Positioning Usage Examples Basic Integration Using Local Configuration Frame Flow Error Handling Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/memory_mem0_71cacf22.txt b/memory_mem0_71cacf22.txt
new file mode 100644
index 0000000000000000000000000000000000000000..57ca5b29b3872d4f4530e35882efd65c0726a3ca
--- /dev/null
+++ b/memory_mem0_71cacf22.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/memory/mem0#memory-operations
+Title: Mem0 - Pipecat
+==================================================
+
+Mem0 - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Memory Mem0 Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Mem0 Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview Mem0MemoryService provides long-term memory capabilities for conversational agents by integrating with Mem0’s API. It automatically stores conversation history and retrieves relevant past context based on the current conversation, enhancing LLM responses with persistent memory across sessions. ​ Installation To use the Mem0 memory service, install the required dependencies: Copy Ask AI pip install "pipecat-ai[mem0]" You’ll also need to set up your Mem0 API key as an environment variable: MEM0_API_KEY . You can obtain a Mem0 API key by signing up at mem0.ai . ​ Mem0MemoryService ​ Constructor Parameters ​ api_key str required Mem0 API key for accessing the service ​ user_id str Unique identifier for the end user to associate with memories ​ agent_id str Identifier for the agent using the memory service ​ run_id str Identifier for the specific conversation session ​ params InputParams Configuration parameters for memory retrieval (see below) ​ local_config dict Configuration for using local LLMs and embedders instead of Mem0’s cloud API (see Local Configuration section) At least one of user_id , agent_id , or run_id must be provided to organize memories. ​ Input Parameters The params object accepts the following configuration settings: ​ search_limit int default: "10" Maximum number of relevant memories to retrieve per query ​ search_threshold float default: "0.1" Relevance threshold for memory retrieval (0.0 to 1.0) ​ api_version str default: "v2" Mem0 API version to use ​ system_prompt str Prefix text to add before retrieved memories ​ add_as_system_message bool default: "True" Whether to add memories as a system message (True) or user message (False) ​ position int default: "1" Position in the context where memories should be inserted ​ Input Frames The service processes the following input frames: ​ OpenAILLMContextFrame Frame Contains OpenAI-specific conversation context ​ LLMMessagesFrame Frame Contains conversation messages in standard format ​ Output Frames The service may produce the following output frames: ​ LLMMessagesFrame Frame Enhanced messages with relevant memories included ​ OpenAILLMContextFrame Frame Enhanced OpenAI context with memories included ​ ErrorFrame Frame Contains error information if memory operations fail ​ Memory Operations The service performs two main operations automatically: ​ Message Storage All conversation messages are stored in Mem0 for future reference. The service: Captures full message history from context frames Associates messages with the specified user/agent/run IDs Stores metadata to enable efficient retrieval ​ Memory Retrieval When a new user message is detected, the service: Uses the message as a search query Retrieves relevant past memories from Mem0 Formats memories with the configured system prompt Adds the formatted memories to the conversation context Passes the enhanced context downstream in the pipeline ​ Pipeline Positioning The memory service should be positioned after the user context aggregator but before the LLM service: Copy Ask AI context_aggregator.user() → memory_service → llm This ensures that: The user’s latest message is included in the context The memory service can enhance the context before the LLM processes it The LLM receives the enhanced context with relevant memories ​ Usage Examples ​ Basic Integration Copy Ask AI from pipecat.services.mem0.memory import Mem0MemoryService from pipecat.pipeline.pipeline import Pipeline # Create the memory service memory = Mem0MemoryService( api_key = os.getenv( "MEM0_API_KEY" ), user_id = "user123" , # Unique user identifier ) # Position the memory service between context aggregator and LLM pipeline = Pipeline([ transport.input(), context_aggregator.user(), memory, # <-- Memory service enhances context here llm, tts, transport.output(), context_aggregator.assistant() ]) ​ Using Local Configuration The local_config parameter allows you to use your own LLM and embedding providers instead of Mem0’s cloud API. This is useful for self-hosted deployments or when you want more control over the memory processing. Copy Ask AI local_config = { "llm" : { "provider" : str , # LLM provider name (e.g., "anthropic", "openai") "config" : { # Provider-specific configuration "model" : str , # Model name "api_key" : str , # API key for the provider # Other provider-specific parameters } }, "embedder" : { "provider" : str , # Embedding provider name (e.g., "openai") "config" : { # Provider-specific configuration "model" : str , # Model name # Other provider-specific parameters } } } # Initialize Mem0 memory service with local configuration memory = Mem0MemoryService( local_config = local_config, # Use local LLM for memory processing user_id = "user123" , # Unique identifier for the user ) When using local_config do not provide the api_key parameter. ​ Frame Flow ​ Error Handling The service includes basic error handling to ensure conversation flow continues even when memory operations fail: Exceptions during memory storage and retrieval are caught and logged If an error occurs during frame processing, an ErrorFrame is emitted with error details The original frame is still passed downstream to prevent the pipeline from stalling Connection and authentication errors from the Mem0 API will be logged but won’t interrupt the conversation While the service attempts to handle errors gracefully, memory operations that fail may result in missing context in conversations. Monitor your application logs for memory-related errors. Tavus Moondream On this page Overview Installation Mem0MemoryService Constructor Parameters Input Parameters Input Frames Output Frames Memory Operations Message Storage Memory Retrieval Pipeline Positioning Usage Examples Basic Integration Using Local Configuration Frame Flow Error Handling Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/memory_mem0_9db48249.txt b/memory_mem0_9db48249.txt
new file mode 100644
index 0000000000000000000000000000000000000000..9bff60cd0bd76285717118312d5ba43a8f55d80d
--- /dev/null
+++ b/memory_mem0_9db48249.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/memory/mem0#pipeline-positioning
+Title: Mem0 - Pipecat
+==================================================
+
+Mem0 - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Memory Mem0 Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Mem0 Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview Mem0MemoryService provides long-term memory capabilities for conversational agents by integrating with Mem0’s API. It automatically stores conversation history and retrieves relevant past context based on the current conversation, enhancing LLM responses with persistent memory across sessions. ​ Installation To use the Mem0 memory service, install the required dependencies: Copy Ask AI pip install "pipecat-ai[mem0]" You’ll also need to set up your Mem0 API key as an environment variable: MEM0_API_KEY . You can obtain a Mem0 API key by signing up at mem0.ai . ​ Mem0MemoryService ​ Constructor Parameters ​ api_key str required Mem0 API key for accessing the service ​ user_id str Unique identifier for the end user to associate with memories ​ agent_id str Identifier for the agent using the memory service ​ run_id str Identifier for the specific conversation session ​ params InputParams Configuration parameters for memory retrieval (see below) ​ local_config dict Configuration for using local LLMs and embedders instead of Mem0’s cloud API (see Local Configuration section) At least one of user_id , agent_id , or run_id must be provided to organize memories. ​ Input Parameters The params object accepts the following configuration settings: ​ search_limit int default: "10" Maximum number of relevant memories to retrieve per query ​ search_threshold float default: "0.1" Relevance threshold for memory retrieval (0.0 to 1.0) ​ api_version str default: "v2" Mem0 API version to use ​ system_prompt str Prefix text to add before retrieved memories ​ add_as_system_message bool default: "True" Whether to add memories as a system message (True) or user message (False) ​ position int default: "1" Position in the context where memories should be inserted ​ Input Frames The service processes the following input frames: ​ OpenAILLMContextFrame Frame Contains OpenAI-specific conversation context ​ LLMMessagesFrame Frame Contains conversation messages in standard format ​ Output Frames The service may produce the following output frames: ​ LLMMessagesFrame Frame Enhanced messages with relevant memories included ​ OpenAILLMContextFrame Frame Enhanced OpenAI context with memories included ​ ErrorFrame Frame Contains error information if memory operations fail ​ Memory Operations The service performs two main operations automatically: ​ Message Storage All conversation messages are stored in Mem0 for future reference. The service: Captures full message history from context frames Associates messages with the specified user/agent/run IDs Stores metadata to enable efficient retrieval ​ Memory Retrieval When a new user message is detected, the service: Uses the message as a search query Retrieves relevant past memories from Mem0 Formats memories with the configured system prompt Adds the formatted memories to the conversation context Passes the enhanced context downstream in the pipeline ​ Pipeline Positioning The memory service should be positioned after the user context aggregator but before the LLM service: Copy Ask AI context_aggregator.user() → memory_service → llm This ensures that: The user’s latest message is included in the context The memory service can enhance the context before the LLM processes it The LLM receives the enhanced context with relevant memories ​ Usage Examples ​ Basic Integration Copy Ask AI from pipecat.services.mem0.memory import Mem0MemoryService from pipecat.pipeline.pipeline import Pipeline # Create the memory service memory = Mem0MemoryService( api_key = os.getenv( "MEM0_API_KEY" ), user_id = "user123" , # Unique user identifier ) # Position the memory service between context aggregator and LLM pipeline = Pipeline([ transport.input(), context_aggregator.user(), memory, # <-- Memory service enhances context here llm, tts, transport.output(), context_aggregator.assistant() ]) ​ Using Local Configuration The local_config parameter allows you to use your own LLM and embedding providers instead of Mem0’s cloud API. This is useful for self-hosted deployments or when you want more control over the memory processing. Copy Ask AI local_config = { "llm" : { "provider" : str , # LLM provider name (e.g., "anthropic", "openai") "config" : { # Provider-specific configuration "model" : str , # Model name "api_key" : str , # API key for the provider # Other provider-specific parameters } }, "embedder" : { "provider" : str , # Embedding provider name (e.g., "openai") "config" : { # Provider-specific configuration "model" : str , # Model name # Other provider-specific parameters } } } # Initialize Mem0 memory service with local configuration memory = Mem0MemoryService( local_config = local_config, # Use local LLM for memory processing user_id = "user123" , # Unique identifier for the user ) When using local_config do not provide the api_key parameter. ​ Frame Flow ​ Error Handling The service includes basic error handling to ensure conversation flow continues even when memory operations fail: Exceptions during memory storage and retrieval are caught and logged If an error occurs during frame processing, an ErrorFrame is emitted with error details The original frame is still passed downstream to prevent the pipeline from stalling Connection and authentication errors from the Mem0 API will be logged but won’t interrupt the conversation While the service attempts to handle errors gracefully, memory operations that fail may result in missing context in conversations. Monitor your application logs for memory-related errors. Tavus Moondream On this page Overview Installation Mem0MemoryService Constructor Parameters Input Parameters Input Frames Output Frames Memory Operations Message Storage Memory Retrieval Pipeline Positioning Usage Examples Basic Integration Using Local Configuration Frame Flow Error Handling Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/memory_mem0_b90d74f3.txt b/memory_mem0_b90d74f3.txt
new file mode 100644
index 0000000000000000000000000000000000000000..acefed134993e8de302f7d3aef80a603aaa99d48
--- /dev/null
+++ b/memory_mem0_b90d74f3.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/memory/mem0#memory-retrieval
+Title: Mem0 - Pipecat
+==================================================
+
+Mem0 - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Memory Mem0 Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Mem0 Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview Mem0MemoryService provides long-term memory capabilities for conversational agents by integrating with Mem0’s API. It automatically stores conversation history and retrieves relevant past context based on the current conversation, enhancing LLM responses with persistent memory across sessions. ​ Installation To use the Mem0 memory service, install the required dependencies: Copy Ask AI pip install "pipecat-ai[mem0]" You’ll also need to set up your Mem0 API key as an environment variable: MEM0_API_KEY . You can obtain a Mem0 API key by signing up at mem0.ai . ​ Mem0MemoryService ​ Constructor Parameters ​ api_key str required Mem0 API key for accessing the service ​ user_id str Unique identifier for the end user to associate with memories ​ agent_id str Identifier for the agent using the memory service ​ run_id str Identifier for the specific conversation session ​ params InputParams Configuration parameters for memory retrieval (see below) ​ local_config dict Configuration for using local LLMs and embedders instead of Mem0’s cloud API (see Local Configuration section) At least one of user_id , agent_id , or run_id must be provided to organize memories. ​ Input Parameters The params object accepts the following configuration settings: ​ search_limit int default: "10" Maximum number of relevant memories to retrieve per query ​ search_threshold float default: "0.1" Relevance threshold for memory retrieval (0.0 to 1.0) ​ api_version str default: "v2" Mem0 API version to use ​ system_prompt str Prefix text to add before retrieved memories ​ add_as_system_message bool default: "True" Whether to add memories as a system message (True) or user message (False) ​ position int default: "1" Position in the context where memories should be inserted ​ Input Frames The service processes the following input frames: ​ OpenAILLMContextFrame Frame Contains OpenAI-specific conversation context ​ LLMMessagesFrame Frame Contains conversation messages in standard format ​ Output Frames The service may produce the following output frames: ​ LLMMessagesFrame Frame Enhanced messages with relevant memories included ​ OpenAILLMContextFrame Frame Enhanced OpenAI context with memories included ​ ErrorFrame Frame Contains error information if memory operations fail ​ Memory Operations The service performs two main operations automatically: ​ Message Storage All conversation messages are stored in Mem0 for future reference. The service: Captures full message history from context frames Associates messages with the specified user/agent/run IDs Stores metadata to enable efficient retrieval ​ Memory Retrieval When a new user message is detected, the service: Uses the message as a search query Retrieves relevant past memories from Mem0 Formats memories with the configured system prompt Adds the formatted memories to the conversation context Passes the enhanced context downstream in the pipeline ​ Pipeline Positioning The memory service should be positioned after the user context aggregator but before the LLM service: Copy Ask AI context_aggregator.user() → memory_service → llm This ensures that: The user’s latest message is included in the context The memory service can enhance the context before the LLM processes it The LLM receives the enhanced context with relevant memories ​ Usage Examples ​ Basic Integration Copy Ask AI from pipecat.services.mem0.memory import Mem0MemoryService from pipecat.pipeline.pipeline import Pipeline # Create the memory service memory = Mem0MemoryService( api_key = os.getenv( "MEM0_API_KEY" ), user_id = "user123" , # Unique user identifier ) # Position the memory service between context aggregator and LLM pipeline = Pipeline([ transport.input(), context_aggregator.user(), memory, # <-- Memory service enhances context here llm, tts, transport.output(), context_aggregator.assistant() ]) ​ Using Local Configuration The local_config parameter allows you to use your own LLM and embedding providers instead of Mem0’s cloud API. This is useful for self-hosted deployments or when you want more control over the memory processing. Copy Ask AI local_config = { "llm" : { "provider" : str , # LLM provider name (e.g., "anthropic", "openai") "config" : { # Provider-specific configuration "model" : str , # Model name "api_key" : str , # API key for the provider # Other provider-specific parameters } }, "embedder" : { "provider" : str , # Embedding provider name (e.g., "openai") "config" : { # Provider-specific configuration "model" : str , # Model name # Other provider-specific parameters } } } # Initialize Mem0 memory service with local configuration memory = Mem0MemoryService( local_config = local_config, # Use local LLM for memory processing user_id = "user123" , # Unique identifier for the user ) When using local_config do not provide the api_key parameter. ​ Frame Flow ​ Error Handling The service includes basic error handling to ensure conversation flow continues even when memory operations fail: Exceptions during memory storage and retrieval are caught and logged If an error occurs during frame processing, an ErrorFrame is emitted with error details The original frame is still passed downstream to prevent the pipeline from stalling Connection and authentication errors from the Mem0 API will be logged but won’t interrupt the conversation While the service attempts to handle errors gracefully, memory operations that fail may result in missing context in conversations. Monitor your application logs for memory-related errors. Tavus Moondream On this page Overview Installation Mem0MemoryService Constructor Parameters Input Parameters Input Frames Output Frames Memory Operations Message Storage Memory Retrieval Pipeline Positioning Usage Examples Basic Integration Using Local Configuration Frame Flow Error Handling Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/memory_mem0_cb147e82.txt b/memory_mem0_cb147e82.txt
new file mode 100644
index 0000000000000000000000000000000000000000..9a27cef1266b1ac4d0dfa0ce26a7fd4057e0c507
--- /dev/null
+++ b/memory_mem0_cb147e82.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/memory/mem0#param-local-config
+Title: Mem0 - Pipecat
+==================================================
+
+Mem0 - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Memory Mem0 Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Mem0 Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview Mem0MemoryService provides long-term memory capabilities for conversational agents by integrating with Mem0’s API. It automatically stores conversation history and retrieves relevant past context based on the current conversation, enhancing LLM responses with persistent memory across sessions. ​ Installation To use the Mem0 memory service, install the required dependencies: Copy Ask AI pip install "pipecat-ai[mem0]" You’ll also need to set up your Mem0 API key as an environment variable: MEM0_API_KEY . You can obtain a Mem0 API key by signing up at mem0.ai . ​ Mem0MemoryService ​ Constructor Parameters ​ api_key str required Mem0 API key for accessing the service ​ user_id str Unique identifier for the end user to associate with memories ​ agent_id str Identifier for the agent using the memory service ​ run_id str Identifier for the specific conversation session ​ params InputParams Configuration parameters for memory retrieval (see below) ​ local_config dict Configuration for using local LLMs and embedders instead of Mem0’s cloud API (see Local Configuration section) At least one of user_id , agent_id , or run_id must be provided to organize memories. ​ Input Parameters The params object accepts the following configuration settings: ​ search_limit int default: "10" Maximum number of relevant memories to retrieve per query ​ search_threshold float default: "0.1" Relevance threshold for memory retrieval (0.0 to 1.0) ​ api_version str default: "v2" Mem0 API version to use ​ system_prompt str Prefix text to add before retrieved memories ​ add_as_system_message bool default: "True" Whether to add memories as a system message (True) or user message (False) ​ position int default: "1" Position in the context where memories should be inserted ​ Input Frames The service processes the following input frames: ​ OpenAILLMContextFrame Frame Contains OpenAI-specific conversation context ​ LLMMessagesFrame Frame Contains conversation messages in standard format ​ Output Frames The service may produce the following output frames: ​ LLMMessagesFrame Frame Enhanced messages with relevant memories included ​ OpenAILLMContextFrame Frame Enhanced OpenAI context with memories included ​ ErrorFrame Frame Contains error information if memory operations fail ​ Memory Operations The service performs two main operations automatically: ​ Message Storage All conversation messages are stored in Mem0 for future reference. The service: Captures full message history from context frames Associates messages with the specified user/agent/run IDs Stores metadata to enable efficient retrieval ​ Memory Retrieval When a new user message is detected, the service: Uses the message as a search query Retrieves relevant past memories from Mem0 Formats memories with the configured system prompt Adds the formatted memories to the conversation context Passes the enhanced context downstream in the pipeline ​ Pipeline Positioning The memory service should be positioned after the user context aggregator but before the LLM service: Copy Ask AI context_aggregator.user() → memory_service → llm This ensures that: The user’s latest message is included in the context The memory service can enhance the context before the LLM processes it The LLM receives the enhanced context with relevant memories ​ Usage Examples ​ Basic Integration Copy Ask AI from pipecat.services.mem0.memory import Mem0MemoryService from pipecat.pipeline.pipeline import Pipeline # Create the memory service memory = Mem0MemoryService( api_key = os.getenv( "MEM0_API_KEY" ), user_id = "user123" , # Unique user identifier ) # Position the memory service between context aggregator and LLM pipeline = Pipeline([ transport.input(), context_aggregator.user(), memory, # <-- Memory service enhances context here llm, tts, transport.output(), context_aggregator.assistant() ]) ​ Using Local Configuration The local_config parameter allows you to use your own LLM and embedding providers instead of Mem0’s cloud API. This is useful for self-hosted deployments or when you want more control over the memory processing. Copy Ask AI local_config = { "llm" : { "provider" : str , # LLM provider name (e.g., "anthropic", "openai") "config" : { # Provider-specific configuration "model" : str , # Model name "api_key" : str , # API key for the provider # Other provider-specific parameters } }, "embedder" : { "provider" : str , # Embedding provider name (e.g., "openai") "config" : { # Provider-specific configuration "model" : str , # Model name # Other provider-specific parameters } } } # Initialize Mem0 memory service with local configuration memory = Mem0MemoryService( local_config = local_config, # Use local LLM for memory processing user_id = "user123" , # Unique identifier for the user ) When using local_config do not provide the api_key parameter. ​ Frame Flow ​ Error Handling The service includes basic error handling to ensure conversation flow continues even when memory operations fail: Exceptions during memory storage and retrieval are caught and logged If an error occurs during frame processing, an ErrorFrame is emitted with error details The original frame is still passed downstream to prevent the pipeline from stalling Connection and authentication errors from the Mem0 API will be logged but won’t interrupt the conversation While the service attempts to handle errors gracefully, memory operations that fail may result in missing context in conversations. Monitor your application logs for memory-related errors. Tavus Moondream On this page Overview Installation Mem0MemoryService Constructor Parameters Input Parameters Input Frames Output Frames Memory Operations Message Storage Memory Retrieval Pipeline Positioning Usage Examples Basic Integration Using Local Configuration Frame Flow Error Handling Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/memory_mem0_cde96ce2.txt b/memory_mem0_cde96ce2.txt
new file mode 100644
index 0000000000000000000000000000000000000000..19d594de7a59283bb621f43611d0f74faf646066
--- /dev/null
+++ b/memory_mem0_cde96ce2.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/memory/mem0#param-open-aillm-context-frame
+Title: Mem0 - Pipecat
+==================================================
+
+Mem0 - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Memory Mem0 Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Mem0 Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview Mem0MemoryService provides long-term memory capabilities for conversational agents by integrating with Mem0’s API. It automatically stores conversation history and retrieves relevant past context based on the current conversation, enhancing LLM responses with persistent memory across sessions. ​ Installation To use the Mem0 memory service, install the required dependencies: Copy Ask AI pip install "pipecat-ai[mem0]" You’ll also need to set up your Mem0 API key as an environment variable: MEM0_API_KEY . You can obtain a Mem0 API key by signing up at mem0.ai . ​ Mem0MemoryService ​ Constructor Parameters ​ api_key str required Mem0 API key for accessing the service ​ user_id str Unique identifier for the end user to associate with memories ​ agent_id str Identifier for the agent using the memory service ​ run_id str Identifier for the specific conversation session ​ params InputParams Configuration parameters for memory retrieval (see below) ​ local_config dict Configuration for using local LLMs and embedders instead of Mem0’s cloud API (see Local Configuration section) At least one of user_id , agent_id , or run_id must be provided to organize memories. ​ Input Parameters The params object accepts the following configuration settings: ​ search_limit int default: "10" Maximum number of relevant memories to retrieve per query ​ search_threshold float default: "0.1" Relevance threshold for memory retrieval (0.0 to 1.0) ​ api_version str default: "v2" Mem0 API version to use ​ system_prompt str Prefix text to add before retrieved memories ​ add_as_system_message bool default: "True" Whether to add memories as a system message (True) or user message (False) ​ position int default: "1" Position in the context where memories should be inserted ​ Input Frames The service processes the following input frames: ​ OpenAILLMContextFrame Frame Contains OpenAI-specific conversation context ​ LLMMessagesFrame Frame Contains conversation messages in standard format ​ Output Frames The service may produce the following output frames: ​ LLMMessagesFrame Frame Enhanced messages with relevant memories included ​ OpenAILLMContextFrame Frame Enhanced OpenAI context with memories included ​ ErrorFrame Frame Contains error information if memory operations fail ​ Memory Operations The service performs two main operations automatically: ​ Message Storage All conversation messages are stored in Mem0 for future reference. The service: Captures full message history from context frames Associates messages with the specified user/agent/run IDs Stores metadata to enable efficient retrieval ​ Memory Retrieval When a new user message is detected, the service: Uses the message as a search query Retrieves relevant past memories from Mem0 Formats memories with the configured system prompt Adds the formatted memories to the conversation context Passes the enhanced context downstream in the pipeline ​ Pipeline Positioning The memory service should be positioned after the user context aggregator but before the LLM service: Copy Ask AI context_aggregator.user() → memory_service → llm This ensures that: The user’s latest message is included in the context The memory service can enhance the context before the LLM processes it The LLM receives the enhanced context with relevant memories ​ Usage Examples ​ Basic Integration Copy Ask AI from pipecat.services.mem0.memory import Mem0MemoryService from pipecat.pipeline.pipeline import Pipeline # Create the memory service memory = Mem0MemoryService( api_key = os.getenv( "MEM0_API_KEY" ), user_id = "user123" , # Unique user identifier ) # Position the memory service between context aggregator and LLM pipeline = Pipeline([ transport.input(), context_aggregator.user(), memory, # <-- Memory service enhances context here llm, tts, transport.output(), context_aggregator.assistant() ]) ​ Using Local Configuration The local_config parameter allows you to use your own LLM and embedding providers instead of Mem0’s cloud API. This is useful for self-hosted deployments or when you want more control over the memory processing. Copy Ask AI local_config = { "llm" : { "provider" : str , # LLM provider name (e.g., "anthropic", "openai") "config" : { # Provider-specific configuration "model" : str , # Model name "api_key" : str , # API key for the provider # Other provider-specific parameters } }, "embedder" : { "provider" : str , # Embedding provider name (e.g., "openai") "config" : { # Provider-specific configuration "model" : str , # Model name # Other provider-specific parameters } } } # Initialize Mem0 memory service with local configuration memory = Mem0MemoryService( local_config = local_config, # Use local LLM for memory processing user_id = "user123" , # Unique identifier for the user ) When using local_config do not provide the api_key parameter. ​ Frame Flow ​ Error Handling The service includes basic error handling to ensure conversation flow continues even when memory operations fail: Exceptions during memory storage and retrieval are caught and logged If an error occurs during frame processing, an ErrorFrame is emitted with error details The original frame is still passed downstream to prevent the pipeline from stalling Connection and authentication errors from the Mem0 API will be logged but won’t interrupt the conversation While the service attempts to handle errors gracefully, memory operations that fail may result in missing context in conversations. Monitor your application logs for memory-related errors. Tavus Moondream On this page Overview Installation Mem0MemoryService Constructor Parameters Input Parameters Input Frames Output Frames Memory Operations Message Storage Memory Retrieval Pipeline Positioning Usage Examples Basic Integration Using Local Configuration Frame Flow Error Handling Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/memory_mem0_d567930b.txt b/memory_mem0_d567930b.txt
new file mode 100644
index 0000000000000000000000000000000000000000..a25744215b60e9bf1f929ca2118fce36dc0bc618
--- /dev/null
+++ b/memory_mem0_d567930b.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/memory/mem0#output-frames
+Title: Mem0 - Pipecat
+==================================================
+
+Mem0 - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Memory Mem0 Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Mem0 Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview Mem0MemoryService provides long-term memory capabilities for conversational agents by integrating with Mem0’s API. It automatically stores conversation history and retrieves relevant past context based on the current conversation, enhancing LLM responses with persistent memory across sessions. ​ Installation To use the Mem0 memory service, install the required dependencies: Copy Ask AI pip install "pipecat-ai[mem0]" You’ll also need to set up your Mem0 API key as an environment variable: MEM0_API_KEY . You can obtain a Mem0 API key by signing up at mem0.ai . ​ Mem0MemoryService ​ Constructor Parameters ​ api_key str required Mem0 API key for accessing the service ​ user_id str Unique identifier for the end user to associate with memories ​ agent_id str Identifier for the agent using the memory service ​ run_id str Identifier for the specific conversation session ​ params InputParams Configuration parameters for memory retrieval (see below) ​ local_config dict Configuration for using local LLMs and embedders instead of Mem0’s cloud API (see Local Configuration section) At least one of user_id , agent_id , or run_id must be provided to organize memories. ​ Input Parameters The params object accepts the following configuration settings: ​ search_limit int default: "10" Maximum number of relevant memories to retrieve per query ​ search_threshold float default: "0.1" Relevance threshold for memory retrieval (0.0 to 1.0) ​ api_version str default: "v2" Mem0 API version to use ​ system_prompt str Prefix text to add before retrieved memories ​ add_as_system_message bool default: "True" Whether to add memories as a system message (True) or user message (False) ​ position int default: "1" Position in the context where memories should be inserted ​ Input Frames The service processes the following input frames: ​ OpenAILLMContextFrame Frame Contains OpenAI-specific conversation context ​ LLMMessagesFrame Frame Contains conversation messages in standard format ​ Output Frames The service may produce the following output frames: ​ LLMMessagesFrame Frame Enhanced messages with relevant memories included ​ OpenAILLMContextFrame Frame Enhanced OpenAI context with memories included ​ ErrorFrame Frame Contains error information if memory operations fail ​ Memory Operations The service performs two main operations automatically: ​ Message Storage All conversation messages are stored in Mem0 for future reference. The service: Captures full message history from context frames Associates messages with the specified user/agent/run IDs Stores metadata to enable efficient retrieval ​ Memory Retrieval When a new user message is detected, the service: Uses the message as a search query Retrieves relevant past memories from Mem0 Formats memories with the configured system prompt Adds the formatted memories to the conversation context Passes the enhanced context downstream in the pipeline ​ Pipeline Positioning The memory service should be positioned after the user context aggregator but before the LLM service: Copy Ask AI context_aggregator.user() → memory_service → llm This ensures that: The user’s latest message is included in the context The memory service can enhance the context before the LLM processes it The LLM receives the enhanced context with relevant memories ​ Usage Examples ​ Basic Integration Copy Ask AI from pipecat.services.mem0.memory import Mem0MemoryService from pipecat.pipeline.pipeline import Pipeline # Create the memory service memory = Mem0MemoryService( api_key = os.getenv( "MEM0_API_KEY" ), user_id = "user123" , # Unique user identifier ) # Position the memory service between context aggregator and LLM pipeline = Pipeline([ transport.input(), context_aggregator.user(), memory, # <-- Memory service enhances context here llm, tts, transport.output(), context_aggregator.assistant() ]) ​ Using Local Configuration The local_config parameter allows you to use your own LLM and embedding providers instead of Mem0’s cloud API. This is useful for self-hosted deployments or when you want more control over the memory processing. Copy Ask AI local_config = { "llm" : { "provider" : str , # LLM provider name (e.g., "anthropic", "openai") "config" : { # Provider-specific configuration "model" : str , # Model name "api_key" : str , # API key for the provider # Other provider-specific parameters } }, "embedder" : { "provider" : str , # Embedding provider name (e.g., "openai") "config" : { # Provider-specific configuration "model" : str , # Model name # Other provider-specific parameters } } } # Initialize Mem0 memory service with local configuration memory = Mem0MemoryService( local_config = local_config, # Use local LLM for memory processing user_id = "user123" , # Unique identifier for the user ) When using local_config do not provide the api_key parameter. ​ Frame Flow ​ Error Handling The service includes basic error handling to ensure conversation flow continues even when memory operations fail: Exceptions during memory storage and retrieval are caught and logged If an error occurs during frame processing, an ErrorFrame is emitted with error details The original frame is still passed downstream to prevent the pipeline from stalling Connection and authentication errors from the Mem0 API will be logged but won’t interrupt the conversation While the service attempts to handle errors gracefully, memory operations that fail may result in missing context in conversations. Monitor your application logs for memory-related errors. Tavus Moondream On this page Overview Installation Mem0MemoryService Constructor Parameters Input Parameters Input Frames Output Frames Memory Operations Message Storage Memory Retrieval Pipeline Positioning Usage Examples Basic Integration Using Local Configuration Frame Flow Error Handling Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/memory_mem0_e26ff14d.txt b/memory_mem0_e26ff14d.txt
new file mode 100644
index 0000000000000000000000000000000000000000..985906e1c24bcd0f55c8b368e7bafbdad41efdf3
--- /dev/null
+++ b/memory_mem0_e26ff14d.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/memory/mem0#overview
+Title: Mem0 - Pipecat
+==================================================
+
+Mem0 - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Memory Mem0 Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Mem0 Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview Mem0MemoryService provides long-term memory capabilities for conversational agents by integrating with Mem0’s API. It automatically stores conversation history and retrieves relevant past context based on the current conversation, enhancing LLM responses with persistent memory across sessions. ​ Installation To use the Mem0 memory service, install the required dependencies: Copy Ask AI pip install "pipecat-ai[mem0]" You’ll also need to set up your Mem0 API key as an environment variable: MEM0_API_KEY . You can obtain a Mem0 API key by signing up at mem0.ai . ​ Mem0MemoryService ​ Constructor Parameters ​ api_key str required Mem0 API key for accessing the service ​ user_id str Unique identifier for the end user to associate with memories ​ agent_id str Identifier for the agent using the memory service ​ run_id str Identifier for the specific conversation session ​ params InputParams Configuration parameters for memory retrieval (see below) ​ local_config dict Configuration for using local LLMs and embedders instead of Mem0’s cloud API (see Local Configuration section) At least one of user_id , agent_id , or run_id must be provided to organize memories. ​ Input Parameters The params object accepts the following configuration settings: ​ search_limit int default: "10" Maximum number of relevant memories to retrieve per query ​ search_threshold float default: "0.1" Relevance threshold for memory retrieval (0.0 to 1.0) ​ api_version str default: "v2" Mem0 API version to use ​ system_prompt str Prefix text to add before retrieved memories ​ add_as_system_message bool default: "True" Whether to add memories as a system message (True) or user message (False) ​ position int default: "1" Position in the context where memories should be inserted ​ Input Frames The service processes the following input frames: ​ OpenAILLMContextFrame Frame Contains OpenAI-specific conversation context ​ LLMMessagesFrame Frame Contains conversation messages in standard format ​ Output Frames The service may produce the following output frames: ​ LLMMessagesFrame Frame Enhanced messages with relevant memories included ​ OpenAILLMContextFrame Frame Enhanced OpenAI context with memories included ​ ErrorFrame Frame Contains error information if memory operations fail ​ Memory Operations The service performs two main operations automatically: ​ Message Storage All conversation messages are stored in Mem0 for future reference. The service: Captures full message history from context frames Associates messages with the specified user/agent/run IDs Stores metadata to enable efficient retrieval ​ Memory Retrieval When a new user message is detected, the service: Uses the message as a search query Retrieves relevant past memories from Mem0 Formats memories with the configured system prompt Adds the formatted memories to the conversation context Passes the enhanced context downstream in the pipeline ​ Pipeline Positioning The memory service should be positioned after the user context aggregator but before the LLM service: Copy Ask AI context_aggregator.user() → memory_service → llm This ensures that: The user’s latest message is included in the context The memory service can enhance the context before the LLM processes it The LLM receives the enhanced context with relevant memories ​ Usage Examples ​ Basic Integration Copy Ask AI from pipecat.services.mem0.memory import Mem0MemoryService from pipecat.pipeline.pipeline import Pipeline # Create the memory service memory = Mem0MemoryService( api_key = os.getenv( "MEM0_API_KEY" ), user_id = "user123" , # Unique user identifier ) # Position the memory service between context aggregator and LLM pipeline = Pipeline([ transport.input(), context_aggregator.user(), memory, # <-- Memory service enhances context here llm, tts, transport.output(), context_aggregator.assistant() ]) ​ Using Local Configuration The local_config parameter allows you to use your own LLM and embedding providers instead of Mem0’s cloud API. This is useful for self-hosted deployments or when you want more control over the memory processing. Copy Ask AI local_config = { "llm" : { "provider" : str , # LLM provider name (e.g., "anthropic", "openai") "config" : { # Provider-specific configuration "model" : str , # Model name "api_key" : str , # API key for the provider # Other provider-specific parameters } }, "embedder" : { "provider" : str , # Embedding provider name (e.g., "openai") "config" : { # Provider-specific configuration "model" : str , # Model name # Other provider-specific parameters } } } # Initialize Mem0 memory service with local configuration memory = Mem0MemoryService( local_config = local_config, # Use local LLM for memory processing user_id = "user123" , # Unique identifier for the user ) When using local_config do not provide the api_key parameter. ​ Frame Flow ​ Error Handling The service includes basic error handling to ensure conversation flow continues even when memory operations fail: Exceptions during memory storage and retrieval are caught and logged If an error occurs during frame processing, an ErrorFrame is emitted with error details The original frame is still passed downstream to prevent the pipeline from stalling Connection and authentication errors from the Mem0 API will be logged but won’t interrupt the conversation While the service attempts to handle errors gracefully, memory operations that fail may result in missing context in conversations. Monitor your application logs for memory-related errors. Tavus Moondream On this page Overview Installation Mem0MemoryService Constructor Parameters Input Parameters Input Frames Output Frames Memory Operations Message Storage Memory Retrieval Pipeline Positioning Usage Examples Basic Integration Using Local Configuration Frame Flow Error Handling Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/observers_debug-observer_85f3b041.txt b/observers_debug-observer_85f3b041.txt
new file mode 100644
index 0000000000000000000000000000000000000000..f38636368e943303602f24714ddfaac991bea8ee
--- /dev/null
+++ b/observers_debug-observer_85f3b041.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/observers/debug-observer#log-output-format
+Title: Debug Log Observer - Pipecat
+==================================================
+
+Debug Log Observer - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Observers Debug Log Observer Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Observer Pattern Debug Observer LLM Observer Transcription Observer Turn Tracking Observer Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline The DebugLogObserver provides detailed logging of frame activity in your Pipecat pipeline, with full visibility into frame content and flexible filtering options. ​ Features Log all frame types and their content Filter by specific frame types Filter by source or destination components Automatic formatting of frame fields Special handling for complex data structures ​ Usage ​ Log All Frames Log all frames passing through the pipeline: Copy Ask AI from pipecat.observers.loggers.debug_log_observer import DebugLogObserver task = PipelineTask( pipeline, params = PipelineParams( observers = [DebugLogObserver()], ), ) ​ Filter by Frame Types Log only specific frame types: Copy Ask AI from pipecat.frames.frames import TranscriptionFrame, InterimTranscriptionFrame from pipecat.observers.loggers.debug_log_observer import DebugLogObserver task = PipelineTask( pipeline, params = PipelineParams( observers = [ DebugLogObserver( frame_types = ( TranscriptionFrame, InterimTranscriptionFrame )) ], ), ) ​ Advanced Source/Destination Filtering Filter frames based on their type and source/destination: Copy Ask AI from pipecat.frames.frames import StartInterruptionFrame, UserStartedSpeakingFrame, LLMTextFrame from pipecat.observers.loggers.debug_log_observer import DebugLogObserver, FrameEndpoint from pipecat.transports.base_output_transport import BaseOutputTransport from pipecat.services.stt_service import STTService task = PipelineTask( pipeline, params = PipelineParams( observers = [ DebugLogObserver( frame_types = { # Only log StartInterruptionFrame when source is BaseOutputTransport StartInterruptionFrame: (BaseOutputTransport, FrameEndpoint. SOURCE ), # Only log UserStartedSpeakingFrame when destination is STTService UserStartedSpeakingFrame: (STTService, FrameEndpoint. DESTINATION ), # Log LLMTextFrame regardless of source or destination LLMTextFrame: None }) ], ), ) ​ Log Output Format The observer logs each frame with its complete details: Copy Ask AI [Source] → [Destination]: [FrameType] [field1: value1, field2: value2, ...] at [timestamp]s For example: Copy Ask AI OpenAILLMService#0 → DailyTransport#0: LLMTextFrame text: 'Hello, how can I help you today?' at 1.24s ​ Configuration Options Parameter Type Description frame_types Tuple[Type[Frame], ...] or Dict[Type[Frame], Optional[Tuple[Type, FrameEndpoint]]] Frame types to log, with optional source/destination filtering exclude_fields Set[str] Field names to exclude from logging (defaults to binary fields) ​ FrameEndpoint Enum The FrameEndpoint enum is used for source/destination filtering: FrameEndpoint.SOURCE : Filter by source component FrameEndpoint.DESTINATION : Filter by destination component Observer Pattern LLM Observer On this page Features Usage Log All Frames Filter by Frame Types Advanced Source/Destination Filtering Log Output Format Configuration Options FrameEndpoint Enum Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/observers_observer-pattern_6dc4f572.txt b/observers_observer-pattern_6dc4f572.txt
new file mode 100644
index 0000000000000000000000000000000000000000..b592914242b6fb6d2d9042dc2e8bf128e4cfa8f3
--- /dev/null
+++ b/observers_observer-pattern_6dc4f572.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/observers/observer-pattern#base-observer
+Title: Observer Pattern - Pipecat
+==================================================
+
+Observer Pattern - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Observers Observer Pattern Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Observer Pattern Debug Observer LLM Observer Transcription Observer Turn Tracking Observer Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline The Observer pattern in Pipecat allows non-intrusive monitoring of frames as they flow through the pipeline. Observers can watch frame traffic without affecting the pipeline’s core functionality. ​ Base Observer All observers must inherit from BaseObserver and implement the on_push_frame method: Copy Ask AI from pipecat.observers.base_observer import BaseObserver class CustomObserver ( BaseObserver ): async def on_push_frame ( self , src : FrameProcessor, dst : FrameProcessor, frame : Frame, direction : FrameDirection, timestamp : int , ): # Your frame observation logic here pass ​ Available Observers Pipecat provides several built-in observers: LLMLogObserver : Logs LLM activity and responses TranscriptionLogObserver : Logs speech-to-text transcription events RTVIObserver : Converts internal frames to RTVI protocol messages for server to client messaging ​ Using Multiple Observers You can attach multiple observers to a pipeline task. Each observer will be notified of all frames: Copy Ask AI task = PipelineTask( pipeline, params = PipelineParams( observers = [LLMLogObserver(), TranscriptionLogObserver(), CustomObserver()], ), ) ​ Example: Debug Observer Here’s an example observer that logs interruptions and bot speaking events: Copy Ask AI class DebugObserver ( BaseObserver ): """Observer to log interruptions and bot speaking events to the console. Logs all frame instances of: - StartInterruptionFrame - BotStartedSpeakingFrame - BotStoppedSpeakingFrame This allows you to see the frame flow from processor to processor through the pipeline for these frames. Log format: [EVENT TYPE]: [source processor] → [destination processor] at [timestamp]s """ async def on_push_frame ( self , src : FrameProcessor, dst : FrameProcessor, frame : Frame, direction : FrameDirection, timestamp : int , ): time_sec = timestamp / 1_000_000_000 arrow = "→" if direction == FrameDirection. DOWNSTREAM else "←" if isinstance (frame, StartInterruptionFrame): logger.info( f "⚡ INTERRUPTION START: { src } { arrow } { dst } at { time_sec :.2f} s" ) elif isinstance (frame, BotStartedSpeakingFrame): logger.info( f "🤖 BOT START SPEAKING: { src } { arrow } { dst } at { time_sec :.2f} s" ) elif isinstance (frame, BotStoppedSpeakingFrame): logger.info( f "🤖 BOT STOP SPEAKING: { src } { arrow } { dst } at { time_sec :.2f} s" ) ​ Common Use Cases Observers are particularly useful for: Debugging frame flow Logging specific events Monitoring pipeline behavior Collecting metrics Converting internal frames to external messages MCPClient Debug Observer On this page Base Observer Available Observers Using Multiple Observers Example: Debug Observer Common Use Cases Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/observers_turn-tracking-observer_1a144d0e.txt b/observers_turn-tracking-observer_1a144d0e.txt
new file mode 100644
index 0000000000000000000000000000000000000000..0a48562713adfaeadebf1b0c4eae63e81c96ad0c
--- /dev/null
+++ b/observers_turn-tracking-observer_1a144d0e.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/observers/turn-tracking-observer
+Title: Turn Tracking Observer - Pipecat
+==================================================
+
+Turn Tracking Observer - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Observers Turn Tracking Observer Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Observer Pattern Debug Observer LLM Observer Transcription Observer Turn Tracking Observer Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline The TurnTrackingObserver monitors and tracks conversational turns in your Pipecat pipeline, providing events when turns start and end. It intelligently identifies when a user-bot interaction cycle begins and completes. ​ Turn Lifecycle A turn represents a complete user-bot interaction cycle: Start : When the user starts speaking (or pipeline starts for first turn) Processing : User speaks, bot processes and responds End : After the bot finishes speaking and either: The user starts speaking again A timeout period elapses with no further activity ​ Events The observer emits two main events: on_turn_started : When a new turn begins Parameters: turn_number (int) on_turn_ended : When a turn completes Parameters: turn_number (int), duration (float, in seconds), was_interrupted (bool) ​ Usage The observer is automatically created when you initialize a PipelineTask with enable_turn_tracking=True (which is the default): Copy Ask AI task = PipelineTask( pipeline, params = PipelineParams( allow_interruptions = True ), # Turn tracking is enabled by default ) # Access the observer turn_observer = task.turn_tracking_observer # Register event handlers @turn_observer.event_handler ( "on_turn_started" ) async def on_turn_started ( observer , turn_number ): logger.info( f "Turn { turn_number } started" ) @turn_observer.event_handler ( "on_turn_ended" ) async def on_turn_ended ( observer , turn_number , duration , was_interrupted ): status = "interrupted" if was_interrupted else "completed" logger.info( f "Turn { turn_number } { status } in { duration :.2f} s" ) ​ Configuration You can configure the observer’s behavior when creating a PipelineTask : Copy Ask AI from pipecat.observers.turn_tracking_observer import TurnTrackingObserver # Create a custom observer instance custom_turn_tracker = TurnTrackingObserver( turn_end_timeout_secs = 3.5 , # Turn end timeout (default: 2.5) ) # Add it as a regular observer task = PipelineTask( pipeline, observers = [custom_turn_tracker], # Disable the default one if adding your own enable_turn_tracking = False , ) ​ Interruptions The observer automatically detects interruptions when the user starts speaking while the bot is still speaking. In this case: The current turn is marked as interrupted ( was_interrupted=True ) A new turn begins immediately ​ How It Works The observer monitors specific frame types to track conversation flow: StartFrame : Initiates the first turn UserStartedSpeakingFrame : Starts user speech or triggers a new turn BotStartedSpeakingFrame : Marks bot speech beginning BotStoppedSpeakingFrame : Starts the turn end timeout After a bot stops speaking, the observer waits for the configured timeout period. If no further bot speech occurs, the turn ends; otherwise, it continues as part of the same turn. ​ Use Cases Analytics : Measure turn durations, interruption rates, and conversation flow Logging : Record turn-based logs for diagnostics and analysis Visualization : Show turn-based conversation timelines in UIs Tracing : Group spans and metrics by conversation turns Transcription Observer Daily REST Helper On this page Turn Lifecycle Events Usage Configuration Interruptions How It Works Use Cases Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/observers_turn-tracking-observer_1f4e11c8.txt b/observers_turn-tracking-observer_1f4e11c8.txt
new file mode 100644
index 0000000000000000000000000000000000000000..7656eca3066a8d5044092e5406dc608096c2f8fb
--- /dev/null
+++ b/observers_turn-tracking-observer_1f4e11c8.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/observers/turn-tracking-observer#configuration
+Title: Turn Tracking Observer - Pipecat
+==================================================
+
+Turn Tracking Observer - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Observers Turn Tracking Observer Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Observer Pattern Debug Observer LLM Observer Transcription Observer Turn Tracking Observer Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline The TurnTrackingObserver monitors and tracks conversational turns in your Pipecat pipeline, providing events when turns start and end. It intelligently identifies when a user-bot interaction cycle begins and completes. ​ Turn Lifecycle A turn represents a complete user-bot interaction cycle: Start : When the user starts speaking (or pipeline starts for first turn) Processing : User speaks, bot processes and responds End : After the bot finishes speaking and either: The user starts speaking again A timeout period elapses with no further activity ​ Events The observer emits two main events: on_turn_started : When a new turn begins Parameters: turn_number (int) on_turn_ended : When a turn completes Parameters: turn_number (int), duration (float, in seconds), was_interrupted (bool) ​ Usage The observer is automatically created when you initialize a PipelineTask with enable_turn_tracking=True (which is the default): Copy Ask AI task = PipelineTask( pipeline, params = PipelineParams( allow_interruptions = True ), # Turn tracking is enabled by default ) # Access the observer turn_observer = task.turn_tracking_observer # Register event handlers @turn_observer.event_handler ( "on_turn_started" ) async def on_turn_started ( observer , turn_number ): logger.info( f "Turn { turn_number } started" ) @turn_observer.event_handler ( "on_turn_ended" ) async def on_turn_ended ( observer , turn_number , duration , was_interrupted ): status = "interrupted" if was_interrupted else "completed" logger.info( f "Turn { turn_number } { status } in { duration :.2f} s" ) ​ Configuration You can configure the observer’s behavior when creating a PipelineTask : Copy Ask AI from pipecat.observers.turn_tracking_observer import TurnTrackingObserver # Create a custom observer instance custom_turn_tracker = TurnTrackingObserver( turn_end_timeout_secs = 3.5 , # Turn end timeout (default: 2.5) ) # Add it as a regular observer task = PipelineTask( pipeline, observers = [custom_turn_tracker], # Disable the default one if adding your own enable_turn_tracking = False , ) ​ Interruptions The observer automatically detects interruptions when the user starts speaking while the bot is still speaking. In this case: The current turn is marked as interrupted ( was_interrupted=True ) A new turn begins immediately ​ How It Works The observer monitors specific frame types to track conversation flow: StartFrame : Initiates the first turn UserStartedSpeakingFrame : Starts user speech or triggers a new turn BotStartedSpeakingFrame : Marks bot speech beginning BotStoppedSpeakingFrame : Starts the turn end timeout After a bot stops speaking, the observer waits for the configured timeout period. If no further bot speech occurs, the turn ends; otherwise, it continues as part of the same turn. ​ Use Cases Analytics : Measure turn durations, interruption rates, and conversation flow Logging : Record turn-based logs for diagnostics and analysis Visualization : Show turn-based conversation timelines in UIs Tracing : Group spans and metrics by conversation turns Transcription Observer Daily REST Helper On this page Turn Lifecycle Events Usage Configuration Interruptions How It Works Use Cases Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/observers_turn-tracking-observer_cf0a6a7a.txt b/observers_turn-tracking-observer_cf0a6a7a.txt
new file mode 100644
index 0000000000000000000000000000000000000000..c770327f92bf0c2f785500f732ea7cd8900940c5
--- /dev/null
+++ b/observers_turn-tracking-observer_cf0a6a7a.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/observers/turn-tracking-observer#usage
+Title: Turn Tracking Observer - Pipecat
+==================================================
+
+Turn Tracking Observer - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Observers Turn Tracking Observer Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Observer Pattern Debug Observer LLM Observer Transcription Observer Turn Tracking Observer Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline The TurnTrackingObserver monitors and tracks conversational turns in your Pipecat pipeline, providing events when turns start and end. It intelligently identifies when a user-bot interaction cycle begins and completes. ​ Turn Lifecycle A turn represents a complete user-bot interaction cycle: Start : When the user starts speaking (or pipeline starts for first turn) Processing : User speaks, bot processes and responds End : After the bot finishes speaking and either: The user starts speaking again A timeout period elapses with no further activity ​ Events The observer emits two main events: on_turn_started : When a new turn begins Parameters: turn_number (int) on_turn_ended : When a turn completes Parameters: turn_number (int), duration (float, in seconds), was_interrupted (bool) ​ Usage The observer is automatically created when you initialize a PipelineTask with enable_turn_tracking=True (which is the default): Copy Ask AI task = PipelineTask( pipeline, params = PipelineParams( allow_interruptions = True ), # Turn tracking is enabled by default ) # Access the observer turn_observer = task.turn_tracking_observer # Register event handlers @turn_observer.event_handler ( "on_turn_started" ) async def on_turn_started ( observer , turn_number ): logger.info( f "Turn { turn_number } started" ) @turn_observer.event_handler ( "on_turn_ended" ) async def on_turn_ended ( observer , turn_number , duration , was_interrupted ): status = "interrupted" if was_interrupted else "completed" logger.info( f "Turn { turn_number } { status } in { duration :.2f} s" ) ​ Configuration You can configure the observer’s behavior when creating a PipelineTask : Copy Ask AI from pipecat.observers.turn_tracking_observer import TurnTrackingObserver # Create a custom observer instance custom_turn_tracker = TurnTrackingObserver( turn_end_timeout_secs = 3.5 , # Turn end timeout (default: 2.5) ) # Add it as a regular observer task = PipelineTask( pipeline, observers = [custom_turn_tracker], # Disable the default one if adding your own enable_turn_tracking = False , ) ​ Interruptions The observer automatically detects interruptions when the user starts speaking while the bot is still speaking. In this case: The current turn is marked as interrupted ( was_interrupted=True ) A new turn begins immediately ​ How It Works The observer monitors specific frame types to track conversation flow: StartFrame : Initiates the first turn UserStartedSpeakingFrame : Starts user speech or triggers a new turn BotStartedSpeakingFrame : Marks bot speech beginning BotStoppedSpeakingFrame : Starts the turn end timeout After a bot stops speaking, the observer waits for the configured timeout period. If no further bot speech occurs, the turn ends; otherwise, it continues as part of the same turn. ​ Use Cases Analytics : Measure turn durations, interruption rates, and conversation flow Logging : Record turn-based logs for diagnostics and analysis Visualization : Show turn-based conversation timelines in UIs Tracing : Group spans and metrics by conversation turns Transcription Observer Daily REST Helper On this page Turn Lifecycle Events Usage Configuration Interruptions How It Works Use Cases Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/pipecat-client-android_indexhtml_832410bf.txt b/pipecat-client-android_indexhtml_832410bf.txt
new file mode 100644
index 0000000000000000000000000000000000000000..8b3841d2498153bfecda8af4a8de297b104c38be
--- /dev/null
+++ b/pipecat-client-android_indexhtml_832410bf.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/client/android/pipecat-client-android/index.html#how-it-works
+Title: Overview - Pipecat
+==================================================
+
+Overview - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Get Started Overview Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Get Started Overview Installation & Setup Quickstart Core Concepts Next Steps & Examples Pipecat is an open source Python framework that handles the complex orchestration of AI services, network transport, audio processing, and multimodal interactions. “Multimodal” means you can use any combination of audio, video, images, and/or text in your interactions. And “real-time” means that things are happening quickly enough that it feels conversational—a “back-and-forth” with a bot, not submitting a query and waiting for results. ​ What You Can Build Voice Assistants Natural, real-time conversations with AI using speech recognition and synthesis Interactive Agents Personal coaches and meeting assistants that can understand context and provide guidance Multimodal Apps Applications that combine voice, video, images, and text for rich interactions Creative Tools Storytelling experiences and social companions that engage users Business Solutions Customer intake flows and support bots for automated business processes Complex Flows Structured conversations using Pipecat Flows for managing complex interactions ​ How It Works The flow of interactions in a Pipecat application is typically straightforward: The bot says something The user says something The bot says something The user says something This continues until the conversation naturally ends. While this flow seems simple, making it feel natural requires sophisticated real-time processing. ​ Real-time Processing Pipecat’s pipeline architecture handles both simple voice interactions and complex multimodal processing. Let’s look at how data flows through the system: Voice app Multimodal app 1 Send Audio Transmit and capture streamed audio from the user 2 Transcribe Speech Convert speech to text as the user is talking 3 Process with LLM Generate responses using a large language model 4 Convert to Speech Transform text responses into natural speech 5 Play Audio Stream the audio response back to the user 1 Send Audio Transmit and capture streamed audio from the user 2 Transcribe Speech Convert speech to text as the user is talking 3 Process with LLM Generate responses using a large language model 4 Convert to Speech Transform text responses into natural speech 5 Play Audio Stream the audio response back to the user 1 Send Audio and Video Transmit and capture audio, video, and image inputs simultaneously 2 Process Streams Handle multiple input streams in parallel 3 Model Processing Send combined inputs to multimodal models (like GPT-4V) 4 Generate Outputs Create various outputs (text, images, audio, etc.) 5 Coordinate Presentation Synchronize and present multiple output types In both cases, Pipecat: Processes responses as they stream in Handles multiple input/output modalities concurrently Manages resource allocation and synchronization Coordinates parallel processing tasks This architecture creates fluid, natural interactions without noticeable delays, whether you’re building a simple voice assistant or a complex multimodal application. Pipecat’s pipeline architecture is particularly valuable for managing the complexity of real-time, multimodal interactions, ensuring smooth data flow and proper synchronization regardless of the input/output types involved. Pipecat handles all this complexity for you, letting you focus on building your application rather than managing the underlying infrastructure. ​ Next Steps Ready to build your first Pipecat application? Installation & Setup Prepare your environment and install required dependencies Quickstart Build and run your first Pipecat application Core Concepts Learn about pipelines, frames, and real-time processing Use Cases Explore example implementations and patterns ​ Join Our Community Discord Community Connect with other developers, share your projects, and get support from the Pipecat team. Installation & Setup On this page What You Can Build How It Works Real-time Processing Next Steps Join Our Community Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/pipecat-client-android_indexhtml_a38c5b88.txt b/pipecat-client-android_indexhtml_a38c5b88.txt
new file mode 100644
index 0000000000000000000000000000000000000000..f0b7316bf072ec5a33c88f4fce36ed12c22b720c
--- /dev/null
+++ b/pipecat-client-android_indexhtml_a38c5b88.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/client/android/pipecat-client-android/index.html
+Title: Overview - Pipecat
+==================================================
+
+Overview - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Get Started Overview Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Get Started Overview Installation & Setup Quickstart Core Concepts Next Steps & Examples Pipecat is an open source Python framework that handles the complex orchestration of AI services, network transport, audio processing, and multimodal interactions. “Multimodal” means you can use any combination of audio, video, images, and/or text in your interactions. And “real-time” means that things are happening quickly enough that it feels conversational—a “back-and-forth” with a bot, not submitting a query and waiting for results. ​ What You Can Build Voice Assistants Natural, real-time conversations with AI using speech recognition and synthesis Interactive Agents Personal coaches and meeting assistants that can understand context and provide guidance Multimodal Apps Applications that combine voice, video, images, and text for rich interactions Creative Tools Storytelling experiences and social companions that engage users Business Solutions Customer intake flows and support bots for automated business processes Complex Flows Structured conversations using Pipecat Flows for managing complex interactions ​ How It Works The flow of interactions in a Pipecat application is typically straightforward: The bot says something The user says something The bot says something The user says something This continues until the conversation naturally ends. While this flow seems simple, making it feel natural requires sophisticated real-time processing. ​ Real-time Processing Pipecat’s pipeline architecture handles both simple voice interactions and complex multimodal processing. Let’s look at how data flows through the system: Voice app Multimodal app 1 Send Audio Transmit and capture streamed audio from the user 2 Transcribe Speech Convert speech to text as the user is talking 3 Process with LLM Generate responses using a large language model 4 Convert to Speech Transform text responses into natural speech 5 Play Audio Stream the audio response back to the user 1 Send Audio Transmit and capture streamed audio from the user 2 Transcribe Speech Convert speech to text as the user is talking 3 Process with LLM Generate responses using a large language model 4 Convert to Speech Transform text responses into natural speech 5 Play Audio Stream the audio response back to the user 1 Send Audio and Video Transmit and capture audio, video, and image inputs simultaneously 2 Process Streams Handle multiple input streams in parallel 3 Model Processing Send combined inputs to multimodal models (like GPT-4V) 4 Generate Outputs Create various outputs (text, images, audio, etc.) 5 Coordinate Presentation Synchronize and present multiple output types In both cases, Pipecat: Processes responses as they stream in Handles multiple input/output modalities concurrently Manages resource allocation and synchronization Coordinates parallel processing tasks This architecture creates fluid, natural interactions without noticeable delays, whether you’re building a simple voice assistant or a complex multimodal application. Pipecat’s pipeline architecture is particularly valuable for managing the complexity of real-time, multimodal interactions, ensuring smooth data flow and proper synchronization regardless of the input/output types involved. Pipecat handles all this complexity for you, letting you focus on building your application rather than managing the underlying infrastructure. ​ Next Steps Ready to build your first Pipecat application? Installation & Setup Prepare your environment and install required dependencies Quickstart Build and run your first Pipecat application Core Concepts Learn about pipelines, frames, and real-time processing Use Cases Explore example implementations and patterns ​ Join Our Community Discord Community Connect with other developers, share your projects, and get support from the Pipecat team. Installation & Setup On this page What You Can Build How It Works Real-time Processing Next Steps Join Our Community Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/pipecat-transport-daily_indexhtml_17ab5517.txt b/pipecat-transport-daily_indexhtml_17ab5517.txt
new file mode 100644
index 0000000000000000000000000000000000000000..804b151f42a84cb3509b92148a2eaf5968f48985
--- /dev/null
+++ b/pipecat-transport-daily_indexhtml_17ab5517.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/client/android/pipecat-transport-daily/index.html#join-our-community
+Title: Overview - Pipecat
+==================================================
+
+Overview - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Get Started Overview Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Get Started Overview Installation & Setup Quickstart Core Concepts Next Steps & Examples Pipecat is an open source Python framework that handles the complex orchestration of AI services, network transport, audio processing, and multimodal interactions. “Multimodal” means you can use any combination of audio, video, images, and/or text in your interactions. And “real-time” means that things are happening quickly enough that it feels conversational—a “back-and-forth” with a bot, not submitting a query and waiting for results. ​ What You Can Build Voice Assistants Natural, real-time conversations with AI using speech recognition and synthesis Interactive Agents Personal coaches and meeting assistants that can understand context and provide guidance Multimodal Apps Applications that combine voice, video, images, and text for rich interactions Creative Tools Storytelling experiences and social companions that engage users Business Solutions Customer intake flows and support bots for automated business processes Complex Flows Structured conversations using Pipecat Flows for managing complex interactions ​ How It Works The flow of interactions in a Pipecat application is typically straightforward: The bot says something The user says something The bot says something The user says something This continues until the conversation naturally ends. While this flow seems simple, making it feel natural requires sophisticated real-time processing. ​ Real-time Processing Pipecat’s pipeline architecture handles both simple voice interactions and complex multimodal processing. Let’s look at how data flows through the system: Voice app Multimodal app 1 Send Audio Transmit and capture streamed audio from the user 2 Transcribe Speech Convert speech to text as the user is talking 3 Process with LLM Generate responses using a large language model 4 Convert to Speech Transform text responses into natural speech 5 Play Audio Stream the audio response back to the user 1 Send Audio Transmit and capture streamed audio from the user 2 Transcribe Speech Convert speech to text as the user is talking 3 Process with LLM Generate responses using a large language model 4 Convert to Speech Transform text responses into natural speech 5 Play Audio Stream the audio response back to the user 1 Send Audio and Video Transmit and capture audio, video, and image inputs simultaneously 2 Process Streams Handle multiple input streams in parallel 3 Model Processing Send combined inputs to multimodal models (like GPT-4V) 4 Generate Outputs Create various outputs (text, images, audio, etc.) 5 Coordinate Presentation Synchronize and present multiple output types In both cases, Pipecat: Processes responses as they stream in Handles multiple input/output modalities concurrently Manages resource allocation and synchronization Coordinates parallel processing tasks This architecture creates fluid, natural interactions without noticeable delays, whether you’re building a simple voice assistant or a complex multimodal application. Pipecat’s pipeline architecture is particularly valuable for managing the complexity of real-time, multimodal interactions, ensuring smooth data flow and proper synchronization regardless of the input/output types involved. Pipecat handles all this complexity for you, letting you focus on building your application rather than managing the underlying infrastructure. ​ Next Steps Ready to build your first Pipecat application? Installation & Setup Prepare your environment and install required dependencies Quickstart Build and run your first Pipecat application Core Concepts Learn about pipelines, frames, and real-time processing Use Cases Explore example implementations and patterns ​ Join Our Community Discord Community Connect with other developers, share your projects, and get support from the Pipecat team. Installation & Setup On this page What You Can Build How It Works Real-time Processing Next Steps Join Our Community Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/pipecat-transport-daily_indexhtml_26a3cb3b.txt b/pipecat-transport-daily_indexhtml_26a3cb3b.txt
new file mode 100644
index 0000000000000000000000000000000000000000..e13404007c6c887fa8b1876c64ece678557e1644
--- /dev/null
+++ b/pipecat-transport-daily_indexhtml_26a3cb3b.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/client/android/pipecat-transport-daily/index.html#what-you-can-build
+Title: Overview - Pipecat
+==================================================
+
+Overview - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Get Started Overview Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Get Started Overview Installation & Setup Quickstart Core Concepts Next Steps & Examples Pipecat is an open source Python framework that handles the complex orchestration of AI services, network transport, audio processing, and multimodal interactions. “Multimodal” means you can use any combination of audio, video, images, and/or text in your interactions. And “real-time” means that things are happening quickly enough that it feels conversational—a “back-and-forth” with a bot, not submitting a query and waiting for results. ​ What You Can Build Voice Assistants Natural, real-time conversations with AI using speech recognition and synthesis Interactive Agents Personal coaches and meeting assistants that can understand context and provide guidance Multimodal Apps Applications that combine voice, video, images, and text for rich interactions Creative Tools Storytelling experiences and social companions that engage users Business Solutions Customer intake flows and support bots for automated business processes Complex Flows Structured conversations using Pipecat Flows for managing complex interactions ​ How It Works The flow of interactions in a Pipecat application is typically straightforward: The bot says something The user says something The bot says something The user says something This continues until the conversation naturally ends. While this flow seems simple, making it feel natural requires sophisticated real-time processing. ​ Real-time Processing Pipecat’s pipeline architecture handles both simple voice interactions and complex multimodal processing. Let’s look at how data flows through the system: Voice app Multimodal app 1 Send Audio Transmit and capture streamed audio from the user 2 Transcribe Speech Convert speech to text as the user is talking 3 Process with LLM Generate responses using a large language model 4 Convert to Speech Transform text responses into natural speech 5 Play Audio Stream the audio response back to the user 1 Send Audio Transmit and capture streamed audio from the user 2 Transcribe Speech Convert speech to text as the user is talking 3 Process with LLM Generate responses using a large language model 4 Convert to Speech Transform text responses into natural speech 5 Play Audio Stream the audio response back to the user 1 Send Audio and Video Transmit and capture audio, video, and image inputs simultaneously 2 Process Streams Handle multiple input streams in parallel 3 Model Processing Send combined inputs to multimodal models (like GPT-4V) 4 Generate Outputs Create various outputs (text, images, audio, etc.) 5 Coordinate Presentation Synchronize and present multiple output types In both cases, Pipecat: Processes responses as they stream in Handles multiple input/output modalities concurrently Manages resource allocation and synchronization Coordinates parallel processing tasks This architecture creates fluid, natural interactions without noticeable delays, whether you’re building a simple voice assistant or a complex multimodal application. Pipecat’s pipeline architecture is particularly valuable for managing the complexity of real-time, multimodal interactions, ensuring smooth data flow and proper synchronization regardless of the input/output types involved. Pipecat handles all this complexity for you, letting you focus on building your application rather than managing the underlying infrastructure. ​ Next Steps Ready to build your first Pipecat application? Installation & Setup Prepare your environment and install required dependencies Quickstart Build and run your first Pipecat application Core Concepts Learn about pipelines, frames, and real-time processing Use Cases Explore example implementations and patterns ​ Join Our Community Discord Community Connect with other developers, share your projects, and get support from the Pipecat team. Installation & Setup On this page What You Can Build How It Works Real-time Processing Next Steps Join Our Community Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/pipecat-transport-daily_indexhtml_97e010d2.txt b/pipecat-transport-daily_indexhtml_97e010d2.txt
new file mode 100644
index 0000000000000000000000000000000000000000..e88f85990aadcb618b32270f1b99ecf144374ea5
--- /dev/null
+++ b/pipecat-transport-daily_indexhtml_97e010d2.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/client/android/pipecat-transport-daily/index.html#next-steps
+Title: Overview - Pipecat
+==================================================
+
+Overview - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Get Started Overview Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Get Started Overview Installation & Setup Quickstart Core Concepts Next Steps & Examples Pipecat is an open source Python framework that handles the complex orchestration of AI services, network transport, audio processing, and multimodal interactions. “Multimodal” means you can use any combination of audio, video, images, and/or text in your interactions. And “real-time” means that things are happening quickly enough that it feels conversational—a “back-and-forth” with a bot, not submitting a query and waiting for results. ​ What You Can Build Voice Assistants Natural, real-time conversations with AI using speech recognition and synthesis Interactive Agents Personal coaches and meeting assistants that can understand context and provide guidance Multimodal Apps Applications that combine voice, video, images, and text for rich interactions Creative Tools Storytelling experiences and social companions that engage users Business Solutions Customer intake flows and support bots for automated business processes Complex Flows Structured conversations using Pipecat Flows for managing complex interactions ​ How It Works The flow of interactions in a Pipecat application is typically straightforward: The bot says something The user says something The bot says something The user says something This continues until the conversation naturally ends. While this flow seems simple, making it feel natural requires sophisticated real-time processing. ​ Real-time Processing Pipecat’s pipeline architecture handles both simple voice interactions and complex multimodal processing. Let’s look at how data flows through the system: Voice app Multimodal app 1 Send Audio Transmit and capture streamed audio from the user 2 Transcribe Speech Convert speech to text as the user is talking 3 Process with LLM Generate responses using a large language model 4 Convert to Speech Transform text responses into natural speech 5 Play Audio Stream the audio response back to the user 1 Send Audio Transmit and capture streamed audio from the user 2 Transcribe Speech Convert speech to text as the user is talking 3 Process with LLM Generate responses using a large language model 4 Convert to Speech Transform text responses into natural speech 5 Play Audio Stream the audio response back to the user 1 Send Audio and Video Transmit and capture audio, video, and image inputs simultaneously 2 Process Streams Handle multiple input streams in parallel 3 Model Processing Send combined inputs to multimodal models (like GPT-4V) 4 Generate Outputs Create various outputs (text, images, audio, etc.) 5 Coordinate Presentation Synchronize and present multiple output types In both cases, Pipecat: Processes responses as they stream in Handles multiple input/output modalities concurrently Manages resource allocation and synchronization Coordinates parallel processing tasks This architecture creates fluid, natural interactions without noticeable delays, whether you’re building a simple voice assistant or a complex multimodal application. Pipecat’s pipeline architecture is particularly valuable for managing the complexity of real-time, multimodal interactions, ensuring smooth data flow and proper synchronization regardless of the input/output types involved. Pipecat handles all this complexity for you, letting you focus on building your application rather than managing the underlying infrastructure. ​ Next Steps Ready to build your first Pipecat application? Installation & Setup Prepare your environment and install required dependencies Quickstart Build and run your first Pipecat application Core Concepts Learn about pipelines, frames, and real-time processing Use Cases Explore example implementations and patterns ​ Join Our Community Discord Community Connect with other developers, share your projects, and get support from the Pipecat team. Installation & Setup On this page What You Can Build How It Works Real-time Processing Next Steps Join Our Community Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/pipecat-transport-gemini-live-websocket_indexhtml_774e88b2.txt b/pipecat-transport-gemini-live-websocket_indexhtml_774e88b2.txt
new file mode 100644
index 0000000000000000000000000000000000000000..b7ee72b3fd50c243b2545814982cc5864819a501
--- /dev/null
+++ b/pipecat-transport-gemini-live-websocket_indexhtml_774e88b2.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/client/android/pipecat-transport-gemini-live-websocket/index.html#what-you-can-build
+Title: Overview - Pipecat
+==================================================
+
+Overview - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Get Started Overview Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Get Started Overview Installation & Setup Quickstart Core Concepts Next Steps & Examples Pipecat is an open source Python framework that handles the complex orchestration of AI services, network transport, audio processing, and multimodal interactions. “Multimodal” means you can use any combination of audio, video, images, and/or text in your interactions. And “real-time” means that things are happening quickly enough that it feels conversational—a “back-and-forth” with a bot, not submitting a query and waiting for results. ​ What You Can Build Voice Assistants Natural, real-time conversations with AI using speech recognition and synthesis Interactive Agents Personal coaches and meeting assistants that can understand context and provide guidance Multimodal Apps Applications that combine voice, video, images, and text for rich interactions Creative Tools Storytelling experiences and social companions that engage users Business Solutions Customer intake flows and support bots for automated business processes Complex Flows Structured conversations using Pipecat Flows for managing complex interactions ​ How It Works The flow of interactions in a Pipecat application is typically straightforward: The bot says something The user says something The bot says something The user says something This continues until the conversation naturally ends. While this flow seems simple, making it feel natural requires sophisticated real-time processing. ​ Real-time Processing Pipecat’s pipeline architecture handles both simple voice interactions and complex multimodal processing. Let’s look at how data flows through the system: Voice app Multimodal app 1 Send Audio Transmit and capture streamed audio from the user 2 Transcribe Speech Convert speech to text as the user is talking 3 Process with LLM Generate responses using a large language model 4 Convert to Speech Transform text responses into natural speech 5 Play Audio Stream the audio response back to the user 1 Send Audio Transmit and capture streamed audio from the user 2 Transcribe Speech Convert speech to text as the user is talking 3 Process with LLM Generate responses using a large language model 4 Convert to Speech Transform text responses into natural speech 5 Play Audio Stream the audio response back to the user 1 Send Audio and Video Transmit and capture audio, video, and image inputs simultaneously 2 Process Streams Handle multiple input streams in parallel 3 Model Processing Send combined inputs to multimodal models (like GPT-4V) 4 Generate Outputs Create various outputs (text, images, audio, etc.) 5 Coordinate Presentation Synchronize and present multiple output types In both cases, Pipecat: Processes responses as they stream in Handles multiple input/output modalities concurrently Manages resource allocation and synchronization Coordinates parallel processing tasks This architecture creates fluid, natural interactions without noticeable delays, whether you’re building a simple voice assistant or a complex multimodal application. Pipecat’s pipeline architecture is particularly valuable for managing the complexity of real-time, multimodal interactions, ensuring smooth data flow and proper synchronization regardless of the input/output types involved. Pipecat handles all this complexity for you, letting you focus on building your application rather than managing the underlying infrastructure. ​ Next Steps Ready to build your first Pipecat application? Installation & Setup Prepare your environment and install required dependencies Quickstart Build and run your first Pipecat application Core Concepts Learn about pipelines, frames, and real-time processing Use Cases Explore example implementations and patterns ​ Join Our Community Discord Community Connect with other developers, share your projects, and get support from the Pipecat team. Installation & Setup On this page What You Can Build How It Works Real-time Processing Next Steps Join Our Community Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/pipecat-transport-gemini-live-websocket_indexhtml_da9d07d0.txt b/pipecat-transport-gemini-live-websocket_indexhtml_da9d07d0.txt
new file mode 100644
index 0000000000000000000000000000000000000000..9be3a11eb590bc43d67e2a560defd6739dc72307
--- /dev/null
+++ b/pipecat-transport-gemini-live-websocket_indexhtml_da9d07d0.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/client/android/pipecat-transport-gemini-live-websocket/index.html#how-it-works
+Title: Overview - Pipecat
+==================================================
+
+Overview - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Get Started Overview Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Get Started Overview Installation & Setup Quickstart Core Concepts Next Steps & Examples Pipecat is an open source Python framework that handles the complex orchestration of AI services, network transport, audio processing, and multimodal interactions. “Multimodal” means you can use any combination of audio, video, images, and/or text in your interactions. And “real-time” means that things are happening quickly enough that it feels conversational—a “back-and-forth” with a bot, not submitting a query and waiting for results. ​ What You Can Build Voice Assistants Natural, real-time conversations with AI using speech recognition and synthesis Interactive Agents Personal coaches and meeting assistants that can understand context and provide guidance Multimodal Apps Applications that combine voice, video, images, and text for rich interactions Creative Tools Storytelling experiences and social companions that engage users Business Solutions Customer intake flows and support bots for automated business processes Complex Flows Structured conversations using Pipecat Flows for managing complex interactions ​ How It Works The flow of interactions in a Pipecat application is typically straightforward: The bot says something The user says something The bot says something The user says something This continues until the conversation naturally ends. While this flow seems simple, making it feel natural requires sophisticated real-time processing. ​ Real-time Processing Pipecat’s pipeline architecture handles both simple voice interactions and complex multimodal processing. Let’s look at how data flows through the system: Voice app Multimodal app 1 Send Audio Transmit and capture streamed audio from the user 2 Transcribe Speech Convert speech to text as the user is talking 3 Process with LLM Generate responses using a large language model 4 Convert to Speech Transform text responses into natural speech 5 Play Audio Stream the audio response back to the user 1 Send Audio Transmit and capture streamed audio from the user 2 Transcribe Speech Convert speech to text as the user is talking 3 Process with LLM Generate responses using a large language model 4 Convert to Speech Transform text responses into natural speech 5 Play Audio Stream the audio response back to the user 1 Send Audio and Video Transmit and capture audio, video, and image inputs simultaneously 2 Process Streams Handle multiple input streams in parallel 3 Model Processing Send combined inputs to multimodal models (like GPT-4V) 4 Generate Outputs Create various outputs (text, images, audio, etc.) 5 Coordinate Presentation Synchronize and present multiple output types In both cases, Pipecat: Processes responses as they stream in Handles multiple input/output modalities concurrently Manages resource allocation and synchronization Coordinates parallel processing tasks This architecture creates fluid, natural interactions without noticeable delays, whether you’re building a simple voice assistant or a complex multimodal application. Pipecat’s pipeline architecture is particularly valuable for managing the complexity of real-time, multimodal interactions, ensuring smooth data flow and proper synchronization regardless of the input/output types involved. Pipecat handles all this complexity for you, letting you focus on building your application rather than managing the underlying infrastructure. ​ Next Steps Ready to build your first Pipecat application? Installation & Setup Prepare your environment and install required dependencies Quickstart Build and run your first Pipecat application Core Concepts Learn about pipelines, frames, and real-time processing Use Cases Explore example implementations and patterns ​ Join Our Community Discord Community Connect with other developers, share your projects, and get support from the Pipecat team. Installation & Setup On this page What You Can Build How It Works Real-time Processing Next Steps Join Our Community Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/pipecat-transport-openai-realtime-webrtc_indexhtml_bf6a944e.txt b/pipecat-transport-openai-realtime-webrtc_indexhtml_bf6a944e.txt
new file mode 100644
index 0000000000000000000000000000000000000000..9a60f346d544e975f8cf803efa0ce09a5c6e2a12
--- /dev/null
+++ b/pipecat-transport-openai-realtime-webrtc_indexhtml_bf6a944e.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/client/android/pipecat-transport-openai-realtime-webrtc/index.html#next-steps
+Title: Overview - Pipecat
+==================================================
+
+Overview - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Get Started Overview Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Get Started Overview Installation & Setup Quickstart Core Concepts Next Steps & Examples Pipecat is an open source Python framework that handles the complex orchestration of AI services, network transport, audio processing, and multimodal interactions. “Multimodal” means you can use any combination of audio, video, images, and/or text in your interactions. And “real-time” means that things are happening quickly enough that it feels conversational—a “back-and-forth” with a bot, not submitting a query and waiting for results. ​ What You Can Build Voice Assistants Natural, real-time conversations with AI using speech recognition and synthesis Interactive Agents Personal coaches and meeting assistants that can understand context and provide guidance Multimodal Apps Applications that combine voice, video, images, and text for rich interactions Creative Tools Storytelling experiences and social companions that engage users Business Solutions Customer intake flows and support bots for automated business processes Complex Flows Structured conversations using Pipecat Flows for managing complex interactions ​ How It Works The flow of interactions in a Pipecat application is typically straightforward: The bot says something The user says something The bot says something The user says something This continues until the conversation naturally ends. While this flow seems simple, making it feel natural requires sophisticated real-time processing. ​ Real-time Processing Pipecat’s pipeline architecture handles both simple voice interactions and complex multimodal processing. Let’s look at how data flows through the system: Voice app Multimodal app 1 Send Audio Transmit and capture streamed audio from the user 2 Transcribe Speech Convert speech to text as the user is talking 3 Process with LLM Generate responses using a large language model 4 Convert to Speech Transform text responses into natural speech 5 Play Audio Stream the audio response back to the user 1 Send Audio Transmit and capture streamed audio from the user 2 Transcribe Speech Convert speech to text as the user is talking 3 Process with LLM Generate responses using a large language model 4 Convert to Speech Transform text responses into natural speech 5 Play Audio Stream the audio response back to the user 1 Send Audio and Video Transmit and capture audio, video, and image inputs simultaneously 2 Process Streams Handle multiple input streams in parallel 3 Model Processing Send combined inputs to multimodal models (like GPT-4V) 4 Generate Outputs Create various outputs (text, images, audio, etc.) 5 Coordinate Presentation Synchronize and present multiple output types In both cases, Pipecat: Processes responses as they stream in Handles multiple input/output modalities concurrently Manages resource allocation and synchronization Coordinates parallel processing tasks This architecture creates fluid, natural interactions without noticeable delays, whether you’re building a simple voice assistant or a complex multimodal application. Pipecat’s pipeline architecture is particularly valuable for managing the complexity of real-time, multimodal interactions, ensuring smooth data flow and proper synchronization regardless of the input/output types involved. Pipecat handles all this complexity for you, letting you focus on building your application rather than managing the underlying infrastructure. ​ Next Steps Ready to build your first Pipecat application? Installation & Setup Prepare your environment and install required dependencies Quickstart Build and run your first Pipecat application Core Concepts Learn about pipelines, frames, and real-time processing Use Cases Explore example implementations and patterns ​ Join Our Community Discord Community Connect with other developers, share your projects, and get support from the Pipecat team. Installation & Setup On this page What You Can Build How It Works Real-time Processing Next Steps Join Our Community Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/pipecat-transport-openai-realtime-webrtc_indexhtml_fe709da3.txt b/pipecat-transport-openai-realtime-webrtc_indexhtml_fe709da3.txt
new file mode 100644
index 0000000000000000000000000000000000000000..5d99ff2be4fc64d5cdcb7ceca2c637637c9765a3
--- /dev/null
+++ b/pipecat-transport-openai-realtime-webrtc_indexhtml_fe709da3.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/client/android/pipecat-transport-openai-realtime-webrtc/index.html#join-our-community
+Title: Overview - Pipecat
+==================================================
+
+Overview - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Get Started Overview Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Get Started Overview Installation & Setup Quickstart Core Concepts Next Steps & Examples Pipecat is an open source Python framework that handles the complex orchestration of AI services, network transport, audio processing, and multimodal interactions. “Multimodal” means you can use any combination of audio, video, images, and/or text in your interactions. And “real-time” means that things are happening quickly enough that it feels conversational—a “back-and-forth” with a bot, not submitting a query and waiting for results. ​ What You Can Build Voice Assistants Natural, real-time conversations with AI using speech recognition and synthesis Interactive Agents Personal coaches and meeting assistants that can understand context and provide guidance Multimodal Apps Applications that combine voice, video, images, and text for rich interactions Creative Tools Storytelling experiences and social companions that engage users Business Solutions Customer intake flows and support bots for automated business processes Complex Flows Structured conversations using Pipecat Flows for managing complex interactions ​ How It Works The flow of interactions in a Pipecat application is typically straightforward: The bot says something The user says something The bot says something The user says something This continues until the conversation naturally ends. While this flow seems simple, making it feel natural requires sophisticated real-time processing. ​ Real-time Processing Pipecat’s pipeline architecture handles both simple voice interactions and complex multimodal processing. Let’s look at how data flows through the system: Voice app Multimodal app 1 Send Audio Transmit and capture streamed audio from the user 2 Transcribe Speech Convert speech to text as the user is talking 3 Process with LLM Generate responses using a large language model 4 Convert to Speech Transform text responses into natural speech 5 Play Audio Stream the audio response back to the user 1 Send Audio Transmit and capture streamed audio from the user 2 Transcribe Speech Convert speech to text as the user is talking 3 Process with LLM Generate responses using a large language model 4 Convert to Speech Transform text responses into natural speech 5 Play Audio Stream the audio response back to the user 1 Send Audio and Video Transmit and capture audio, video, and image inputs simultaneously 2 Process Streams Handle multiple input streams in parallel 3 Model Processing Send combined inputs to multimodal models (like GPT-4V) 4 Generate Outputs Create various outputs (text, images, audio, etc.) 5 Coordinate Presentation Synchronize and present multiple output types In both cases, Pipecat: Processes responses as they stream in Handles multiple input/output modalities concurrently Manages resource allocation and synchronization Coordinates parallel processing tasks This architecture creates fluid, natural interactions without noticeable delays, whether you’re building a simple voice assistant or a complex multimodal application. Pipecat’s pipeline architecture is particularly valuable for managing the complexity of real-time, multimodal interactions, ensuring smooth data flow and proper synchronization regardless of the input/output types involved. Pipecat handles all this complexity for you, letting you focus on building your application rather than managing the underlying infrastructure. ​ Next Steps Ready to build your first Pipecat application? Installation & Setup Prepare your environment and install required dependencies Quickstart Build and run your first Pipecat application Core Concepts Learn about pipelines, frames, and real-time processing Use Cases Explore example implementations and patterns ​ Join Our Community Discord Community Connect with other developers, share your projects, and get support from the Pipecat team. Installation & Setup On this page What You Can Build How It Works Real-time Processing Next Steps Join Our Community Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/pipeline_heartbeats_549239c2.txt b/pipeline_heartbeats_549239c2.txt
new file mode 100644
index 0000000000000000000000000000000000000000..785f8f288ec657485ad273b53154a53f089a6bbd
--- /dev/null
+++ b/pipeline_heartbeats_549239c2.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/pipeline/heartbeats#configuration
+Title: Pipeline Heartbeats - Pipecat
+==================================================
+
+Pipeline Heartbeats - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Pipeline Pipeline Heartbeats Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview Pipeline heartbeats provide a way to monitor the health of your pipeline by sending periodic heartbeat frames through the system. When enabled, the pipeline will send heartbeat frames every second and monitor their progress through the pipeline. ​ Enabling Heartbeats Heartbeats can be enabled by setting enable_heartbeats to True in the PipelineParams : Copy Ask AI from pipecat.pipeline.task import PipelineParams, PipelineTask pipeline = Pipeline([ ... ]) params = params = PipelineParams( enable_heartbeats = True ) task = PipelineTask(pipeline, params) ​ How It Works When heartbeats are enabled: The pipeline sends a HeartbeatFrame every second The frame traverses through all processors in the pipeline, from source to sink The pipeline monitors how long it takes for heartbeat frames to complete their journey If a heartbeat frame isn’t received within 5 seconds, a warning is logged ​ Monitoring Output The system will log: Trace-level logs showing heartbeat processing time Warning messages if heartbeats aren’t received within the monitoring window Example warning message: Copy Ask AI WARNING PipelineTask#1: heartbeat frame not received for more than 5.0 seconds ​ Use Cases Heartbeat monitoring is useful for: Detecting pipeline stalls or blockages Monitoring processing latency through the pipeline Identifying performance issues in specific processors Ensuring the pipeline remains responsive ​ Configuration The heartbeat system uses two key timing constants: HEARTBEAT_SECONDS = 1.0 - Interval between heartbeat frames HEARTBEAT_MONITOR_SECONDS = 10.0 - Time before warning if no heartbeat received These values are currently fixed but may be configurable in future versions. Pipeline Idle Detection ParallelPipeline On this page Overview Enabling Heartbeats How It Works Monitoring Output Use Cases Configuration Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/pipeline_heartbeats_d8bb0f46.txt b/pipeline_heartbeats_d8bb0f46.txt
new file mode 100644
index 0000000000000000000000000000000000000000..b160a16dfc1e4572c9848781c96e51e3e4143026
--- /dev/null
+++ b/pipeline_heartbeats_d8bb0f46.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/pipeline/heartbeats
+Title: Pipeline Heartbeats - Pipecat
+==================================================
+
+Pipeline Heartbeats - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Pipeline Pipeline Heartbeats Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview Pipeline heartbeats provide a way to monitor the health of your pipeline by sending periodic heartbeat frames through the system. When enabled, the pipeline will send heartbeat frames every second and monitor their progress through the pipeline. ​ Enabling Heartbeats Heartbeats can be enabled by setting enable_heartbeats to True in the PipelineParams : Copy Ask AI from pipecat.pipeline.task import PipelineParams, PipelineTask pipeline = Pipeline([ ... ]) params = params = PipelineParams( enable_heartbeats = True ) task = PipelineTask(pipeline, params) ​ How It Works When heartbeats are enabled: The pipeline sends a HeartbeatFrame every second The frame traverses through all processors in the pipeline, from source to sink The pipeline monitors how long it takes for heartbeat frames to complete their journey If a heartbeat frame isn’t received within 5 seconds, a warning is logged ​ Monitoring Output The system will log: Trace-level logs showing heartbeat processing time Warning messages if heartbeats aren’t received within the monitoring window Example warning message: Copy Ask AI WARNING PipelineTask#1: heartbeat frame not received for more than 5.0 seconds ​ Use Cases Heartbeat monitoring is useful for: Detecting pipeline stalls or blockages Monitoring processing latency through the pipeline Identifying performance issues in specific processors Ensuring the pipeline remains responsive ​ Configuration The heartbeat system uses two key timing constants: HEARTBEAT_SECONDS = 1.0 - Interval between heartbeat frames HEARTBEAT_MONITOR_SECONDS = 10.0 - Time before warning if no heartbeat received These values are currently fixed but may be configurable in future versions. Pipeline Idle Detection ParallelPipeline On this page Overview Enabling Heartbeats How It Works Monitoring Output Use Cases Configuration Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/pipeline_heartbeats_f35f7e52.txt b/pipeline_heartbeats_f35f7e52.txt
new file mode 100644
index 0000000000000000000000000000000000000000..42eda3286175359f521a6dcfeca6feb9e0d289ae
--- /dev/null
+++ b/pipeline_heartbeats_f35f7e52.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/pipeline/heartbeats#how-it-works
+Title: Pipeline Heartbeats - Pipecat
+==================================================
+
+Pipeline Heartbeats - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Pipeline Pipeline Heartbeats Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview Pipeline heartbeats provide a way to monitor the health of your pipeline by sending periodic heartbeat frames through the system. When enabled, the pipeline will send heartbeat frames every second and monitor their progress through the pipeline. ​ Enabling Heartbeats Heartbeats can be enabled by setting enable_heartbeats to True in the PipelineParams : Copy Ask AI from pipecat.pipeline.task import PipelineParams, PipelineTask pipeline = Pipeline([ ... ]) params = params = PipelineParams( enable_heartbeats = True ) task = PipelineTask(pipeline, params) ​ How It Works When heartbeats are enabled: The pipeline sends a HeartbeatFrame every second The frame traverses through all processors in the pipeline, from source to sink The pipeline monitors how long it takes for heartbeat frames to complete their journey If a heartbeat frame isn’t received within 5 seconds, a warning is logged ​ Monitoring Output The system will log: Trace-level logs showing heartbeat processing time Warning messages if heartbeats aren’t received within the monitoring window Example warning message: Copy Ask AI WARNING PipelineTask#1: heartbeat frame not received for more than 5.0 seconds ​ Use Cases Heartbeat monitoring is useful for: Detecting pipeline stalls or blockages Monitoring processing latency through the pipeline Identifying performance issues in specific processors Ensuring the pipeline remains responsive ​ Configuration The heartbeat system uses two key timing constants: HEARTBEAT_SECONDS = 1.0 - Interval between heartbeat frames HEARTBEAT_MONITOR_SECONDS = 10.0 - Time before warning if no heartbeat received These values are currently fixed but may be configurable in future versions. Pipeline Idle Detection ParallelPipeline On this page Overview Enabling Heartbeats How It Works Monitoring Output Use Cases Configuration Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/pipeline_parallel-pipeline_4c87f111.txt b/pipeline_parallel-pipeline_4c87f111.txt
new file mode 100644
index 0000000000000000000000000000000000000000..fa172af57d8f28418cc7846710a02da3a839b4c7
--- /dev/null
+++ b/pipeline_parallel-pipeline_4c87f111.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/pipeline/parallel-pipeline#cross-branch-communication
+Title: ParallelPipeline - Pipecat
+==================================================
+
+ParallelPipeline - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Pipeline ParallelPipeline Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview ParallelPipeline allows you to create multiple independent processing branches that run simultaneously, sharing input and coordinating output. It’s particularly useful for multi-agent systems, parallel stream processing, and creating redundant service paths. Each branch receives the same downstream frames, processes them independently, and the results are merged back into a single stream. System frames (like StartFrame and EndFrame ) are synchronized across all branches. ​ Constructor Parameters ​ *args List[List[FrameProcessor]] required Multiple lists of processors, where each list defines a parallel branch. All branches execute simultaneously when frames flow through the pipeline. ​ Usage Examples ​ Multi-Agent Conversation Create a conversation with two AI agents that can interact with the user independently: Copy Ask AI pipeline = Pipeline([ transport.input(), ParallelPipeline( # Agent 1: Customer service representative [ stt_1, context_aggregator.user_a(), llm_agent_1, tts_agent_1, ], # Agent 2: Technical specialist [ stt_2, context_aggregator.user_b(), llm_agent_2, tts_agent_2, ] ), transport.output(), ]) ​ Redundant Services with Failover Set up redundant services with automatic failover: Copy Ask AI pipeline = Pipeline([ transport.input(), stt, ParallelPipeline( # Primary LLM service [ gate_primary, primary_llm, error_detector, ], # Backup LLM service (used only if primary fails) [ gate_backup, backup_llm, fallback_processor, ] ), tts, transport.output(), ]) ​ Cross-Branch Communication Using Producer/Consumer processors to share data between branches: Copy Ask AI # Create producer/consumer pair for cross-branch communication frame_producer = ProducerProcessor( filter = is_important_frame) frame_consumer = ConsumerProcessor( producer = frame_producer) pipeline = Pipeline([ transport.input(), ParallelPipeline( # Branch that generates important frames [ stt, llm, tts, frame_producer, # Share frames with other branch ], # Branch that consumes those frames [ frame_consumer, # Receive frames from other branch llm, # Speech to Speech LLM (audio in) ] ), transport.output(), ]) ​ How It Works ParallelPipeline adds special source and sink processors to each branch System frames (like StartFrame and EndFrame ) are sent to all branches Other frames flow downstream to all branch sources Results from each branch are collected at the sinks The pipeline ensures EndFrame s are only passed through after all branches complete Pipeline Heartbeats On this page Overview Constructor Parameters Usage Examples Multi-Agent Conversation Redundant Services with Failover Cross-Branch Communication How It Works Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/pipeline_parallel-pipeline_56598810.txt b/pipeline_parallel-pipeline_56598810.txt
new file mode 100644
index 0000000000000000000000000000000000000000..d81f7bc4b988fd23d5c2d52b0cfc4d163318eeb1
--- /dev/null
+++ b/pipeline_parallel-pipeline_56598810.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/pipeline/parallel-pipeline#overview
+Title: ParallelPipeline - Pipecat
+==================================================
+
+ParallelPipeline - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Pipeline ParallelPipeline Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview ParallelPipeline allows you to create multiple independent processing branches that run simultaneously, sharing input and coordinating output. It’s particularly useful for multi-agent systems, parallel stream processing, and creating redundant service paths. Each branch receives the same downstream frames, processes them independently, and the results are merged back into a single stream. System frames (like StartFrame and EndFrame ) are synchronized across all branches. ​ Constructor Parameters ​ *args List[List[FrameProcessor]] required Multiple lists of processors, where each list defines a parallel branch. All branches execute simultaneously when frames flow through the pipeline. ​ Usage Examples ​ Multi-Agent Conversation Create a conversation with two AI agents that can interact with the user independently: Copy Ask AI pipeline = Pipeline([ transport.input(), ParallelPipeline( # Agent 1: Customer service representative [ stt_1, context_aggregator.user_a(), llm_agent_1, tts_agent_1, ], # Agent 2: Technical specialist [ stt_2, context_aggregator.user_b(), llm_agent_2, tts_agent_2, ] ), transport.output(), ]) ​ Redundant Services with Failover Set up redundant services with automatic failover: Copy Ask AI pipeline = Pipeline([ transport.input(), stt, ParallelPipeline( # Primary LLM service [ gate_primary, primary_llm, error_detector, ], # Backup LLM service (used only if primary fails) [ gate_backup, backup_llm, fallback_processor, ] ), tts, transport.output(), ]) ​ Cross-Branch Communication Using Producer/Consumer processors to share data between branches: Copy Ask AI # Create producer/consumer pair for cross-branch communication frame_producer = ProducerProcessor( filter = is_important_frame) frame_consumer = ConsumerProcessor( producer = frame_producer) pipeline = Pipeline([ transport.input(), ParallelPipeline( # Branch that generates important frames [ stt, llm, tts, frame_producer, # Share frames with other branch ], # Branch that consumes those frames [ frame_consumer, # Receive frames from other branch llm, # Speech to Speech LLM (audio in) ] ), transport.output(), ]) ​ How It Works ParallelPipeline adds special source and sink processors to each branch System frames (like StartFrame and EndFrame ) are sent to all branches Other frames flow downstream to all branch sources Results from each branch are collected at the sinks The pipeline ensures EndFrame s are only passed through after all branches complete Pipeline Heartbeats On this page Overview Constructor Parameters Usage Examples Multi-Agent Conversation Redundant Services with Failover Cross-Branch Communication How It Works Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/pipeline_parallel-pipeline_57bf8e1d.txt b/pipeline_parallel-pipeline_57bf8e1d.txt
new file mode 100644
index 0000000000000000000000000000000000000000..8445a55d677415e8ad9946366a5c438d2e9bdddc
--- /dev/null
+++ b/pipeline_parallel-pipeline_57bf8e1d.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/pipeline/parallel-pipeline#constructor-parameters
+Title: ParallelPipeline - Pipecat
+==================================================
+
+ParallelPipeline - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Pipeline ParallelPipeline Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview ParallelPipeline allows you to create multiple independent processing branches that run simultaneously, sharing input and coordinating output. It’s particularly useful for multi-agent systems, parallel stream processing, and creating redundant service paths. Each branch receives the same downstream frames, processes them independently, and the results are merged back into a single stream. System frames (like StartFrame and EndFrame ) are synchronized across all branches. ​ Constructor Parameters ​ *args List[List[FrameProcessor]] required Multiple lists of processors, where each list defines a parallel branch. All branches execute simultaneously when frames flow through the pipeline. ​ Usage Examples ​ Multi-Agent Conversation Create a conversation with two AI agents that can interact with the user independently: Copy Ask AI pipeline = Pipeline([ transport.input(), ParallelPipeline( # Agent 1: Customer service representative [ stt_1, context_aggregator.user_a(), llm_agent_1, tts_agent_1, ], # Agent 2: Technical specialist [ stt_2, context_aggregator.user_b(), llm_agent_2, tts_agent_2, ] ), transport.output(), ]) ​ Redundant Services with Failover Set up redundant services with automatic failover: Copy Ask AI pipeline = Pipeline([ transport.input(), stt, ParallelPipeline( # Primary LLM service [ gate_primary, primary_llm, error_detector, ], # Backup LLM service (used only if primary fails) [ gate_backup, backup_llm, fallback_processor, ] ), tts, transport.output(), ]) ​ Cross-Branch Communication Using Producer/Consumer processors to share data between branches: Copy Ask AI # Create producer/consumer pair for cross-branch communication frame_producer = ProducerProcessor( filter = is_important_frame) frame_consumer = ConsumerProcessor( producer = frame_producer) pipeline = Pipeline([ transport.input(), ParallelPipeline( # Branch that generates important frames [ stt, llm, tts, frame_producer, # Share frames with other branch ], # Branch that consumes those frames [ frame_consumer, # Receive frames from other branch llm, # Speech to Speech LLM (audio in) ] ), transport.output(), ]) ​ How It Works ParallelPipeline adds special source and sink processors to each branch System frames (like StartFrame and EndFrame ) are sent to all branches Other frames flow downstream to all branch sources Results from each branch are collected at the sinks The pipeline ensures EndFrame s are only passed through after all branches complete Pipeline Heartbeats On this page Overview Constructor Parameters Usage Examples Multi-Agent Conversation Redundant Services with Failover Cross-Branch Communication How It Works Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/pipeline_parallel-pipeline_9aee58a2.txt b/pipeline_parallel-pipeline_9aee58a2.txt
new file mode 100644
index 0000000000000000000000000000000000000000..e506ae6418279d006cd1cfb467fad30146738c82
--- /dev/null
+++ b/pipeline_parallel-pipeline_9aee58a2.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/pipeline/parallel-pipeline
+Title: ParallelPipeline - Pipecat
+==================================================
+
+ParallelPipeline - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Pipeline ParallelPipeline Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview ParallelPipeline allows you to create multiple independent processing branches that run simultaneously, sharing input and coordinating output. It’s particularly useful for multi-agent systems, parallel stream processing, and creating redundant service paths. Each branch receives the same downstream frames, processes them independently, and the results are merged back into a single stream. System frames (like StartFrame and EndFrame ) are synchronized across all branches. ​ Constructor Parameters ​ *args List[List[FrameProcessor]] required Multiple lists of processors, where each list defines a parallel branch. All branches execute simultaneously when frames flow through the pipeline. ​ Usage Examples ​ Multi-Agent Conversation Create a conversation with two AI agents that can interact with the user independently: Copy Ask AI pipeline = Pipeline([ transport.input(), ParallelPipeline( # Agent 1: Customer service representative [ stt_1, context_aggregator.user_a(), llm_agent_1, tts_agent_1, ], # Agent 2: Technical specialist [ stt_2, context_aggregator.user_b(), llm_agent_2, tts_agent_2, ] ), transport.output(), ]) ​ Redundant Services with Failover Set up redundant services with automatic failover: Copy Ask AI pipeline = Pipeline([ transport.input(), stt, ParallelPipeline( # Primary LLM service [ gate_primary, primary_llm, error_detector, ], # Backup LLM service (used only if primary fails) [ gate_backup, backup_llm, fallback_processor, ] ), tts, transport.output(), ]) ​ Cross-Branch Communication Using Producer/Consumer processors to share data between branches: Copy Ask AI # Create producer/consumer pair for cross-branch communication frame_producer = ProducerProcessor( filter = is_important_frame) frame_consumer = ConsumerProcessor( producer = frame_producer) pipeline = Pipeline([ transport.input(), ParallelPipeline( # Branch that generates important frames [ stt, llm, tts, frame_producer, # Share frames with other branch ], # Branch that consumes those frames [ frame_consumer, # Receive frames from other branch llm, # Speech to Speech LLM (audio in) ] ), transport.output(), ]) ​ How It Works ParallelPipeline adds special source and sink processors to each branch System frames (like StartFrame and EndFrame ) are sent to all branches Other frames flow downstream to all branch sources Results from each branch are collected at the sinks The pipeline ensures EndFrame s are only passed through after all branches complete Pipeline Heartbeats On this page Overview Constructor Parameters Usage Examples Multi-Agent Conversation Redundant Services with Failover Cross-Branch Communication How It Works Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/pipeline_pipeline-idle-detection_3cfb3ba8.txt b/pipeline_pipeline-idle-detection_3cfb3ba8.txt
new file mode 100644
index 0000000000000000000000000000000000000000..ea59e62760cb6cb20f7b3057ffd98a53b284691c
--- /dev/null
+++ b/pipeline_pipeline-idle-detection_3cfb3ba8.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/pipeline/pipeline-idle-detection
+Title: Pipeline Idle Detection - Pipecat
+==================================================
+
+Pipeline Idle Detection - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Pipeline Pipeline Idle Detection Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview Pipeline idle detection is a feature that monitors activity in your pipeline and can automatically cancel tasks when no meaningful bot interactions are occurring. This helps prevent pipelines from running indefinitely when a conversation has naturally ended but wasn’t properly terminated. ​ How It Works The system monitors specific “activity frames” that indicate the bot is actively engaged in the conversation. By default, these are: BotSpeakingFrame - When the bot is speaking LLMFullResponseEndFrame - When the LLM has completed a response If no activity frames are detected within the configured timeout period (5 minutes by default), the system considers the pipeline idle and can automatically terminate it. Idle detection only starts after the pipeline has begun processing frames. The idle timer resets whenever an activity frame (as specified in idle_timeout_frames ) is received. ​ Configuration You can configure idle detection behavior when creating a PipelineTask : Copy Ask AI from pipecat.pipeline.task import PipelineParams, PipelineTask # Default configuration - cancel after 5 minutes of inactivity task = PipelineTask(pipeline) # Custom configuration task = PipelineTask( pipeline, params = PipelineParams( allow_interruptions = True ), idle_timeout_secs = 600 , # 10 minute timeout idle_timeout_frames = (BotSpeakingFrame,), # Only monitor bot speaking cancel_on_idle_timeout = False , # Don't auto-cancel, just notify ) ​ Configuration Parameters ​ idle_timeout_secs Optional[float] default: "300" Timeout in seconds before considering the pipeline idle. Set to None to disable idle detection. ​ idle_timeout_frames Tuple[Type[Frame], ...] default: "(BotSpeakingFrame, LLMFullResponseEndFrame)" Frame types that should prevent the pipeline from being considered idle. ​ cancel_on_idle_timeout bool default: "True" Whether to automatically cancel the pipeline task when idle timeout is reached. ​ Handling Idle Timeouts You can respond to idle timeout events by adding an event handler: Copy Ask AI @task.event_handler ( "on_idle_timeout" ) async def on_idle_timeout ( task ): logger.info( "Pipeline has been idle for too long" ) # Perform any custom cleanup or logging # Note: If cancel_on_idle_timeout=True, the pipeline will be cancelled after this handler runs ​ Example Implementation Here’s a complete example showing how to configure idle detection with custom handling: Copy Ask AI from pipecat.frames.frames import BotSpeakingFrame, LLMFullResponseEndFrame, TTSSpeakFrame from pipecat.pipeline.runner import PipelineRunner from pipecat.pipeline.task import PipelineParams, PipelineTask # Create pipeline pipeline = Pipeline([ ... ]) # Configure task with custom idle settings task = PipelineTask( pipeline, params = PipelineParams( allow_interruptions = True ), idle_timeout_secs = 180 , # 3 minutes cancel_on_idle_timeout = False # Don't auto-cancel ) # Add event handler for idle timeout @task.event_handler ( "on_idle_timeout" ) async def on_idle_timeout ( task ): logger.info( "Conversation has been idle for 3 minutes" ) # Add a farewell message await task.queue_frame(TTSSpeakFrame( "I haven't heard from you in a while. Goodbye!" )) # Then end the conversation gracefully await task.stop_when_done() runner = PipelineRunner() await runner.run(task) PipelineTask Pipeline Heartbeats On this page Overview How It Works Configuration Configuration Parameters Handling Idle Timeouts Example Implementation Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/pipeline_pipeline-idle-detection_9221d7b0.txt b/pipeline_pipeline-idle-detection_9221d7b0.txt
new file mode 100644
index 0000000000000000000000000000000000000000..b959df24dee9f3b60737431acf0c8a50b00048ad
--- /dev/null
+++ b/pipeline_pipeline-idle-detection_9221d7b0.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/pipeline/pipeline-idle-detection#configuration
+Title: Pipeline Idle Detection - Pipecat
+==================================================
+
+Pipeline Idle Detection - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Pipeline Pipeline Idle Detection Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview Pipeline idle detection is a feature that monitors activity in your pipeline and can automatically cancel tasks when no meaningful bot interactions are occurring. This helps prevent pipelines from running indefinitely when a conversation has naturally ended but wasn’t properly terminated. ​ How It Works The system monitors specific “activity frames” that indicate the bot is actively engaged in the conversation. By default, these are: BotSpeakingFrame - When the bot is speaking LLMFullResponseEndFrame - When the LLM has completed a response If no activity frames are detected within the configured timeout period (5 minutes by default), the system considers the pipeline idle and can automatically terminate it. Idle detection only starts after the pipeline has begun processing frames. The idle timer resets whenever an activity frame (as specified in idle_timeout_frames ) is received. ​ Configuration You can configure idle detection behavior when creating a PipelineTask : Copy Ask AI from pipecat.pipeline.task import PipelineParams, PipelineTask # Default configuration - cancel after 5 minutes of inactivity task = PipelineTask(pipeline) # Custom configuration task = PipelineTask( pipeline, params = PipelineParams( allow_interruptions = True ), idle_timeout_secs = 600 , # 10 minute timeout idle_timeout_frames = (BotSpeakingFrame,), # Only monitor bot speaking cancel_on_idle_timeout = False , # Don't auto-cancel, just notify ) ​ Configuration Parameters ​ idle_timeout_secs Optional[float] default: "300" Timeout in seconds before considering the pipeline idle. Set to None to disable idle detection. ​ idle_timeout_frames Tuple[Type[Frame], ...] default: "(BotSpeakingFrame, LLMFullResponseEndFrame)" Frame types that should prevent the pipeline from being considered idle. ​ cancel_on_idle_timeout bool default: "True" Whether to automatically cancel the pipeline task when idle timeout is reached. ​ Handling Idle Timeouts You can respond to idle timeout events by adding an event handler: Copy Ask AI @task.event_handler ( "on_idle_timeout" ) async def on_idle_timeout ( task ): logger.info( "Pipeline has been idle for too long" ) # Perform any custom cleanup or logging # Note: If cancel_on_idle_timeout=True, the pipeline will be cancelled after this handler runs ​ Example Implementation Here’s a complete example showing how to configure idle detection with custom handling: Copy Ask AI from pipecat.frames.frames import BotSpeakingFrame, LLMFullResponseEndFrame, TTSSpeakFrame from pipecat.pipeline.runner import PipelineRunner from pipecat.pipeline.task import PipelineParams, PipelineTask # Create pipeline pipeline = Pipeline([ ... ]) # Configure task with custom idle settings task = PipelineTask( pipeline, params = PipelineParams( allow_interruptions = True ), idle_timeout_secs = 180 , # 3 minutes cancel_on_idle_timeout = False # Don't auto-cancel ) # Add event handler for idle timeout @task.event_handler ( "on_idle_timeout" ) async def on_idle_timeout ( task ): logger.info( "Conversation has been idle for 3 minutes" ) # Add a farewell message await task.queue_frame(TTSSpeakFrame( "I haven't heard from you in a while. Goodbye!" )) # Then end the conversation gracefully await task.stop_when_done() runner = PipelineRunner() await runner.run(task) PipelineTask Pipeline Heartbeats On this page Overview How It Works Configuration Configuration Parameters Handling Idle Timeouts Example Implementation Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/pipeline_pipeline-idle-detection_9f19f9fc.txt b/pipeline_pipeline-idle-detection_9f19f9fc.txt
new file mode 100644
index 0000000000000000000000000000000000000000..2455ece1efc09b590662dfbd6b48ac1b42ad3921
--- /dev/null
+++ b/pipeline_pipeline-idle-detection_9f19f9fc.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/pipeline/pipeline-idle-detection#how-it-works
+Title: Pipeline Idle Detection - Pipecat
+==================================================
+
+Pipeline Idle Detection - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Pipeline Pipeline Idle Detection Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview Pipeline idle detection is a feature that monitors activity in your pipeline and can automatically cancel tasks when no meaningful bot interactions are occurring. This helps prevent pipelines from running indefinitely when a conversation has naturally ended but wasn’t properly terminated. ​ How It Works The system monitors specific “activity frames” that indicate the bot is actively engaged in the conversation. By default, these are: BotSpeakingFrame - When the bot is speaking LLMFullResponseEndFrame - When the LLM has completed a response If no activity frames are detected within the configured timeout period (5 minutes by default), the system considers the pipeline idle and can automatically terminate it. Idle detection only starts after the pipeline has begun processing frames. The idle timer resets whenever an activity frame (as specified in idle_timeout_frames ) is received. ​ Configuration You can configure idle detection behavior when creating a PipelineTask : Copy Ask AI from pipecat.pipeline.task import PipelineParams, PipelineTask # Default configuration - cancel after 5 minutes of inactivity task = PipelineTask(pipeline) # Custom configuration task = PipelineTask( pipeline, params = PipelineParams( allow_interruptions = True ), idle_timeout_secs = 600 , # 10 minute timeout idle_timeout_frames = (BotSpeakingFrame,), # Only monitor bot speaking cancel_on_idle_timeout = False , # Don't auto-cancel, just notify ) ​ Configuration Parameters ​ idle_timeout_secs Optional[float] default: "300" Timeout in seconds before considering the pipeline idle. Set to None to disable idle detection. ​ idle_timeout_frames Tuple[Type[Frame], ...] default: "(BotSpeakingFrame, LLMFullResponseEndFrame)" Frame types that should prevent the pipeline from being considered idle. ​ cancel_on_idle_timeout bool default: "True" Whether to automatically cancel the pipeline task when idle timeout is reached. ​ Handling Idle Timeouts You can respond to idle timeout events by adding an event handler: Copy Ask AI @task.event_handler ( "on_idle_timeout" ) async def on_idle_timeout ( task ): logger.info( "Pipeline has been idle for too long" ) # Perform any custom cleanup or logging # Note: If cancel_on_idle_timeout=True, the pipeline will be cancelled after this handler runs ​ Example Implementation Here’s a complete example showing how to configure idle detection with custom handling: Copy Ask AI from pipecat.frames.frames import BotSpeakingFrame, LLMFullResponseEndFrame, TTSSpeakFrame from pipecat.pipeline.runner import PipelineRunner from pipecat.pipeline.task import PipelineParams, PipelineTask # Create pipeline pipeline = Pipeline([ ... ]) # Configure task with custom idle settings task = PipelineTask( pipeline, params = PipelineParams( allow_interruptions = True ), idle_timeout_secs = 180 , # 3 minutes cancel_on_idle_timeout = False # Don't auto-cancel ) # Add event handler for idle timeout @task.event_handler ( "on_idle_timeout" ) async def on_idle_timeout ( task ): logger.info( "Conversation has been idle for 3 minutes" ) # Add a farewell message await task.queue_frame(TTSSpeakFrame( "I haven't heard from you in a while. Goodbye!" )) # Then end the conversation gracefully await task.stop_when_done() runner = PipelineRunner() await runner.run(task) PipelineTask Pipeline Heartbeats On this page Overview How It Works Configuration Configuration Parameters Handling Idle Timeouts Example Implementation Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/pipeline_pipeline-params_41fd74e2.txt b/pipeline_pipeline-params_41fd74e2.txt
new file mode 100644
index 0000000000000000000000000000000000000000..e3c1ba0978be6977dc672348a1f2f4feb33ff193
--- /dev/null
+++ b/pipeline_pipeline-params_41fd74e2.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/pipeline/pipeline-params#how-parameters-are-used
+Title: PipelineParams - Pipecat
+==================================================
+
+PipelineParams - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Pipeline PipelineParams Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview The PipelineParams class provides a structured way to configure various aspects of pipeline execution. These parameters control behaviors like audio settings, metrics collection, heartbeat monitoring, and interruption handling. ​ Basic Usage Copy Ask AI from pipecat.pipeline.task import PipelineParams, PipelineTask # Create with default parameters params = PipelineParams() # Or customize specific parameters params = PipelineParams( allow_interruptions = True , audio_in_sample_rate = 16000 , enable_metrics = True ) # Pass to PipelineTask pipeline = Pipeline([ ... ]) task = PipelineTask(pipeline, params = params) ​ Available Parameters ​ allow_interruptions bool default: "False" Whether to allow pipeline interruptions. When enabled, a user’s speech will immediately interrupt the bot’s response. ​ audio_in_sample_rate int default: "16000" Input audio sample rate in Hz. Setting the audio_in_sample_rate as a PipelineParam sets the input sample rate for all corresponding services in the pipeline. ​ audio_out_sample_rate int default: "24000" Output audio sample rate in Hz. Setting the audio_out_sample_rate as a PipelineParam sets the output sample rate for all corresponding services in the pipeline. ​ enable_heartbeats bool default: "False" Whether to enable heartbeat monitoring to detect pipeline stalls. See Heartbeats for details. ​ heartbeats_period_secs float default: "1.0" Period between heartbeats in seconds (when heartbeats are enabled). ​ enable_metrics bool default: "False" Whether to enable metrics collection for pipeline performance. ​ enable_usage_metrics bool default: "False" Whether to enable usage metrics tracking. ​ report_only_initial_ttfb bool default: "False" Whether to report only initial time to first byte metric. ​ send_initial_empty_metrics bool default: "True" Whether to send initial empty metrics frame at pipeline start. ​ start_metadata Dict[str, Any] default: "{}" Additional metadata to include in the StartFrame. ​ Common Configurations ​ Audio Processing Configuration You can set the audio input and output sample rates in the PipelineParams to set the sample rate for all input and output services in the pipeline. This acts as a convenience to avoid setting the sample rate for each service individually. Note, if services are set individually, they will supersede the values set in PipelineParams . Copy Ask AI params = PipelineParams( audio_in_sample_rate = 8000 , # Lower quality input audio audio_out_sample_rate = 8000 # High quality output audio ) ​ Performance Monitoring Configuration Pipeline heartbeats provide a way to monitor the health of your pipeline by sending periodic heartbeat frames through the system. When enabled, the pipeline will send heartbeat frames every second and monitor their progress through the pipeline. Copy Ask AI params = PipelineParams( enable_heartbeats = True , heartbeats_period_secs = 2.0 , # Send heartbeats every 2 seconds enable_metrics = True ) ​ How Parameters Are Used The parameters you set in PipelineParams are passed to various components of the pipeline: StartFrame : Many parameters are included in the StartFrame that initializes the pipeline Metrics Collection : Metrics settings configure what performance data is gathered Heartbeat Monitoring : Controls the pipeline’s health monitoring system Audio Processing : Sample rates affect how audio is processed throughout the pipeline ​ Complete Example Copy Ask AI from pipecat.frames.frames import TTSSpeakFrame from pipecat.observers.file_observer import FileObserver from pipecat.pipeline.task import PipelineParams, PipelineTask from pipecat.pipeline.runner import PipelineRunner # Create comprehensive parameters params = PipelineParams( allow_interruptions = True , audio_in_sample_rate = 8000 , audio_out_sample_rate = 8000 , enable_heartbeats = True , enable_metrics = True , enable_usage_metrics = True , heartbeats_period_secs = 1.0 , report_only_initial_ttfb = False , start_metadata = { "conversation_id" : "conv-123" , "session_data" : { "user_id" : "user-456" , "start_time" : "2023-10-25T14:30:00Z" } } ) # Create pipeline and task pipeline = Pipeline([ ... ]) task = PipelineTask( pipeline, params = params, observers = [FileObserver( "pipeline_logs.jsonl" )] ) # Run the pipeline runner = PipelineRunner() await runner.run(task) ​ Additional Information Parameters are immutable once the pipeline starts The start_metadata dictionary can contain any serializable data For metrics collection to work properly, enable_metrics must be set to True Pipecat Flows PipelineTask On this page Overview Basic Usage Available Parameters Common Configurations Audio Processing Configuration Performance Monitoring Configuration How Parameters Are Used Complete Example Additional Information Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/pipeline_pipeline-params_48ef8fa3.txt b/pipeline_pipeline-params_48ef8fa3.txt
new file mode 100644
index 0000000000000000000000000000000000000000..53174cecad108af5464c06bef9e2550a9f348636
--- /dev/null
+++ b/pipeline_pipeline-params_48ef8fa3.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/pipeline/pipeline-params#param-enable-usage-metrics
+Title: PipelineParams - Pipecat
+==================================================
+
+PipelineParams - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Pipeline PipelineParams Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview The PipelineParams class provides a structured way to configure various aspects of pipeline execution. These parameters control behaviors like audio settings, metrics collection, heartbeat monitoring, and interruption handling. ​ Basic Usage Copy Ask AI from pipecat.pipeline.task import PipelineParams, PipelineTask # Create with default parameters params = PipelineParams() # Or customize specific parameters params = PipelineParams( allow_interruptions = True , audio_in_sample_rate = 16000 , enable_metrics = True ) # Pass to PipelineTask pipeline = Pipeline([ ... ]) task = PipelineTask(pipeline, params = params) ​ Available Parameters ​ allow_interruptions bool default: "False" Whether to allow pipeline interruptions. When enabled, a user’s speech will immediately interrupt the bot’s response. ​ audio_in_sample_rate int default: "16000" Input audio sample rate in Hz. Setting the audio_in_sample_rate as a PipelineParam sets the input sample rate for all corresponding services in the pipeline. ​ audio_out_sample_rate int default: "24000" Output audio sample rate in Hz. Setting the audio_out_sample_rate as a PipelineParam sets the output sample rate for all corresponding services in the pipeline. ​ enable_heartbeats bool default: "False" Whether to enable heartbeat monitoring to detect pipeline stalls. See Heartbeats for details. ​ heartbeats_period_secs float default: "1.0" Period between heartbeats in seconds (when heartbeats are enabled). ​ enable_metrics bool default: "False" Whether to enable metrics collection for pipeline performance. ​ enable_usage_metrics bool default: "False" Whether to enable usage metrics tracking. ​ report_only_initial_ttfb bool default: "False" Whether to report only initial time to first byte metric. ​ send_initial_empty_metrics bool default: "True" Whether to send initial empty metrics frame at pipeline start. ​ start_metadata Dict[str, Any] default: "{}" Additional metadata to include in the StartFrame. ​ Common Configurations ​ Audio Processing Configuration You can set the audio input and output sample rates in the PipelineParams to set the sample rate for all input and output services in the pipeline. This acts as a convenience to avoid setting the sample rate for each service individually. Note, if services are set individually, they will supersede the values set in PipelineParams . Copy Ask AI params = PipelineParams( audio_in_sample_rate = 8000 , # Lower quality input audio audio_out_sample_rate = 8000 # High quality output audio ) ​ Performance Monitoring Configuration Pipeline heartbeats provide a way to monitor the health of your pipeline by sending periodic heartbeat frames through the system. When enabled, the pipeline will send heartbeat frames every second and monitor their progress through the pipeline. Copy Ask AI params = PipelineParams( enable_heartbeats = True , heartbeats_period_secs = 2.0 , # Send heartbeats every 2 seconds enable_metrics = True ) ​ How Parameters Are Used The parameters you set in PipelineParams are passed to various components of the pipeline: StartFrame : Many parameters are included in the StartFrame that initializes the pipeline Metrics Collection : Metrics settings configure what performance data is gathered Heartbeat Monitoring : Controls the pipeline’s health monitoring system Audio Processing : Sample rates affect how audio is processed throughout the pipeline ​ Complete Example Copy Ask AI from pipecat.frames.frames import TTSSpeakFrame from pipecat.observers.file_observer import FileObserver from pipecat.pipeline.task import PipelineParams, PipelineTask from pipecat.pipeline.runner import PipelineRunner # Create comprehensive parameters params = PipelineParams( allow_interruptions = True , audio_in_sample_rate = 8000 , audio_out_sample_rate = 8000 , enable_heartbeats = True , enable_metrics = True , enable_usage_metrics = True , heartbeats_period_secs = 1.0 , report_only_initial_ttfb = False , start_metadata = { "conversation_id" : "conv-123" , "session_data" : { "user_id" : "user-456" , "start_time" : "2023-10-25T14:30:00Z" } } ) # Create pipeline and task pipeline = Pipeline([ ... ]) task = PipelineTask( pipeline, params = params, observers = [FileObserver( "pipeline_logs.jsonl" )] ) # Run the pipeline runner = PipelineRunner() await runner.run(task) ​ Additional Information Parameters are immutable once the pipeline starts The start_metadata dictionary can contain any serializable data For metrics collection to work properly, enable_metrics must be set to True Pipecat Flows PipelineTask On this page Overview Basic Usage Available Parameters Common Configurations Audio Processing Configuration Performance Monitoring Configuration How Parameters Are Used Complete Example Additional Information Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/pipeline_pipeline-params_797979bd.txt b/pipeline_pipeline-params_797979bd.txt
new file mode 100644
index 0000000000000000000000000000000000000000..b9387c950b44c856d88fde80810619ef38662885
--- /dev/null
+++ b/pipeline_pipeline-params_797979bd.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/pipeline/pipeline-params#common-configurations
+Title: PipelineParams - Pipecat
+==================================================
+
+PipelineParams - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Pipeline PipelineParams Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview The PipelineParams class provides a structured way to configure various aspects of pipeline execution. These parameters control behaviors like audio settings, metrics collection, heartbeat monitoring, and interruption handling. ​ Basic Usage Copy Ask AI from pipecat.pipeline.task import PipelineParams, PipelineTask # Create with default parameters params = PipelineParams() # Or customize specific parameters params = PipelineParams( allow_interruptions = True , audio_in_sample_rate = 16000 , enable_metrics = True ) # Pass to PipelineTask pipeline = Pipeline([ ... ]) task = PipelineTask(pipeline, params = params) ​ Available Parameters ​ allow_interruptions bool default: "False" Whether to allow pipeline interruptions. When enabled, a user’s speech will immediately interrupt the bot’s response. ​ audio_in_sample_rate int default: "16000" Input audio sample rate in Hz. Setting the audio_in_sample_rate as a PipelineParam sets the input sample rate for all corresponding services in the pipeline. ​ audio_out_sample_rate int default: "24000" Output audio sample rate in Hz. Setting the audio_out_sample_rate as a PipelineParam sets the output sample rate for all corresponding services in the pipeline. ​ enable_heartbeats bool default: "False" Whether to enable heartbeat monitoring to detect pipeline stalls. See Heartbeats for details. ​ heartbeats_period_secs float default: "1.0" Period between heartbeats in seconds (when heartbeats are enabled). ​ enable_metrics bool default: "False" Whether to enable metrics collection for pipeline performance. ​ enable_usage_metrics bool default: "False" Whether to enable usage metrics tracking. ​ report_only_initial_ttfb bool default: "False" Whether to report only initial time to first byte metric. ​ send_initial_empty_metrics bool default: "True" Whether to send initial empty metrics frame at pipeline start. ​ start_metadata Dict[str, Any] default: "{}" Additional metadata to include in the StartFrame. ​ Common Configurations ​ Audio Processing Configuration You can set the audio input and output sample rates in the PipelineParams to set the sample rate for all input and output services in the pipeline. This acts as a convenience to avoid setting the sample rate for each service individually. Note, if services are set individually, they will supersede the values set in PipelineParams . Copy Ask AI params = PipelineParams( audio_in_sample_rate = 8000 , # Lower quality input audio audio_out_sample_rate = 8000 # High quality output audio ) ​ Performance Monitoring Configuration Pipeline heartbeats provide a way to monitor the health of your pipeline by sending periodic heartbeat frames through the system. When enabled, the pipeline will send heartbeat frames every second and monitor their progress through the pipeline. Copy Ask AI params = PipelineParams( enable_heartbeats = True , heartbeats_period_secs = 2.0 , # Send heartbeats every 2 seconds enable_metrics = True ) ​ How Parameters Are Used The parameters you set in PipelineParams are passed to various components of the pipeline: StartFrame : Many parameters are included in the StartFrame that initializes the pipeline Metrics Collection : Metrics settings configure what performance data is gathered Heartbeat Monitoring : Controls the pipeline’s health monitoring system Audio Processing : Sample rates affect how audio is processed throughout the pipeline ​ Complete Example Copy Ask AI from pipecat.frames.frames import TTSSpeakFrame from pipecat.observers.file_observer import FileObserver from pipecat.pipeline.task import PipelineParams, PipelineTask from pipecat.pipeline.runner import PipelineRunner # Create comprehensive parameters params = PipelineParams( allow_interruptions = True , audio_in_sample_rate = 8000 , audio_out_sample_rate = 8000 , enable_heartbeats = True , enable_metrics = True , enable_usage_metrics = True , heartbeats_period_secs = 1.0 , report_only_initial_ttfb = False , start_metadata = { "conversation_id" : "conv-123" , "session_data" : { "user_id" : "user-456" , "start_time" : "2023-10-25T14:30:00Z" } } ) # Create pipeline and task pipeline = Pipeline([ ... ]) task = PipelineTask( pipeline, params = params, observers = [FileObserver( "pipeline_logs.jsonl" )] ) # Run the pipeline runner = PipelineRunner() await runner.run(task) ​ Additional Information Parameters are immutable once the pipeline starts The start_metadata dictionary can contain any serializable data For metrics collection to work properly, enable_metrics must be set to True Pipecat Flows PipelineTask On this page Overview Basic Usage Available Parameters Common Configurations Audio Processing Configuration Performance Monitoring Configuration How Parameters Are Used Complete Example Additional Information Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/pipeline_pipeline-params_7e321bb5.txt b/pipeline_pipeline-params_7e321bb5.txt
new file mode 100644
index 0000000000000000000000000000000000000000..7f3f056724ee4008795c1d789d032ec6d27d72dc
--- /dev/null
+++ b/pipeline_pipeline-params_7e321bb5.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/pipeline/pipeline-params#param-report-only-initial-ttfb
+Title: PipelineParams - Pipecat
+==================================================
+
+PipelineParams - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Pipeline PipelineParams Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview The PipelineParams class provides a structured way to configure various aspects of pipeline execution. These parameters control behaviors like audio settings, metrics collection, heartbeat monitoring, and interruption handling. ​ Basic Usage Copy Ask AI from pipecat.pipeline.task import PipelineParams, PipelineTask # Create with default parameters params = PipelineParams() # Or customize specific parameters params = PipelineParams( allow_interruptions = True , audio_in_sample_rate = 16000 , enable_metrics = True ) # Pass to PipelineTask pipeline = Pipeline([ ... ]) task = PipelineTask(pipeline, params = params) ​ Available Parameters ​ allow_interruptions bool default: "False" Whether to allow pipeline interruptions. When enabled, a user’s speech will immediately interrupt the bot’s response. ​ audio_in_sample_rate int default: "16000" Input audio sample rate in Hz. Setting the audio_in_sample_rate as a PipelineParam sets the input sample rate for all corresponding services in the pipeline. ​ audio_out_sample_rate int default: "24000" Output audio sample rate in Hz. Setting the audio_out_sample_rate as a PipelineParam sets the output sample rate for all corresponding services in the pipeline. ​ enable_heartbeats bool default: "False" Whether to enable heartbeat monitoring to detect pipeline stalls. See Heartbeats for details. ​ heartbeats_period_secs float default: "1.0" Period between heartbeats in seconds (when heartbeats are enabled). ​ enable_metrics bool default: "False" Whether to enable metrics collection for pipeline performance. ​ enable_usage_metrics bool default: "False" Whether to enable usage metrics tracking. ​ report_only_initial_ttfb bool default: "False" Whether to report only initial time to first byte metric. ​ send_initial_empty_metrics bool default: "True" Whether to send initial empty metrics frame at pipeline start. ​ start_metadata Dict[str, Any] default: "{}" Additional metadata to include in the StartFrame. ​ Common Configurations ​ Audio Processing Configuration You can set the audio input and output sample rates in the PipelineParams to set the sample rate for all input and output services in the pipeline. This acts as a convenience to avoid setting the sample rate for each service individually. Note, if services are set individually, they will supersede the values set in PipelineParams . Copy Ask AI params = PipelineParams( audio_in_sample_rate = 8000 , # Lower quality input audio audio_out_sample_rate = 8000 # High quality output audio ) ​ Performance Monitoring Configuration Pipeline heartbeats provide a way to monitor the health of your pipeline by sending periodic heartbeat frames through the system. When enabled, the pipeline will send heartbeat frames every second and monitor their progress through the pipeline. Copy Ask AI params = PipelineParams( enable_heartbeats = True , heartbeats_period_secs = 2.0 , # Send heartbeats every 2 seconds enable_metrics = True ) ​ How Parameters Are Used The parameters you set in PipelineParams are passed to various components of the pipeline: StartFrame : Many parameters are included in the StartFrame that initializes the pipeline Metrics Collection : Metrics settings configure what performance data is gathered Heartbeat Monitoring : Controls the pipeline’s health monitoring system Audio Processing : Sample rates affect how audio is processed throughout the pipeline ​ Complete Example Copy Ask AI from pipecat.frames.frames import TTSSpeakFrame from pipecat.observers.file_observer import FileObserver from pipecat.pipeline.task import PipelineParams, PipelineTask from pipecat.pipeline.runner import PipelineRunner # Create comprehensive parameters params = PipelineParams( allow_interruptions = True , audio_in_sample_rate = 8000 , audio_out_sample_rate = 8000 , enable_heartbeats = True , enable_metrics = True , enable_usage_metrics = True , heartbeats_period_secs = 1.0 , report_only_initial_ttfb = False , start_metadata = { "conversation_id" : "conv-123" , "session_data" : { "user_id" : "user-456" , "start_time" : "2023-10-25T14:30:00Z" } } ) # Create pipeline and task pipeline = Pipeline([ ... ]) task = PipelineTask( pipeline, params = params, observers = [FileObserver( "pipeline_logs.jsonl" )] ) # Run the pipeline runner = PipelineRunner() await runner.run(task) ​ Additional Information Parameters are immutable once the pipeline starts The start_metadata dictionary can contain any serializable data For metrics collection to work properly, enable_metrics must be set to True Pipecat Flows PipelineTask On this page Overview Basic Usage Available Parameters Common Configurations Audio Processing Configuration Performance Monitoring Configuration How Parameters Are Used Complete Example Additional Information Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/pipeline_pipeline-params_c0e07e7f.txt b/pipeline_pipeline-params_c0e07e7f.txt
new file mode 100644
index 0000000000000000000000000000000000000000..26574c8a21dd7fca0f074efa0004f2fcdac5a10e
--- /dev/null
+++ b/pipeline_pipeline-params_c0e07e7f.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/pipeline/pipeline-params#param-heartbeats-period-secs
+Title: PipelineParams - Pipecat
+==================================================
+
+PipelineParams - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Pipeline PipelineParams Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview The PipelineParams class provides a structured way to configure various aspects of pipeline execution. These parameters control behaviors like audio settings, metrics collection, heartbeat monitoring, and interruption handling. ​ Basic Usage Copy Ask AI from pipecat.pipeline.task import PipelineParams, PipelineTask # Create with default parameters params = PipelineParams() # Or customize specific parameters params = PipelineParams( allow_interruptions = True , audio_in_sample_rate = 16000 , enable_metrics = True ) # Pass to PipelineTask pipeline = Pipeline([ ... ]) task = PipelineTask(pipeline, params = params) ​ Available Parameters ​ allow_interruptions bool default: "False" Whether to allow pipeline interruptions. When enabled, a user’s speech will immediately interrupt the bot’s response. ​ audio_in_sample_rate int default: "16000" Input audio sample rate in Hz. Setting the audio_in_sample_rate as a PipelineParam sets the input sample rate for all corresponding services in the pipeline. ​ audio_out_sample_rate int default: "24000" Output audio sample rate in Hz. Setting the audio_out_sample_rate as a PipelineParam sets the output sample rate for all corresponding services in the pipeline. ​ enable_heartbeats bool default: "False" Whether to enable heartbeat monitoring to detect pipeline stalls. See Heartbeats for details. ​ heartbeats_period_secs float default: "1.0" Period between heartbeats in seconds (when heartbeats are enabled). ​ enable_metrics bool default: "False" Whether to enable metrics collection for pipeline performance. ​ enable_usage_metrics bool default: "False" Whether to enable usage metrics tracking. ​ report_only_initial_ttfb bool default: "False" Whether to report only initial time to first byte metric. ​ send_initial_empty_metrics bool default: "True" Whether to send initial empty metrics frame at pipeline start. ​ start_metadata Dict[str, Any] default: "{}" Additional metadata to include in the StartFrame. ​ Common Configurations ​ Audio Processing Configuration You can set the audio input and output sample rates in the PipelineParams to set the sample rate for all input and output services in the pipeline. This acts as a convenience to avoid setting the sample rate for each service individually. Note, if services are set individually, they will supersede the values set in PipelineParams . Copy Ask AI params = PipelineParams( audio_in_sample_rate = 8000 , # Lower quality input audio audio_out_sample_rate = 8000 # High quality output audio ) ​ Performance Monitoring Configuration Pipeline heartbeats provide a way to monitor the health of your pipeline by sending periodic heartbeat frames through the system. When enabled, the pipeline will send heartbeat frames every second and monitor their progress through the pipeline. Copy Ask AI params = PipelineParams( enable_heartbeats = True , heartbeats_period_secs = 2.0 , # Send heartbeats every 2 seconds enable_metrics = True ) ​ How Parameters Are Used The parameters you set in PipelineParams are passed to various components of the pipeline: StartFrame : Many parameters are included in the StartFrame that initializes the pipeline Metrics Collection : Metrics settings configure what performance data is gathered Heartbeat Monitoring : Controls the pipeline’s health monitoring system Audio Processing : Sample rates affect how audio is processed throughout the pipeline ​ Complete Example Copy Ask AI from pipecat.frames.frames import TTSSpeakFrame from pipecat.observers.file_observer import FileObserver from pipecat.pipeline.task import PipelineParams, PipelineTask from pipecat.pipeline.runner import PipelineRunner # Create comprehensive parameters params = PipelineParams( allow_interruptions = True , audio_in_sample_rate = 8000 , audio_out_sample_rate = 8000 , enable_heartbeats = True , enable_metrics = True , enable_usage_metrics = True , heartbeats_period_secs = 1.0 , report_only_initial_ttfb = False , start_metadata = { "conversation_id" : "conv-123" , "session_data" : { "user_id" : "user-456" , "start_time" : "2023-10-25T14:30:00Z" } } ) # Create pipeline and task pipeline = Pipeline([ ... ]) task = PipelineTask( pipeline, params = params, observers = [FileObserver( "pipeline_logs.jsonl" )] ) # Run the pipeline runner = PipelineRunner() await runner.run(task) ​ Additional Information Parameters are immutable once the pipeline starts The start_metadata dictionary can contain any serializable data For metrics collection to work properly, enable_metrics must be set to True Pipecat Flows PipelineTask On this page Overview Basic Usage Available Parameters Common Configurations Audio Processing Configuration Performance Monitoring Configuration How Parameters Are Used Complete Example Additional Information Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/pipeline_pipeline-task_310c3547.txt b/pipeline_pipeline-task_310c3547.txt
new file mode 100644
index 0000000000000000000000000000000000000000..37a8a2e1443fffaa2cc707ac3c64b602a908d270
--- /dev/null
+++ b/pipeline_pipeline-task_310c3547.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/pipeline/pipeline-task#param-params
+Title: PipelineTask - Pipecat
+==================================================
+
+PipelineTask - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Pipeline PipelineTask Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview PipelineTask is the central class for managing pipeline execution. It handles the lifecycle of the pipeline, processes frames in both directions, manages task cancellation, and provides event handlers for monitoring pipeline activity. ​ Basic Usage Copy Ask AI from pipecat.pipeline.pipeline import Pipeline from pipecat.pipeline.runner import PipelineRunner from pipecat.pipeline.task import PipelineParams, PipelineTask # Create a pipeline pipeline = Pipeline([ ... ]) # Create a task with the pipeline task = PipelineTask(pipeline) # Queue frames for processing await task.queue_frame(TTSSpeakFrame( "Hello, how can I help you today?" )) # Run the pipeline runner = PipelineRunner() await runner.run(task) ​ Constructor Parameters ​ pipeline BasePipeline required The pipeline to execute. ​ params PipelineParams default: "PipelineParams()" Configuration parameters for the pipeline. See PipelineParams for details. ​ observers List[BaseObserver] default: "[]" List of observers for monitoring pipeline execution. See Observers for details. ​ clock BaseClock default: "SystemClock()" Clock implementation for timing operations. ​ task_manager Optional[BaseTaskManager] default: "None" Custom task manager for handling asyncio tasks. If None, a default TaskManager is used. ​ check_dangling_tasks bool default: "True" Whether to check for processors’ tasks finishing properly. ​ idle_timeout_secs Optional[float] default: "300" Timeout in seconds before considering the pipeline idle. Set to None to disable idle detection. See Pipeline Idle Detection for details. ​ idle_timeout_frames Tuple[Type[Frame], ...] default: "(BotSpeakingFrame, LLMFullResponseEndFrame)" Frame types that should prevent the pipeline from being considered idle. See Pipeline Idle Detection for details. ​ cancel_on_idle_timeout bool default: "True" Whether to automatically cancel the pipeline task when idle timeout is reached. See Pipeline Idle Detection for details. ​ enable_tracing bool default: "False" Whether to enable OpenTelemetry tracing. See The OpenTelemetry guide for details. ​ enable_turn_tracking bool default: "False" Whether to enable turn tracking. See The OpenTelemetry guide for details. ​ conversation_id Optional[str] default: "None" Custom ID for the conversation. If not provided, a UUID will be generated. See The OpenTelemetry guide for details. ​ additional_span_attributes Optional[dict] default: "None" Any additional attributes to add to top-level OpenTelemetry conversation span. See The OpenTelemetry guide for details. ​ Methods ​ Task Lifecycle Management ​ run() async Starts and manages the pipeline execution until completion or cancellation. Copy Ask AI await task.run() ​ stop_when_done() async Sends an EndFrame to the pipeline to gracefully stop the task after all queued frames have been processed. Copy Ask AI await task.stop_when_done() ​ cancel() async Stops the running pipeline immediately by sending a CancelFrame. Copy Ask AI await task.cancel() ​ has_finished() bool Returns whether the task has finished (all processors have stopped). Copy Ask AI if task.has_finished(): print ( "Task is complete" ) ​ Frame Management ​ queue_frame() async Queues a single frame to be pushed down the pipeline. Copy Ask AI await task.queue_frame(TTSSpeakFrame( "Hello!" )) ​ queue_frames() async Queues multiple frames to be pushed down the pipeline. Copy Ask AI frames = [TTSSpeakFrame( "Hello!" ), TTSSpeakFrame( "How are you?" )] await task.queue_frames(frames) ​ Event Handlers PipelineTask provides an event handler that can be registered using the event_handler decorator: ​ on_idle_timeout Triggered when no activity frames (as specified by idle_timeout_frames ) have been received within the idle timeout period. Copy Ask AI @task.event_handler ( "on_idle_timeout" ) async def on_idle_timeout ( task ): print ( "Pipeline has been idle too long" ) await task.queue_frame(TTSSpeakFrame( "Are you still there?" )) PipelineParams Pipeline Idle Detection On this page Overview Basic Usage Constructor Parameters Methods Task Lifecycle Management Frame Management Event Handlers on_idle_timeout Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/pipeline_pipeline-task_33e64eaa.txt b/pipeline_pipeline-task_33e64eaa.txt
new file mode 100644
index 0000000000000000000000000000000000000000..6b08f40a76209a9206fa5718e5172d65d48d34e9
--- /dev/null
+++ b/pipeline_pipeline-task_33e64eaa.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/pipeline/pipeline-task#on-idle-timeout
+Title: PipelineTask - Pipecat
+==================================================
+
+PipelineTask - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Pipeline PipelineTask Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview PipelineTask is the central class for managing pipeline execution. It handles the lifecycle of the pipeline, processes frames in both directions, manages task cancellation, and provides event handlers for monitoring pipeline activity. ​ Basic Usage Copy Ask AI from pipecat.pipeline.pipeline import Pipeline from pipecat.pipeline.runner import PipelineRunner from pipecat.pipeline.task import PipelineParams, PipelineTask # Create a pipeline pipeline = Pipeline([ ... ]) # Create a task with the pipeline task = PipelineTask(pipeline) # Queue frames for processing await task.queue_frame(TTSSpeakFrame( "Hello, how can I help you today?" )) # Run the pipeline runner = PipelineRunner() await runner.run(task) ​ Constructor Parameters ​ pipeline BasePipeline required The pipeline to execute. ​ params PipelineParams default: "PipelineParams()" Configuration parameters for the pipeline. See PipelineParams for details. ​ observers List[BaseObserver] default: "[]" List of observers for monitoring pipeline execution. See Observers for details. ​ clock BaseClock default: "SystemClock()" Clock implementation for timing operations. ​ task_manager Optional[BaseTaskManager] default: "None" Custom task manager for handling asyncio tasks. If None, a default TaskManager is used. ​ check_dangling_tasks bool default: "True" Whether to check for processors’ tasks finishing properly. ​ idle_timeout_secs Optional[float] default: "300" Timeout in seconds before considering the pipeline idle. Set to None to disable idle detection. See Pipeline Idle Detection for details. ​ idle_timeout_frames Tuple[Type[Frame], ...] default: "(BotSpeakingFrame, LLMFullResponseEndFrame)" Frame types that should prevent the pipeline from being considered idle. See Pipeline Idle Detection for details. ​ cancel_on_idle_timeout bool default: "True" Whether to automatically cancel the pipeline task when idle timeout is reached. See Pipeline Idle Detection for details. ​ enable_tracing bool default: "False" Whether to enable OpenTelemetry tracing. See The OpenTelemetry guide for details. ​ enable_turn_tracking bool default: "False" Whether to enable turn tracking. See The OpenTelemetry guide for details. ​ conversation_id Optional[str] default: "None" Custom ID for the conversation. If not provided, a UUID will be generated. See The OpenTelemetry guide for details. ​ additional_span_attributes Optional[dict] default: "None" Any additional attributes to add to top-level OpenTelemetry conversation span. See The OpenTelemetry guide for details. ​ Methods ​ Task Lifecycle Management ​ run() async Starts and manages the pipeline execution until completion or cancellation. Copy Ask AI await task.run() ​ stop_when_done() async Sends an EndFrame to the pipeline to gracefully stop the task after all queued frames have been processed. Copy Ask AI await task.stop_when_done() ​ cancel() async Stops the running pipeline immediately by sending a CancelFrame. Copy Ask AI await task.cancel() ​ has_finished() bool Returns whether the task has finished (all processors have stopped). Copy Ask AI if task.has_finished(): print ( "Task is complete" ) ​ Frame Management ​ queue_frame() async Queues a single frame to be pushed down the pipeline. Copy Ask AI await task.queue_frame(TTSSpeakFrame( "Hello!" )) ​ queue_frames() async Queues multiple frames to be pushed down the pipeline. Copy Ask AI frames = [TTSSpeakFrame( "Hello!" ), TTSSpeakFrame( "How are you?" )] await task.queue_frames(frames) ​ Event Handlers PipelineTask provides an event handler that can be registered using the event_handler decorator: ​ on_idle_timeout Triggered when no activity frames (as specified by idle_timeout_frames ) have been received within the idle timeout period. Copy Ask AI @task.event_handler ( "on_idle_timeout" ) async def on_idle_timeout ( task ): print ( "Pipeline has been idle too long" ) await task.queue_frame(TTSSpeakFrame( "Are you still there?" )) PipelineParams Pipeline Idle Detection On this page Overview Basic Usage Constructor Parameters Methods Task Lifecycle Management Frame Management Event Handlers on_idle_timeout Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/pipeline_pipeline-task_4563c25e.txt b/pipeline_pipeline-task_4563c25e.txt
new file mode 100644
index 0000000000000000000000000000000000000000..1949dfcc5698ac5110ca264e380ea0717254e219
--- /dev/null
+++ b/pipeline_pipeline-task_4563c25e.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/pipeline/pipeline-task#methods
+Title: PipelineTask - Pipecat
+==================================================
+
+PipelineTask - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Pipeline PipelineTask Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview PipelineTask is the central class for managing pipeline execution. It handles the lifecycle of the pipeline, processes frames in both directions, manages task cancellation, and provides event handlers for monitoring pipeline activity. ​ Basic Usage Copy Ask AI from pipecat.pipeline.pipeline import Pipeline from pipecat.pipeline.runner import PipelineRunner from pipecat.pipeline.task import PipelineParams, PipelineTask # Create a pipeline pipeline = Pipeline([ ... ]) # Create a task with the pipeline task = PipelineTask(pipeline) # Queue frames for processing await task.queue_frame(TTSSpeakFrame( "Hello, how can I help you today?" )) # Run the pipeline runner = PipelineRunner() await runner.run(task) ​ Constructor Parameters ​ pipeline BasePipeline required The pipeline to execute. ​ params PipelineParams default: "PipelineParams()" Configuration parameters for the pipeline. See PipelineParams for details. ​ observers List[BaseObserver] default: "[]" List of observers for monitoring pipeline execution. See Observers for details. ​ clock BaseClock default: "SystemClock()" Clock implementation for timing operations. ​ task_manager Optional[BaseTaskManager] default: "None" Custom task manager for handling asyncio tasks. If None, a default TaskManager is used. ​ check_dangling_tasks bool default: "True" Whether to check for processors’ tasks finishing properly. ​ idle_timeout_secs Optional[float] default: "300" Timeout in seconds before considering the pipeline idle. Set to None to disable idle detection. See Pipeline Idle Detection for details. ​ idle_timeout_frames Tuple[Type[Frame], ...] default: "(BotSpeakingFrame, LLMFullResponseEndFrame)" Frame types that should prevent the pipeline from being considered idle. See Pipeline Idle Detection for details. ​ cancel_on_idle_timeout bool default: "True" Whether to automatically cancel the pipeline task when idle timeout is reached. See Pipeline Idle Detection for details. ​ enable_tracing bool default: "False" Whether to enable OpenTelemetry tracing. See The OpenTelemetry guide for details. ​ enable_turn_tracking bool default: "False" Whether to enable turn tracking. See The OpenTelemetry guide for details. ​ conversation_id Optional[str] default: "None" Custom ID for the conversation. If not provided, a UUID will be generated. See The OpenTelemetry guide for details. ​ additional_span_attributes Optional[dict] default: "None" Any additional attributes to add to top-level OpenTelemetry conversation span. See The OpenTelemetry guide for details. ​ Methods ​ Task Lifecycle Management ​ run() async Starts and manages the pipeline execution until completion or cancellation. Copy Ask AI await task.run() ​ stop_when_done() async Sends an EndFrame to the pipeline to gracefully stop the task after all queued frames have been processed. Copy Ask AI await task.stop_when_done() ​ cancel() async Stops the running pipeline immediately by sending a CancelFrame. Copy Ask AI await task.cancel() ​ has_finished() bool Returns whether the task has finished (all processors have stopped). Copy Ask AI if task.has_finished(): print ( "Task is complete" ) ​ Frame Management ​ queue_frame() async Queues a single frame to be pushed down the pipeline. Copy Ask AI await task.queue_frame(TTSSpeakFrame( "Hello!" )) ​ queue_frames() async Queues multiple frames to be pushed down the pipeline. Copy Ask AI frames = [TTSSpeakFrame( "Hello!" ), TTSSpeakFrame( "How are you?" )] await task.queue_frames(frames) ​ Event Handlers PipelineTask provides an event handler that can be registered using the event_handler decorator: ​ on_idle_timeout Triggered when no activity frames (as specified by idle_timeout_frames ) have been received within the idle timeout period. Copy Ask AI @task.event_handler ( "on_idle_timeout" ) async def on_idle_timeout ( task ): print ( "Pipeline has been idle too long" ) await task.queue_frame(TTSSpeakFrame( "Are you still there?" )) PipelineParams Pipeline Idle Detection On this page Overview Basic Usage Constructor Parameters Methods Task Lifecycle Management Frame Management Event Handlers on_idle_timeout Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/pipeline_pipeline-task_6496025a.txt b/pipeline_pipeline-task_6496025a.txt
new file mode 100644
index 0000000000000000000000000000000000000000..8899f548efed001e3e26388b7e43aa1788f8163d
--- /dev/null
+++ b/pipeline_pipeline-task_6496025a.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/pipeline/pipeline-task#param-idle-timeout-secs
+Title: PipelineTask - Pipecat
+==================================================
+
+PipelineTask - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Pipeline PipelineTask Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview PipelineTask is the central class for managing pipeline execution. It handles the lifecycle of the pipeline, processes frames in both directions, manages task cancellation, and provides event handlers for monitoring pipeline activity. ​ Basic Usage Copy Ask AI from pipecat.pipeline.pipeline import Pipeline from pipecat.pipeline.runner import PipelineRunner from pipecat.pipeline.task import PipelineParams, PipelineTask # Create a pipeline pipeline = Pipeline([ ... ]) # Create a task with the pipeline task = PipelineTask(pipeline) # Queue frames for processing await task.queue_frame(TTSSpeakFrame( "Hello, how can I help you today?" )) # Run the pipeline runner = PipelineRunner() await runner.run(task) ​ Constructor Parameters ​ pipeline BasePipeline required The pipeline to execute. ​ params PipelineParams default: "PipelineParams()" Configuration parameters for the pipeline. See PipelineParams for details. ​ observers List[BaseObserver] default: "[]" List of observers for monitoring pipeline execution. See Observers for details. ​ clock BaseClock default: "SystemClock()" Clock implementation for timing operations. ​ task_manager Optional[BaseTaskManager] default: "None" Custom task manager for handling asyncio tasks. If None, a default TaskManager is used. ​ check_dangling_tasks bool default: "True" Whether to check for processors’ tasks finishing properly. ​ idle_timeout_secs Optional[float] default: "300" Timeout in seconds before considering the pipeline idle. Set to None to disable idle detection. See Pipeline Idle Detection for details. ​ idle_timeout_frames Tuple[Type[Frame], ...] default: "(BotSpeakingFrame, LLMFullResponseEndFrame)" Frame types that should prevent the pipeline from being considered idle. See Pipeline Idle Detection for details. ​ cancel_on_idle_timeout bool default: "True" Whether to automatically cancel the pipeline task when idle timeout is reached. See Pipeline Idle Detection for details. ​ enable_tracing bool default: "False" Whether to enable OpenTelemetry tracing. See The OpenTelemetry guide for details. ​ enable_turn_tracking bool default: "False" Whether to enable turn tracking. See The OpenTelemetry guide for details. ​ conversation_id Optional[str] default: "None" Custom ID for the conversation. If not provided, a UUID will be generated. See The OpenTelemetry guide for details. ​ additional_span_attributes Optional[dict] default: "None" Any additional attributes to add to top-level OpenTelemetry conversation span. See The OpenTelemetry guide for details. ​ Methods ​ Task Lifecycle Management ​ run() async Starts and manages the pipeline execution until completion or cancellation. Copy Ask AI await task.run() ​ stop_when_done() async Sends an EndFrame to the pipeline to gracefully stop the task after all queued frames have been processed. Copy Ask AI await task.stop_when_done() ​ cancel() async Stops the running pipeline immediately by sending a CancelFrame. Copy Ask AI await task.cancel() ​ has_finished() bool Returns whether the task has finished (all processors have stopped). Copy Ask AI if task.has_finished(): print ( "Task is complete" ) ​ Frame Management ​ queue_frame() async Queues a single frame to be pushed down the pipeline. Copy Ask AI await task.queue_frame(TTSSpeakFrame( "Hello!" )) ​ queue_frames() async Queues multiple frames to be pushed down the pipeline. Copy Ask AI frames = [TTSSpeakFrame( "Hello!" ), TTSSpeakFrame( "How are you?" )] await task.queue_frames(frames) ​ Event Handlers PipelineTask provides an event handler that can be registered using the event_handler decorator: ​ on_idle_timeout Triggered when no activity frames (as specified by idle_timeout_frames ) have been received within the idle timeout period. Copy Ask AI @task.event_handler ( "on_idle_timeout" ) async def on_idle_timeout ( task ): print ( "Pipeline has been idle too long" ) await task.queue_frame(TTSSpeakFrame( "Are you still there?" )) PipelineParams Pipeline Idle Detection On this page Overview Basic Usage Constructor Parameters Methods Task Lifecycle Management Frame Management Event Handlers on_idle_timeout Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/pipeline_pipeline-task_7cfe1600.txt b/pipeline_pipeline-task_7cfe1600.txt
new file mode 100644
index 0000000000000000000000000000000000000000..b0c2ffbd0446e87e4ff97affd93ed03650d5627a
--- /dev/null
+++ b/pipeline_pipeline-task_7cfe1600.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/pipeline/pipeline-task#frame-management
+Title: PipelineTask - Pipecat
+==================================================
+
+PipelineTask - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Pipeline PipelineTask Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview PipelineTask is the central class for managing pipeline execution. It handles the lifecycle of the pipeline, processes frames in both directions, manages task cancellation, and provides event handlers for monitoring pipeline activity. ​ Basic Usage Copy Ask AI from pipecat.pipeline.pipeline import Pipeline from pipecat.pipeline.runner import PipelineRunner from pipecat.pipeline.task import PipelineParams, PipelineTask # Create a pipeline pipeline = Pipeline([ ... ]) # Create a task with the pipeline task = PipelineTask(pipeline) # Queue frames for processing await task.queue_frame(TTSSpeakFrame( "Hello, how can I help you today?" )) # Run the pipeline runner = PipelineRunner() await runner.run(task) ​ Constructor Parameters ​ pipeline BasePipeline required The pipeline to execute. ​ params PipelineParams default: "PipelineParams()" Configuration parameters for the pipeline. See PipelineParams for details. ​ observers List[BaseObserver] default: "[]" List of observers for monitoring pipeline execution. See Observers for details. ​ clock BaseClock default: "SystemClock()" Clock implementation for timing operations. ​ task_manager Optional[BaseTaskManager] default: "None" Custom task manager for handling asyncio tasks. If None, a default TaskManager is used. ​ check_dangling_tasks bool default: "True" Whether to check for processors’ tasks finishing properly. ​ idle_timeout_secs Optional[float] default: "300" Timeout in seconds before considering the pipeline idle. Set to None to disable idle detection. See Pipeline Idle Detection for details. ​ idle_timeout_frames Tuple[Type[Frame], ...] default: "(BotSpeakingFrame, LLMFullResponseEndFrame)" Frame types that should prevent the pipeline from being considered idle. See Pipeline Idle Detection for details. ​ cancel_on_idle_timeout bool default: "True" Whether to automatically cancel the pipeline task when idle timeout is reached. See Pipeline Idle Detection for details. ​ enable_tracing bool default: "False" Whether to enable OpenTelemetry tracing. See The OpenTelemetry guide for details. ​ enable_turn_tracking bool default: "False" Whether to enable turn tracking. See The OpenTelemetry guide for details. ​ conversation_id Optional[str] default: "None" Custom ID for the conversation. If not provided, a UUID will be generated. See The OpenTelemetry guide for details. ​ additional_span_attributes Optional[dict] default: "None" Any additional attributes to add to top-level OpenTelemetry conversation span. See The OpenTelemetry guide for details. ​ Methods ​ Task Lifecycle Management ​ run() async Starts and manages the pipeline execution until completion or cancellation. Copy Ask AI await task.run() ​ stop_when_done() async Sends an EndFrame to the pipeline to gracefully stop the task after all queued frames have been processed. Copy Ask AI await task.stop_when_done() ​ cancel() async Stops the running pipeline immediately by sending a CancelFrame. Copy Ask AI await task.cancel() ​ has_finished() bool Returns whether the task has finished (all processors have stopped). Copy Ask AI if task.has_finished(): print ( "Task is complete" ) ​ Frame Management ​ queue_frame() async Queues a single frame to be pushed down the pipeline. Copy Ask AI await task.queue_frame(TTSSpeakFrame( "Hello!" )) ​ queue_frames() async Queues multiple frames to be pushed down the pipeline. Copy Ask AI frames = [TTSSpeakFrame( "Hello!" ), TTSSpeakFrame( "How are you?" )] await task.queue_frames(frames) ​ Event Handlers PipelineTask provides an event handler that can be registered using the event_handler decorator: ​ on_idle_timeout Triggered when no activity frames (as specified by idle_timeout_frames ) have been received within the idle timeout period. Copy Ask AI @task.event_handler ( "on_idle_timeout" ) async def on_idle_timeout ( task ): print ( "Pipeline has been idle too long" ) await task.queue_frame(TTSSpeakFrame( "Are you still there?" )) PipelineParams Pipeline Idle Detection On this page Overview Basic Usage Constructor Parameters Methods Task Lifecycle Management Frame Management Event Handlers on_idle_timeout Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/pipeline_pipeline-task_81448819.txt b/pipeline_pipeline-task_81448819.txt
new file mode 100644
index 0000000000000000000000000000000000000000..882028af8df0031e4e5de88efb1b6cc28269e4a3
--- /dev/null
+++ b/pipeline_pipeline-task_81448819.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/pipeline/pipeline-task#param-stop-when-done
+Title: PipelineTask - Pipecat
+==================================================
+
+PipelineTask - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Pipeline PipelineTask Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview PipelineTask is the central class for managing pipeline execution. It handles the lifecycle of the pipeline, processes frames in both directions, manages task cancellation, and provides event handlers for monitoring pipeline activity. ​ Basic Usage Copy Ask AI from pipecat.pipeline.pipeline import Pipeline from pipecat.pipeline.runner import PipelineRunner from pipecat.pipeline.task import PipelineParams, PipelineTask # Create a pipeline pipeline = Pipeline([ ... ]) # Create a task with the pipeline task = PipelineTask(pipeline) # Queue frames for processing await task.queue_frame(TTSSpeakFrame( "Hello, how can I help you today?" )) # Run the pipeline runner = PipelineRunner() await runner.run(task) ​ Constructor Parameters ​ pipeline BasePipeline required The pipeline to execute. ​ params PipelineParams default: "PipelineParams()" Configuration parameters for the pipeline. See PipelineParams for details. ​ observers List[BaseObserver] default: "[]" List of observers for monitoring pipeline execution. See Observers for details. ​ clock BaseClock default: "SystemClock()" Clock implementation for timing operations. ​ task_manager Optional[BaseTaskManager] default: "None" Custom task manager for handling asyncio tasks. If None, a default TaskManager is used. ​ check_dangling_tasks bool default: "True" Whether to check for processors’ tasks finishing properly. ​ idle_timeout_secs Optional[float] default: "300" Timeout in seconds before considering the pipeline idle. Set to None to disable idle detection. See Pipeline Idle Detection for details. ​ idle_timeout_frames Tuple[Type[Frame], ...] default: "(BotSpeakingFrame, LLMFullResponseEndFrame)" Frame types that should prevent the pipeline from being considered idle. See Pipeline Idle Detection for details. ​ cancel_on_idle_timeout bool default: "True" Whether to automatically cancel the pipeline task when idle timeout is reached. See Pipeline Idle Detection for details. ​ enable_tracing bool default: "False" Whether to enable OpenTelemetry tracing. See The OpenTelemetry guide for details. ​ enable_turn_tracking bool default: "False" Whether to enable turn tracking. See The OpenTelemetry guide for details. ​ conversation_id Optional[str] default: "None" Custom ID for the conversation. If not provided, a UUID will be generated. See The OpenTelemetry guide for details. ​ additional_span_attributes Optional[dict] default: "None" Any additional attributes to add to top-level OpenTelemetry conversation span. See The OpenTelemetry guide for details. ​ Methods ​ Task Lifecycle Management ​ run() async Starts and manages the pipeline execution until completion or cancellation. Copy Ask AI await task.run() ​ stop_when_done() async Sends an EndFrame to the pipeline to gracefully stop the task after all queued frames have been processed. Copy Ask AI await task.stop_when_done() ​ cancel() async Stops the running pipeline immediately by sending a CancelFrame. Copy Ask AI await task.cancel() ​ has_finished() bool Returns whether the task has finished (all processors have stopped). Copy Ask AI if task.has_finished(): print ( "Task is complete" ) ​ Frame Management ​ queue_frame() async Queues a single frame to be pushed down the pipeline. Copy Ask AI await task.queue_frame(TTSSpeakFrame( "Hello!" )) ​ queue_frames() async Queues multiple frames to be pushed down the pipeline. Copy Ask AI frames = [TTSSpeakFrame( "Hello!" ), TTSSpeakFrame( "How are you?" )] await task.queue_frames(frames) ​ Event Handlers PipelineTask provides an event handler that can be registered using the event_handler decorator: ​ on_idle_timeout Triggered when no activity frames (as specified by idle_timeout_frames ) have been received within the idle timeout period. Copy Ask AI @task.event_handler ( "on_idle_timeout" ) async def on_idle_timeout ( task ): print ( "Pipeline has been idle too long" ) await task.queue_frame(TTSSpeakFrame( "Are you still there?" )) PipelineParams Pipeline Idle Detection On this page Overview Basic Usage Constructor Parameters Methods Task Lifecycle Management Frame Management Event Handlers on_idle_timeout Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/pipeline_pipeline-task_8fce26ac.txt b/pipeline_pipeline-task_8fce26ac.txt
new file mode 100644
index 0000000000000000000000000000000000000000..29133ab4a56349e4e36e44e5fd268fc394c779b8
--- /dev/null
+++ b/pipeline_pipeline-task_8fce26ac.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/pipeline/pipeline-task#event-handlers
+Title: PipelineTask - Pipecat
+==================================================
+
+PipelineTask - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Pipeline PipelineTask Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview PipelineTask is the central class for managing pipeline execution. It handles the lifecycle of the pipeline, processes frames in both directions, manages task cancellation, and provides event handlers for monitoring pipeline activity. ​ Basic Usage Copy Ask AI from pipecat.pipeline.pipeline import Pipeline from pipecat.pipeline.runner import PipelineRunner from pipecat.pipeline.task import PipelineParams, PipelineTask # Create a pipeline pipeline = Pipeline([ ... ]) # Create a task with the pipeline task = PipelineTask(pipeline) # Queue frames for processing await task.queue_frame(TTSSpeakFrame( "Hello, how can I help you today?" )) # Run the pipeline runner = PipelineRunner() await runner.run(task) ​ Constructor Parameters ​ pipeline BasePipeline required The pipeline to execute. ​ params PipelineParams default: "PipelineParams()" Configuration parameters for the pipeline. See PipelineParams for details. ​ observers List[BaseObserver] default: "[]" List of observers for monitoring pipeline execution. See Observers for details. ​ clock BaseClock default: "SystemClock()" Clock implementation for timing operations. ​ task_manager Optional[BaseTaskManager] default: "None" Custom task manager for handling asyncio tasks. If None, a default TaskManager is used. ​ check_dangling_tasks bool default: "True" Whether to check for processors’ tasks finishing properly. ​ idle_timeout_secs Optional[float] default: "300" Timeout in seconds before considering the pipeline idle. Set to None to disable idle detection. See Pipeline Idle Detection for details. ​ idle_timeout_frames Tuple[Type[Frame], ...] default: "(BotSpeakingFrame, LLMFullResponseEndFrame)" Frame types that should prevent the pipeline from being considered idle. See Pipeline Idle Detection for details. ​ cancel_on_idle_timeout bool default: "True" Whether to automatically cancel the pipeline task when idle timeout is reached. See Pipeline Idle Detection for details. ​ enable_tracing bool default: "False" Whether to enable OpenTelemetry tracing. See The OpenTelemetry guide for details. ​ enable_turn_tracking bool default: "False" Whether to enable turn tracking. See The OpenTelemetry guide for details. ​ conversation_id Optional[str] default: "None" Custom ID for the conversation. If not provided, a UUID will be generated. See The OpenTelemetry guide for details. ​ additional_span_attributes Optional[dict] default: "None" Any additional attributes to add to top-level OpenTelemetry conversation span. See The OpenTelemetry guide for details. ​ Methods ​ Task Lifecycle Management ​ run() async Starts and manages the pipeline execution until completion or cancellation. Copy Ask AI await task.run() ​ stop_when_done() async Sends an EndFrame to the pipeline to gracefully stop the task after all queued frames have been processed. Copy Ask AI await task.stop_when_done() ​ cancel() async Stops the running pipeline immediately by sending a CancelFrame. Copy Ask AI await task.cancel() ​ has_finished() bool Returns whether the task has finished (all processors have stopped). Copy Ask AI if task.has_finished(): print ( "Task is complete" ) ​ Frame Management ​ queue_frame() async Queues a single frame to be pushed down the pipeline. Copy Ask AI await task.queue_frame(TTSSpeakFrame( "Hello!" )) ​ queue_frames() async Queues multiple frames to be pushed down the pipeline. Copy Ask AI frames = [TTSSpeakFrame( "Hello!" ), TTSSpeakFrame( "How are you?" )] await task.queue_frames(frames) ​ Event Handlers PipelineTask provides an event handler that can be registered using the event_handler decorator: ​ on_idle_timeout Triggered when no activity frames (as specified by idle_timeout_frames ) have been received within the idle timeout period. Copy Ask AI @task.event_handler ( "on_idle_timeout" ) async def on_idle_timeout ( task ): print ( "Pipeline has been idle too long" ) await task.queue_frame(TTSSpeakFrame( "Are you still there?" )) PipelineParams Pipeline Idle Detection On this page Overview Basic Usage Constructor Parameters Methods Task Lifecycle Management Frame Management Event Handlers on_idle_timeout Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/pipeline_pipeline-task_987a4cfa.txt b/pipeline_pipeline-task_987a4cfa.txt
new file mode 100644
index 0000000000000000000000000000000000000000..91523da50413a4b29522e1dcdab96079de411001
--- /dev/null
+++ b/pipeline_pipeline-task_987a4cfa.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/pipeline/pipeline-task#param-conversation-id
+Title: PipelineTask - Pipecat
+==================================================
+
+PipelineTask - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Pipeline PipelineTask Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview PipelineTask is the central class for managing pipeline execution. It handles the lifecycle of the pipeline, processes frames in both directions, manages task cancellation, and provides event handlers for monitoring pipeline activity. ​ Basic Usage Copy Ask AI from pipecat.pipeline.pipeline import Pipeline from pipecat.pipeline.runner import PipelineRunner from pipecat.pipeline.task import PipelineParams, PipelineTask # Create a pipeline pipeline = Pipeline([ ... ]) # Create a task with the pipeline task = PipelineTask(pipeline) # Queue frames for processing await task.queue_frame(TTSSpeakFrame( "Hello, how can I help you today?" )) # Run the pipeline runner = PipelineRunner() await runner.run(task) ​ Constructor Parameters ​ pipeline BasePipeline required The pipeline to execute. ​ params PipelineParams default: "PipelineParams()" Configuration parameters for the pipeline. See PipelineParams for details. ​ observers List[BaseObserver] default: "[]" List of observers for monitoring pipeline execution. See Observers for details. ​ clock BaseClock default: "SystemClock()" Clock implementation for timing operations. ​ task_manager Optional[BaseTaskManager] default: "None" Custom task manager for handling asyncio tasks. If None, a default TaskManager is used. ​ check_dangling_tasks bool default: "True" Whether to check for processors’ tasks finishing properly. ​ idle_timeout_secs Optional[float] default: "300" Timeout in seconds before considering the pipeline idle. Set to None to disable idle detection. See Pipeline Idle Detection for details. ​ idle_timeout_frames Tuple[Type[Frame], ...] default: "(BotSpeakingFrame, LLMFullResponseEndFrame)" Frame types that should prevent the pipeline from being considered idle. See Pipeline Idle Detection for details. ​ cancel_on_idle_timeout bool default: "True" Whether to automatically cancel the pipeline task when idle timeout is reached. See Pipeline Idle Detection for details. ​ enable_tracing bool default: "False" Whether to enable OpenTelemetry tracing. See The OpenTelemetry guide for details. ​ enable_turn_tracking bool default: "False" Whether to enable turn tracking. See The OpenTelemetry guide for details. ​ conversation_id Optional[str] default: "None" Custom ID for the conversation. If not provided, a UUID will be generated. See The OpenTelemetry guide for details. ​ additional_span_attributes Optional[dict] default: "None" Any additional attributes to add to top-level OpenTelemetry conversation span. See The OpenTelemetry guide for details. ​ Methods ​ Task Lifecycle Management ​ run() async Starts and manages the pipeline execution until completion or cancellation. Copy Ask AI await task.run() ​ stop_when_done() async Sends an EndFrame to the pipeline to gracefully stop the task after all queued frames have been processed. Copy Ask AI await task.stop_when_done() ​ cancel() async Stops the running pipeline immediately by sending a CancelFrame. Copy Ask AI await task.cancel() ​ has_finished() bool Returns whether the task has finished (all processors have stopped). Copy Ask AI if task.has_finished(): print ( "Task is complete" ) ​ Frame Management ​ queue_frame() async Queues a single frame to be pushed down the pipeline. Copy Ask AI await task.queue_frame(TTSSpeakFrame( "Hello!" )) ​ queue_frames() async Queues multiple frames to be pushed down the pipeline. Copy Ask AI frames = [TTSSpeakFrame( "Hello!" ), TTSSpeakFrame( "How are you?" )] await task.queue_frames(frames) ​ Event Handlers PipelineTask provides an event handler that can be registered using the event_handler decorator: ​ on_idle_timeout Triggered when no activity frames (as specified by idle_timeout_frames ) have been received within the idle timeout period. Copy Ask AI @task.event_handler ( "on_idle_timeout" ) async def on_idle_timeout ( task ): print ( "Pipeline has been idle too long" ) await task.queue_frame(TTSSpeakFrame( "Are you still there?" )) PipelineParams Pipeline Idle Detection On this page Overview Basic Usage Constructor Parameters Methods Task Lifecycle Management Frame Management Event Handlers on_idle_timeout Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/pipeline_pipeline-task_ae2e01df.txt b/pipeline_pipeline-task_ae2e01df.txt
new file mode 100644
index 0000000000000000000000000000000000000000..c81aeb31df0ad9c1db5b9151e60ca44453c0150d
--- /dev/null
+++ b/pipeline_pipeline-task_ae2e01df.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/pipeline/pipeline-task#param-task-manager
+Title: PipelineTask - Pipecat
+==================================================
+
+PipelineTask - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Pipeline PipelineTask Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview PipelineTask is the central class for managing pipeline execution. It handles the lifecycle of the pipeline, processes frames in both directions, manages task cancellation, and provides event handlers for monitoring pipeline activity. ​ Basic Usage Copy Ask AI from pipecat.pipeline.pipeline import Pipeline from pipecat.pipeline.runner import PipelineRunner from pipecat.pipeline.task import PipelineParams, PipelineTask # Create a pipeline pipeline = Pipeline([ ... ]) # Create a task with the pipeline task = PipelineTask(pipeline) # Queue frames for processing await task.queue_frame(TTSSpeakFrame( "Hello, how can I help you today?" )) # Run the pipeline runner = PipelineRunner() await runner.run(task) ​ Constructor Parameters ​ pipeline BasePipeline required The pipeline to execute. ​ params PipelineParams default: "PipelineParams()" Configuration parameters for the pipeline. See PipelineParams for details. ​ observers List[BaseObserver] default: "[]" List of observers for monitoring pipeline execution. See Observers for details. ​ clock BaseClock default: "SystemClock()" Clock implementation for timing operations. ​ task_manager Optional[BaseTaskManager] default: "None" Custom task manager for handling asyncio tasks. If None, a default TaskManager is used. ​ check_dangling_tasks bool default: "True" Whether to check for processors’ tasks finishing properly. ​ idle_timeout_secs Optional[float] default: "300" Timeout in seconds before considering the pipeline idle. Set to None to disable idle detection. See Pipeline Idle Detection for details. ​ idle_timeout_frames Tuple[Type[Frame], ...] default: "(BotSpeakingFrame, LLMFullResponseEndFrame)" Frame types that should prevent the pipeline from being considered idle. See Pipeline Idle Detection for details. ​ cancel_on_idle_timeout bool default: "True" Whether to automatically cancel the pipeline task when idle timeout is reached. See Pipeline Idle Detection for details. ​ enable_tracing bool default: "False" Whether to enable OpenTelemetry tracing. See The OpenTelemetry guide for details. ​ enable_turn_tracking bool default: "False" Whether to enable turn tracking. See The OpenTelemetry guide for details. ​ conversation_id Optional[str] default: "None" Custom ID for the conversation. If not provided, a UUID will be generated. See The OpenTelemetry guide for details. ​ additional_span_attributes Optional[dict] default: "None" Any additional attributes to add to top-level OpenTelemetry conversation span. See The OpenTelemetry guide for details. ​ Methods ​ Task Lifecycle Management ​ run() async Starts and manages the pipeline execution until completion or cancellation. Copy Ask AI await task.run() ​ stop_when_done() async Sends an EndFrame to the pipeline to gracefully stop the task after all queued frames have been processed. Copy Ask AI await task.stop_when_done() ​ cancel() async Stops the running pipeline immediately by sending a CancelFrame. Copy Ask AI await task.cancel() ​ has_finished() bool Returns whether the task has finished (all processors have stopped). Copy Ask AI if task.has_finished(): print ( "Task is complete" ) ​ Frame Management ​ queue_frame() async Queues a single frame to be pushed down the pipeline. Copy Ask AI await task.queue_frame(TTSSpeakFrame( "Hello!" )) ​ queue_frames() async Queues multiple frames to be pushed down the pipeline. Copy Ask AI frames = [TTSSpeakFrame( "Hello!" ), TTSSpeakFrame( "How are you?" )] await task.queue_frames(frames) ​ Event Handlers PipelineTask provides an event handler that can be registered using the event_handler decorator: ​ on_idle_timeout Triggered when no activity frames (as specified by idle_timeout_frames ) have been received within the idle timeout period. Copy Ask AI @task.event_handler ( "on_idle_timeout" ) async def on_idle_timeout ( task ): print ( "Pipeline has been idle too long" ) await task.queue_frame(TTSSpeakFrame( "Are you still there?" )) PipelineParams Pipeline Idle Detection On this page Overview Basic Usage Constructor Parameters Methods Task Lifecycle Management Frame Management Event Handlers on_idle_timeout Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/pipeline_pipeline-task_b0de8b66.txt b/pipeline_pipeline-task_b0de8b66.txt
new file mode 100644
index 0000000000000000000000000000000000000000..e3e93adc7bbb168d2a356e617e0709ed23a7090e
--- /dev/null
+++ b/pipeline_pipeline-task_b0de8b66.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/pipeline/pipeline-task#param-idle-timeout-frames
+Title: PipelineTask - Pipecat
+==================================================
+
+PipelineTask - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Pipeline PipelineTask Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview PipelineTask is the central class for managing pipeline execution. It handles the lifecycle of the pipeline, processes frames in both directions, manages task cancellation, and provides event handlers for monitoring pipeline activity. ​ Basic Usage Copy Ask AI from pipecat.pipeline.pipeline import Pipeline from pipecat.pipeline.runner import PipelineRunner from pipecat.pipeline.task import PipelineParams, PipelineTask # Create a pipeline pipeline = Pipeline([ ... ]) # Create a task with the pipeline task = PipelineTask(pipeline) # Queue frames for processing await task.queue_frame(TTSSpeakFrame( "Hello, how can I help you today?" )) # Run the pipeline runner = PipelineRunner() await runner.run(task) ​ Constructor Parameters ​ pipeline BasePipeline required The pipeline to execute. ​ params PipelineParams default: "PipelineParams()" Configuration parameters for the pipeline. See PipelineParams for details. ​ observers List[BaseObserver] default: "[]" List of observers for monitoring pipeline execution. See Observers for details. ​ clock BaseClock default: "SystemClock()" Clock implementation for timing operations. ​ task_manager Optional[BaseTaskManager] default: "None" Custom task manager for handling asyncio tasks. If None, a default TaskManager is used. ​ check_dangling_tasks bool default: "True" Whether to check for processors’ tasks finishing properly. ​ idle_timeout_secs Optional[float] default: "300" Timeout in seconds before considering the pipeline idle. Set to None to disable idle detection. See Pipeline Idle Detection for details. ​ idle_timeout_frames Tuple[Type[Frame], ...] default: "(BotSpeakingFrame, LLMFullResponseEndFrame)" Frame types that should prevent the pipeline from being considered idle. See Pipeline Idle Detection for details. ​ cancel_on_idle_timeout bool default: "True" Whether to automatically cancel the pipeline task when idle timeout is reached. See Pipeline Idle Detection for details. ​ enable_tracing bool default: "False" Whether to enable OpenTelemetry tracing. See The OpenTelemetry guide for details. ​ enable_turn_tracking bool default: "False" Whether to enable turn tracking. See The OpenTelemetry guide for details. ​ conversation_id Optional[str] default: "None" Custom ID for the conversation. If not provided, a UUID will be generated. See The OpenTelemetry guide for details. ​ additional_span_attributes Optional[dict] default: "None" Any additional attributes to add to top-level OpenTelemetry conversation span. See The OpenTelemetry guide for details. ​ Methods ​ Task Lifecycle Management ​ run() async Starts and manages the pipeline execution until completion or cancellation. Copy Ask AI await task.run() ​ stop_when_done() async Sends an EndFrame to the pipeline to gracefully stop the task after all queued frames have been processed. Copy Ask AI await task.stop_when_done() ​ cancel() async Stops the running pipeline immediately by sending a CancelFrame. Copy Ask AI await task.cancel() ​ has_finished() bool Returns whether the task has finished (all processors have stopped). Copy Ask AI if task.has_finished(): print ( "Task is complete" ) ​ Frame Management ​ queue_frame() async Queues a single frame to be pushed down the pipeline. Copy Ask AI await task.queue_frame(TTSSpeakFrame( "Hello!" )) ​ queue_frames() async Queues multiple frames to be pushed down the pipeline. Copy Ask AI frames = [TTSSpeakFrame( "Hello!" ), TTSSpeakFrame( "How are you?" )] await task.queue_frames(frames) ​ Event Handlers PipelineTask provides an event handler that can be registered using the event_handler decorator: ​ on_idle_timeout Triggered when no activity frames (as specified by idle_timeout_frames ) have been received within the idle timeout period. Copy Ask AI @task.event_handler ( "on_idle_timeout" ) async def on_idle_timeout ( task ): print ( "Pipeline has been idle too long" ) await task.queue_frame(TTSSpeakFrame( "Are you still there?" )) PipelineParams Pipeline Idle Detection On this page Overview Basic Usage Constructor Parameters Methods Task Lifecycle Management Frame Management Event Handlers on_idle_timeout Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/pipeline_pipeline-task_b4e9648c.txt b/pipeline_pipeline-task_b4e9648c.txt
new file mode 100644
index 0000000000000000000000000000000000000000..1f74525e148a6b7b13ba46aaad68fc4e1a903c23
--- /dev/null
+++ b/pipeline_pipeline-task_b4e9648c.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/pipeline/pipeline-task#param-queue-frames
+Title: PipelineTask - Pipecat
+==================================================
+
+PipelineTask - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Pipeline PipelineTask Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview PipelineTask is the central class for managing pipeline execution. It handles the lifecycle of the pipeline, processes frames in both directions, manages task cancellation, and provides event handlers for monitoring pipeline activity. ​ Basic Usage Copy Ask AI from pipecat.pipeline.pipeline import Pipeline from pipecat.pipeline.runner import PipelineRunner from pipecat.pipeline.task import PipelineParams, PipelineTask # Create a pipeline pipeline = Pipeline([ ... ]) # Create a task with the pipeline task = PipelineTask(pipeline) # Queue frames for processing await task.queue_frame(TTSSpeakFrame( "Hello, how can I help you today?" )) # Run the pipeline runner = PipelineRunner() await runner.run(task) ​ Constructor Parameters ​ pipeline BasePipeline required The pipeline to execute. ​ params PipelineParams default: "PipelineParams()" Configuration parameters for the pipeline. See PipelineParams for details. ​ observers List[BaseObserver] default: "[]" List of observers for monitoring pipeline execution. See Observers for details. ​ clock BaseClock default: "SystemClock()" Clock implementation for timing operations. ​ task_manager Optional[BaseTaskManager] default: "None" Custom task manager for handling asyncio tasks. If None, a default TaskManager is used. ​ check_dangling_tasks bool default: "True" Whether to check for processors’ tasks finishing properly. ​ idle_timeout_secs Optional[float] default: "300" Timeout in seconds before considering the pipeline idle. Set to None to disable idle detection. See Pipeline Idle Detection for details. ​ idle_timeout_frames Tuple[Type[Frame], ...] default: "(BotSpeakingFrame, LLMFullResponseEndFrame)" Frame types that should prevent the pipeline from being considered idle. See Pipeline Idle Detection for details. ​ cancel_on_idle_timeout bool default: "True" Whether to automatically cancel the pipeline task when idle timeout is reached. See Pipeline Idle Detection for details. ​ enable_tracing bool default: "False" Whether to enable OpenTelemetry tracing. See The OpenTelemetry guide for details. ​ enable_turn_tracking bool default: "False" Whether to enable turn tracking. See The OpenTelemetry guide for details. ​ conversation_id Optional[str] default: "None" Custom ID for the conversation. If not provided, a UUID will be generated. See The OpenTelemetry guide for details. ​ additional_span_attributes Optional[dict] default: "None" Any additional attributes to add to top-level OpenTelemetry conversation span. See The OpenTelemetry guide for details. ​ Methods ​ Task Lifecycle Management ​ run() async Starts and manages the pipeline execution until completion or cancellation. Copy Ask AI await task.run() ​ stop_when_done() async Sends an EndFrame to the pipeline to gracefully stop the task after all queued frames have been processed. Copy Ask AI await task.stop_when_done() ​ cancel() async Stops the running pipeline immediately by sending a CancelFrame. Copy Ask AI await task.cancel() ​ has_finished() bool Returns whether the task has finished (all processors have stopped). Copy Ask AI if task.has_finished(): print ( "Task is complete" ) ​ Frame Management ​ queue_frame() async Queues a single frame to be pushed down the pipeline. Copy Ask AI await task.queue_frame(TTSSpeakFrame( "Hello!" )) ​ queue_frames() async Queues multiple frames to be pushed down the pipeline. Copy Ask AI frames = [TTSSpeakFrame( "Hello!" ), TTSSpeakFrame( "How are you?" )] await task.queue_frames(frames) ​ Event Handlers PipelineTask provides an event handler that can be registered using the event_handler decorator: ​ on_idle_timeout Triggered when no activity frames (as specified by idle_timeout_frames ) have been received within the idle timeout period. Copy Ask AI @task.event_handler ( "on_idle_timeout" ) async def on_idle_timeout ( task ): print ( "Pipeline has been idle too long" ) await task.queue_frame(TTSSpeakFrame( "Are you still there?" )) PipelineParams Pipeline Idle Detection On this page Overview Basic Usage Constructor Parameters Methods Task Lifecycle Management Frame Management Event Handlers on_idle_timeout Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/pipeline_pipeline-task_c48a2c87.txt b/pipeline_pipeline-task_c48a2c87.txt
new file mode 100644
index 0000000000000000000000000000000000000000..f90e63e1e7b1ea7fb64ad834769394a7d15f5281
--- /dev/null
+++ b/pipeline_pipeline-task_c48a2c87.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/pipeline/pipeline-task#param-enable-tracing
+Title: PipelineTask - Pipecat
+==================================================
+
+PipelineTask - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Pipeline PipelineTask Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview PipelineTask is the central class for managing pipeline execution. It handles the lifecycle of the pipeline, processes frames in both directions, manages task cancellation, and provides event handlers for monitoring pipeline activity. ​ Basic Usage Copy Ask AI from pipecat.pipeline.pipeline import Pipeline from pipecat.pipeline.runner import PipelineRunner from pipecat.pipeline.task import PipelineParams, PipelineTask # Create a pipeline pipeline = Pipeline([ ... ]) # Create a task with the pipeline task = PipelineTask(pipeline) # Queue frames for processing await task.queue_frame(TTSSpeakFrame( "Hello, how can I help you today?" )) # Run the pipeline runner = PipelineRunner() await runner.run(task) ​ Constructor Parameters ​ pipeline BasePipeline required The pipeline to execute. ​ params PipelineParams default: "PipelineParams()" Configuration parameters for the pipeline. See PipelineParams for details. ​ observers List[BaseObserver] default: "[]" List of observers for monitoring pipeline execution. See Observers for details. ​ clock BaseClock default: "SystemClock()" Clock implementation for timing operations. ​ task_manager Optional[BaseTaskManager] default: "None" Custom task manager for handling asyncio tasks. If None, a default TaskManager is used. ​ check_dangling_tasks bool default: "True" Whether to check for processors’ tasks finishing properly. ​ idle_timeout_secs Optional[float] default: "300" Timeout in seconds before considering the pipeline idle. Set to None to disable idle detection. See Pipeline Idle Detection for details. ​ idle_timeout_frames Tuple[Type[Frame], ...] default: "(BotSpeakingFrame, LLMFullResponseEndFrame)" Frame types that should prevent the pipeline from being considered idle. See Pipeline Idle Detection for details. ​ cancel_on_idle_timeout bool default: "True" Whether to automatically cancel the pipeline task when idle timeout is reached. See Pipeline Idle Detection for details. ​ enable_tracing bool default: "False" Whether to enable OpenTelemetry tracing. See The OpenTelemetry guide for details. ​ enable_turn_tracking bool default: "False" Whether to enable turn tracking. See The OpenTelemetry guide for details. ​ conversation_id Optional[str] default: "None" Custom ID for the conversation. If not provided, a UUID will be generated. See The OpenTelemetry guide for details. ​ additional_span_attributes Optional[dict] default: "None" Any additional attributes to add to top-level OpenTelemetry conversation span. See The OpenTelemetry guide for details. ​ Methods ​ Task Lifecycle Management ​ run() async Starts and manages the pipeline execution until completion or cancellation. Copy Ask AI await task.run() ​ stop_when_done() async Sends an EndFrame to the pipeline to gracefully stop the task after all queued frames have been processed. Copy Ask AI await task.stop_when_done() ​ cancel() async Stops the running pipeline immediately by sending a CancelFrame. Copy Ask AI await task.cancel() ​ has_finished() bool Returns whether the task has finished (all processors have stopped). Copy Ask AI if task.has_finished(): print ( "Task is complete" ) ​ Frame Management ​ queue_frame() async Queues a single frame to be pushed down the pipeline. Copy Ask AI await task.queue_frame(TTSSpeakFrame( "Hello!" )) ​ queue_frames() async Queues multiple frames to be pushed down the pipeline. Copy Ask AI frames = [TTSSpeakFrame( "Hello!" ), TTSSpeakFrame( "How are you?" )] await task.queue_frames(frames) ​ Event Handlers PipelineTask provides an event handler that can be registered using the event_handler decorator: ​ on_idle_timeout Triggered when no activity frames (as specified by idle_timeout_frames ) have been received within the idle timeout period. Copy Ask AI @task.event_handler ( "on_idle_timeout" ) async def on_idle_timeout ( task ): print ( "Pipeline has been idle too long" ) await task.queue_frame(TTSSpeakFrame( "Are you still there?" )) PipelineParams Pipeline Idle Detection On this page Overview Basic Usage Constructor Parameters Methods Task Lifecycle Management Frame Management Event Handlers on_idle_timeout Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/pipeline_pipeline-task_e5b8c9ec.txt b/pipeline_pipeline-task_e5b8c9ec.txt
new file mode 100644
index 0000000000000000000000000000000000000000..beec1b1520d317fdd80decad4485dd16dfc52260
--- /dev/null
+++ b/pipeline_pipeline-task_e5b8c9ec.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/pipeline/pipeline-task#param-pipeline
+Title: PipelineTask - Pipecat
+==================================================
+
+PipelineTask - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Pipeline PipelineTask Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview PipelineTask is the central class for managing pipeline execution. It handles the lifecycle of the pipeline, processes frames in both directions, manages task cancellation, and provides event handlers for monitoring pipeline activity. ​ Basic Usage Copy Ask AI from pipecat.pipeline.pipeline import Pipeline from pipecat.pipeline.runner import PipelineRunner from pipecat.pipeline.task import PipelineParams, PipelineTask # Create a pipeline pipeline = Pipeline([ ... ]) # Create a task with the pipeline task = PipelineTask(pipeline) # Queue frames for processing await task.queue_frame(TTSSpeakFrame( "Hello, how can I help you today?" )) # Run the pipeline runner = PipelineRunner() await runner.run(task) ​ Constructor Parameters ​ pipeline BasePipeline required The pipeline to execute. ​ params PipelineParams default: "PipelineParams()" Configuration parameters for the pipeline. See PipelineParams for details. ​ observers List[BaseObserver] default: "[]" List of observers for monitoring pipeline execution. See Observers for details. ​ clock BaseClock default: "SystemClock()" Clock implementation for timing operations. ​ task_manager Optional[BaseTaskManager] default: "None" Custom task manager for handling asyncio tasks. If None, a default TaskManager is used. ​ check_dangling_tasks bool default: "True" Whether to check for processors’ tasks finishing properly. ​ idle_timeout_secs Optional[float] default: "300" Timeout in seconds before considering the pipeline idle. Set to None to disable idle detection. See Pipeline Idle Detection for details. ​ idle_timeout_frames Tuple[Type[Frame], ...] default: "(BotSpeakingFrame, LLMFullResponseEndFrame)" Frame types that should prevent the pipeline from being considered idle. See Pipeline Idle Detection for details. ​ cancel_on_idle_timeout bool default: "True" Whether to automatically cancel the pipeline task when idle timeout is reached. See Pipeline Idle Detection for details. ​ enable_tracing bool default: "False" Whether to enable OpenTelemetry tracing. See The OpenTelemetry guide for details. ​ enable_turn_tracking bool default: "False" Whether to enable turn tracking. See The OpenTelemetry guide for details. ​ conversation_id Optional[str] default: "None" Custom ID for the conversation. If not provided, a UUID will be generated. See The OpenTelemetry guide for details. ​ additional_span_attributes Optional[dict] default: "None" Any additional attributes to add to top-level OpenTelemetry conversation span. See The OpenTelemetry guide for details. ​ Methods ​ Task Lifecycle Management ​ run() async Starts and manages the pipeline execution until completion or cancellation. Copy Ask AI await task.run() ​ stop_when_done() async Sends an EndFrame to the pipeline to gracefully stop the task after all queued frames have been processed. Copy Ask AI await task.stop_when_done() ​ cancel() async Stops the running pipeline immediately by sending a CancelFrame. Copy Ask AI await task.cancel() ​ has_finished() bool Returns whether the task has finished (all processors have stopped). Copy Ask AI if task.has_finished(): print ( "Task is complete" ) ​ Frame Management ​ queue_frame() async Queues a single frame to be pushed down the pipeline. Copy Ask AI await task.queue_frame(TTSSpeakFrame( "Hello!" )) ​ queue_frames() async Queues multiple frames to be pushed down the pipeline. Copy Ask AI frames = [TTSSpeakFrame( "Hello!" ), TTSSpeakFrame( "How are you?" )] await task.queue_frames(frames) ​ Event Handlers PipelineTask provides an event handler that can be registered using the event_handler decorator: ​ on_idle_timeout Triggered when no activity frames (as specified by idle_timeout_frames ) have been received within the idle timeout period. Copy Ask AI @task.event_handler ( "on_idle_timeout" ) async def on_idle_timeout ( task ): print ( "Pipeline has been idle too long" ) await task.queue_frame(TTSSpeakFrame( "Are you still there?" )) PipelineParams Pipeline Idle Detection On this page Overview Basic Usage Constructor Parameters Methods Task Lifecycle Management Frame Management Event Handlers on_idle_timeout Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/react-native_introduction_7b5bf05a.txt b/react-native_introduction_7b5bf05a.txt
new file mode 100644
index 0000000000000000000000000000000000000000..6b30a5f05113437b3e8a0955f3ffe651e6a8e9c6
--- /dev/null
+++ b/react-native_introduction_7b5bf05a.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/client/react-native/introduction#requirements
+Title: SDK Introduction - Pipecat
+==================================================
+
+SDK Introduction - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation React Native SDK SDK Introduction Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Client SDKs The RTVI Standard RTVIClient Migration Guide Javascript SDK SDK Introduction API Reference Transport packages React SDK SDK Introduction API Reference React Native SDK SDK Introduction API Reference iOS SDK SDK Introduction API Reference Transport packages Android SDK SDK Introduction API Reference Transport packages C++ SDK SDK Introduction Daily WebRTC Transport The Pipecat React Native SDK leverages the Pipecat JavaScript SDK to provide seamless integration for React Native applications. Since the JavaScript SDK is designed to work across both web and React Native platforms, the core functionalities remain the same: Device and media stream management Managing bot configuration Sending actions to the bot Handling bot messages and responses Managing session state and errors The primary difference lies in the transport layer, which is tailored to support the unique requirements of the React Native environment. For example, when using the SDK with React Native, you would install RNDailyTransport instead of DailyTransport . ​ Installation Install the SDK and a transport implementation (e.g. Daily for WebRTC): Copy Ask AI npm i @pipecat-ai/react-native-daily-transport npm i @daily-co/react-native-daily-js@^0.70.0 npm i @daily-co/react-native-webrtc@^118.0.3-daily.2 npm i @react-native-async-storage/async-storage@^1.23.1 npm i react-native-background-timer@^2.4.1 npm i react-native-get-random-values@^1.11.0 Installing @pipecat-ai/react-native-daily-transport automatically includes the corresponding version of the JavaScript SDK. If you are using Expo, you will also need to add the following dependencies: Copy Ask AI npm i @config-plugins/react-native-webrtc@^10.0.0 npm i @daily-co/config-plugin-rn-daily-js@0.0.7 ​ Requirements This package introduces some constraints on what OS/SDK versions your project can support: iOS: Deployment target >= 13 Android: minSdkVersion >= 24 ​ Quick start Here’s a simple example using Daily as the transport layer: Copy Ask AI import { RNDailyTransport } from '@pipecat-ai/react-native-daily-transport' ; import { RTVIClient } from '@pipecat-ai/client-js' ; // Create and configure the client let voiceClient = new RTVIClient ({ params: { baseUrl: process . env . PIPECAT_API_URL || "/api" , }, transport: new RNDailyTransport (), enableMic: true }); // Connect to your bot await voiceClient . connect (); You can find a basic working example here and a more comprehensive example here . ​ Explore the SDK The Pipecat React Native SDK leverages the Pipecat JavaScript SDK for seamless integration with React Native applications. For detailed information, refer to our JavaScript documentation. Just ensure you use the appropriate transport layer for React Native. Client Constructor Configure your client instance with transport and callbacks Client Methods Core methods for interacting with your bot API Reference Detailed documentation of all available APIs Helpers Utility functions for common operations Hooks API Reference On this page Installation Requirements Quick start Explore the SDK Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/react_components_0a2db5b0.txt b/react_components_0a2db5b0.txt
new file mode 100644
index 0000000000000000000000000000000000000000..8aeeee793b2db18a32006e8e7476450e24629193
--- /dev/null
+++ b/react_components_0a2db5b0.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/client/react/components#param-disabled-1
+Title: Components - Pipecat
+==================================================
+
+Components - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation API Reference Components Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Client SDKs The RTVI Standard RTVIClient Migration Guide Javascript SDK SDK Introduction API Reference Transport packages React SDK SDK Introduction API Reference Components Hooks React Native SDK SDK Introduction API Reference iOS SDK SDK Introduction API Reference Transport packages Android SDK SDK Introduction API Reference Transport packages C++ SDK SDK Introduction Daily WebRTC Transport The Pipecat React SDK provides several components for handling audio, video, and visualization in your application. ​ PipecatClientProvider The root component for providing Pipecat client context to your application. Copy Ask AI < PipecatClientProvider client = { pcClient } > { /* Child components */ } </ PipecatClientProvider > Props ​ client PipecatClient required A singleton instance of PipecatClient ​ PipecatClientAudio Creates a new <audio> element that mounts the bot’s audio track. Copy Ask AI < PipecatClientAudio /> Props No props required ​ PipecatClientVideo Creates a new <video> element that renders either the bot or local participant’s video track. Copy Ask AI < PipecatClientVideo participant = "local" fit = "cover" mirror onResize = { ({ aspectRatio , height , width }) => { console . log ( "Video dimensions changed:" , { aspectRatio , height , width }); } } /> Props ​ participant ('local' | 'bot') required Defines which participant’s video track is rendered ​ fit ('contain' | 'cover') Defines whether the video should be fully contained or cover the box. Default: ‘contain’ ​ mirror boolean Forces the video to be mirrored, if set ​ onResize(dimensions: object) function Triggered whenever the video’s rendered width or height changes ​ PipecatClientCamToggle A headless component to read and set the local participant’s camera state. Copy Ask AI < PipecatClientCamToggle onCamEnabledChanged = { ( enabled ) => console . log ( "Camera enabled:" , enabled ) } disabled = { false } > { ({ disabled , isCamEnabled , onClick }) => ( < button disabled = { disabled } onClick = { onClick } > { isCamEnabled ? "Disable Camera" : "Enable Camera" } </ button > ) } </ PipecatClientCamToggle > Props ​ onCamEnabledChanged(enabled: boolean) function Triggered whenever the local participant’s camera state changes ​ disabled boolean If true, the component will not allow toggling the camera state. Default: false ​ children function A render prop that provides state and handlers to the children ​ PipecatClientMicToggle A headless component to read and set the local participant’s microphone state. Copy Ask AI < PipecatClientMicToggle onMicEnabledChanged = { ( enabled ) => console . log ( "Microphone enabled:" , enabled ) } disabled = { false } > { ({ disabled , isMicEnabled , onClick }) => ( < button disabled = { disabled } onClick = { onClick } > { isMicEnabled ? "Disable Microphone" : "Enable Microphone" } </ button > ) } </ PipecatClientMicToggle > Props ​ onMicEnabledChanged(enabled: boolean) function Triggered whenever the local participant’s microphone state changes ​ disabled boolean If true, the component will not allow toggling the microphone state. Default: false ​ children function A render prop that provides state and handlers to the children ​ VoiceVisualizer Renders a visual representation of audio input levels on a <canvas> element. Copy Ask AI < VoiceVisualizer participantType = "local" backgroundColor = "white" barColor = "black" barGap = { 1 } barWidth = { 4 } barMaxHeight = { 24 } /> Props ​ participantType string required The participant type to visualize audio for ​ backgroundColor string The background color of the canvas. Default: ‘transparent’ ​ barColor string The color of the audio level bars. Default: ‘black’ ​ barCount number The number of bars to display. Default: 5 ​ barGap number The gap between bars in pixels. Default: 12 ​ barWidth number The width of each bar in pixels. Default: 30 ​ barMaxHeight number The maximum height at full volume of each bar in pixels. Default: 120 SDK Introduction Hooks On this page PipecatClientProvider PipecatClientAudio PipecatClientVideo PipecatClientCamToggle PipecatClientMicToggle VoiceVisualizer Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/react_components_0e5ef51f.txt b/react_components_0e5ef51f.txt
new file mode 100644
index 0000000000000000000000000000000000000000..7491f8d744eaaf54e488c9aca2bd94dc548db592
--- /dev/null
+++ b/react_components_0e5ef51f.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/client/react/components#param-on-resize-dimensions-object
+Title: Components - Pipecat
+==================================================
+
+Components - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation API Reference Components Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Client SDKs The RTVI Standard RTVIClient Migration Guide Javascript SDK SDK Introduction API Reference Transport packages React SDK SDK Introduction API Reference Components Hooks React Native SDK SDK Introduction API Reference iOS SDK SDK Introduction API Reference Transport packages Android SDK SDK Introduction API Reference Transport packages C++ SDK SDK Introduction Daily WebRTC Transport The Pipecat React SDK provides several components for handling audio, video, and visualization in your application. ​ PipecatClientProvider The root component for providing Pipecat client context to your application. Copy Ask AI < PipecatClientProvider client = { pcClient } > { /* Child components */ } </ PipecatClientProvider > Props ​ client PipecatClient required A singleton instance of PipecatClient ​ PipecatClientAudio Creates a new <audio> element that mounts the bot’s audio track. Copy Ask AI < PipecatClientAudio /> Props No props required ​ PipecatClientVideo Creates a new <video> element that renders either the bot or local participant’s video track. Copy Ask AI < PipecatClientVideo participant = "local" fit = "cover" mirror onResize = { ({ aspectRatio , height , width }) => { console . log ( "Video dimensions changed:" , { aspectRatio , height , width }); } } /> Props ​ participant ('local' | 'bot') required Defines which participant’s video track is rendered ​ fit ('contain' | 'cover') Defines whether the video should be fully contained or cover the box. Default: ‘contain’ ​ mirror boolean Forces the video to be mirrored, if set ​ onResize(dimensions: object) function Triggered whenever the video’s rendered width or height changes ​ PipecatClientCamToggle A headless component to read and set the local participant’s camera state. Copy Ask AI < PipecatClientCamToggle onCamEnabledChanged = { ( enabled ) => console . log ( "Camera enabled:" , enabled ) } disabled = { false } > { ({ disabled , isCamEnabled , onClick }) => ( < button disabled = { disabled } onClick = { onClick } > { isCamEnabled ? "Disable Camera" : "Enable Camera" } </ button > ) } </ PipecatClientCamToggle > Props ​ onCamEnabledChanged(enabled: boolean) function Triggered whenever the local participant’s camera state changes ​ disabled boolean If true, the component will not allow toggling the camera state. Default: false ​ children function A render prop that provides state and handlers to the children ​ PipecatClientMicToggle A headless component to read and set the local participant’s microphone state. Copy Ask AI < PipecatClientMicToggle onMicEnabledChanged = { ( enabled ) => console . log ( "Microphone enabled:" , enabled ) } disabled = { false } > { ({ disabled , isMicEnabled , onClick }) => ( < button disabled = { disabled } onClick = { onClick } > { isMicEnabled ? "Disable Microphone" : "Enable Microphone" } </ button > ) } </ PipecatClientMicToggle > Props ​ onMicEnabledChanged(enabled: boolean) function Triggered whenever the local participant’s microphone state changes ​ disabled boolean If true, the component will not allow toggling the microphone state. Default: false ​ children function A render prop that provides state and handlers to the children ​ VoiceVisualizer Renders a visual representation of audio input levels on a <canvas> element. Copy Ask AI < VoiceVisualizer participantType = "local" backgroundColor = "white" barColor = "black" barGap = { 1 } barWidth = { 4 } barMaxHeight = { 24 } /> Props ​ participantType string required The participant type to visualize audio for ​ backgroundColor string The background color of the canvas. Default: ‘transparent’ ​ barColor string The color of the audio level bars. Default: ‘black’ ​ barCount number The number of bars to display. Default: 5 ​ barGap number The gap between bars in pixels. Default: 12 ​ barWidth number The width of each bar in pixels. Default: 30 ​ barMaxHeight number The maximum height at full volume of each bar in pixels. Default: 120 SDK Introduction Hooks On this page PipecatClientProvider PipecatClientAudio PipecatClientVideo PipecatClientCamToggle PipecatClientMicToggle VoiceVisualizer Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/react_components_2e084ebd.txt b/react_components_2e084ebd.txt
new file mode 100644
index 0000000000000000000000000000000000000000..2c3c3d18f043ae7bdc5b0b683914d400e7d46deb
--- /dev/null
+++ b/react_components_2e084ebd.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/client/react/components#param-client
+Title: Components - Pipecat
+==================================================
+
+Components - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation API Reference Components Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Client SDKs The RTVI Standard RTVIClient Migration Guide Javascript SDK SDK Introduction API Reference Transport packages React SDK SDK Introduction API Reference Components Hooks React Native SDK SDK Introduction API Reference iOS SDK SDK Introduction API Reference Transport packages Android SDK SDK Introduction API Reference Transport packages C++ SDK SDK Introduction Daily WebRTC Transport The Pipecat React SDK provides several components for handling audio, video, and visualization in your application. ​ PipecatClientProvider The root component for providing Pipecat client context to your application. Copy Ask AI < PipecatClientProvider client = { pcClient } > { /* Child components */ } </ PipecatClientProvider > Props ​ client PipecatClient required A singleton instance of PipecatClient ​ PipecatClientAudio Creates a new <audio> element that mounts the bot’s audio track. Copy Ask AI < PipecatClientAudio /> Props No props required ​ PipecatClientVideo Creates a new <video> element that renders either the bot or local participant’s video track. Copy Ask AI < PipecatClientVideo participant = "local" fit = "cover" mirror onResize = { ({ aspectRatio , height , width }) => { console . log ( "Video dimensions changed:" , { aspectRatio , height , width }); } } /> Props ​ participant ('local' | 'bot') required Defines which participant’s video track is rendered ​ fit ('contain' | 'cover') Defines whether the video should be fully contained or cover the box. Default: ‘contain’ ​ mirror boolean Forces the video to be mirrored, if set ​ onResize(dimensions: object) function Triggered whenever the video’s rendered width or height changes ​ PipecatClientCamToggle A headless component to read and set the local participant’s camera state. Copy Ask AI < PipecatClientCamToggle onCamEnabledChanged = { ( enabled ) => console . log ( "Camera enabled:" , enabled ) } disabled = { false } > { ({ disabled , isCamEnabled , onClick }) => ( < button disabled = { disabled } onClick = { onClick } > { isCamEnabled ? "Disable Camera" : "Enable Camera" } </ button > ) } </ PipecatClientCamToggle > Props ​ onCamEnabledChanged(enabled: boolean) function Triggered whenever the local participant’s camera state changes ​ disabled boolean If true, the component will not allow toggling the camera state. Default: false ​ children function A render prop that provides state and handlers to the children ​ PipecatClientMicToggle A headless component to read and set the local participant’s microphone state. Copy Ask AI < PipecatClientMicToggle onMicEnabledChanged = { ( enabled ) => console . log ( "Microphone enabled:" , enabled ) } disabled = { false } > { ({ disabled , isMicEnabled , onClick }) => ( < button disabled = { disabled } onClick = { onClick } > { isMicEnabled ? "Disable Microphone" : "Enable Microphone" } </ button > ) } </ PipecatClientMicToggle > Props ​ onMicEnabledChanged(enabled: boolean) function Triggered whenever the local participant’s microphone state changes ​ disabled boolean If true, the component will not allow toggling the microphone state. Default: false ​ children function A render prop that provides state and handlers to the children ​ VoiceVisualizer Renders a visual representation of audio input levels on a <canvas> element. Copy Ask AI < VoiceVisualizer participantType = "local" backgroundColor = "white" barColor = "black" barGap = { 1 } barWidth = { 4 } barMaxHeight = { 24 } /> Props ​ participantType string required The participant type to visualize audio for ​ backgroundColor string The background color of the canvas. Default: ‘transparent’ ​ barColor string The color of the audio level bars. Default: ‘black’ ​ barCount number The number of bars to display. Default: 5 ​ barGap number The gap between bars in pixels. Default: 12 ​ barWidth number The width of each bar in pixels. Default: 30 ​ barMaxHeight number The maximum height at full volume of each bar in pixels. Default: 120 SDK Introduction Hooks On this page PipecatClientProvider PipecatClientAudio PipecatClientVideo PipecatClientCamToggle PipecatClientMicToggle VoiceVisualizer Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/react_components_4ef4b57b.txt b/react_components_4ef4b57b.txt
new file mode 100644
index 0000000000000000000000000000000000000000..20e448976dfefec138ee838151ae793c554fdcd4
--- /dev/null
+++ b/react_components_4ef4b57b.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/client/react/components#pipecatclientvideo
+Title: Components - Pipecat
+==================================================
+
+Components - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation API Reference Components Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Client SDKs The RTVI Standard RTVIClient Migration Guide Javascript SDK SDK Introduction API Reference Transport packages React SDK SDK Introduction API Reference Components Hooks React Native SDK SDK Introduction API Reference iOS SDK SDK Introduction API Reference Transport packages Android SDK SDK Introduction API Reference Transport packages C++ SDK SDK Introduction Daily WebRTC Transport The Pipecat React SDK provides several components for handling audio, video, and visualization in your application. ​ PipecatClientProvider The root component for providing Pipecat client context to your application. Copy Ask AI < PipecatClientProvider client = { pcClient } > { /* Child components */ } </ PipecatClientProvider > Props ​ client PipecatClient required A singleton instance of PipecatClient ​ PipecatClientAudio Creates a new <audio> element that mounts the bot’s audio track. Copy Ask AI < PipecatClientAudio /> Props No props required ​ PipecatClientVideo Creates a new <video> element that renders either the bot or local participant’s video track. Copy Ask AI < PipecatClientVideo participant = "local" fit = "cover" mirror onResize = { ({ aspectRatio , height , width }) => { console . log ( "Video dimensions changed:" , { aspectRatio , height , width }); } } /> Props ​ participant ('local' | 'bot') required Defines which participant’s video track is rendered ​ fit ('contain' | 'cover') Defines whether the video should be fully contained or cover the box. Default: ‘contain’ ​ mirror boolean Forces the video to be mirrored, if set ​ onResize(dimensions: object) function Triggered whenever the video’s rendered width or height changes ​ PipecatClientCamToggle A headless component to read and set the local participant’s camera state. Copy Ask AI < PipecatClientCamToggle onCamEnabledChanged = { ( enabled ) => console . log ( "Camera enabled:" , enabled ) } disabled = { false } > { ({ disabled , isCamEnabled , onClick }) => ( < button disabled = { disabled } onClick = { onClick } > { isCamEnabled ? "Disable Camera" : "Enable Camera" } </ button > ) } </ PipecatClientCamToggle > Props ​ onCamEnabledChanged(enabled: boolean) function Triggered whenever the local participant’s camera state changes ​ disabled boolean If true, the component will not allow toggling the camera state. Default: false ​ children function A render prop that provides state and handlers to the children ​ PipecatClientMicToggle A headless component to read and set the local participant’s microphone state. Copy Ask AI < PipecatClientMicToggle onMicEnabledChanged = { ( enabled ) => console . log ( "Microphone enabled:" , enabled ) } disabled = { false } > { ({ disabled , isMicEnabled , onClick }) => ( < button disabled = { disabled } onClick = { onClick } > { isMicEnabled ? "Disable Microphone" : "Enable Microphone" } </ button > ) } </ PipecatClientMicToggle > Props ​ onMicEnabledChanged(enabled: boolean) function Triggered whenever the local participant’s microphone state changes ​ disabled boolean If true, the component will not allow toggling the microphone state. Default: false ​ children function A render prop that provides state and handlers to the children ​ VoiceVisualizer Renders a visual representation of audio input levels on a <canvas> element. Copy Ask AI < VoiceVisualizer participantType = "local" backgroundColor = "white" barColor = "black" barGap = { 1 } barWidth = { 4 } barMaxHeight = { 24 } /> Props ​ participantType string required The participant type to visualize audio for ​ backgroundColor string The background color of the canvas. Default: ‘transparent’ ​ barColor string The color of the audio level bars. Default: ‘black’ ​ barCount number The number of bars to display. Default: 5 ​ barGap number The gap between bars in pixels. Default: 12 ​ barWidth number The width of each bar in pixels. Default: 30 ​ barMaxHeight number The maximum height at full volume of each bar in pixels. Default: 120 SDK Introduction Hooks On this page PipecatClientProvider PipecatClientAudio PipecatClientVideo PipecatClientCamToggle PipecatClientMicToggle VoiceVisualizer Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/react_components_677539e7.txt b/react_components_677539e7.txt
new file mode 100644
index 0000000000000000000000000000000000000000..8973d07a4b60867ab4f5dc3d1ad5aa61ace8b96a
--- /dev/null
+++ b/react_components_677539e7.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/client/react/components#param-on-cam-enabled-changed-enabled-boolean
+Title: Components - Pipecat
+==================================================
+
+Components - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation API Reference Components Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Client SDKs The RTVI Standard RTVIClient Migration Guide Javascript SDK SDK Introduction API Reference Transport packages React SDK SDK Introduction API Reference Components Hooks React Native SDK SDK Introduction API Reference iOS SDK SDK Introduction API Reference Transport packages Android SDK SDK Introduction API Reference Transport packages C++ SDK SDK Introduction Daily WebRTC Transport The Pipecat React SDK provides several components for handling audio, video, and visualization in your application. ​ PipecatClientProvider The root component for providing Pipecat client context to your application. Copy Ask AI < PipecatClientProvider client = { pcClient } > { /* Child components */ } </ PipecatClientProvider > Props ​ client PipecatClient required A singleton instance of PipecatClient ​ PipecatClientAudio Creates a new <audio> element that mounts the bot’s audio track. Copy Ask AI < PipecatClientAudio /> Props No props required ​ PipecatClientVideo Creates a new <video> element that renders either the bot or local participant’s video track. Copy Ask AI < PipecatClientVideo participant = "local" fit = "cover" mirror onResize = { ({ aspectRatio , height , width }) => { console . log ( "Video dimensions changed:" , { aspectRatio , height , width }); } } /> Props ​ participant ('local' | 'bot') required Defines which participant’s video track is rendered ​ fit ('contain' | 'cover') Defines whether the video should be fully contained or cover the box. Default: ‘contain’ ​ mirror boolean Forces the video to be mirrored, if set ​ onResize(dimensions: object) function Triggered whenever the video’s rendered width or height changes ​ PipecatClientCamToggle A headless component to read and set the local participant’s camera state. Copy Ask AI < PipecatClientCamToggle onCamEnabledChanged = { ( enabled ) => console . log ( "Camera enabled:" , enabled ) } disabled = { false } > { ({ disabled , isCamEnabled , onClick }) => ( < button disabled = { disabled } onClick = { onClick } > { isCamEnabled ? "Disable Camera" : "Enable Camera" } </ button > ) } </ PipecatClientCamToggle > Props ​ onCamEnabledChanged(enabled: boolean) function Triggered whenever the local participant’s camera state changes ​ disabled boolean If true, the component will not allow toggling the camera state. Default: false ​ children function A render prop that provides state and handlers to the children ​ PipecatClientMicToggle A headless component to read and set the local participant’s microphone state. Copy Ask AI < PipecatClientMicToggle onMicEnabledChanged = { ( enabled ) => console . log ( "Microphone enabled:" , enabled ) } disabled = { false } > { ({ disabled , isMicEnabled , onClick }) => ( < button disabled = { disabled } onClick = { onClick } > { isMicEnabled ? "Disable Microphone" : "Enable Microphone" } </ button > ) } </ PipecatClientMicToggle > Props ​ onMicEnabledChanged(enabled: boolean) function Triggered whenever the local participant’s microphone state changes ​ disabled boolean If true, the component will not allow toggling the microphone state. Default: false ​ children function A render prop that provides state and handlers to the children ​ VoiceVisualizer Renders a visual representation of audio input levels on a <canvas> element. Copy Ask AI < VoiceVisualizer participantType = "local" backgroundColor = "white" barColor = "black" barGap = { 1 } barWidth = { 4 } barMaxHeight = { 24 } /> Props ​ participantType string required The participant type to visualize audio for ​ backgroundColor string The background color of the canvas. Default: ‘transparent’ ​ barColor string The color of the audio level bars. Default: ‘black’ ​ barCount number The number of bars to display. Default: 5 ​ barGap number The gap between bars in pixels. Default: 12 ​ barWidth number The width of each bar in pixels. Default: 30 ​ barMaxHeight number The maximum height at full volume of each bar in pixels. Default: 120 SDK Introduction Hooks On this page PipecatClientProvider PipecatClientAudio PipecatClientVideo PipecatClientCamToggle PipecatClientMicToggle VoiceVisualizer Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/react_components_6a5009ab.txt b/react_components_6a5009ab.txt
new file mode 100644
index 0000000000000000000000000000000000000000..0ad3033a914628f715b710f5e8f565a617f653fb
--- /dev/null
+++ b/react_components_6a5009ab.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/client/react/components#param-mirror
+Title: Components - Pipecat
+==================================================
+
+Components - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation API Reference Components Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Client SDKs The RTVI Standard RTVIClient Migration Guide Javascript SDK SDK Introduction API Reference Transport packages React SDK SDK Introduction API Reference Components Hooks React Native SDK SDK Introduction API Reference iOS SDK SDK Introduction API Reference Transport packages Android SDK SDK Introduction API Reference Transport packages C++ SDK SDK Introduction Daily WebRTC Transport The Pipecat React SDK provides several components for handling audio, video, and visualization in your application. ​ PipecatClientProvider The root component for providing Pipecat client context to your application. Copy Ask AI < PipecatClientProvider client = { pcClient } > { /* Child components */ } </ PipecatClientProvider > Props ​ client PipecatClient required A singleton instance of PipecatClient ​ PipecatClientAudio Creates a new <audio> element that mounts the bot’s audio track. Copy Ask AI < PipecatClientAudio /> Props No props required ​ PipecatClientVideo Creates a new <video> element that renders either the bot or local participant’s video track. Copy Ask AI < PipecatClientVideo participant = "local" fit = "cover" mirror onResize = { ({ aspectRatio , height , width }) => { console . log ( "Video dimensions changed:" , { aspectRatio , height , width }); } } /> Props ​ participant ('local' | 'bot') required Defines which participant’s video track is rendered ​ fit ('contain' | 'cover') Defines whether the video should be fully contained or cover the box. Default: ‘contain’ ​ mirror boolean Forces the video to be mirrored, if set ​ onResize(dimensions: object) function Triggered whenever the video’s rendered width or height changes ​ PipecatClientCamToggle A headless component to read and set the local participant’s camera state. Copy Ask AI < PipecatClientCamToggle onCamEnabledChanged = { ( enabled ) => console . log ( "Camera enabled:" , enabled ) } disabled = { false } > { ({ disabled , isCamEnabled , onClick }) => ( < button disabled = { disabled } onClick = { onClick } > { isCamEnabled ? "Disable Camera" : "Enable Camera" } </ button > ) } </ PipecatClientCamToggle > Props ​ onCamEnabledChanged(enabled: boolean) function Triggered whenever the local participant’s camera state changes ​ disabled boolean If true, the component will not allow toggling the camera state. Default: false ​ children function A render prop that provides state and handlers to the children ​ PipecatClientMicToggle A headless component to read and set the local participant’s microphone state. Copy Ask AI < PipecatClientMicToggle onMicEnabledChanged = { ( enabled ) => console . log ( "Microphone enabled:" , enabled ) } disabled = { false } > { ({ disabled , isMicEnabled , onClick }) => ( < button disabled = { disabled } onClick = { onClick } > { isMicEnabled ? "Disable Microphone" : "Enable Microphone" } </ button > ) } </ PipecatClientMicToggle > Props ​ onMicEnabledChanged(enabled: boolean) function Triggered whenever the local participant’s microphone state changes ​ disabled boolean If true, the component will not allow toggling the microphone state. Default: false ​ children function A render prop that provides state and handlers to the children ​ VoiceVisualizer Renders a visual representation of audio input levels on a <canvas> element. Copy Ask AI < VoiceVisualizer participantType = "local" backgroundColor = "white" barColor = "black" barGap = { 1 } barWidth = { 4 } barMaxHeight = { 24 } /> Props ​ participantType string required The participant type to visualize audio for ​ backgroundColor string The background color of the canvas. Default: ‘transparent’ ​ barColor string The color of the audio level bars. Default: ‘black’ ​ barCount number The number of bars to display. Default: 5 ​ barGap number The gap between bars in pixels. Default: 12 ​ barWidth number The width of each bar in pixels. Default: 30 ​ barMaxHeight number The maximum height at full volume of each bar in pixels. Default: 120 SDK Introduction Hooks On this page PipecatClientProvider PipecatClientAudio PipecatClientVideo PipecatClientCamToggle PipecatClientMicToggle VoiceVisualizer Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/react_components_736b609a.txt b/react_components_736b609a.txt
new file mode 100644
index 0000000000000000000000000000000000000000..e657851ba0779f8242594f243fe4f0dfaccaa179
--- /dev/null
+++ b/react_components_736b609a.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/client/react/components#param-bar-width
+Title: Components - Pipecat
+==================================================
+
+Components - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation API Reference Components Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Client SDKs The RTVI Standard RTVIClient Migration Guide Javascript SDK SDK Introduction API Reference Transport packages React SDK SDK Introduction API Reference Components Hooks React Native SDK SDK Introduction API Reference iOS SDK SDK Introduction API Reference Transport packages Android SDK SDK Introduction API Reference Transport packages C++ SDK SDK Introduction Daily WebRTC Transport The Pipecat React SDK provides several components for handling audio, video, and visualization in your application. ​ PipecatClientProvider The root component for providing Pipecat client context to your application. Copy Ask AI < PipecatClientProvider client = { pcClient } > { /* Child components */ } </ PipecatClientProvider > Props ​ client PipecatClient required A singleton instance of PipecatClient ​ PipecatClientAudio Creates a new <audio> element that mounts the bot’s audio track. Copy Ask AI < PipecatClientAudio /> Props No props required ​ PipecatClientVideo Creates a new <video> element that renders either the bot or local participant’s video track. Copy Ask AI < PipecatClientVideo participant = "local" fit = "cover" mirror onResize = { ({ aspectRatio , height , width }) => { console . log ( "Video dimensions changed:" , { aspectRatio , height , width }); } } /> Props ​ participant ('local' | 'bot') required Defines which participant’s video track is rendered ​ fit ('contain' | 'cover') Defines whether the video should be fully contained or cover the box. Default: ‘contain’ ​ mirror boolean Forces the video to be mirrored, if set ​ onResize(dimensions: object) function Triggered whenever the video’s rendered width or height changes ​ PipecatClientCamToggle A headless component to read and set the local participant’s camera state. Copy Ask AI < PipecatClientCamToggle onCamEnabledChanged = { ( enabled ) => console . log ( "Camera enabled:" , enabled ) } disabled = { false } > { ({ disabled , isCamEnabled , onClick }) => ( < button disabled = { disabled } onClick = { onClick } > { isCamEnabled ? "Disable Camera" : "Enable Camera" } </ button > ) } </ PipecatClientCamToggle > Props ​ onCamEnabledChanged(enabled: boolean) function Triggered whenever the local participant’s camera state changes ​ disabled boolean If true, the component will not allow toggling the camera state. Default: false ​ children function A render prop that provides state and handlers to the children ​ PipecatClientMicToggle A headless component to read and set the local participant’s microphone state. Copy Ask AI < PipecatClientMicToggle onMicEnabledChanged = { ( enabled ) => console . log ( "Microphone enabled:" , enabled ) } disabled = { false } > { ({ disabled , isMicEnabled , onClick }) => ( < button disabled = { disabled } onClick = { onClick } > { isMicEnabled ? "Disable Microphone" : "Enable Microphone" } </ button > ) } </ PipecatClientMicToggle > Props ​ onMicEnabledChanged(enabled: boolean) function Triggered whenever the local participant’s microphone state changes ​ disabled boolean If true, the component will not allow toggling the microphone state. Default: false ​ children function A render prop that provides state and handlers to the children ​ VoiceVisualizer Renders a visual representation of audio input levels on a <canvas> element. Copy Ask AI < VoiceVisualizer participantType = "local" backgroundColor = "white" barColor = "black" barGap = { 1 } barWidth = { 4 } barMaxHeight = { 24 } /> Props ​ participantType string required The participant type to visualize audio for ​ backgroundColor string The background color of the canvas. Default: ‘transparent’ ​ barColor string The color of the audio level bars. Default: ‘black’ ​ barCount number The number of bars to display. Default: 5 ​ barGap number The gap between bars in pixels. Default: 12 ​ barWidth number The width of each bar in pixels. Default: 30 ​ barMaxHeight number The maximum height at full volume of each bar in pixels. Default: 120 SDK Introduction Hooks On this page PipecatClientProvider PipecatClientAudio PipecatClientVideo PipecatClientCamToggle PipecatClientMicToggle VoiceVisualizer Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/react_components_ad7986fa.txt b/react_components_ad7986fa.txt
new file mode 100644
index 0000000000000000000000000000000000000000..999092278fff061cef7bc447d1a00bf28f86f06f
--- /dev/null
+++ b/react_components_ad7986fa.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/client/react/components#param-on-mic-enabled-changed-enabled-boolean
+Title: Components - Pipecat
+==================================================
+
+Components - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation API Reference Components Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Client SDKs The RTVI Standard RTVIClient Migration Guide Javascript SDK SDK Introduction API Reference Transport packages React SDK SDK Introduction API Reference Components Hooks React Native SDK SDK Introduction API Reference iOS SDK SDK Introduction API Reference Transport packages Android SDK SDK Introduction API Reference Transport packages C++ SDK SDK Introduction Daily WebRTC Transport The Pipecat React SDK provides several components for handling audio, video, and visualization in your application. ​ PipecatClientProvider The root component for providing Pipecat client context to your application. Copy Ask AI < PipecatClientProvider client = { pcClient } > { /* Child components */ } </ PipecatClientProvider > Props ​ client PipecatClient required A singleton instance of PipecatClient ​ PipecatClientAudio Creates a new <audio> element that mounts the bot’s audio track. Copy Ask AI < PipecatClientAudio /> Props No props required ​ PipecatClientVideo Creates a new <video> element that renders either the bot or local participant’s video track. Copy Ask AI < PipecatClientVideo participant = "local" fit = "cover" mirror onResize = { ({ aspectRatio , height , width }) => { console . log ( "Video dimensions changed:" , { aspectRatio , height , width }); } } /> Props ​ participant ('local' | 'bot') required Defines which participant’s video track is rendered ​ fit ('contain' | 'cover') Defines whether the video should be fully contained or cover the box. Default: ‘contain’ ​ mirror boolean Forces the video to be mirrored, if set ​ onResize(dimensions: object) function Triggered whenever the video’s rendered width or height changes ​ PipecatClientCamToggle A headless component to read and set the local participant’s camera state. Copy Ask AI < PipecatClientCamToggle onCamEnabledChanged = { ( enabled ) => console . log ( "Camera enabled:" , enabled ) } disabled = { false } > { ({ disabled , isCamEnabled , onClick }) => ( < button disabled = { disabled } onClick = { onClick } > { isCamEnabled ? "Disable Camera" : "Enable Camera" } </ button > ) } </ PipecatClientCamToggle > Props ​ onCamEnabledChanged(enabled: boolean) function Triggered whenever the local participant’s camera state changes ​ disabled boolean If true, the component will not allow toggling the camera state. Default: false ​ children function A render prop that provides state and handlers to the children ​ PipecatClientMicToggle A headless component to read and set the local participant’s microphone state. Copy Ask AI < PipecatClientMicToggle onMicEnabledChanged = { ( enabled ) => console . log ( "Microphone enabled:" , enabled ) } disabled = { false } > { ({ disabled , isMicEnabled , onClick }) => ( < button disabled = { disabled } onClick = { onClick } > { isMicEnabled ? "Disable Microphone" : "Enable Microphone" } </ button > ) } </ PipecatClientMicToggle > Props ​ onMicEnabledChanged(enabled: boolean) function Triggered whenever the local participant’s microphone state changes ​ disabled boolean If true, the component will not allow toggling the microphone state. Default: false ​ children function A render prop that provides state and handlers to the children ​ VoiceVisualizer Renders a visual representation of audio input levels on a <canvas> element. Copy Ask AI < VoiceVisualizer participantType = "local" backgroundColor = "white" barColor = "black" barGap = { 1 } barWidth = { 4 } barMaxHeight = { 24 } /> Props ​ participantType string required The participant type to visualize audio for ​ backgroundColor string The background color of the canvas. Default: ‘transparent’ ​ barColor string The color of the audio level bars. Default: ‘black’ ​ barCount number The number of bars to display. Default: 5 ​ barGap number The gap between bars in pixels. Default: 12 ​ barWidth number The width of each bar in pixels. Default: 30 ​ barMaxHeight number The maximum height at full volume of each bar in pixels. Default: 120 SDK Introduction Hooks On this page PipecatClientProvider PipecatClientAudio PipecatClientVideo PipecatClientCamToggle PipecatClientMicToggle VoiceVisualizer Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/react_components_d4799f66.txt b/react_components_d4799f66.txt
new file mode 100644
index 0000000000000000000000000000000000000000..6efb0dc2cb8470354aff5645876d61c3a09b0c25
--- /dev/null
+++ b/react_components_d4799f66.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/client/react/components#param-fit
+Title: Components - Pipecat
+==================================================
+
+Components - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation API Reference Components Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Client SDKs The RTVI Standard RTVIClient Migration Guide Javascript SDK SDK Introduction API Reference Transport packages React SDK SDK Introduction API Reference Components Hooks React Native SDK SDK Introduction API Reference iOS SDK SDK Introduction API Reference Transport packages Android SDK SDK Introduction API Reference Transport packages C++ SDK SDK Introduction Daily WebRTC Transport The Pipecat React SDK provides several components for handling audio, video, and visualization in your application. ​ PipecatClientProvider The root component for providing Pipecat client context to your application. Copy Ask AI < PipecatClientProvider client = { pcClient } > { /* Child components */ } </ PipecatClientProvider > Props ​ client PipecatClient required A singleton instance of PipecatClient ​ PipecatClientAudio Creates a new <audio> element that mounts the bot’s audio track. Copy Ask AI < PipecatClientAudio /> Props No props required ​ PipecatClientVideo Creates a new <video> element that renders either the bot or local participant’s video track. Copy Ask AI < PipecatClientVideo participant = "local" fit = "cover" mirror onResize = { ({ aspectRatio , height , width }) => { console . log ( "Video dimensions changed:" , { aspectRatio , height , width }); } } /> Props ​ participant ('local' | 'bot') required Defines which participant’s video track is rendered ​ fit ('contain' | 'cover') Defines whether the video should be fully contained or cover the box. Default: ‘contain’ ​ mirror boolean Forces the video to be mirrored, if set ​ onResize(dimensions: object) function Triggered whenever the video’s rendered width or height changes ​ PipecatClientCamToggle A headless component to read and set the local participant’s camera state. Copy Ask AI < PipecatClientCamToggle onCamEnabledChanged = { ( enabled ) => console . log ( "Camera enabled:" , enabled ) } disabled = { false } > { ({ disabled , isCamEnabled , onClick }) => ( < button disabled = { disabled } onClick = { onClick } > { isCamEnabled ? "Disable Camera" : "Enable Camera" } </ button > ) } </ PipecatClientCamToggle > Props ​ onCamEnabledChanged(enabled: boolean) function Triggered whenever the local participant’s camera state changes ​ disabled boolean If true, the component will not allow toggling the camera state. Default: false ​ children function A render prop that provides state and handlers to the children ​ PipecatClientMicToggle A headless component to read and set the local participant’s microphone state. Copy Ask AI < PipecatClientMicToggle onMicEnabledChanged = { ( enabled ) => console . log ( "Microphone enabled:" , enabled ) } disabled = { false } > { ({ disabled , isMicEnabled , onClick }) => ( < button disabled = { disabled } onClick = { onClick } > { isMicEnabled ? "Disable Microphone" : "Enable Microphone" } </ button > ) } </ PipecatClientMicToggle > Props ​ onMicEnabledChanged(enabled: boolean) function Triggered whenever the local participant’s microphone state changes ​ disabled boolean If true, the component will not allow toggling the microphone state. Default: false ​ children function A render prop that provides state and handlers to the children ​ VoiceVisualizer Renders a visual representation of audio input levels on a <canvas> element. Copy Ask AI < VoiceVisualizer participantType = "local" backgroundColor = "white" barColor = "black" barGap = { 1 } barWidth = { 4 } barMaxHeight = { 24 } /> Props ​ participantType string required The participant type to visualize audio for ​ backgroundColor string The background color of the canvas. Default: ‘transparent’ ​ barColor string The color of the audio level bars. Default: ‘black’ ​ barCount number The number of bars to display. Default: 5 ​ barGap number The gap between bars in pixels. Default: 12 ​ barWidth number The width of each bar in pixels. Default: 30 ​ barMaxHeight number The maximum height at full volume of each bar in pixels. Default: 120 SDK Introduction Hooks On this page PipecatClientProvider PipecatClientAudio PipecatClientVideo PipecatClientCamToggle PipecatClientMicToggle VoiceVisualizer Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/react_hooks_375a597c.txt b/react_hooks_375a597c.txt
new file mode 100644
index 0000000000000000000000000000000000000000..589deff1147745ecd69f00bd92370ab751cd7d82
--- /dev/null
+++ b/react_hooks_375a597c.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/client/react/hooks
+Title: Hooks - Pipecat
+==================================================
+
+Hooks - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation API Reference Hooks Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Client SDKs The RTVI Standard RTVIClient Migration Guide Javascript SDK SDK Introduction API Reference Transport packages React SDK SDK Introduction API Reference Components Hooks React Native SDK SDK Introduction API Reference iOS SDK SDK Introduction API Reference Transport packages Android SDK SDK Introduction API Reference Transport packages C++ SDK SDK Introduction Daily WebRTC Transport The Pipecat React SDK provides hooks for accessing client functionality, managing media devices, and handling events. ​ usePipecatClient Provides access to the PipecatClient instance originally passed to PipecatClientProvider . Copy Ask AI import { usePipecatClient } from "@pipecat-ai/client-react" ; function MyComponent () { const pcClient = usePipecatClient (); await pcClient . connect ({ endpoint: 'https://your-pipecat-api-url/connect' , requestData: { // Any custom data your /connect endpoint requires } }); } ​ useRTVIClientEvent Allows subscribing to RTVI client events. It is advised to wrap handlers with useCallback . Copy Ask AI import { useCallback } from "react" ; import { RTVIEvent , TransportState } from "@pipecat-ai/client-js" ; import { useRTVIClientEvent } from "@pipecat-ai/client-react" ; function EventListener () { useRTVIClientEvent ( RTVIEvent . TransportStateChanged , useCallback (( transportState : TransportState ) => { console . log ( "Transport state changed to" , transportState ); }, []) ); } Arguments ​ event RTVIEvent required ​ handler function required ​ usePipecatClientMediaDevices Manage and list available media devices. Copy Ask AI import { usePipecatClientMediaDevices } from "@pipecat-ai/client-react" ; function DeviceSelector () { const { availableCams , availableMics , selectedCam , selectedMic , updateCam , updateMic , } = usePipecatClientMediaDevices (); return ( <> < select name = "cam" onChange = { ( ev ) => updateCam ( ev . target . value ) } value = { selectedCam ?. deviceId } > { availableCams . map (( cam ) => ( < option key = { cam . deviceId } value = { cam . deviceId } > { cam . label } </ option > )) } </ select > < select name = "mic" onChange = { ( ev ) => updateMic ( ev . target . value ) } value = { selectedMic ?. deviceId } > { availableMics . map (( mic ) => ( < option key = { mic . deviceId } value = { mic . deviceId } > { mic . label } </ option > )) } </ select > </> ); } ​ usePipecatClientMediaTrack Access audio and video tracks. Copy Ask AI import { usePipecatClientMediaTrack } from "@pipecat-ai/client-react" ; function MyTracks () { const localAudioTrack = usePipecatClientMediaTrack ( "audio" , "local" ); const botAudioTrack = usePipecatClientMediaTrack ( "audio" , "bot" ); } Arguments ​ trackType 'audio' | 'video' required ​ participantType 'bot' | 'local' required ​ usePipecatClientTransportState Returns the current transport state. Copy Ask AI import { usePipecatClientTransportState } from "@pipecat-ai/client-react" ; function ConnectionStatus () { const transportState = usePipecatClientTransportState (); } ​ usePipecatClientCamControl Controls the local participant’s camera state. Copy Ask AI import { usePipecatClientCamControl } from "@pipecat-ai/client-react" ; function CamToggle () { const { enableCam , isCamEnabled } = usePipecatClientCamControl (); return ( < button onClick = { () => enableCam ( ! isCamEnabled ) } > { isCamEnabled ? "Disable Camera" : "Enable Camera" } </ button > ); } ​ usePipecatClientMicControl Controls the local participant’s microphone state. Copy Ask AI import { usePipecatClientMicControl } from "@pipecat-ai/client-react" ; function MicToggle () { const { enableMic , isMicEnabled } = usePipecatClientMicControl (); return ( < button onClick = { () => enableMic ( ! isMicEnabled ) } > { isMicEnabled ? "Disable Microphone" : "Enable Microphone" } </ button > ); } Components SDK Introduction On this page usePipecatClient useRTVIClientEvent usePipecatClientMediaDevices usePipecatClientMediaTrack usePipecatClientTransportState usePipecatClientCamControl usePipecatClientMicControl Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/react_hooks_5184905c.txt b/react_hooks_5184905c.txt
new file mode 100644
index 0000000000000000000000000000000000000000..4c062880fd92eb29b4c2737e8251333087dfb70c
--- /dev/null
+++ b/react_hooks_5184905c.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/client/react/hooks#usepipecatclientcamcontrol
+Title: Hooks - Pipecat
+==================================================
+
+Hooks - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation API Reference Hooks Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Client SDKs The RTVI Standard RTVIClient Migration Guide Javascript SDK SDK Introduction API Reference Transport packages React SDK SDK Introduction API Reference Components Hooks React Native SDK SDK Introduction API Reference iOS SDK SDK Introduction API Reference Transport packages Android SDK SDK Introduction API Reference Transport packages C++ SDK SDK Introduction Daily WebRTC Transport The Pipecat React SDK provides hooks for accessing client functionality, managing media devices, and handling events. ​ usePipecatClient Provides access to the PipecatClient instance originally passed to PipecatClientProvider . Copy Ask AI import { usePipecatClient } from "@pipecat-ai/client-react" ; function MyComponent () { const pcClient = usePipecatClient (); await pcClient . connect ({ endpoint: 'https://your-pipecat-api-url/connect' , requestData: { // Any custom data your /connect endpoint requires } }); } ​ useRTVIClientEvent Allows subscribing to RTVI client events. It is advised to wrap handlers with useCallback . Copy Ask AI import { useCallback } from "react" ; import { RTVIEvent , TransportState } from "@pipecat-ai/client-js" ; import { useRTVIClientEvent } from "@pipecat-ai/client-react" ; function EventListener () { useRTVIClientEvent ( RTVIEvent . TransportStateChanged , useCallback (( transportState : TransportState ) => { console . log ( "Transport state changed to" , transportState ); }, []) ); } Arguments ​ event RTVIEvent required ​ handler function required ​ usePipecatClientMediaDevices Manage and list available media devices. Copy Ask AI import { usePipecatClientMediaDevices } from "@pipecat-ai/client-react" ; function DeviceSelector () { const { availableCams , availableMics , selectedCam , selectedMic , updateCam , updateMic , } = usePipecatClientMediaDevices (); return ( <> < select name = "cam" onChange = { ( ev ) => updateCam ( ev . target . value ) } value = { selectedCam ?. deviceId } > { availableCams . map (( cam ) => ( < option key = { cam . deviceId } value = { cam . deviceId } > { cam . label } </ option > )) } </ select > < select name = "mic" onChange = { ( ev ) => updateMic ( ev . target . value ) } value = { selectedMic ?. deviceId } > { availableMics . map (( mic ) => ( < option key = { mic . deviceId } value = { mic . deviceId } > { mic . label } </ option > )) } </ select > </> ); } ​ usePipecatClientMediaTrack Access audio and video tracks. Copy Ask AI import { usePipecatClientMediaTrack } from "@pipecat-ai/client-react" ; function MyTracks () { const localAudioTrack = usePipecatClientMediaTrack ( "audio" , "local" ); const botAudioTrack = usePipecatClientMediaTrack ( "audio" , "bot" ); } Arguments ​ trackType 'audio' | 'video' required ​ participantType 'bot' | 'local' required ​ usePipecatClientTransportState Returns the current transport state. Copy Ask AI import { usePipecatClientTransportState } from "@pipecat-ai/client-react" ; function ConnectionStatus () { const transportState = usePipecatClientTransportState (); } ​ usePipecatClientCamControl Controls the local participant’s camera state. Copy Ask AI import { usePipecatClientCamControl } from "@pipecat-ai/client-react" ; function CamToggle () { const { enableCam , isCamEnabled } = usePipecatClientCamControl (); return ( < button onClick = { () => enableCam ( ! isCamEnabled ) } > { isCamEnabled ? "Disable Camera" : "Enable Camera" } </ button > ); } ​ usePipecatClientMicControl Controls the local participant’s microphone state. Copy Ask AI import { usePipecatClientMicControl } from "@pipecat-ai/client-react" ; function MicToggle () { const { enableMic , isMicEnabled } = usePipecatClientMicControl (); return ( < button onClick = { () => enableMic ( ! isMicEnabled ) } > { isMicEnabled ? "Disable Microphone" : "Enable Microphone" } </ button > ); } Components SDK Introduction On this page usePipecatClient useRTVIClientEvent usePipecatClientMediaDevices usePipecatClientMediaTrack usePipecatClientTransportState usePipecatClientCamControl usePipecatClientMicControl Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/react_hooks_803b2837.txt b/react_hooks_803b2837.txt
new file mode 100644
index 0000000000000000000000000000000000000000..1e9a6bc38850fd73eb469f53cd47b38230e1fc24
--- /dev/null
+++ b/react_hooks_803b2837.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/client/react/hooks#usepipecatclientmiccontrol
+Title: Hooks - Pipecat
+==================================================
+
+Hooks - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation API Reference Hooks Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Client SDKs The RTVI Standard RTVIClient Migration Guide Javascript SDK SDK Introduction API Reference Transport packages React SDK SDK Introduction API Reference Components Hooks React Native SDK SDK Introduction API Reference iOS SDK SDK Introduction API Reference Transport packages Android SDK SDK Introduction API Reference Transport packages C++ SDK SDK Introduction Daily WebRTC Transport The Pipecat React SDK provides hooks for accessing client functionality, managing media devices, and handling events. ​ usePipecatClient Provides access to the PipecatClient instance originally passed to PipecatClientProvider . Copy Ask AI import { usePipecatClient } from "@pipecat-ai/client-react" ; function MyComponent () { const pcClient = usePipecatClient (); await pcClient . connect ({ endpoint: 'https://your-pipecat-api-url/connect' , requestData: { // Any custom data your /connect endpoint requires } }); } ​ useRTVIClientEvent Allows subscribing to RTVI client events. It is advised to wrap handlers with useCallback . Copy Ask AI import { useCallback } from "react" ; import { RTVIEvent , TransportState } from "@pipecat-ai/client-js" ; import { useRTVIClientEvent } from "@pipecat-ai/client-react" ; function EventListener () { useRTVIClientEvent ( RTVIEvent . TransportStateChanged , useCallback (( transportState : TransportState ) => { console . log ( "Transport state changed to" , transportState ); }, []) ); } Arguments ​ event RTVIEvent required ​ handler function required ​ usePipecatClientMediaDevices Manage and list available media devices. Copy Ask AI import { usePipecatClientMediaDevices } from "@pipecat-ai/client-react" ; function DeviceSelector () { const { availableCams , availableMics , selectedCam , selectedMic , updateCam , updateMic , } = usePipecatClientMediaDevices (); return ( <> < select name = "cam" onChange = { ( ev ) => updateCam ( ev . target . value ) } value = { selectedCam ?. deviceId } > { availableCams . map (( cam ) => ( < option key = { cam . deviceId } value = { cam . deviceId } > { cam . label } </ option > )) } </ select > < select name = "mic" onChange = { ( ev ) => updateMic ( ev . target . value ) } value = { selectedMic ?. deviceId } > { availableMics . map (( mic ) => ( < option key = { mic . deviceId } value = { mic . deviceId } > { mic . label } </ option > )) } </ select > </> ); } ​ usePipecatClientMediaTrack Access audio and video tracks. Copy Ask AI import { usePipecatClientMediaTrack } from "@pipecat-ai/client-react" ; function MyTracks () { const localAudioTrack = usePipecatClientMediaTrack ( "audio" , "local" ); const botAudioTrack = usePipecatClientMediaTrack ( "audio" , "bot" ); } Arguments ​ trackType 'audio' | 'video' required ​ participantType 'bot' | 'local' required ​ usePipecatClientTransportState Returns the current transport state. Copy Ask AI import { usePipecatClientTransportState } from "@pipecat-ai/client-react" ; function ConnectionStatus () { const transportState = usePipecatClientTransportState (); } ​ usePipecatClientCamControl Controls the local participant’s camera state. Copy Ask AI import { usePipecatClientCamControl } from "@pipecat-ai/client-react" ; function CamToggle () { const { enableCam , isCamEnabled } = usePipecatClientCamControl (); return ( < button onClick = { () => enableCam ( ! isCamEnabled ) } > { isCamEnabled ? "Disable Camera" : "Enable Camera" } </ button > ); } ​ usePipecatClientMicControl Controls the local participant’s microphone state. Copy Ask AI import { usePipecatClientMicControl } from "@pipecat-ai/client-react" ; function MicToggle () { const { enableMic , isMicEnabled } = usePipecatClientMicControl (); return ( < button onClick = { () => enableMic ( ! isMicEnabled ) } > { isMicEnabled ? "Disable Microphone" : "Enable Microphone" } </ button > ); } Components SDK Introduction On this page usePipecatClient useRTVIClientEvent usePipecatClientMediaDevices usePipecatClientMediaTrack usePipecatClientTransportState usePipecatClientCamControl usePipecatClientMicControl Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/react_introduction_81fae952.txt b/react_introduction_81fae952.txt
new file mode 100644
index 0000000000000000000000000000000000000000..a021204819fde5ac575aa1d2b5faf623f77cd655
--- /dev/null
+++ b/react_introduction_81fae952.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/client/react/introduction#installation
+Title: SDK Introduction - Pipecat
+==================================================
+
+SDK Introduction - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation React SDK SDK Introduction Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Client SDKs The RTVI Standard RTVIClient Migration Guide Javascript SDK SDK Introduction API Reference Transport packages React SDK SDK Introduction API Reference React Native SDK SDK Introduction API Reference iOS SDK SDK Introduction API Reference Transport packages Android SDK SDK Introduction API Reference Transport packages C++ SDK SDK Introduction Daily WebRTC Transport The Pipecat React SDK provides React-specific components and hooks for building voice and multimodal AI applications. It wraps the core JavaScript SDK functionality in an idiomatic React interface that handles: React context for client state management Components for audio and video rendering Hooks for accessing client functionality Media device management Event handling through hooks ​ Installation Install the SDK, core client, and a transport implementation (e.g. Daily for WebRTC): Copy Ask AI npm install @pipecat-ai/client-js npm install @pipecat-ai/client-react npm install @pipecat-ai/daily-transport ​ Example Here’s a simple example using Daily as the transport layer: Copy Ask AI import { PipecatClient } from "@pipecat-ai/client-js" ; import { PipecatClientProvider , PipecatClientAudio , usePipecatClient , } from "@pipecat-ai/client-react" ; import { DailyTransport } from "@pipecat-ai/daily-transport" ; // Create the client instance const client = new PipecatClient ({ transport: new DailyTransport (), enableMic: true , }); // Root component wraps the app with the provider function App () { return ( < PipecatClientProvider client = { client } > < VoiceBot /> < PipecatClientAudio /> </ PipecatClientProvider > ); } // Component using the client function VoiceBot () { const client = usePipecatClient (); const handleClick = async () => { await client . connect ({ endpoint: ` ${ process . env . PIPECAT_API_URL || "/api" } /connect` }); }; return ( < button onClick = { handleClick } > Start Conversation </ button > ; ); } ​ Explore the SDK Components Ready-to-use components for audio, video, and visualization Hooks React hooks for accessing client functionality The Pipecat React SDK builds on top of the JavaScript SDK to provide an idiomatic React interface while maintaining compatibility with the RTVI standard. OpenAIRealTimeWebRTCTransport Components On this page Installation Example Explore the SDK Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/react_introduction_a9c23812.txt b/react_introduction_a9c23812.txt
new file mode 100644
index 0000000000000000000000000000000000000000..0659f44e333387d222be3481f9626b475a081c56
--- /dev/null
+++ b/react_introduction_a9c23812.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/client/react/introduction
+Title: SDK Introduction - Pipecat
+==================================================
+
+SDK Introduction - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation React SDK SDK Introduction Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Client SDKs The RTVI Standard RTVIClient Migration Guide Javascript SDK SDK Introduction API Reference Transport packages React SDK SDK Introduction API Reference React Native SDK SDK Introduction API Reference iOS SDK SDK Introduction API Reference Transport packages Android SDK SDK Introduction API Reference Transport packages C++ SDK SDK Introduction Daily WebRTC Transport The Pipecat React SDK provides React-specific components and hooks for building voice and multimodal AI applications. It wraps the core JavaScript SDK functionality in an idiomatic React interface that handles: React context for client state management Components for audio and video rendering Hooks for accessing client functionality Media device management Event handling through hooks ​ Installation Install the SDK, core client, and a transport implementation (e.g. Daily for WebRTC): Copy Ask AI npm install @pipecat-ai/client-js npm install @pipecat-ai/client-react npm install @pipecat-ai/daily-transport ​ Example Here’s a simple example using Daily as the transport layer: Copy Ask AI import { PipecatClient } from "@pipecat-ai/client-js" ; import { PipecatClientProvider , PipecatClientAudio , usePipecatClient , } from "@pipecat-ai/client-react" ; import { DailyTransport } from "@pipecat-ai/daily-transport" ; // Create the client instance const client = new PipecatClient ({ transport: new DailyTransport (), enableMic: true , }); // Root component wraps the app with the provider function App () { return ( < PipecatClientProvider client = { client } > < VoiceBot /> < PipecatClientAudio /> </ PipecatClientProvider > ); } // Component using the client function VoiceBot () { const client = usePipecatClient (); const handleClick = async () => { await client . connect ({ endpoint: ` ${ process . env . PIPECAT_API_URL || "/api" } /connect` }); }; return ( < button onClick = { handleClick } > Start Conversation </ button > ; ); } ​ Explore the SDK Components Ready-to-use components for audio, video, and visualization Hooks React hooks for accessing client functionality The Pipecat React SDK builds on top of the JavaScript SDK to provide an idiomatic React interface while maintaining compatibility with the RTVI standard. OpenAIRealTimeWebRTCTransport Components On this page Installation Example Explore the SDK Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/react_migration-guide_6446cbf3.txt b/react_migration-guide_6446cbf3.txt
new file mode 100644
index 0000000000000000000000000000000000000000..19d5529c9bf975d25dee22c1d38e8112218c6b0f
--- /dev/null
+++ b/react_migration-guide_6446cbf3.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/client/react/migration-guide#breaking-changes
+Title: RTVIClient Migration Guide for React - Pipecat
+==================================================
+
+RTVIClient Migration Guide for React - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation RTVIClient Migration Guide for React Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Client SDKs The RTVI Standard RTVIClient Migration Guide Javascript SDK SDK Introduction API Reference Transport packages React SDK SDK Introduction API Reference React Native SDK SDK Introduction API Reference iOS SDK SDK Introduction API Reference Transport packages Android SDK SDK Introduction API Reference Transport packages C++ SDK SDK Introduction Daily WebRTC Transport This guide covers migrating from RTVIClient to the new PipecatClient in a React application. The new client introduces simplified configuration and improved client-server messaging. For an overview of the changes, see the top-level RTVIClient Migration Guide . ​ Key Changes Package and Class Names Copy Ask AI // Old import { RTVIClient } from '@pipecat-ai/client-js' ; // New import { PipecatClient } from '@pipecat-ai/client-js' ; React Components and Hooks Copy Ask AI // Old import { RTVIClientProvider , RTVIClientAudio , RTVIClientVideo , useRTVIClient , useRTVIClientTransportState } from '@pipecat-ai/client-react' ; // New import { PipecatClientProvider , PipecatClientAudio , PipecatClientVideo , usePipecatClient , usePipecatClientTransportState } from '@pipecat-ai/client-react' ; Client and Transport Configuration Copy Ask AI // Old const transport = new DailyTransport (); const client = new RTVIClient ({ transport , params: { baseUrl: 'http://localhost:7860' , endpoints: { connect: '/connect' } } }); // New const client = new PipecatClient ({ transport: new DailyTransport (), // Connection params moved to connect() call }); Connection Method Copy Ask AI // Old await client . connect (); // New await client . connect ({ endpoint: 'http://localhost:7860/connect' , requestData: { // Any custom data your /connect endpoint requires llm_provider: 'openai' , initial_prompt: "You are a pirate captain" , // Any additional data } }); Function Call Handling Copy Ask AI // Old let llmHelper = new LLMHelper ({}); llmHelper . handleFunctionCall ( async ( data ) => { return await this . handleFunctionCall ( data . functionName , data . arguments ); }); client . registerHelper ( 'openai' , llmHelper ); // New client . registerFunctionCallHandler ( 'functionName' , async ( data ) => { // Handle function call return result ; }); ​ Breaking Changes Configuration Structure : Connection parameters are now passed to connect() instead of constructor Helper System : Removed in favor of direct PipecatClient member functions or client-server messaging. Component Names : All React components renamed from RTVI prefix to Pipecat prefix Hook Names : All React hooks renamed from useRTVI prefix to usePipecat prefix ​ Migration Steps Update package imports to use new names Move connection configuration from constructor to connect() method Replace any helper classes with corresponding PipecatClient methods or custom messaging Update React component and hook names Update any TypeScript types referencing old names On this page Key Changes Breaking Changes Migration Steps Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/rtvi_introduction_0d20dbf2.txt b/rtvi_introduction_0d20dbf2.txt
new file mode 100644
index 0000000000000000000000000000000000000000..f02192caa16fdf501c31acab66318411f6ae31aa
--- /dev/null
+++ b/rtvi_introduction_0d20dbf2.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/frameworks/rtvi/introduction#key-components
+Title: RTVI (Real-Time Voice Interaction) - Pipecat
+==================================================
+
+RTVI (Real-Time Voice Interaction) - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation RTVI RTVI (Real-Time Voice Interaction) Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Overview RTVIProcessor RTVI Observer Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline Pipecat’s RTVI (Real-Time Voice Interaction) protocol provides a standardized communication layer between clients and servers for building real-time voice and multimodal applications. It handles the synchronization of user and bot interactions, transcriptions, LLM processing, and text-to-speech delivery. Speaking States Track when users and bots start/stop speaking for natural turn-taking Transcription Handle real-time transcriptions from both users and bots LLM Processing Manage LLM responses and function calls with proper client notifications TTS Management Control text-to-speech state and audio delivery ​ Architecture RTVI operates with two primary components: RTVIProcessor - A frame processor residing in the pipeline that serves as the entry point for sending and receiving messages to/from the client. RTVIObserver - An observer that monitors pipeline events and translates them into client-compatible messages, handling: Speaking state changes Transcription updates LLM responses TTS events Performance metrics ​ Basic Example Here’s how to set up RTVI in your Pipecat application: Copy Ask AI from pipecat.processors.frameworks.rtvi import RTVIConfig, RTVIObserver, RTVIProcessor # Create the RTVI processor rtvi = RTVIProcessor( config = RTVIConfig( config = [])) # Include the RTVIProcessor in your pipeline pipeline = Pipeline( [ transport.input(), rtvi, stt, context_aggregator.user(), llm, tts, transport.output(), context_aggregator.assistant(), ] ) # Add the RTVIObserver to your pipeline task task = PipelineTask( pipeline, params = PipelineParams( allow_interruptions = True , enable_metrics = True , ), observers = [RTVIObserver(rtvi)], ) # Handle client connection @rtvi.event_handler ( "on_client_ready" ) async def on_client_ready ( rtvi ): # Signal bot is ready to receive messages await rtvi.set_bot_ready() # Initialize the conversation await task.queue_frames([context_aggregator.user().get_context_frame()]) # Handle participant disconnection @transport.event_handler ( "on_participant_left" ) async def on_participant_left ( transport , participant , reason ): await task.cancel() # Run the pipeline runner = PipelineRunner() await runner.run(task) ​ Protocol Flow Client connects and sends a client-ready message Server responds with bot-ready and initial configuration Client and server exchange real-time events: Speaking state changes ( user/bot-started/stopped-speaking ) Transcriptions ( user/bot-transcription ) LLM processing ( bot-llm-text , llm-function-call ) TTS events ( bot-tts-text , bot-tts-audio ) ​ Key Components RTVIProcessor Configure and manage RTVI services, actions, and client communication RTVIObserver Translate internal pipeline events to standardized client messages ​ Client Integration RTVI is implemented in Pipecat client SDKs, providing a high-level API to interact with the protocol. Visit the Pipecat Client SDKs documentation: Client SDKs Learn how to implement RTVI on the client-side with our JavaScript, React, and mobile SDKs Interruption Strategies RTVIProcessor On this page Architecture Basic Example Protocol Flow Key Components Client Integration Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/rtvi_introduction_2038f975.txt b/rtvi_introduction_2038f975.txt
new file mode 100644
index 0000000000000000000000000000000000000000..45c406e6ec7a795954534609235c8f40a19f5389
--- /dev/null
+++ b/rtvi_introduction_2038f975.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/frameworks/rtvi/introduction
+Title: RTVI (Real-Time Voice Interaction) - Pipecat
+==================================================
+
+RTVI (Real-Time Voice Interaction) - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation RTVI RTVI (Real-Time Voice Interaction) Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Overview RTVIProcessor RTVI Observer Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline Pipecat’s RTVI (Real-Time Voice Interaction) protocol provides a standardized communication layer between clients and servers for building real-time voice and multimodal applications. It handles the synchronization of user and bot interactions, transcriptions, LLM processing, and text-to-speech delivery. Speaking States Track when users and bots start/stop speaking for natural turn-taking Transcription Handle real-time transcriptions from both users and bots LLM Processing Manage LLM responses and function calls with proper client notifications TTS Management Control text-to-speech state and audio delivery ​ Architecture RTVI operates with two primary components: RTVIProcessor - A frame processor residing in the pipeline that serves as the entry point for sending and receiving messages to/from the client. RTVIObserver - An observer that monitors pipeline events and translates them into client-compatible messages, handling: Speaking state changes Transcription updates LLM responses TTS events Performance metrics ​ Basic Example Here’s how to set up RTVI in your Pipecat application: Copy Ask AI from pipecat.processors.frameworks.rtvi import RTVIConfig, RTVIObserver, RTVIProcessor # Create the RTVI processor rtvi = RTVIProcessor( config = RTVIConfig( config = [])) # Include the RTVIProcessor in your pipeline pipeline = Pipeline( [ transport.input(), rtvi, stt, context_aggregator.user(), llm, tts, transport.output(), context_aggregator.assistant(), ] ) # Add the RTVIObserver to your pipeline task task = PipelineTask( pipeline, params = PipelineParams( allow_interruptions = True , enable_metrics = True , ), observers = [RTVIObserver(rtvi)], ) # Handle client connection @rtvi.event_handler ( "on_client_ready" ) async def on_client_ready ( rtvi ): # Signal bot is ready to receive messages await rtvi.set_bot_ready() # Initialize the conversation await task.queue_frames([context_aggregator.user().get_context_frame()]) # Handle participant disconnection @transport.event_handler ( "on_participant_left" ) async def on_participant_left ( transport , participant , reason ): await task.cancel() # Run the pipeline runner = PipelineRunner() await runner.run(task) ​ Protocol Flow Client connects and sends a client-ready message Server responds with bot-ready and initial configuration Client and server exchange real-time events: Speaking state changes ( user/bot-started/stopped-speaking ) Transcriptions ( user/bot-transcription ) LLM processing ( bot-llm-text , llm-function-call ) TTS events ( bot-tts-text , bot-tts-audio ) ​ Key Components RTVIProcessor Configure and manage RTVI services, actions, and client communication RTVIObserver Translate internal pipeline events to standardized client messages ​ Client Integration RTVI is implemented in Pipecat client SDKs, providing a high-level API to interact with the protocol. Visit the Pipecat Client SDKs documentation: Client SDKs Learn how to implement RTVI on the client-side with our JavaScript, React, and mobile SDKs Interruption Strategies RTVIProcessor On this page Architecture Basic Example Protocol Flow Key Components Client Integration Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/rtvi_introduction_8d77072d.txt b/rtvi_introduction_8d77072d.txt
new file mode 100644
index 0000000000000000000000000000000000000000..89d717715b7f6150554f6b16189e023f35b43012
--- /dev/null
+++ b/rtvi_introduction_8d77072d.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/frameworks/rtvi/introduction#client-integration
+Title: RTVI (Real-Time Voice Interaction) - Pipecat
+==================================================
+
+RTVI (Real-Time Voice Interaction) - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation RTVI RTVI (Real-Time Voice Interaction) Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Overview RTVIProcessor RTVI Observer Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline Pipecat’s RTVI (Real-Time Voice Interaction) protocol provides a standardized communication layer between clients and servers for building real-time voice and multimodal applications. It handles the synchronization of user and bot interactions, transcriptions, LLM processing, and text-to-speech delivery. Speaking States Track when users and bots start/stop speaking for natural turn-taking Transcription Handle real-time transcriptions from both users and bots LLM Processing Manage LLM responses and function calls with proper client notifications TTS Management Control text-to-speech state and audio delivery ​ Architecture RTVI operates with two primary components: RTVIProcessor - A frame processor residing in the pipeline that serves as the entry point for sending and receiving messages to/from the client. RTVIObserver - An observer that monitors pipeline events and translates them into client-compatible messages, handling: Speaking state changes Transcription updates LLM responses TTS events Performance metrics ​ Basic Example Here’s how to set up RTVI in your Pipecat application: Copy Ask AI from pipecat.processors.frameworks.rtvi import RTVIConfig, RTVIObserver, RTVIProcessor # Create the RTVI processor rtvi = RTVIProcessor( config = RTVIConfig( config = [])) # Include the RTVIProcessor in your pipeline pipeline = Pipeline( [ transport.input(), rtvi, stt, context_aggregator.user(), llm, tts, transport.output(), context_aggregator.assistant(), ] ) # Add the RTVIObserver to your pipeline task task = PipelineTask( pipeline, params = PipelineParams( allow_interruptions = True , enable_metrics = True , ), observers = [RTVIObserver(rtvi)], ) # Handle client connection @rtvi.event_handler ( "on_client_ready" ) async def on_client_ready ( rtvi ): # Signal bot is ready to receive messages await rtvi.set_bot_ready() # Initialize the conversation await task.queue_frames([context_aggregator.user().get_context_frame()]) # Handle participant disconnection @transport.event_handler ( "on_participant_left" ) async def on_participant_left ( transport , participant , reason ): await task.cancel() # Run the pipeline runner = PipelineRunner() await runner.run(task) ​ Protocol Flow Client connects and sends a client-ready message Server responds with bot-ready and initial configuration Client and server exchange real-time events: Speaking state changes ( user/bot-started/stopped-speaking ) Transcriptions ( user/bot-transcription ) LLM processing ( bot-llm-text , llm-function-call ) TTS events ( bot-tts-text , bot-tts-audio ) ​ Key Components RTVIProcessor Configure and manage RTVI services, actions, and client communication RTVIObserver Translate internal pipeline events to standardized client messages ​ Client Integration RTVI is implemented in Pipecat client SDKs, providing a high-level API to interact with the protocol. Visit the Pipecat Client SDKs documentation: Client SDKs Learn how to implement RTVI on the client-side with our JavaScript, React, and mobile SDKs Interruption Strategies RTVIProcessor On this page Architecture Basic Example Protocol Flow Key Components Client Integration Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/rtvi_rtvi-processor_4f5d38e0.txt b/rtvi_rtvi-processor_4f5d38e0.txt
new file mode 100644
index 0000000000000000000000000000000000000000..15b53754452d06f7e9f651db873cf426d9e821e9
--- /dev/null
+++ b/rtvi_rtvi-processor_4f5d38e0.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/frameworks/rtvi/rtvi-processor#actions
+Title: RTVIProcessor - Pipecat
+==================================================
+
+RTVIProcessor - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation RTVI RTVIProcessor Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Overview RTVIProcessor RTVI Observer Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline The RTVIProcessor manages bidirectional communication between clients and your Pipecat application. It processes client messages, handles service configuration, executes actions, and coordinates function calls. ​ Initialization Add the RTVIProcessor to your pipeline: Copy Ask AI from pipecat.processors.rtvi import RTVIProcessor, RTVIConfig, RTVIServiceConfig # Create the RTVIProcessor rtvi = RTVIProcessor( config = RTVIConfig( config = [])) # Add to pipeline pipeline = Pipeline([ transport.input(), rtvi, stt, # ... other processors ... transport.output() ]) ​ Readiness Protocol ​ Client Ready State Clients indicate readiness by sending a client-ready message, triggering the on_client_ready event in the processor: Copy Ask AI @rtvi.event_handler ( "on_client_ready" ) async def on_client_ready ( rtvi ): # Handle client ready state await rtvi.set_bot_ready() # Initialize conversation await task.queue_frames([ ... ]) ​ Bot Ready State The server must mark the bot as ready before it can process client messages: Copy Ask AI await rtvi.set_bot_ready() When marked ready, the bot sends a response containing: RTVI protocol version Current service configuration Available actions ​ Services Services represent configurable components of your application that clients can interact with. ​ Registering Services Copy Ask AI # 1. Define option handler async def handle_voice_option ( processor , service , option ): voice_id = option.value # Apply configuration change logger.info( f "Voice ID updated to: { voice_id } " ) # 2. Create RTVIService voice_service = RTVIService( name = "voice" , options = [ RTVIServiceOption( name = "voice_id" , type = "string" , handler = handle_voice_option ) ] ) # 3. Register with processor rtvi.register_service(voice_service) ​ Option Types Services support multiple data types for configuration: Copy Ask AI RTVIServiceOption( name = "temperature" , type = "number" , # number, string, bool, array, object handler = handle_temperature ) Option handlers receive: The processor instance The service name The option configuration with new value ​ Actions Actions are server-side functions that clients can trigger with arguments. ​ Registering Actions Copy Ask AI # 1. Define handler function async def handle_print_message ( processor , service , arguments ): message = arguments.get( "message" , "Default message" ) logger.info( f "Print action triggered with message: { message } " ) return True # 2. Create and register RTVIAction print_action = RTVIAction( service = "conversation" , action = "print_message" , arguments = [ RTVIActionArgument( name = "message" , type = "string" ) ], result = "bool" , handler = handle_print_message ) rtvi.register_action(print_action) ​ Action Arguments Actions can accept typed arguments from clients: Copy Ask AI search_action = RTVIAction( service = "knowledge" , action = "search" , arguments = [ RTVIActionArgument( name = "query" , type = "string" ), RTVIActionArgument( name = "limit" , type = "number" ) ], result = "array" , handler = handle_search ) ​ Function Calls Handle LLM function calls with client interaction: Copy Ask AI await processor.handle_function_call( function_name = function_name, tool_call_id = tool_call_id, arguments = arguments, ) await processor.handle_function_call(params) The function call process: LLM requests a function call Processor notifies client with llm-function-call message Client executes function and returns result Result is passed back to LLM via FunctionCallResultFrame Conversation continues ​ Error Handling Send error messages to clients: Copy Ask AI # General error await processor.send_error( "Invalid configuration" ) # Request-specific error await processor._send_error_response(request_id, "Invalid action arguments" ) Error categories: Configuration errors Action execution errors Function call errors Protocol errors Fatal and non-fatal errors ​ Bot Control Manage bot state and handle interruptions: Copy Ask AI # Set bot as ready await processor.set_bot_ready() # Handle interruptions await processor.interrupt_bot() ​ Custom Messaging Send custom messages from server to client: Copy Ask AI from pipecat.processors.frameworks.rtvi import RTVIServerMessageFrame # Send a custom message frame = RTVIServerMessageFrame( data = { "type" : "custom-event" , "payload" : { "key" : "value" } } ) await rtvi.push_frame(frame) The client receives these messages through the onServerMessage callback or serverMessage event. Overview RTVI Observer On this page Initialization Readiness Protocol Client Ready State Bot Ready State Services Registering Services Option Types Actions Registering Actions Action Arguments Function Calls Error Handling Bot Control Custom Messaging Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/rtvi_rtvi-processor_6872c7e1.txt b/rtvi_rtvi-processor_6872c7e1.txt
new file mode 100644
index 0000000000000000000000000000000000000000..76bc911d3c72104a3f010ef99eb9a315e707a76f
--- /dev/null
+++ b/rtvi_rtvi-processor_6872c7e1.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/frameworks/rtvi/rtvi-processor#bot-ready-state
+Title: RTVIProcessor - Pipecat
+==================================================
+
+RTVIProcessor - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation RTVI RTVIProcessor Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Overview RTVIProcessor RTVI Observer Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline The RTVIProcessor manages bidirectional communication between clients and your Pipecat application. It processes client messages, handles service configuration, executes actions, and coordinates function calls. ​ Initialization Add the RTVIProcessor to your pipeline: Copy Ask AI from pipecat.processors.rtvi import RTVIProcessor, RTVIConfig, RTVIServiceConfig # Create the RTVIProcessor rtvi = RTVIProcessor( config = RTVIConfig( config = [])) # Add to pipeline pipeline = Pipeline([ transport.input(), rtvi, stt, # ... other processors ... transport.output() ]) ​ Readiness Protocol ​ Client Ready State Clients indicate readiness by sending a client-ready message, triggering the on_client_ready event in the processor: Copy Ask AI @rtvi.event_handler ( "on_client_ready" ) async def on_client_ready ( rtvi ): # Handle client ready state await rtvi.set_bot_ready() # Initialize conversation await task.queue_frames([ ... ]) ​ Bot Ready State The server must mark the bot as ready before it can process client messages: Copy Ask AI await rtvi.set_bot_ready() When marked ready, the bot sends a response containing: RTVI protocol version Current service configuration Available actions ​ Services Services represent configurable components of your application that clients can interact with. ​ Registering Services Copy Ask AI # 1. Define option handler async def handle_voice_option ( processor , service , option ): voice_id = option.value # Apply configuration change logger.info( f "Voice ID updated to: { voice_id } " ) # 2. Create RTVIService voice_service = RTVIService( name = "voice" , options = [ RTVIServiceOption( name = "voice_id" , type = "string" , handler = handle_voice_option ) ] ) # 3. Register with processor rtvi.register_service(voice_service) ​ Option Types Services support multiple data types for configuration: Copy Ask AI RTVIServiceOption( name = "temperature" , type = "number" , # number, string, bool, array, object handler = handle_temperature ) Option handlers receive: The processor instance The service name The option configuration with new value ​ Actions Actions are server-side functions that clients can trigger with arguments. ​ Registering Actions Copy Ask AI # 1. Define handler function async def handle_print_message ( processor , service , arguments ): message = arguments.get( "message" , "Default message" ) logger.info( f "Print action triggered with message: { message } " ) return True # 2. Create and register RTVIAction print_action = RTVIAction( service = "conversation" , action = "print_message" , arguments = [ RTVIActionArgument( name = "message" , type = "string" ) ], result = "bool" , handler = handle_print_message ) rtvi.register_action(print_action) ​ Action Arguments Actions can accept typed arguments from clients: Copy Ask AI search_action = RTVIAction( service = "knowledge" , action = "search" , arguments = [ RTVIActionArgument( name = "query" , type = "string" ), RTVIActionArgument( name = "limit" , type = "number" ) ], result = "array" , handler = handle_search ) ​ Function Calls Handle LLM function calls with client interaction: Copy Ask AI await processor.handle_function_call( function_name = function_name, tool_call_id = tool_call_id, arguments = arguments, ) await processor.handle_function_call(params) The function call process: LLM requests a function call Processor notifies client with llm-function-call message Client executes function and returns result Result is passed back to LLM via FunctionCallResultFrame Conversation continues ​ Error Handling Send error messages to clients: Copy Ask AI # General error await processor.send_error( "Invalid configuration" ) # Request-specific error await processor._send_error_response(request_id, "Invalid action arguments" ) Error categories: Configuration errors Action execution errors Function call errors Protocol errors Fatal and non-fatal errors ​ Bot Control Manage bot state and handle interruptions: Copy Ask AI # Set bot as ready await processor.set_bot_ready() # Handle interruptions await processor.interrupt_bot() ​ Custom Messaging Send custom messages from server to client: Copy Ask AI from pipecat.processors.frameworks.rtvi import RTVIServerMessageFrame # Send a custom message frame = RTVIServerMessageFrame( data = { "type" : "custom-event" , "payload" : { "key" : "value" } } ) await rtvi.push_frame(frame) The client receives these messages through the onServerMessage callback or serverMessage event. Overview RTVI Observer On this page Initialization Readiness Protocol Client Ready State Bot Ready State Services Registering Services Option Types Actions Registering Actions Action Arguments Function Calls Error Handling Bot Control Custom Messaging Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/rtvi_rtvi-processor_9c815016.txt b/rtvi_rtvi-processor_9c815016.txt
new file mode 100644
index 0000000000000000000000000000000000000000..ab8f1cf5258a69089abc4fb569bb923f821fdb5c
--- /dev/null
+++ b/rtvi_rtvi-processor_9c815016.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/frameworks/rtvi/rtvi-processor#registering-services
+Title: RTVIProcessor - Pipecat
+==================================================
+
+RTVIProcessor - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation RTVI RTVIProcessor Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Overview RTVIProcessor RTVI Observer Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline The RTVIProcessor manages bidirectional communication between clients and your Pipecat application. It processes client messages, handles service configuration, executes actions, and coordinates function calls. ​ Initialization Add the RTVIProcessor to your pipeline: Copy Ask AI from pipecat.processors.rtvi import RTVIProcessor, RTVIConfig, RTVIServiceConfig # Create the RTVIProcessor rtvi = RTVIProcessor( config = RTVIConfig( config = [])) # Add to pipeline pipeline = Pipeline([ transport.input(), rtvi, stt, # ... other processors ... transport.output() ]) ​ Readiness Protocol ​ Client Ready State Clients indicate readiness by sending a client-ready message, triggering the on_client_ready event in the processor: Copy Ask AI @rtvi.event_handler ( "on_client_ready" ) async def on_client_ready ( rtvi ): # Handle client ready state await rtvi.set_bot_ready() # Initialize conversation await task.queue_frames([ ... ]) ​ Bot Ready State The server must mark the bot as ready before it can process client messages: Copy Ask AI await rtvi.set_bot_ready() When marked ready, the bot sends a response containing: RTVI protocol version Current service configuration Available actions ​ Services Services represent configurable components of your application that clients can interact with. ​ Registering Services Copy Ask AI # 1. Define option handler async def handle_voice_option ( processor , service , option ): voice_id = option.value # Apply configuration change logger.info( f "Voice ID updated to: { voice_id } " ) # 2. Create RTVIService voice_service = RTVIService( name = "voice" , options = [ RTVIServiceOption( name = "voice_id" , type = "string" , handler = handle_voice_option ) ] ) # 3. Register with processor rtvi.register_service(voice_service) ​ Option Types Services support multiple data types for configuration: Copy Ask AI RTVIServiceOption( name = "temperature" , type = "number" , # number, string, bool, array, object handler = handle_temperature ) Option handlers receive: The processor instance The service name The option configuration with new value ​ Actions Actions are server-side functions that clients can trigger with arguments. ​ Registering Actions Copy Ask AI # 1. Define handler function async def handle_print_message ( processor , service , arguments ): message = arguments.get( "message" , "Default message" ) logger.info( f "Print action triggered with message: { message } " ) return True # 2. Create and register RTVIAction print_action = RTVIAction( service = "conversation" , action = "print_message" , arguments = [ RTVIActionArgument( name = "message" , type = "string" ) ], result = "bool" , handler = handle_print_message ) rtvi.register_action(print_action) ​ Action Arguments Actions can accept typed arguments from clients: Copy Ask AI search_action = RTVIAction( service = "knowledge" , action = "search" , arguments = [ RTVIActionArgument( name = "query" , type = "string" ), RTVIActionArgument( name = "limit" , type = "number" ) ], result = "array" , handler = handle_search ) ​ Function Calls Handle LLM function calls with client interaction: Copy Ask AI await processor.handle_function_call( function_name = function_name, tool_call_id = tool_call_id, arguments = arguments, ) await processor.handle_function_call(params) The function call process: LLM requests a function call Processor notifies client with llm-function-call message Client executes function and returns result Result is passed back to LLM via FunctionCallResultFrame Conversation continues ​ Error Handling Send error messages to clients: Copy Ask AI # General error await processor.send_error( "Invalid configuration" ) # Request-specific error await processor._send_error_response(request_id, "Invalid action arguments" ) Error categories: Configuration errors Action execution errors Function call errors Protocol errors Fatal and non-fatal errors ​ Bot Control Manage bot state and handle interruptions: Copy Ask AI # Set bot as ready await processor.set_bot_ready() # Handle interruptions await processor.interrupt_bot() ​ Custom Messaging Send custom messages from server to client: Copy Ask AI from pipecat.processors.frameworks.rtvi import RTVIServerMessageFrame # Send a custom message frame = RTVIServerMessageFrame( data = { "type" : "custom-event" , "payload" : { "key" : "value" } } ) await rtvi.push_frame(frame) The client receives these messages through the onServerMessage callback or serverMessage event. Overview RTVI Observer On this page Initialization Readiness Protocol Client Ready State Bot Ready State Services Registering Services Option Types Actions Registering Actions Action Arguments Function Calls Error Handling Bot Control Custom Messaging Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/rtvi_rtvi-processor_beb3de32.txt b/rtvi_rtvi-processor_beb3de32.txt
new file mode 100644
index 0000000000000000000000000000000000000000..a78e244cece08f5b0ec4502230d661b7bdb40952
--- /dev/null
+++ b/rtvi_rtvi-processor_beb3de32.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/frameworks/rtvi/rtvi-processor#action-arguments
+Title: RTVIProcessor - Pipecat
+==================================================
+
+RTVIProcessor - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation RTVI RTVIProcessor Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Overview RTVIProcessor RTVI Observer Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline The RTVIProcessor manages bidirectional communication between clients and your Pipecat application. It processes client messages, handles service configuration, executes actions, and coordinates function calls. ​ Initialization Add the RTVIProcessor to your pipeline: Copy Ask AI from pipecat.processors.rtvi import RTVIProcessor, RTVIConfig, RTVIServiceConfig # Create the RTVIProcessor rtvi = RTVIProcessor( config = RTVIConfig( config = [])) # Add to pipeline pipeline = Pipeline([ transport.input(), rtvi, stt, # ... other processors ... transport.output() ]) ​ Readiness Protocol ​ Client Ready State Clients indicate readiness by sending a client-ready message, triggering the on_client_ready event in the processor: Copy Ask AI @rtvi.event_handler ( "on_client_ready" ) async def on_client_ready ( rtvi ): # Handle client ready state await rtvi.set_bot_ready() # Initialize conversation await task.queue_frames([ ... ]) ​ Bot Ready State The server must mark the bot as ready before it can process client messages: Copy Ask AI await rtvi.set_bot_ready() When marked ready, the bot sends a response containing: RTVI protocol version Current service configuration Available actions ​ Services Services represent configurable components of your application that clients can interact with. ​ Registering Services Copy Ask AI # 1. Define option handler async def handle_voice_option ( processor , service , option ): voice_id = option.value # Apply configuration change logger.info( f "Voice ID updated to: { voice_id } " ) # 2. Create RTVIService voice_service = RTVIService( name = "voice" , options = [ RTVIServiceOption( name = "voice_id" , type = "string" , handler = handle_voice_option ) ] ) # 3. Register with processor rtvi.register_service(voice_service) ​ Option Types Services support multiple data types for configuration: Copy Ask AI RTVIServiceOption( name = "temperature" , type = "number" , # number, string, bool, array, object handler = handle_temperature ) Option handlers receive: The processor instance The service name The option configuration with new value ​ Actions Actions are server-side functions that clients can trigger with arguments. ​ Registering Actions Copy Ask AI # 1. Define handler function async def handle_print_message ( processor , service , arguments ): message = arguments.get( "message" , "Default message" ) logger.info( f "Print action triggered with message: { message } " ) return True # 2. Create and register RTVIAction print_action = RTVIAction( service = "conversation" , action = "print_message" , arguments = [ RTVIActionArgument( name = "message" , type = "string" ) ], result = "bool" , handler = handle_print_message ) rtvi.register_action(print_action) ​ Action Arguments Actions can accept typed arguments from clients: Copy Ask AI search_action = RTVIAction( service = "knowledge" , action = "search" , arguments = [ RTVIActionArgument( name = "query" , type = "string" ), RTVIActionArgument( name = "limit" , type = "number" ) ], result = "array" , handler = handle_search ) ​ Function Calls Handle LLM function calls with client interaction: Copy Ask AI await processor.handle_function_call( function_name = function_name, tool_call_id = tool_call_id, arguments = arguments, ) await processor.handle_function_call(params) The function call process: LLM requests a function call Processor notifies client with llm-function-call message Client executes function and returns result Result is passed back to LLM via FunctionCallResultFrame Conversation continues ​ Error Handling Send error messages to clients: Copy Ask AI # General error await processor.send_error( "Invalid configuration" ) # Request-specific error await processor._send_error_response(request_id, "Invalid action arguments" ) Error categories: Configuration errors Action execution errors Function call errors Protocol errors Fatal and non-fatal errors ​ Bot Control Manage bot state and handle interruptions: Copy Ask AI # Set bot as ready await processor.set_bot_ready() # Handle interruptions await processor.interrupt_bot() ​ Custom Messaging Send custom messages from server to client: Copy Ask AI from pipecat.processors.frameworks.rtvi import RTVIServerMessageFrame # Send a custom message frame = RTVIServerMessageFrame( data = { "type" : "custom-event" , "payload" : { "key" : "value" } } ) await rtvi.push_frame(frame) The client receives these messages through the onServerMessage callback or serverMessage event. Overview RTVI Observer On this page Initialization Readiness Protocol Client Ready State Bot Ready State Services Registering Services Option Types Actions Registering Actions Action Arguments Function Calls Error Handling Bot Control Custom Messaging Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/rtvi_rtvi-processor_c4f27814.txt b/rtvi_rtvi-processor_c4f27814.txt
new file mode 100644
index 0000000000000000000000000000000000000000..5f4cf726118aa1aa1f06a1c9fdc38f459c0df7d0
--- /dev/null
+++ b/rtvi_rtvi-processor_c4f27814.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/frameworks/rtvi/rtvi-processor#custom-messaging
+Title: RTVIProcessor - Pipecat
+==================================================
+
+RTVIProcessor - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation RTVI RTVIProcessor Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Overview RTVIProcessor RTVI Observer Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline The RTVIProcessor manages bidirectional communication between clients and your Pipecat application. It processes client messages, handles service configuration, executes actions, and coordinates function calls. ​ Initialization Add the RTVIProcessor to your pipeline: Copy Ask AI from pipecat.processors.rtvi import RTVIProcessor, RTVIConfig, RTVIServiceConfig # Create the RTVIProcessor rtvi = RTVIProcessor( config = RTVIConfig( config = [])) # Add to pipeline pipeline = Pipeline([ transport.input(), rtvi, stt, # ... other processors ... transport.output() ]) ​ Readiness Protocol ​ Client Ready State Clients indicate readiness by sending a client-ready message, triggering the on_client_ready event in the processor: Copy Ask AI @rtvi.event_handler ( "on_client_ready" ) async def on_client_ready ( rtvi ): # Handle client ready state await rtvi.set_bot_ready() # Initialize conversation await task.queue_frames([ ... ]) ​ Bot Ready State The server must mark the bot as ready before it can process client messages: Copy Ask AI await rtvi.set_bot_ready() When marked ready, the bot sends a response containing: RTVI protocol version Current service configuration Available actions ​ Services Services represent configurable components of your application that clients can interact with. ​ Registering Services Copy Ask AI # 1. Define option handler async def handle_voice_option ( processor , service , option ): voice_id = option.value # Apply configuration change logger.info( f "Voice ID updated to: { voice_id } " ) # 2. Create RTVIService voice_service = RTVIService( name = "voice" , options = [ RTVIServiceOption( name = "voice_id" , type = "string" , handler = handle_voice_option ) ] ) # 3. Register with processor rtvi.register_service(voice_service) ​ Option Types Services support multiple data types for configuration: Copy Ask AI RTVIServiceOption( name = "temperature" , type = "number" , # number, string, bool, array, object handler = handle_temperature ) Option handlers receive: The processor instance The service name The option configuration with new value ​ Actions Actions are server-side functions that clients can trigger with arguments. ​ Registering Actions Copy Ask AI # 1. Define handler function async def handle_print_message ( processor , service , arguments ): message = arguments.get( "message" , "Default message" ) logger.info( f "Print action triggered with message: { message } " ) return True # 2. Create and register RTVIAction print_action = RTVIAction( service = "conversation" , action = "print_message" , arguments = [ RTVIActionArgument( name = "message" , type = "string" ) ], result = "bool" , handler = handle_print_message ) rtvi.register_action(print_action) ​ Action Arguments Actions can accept typed arguments from clients: Copy Ask AI search_action = RTVIAction( service = "knowledge" , action = "search" , arguments = [ RTVIActionArgument( name = "query" , type = "string" ), RTVIActionArgument( name = "limit" , type = "number" ) ], result = "array" , handler = handle_search ) ​ Function Calls Handle LLM function calls with client interaction: Copy Ask AI await processor.handle_function_call( function_name = function_name, tool_call_id = tool_call_id, arguments = arguments, ) await processor.handle_function_call(params) The function call process: LLM requests a function call Processor notifies client with llm-function-call message Client executes function and returns result Result is passed back to LLM via FunctionCallResultFrame Conversation continues ​ Error Handling Send error messages to clients: Copy Ask AI # General error await processor.send_error( "Invalid configuration" ) # Request-specific error await processor._send_error_response(request_id, "Invalid action arguments" ) Error categories: Configuration errors Action execution errors Function call errors Protocol errors Fatal and non-fatal errors ​ Bot Control Manage bot state and handle interruptions: Copy Ask AI # Set bot as ready await processor.set_bot_ready() # Handle interruptions await processor.interrupt_bot() ​ Custom Messaging Send custom messages from server to client: Copy Ask AI from pipecat.processors.frameworks.rtvi import RTVIServerMessageFrame # Send a custom message frame = RTVIServerMessageFrame( data = { "type" : "custom-event" , "payload" : { "key" : "value" } } ) await rtvi.push_frame(frame) The client receives these messages through the onServerMessage callback or serverMessage event. Overview RTVI Observer On this page Initialization Readiness Protocol Client Ready State Bot Ready State Services Registering Services Option Types Actions Registering Actions Action Arguments Function Calls Error Handling Bot Control Custom Messaging Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/s2s_aws_0df3e38b.txt b/s2s_aws_0df3e38b.txt
new file mode 100644
index 0000000000000000000000000000000000000000..90f331748f537d35348f71adc15d34e3a723a5fe
--- /dev/null
+++ b/s2s_aws_0df3e38b.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/s2s/aws#param-open-aillm-context-frame
+Title: AWS Nova Sonic - Pipecat
+==================================================
+
+AWS Nova Sonic - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Speech AWS Nova Sonic Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech AWS Nova Sonic Gemini Multimodal Live OpenAI Realtime Beta Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline The AWSNovaSonicLLMService enables natural, real-time conversations with AWS Nova Sonic. It provides built-in audio transcription, voice activity detection, and context management for creating interactive AI experiences. It provides: Real-time Interaction Stream audio in real-time with low latency response times Speech Processing Built-in speech-to-text and text-to-speech capabilities with multiple voice options Voice Activity Detection Automatic detection of speech start/stop for natural conversations Context Management Intelligent handling of conversation history and system instructions ​ Installation To use AWSNovaSonicLLMService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[aws-nova-sonic]" We recommend setting up your AWS credentials as environment variables, as you’ll need them to initialize AWSNovaSonicLLMService : AWS_SECRET_ACCESS_KEY AWS_ACCESS_KEY_ID AWS_REGION ​ Basic Usage Here’s a simple example of setting up a conversational AI bot with AWS Nova Sonic: Copy Ask AI from pipecat.services.aws_nova_sonic.aws import AWSNovaSonicLLMService llm = AWSNovaSonicLLMService( secret_access_key = os.getenv( "AWS_SECRET_ACCESS_KEY" ), access_key_id = os.getenv( "AWS_ACCESS_KEY_ID" ), region = os.getenv( "AWS_REGION" ) voice_id = "tiffany" , # Voices: matthew, tiffany, amy ) ​ Configuration ​ Constructor Parameters ​ secret_access_key str required Your AWS secret access key ​ access_key_id str required Your AWS access key ID ​ region str required Specify the AWS region for the service (e.g., "us-east-1" ). Note that the service may not be available in all AWS regions: check the AWS Bedrock User Guide’s support table . ​ model str default: "amazon.nova-sonic-v1:0" AWS Nova Sonic model to use. Note that "amazon.nova-sonic-v1:0" is the only supported model as of 2025-05-08. ​ voice_id str default: "matthew" Voice for text-to-speech (options: "matthew" , "tiffany" , "amy" ) ​ params Params Configuration for model parameters ​ system_instruction str High-level instructions that guide the model’s behavior. Note that more commonly these instructions will be included as part of the context provided to kick off the conversation. ​ tools ToolsSchema List of function definitions for tool/function calling. Note that more commonly tools will be included as part of the context provided to kick off the conversation. ​ send_transcription_frames bool default: "True" Whether to emit transcription frames ​ Model Parameters The Params object configures the behavior of the AWS Nova Sonic model. It is strongly recommended to stick with default values (most easily by omitting params when constructing AWSNovaSonicLLMService ) unless you have a good understanding of the parameters and their impact. Deviating from the defaults may lead to unexpected behavior. ​ temperature float default: "0.7" Controls randomness in responses. Range: 0.0 to 2.0 ​ max_tokens int default: "1024" Maximum number of tokens to generate ​ top_p float default: "0.9" Cumulative probability cutoff for token selection. Range: 0.0 to 1.0 ​ input_sample_rate int default: "16000" Sample rate for input audio ​ output_sample_rate int default: "24000" Sample rate for output audio ​ input_sample_size int default: "16" Bit depth for input audio ​ input_channel_count int default: "1" Number of channels for input audio ​ output_sample_size int default: "16" Bit depth for output audio ​ output_channel_count int default: "1" Number of channels for output audio ​ Frame Types ​ Input Frames ​ InputAudioRawFrame Frame Raw audio data for speech input ​ OpenAILLMContextFrame Frame Contains conversation context ​ BotStoppedSpeakingFrame Frame Signals the bot has stopped speaking ​ Output Frames ​ TTSAudioRawFrame Frame Generated speech audio ​ LLMFullResponseStartFrame Frame Signals the start of a response from the LLM ​ LLMFullResponseEndFrame Frame Signals the end of a response from the LLM ​ TTSStartedFrame Frame Signals start of speech synthesis (coincides with the start of the LLM response, as this is a speech-to-speech model) ​ TTSStoppedFrame Frame Signals end of speech synthesis (coincides with the end of the LLM response, as this is a speech-to-speech model) ​ LLMTextFrame Frame Generated text responses from the LLM ​ TTSTextFrame Frame Generated text responses ​ TranscriptionFrame Frame Speech transcriptions. Only output if send_transcription_frames is True . ​ Function Calling This service supports function calling (also known as tool calling) which allows the LLM to request information from external services and APIs. For example, you can enable your bot to: Check current weather conditions Query databases Access external APIs Perform custom actions See the Function Calling guide for: Detailed implementation instructions Provider-specific function definitions Handler registration examples Control over function call behavior Complete usage examples ​ Next Steps ​ Examples Foundational Example Basic implementation showing core features Persistent Content Example Implementation showing saving and loading conversation history XTTS Gemini Multimodal Live On this page Installation Basic Usage Configuration Constructor Parameters Model Parameters Frame Types Input Frames Output Frames Function Calling Next Steps Examples Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/s2s_aws_7132889e.txt b/s2s_aws_7132889e.txt
new file mode 100644
index 0000000000000000000000000000000000000000..85c2122274fbd602daf84ef68708b3db0df470a8
--- /dev/null
+++ b/s2s_aws_7132889e.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/s2s/aws#input-frames
+Title: AWS Nova Sonic - Pipecat
+==================================================
+
+AWS Nova Sonic - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Speech AWS Nova Sonic Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech AWS Nova Sonic Gemini Multimodal Live OpenAI Realtime Beta Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline The AWSNovaSonicLLMService enables natural, real-time conversations with AWS Nova Sonic. It provides built-in audio transcription, voice activity detection, and context management for creating interactive AI experiences. It provides: Real-time Interaction Stream audio in real-time with low latency response times Speech Processing Built-in speech-to-text and text-to-speech capabilities with multiple voice options Voice Activity Detection Automatic detection of speech start/stop for natural conversations Context Management Intelligent handling of conversation history and system instructions ​ Installation To use AWSNovaSonicLLMService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[aws-nova-sonic]" We recommend setting up your AWS credentials as environment variables, as you’ll need them to initialize AWSNovaSonicLLMService : AWS_SECRET_ACCESS_KEY AWS_ACCESS_KEY_ID AWS_REGION ​ Basic Usage Here’s a simple example of setting up a conversational AI bot with AWS Nova Sonic: Copy Ask AI from pipecat.services.aws_nova_sonic.aws import AWSNovaSonicLLMService llm = AWSNovaSonicLLMService( secret_access_key = os.getenv( "AWS_SECRET_ACCESS_KEY" ), access_key_id = os.getenv( "AWS_ACCESS_KEY_ID" ), region = os.getenv( "AWS_REGION" ) voice_id = "tiffany" , # Voices: matthew, tiffany, amy ) ​ Configuration ​ Constructor Parameters ​ secret_access_key str required Your AWS secret access key ​ access_key_id str required Your AWS access key ID ​ region str required Specify the AWS region for the service (e.g., "us-east-1" ). Note that the service may not be available in all AWS regions: check the AWS Bedrock User Guide’s support table . ​ model str default: "amazon.nova-sonic-v1:0" AWS Nova Sonic model to use. Note that "amazon.nova-sonic-v1:0" is the only supported model as of 2025-05-08. ​ voice_id str default: "matthew" Voice for text-to-speech (options: "matthew" , "tiffany" , "amy" ) ​ params Params Configuration for model parameters ​ system_instruction str High-level instructions that guide the model’s behavior. Note that more commonly these instructions will be included as part of the context provided to kick off the conversation. ​ tools ToolsSchema List of function definitions for tool/function calling. Note that more commonly tools will be included as part of the context provided to kick off the conversation. ​ send_transcription_frames bool default: "True" Whether to emit transcription frames ​ Model Parameters The Params object configures the behavior of the AWS Nova Sonic model. It is strongly recommended to stick with default values (most easily by omitting params when constructing AWSNovaSonicLLMService ) unless you have a good understanding of the parameters and their impact. Deviating from the defaults may lead to unexpected behavior. ​ temperature float default: "0.7" Controls randomness in responses. Range: 0.0 to 2.0 ​ max_tokens int default: "1024" Maximum number of tokens to generate ​ top_p float default: "0.9" Cumulative probability cutoff for token selection. Range: 0.0 to 1.0 ​ input_sample_rate int default: "16000" Sample rate for input audio ​ output_sample_rate int default: "24000" Sample rate for output audio ​ input_sample_size int default: "16" Bit depth for input audio ​ input_channel_count int default: "1" Number of channels for input audio ​ output_sample_size int default: "16" Bit depth for output audio ​ output_channel_count int default: "1" Number of channels for output audio ​ Frame Types ​ Input Frames ​ InputAudioRawFrame Frame Raw audio data for speech input ​ OpenAILLMContextFrame Frame Contains conversation context ​ BotStoppedSpeakingFrame Frame Signals the bot has stopped speaking ​ Output Frames ​ TTSAudioRawFrame Frame Generated speech audio ​ LLMFullResponseStartFrame Frame Signals the start of a response from the LLM ​ LLMFullResponseEndFrame Frame Signals the end of a response from the LLM ​ TTSStartedFrame Frame Signals start of speech synthesis (coincides with the start of the LLM response, as this is a speech-to-speech model) ​ TTSStoppedFrame Frame Signals end of speech synthesis (coincides with the end of the LLM response, as this is a speech-to-speech model) ​ LLMTextFrame Frame Generated text responses from the LLM ​ TTSTextFrame Frame Generated text responses ​ TranscriptionFrame Frame Speech transcriptions. Only output if send_transcription_frames is True . ​ Function Calling This service supports function calling (also known as tool calling) which allows the LLM to request information from external services and APIs. For example, you can enable your bot to: Check current weather conditions Query databases Access external APIs Perform custom actions See the Function Calling guide for: Detailed implementation instructions Provider-specific function definitions Handler registration examples Control over function call behavior Complete usage examples ​ Next Steps ​ Examples Foundational Example Basic implementation showing core features Persistent Content Example Implementation showing saving and loading conversation history XTTS Gemini Multimodal Live On this page Installation Basic Usage Configuration Constructor Parameters Model Parameters Frame Types Input Frames Output Frames Function Calling Next Steps Examples Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/s2s_aws_77418186.txt b/s2s_aws_77418186.txt
new file mode 100644
index 0000000000000000000000000000000000000000..8679c421bbb3c62349491f8d741c56c0cb8a430a
--- /dev/null
+++ b/s2s_aws_77418186.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/s2s/aws#basic-usage
+Title: AWS Nova Sonic - Pipecat
+==================================================
+
+AWS Nova Sonic - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Speech AWS Nova Sonic Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech AWS Nova Sonic Gemini Multimodal Live OpenAI Realtime Beta Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline The AWSNovaSonicLLMService enables natural, real-time conversations with AWS Nova Sonic. It provides built-in audio transcription, voice activity detection, and context management for creating interactive AI experiences. It provides: Real-time Interaction Stream audio in real-time with low latency response times Speech Processing Built-in speech-to-text and text-to-speech capabilities with multiple voice options Voice Activity Detection Automatic detection of speech start/stop for natural conversations Context Management Intelligent handling of conversation history and system instructions ​ Installation To use AWSNovaSonicLLMService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[aws-nova-sonic]" We recommend setting up your AWS credentials as environment variables, as you’ll need them to initialize AWSNovaSonicLLMService : AWS_SECRET_ACCESS_KEY AWS_ACCESS_KEY_ID AWS_REGION ​ Basic Usage Here’s a simple example of setting up a conversational AI bot with AWS Nova Sonic: Copy Ask AI from pipecat.services.aws_nova_sonic.aws import AWSNovaSonicLLMService llm = AWSNovaSonicLLMService( secret_access_key = os.getenv( "AWS_SECRET_ACCESS_KEY" ), access_key_id = os.getenv( "AWS_ACCESS_KEY_ID" ), region = os.getenv( "AWS_REGION" ) voice_id = "tiffany" , # Voices: matthew, tiffany, amy ) ​ Configuration ​ Constructor Parameters ​ secret_access_key str required Your AWS secret access key ​ access_key_id str required Your AWS access key ID ​ region str required Specify the AWS region for the service (e.g., "us-east-1" ). Note that the service may not be available in all AWS regions: check the AWS Bedrock User Guide’s support table . ​ model str default: "amazon.nova-sonic-v1:0" AWS Nova Sonic model to use. Note that "amazon.nova-sonic-v1:0" is the only supported model as of 2025-05-08. ​ voice_id str default: "matthew" Voice for text-to-speech (options: "matthew" , "tiffany" , "amy" ) ​ params Params Configuration for model parameters ​ system_instruction str High-level instructions that guide the model’s behavior. Note that more commonly these instructions will be included as part of the context provided to kick off the conversation. ​ tools ToolsSchema List of function definitions for tool/function calling. Note that more commonly tools will be included as part of the context provided to kick off the conversation. ​ send_transcription_frames bool default: "True" Whether to emit transcription frames ​ Model Parameters The Params object configures the behavior of the AWS Nova Sonic model. It is strongly recommended to stick with default values (most easily by omitting params when constructing AWSNovaSonicLLMService ) unless you have a good understanding of the parameters and their impact. Deviating from the defaults may lead to unexpected behavior. ​ temperature float default: "0.7" Controls randomness in responses. Range: 0.0 to 2.0 ​ max_tokens int default: "1024" Maximum number of tokens to generate ​ top_p float default: "0.9" Cumulative probability cutoff for token selection. Range: 0.0 to 1.0 ​ input_sample_rate int default: "16000" Sample rate for input audio ​ output_sample_rate int default: "24000" Sample rate for output audio ​ input_sample_size int default: "16" Bit depth for input audio ​ input_channel_count int default: "1" Number of channels for input audio ​ output_sample_size int default: "16" Bit depth for output audio ​ output_channel_count int default: "1" Number of channels for output audio ​ Frame Types ​ Input Frames ​ InputAudioRawFrame Frame Raw audio data for speech input ​ OpenAILLMContextFrame Frame Contains conversation context ​ BotStoppedSpeakingFrame Frame Signals the bot has stopped speaking ​ Output Frames ​ TTSAudioRawFrame Frame Generated speech audio ​ LLMFullResponseStartFrame Frame Signals the start of a response from the LLM ​ LLMFullResponseEndFrame Frame Signals the end of a response from the LLM ​ TTSStartedFrame Frame Signals start of speech synthesis (coincides with the start of the LLM response, as this is a speech-to-speech model) ​ TTSStoppedFrame Frame Signals end of speech synthesis (coincides with the end of the LLM response, as this is a speech-to-speech model) ​ LLMTextFrame Frame Generated text responses from the LLM ​ TTSTextFrame Frame Generated text responses ​ TranscriptionFrame Frame Speech transcriptions. Only output if send_transcription_frames is True . ​ Function Calling This service supports function calling (also known as tool calling) which allows the LLM to request information from external services and APIs. For example, you can enable your bot to: Check current weather conditions Query databases Access external APIs Perform custom actions See the Function Calling guide for: Detailed implementation instructions Provider-specific function definitions Handler registration examples Control over function call behavior Complete usage examples ​ Next Steps ​ Examples Foundational Example Basic implementation showing core features Persistent Content Example Implementation showing saving and loading conversation history XTTS Gemini Multimodal Live On this page Installation Basic Usage Configuration Constructor Parameters Model Parameters Frame Types Input Frames Output Frames Function Calling Next Steps Examples Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/s2s_aws_813e9738.txt b/s2s_aws_813e9738.txt
new file mode 100644
index 0000000000000000000000000000000000000000..d763423c099e2bd3396d420156b6051d4a173670
--- /dev/null
+++ b/s2s_aws_813e9738.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/s2s/aws#examples
+Title: AWS Nova Sonic - Pipecat
+==================================================
+
+AWS Nova Sonic - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Speech AWS Nova Sonic Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech AWS Nova Sonic Gemini Multimodal Live OpenAI Realtime Beta Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline The AWSNovaSonicLLMService enables natural, real-time conversations with AWS Nova Sonic. It provides built-in audio transcription, voice activity detection, and context management for creating interactive AI experiences. It provides: Real-time Interaction Stream audio in real-time with low latency response times Speech Processing Built-in speech-to-text and text-to-speech capabilities with multiple voice options Voice Activity Detection Automatic detection of speech start/stop for natural conversations Context Management Intelligent handling of conversation history and system instructions ​ Installation To use AWSNovaSonicLLMService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[aws-nova-sonic]" We recommend setting up your AWS credentials as environment variables, as you’ll need them to initialize AWSNovaSonicLLMService : AWS_SECRET_ACCESS_KEY AWS_ACCESS_KEY_ID AWS_REGION ​ Basic Usage Here’s a simple example of setting up a conversational AI bot with AWS Nova Sonic: Copy Ask AI from pipecat.services.aws_nova_sonic.aws import AWSNovaSonicLLMService llm = AWSNovaSonicLLMService( secret_access_key = os.getenv( "AWS_SECRET_ACCESS_KEY" ), access_key_id = os.getenv( "AWS_ACCESS_KEY_ID" ), region = os.getenv( "AWS_REGION" ) voice_id = "tiffany" , # Voices: matthew, tiffany, amy ) ​ Configuration ​ Constructor Parameters ​ secret_access_key str required Your AWS secret access key ​ access_key_id str required Your AWS access key ID ​ region str required Specify the AWS region for the service (e.g., "us-east-1" ). Note that the service may not be available in all AWS regions: check the AWS Bedrock User Guide’s support table . ​ model str default: "amazon.nova-sonic-v1:0" AWS Nova Sonic model to use. Note that "amazon.nova-sonic-v1:0" is the only supported model as of 2025-05-08. ​ voice_id str default: "matthew" Voice for text-to-speech (options: "matthew" , "tiffany" , "amy" ) ​ params Params Configuration for model parameters ​ system_instruction str High-level instructions that guide the model’s behavior. Note that more commonly these instructions will be included as part of the context provided to kick off the conversation. ​ tools ToolsSchema List of function definitions for tool/function calling. Note that more commonly tools will be included as part of the context provided to kick off the conversation. ​ send_transcription_frames bool default: "True" Whether to emit transcription frames ​ Model Parameters The Params object configures the behavior of the AWS Nova Sonic model. It is strongly recommended to stick with default values (most easily by omitting params when constructing AWSNovaSonicLLMService ) unless you have a good understanding of the parameters and their impact. Deviating from the defaults may lead to unexpected behavior. ​ temperature float default: "0.7" Controls randomness in responses. Range: 0.0 to 2.0 ​ max_tokens int default: "1024" Maximum number of tokens to generate ​ top_p float default: "0.9" Cumulative probability cutoff for token selection. Range: 0.0 to 1.0 ​ input_sample_rate int default: "16000" Sample rate for input audio ​ output_sample_rate int default: "24000" Sample rate for output audio ​ input_sample_size int default: "16" Bit depth for input audio ​ input_channel_count int default: "1" Number of channels for input audio ​ output_sample_size int default: "16" Bit depth for output audio ​ output_channel_count int default: "1" Number of channels for output audio ​ Frame Types ​ Input Frames ​ InputAudioRawFrame Frame Raw audio data for speech input ​ OpenAILLMContextFrame Frame Contains conversation context ​ BotStoppedSpeakingFrame Frame Signals the bot has stopped speaking ​ Output Frames ​ TTSAudioRawFrame Frame Generated speech audio ​ LLMFullResponseStartFrame Frame Signals the start of a response from the LLM ​ LLMFullResponseEndFrame Frame Signals the end of a response from the LLM ​ TTSStartedFrame Frame Signals start of speech synthesis (coincides with the start of the LLM response, as this is a speech-to-speech model) ​ TTSStoppedFrame Frame Signals end of speech synthesis (coincides with the end of the LLM response, as this is a speech-to-speech model) ​ LLMTextFrame Frame Generated text responses from the LLM ​ TTSTextFrame Frame Generated text responses ​ TranscriptionFrame Frame Speech transcriptions. Only output if send_transcription_frames is True . ​ Function Calling This service supports function calling (also known as tool calling) which allows the LLM to request information from external services and APIs. For example, you can enable your bot to: Check current weather conditions Query databases Access external APIs Perform custom actions See the Function Calling guide for: Detailed implementation instructions Provider-specific function definitions Handler registration examples Control over function call behavior Complete usage examples ​ Next Steps ​ Examples Foundational Example Basic implementation showing core features Persistent Content Example Implementation showing saving and loading conversation history XTTS Gemini Multimodal Live On this page Installation Basic Usage Configuration Constructor Parameters Model Parameters Frame Types Input Frames Output Frames Function Calling Next Steps Examples Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/s2s_aws_8a867500.txt b/s2s_aws_8a867500.txt
new file mode 100644
index 0000000000000000000000000000000000000000..551e302aaf1db4db88e6acb6475f2158e4cf78b2
--- /dev/null
+++ b/s2s_aws_8a867500.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/s2s/aws#configuration
+Title: AWS Nova Sonic - Pipecat
+==================================================
+
+AWS Nova Sonic - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Speech AWS Nova Sonic Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech AWS Nova Sonic Gemini Multimodal Live OpenAI Realtime Beta Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline The AWSNovaSonicLLMService enables natural, real-time conversations with AWS Nova Sonic. It provides built-in audio transcription, voice activity detection, and context management for creating interactive AI experiences. It provides: Real-time Interaction Stream audio in real-time with low latency response times Speech Processing Built-in speech-to-text and text-to-speech capabilities with multiple voice options Voice Activity Detection Automatic detection of speech start/stop for natural conversations Context Management Intelligent handling of conversation history and system instructions ​ Installation To use AWSNovaSonicLLMService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[aws-nova-sonic]" We recommend setting up your AWS credentials as environment variables, as you’ll need them to initialize AWSNovaSonicLLMService : AWS_SECRET_ACCESS_KEY AWS_ACCESS_KEY_ID AWS_REGION ​ Basic Usage Here’s a simple example of setting up a conversational AI bot with AWS Nova Sonic: Copy Ask AI from pipecat.services.aws_nova_sonic.aws import AWSNovaSonicLLMService llm = AWSNovaSonicLLMService( secret_access_key = os.getenv( "AWS_SECRET_ACCESS_KEY" ), access_key_id = os.getenv( "AWS_ACCESS_KEY_ID" ), region = os.getenv( "AWS_REGION" ) voice_id = "tiffany" , # Voices: matthew, tiffany, amy ) ​ Configuration ​ Constructor Parameters ​ secret_access_key str required Your AWS secret access key ​ access_key_id str required Your AWS access key ID ​ region str required Specify the AWS region for the service (e.g., "us-east-1" ). Note that the service may not be available in all AWS regions: check the AWS Bedrock User Guide’s support table . ​ model str default: "amazon.nova-sonic-v1:0" AWS Nova Sonic model to use. Note that "amazon.nova-sonic-v1:0" is the only supported model as of 2025-05-08. ​ voice_id str default: "matthew" Voice for text-to-speech (options: "matthew" , "tiffany" , "amy" ) ​ params Params Configuration for model parameters ​ system_instruction str High-level instructions that guide the model’s behavior. Note that more commonly these instructions will be included as part of the context provided to kick off the conversation. ​ tools ToolsSchema List of function definitions for tool/function calling. Note that more commonly tools will be included as part of the context provided to kick off the conversation. ​ send_transcription_frames bool default: "True" Whether to emit transcription frames ​ Model Parameters The Params object configures the behavior of the AWS Nova Sonic model. It is strongly recommended to stick with default values (most easily by omitting params when constructing AWSNovaSonicLLMService ) unless you have a good understanding of the parameters and their impact. Deviating from the defaults may lead to unexpected behavior. ​ temperature float default: "0.7" Controls randomness in responses. Range: 0.0 to 2.0 ​ max_tokens int default: "1024" Maximum number of tokens to generate ​ top_p float default: "0.9" Cumulative probability cutoff for token selection. Range: 0.0 to 1.0 ​ input_sample_rate int default: "16000" Sample rate for input audio ​ output_sample_rate int default: "24000" Sample rate for output audio ​ input_sample_size int default: "16" Bit depth for input audio ​ input_channel_count int default: "1" Number of channels for input audio ​ output_sample_size int default: "16" Bit depth for output audio ​ output_channel_count int default: "1" Number of channels for output audio ​ Frame Types ​ Input Frames ​ InputAudioRawFrame Frame Raw audio data for speech input ​ OpenAILLMContextFrame Frame Contains conversation context ​ BotStoppedSpeakingFrame Frame Signals the bot has stopped speaking ​ Output Frames ​ TTSAudioRawFrame Frame Generated speech audio ​ LLMFullResponseStartFrame Frame Signals the start of a response from the LLM ​ LLMFullResponseEndFrame Frame Signals the end of a response from the LLM ​ TTSStartedFrame Frame Signals start of speech synthesis (coincides with the start of the LLM response, as this is a speech-to-speech model) ​ TTSStoppedFrame Frame Signals end of speech synthesis (coincides with the end of the LLM response, as this is a speech-to-speech model) ​ LLMTextFrame Frame Generated text responses from the LLM ​ TTSTextFrame Frame Generated text responses ​ TranscriptionFrame Frame Speech transcriptions. Only output if send_transcription_frames is True . ​ Function Calling This service supports function calling (also known as tool calling) which allows the LLM to request information from external services and APIs. For example, you can enable your bot to: Check current weather conditions Query databases Access external APIs Perform custom actions See the Function Calling guide for: Detailed implementation instructions Provider-specific function definitions Handler registration examples Control over function call behavior Complete usage examples ​ Next Steps ​ Examples Foundational Example Basic implementation showing core features Persistent Content Example Implementation showing saving and loading conversation history XTTS Gemini Multimodal Live On this page Installation Basic Usage Configuration Constructor Parameters Model Parameters Frame Types Input Frames Output Frames Function Calling Next Steps Examples Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/s2s_aws_90e24cf3.txt b/s2s_aws_90e24cf3.txt
new file mode 100644
index 0000000000000000000000000000000000000000..3c27b121238e3238fb29836d1c9f1017bd575a19
--- /dev/null
+++ b/s2s_aws_90e24cf3.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/s2s/aws#param-input-channel-count
+Title: AWS Nova Sonic - Pipecat
+==================================================
+
+AWS Nova Sonic - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Speech AWS Nova Sonic Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech AWS Nova Sonic Gemini Multimodal Live OpenAI Realtime Beta Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline The AWSNovaSonicLLMService enables natural, real-time conversations with AWS Nova Sonic. It provides built-in audio transcription, voice activity detection, and context management for creating interactive AI experiences. It provides: Real-time Interaction Stream audio in real-time with low latency response times Speech Processing Built-in speech-to-text and text-to-speech capabilities with multiple voice options Voice Activity Detection Automatic detection of speech start/stop for natural conversations Context Management Intelligent handling of conversation history and system instructions ​ Installation To use AWSNovaSonicLLMService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[aws-nova-sonic]" We recommend setting up your AWS credentials as environment variables, as you’ll need them to initialize AWSNovaSonicLLMService : AWS_SECRET_ACCESS_KEY AWS_ACCESS_KEY_ID AWS_REGION ​ Basic Usage Here’s a simple example of setting up a conversational AI bot with AWS Nova Sonic: Copy Ask AI from pipecat.services.aws_nova_sonic.aws import AWSNovaSonicLLMService llm = AWSNovaSonicLLMService( secret_access_key = os.getenv( "AWS_SECRET_ACCESS_KEY" ), access_key_id = os.getenv( "AWS_ACCESS_KEY_ID" ), region = os.getenv( "AWS_REGION" ) voice_id = "tiffany" , # Voices: matthew, tiffany, amy ) ​ Configuration ​ Constructor Parameters ​ secret_access_key str required Your AWS secret access key ​ access_key_id str required Your AWS access key ID ​ region str required Specify the AWS region for the service (e.g., "us-east-1" ). Note that the service may not be available in all AWS regions: check the AWS Bedrock User Guide’s support table . ​ model str default: "amazon.nova-sonic-v1:0" AWS Nova Sonic model to use. Note that "amazon.nova-sonic-v1:0" is the only supported model as of 2025-05-08. ​ voice_id str default: "matthew" Voice for text-to-speech (options: "matthew" , "tiffany" , "amy" ) ​ params Params Configuration for model parameters ​ system_instruction str High-level instructions that guide the model’s behavior. Note that more commonly these instructions will be included as part of the context provided to kick off the conversation. ​ tools ToolsSchema List of function definitions for tool/function calling. Note that more commonly tools will be included as part of the context provided to kick off the conversation. ​ send_transcription_frames bool default: "True" Whether to emit transcription frames ​ Model Parameters The Params object configures the behavior of the AWS Nova Sonic model. It is strongly recommended to stick with default values (most easily by omitting params when constructing AWSNovaSonicLLMService ) unless you have a good understanding of the parameters and their impact. Deviating from the defaults may lead to unexpected behavior. ​ temperature float default: "0.7" Controls randomness in responses. Range: 0.0 to 2.0 ​ max_tokens int default: "1024" Maximum number of tokens to generate ​ top_p float default: "0.9" Cumulative probability cutoff for token selection. Range: 0.0 to 1.0 ​ input_sample_rate int default: "16000" Sample rate for input audio ​ output_sample_rate int default: "24000" Sample rate for output audio ​ input_sample_size int default: "16" Bit depth for input audio ​ input_channel_count int default: "1" Number of channels for input audio ​ output_sample_size int default: "16" Bit depth for output audio ​ output_channel_count int default: "1" Number of channels for output audio ​ Frame Types ​ Input Frames ​ InputAudioRawFrame Frame Raw audio data for speech input ​ OpenAILLMContextFrame Frame Contains conversation context ​ BotStoppedSpeakingFrame Frame Signals the bot has stopped speaking ​ Output Frames ​ TTSAudioRawFrame Frame Generated speech audio ​ LLMFullResponseStartFrame Frame Signals the start of a response from the LLM ​ LLMFullResponseEndFrame Frame Signals the end of a response from the LLM ​ TTSStartedFrame Frame Signals start of speech synthesis (coincides with the start of the LLM response, as this is a speech-to-speech model) ​ TTSStoppedFrame Frame Signals end of speech synthesis (coincides with the end of the LLM response, as this is a speech-to-speech model) ​ LLMTextFrame Frame Generated text responses from the LLM ​ TTSTextFrame Frame Generated text responses ​ TranscriptionFrame Frame Speech transcriptions. Only output if send_transcription_frames is True . ​ Function Calling This service supports function calling (also known as tool calling) which allows the LLM to request information from external services and APIs. For example, you can enable your bot to: Check current weather conditions Query databases Access external APIs Perform custom actions See the Function Calling guide for: Detailed implementation instructions Provider-specific function definitions Handler registration examples Control over function call behavior Complete usage examples ​ Next Steps ​ Examples Foundational Example Basic implementation showing core features Persistent Content Example Implementation showing saving and loading conversation history XTTS Gemini Multimodal Live On this page Installation Basic Usage Configuration Constructor Parameters Model Parameters Frame Types Input Frames Output Frames Function Calling Next Steps Examples Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/s2s_aws_a33e104e.txt b/s2s_aws_a33e104e.txt
new file mode 100644
index 0000000000000000000000000000000000000000..9f1b83588efb413adbc1a600c3f30c41c4474a98
--- /dev/null
+++ b/s2s_aws_a33e104e.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/s2s/aws#param-tts-text-frame
+Title: AWS Nova Sonic - Pipecat
+==================================================
+
+AWS Nova Sonic - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Speech AWS Nova Sonic Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech AWS Nova Sonic Gemini Multimodal Live OpenAI Realtime Beta Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline The AWSNovaSonicLLMService enables natural, real-time conversations with AWS Nova Sonic. It provides built-in audio transcription, voice activity detection, and context management for creating interactive AI experiences. It provides: Real-time Interaction Stream audio in real-time with low latency response times Speech Processing Built-in speech-to-text and text-to-speech capabilities with multiple voice options Voice Activity Detection Automatic detection of speech start/stop for natural conversations Context Management Intelligent handling of conversation history and system instructions ​ Installation To use AWSNovaSonicLLMService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[aws-nova-sonic]" We recommend setting up your AWS credentials as environment variables, as you’ll need them to initialize AWSNovaSonicLLMService : AWS_SECRET_ACCESS_KEY AWS_ACCESS_KEY_ID AWS_REGION ​ Basic Usage Here’s a simple example of setting up a conversational AI bot with AWS Nova Sonic: Copy Ask AI from pipecat.services.aws_nova_sonic.aws import AWSNovaSonicLLMService llm = AWSNovaSonicLLMService( secret_access_key = os.getenv( "AWS_SECRET_ACCESS_KEY" ), access_key_id = os.getenv( "AWS_ACCESS_KEY_ID" ), region = os.getenv( "AWS_REGION" ) voice_id = "tiffany" , # Voices: matthew, tiffany, amy ) ​ Configuration ​ Constructor Parameters ​ secret_access_key str required Your AWS secret access key ​ access_key_id str required Your AWS access key ID ​ region str required Specify the AWS region for the service (e.g., "us-east-1" ). Note that the service may not be available in all AWS regions: check the AWS Bedrock User Guide’s support table . ​ model str default: "amazon.nova-sonic-v1:0" AWS Nova Sonic model to use. Note that "amazon.nova-sonic-v1:0" is the only supported model as of 2025-05-08. ​ voice_id str default: "matthew" Voice for text-to-speech (options: "matthew" , "tiffany" , "amy" ) ​ params Params Configuration for model parameters ​ system_instruction str High-level instructions that guide the model’s behavior. Note that more commonly these instructions will be included as part of the context provided to kick off the conversation. ​ tools ToolsSchema List of function definitions for tool/function calling. Note that more commonly tools will be included as part of the context provided to kick off the conversation. ​ send_transcription_frames bool default: "True" Whether to emit transcription frames ​ Model Parameters The Params object configures the behavior of the AWS Nova Sonic model. It is strongly recommended to stick with default values (most easily by omitting params when constructing AWSNovaSonicLLMService ) unless you have a good understanding of the parameters and their impact. Deviating from the defaults may lead to unexpected behavior. ​ temperature float default: "0.7" Controls randomness in responses. Range: 0.0 to 2.0 ​ max_tokens int default: "1024" Maximum number of tokens to generate ​ top_p float default: "0.9" Cumulative probability cutoff for token selection. Range: 0.0 to 1.0 ​ input_sample_rate int default: "16000" Sample rate for input audio ​ output_sample_rate int default: "24000" Sample rate for output audio ​ input_sample_size int default: "16" Bit depth for input audio ​ input_channel_count int default: "1" Number of channels for input audio ​ output_sample_size int default: "16" Bit depth for output audio ​ output_channel_count int default: "1" Number of channels for output audio ​ Frame Types ​ Input Frames ​ InputAudioRawFrame Frame Raw audio data for speech input ​ OpenAILLMContextFrame Frame Contains conversation context ​ BotStoppedSpeakingFrame Frame Signals the bot has stopped speaking ​ Output Frames ​ TTSAudioRawFrame Frame Generated speech audio ​ LLMFullResponseStartFrame Frame Signals the start of a response from the LLM ​ LLMFullResponseEndFrame Frame Signals the end of a response from the LLM ​ TTSStartedFrame Frame Signals start of speech synthesis (coincides with the start of the LLM response, as this is a speech-to-speech model) ​ TTSStoppedFrame Frame Signals end of speech synthesis (coincides with the end of the LLM response, as this is a speech-to-speech model) ​ LLMTextFrame Frame Generated text responses from the LLM ​ TTSTextFrame Frame Generated text responses ​ TranscriptionFrame Frame Speech transcriptions. Only output if send_transcription_frames is True . ​ Function Calling This service supports function calling (also known as tool calling) which allows the LLM to request information from external services and APIs. For example, you can enable your bot to: Check current weather conditions Query databases Access external APIs Perform custom actions See the Function Calling guide for: Detailed implementation instructions Provider-specific function definitions Handler registration examples Control over function call behavior Complete usage examples ​ Next Steps ​ Examples Foundational Example Basic implementation showing core features Persistent Content Example Implementation showing saving and loading conversation history XTTS Gemini Multimodal Live On this page Installation Basic Usage Configuration Constructor Parameters Model Parameters Frame Types Input Frames Output Frames Function Calling Next Steps Examples Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/s2s_aws_a9a31e38.txt b/s2s_aws_a9a31e38.txt
new file mode 100644
index 0000000000000000000000000000000000000000..dd7b2e35028fac6bfa069333e5c94c0be6139942
--- /dev/null
+++ b/s2s_aws_a9a31e38.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/s2s/aws#param-voice-id
+Title: AWS Nova Sonic - Pipecat
+==================================================
+
+AWS Nova Sonic - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Speech AWS Nova Sonic Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech AWS Nova Sonic Gemini Multimodal Live OpenAI Realtime Beta Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline The AWSNovaSonicLLMService enables natural, real-time conversations with AWS Nova Sonic. It provides built-in audio transcription, voice activity detection, and context management for creating interactive AI experiences. It provides: Real-time Interaction Stream audio in real-time with low latency response times Speech Processing Built-in speech-to-text and text-to-speech capabilities with multiple voice options Voice Activity Detection Automatic detection of speech start/stop for natural conversations Context Management Intelligent handling of conversation history and system instructions ​ Installation To use AWSNovaSonicLLMService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[aws-nova-sonic]" We recommend setting up your AWS credentials as environment variables, as you’ll need them to initialize AWSNovaSonicLLMService : AWS_SECRET_ACCESS_KEY AWS_ACCESS_KEY_ID AWS_REGION ​ Basic Usage Here’s a simple example of setting up a conversational AI bot with AWS Nova Sonic: Copy Ask AI from pipecat.services.aws_nova_sonic.aws import AWSNovaSonicLLMService llm = AWSNovaSonicLLMService( secret_access_key = os.getenv( "AWS_SECRET_ACCESS_KEY" ), access_key_id = os.getenv( "AWS_ACCESS_KEY_ID" ), region = os.getenv( "AWS_REGION" ) voice_id = "tiffany" , # Voices: matthew, tiffany, amy ) ​ Configuration ​ Constructor Parameters ​ secret_access_key str required Your AWS secret access key ​ access_key_id str required Your AWS access key ID ​ region str required Specify the AWS region for the service (e.g., "us-east-1" ). Note that the service may not be available in all AWS regions: check the AWS Bedrock User Guide’s support table . ​ model str default: "amazon.nova-sonic-v1:0" AWS Nova Sonic model to use. Note that "amazon.nova-sonic-v1:0" is the only supported model as of 2025-05-08. ​ voice_id str default: "matthew" Voice for text-to-speech (options: "matthew" , "tiffany" , "amy" ) ​ params Params Configuration for model parameters ​ system_instruction str High-level instructions that guide the model’s behavior. Note that more commonly these instructions will be included as part of the context provided to kick off the conversation. ​ tools ToolsSchema List of function definitions for tool/function calling. Note that more commonly tools will be included as part of the context provided to kick off the conversation. ​ send_transcription_frames bool default: "True" Whether to emit transcription frames ​ Model Parameters The Params object configures the behavior of the AWS Nova Sonic model. It is strongly recommended to stick with default values (most easily by omitting params when constructing AWSNovaSonicLLMService ) unless you have a good understanding of the parameters and their impact. Deviating from the defaults may lead to unexpected behavior. ​ temperature float default: "0.7" Controls randomness in responses. Range: 0.0 to 2.0 ​ max_tokens int default: "1024" Maximum number of tokens to generate ​ top_p float default: "0.9" Cumulative probability cutoff for token selection. Range: 0.0 to 1.0 ​ input_sample_rate int default: "16000" Sample rate for input audio ​ output_sample_rate int default: "24000" Sample rate for output audio ​ input_sample_size int default: "16" Bit depth for input audio ​ input_channel_count int default: "1" Number of channels for input audio ​ output_sample_size int default: "16" Bit depth for output audio ​ output_channel_count int default: "1" Number of channels for output audio ​ Frame Types ​ Input Frames ​ InputAudioRawFrame Frame Raw audio data for speech input ​ OpenAILLMContextFrame Frame Contains conversation context ​ BotStoppedSpeakingFrame Frame Signals the bot has stopped speaking ​ Output Frames ​ TTSAudioRawFrame Frame Generated speech audio ​ LLMFullResponseStartFrame Frame Signals the start of a response from the LLM ​ LLMFullResponseEndFrame Frame Signals the end of a response from the LLM ​ TTSStartedFrame Frame Signals start of speech synthesis (coincides with the start of the LLM response, as this is a speech-to-speech model) ​ TTSStoppedFrame Frame Signals end of speech synthesis (coincides with the end of the LLM response, as this is a speech-to-speech model) ​ LLMTextFrame Frame Generated text responses from the LLM ​ TTSTextFrame Frame Generated text responses ​ TranscriptionFrame Frame Speech transcriptions. Only output if send_transcription_frames is True . ​ Function Calling This service supports function calling (also known as tool calling) which allows the LLM to request information from external services and APIs. For example, you can enable your bot to: Check current weather conditions Query databases Access external APIs Perform custom actions See the Function Calling guide for: Detailed implementation instructions Provider-specific function definitions Handler registration examples Control over function call behavior Complete usage examples ​ Next Steps ​ Examples Foundational Example Basic implementation showing core features Persistent Content Example Implementation showing saving and loading conversation history XTTS Gemini Multimodal Live On this page Installation Basic Usage Configuration Constructor Parameters Model Parameters Frame Types Input Frames Output Frames Function Calling Next Steps Examples Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/s2s_aws_af2f152e.txt b/s2s_aws_af2f152e.txt
new file mode 100644
index 0000000000000000000000000000000000000000..6ab0873854d8d2604de6731d928a96852717f886
--- /dev/null
+++ b/s2s_aws_af2f152e.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/s2s/aws#param-llm-text-frame
+Title: AWS Nova Sonic - Pipecat
+==================================================
+
+AWS Nova Sonic - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Speech AWS Nova Sonic Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech AWS Nova Sonic Gemini Multimodal Live OpenAI Realtime Beta Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline The AWSNovaSonicLLMService enables natural, real-time conversations with AWS Nova Sonic. It provides built-in audio transcription, voice activity detection, and context management for creating interactive AI experiences. It provides: Real-time Interaction Stream audio in real-time with low latency response times Speech Processing Built-in speech-to-text and text-to-speech capabilities with multiple voice options Voice Activity Detection Automatic detection of speech start/stop for natural conversations Context Management Intelligent handling of conversation history and system instructions ​ Installation To use AWSNovaSonicLLMService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[aws-nova-sonic]" We recommend setting up your AWS credentials as environment variables, as you’ll need them to initialize AWSNovaSonicLLMService : AWS_SECRET_ACCESS_KEY AWS_ACCESS_KEY_ID AWS_REGION ​ Basic Usage Here’s a simple example of setting up a conversational AI bot with AWS Nova Sonic: Copy Ask AI from pipecat.services.aws_nova_sonic.aws import AWSNovaSonicLLMService llm = AWSNovaSonicLLMService( secret_access_key = os.getenv( "AWS_SECRET_ACCESS_KEY" ), access_key_id = os.getenv( "AWS_ACCESS_KEY_ID" ), region = os.getenv( "AWS_REGION" ) voice_id = "tiffany" , # Voices: matthew, tiffany, amy ) ​ Configuration ​ Constructor Parameters ​ secret_access_key str required Your AWS secret access key ​ access_key_id str required Your AWS access key ID ​ region str required Specify the AWS region for the service (e.g., "us-east-1" ). Note that the service may not be available in all AWS regions: check the AWS Bedrock User Guide’s support table . ​ model str default: "amazon.nova-sonic-v1:0" AWS Nova Sonic model to use. Note that "amazon.nova-sonic-v1:0" is the only supported model as of 2025-05-08. ​ voice_id str default: "matthew" Voice for text-to-speech (options: "matthew" , "tiffany" , "amy" ) ​ params Params Configuration for model parameters ​ system_instruction str High-level instructions that guide the model’s behavior. Note that more commonly these instructions will be included as part of the context provided to kick off the conversation. ​ tools ToolsSchema List of function definitions for tool/function calling. Note that more commonly tools will be included as part of the context provided to kick off the conversation. ​ send_transcription_frames bool default: "True" Whether to emit transcription frames ​ Model Parameters The Params object configures the behavior of the AWS Nova Sonic model. It is strongly recommended to stick with default values (most easily by omitting params when constructing AWSNovaSonicLLMService ) unless you have a good understanding of the parameters and their impact. Deviating from the defaults may lead to unexpected behavior. ​ temperature float default: "0.7" Controls randomness in responses. Range: 0.0 to 2.0 ​ max_tokens int default: "1024" Maximum number of tokens to generate ​ top_p float default: "0.9" Cumulative probability cutoff for token selection. Range: 0.0 to 1.0 ​ input_sample_rate int default: "16000" Sample rate for input audio ​ output_sample_rate int default: "24000" Sample rate for output audio ​ input_sample_size int default: "16" Bit depth for input audio ​ input_channel_count int default: "1" Number of channels for input audio ​ output_sample_size int default: "16" Bit depth for output audio ​ output_channel_count int default: "1" Number of channels for output audio ​ Frame Types ​ Input Frames ​ InputAudioRawFrame Frame Raw audio data for speech input ​ OpenAILLMContextFrame Frame Contains conversation context ​ BotStoppedSpeakingFrame Frame Signals the bot has stopped speaking ​ Output Frames ​ TTSAudioRawFrame Frame Generated speech audio ​ LLMFullResponseStartFrame Frame Signals the start of a response from the LLM ​ LLMFullResponseEndFrame Frame Signals the end of a response from the LLM ​ TTSStartedFrame Frame Signals start of speech synthesis (coincides with the start of the LLM response, as this is a speech-to-speech model) ​ TTSStoppedFrame Frame Signals end of speech synthesis (coincides with the end of the LLM response, as this is a speech-to-speech model) ​ LLMTextFrame Frame Generated text responses from the LLM ​ TTSTextFrame Frame Generated text responses ​ TranscriptionFrame Frame Speech transcriptions. Only output if send_transcription_frames is True . ​ Function Calling This service supports function calling (also known as tool calling) which allows the LLM to request information from external services and APIs. For example, you can enable your bot to: Check current weather conditions Query databases Access external APIs Perform custom actions See the Function Calling guide for: Detailed implementation instructions Provider-specific function definitions Handler registration examples Control over function call behavior Complete usage examples ​ Next Steps ​ Examples Foundational Example Basic implementation showing core features Persistent Content Example Implementation showing saving and loading conversation history XTTS Gemini Multimodal Live On this page Installation Basic Usage Configuration Constructor Parameters Model Parameters Frame Types Input Frames Output Frames Function Calling Next Steps Examples Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/s2s_aws_bc64fa52.txt b/s2s_aws_bc64fa52.txt
new file mode 100644
index 0000000000000000000000000000000000000000..f0888c385c4597e7be8a05294d5d9029392e32f1
--- /dev/null
+++ b/s2s_aws_bc64fa52.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/s2s/aws#param-bot-stopped-speaking-frame
+Title: AWS Nova Sonic - Pipecat
+==================================================
+
+AWS Nova Sonic - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Speech AWS Nova Sonic Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech AWS Nova Sonic Gemini Multimodal Live OpenAI Realtime Beta Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline The AWSNovaSonicLLMService enables natural, real-time conversations with AWS Nova Sonic. It provides built-in audio transcription, voice activity detection, and context management for creating interactive AI experiences. It provides: Real-time Interaction Stream audio in real-time with low latency response times Speech Processing Built-in speech-to-text and text-to-speech capabilities with multiple voice options Voice Activity Detection Automatic detection of speech start/stop for natural conversations Context Management Intelligent handling of conversation history and system instructions ​ Installation To use AWSNovaSonicLLMService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[aws-nova-sonic]" We recommend setting up your AWS credentials as environment variables, as you’ll need them to initialize AWSNovaSonicLLMService : AWS_SECRET_ACCESS_KEY AWS_ACCESS_KEY_ID AWS_REGION ​ Basic Usage Here’s a simple example of setting up a conversational AI bot with AWS Nova Sonic: Copy Ask AI from pipecat.services.aws_nova_sonic.aws import AWSNovaSonicLLMService llm = AWSNovaSonicLLMService( secret_access_key = os.getenv( "AWS_SECRET_ACCESS_KEY" ), access_key_id = os.getenv( "AWS_ACCESS_KEY_ID" ), region = os.getenv( "AWS_REGION" ) voice_id = "tiffany" , # Voices: matthew, tiffany, amy ) ​ Configuration ​ Constructor Parameters ​ secret_access_key str required Your AWS secret access key ​ access_key_id str required Your AWS access key ID ​ region str required Specify the AWS region for the service (e.g., "us-east-1" ). Note that the service may not be available in all AWS regions: check the AWS Bedrock User Guide’s support table . ​ model str default: "amazon.nova-sonic-v1:0" AWS Nova Sonic model to use. Note that "amazon.nova-sonic-v1:0" is the only supported model as of 2025-05-08. ​ voice_id str default: "matthew" Voice for text-to-speech (options: "matthew" , "tiffany" , "amy" ) ​ params Params Configuration for model parameters ​ system_instruction str High-level instructions that guide the model’s behavior. Note that more commonly these instructions will be included as part of the context provided to kick off the conversation. ​ tools ToolsSchema List of function definitions for tool/function calling. Note that more commonly tools will be included as part of the context provided to kick off the conversation. ​ send_transcription_frames bool default: "True" Whether to emit transcription frames ​ Model Parameters The Params object configures the behavior of the AWS Nova Sonic model. It is strongly recommended to stick with default values (most easily by omitting params when constructing AWSNovaSonicLLMService ) unless you have a good understanding of the parameters and their impact. Deviating from the defaults may lead to unexpected behavior. ​ temperature float default: "0.7" Controls randomness in responses. Range: 0.0 to 2.0 ​ max_tokens int default: "1024" Maximum number of tokens to generate ​ top_p float default: "0.9" Cumulative probability cutoff for token selection. Range: 0.0 to 1.0 ​ input_sample_rate int default: "16000" Sample rate for input audio ​ output_sample_rate int default: "24000" Sample rate for output audio ​ input_sample_size int default: "16" Bit depth for input audio ​ input_channel_count int default: "1" Number of channels for input audio ​ output_sample_size int default: "16" Bit depth for output audio ​ output_channel_count int default: "1" Number of channels for output audio ​ Frame Types ​ Input Frames ​ InputAudioRawFrame Frame Raw audio data for speech input ​ OpenAILLMContextFrame Frame Contains conversation context ​ BotStoppedSpeakingFrame Frame Signals the bot has stopped speaking ​ Output Frames ​ TTSAudioRawFrame Frame Generated speech audio ​ LLMFullResponseStartFrame Frame Signals the start of a response from the LLM ​ LLMFullResponseEndFrame Frame Signals the end of a response from the LLM ​ TTSStartedFrame Frame Signals start of speech synthesis (coincides with the start of the LLM response, as this is a speech-to-speech model) ​ TTSStoppedFrame Frame Signals end of speech synthesis (coincides with the end of the LLM response, as this is a speech-to-speech model) ​ LLMTextFrame Frame Generated text responses from the LLM ​ TTSTextFrame Frame Generated text responses ​ TranscriptionFrame Frame Speech transcriptions. Only output if send_transcription_frames is True . ​ Function Calling This service supports function calling (also known as tool calling) which allows the LLM to request information from external services and APIs. For example, you can enable your bot to: Check current weather conditions Query databases Access external APIs Perform custom actions See the Function Calling guide for: Detailed implementation instructions Provider-specific function definitions Handler registration examples Control over function call behavior Complete usage examples ​ Next Steps ​ Examples Foundational Example Basic implementation showing core features Persistent Content Example Implementation showing saving and loading conversation history XTTS Gemini Multimodal Live On this page Installation Basic Usage Configuration Constructor Parameters Model Parameters Frame Types Input Frames Output Frames Function Calling Next Steps Examples Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/s2s_aws_ca053c8f.txt b/s2s_aws_ca053c8f.txt
new file mode 100644
index 0000000000000000000000000000000000000000..132e24d36eee8b7655903ccc891e187b7191c279
--- /dev/null
+++ b/s2s_aws_ca053c8f.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/s2s/aws#param-temperature
+Title: AWS Nova Sonic - Pipecat
+==================================================
+
+AWS Nova Sonic - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Speech AWS Nova Sonic Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech AWS Nova Sonic Gemini Multimodal Live OpenAI Realtime Beta Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline The AWSNovaSonicLLMService enables natural, real-time conversations with AWS Nova Sonic. It provides built-in audio transcription, voice activity detection, and context management for creating interactive AI experiences. It provides: Real-time Interaction Stream audio in real-time with low latency response times Speech Processing Built-in speech-to-text and text-to-speech capabilities with multiple voice options Voice Activity Detection Automatic detection of speech start/stop for natural conversations Context Management Intelligent handling of conversation history and system instructions ​ Installation To use AWSNovaSonicLLMService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[aws-nova-sonic]" We recommend setting up your AWS credentials as environment variables, as you’ll need them to initialize AWSNovaSonicLLMService : AWS_SECRET_ACCESS_KEY AWS_ACCESS_KEY_ID AWS_REGION ​ Basic Usage Here’s a simple example of setting up a conversational AI bot with AWS Nova Sonic: Copy Ask AI from pipecat.services.aws_nova_sonic.aws import AWSNovaSonicLLMService llm = AWSNovaSonicLLMService( secret_access_key = os.getenv( "AWS_SECRET_ACCESS_KEY" ), access_key_id = os.getenv( "AWS_ACCESS_KEY_ID" ), region = os.getenv( "AWS_REGION" ) voice_id = "tiffany" , # Voices: matthew, tiffany, amy ) ​ Configuration ​ Constructor Parameters ​ secret_access_key str required Your AWS secret access key ​ access_key_id str required Your AWS access key ID ​ region str required Specify the AWS region for the service (e.g., "us-east-1" ). Note that the service may not be available in all AWS regions: check the AWS Bedrock User Guide’s support table . ​ model str default: "amazon.nova-sonic-v1:0" AWS Nova Sonic model to use. Note that "amazon.nova-sonic-v1:0" is the only supported model as of 2025-05-08. ​ voice_id str default: "matthew" Voice for text-to-speech (options: "matthew" , "tiffany" , "amy" ) ​ params Params Configuration for model parameters ​ system_instruction str High-level instructions that guide the model’s behavior. Note that more commonly these instructions will be included as part of the context provided to kick off the conversation. ​ tools ToolsSchema List of function definitions for tool/function calling. Note that more commonly tools will be included as part of the context provided to kick off the conversation. ​ send_transcription_frames bool default: "True" Whether to emit transcription frames ​ Model Parameters The Params object configures the behavior of the AWS Nova Sonic model. It is strongly recommended to stick with default values (most easily by omitting params when constructing AWSNovaSonicLLMService ) unless you have a good understanding of the parameters and their impact. Deviating from the defaults may lead to unexpected behavior. ​ temperature float default: "0.7" Controls randomness in responses. Range: 0.0 to 2.0 ​ max_tokens int default: "1024" Maximum number of tokens to generate ​ top_p float default: "0.9" Cumulative probability cutoff for token selection. Range: 0.0 to 1.0 ​ input_sample_rate int default: "16000" Sample rate for input audio ​ output_sample_rate int default: "24000" Sample rate for output audio ​ input_sample_size int default: "16" Bit depth for input audio ​ input_channel_count int default: "1" Number of channels for input audio ​ output_sample_size int default: "16" Bit depth for output audio ​ output_channel_count int default: "1" Number of channels for output audio ​ Frame Types ​ Input Frames ​ InputAudioRawFrame Frame Raw audio data for speech input ​ OpenAILLMContextFrame Frame Contains conversation context ​ BotStoppedSpeakingFrame Frame Signals the bot has stopped speaking ​ Output Frames ​ TTSAudioRawFrame Frame Generated speech audio ​ LLMFullResponseStartFrame Frame Signals the start of a response from the LLM ​ LLMFullResponseEndFrame Frame Signals the end of a response from the LLM ​ TTSStartedFrame Frame Signals start of speech synthesis (coincides with the start of the LLM response, as this is a speech-to-speech model) ​ TTSStoppedFrame Frame Signals end of speech synthesis (coincides with the end of the LLM response, as this is a speech-to-speech model) ​ LLMTextFrame Frame Generated text responses from the LLM ​ TTSTextFrame Frame Generated text responses ​ TranscriptionFrame Frame Speech transcriptions. Only output if send_transcription_frames is True . ​ Function Calling This service supports function calling (also known as tool calling) which allows the LLM to request information from external services and APIs. For example, you can enable your bot to: Check current weather conditions Query databases Access external APIs Perform custom actions See the Function Calling guide for: Detailed implementation instructions Provider-specific function definitions Handler registration examples Control over function call behavior Complete usage examples ​ Next Steps ​ Examples Foundational Example Basic implementation showing core features Persistent Content Example Implementation showing saving and loading conversation history XTTS Gemini Multimodal Live On this page Installation Basic Usage Configuration Constructor Parameters Model Parameters Frame Types Input Frames Output Frames Function Calling Next Steps Examples Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/s2s_aws_cb1e7390.txt b/s2s_aws_cb1e7390.txt
new file mode 100644
index 0000000000000000000000000000000000000000..fd8188c7304c8261127696bce168ca520f688935
--- /dev/null
+++ b/s2s_aws_cb1e7390.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/s2s/aws#param-output-channel-count
+Title: AWS Nova Sonic - Pipecat
+==================================================
+
+AWS Nova Sonic - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Speech AWS Nova Sonic Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech AWS Nova Sonic Gemini Multimodal Live OpenAI Realtime Beta Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline The AWSNovaSonicLLMService enables natural, real-time conversations with AWS Nova Sonic. It provides built-in audio transcription, voice activity detection, and context management for creating interactive AI experiences. It provides: Real-time Interaction Stream audio in real-time with low latency response times Speech Processing Built-in speech-to-text and text-to-speech capabilities with multiple voice options Voice Activity Detection Automatic detection of speech start/stop for natural conversations Context Management Intelligent handling of conversation history and system instructions ​ Installation To use AWSNovaSonicLLMService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[aws-nova-sonic]" We recommend setting up your AWS credentials as environment variables, as you’ll need them to initialize AWSNovaSonicLLMService : AWS_SECRET_ACCESS_KEY AWS_ACCESS_KEY_ID AWS_REGION ​ Basic Usage Here’s a simple example of setting up a conversational AI bot with AWS Nova Sonic: Copy Ask AI from pipecat.services.aws_nova_sonic.aws import AWSNovaSonicLLMService llm = AWSNovaSonicLLMService( secret_access_key = os.getenv( "AWS_SECRET_ACCESS_KEY" ), access_key_id = os.getenv( "AWS_ACCESS_KEY_ID" ), region = os.getenv( "AWS_REGION" ) voice_id = "tiffany" , # Voices: matthew, tiffany, amy ) ​ Configuration ​ Constructor Parameters ​ secret_access_key str required Your AWS secret access key ​ access_key_id str required Your AWS access key ID ​ region str required Specify the AWS region for the service (e.g., "us-east-1" ). Note that the service may not be available in all AWS regions: check the AWS Bedrock User Guide’s support table . ​ model str default: "amazon.nova-sonic-v1:0" AWS Nova Sonic model to use. Note that "amazon.nova-sonic-v1:0" is the only supported model as of 2025-05-08. ​ voice_id str default: "matthew" Voice for text-to-speech (options: "matthew" , "tiffany" , "amy" ) ​ params Params Configuration for model parameters ​ system_instruction str High-level instructions that guide the model’s behavior. Note that more commonly these instructions will be included as part of the context provided to kick off the conversation. ​ tools ToolsSchema List of function definitions for tool/function calling. Note that more commonly tools will be included as part of the context provided to kick off the conversation. ​ send_transcription_frames bool default: "True" Whether to emit transcription frames ​ Model Parameters The Params object configures the behavior of the AWS Nova Sonic model. It is strongly recommended to stick with default values (most easily by omitting params when constructing AWSNovaSonicLLMService ) unless you have a good understanding of the parameters and their impact. Deviating from the defaults may lead to unexpected behavior. ​ temperature float default: "0.7" Controls randomness in responses. Range: 0.0 to 2.0 ​ max_tokens int default: "1024" Maximum number of tokens to generate ​ top_p float default: "0.9" Cumulative probability cutoff for token selection. Range: 0.0 to 1.0 ​ input_sample_rate int default: "16000" Sample rate for input audio ​ output_sample_rate int default: "24000" Sample rate for output audio ​ input_sample_size int default: "16" Bit depth for input audio ​ input_channel_count int default: "1" Number of channels for input audio ​ output_sample_size int default: "16" Bit depth for output audio ​ output_channel_count int default: "1" Number of channels for output audio ​ Frame Types ​ Input Frames ​ InputAudioRawFrame Frame Raw audio data for speech input ​ OpenAILLMContextFrame Frame Contains conversation context ​ BotStoppedSpeakingFrame Frame Signals the bot has stopped speaking ​ Output Frames ​ TTSAudioRawFrame Frame Generated speech audio ​ LLMFullResponseStartFrame Frame Signals the start of a response from the LLM ​ LLMFullResponseEndFrame Frame Signals the end of a response from the LLM ​ TTSStartedFrame Frame Signals start of speech synthesis (coincides with the start of the LLM response, as this is a speech-to-speech model) ​ TTSStoppedFrame Frame Signals end of speech synthesis (coincides with the end of the LLM response, as this is a speech-to-speech model) ​ LLMTextFrame Frame Generated text responses from the LLM ​ TTSTextFrame Frame Generated text responses ​ TranscriptionFrame Frame Speech transcriptions. Only output if send_transcription_frames is True . ​ Function Calling This service supports function calling (also known as tool calling) which allows the LLM to request information from external services and APIs. For example, you can enable your bot to: Check current weather conditions Query databases Access external APIs Perform custom actions See the Function Calling guide for: Detailed implementation instructions Provider-specific function definitions Handler registration examples Control over function call behavior Complete usage examples ​ Next Steps ​ Examples Foundational Example Basic implementation showing core features Persistent Content Example Implementation showing saving and loading conversation history XTTS Gemini Multimodal Live On this page Installation Basic Usage Configuration Constructor Parameters Model Parameters Frame Types Input Frames Output Frames Function Calling Next Steps Examples Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/s2s_aws_cbf6802b.txt b/s2s_aws_cbf6802b.txt
new file mode 100644
index 0000000000000000000000000000000000000000..adc32dfe015f4ea9ced104221f39d97f413661f9
--- /dev/null
+++ b/s2s_aws_cbf6802b.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/s2s/aws#model-parameters
+Title: AWS Nova Sonic - Pipecat
+==================================================
+
+AWS Nova Sonic - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Speech AWS Nova Sonic Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech AWS Nova Sonic Gemini Multimodal Live OpenAI Realtime Beta Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline The AWSNovaSonicLLMService enables natural, real-time conversations with AWS Nova Sonic. It provides built-in audio transcription, voice activity detection, and context management for creating interactive AI experiences. It provides: Real-time Interaction Stream audio in real-time with low latency response times Speech Processing Built-in speech-to-text and text-to-speech capabilities with multiple voice options Voice Activity Detection Automatic detection of speech start/stop for natural conversations Context Management Intelligent handling of conversation history and system instructions ​ Installation To use AWSNovaSonicLLMService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[aws-nova-sonic]" We recommend setting up your AWS credentials as environment variables, as you’ll need them to initialize AWSNovaSonicLLMService : AWS_SECRET_ACCESS_KEY AWS_ACCESS_KEY_ID AWS_REGION ​ Basic Usage Here’s a simple example of setting up a conversational AI bot with AWS Nova Sonic: Copy Ask AI from pipecat.services.aws_nova_sonic.aws import AWSNovaSonicLLMService llm = AWSNovaSonicLLMService( secret_access_key = os.getenv( "AWS_SECRET_ACCESS_KEY" ), access_key_id = os.getenv( "AWS_ACCESS_KEY_ID" ), region = os.getenv( "AWS_REGION" ) voice_id = "tiffany" , # Voices: matthew, tiffany, amy ) ​ Configuration ​ Constructor Parameters ​ secret_access_key str required Your AWS secret access key ​ access_key_id str required Your AWS access key ID ​ region str required Specify the AWS region for the service (e.g., "us-east-1" ). Note that the service may not be available in all AWS regions: check the AWS Bedrock User Guide’s support table . ​ model str default: "amazon.nova-sonic-v1:0" AWS Nova Sonic model to use. Note that "amazon.nova-sonic-v1:0" is the only supported model as of 2025-05-08. ​ voice_id str default: "matthew" Voice for text-to-speech (options: "matthew" , "tiffany" , "amy" ) ​ params Params Configuration for model parameters ​ system_instruction str High-level instructions that guide the model’s behavior. Note that more commonly these instructions will be included as part of the context provided to kick off the conversation. ​ tools ToolsSchema List of function definitions for tool/function calling. Note that more commonly tools will be included as part of the context provided to kick off the conversation. ​ send_transcription_frames bool default: "True" Whether to emit transcription frames ​ Model Parameters The Params object configures the behavior of the AWS Nova Sonic model. It is strongly recommended to stick with default values (most easily by omitting params when constructing AWSNovaSonicLLMService ) unless you have a good understanding of the parameters and their impact. Deviating from the defaults may lead to unexpected behavior. ​ temperature float default: "0.7" Controls randomness in responses. Range: 0.0 to 2.0 ​ max_tokens int default: "1024" Maximum number of tokens to generate ​ top_p float default: "0.9" Cumulative probability cutoff for token selection. Range: 0.0 to 1.0 ​ input_sample_rate int default: "16000" Sample rate for input audio ​ output_sample_rate int default: "24000" Sample rate for output audio ​ input_sample_size int default: "16" Bit depth for input audio ​ input_channel_count int default: "1" Number of channels for input audio ​ output_sample_size int default: "16" Bit depth for output audio ​ output_channel_count int default: "1" Number of channels for output audio ​ Frame Types ​ Input Frames ​ InputAudioRawFrame Frame Raw audio data for speech input ​ OpenAILLMContextFrame Frame Contains conversation context ​ BotStoppedSpeakingFrame Frame Signals the bot has stopped speaking ​ Output Frames ​ TTSAudioRawFrame Frame Generated speech audio ​ LLMFullResponseStartFrame Frame Signals the start of a response from the LLM ​ LLMFullResponseEndFrame Frame Signals the end of a response from the LLM ​ TTSStartedFrame Frame Signals start of speech synthesis (coincides with the start of the LLM response, as this is a speech-to-speech model) ​ TTSStoppedFrame Frame Signals end of speech synthesis (coincides with the end of the LLM response, as this is a speech-to-speech model) ​ LLMTextFrame Frame Generated text responses from the LLM ​ TTSTextFrame Frame Generated text responses ​ TranscriptionFrame Frame Speech transcriptions. Only output if send_transcription_frames is True . ​ Function Calling This service supports function calling (also known as tool calling) which allows the LLM to request information from external services and APIs. For example, you can enable your bot to: Check current weather conditions Query databases Access external APIs Perform custom actions See the Function Calling guide for: Detailed implementation instructions Provider-specific function definitions Handler registration examples Control over function call behavior Complete usage examples ​ Next Steps ​ Examples Foundational Example Basic implementation showing core features Persistent Content Example Implementation showing saving and loading conversation history XTTS Gemini Multimodal Live On this page Installation Basic Usage Configuration Constructor Parameters Model Parameters Frame Types Input Frames Output Frames Function Calling Next Steps Examples Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/s2s_aws_ded5ad10.txt b/s2s_aws_ded5ad10.txt
new file mode 100644
index 0000000000000000000000000000000000000000..7107071d55a781b00dccb94b02d17da67568f6a9
--- /dev/null
+++ b/s2s_aws_ded5ad10.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/s2s/aws#param-llm-full-response-end-frame
+Title: AWS Nova Sonic - Pipecat
+==================================================
+
+AWS Nova Sonic - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Speech AWS Nova Sonic Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech AWS Nova Sonic Gemini Multimodal Live OpenAI Realtime Beta Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline The AWSNovaSonicLLMService enables natural, real-time conversations with AWS Nova Sonic. It provides built-in audio transcription, voice activity detection, and context management for creating interactive AI experiences. It provides: Real-time Interaction Stream audio in real-time with low latency response times Speech Processing Built-in speech-to-text and text-to-speech capabilities with multiple voice options Voice Activity Detection Automatic detection of speech start/stop for natural conversations Context Management Intelligent handling of conversation history and system instructions ​ Installation To use AWSNovaSonicLLMService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[aws-nova-sonic]" We recommend setting up your AWS credentials as environment variables, as you’ll need them to initialize AWSNovaSonicLLMService : AWS_SECRET_ACCESS_KEY AWS_ACCESS_KEY_ID AWS_REGION ​ Basic Usage Here’s a simple example of setting up a conversational AI bot with AWS Nova Sonic: Copy Ask AI from pipecat.services.aws_nova_sonic.aws import AWSNovaSonicLLMService llm = AWSNovaSonicLLMService( secret_access_key = os.getenv( "AWS_SECRET_ACCESS_KEY" ), access_key_id = os.getenv( "AWS_ACCESS_KEY_ID" ), region = os.getenv( "AWS_REGION" ) voice_id = "tiffany" , # Voices: matthew, tiffany, amy ) ​ Configuration ​ Constructor Parameters ​ secret_access_key str required Your AWS secret access key ​ access_key_id str required Your AWS access key ID ​ region str required Specify the AWS region for the service (e.g., "us-east-1" ). Note that the service may not be available in all AWS regions: check the AWS Bedrock User Guide’s support table . ​ model str default: "amazon.nova-sonic-v1:0" AWS Nova Sonic model to use. Note that "amazon.nova-sonic-v1:0" is the only supported model as of 2025-05-08. ​ voice_id str default: "matthew" Voice for text-to-speech (options: "matthew" , "tiffany" , "amy" ) ​ params Params Configuration for model parameters ​ system_instruction str High-level instructions that guide the model’s behavior. Note that more commonly these instructions will be included as part of the context provided to kick off the conversation. ​ tools ToolsSchema List of function definitions for tool/function calling. Note that more commonly tools will be included as part of the context provided to kick off the conversation. ​ send_transcription_frames bool default: "True" Whether to emit transcription frames ​ Model Parameters The Params object configures the behavior of the AWS Nova Sonic model. It is strongly recommended to stick with default values (most easily by omitting params when constructing AWSNovaSonicLLMService ) unless you have a good understanding of the parameters and their impact. Deviating from the defaults may lead to unexpected behavior. ​ temperature float default: "0.7" Controls randomness in responses. Range: 0.0 to 2.0 ​ max_tokens int default: "1024" Maximum number of tokens to generate ​ top_p float default: "0.9" Cumulative probability cutoff for token selection. Range: 0.0 to 1.0 ​ input_sample_rate int default: "16000" Sample rate for input audio ​ output_sample_rate int default: "24000" Sample rate for output audio ​ input_sample_size int default: "16" Bit depth for input audio ​ input_channel_count int default: "1" Number of channels for input audio ​ output_sample_size int default: "16" Bit depth for output audio ​ output_channel_count int default: "1" Number of channels for output audio ​ Frame Types ​ Input Frames ​ InputAudioRawFrame Frame Raw audio data for speech input ​ OpenAILLMContextFrame Frame Contains conversation context ​ BotStoppedSpeakingFrame Frame Signals the bot has stopped speaking ​ Output Frames ​ TTSAudioRawFrame Frame Generated speech audio ​ LLMFullResponseStartFrame Frame Signals the start of a response from the LLM ​ LLMFullResponseEndFrame Frame Signals the end of a response from the LLM ​ TTSStartedFrame Frame Signals start of speech synthesis (coincides with the start of the LLM response, as this is a speech-to-speech model) ​ TTSStoppedFrame Frame Signals end of speech synthesis (coincides with the end of the LLM response, as this is a speech-to-speech model) ​ LLMTextFrame Frame Generated text responses from the LLM ​ TTSTextFrame Frame Generated text responses ​ TranscriptionFrame Frame Speech transcriptions. Only output if send_transcription_frames is True . ​ Function Calling This service supports function calling (also known as tool calling) which allows the LLM to request information from external services and APIs. For example, you can enable your bot to: Check current weather conditions Query databases Access external APIs Perform custom actions See the Function Calling guide for: Detailed implementation instructions Provider-specific function definitions Handler registration examples Control over function call behavior Complete usage examples ​ Next Steps ​ Examples Foundational Example Basic implementation showing core features Persistent Content Example Implementation showing saving and loading conversation history XTTS Gemini Multimodal Live On this page Installation Basic Usage Configuration Constructor Parameters Model Parameters Frame Types Input Frames Output Frames Function Calling Next Steps Examples Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/s2s_aws_f4a11d50.txt b/s2s_aws_f4a11d50.txt
new file mode 100644
index 0000000000000000000000000000000000000000..bf6ad4ea88b3ce1dee522f816e85f5fbac09ebc8
--- /dev/null
+++ b/s2s_aws_f4a11d50.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/s2s/aws#param-tts-started-frame
+Title: AWS Nova Sonic - Pipecat
+==================================================
+
+AWS Nova Sonic - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Speech AWS Nova Sonic Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech AWS Nova Sonic Gemini Multimodal Live OpenAI Realtime Beta Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline The AWSNovaSonicLLMService enables natural, real-time conversations with AWS Nova Sonic. It provides built-in audio transcription, voice activity detection, and context management for creating interactive AI experiences. It provides: Real-time Interaction Stream audio in real-time with low latency response times Speech Processing Built-in speech-to-text and text-to-speech capabilities with multiple voice options Voice Activity Detection Automatic detection of speech start/stop for natural conversations Context Management Intelligent handling of conversation history and system instructions ​ Installation To use AWSNovaSonicLLMService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[aws-nova-sonic]" We recommend setting up your AWS credentials as environment variables, as you’ll need them to initialize AWSNovaSonicLLMService : AWS_SECRET_ACCESS_KEY AWS_ACCESS_KEY_ID AWS_REGION ​ Basic Usage Here’s a simple example of setting up a conversational AI bot with AWS Nova Sonic: Copy Ask AI from pipecat.services.aws_nova_sonic.aws import AWSNovaSonicLLMService llm = AWSNovaSonicLLMService( secret_access_key = os.getenv( "AWS_SECRET_ACCESS_KEY" ), access_key_id = os.getenv( "AWS_ACCESS_KEY_ID" ), region = os.getenv( "AWS_REGION" ) voice_id = "tiffany" , # Voices: matthew, tiffany, amy ) ​ Configuration ​ Constructor Parameters ​ secret_access_key str required Your AWS secret access key ​ access_key_id str required Your AWS access key ID ​ region str required Specify the AWS region for the service (e.g., "us-east-1" ). Note that the service may not be available in all AWS regions: check the AWS Bedrock User Guide’s support table . ​ model str default: "amazon.nova-sonic-v1:0" AWS Nova Sonic model to use. Note that "amazon.nova-sonic-v1:0" is the only supported model as of 2025-05-08. ​ voice_id str default: "matthew" Voice for text-to-speech (options: "matthew" , "tiffany" , "amy" ) ​ params Params Configuration for model parameters ​ system_instruction str High-level instructions that guide the model’s behavior. Note that more commonly these instructions will be included as part of the context provided to kick off the conversation. ​ tools ToolsSchema List of function definitions for tool/function calling. Note that more commonly tools will be included as part of the context provided to kick off the conversation. ​ send_transcription_frames bool default: "True" Whether to emit transcription frames ​ Model Parameters The Params object configures the behavior of the AWS Nova Sonic model. It is strongly recommended to stick with default values (most easily by omitting params when constructing AWSNovaSonicLLMService ) unless you have a good understanding of the parameters and their impact. Deviating from the defaults may lead to unexpected behavior. ​ temperature float default: "0.7" Controls randomness in responses. Range: 0.0 to 2.0 ​ max_tokens int default: "1024" Maximum number of tokens to generate ​ top_p float default: "0.9" Cumulative probability cutoff for token selection. Range: 0.0 to 1.0 ​ input_sample_rate int default: "16000" Sample rate for input audio ​ output_sample_rate int default: "24000" Sample rate for output audio ​ input_sample_size int default: "16" Bit depth for input audio ​ input_channel_count int default: "1" Number of channels for input audio ​ output_sample_size int default: "16" Bit depth for output audio ​ output_channel_count int default: "1" Number of channels for output audio ​ Frame Types ​ Input Frames ​ InputAudioRawFrame Frame Raw audio data for speech input ​ OpenAILLMContextFrame Frame Contains conversation context ​ BotStoppedSpeakingFrame Frame Signals the bot has stopped speaking ​ Output Frames ​ TTSAudioRawFrame Frame Generated speech audio ​ LLMFullResponseStartFrame Frame Signals the start of a response from the LLM ​ LLMFullResponseEndFrame Frame Signals the end of a response from the LLM ​ TTSStartedFrame Frame Signals start of speech synthesis (coincides with the start of the LLM response, as this is a speech-to-speech model) ​ TTSStoppedFrame Frame Signals end of speech synthesis (coincides with the end of the LLM response, as this is a speech-to-speech model) ​ LLMTextFrame Frame Generated text responses from the LLM ​ TTSTextFrame Frame Generated text responses ​ TranscriptionFrame Frame Speech transcriptions. Only output if send_transcription_frames is True . ​ Function Calling This service supports function calling (also known as tool calling) which allows the LLM to request information from external services and APIs. For example, you can enable your bot to: Check current weather conditions Query databases Access external APIs Perform custom actions See the Function Calling guide for: Detailed implementation instructions Provider-specific function definitions Handler registration examples Control over function call behavior Complete usage examples ​ Next Steps ​ Examples Foundational Example Basic implementation showing core features Persistent Content Example Implementation showing saving and loading conversation history XTTS Gemini Multimodal Live On this page Installation Basic Usage Configuration Constructor Parameters Model Parameters Frame Types Input Frames Output Frames Function Calling Next Steps Examples Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/s2s_aws_f4c1722d.txt b/s2s_aws_f4c1722d.txt
new file mode 100644
index 0000000000000000000000000000000000000000..eb065248cbfe6216a484e6034509043b994fa00a
--- /dev/null
+++ b/s2s_aws_f4c1722d.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/s2s/aws#param-input-sample-rate
+Title: AWS Nova Sonic - Pipecat
+==================================================
+
+AWS Nova Sonic - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Speech AWS Nova Sonic Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech AWS Nova Sonic Gemini Multimodal Live OpenAI Realtime Beta Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline The AWSNovaSonicLLMService enables natural, real-time conversations with AWS Nova Sonic. It provides built-in audio transcription, voice activity detection, and context management for creating interactive AI experiences. It provides: Real-time Interaction Stream audio in real-time with low latency response times Speech Processing Built-in speech-to-text and text-to-speech capabilities with multiple voice options Voice Activity Detection Automatic detection of speech start/stop for natural conversations Context Management Intelligent handling of conversation history and system instructions ​ Installation To use AWSNovaSonicLLMService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[aws-nova-sonic]" We recommend setting up your AWS credentials as environment variables, as you’ll need them to initialize AWSNovaSonicLLMService : AWS_SECRET_ACCESS_KEY AWS_ACCESS_KEY_ID AWS_REGION ​ Basic Usage Here’s a simple example of setting up a conversational AI bot with AWS Nova Sonic: Copy Ask AI from pipecat.services.aws_nova_sonic.aws import AWSNovaSonicLLMService llm = AWSNovaSonicLLMService( secret_access_key = os.getenv( "AWS_SECRET_ACCESS_KEY" ), access_key_id = os.getenv( "AWS_ACCESS_KEY_ID" ), region = os.getenv( "AWS_REGION" ) voice_id = "tiffany" , # Voices: matthew, tiffany, amy ) ​ Configuration ​ Constructor Parameters ​ secret_access_key str required Your AWS secret access key ​ access_key_id str required Your AWS access key ID ​ region str required Specify the AWS region for the service (e.g., "us-east-1" ). Note that the service may not be available in all AWS regions: check the AWS Bedrock User Guide’s support table . ​ model str default: "amazon.nova-sonic-v1:0" AWS Nova Sonic model to use. Note that "amazon.nova-sonic-v1:0" is the only supported model as of 2025-05-08. ​ voice_id str default: "matthew" Voice for text-to-speech (options: "matthew" , "tiffany" , "amy" ) ​ params Params Configuration for model parameters ​ system_instruction str High-level instructions that guide the model’s behavior. Note that more commonly these instructions will be included as part of the context provided to kick off the conversation. ​ tools ToolsSchema List of function definitions for tool/function calling. Note that more commonly tools will be included as part of the context provided to kick off the conversation. ​ send_transcription_frames bool default: "True" Whether to emit transcription frames ​ Model Parameters The Params object configures the behavior of the AWS Nova Sonic model. It is strongly recommended to stick with default values (most easily by omitting params when constructing AWSNovaSonicLLMService ) unless you have a good understanding of the parameters and their impact. Deviating from the defaults may lead to unexpected behavior. ​ temperature float default: "0.7" Controls randomness in responses. Range: 0.0 to 2.0 ​ max_tokens int default: "1024" Maximum number of tokens to generate ​ top_p float default: "0.9" Cumulative probability cutoff for token selection. Range: 0.0 to 1.0 ​ input_sample_rate int default: "16000" Sample rate for input audio ​ output_sample_rate int default: "24000" Sample rate for output audio ​ input_sample_size int default: "16" Bit depth for input audio ​ input_channel_count int default: "1" Number of channels for input audio ​ output_sample_size int default: "16" Bit depth for output audio ​ output_channel_count int default: "1" Number of channels for output audio ​ Frame Types ​ Input Frames ​ InputAudioRawFrame Frame Raw audio data for speech input ​ OpenAILLMContextFrame Frame Contains conversation context ​ BotStoppedSpeakingFrame Frame Signals the bot has stopped speaking ​ Output Frames ​ TTSAudioRawFrame Frame Generated speech audio ​ LLMFullResponseStartFrame Frame Signals the start of a response from the LLM ​ LLMFullResponseEndFrame Frame Signals the end of a response from the LLM ​ TTSStartedFrame Frame Signals start of speech synthesis (coincides with the start of the LLM response, as this is a speech-to-speech model) ​ TTSStoppedFrame Frame Signals end of speech synthesis (coincides with the end of the LLM response, as this is a speech-to-speech model) ​ LLMTextFrame Frame Generated text responses from the LLM ​ TTSTextFrame Frame Generated text responses ​ TranscriptionFrame Frame Speech transcriptions. Only output if send_transcription_frames is True . ​ Function Calling This service supports function calling (also known as tool calling) which allows the LLM to request information from external services and APIs. For example, you can enable your bot to: Check current weather conditions Query databases Access external APIs Perform custom actions See the Function Calling guide for: Detailed implementation instructions Provider-specific function definitions Handler registration examples Control over function call behavior Complete usage examples ​ Next Steps ​ Examples Foundational Example Basic implementation showing core features Persistent Content Example Implementation showing saving and loading conversation history XTTS Gemini Multimodal Live On this page Installation Basic Usage Configuration Constructor Parameters Model Parameters Frame Types Input Frames Output Frames Function Calling Next Steps Examples Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/s2s_aws_f59ce296.txt b/s2s_aws_f59ce296.txt
new file mode 100644
index 0000000000000000000000000000000000000000..e2e02e35f99fd27c7473cb60c4975d657e40a21c
--- /dev/null
+++ b/s2s_aws_f59ce296.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/s2s/aws#output-frames
+Title: AWS Nova Sonic - Pipecat
+==================================================
+
+AWS Nova Sonic - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Speech AWS Nova Sonic Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech AWS Nova Sonic Gemini Multimodal Live OpenAI Realtime Beta Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline The AWSNovaSonicLLMService enables natural, real-time conversations with AWS Nova Sonic. It provides built-in audio transcription, voice activity detection, and context management for creating interactive AI experiences. It provides: Real-time Interaction Stream audio in real-time with low latency response times Speech Processing Built-in speech-to-text and text-to-speech capabilities with multiple voice options Voice Activity Detection Automatic detection of speech start/stop for natural conversations Context Management Intelligent handling of conversation history and system instructions ​ Installation To use AWSNovaSonicLLMService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[aws-nova-sonic]" We recommend setting up your AWS credentials as environment variables, as you’ll need them to initialize AWSNovaSonicLLMService : AWS_SECRET_ACCESS_KEY AWS_ACCESS_KEY_ID AWS_REGION ​ Basic Usage Here’s a simple example of setting up a conversational AI bot with AWS Nova Sonic: Copy Ask AI from pipecat.services.aws_nova_sonic.aws import AWSNovaSonicLLMService llm = AWSNovaSonicLLMService( secret_access_key = os.getenv( "AWS_SECRET_ACCESS_KEY" ), access_key_id = os.getenv( "AWS_ACCESS_KEY_ID" ), region = os.getenv( "AWS_REGION" ) voice_id = "tiffany" , # Voices: matthew, tiffany, amy ) ​ Configuration ​ Constructor Parameters ​ secret_access_key str required Your AWS secret access key ​ access_key_id str required Your AWS access key ID ​ region str required Specify the AWS region for the service (e.g., "us-east-1" ). Note that the service may not be available in all AWS regions: check the AWS Bedrock User Guide’s support table . ​ model str default: "amazon.nova-sonic-v1:0" AWS Nova Sonic model to use. Note that "amazon.nova-sonic-v1:0" is the only supported model as of 2025-05-08. ​ voice_id str default: "matthew" Voice for text-to-speech (options: "matthew" , "tiffany" , "amy" ) ​ params Params Configuration for model parameters ​ system_instruction str High-level instructions that guide the model’s behavior. Note that more commonly these instructions will be included as part of the context provided to kick off the conversation. ​ tools ToolsSchema List of function definitions for tool/function calling. Note that more commonly tools will be included as part of the context provided to kick off the conversation. ​ send_transcription_frames bool default: "True" Whether to emit transcription frames ​ Model Parameters The Params object configures the behavior of the AWS Nova Sonic model. It is strongly recommended to stick with default values (most easily by omitting params when constructing AWSNovaSonicLLMService ) unless you have a good understanding of the parameters and their impact. Deviating from the defaults may lead to unexpected behavior. ​ temperature float default: "0.7" Controls randomness in responses. Range: 0.0 to 2.0 ​ max_tokens int default: "1024" Maximum number of tokens to generate ​ top_p float default: "0.9" Cumulative probability cutoff for token selection. Range: 0.0 to 1.0 ​ input_sample_rate int default: "16000" Sample rate for input audio ​ output_sample_rate int default: "24000" Sample rate for output audio ​ input_sample_size int default: "16" Bit depth for input audio ​ input_channel_count int default: "1" Number of channels for input audio ​ output_sample_size int default: "16" Bit depth for output audio ​ output_channel_count int default: "1" Number of channels for output audio ​ Frame Types ​ Input Frames ​ InputAudioRawFrame Frame Raw audio data for speech input ​ OpenAILLMContextFrame Frame Contains conversation context ​ BotStoppedSpeakingFrame Frame Signals the bot has stopped speaking ​ Output Frames ​ TTSAudioRawFrame Frame Generated speech audio ​ LLMFullResponseStartFrame Frame Signals the start of a response from the LLM ​ LLMFullResponseEndFrame Frame Signals the end of a response from the LLM ​ TTSStartedFrame Frame Signals start of speech synthesis (coincides with the start of the LLM response, as this is a speech-to-speech model) ​ TTSStoppedFrame Frame Signals end of speech synthesis (coincides with the end of the LLM response, as this is a speech-to-speech model) ​ LLMTextFrame Frame Generated text responses from the LLM ​ TTSTextFrame Frame Generated text responses ​ TranscriptionFrame Frame Speech transcriptions. Only output if send_transcription_frames is True . ​ Function Calling This service supports function calling (also known as tool calling) which allows the LLM to request information from external services and APIs. For example, you can enable your bot to: Check current weather conditions Query databases Access external APIs Perform custom actions See the Function Calling guide for: Detailed implementation instructions Provider-specific function definitions Handler registration examples Control over function call behavior Complete usage examples ​ Next Steps ​ Examples Foundational Example Basic implementation showing core features Persistent Content Example Implementation showing saving and loading conversation history XTTS Gemini Multimodal Live On this page Installation Basic Usage Configuration Constructor Parameters Model Parameters Frame Types Input Frames Output Frames Function Calling Next Steps Examples Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/s2s_gemini_0e1a2465.txt b/s2s_gemini_0e1a2465.txt
new file mode 100644
index 0000000000000000000000000000000000000000..a10387a4717fe50a902c8a9f1a1676b3289110bc
--- /dev/null
+++ b/s2s_gemini_0e1a2465.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/s2s/gemini#param-base-url
+Title: Gemini Multimodal Live - Pipecat
+==================================================
+
+Gemini Multimodal Live - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Speech Gemini Multimodal Live Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech AWS Nova Sonic Gemini Multimodal Live OpenAI Realtime Beta Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline The GeminiMultimodalLiveLLMService enables natural, real-time conversations with Google’s Gemini model. It provides built-in audio transcription, voice activity detection, and context management for creating interactive AI experiences. It provides: Real-time Interaction Stream audio and video in real-time with low latency response times Speech Processing Built-in speech-to-text and text-to-speech capabilities with multiple voice options Voice Activity Detection Automatic detection of speech start/stop for natural conversations Context Management Intelligent handling of conversation history and system instructions Want to start building? Check out our Gemini Multimodal Live Guide . ​ Installation To use GeminiMultimodalLiveLLMService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[google]" You’ll need to set up your Google API key as an environment variable: GOOGLE_API_KEY . ​ Basic Usage Here’s a simple example of setting up a conversational AI bot with Gemini Multimodal Live: Copy Ask AI from pipecat.services.gemini_multimodal_live.gemini import ( GeminiMultimodalLiveLLMService, InputParams, GeminiMultimodalModalities ) llm = GeminiMultimodalLiveLLMService( api_key = os.getenv( "GOOGLE_API_KEY" ), voice_id = "Aoede" , # Voices: Aoede, Charon, Fenrir, Kore, Puck params = InputParams( temperature = 0.7 , # Set model input params language = Language. EN_US , # Set language (30+ languages supported) modalities = GeminiMultimodalModalities. AUDIO # Response modality ) ) ​ Configuration ​ Constructor Parameters ​ api_key str required Your Google API key ​ base_url str API endpoint URL ​ model str Gemini model to use (upgraded to new v1beta model) ​ voice_id str default: "Charon" Voice for text-to-speech (options: Aoede, Charon, Fenrir, Kore, Puck) Copy Ask AI llm = GeminiMultimodalLiveLLMService( api_key = os.getenv( "GOOGLE_API_KEY" ), voice_id = "Puck" , # Choose your preferred voice ) ​ system_instruction str High-level instructions that guide the model’s behavior Copy Ask AI llm = GeminiMultimodalLiveLLMService( api_key = os.getenv( "GOOGLE_API_KEY" ), system_instruction = "Talk like a pirate." , ) ​ start_audio_paused bool default: "False" Whether to start with audio input paused ​ start_video_paused bool default: "False" Whether to start with video input paused ​ tools Union[List[dict], ToolsSchema] Tools/functions available to the model ​ inference_on_context_initialization bool default: "True" Whether to generate a response when context is first set ​ Input Parameters ​ frequency_penalty float default: "None" Penalizes repeated tokens. Range: 0.0 to 2.0 ​ max_tokens int default: "4096" Maximum number of tokens to generate ​ modalities GeminiMultimodalModalities default: "AUDIO" Response modalities to include (options: AUDIO , TEXT ). ​ presence_penalty float default: "None" Penalizes tokens based on their presence in the text. Range: 0.0 to 2.0 ​ temperature float default: "None" Controls randomness in responses. Range: 0.0 to 2.0 ​ language Language default: "Language.EN_US" Language for generation. Over 30 languages are supported. ​ media_resolution GeminiMediaResolution default: "UNSPECIFIED" Controls image processing quality and token usage: LOW : Uses 64 tokens MEDIUM : Uses 256 tokens HIGH : Zoomed reframing with 256 tokens ​ vad GeminiVADParams Voice Activity Detection configuration: disabled : Toggle VAD on/off start_sensitivity : How quickly speech is detected (HIGH/LOW) end_sensitivity : How quickly turns end after pauses (HIGH/LOW) prefix_padding_ms : Milliseconds of audio to keep before speech silence_duration_ms : Milliseconds of silence to end a turn Copy Ask AI from pipecat.services.gemini_multimodal_live.events import ( StartSensitivity, EndSensitivity ) from pipecat.services.gemini_multimodal_live.gemini import ( GeminiVADParams, GeminiMediaResolution, ) llm = GeminiMultimodalLiveLLMService( api_key = os.getenv( "GOOGLE_API_KEY" ), params = InputParams( temperature = 0.7 , language = Language. ES , # Spanish language media_resolution = GeminiMediaResolution. HIGH , # Higher quality image processing vad = GeminiVADParams( start_sensitivity = StartSensitivity. HIGH , # Detect speech quickly end_sensitivity = EndSensitivity. LOW , # Allow longer pauses prefix_padding_ms = 300 , # Keep 300ms before speech silence_duration_ms = 1000 , # End turn after 1s silence ) ) ) ​ top_k int default: "None" Limits vocabulary to k most likely tokens. Minimum: 0 ​ top_p float default: "None" Cumulative probability cutoff for token selection. Range: 0.0 to 1.0 ​ context_window_compression ContextWindowCompressionParams Parameters for managing the context window: - enabled : Enable/disable compression (default: False) - trigger_tokens : Number of tokens that trigger compression (default: None, uses 80% of context window) Copy Ask AI from pipecat.services.gemini_multimodal_live.gemini import ( ContextWindowCompressionParams ) llm = GeminiMultimodalLiveLLMService( api_key = os.getenv( "GOOGLE_API_KEY" ), params = InputParams( top_p = 0.9 , # More focused token selection top_k = 40 , # Limit vocabulary options context_window_compression = ContextWindowCompressionParams( enabled = True , trigger_tokens = 8000 # Compress when reaching 8000 tokens ) ) ) ​ Methods ​ set_audio_input_paused(paused: bool) method Pause or unpause audio input processing ​ set_video_input_paused(paused: bool) method Pause or unpause video input processing ​ set_model_modalities(modalities: GeminiMultimodalModalities) method Change the response modality (TEXT or AUDIO) ​ set_language(language: Language) method Change the language for generation ​ set_context(context: OpenAILLMContext) method Set the conversation context explicitly ​ create_context_aggregator(context: OpenAILLMContext, user_params: LLMUserAggregatorParams, assistant_params: LLMAssistantAggregatorParams) method Create context aggregators for managing conversation state ​ Frame Types ​ Input Frames ​ InputAudioRawFrame Frame Raw audio data for speech input ​ InputImageRawFrame Frame Raw image data for visual input ​ StartInterruptionFrame Frame Signals start of user interruption ​ UserStartedSpeakingFrame Frame Signals user started speaking ​ UserStoppedSpeakingFrame Frame Signals user stopped speaking ​ OpenAILLMContextFrame Frame Contains conversation context ​ LLMMessagesAppendFrame Frame Adds messages to the conversation ​ LLMUpdateSettingsFrame Frame Updates LLM settings ​ LLMSetToolsFrame Frame Sets available tools for the LLM ​ Output Frames ​ TTSAudioRawFrame Frame Generated speech audio ​ TTSStartedFrame Frame Signals start of speech synthesis ​ TTSStoppedFrame Frame Signals end of speech synthesis ​ LLMTextFrame Frame Generated text responses from the LLM ​ TTSTextFrame Frame Text used for speech synthesis ​ TranscriptionFrame Frame Speech transcriptions from user audio ​ LLMFullResponseStartFrame Frame Signals the start of a complete LLM response ​ LLMFullResponseEndFrame Frame Signals the end of a complete LLM response ​ Function Calling This service supports function calling (also known as tool calling) which allows the LLM to request information from external services and APIs. For example, you can enable your bot to: Check current weather conditions Query databases Access external APIs Perform custom actions See the Function Calling guide for: Detailed implementation instructions Provider-specific function definitions Handler registration examples Control over function call behavior Complete usage examples ​ Token Usage Tracking Gemini Multimodal Live automatically tracks token usage metrics, providing: Prompt token counts Completion token counts Total token counts Detailed token breakdowns by modality (text, audio) These metrics can be used for monitoring usage, optimizing costs, and understanding model performance. ​ Language Support Gemini Multimodal Live supports the following languages: Language Code Description Gemini Code Language.AR Arabic ar-XA Language.BN_IN Bengali (India) bn-IN Language.CMN_CN Chinese (Mandarin) cmn-CN Language.DE_DE German (Germany) de-DE Language.EN_US English (US) en-US Language.EN_AU English (Australia) en-AU Language.EN_GB English (UK) en-GB Language.EN_IN English (India) en-IN Language.ES_ES Spanish (Spain) es-ES Language.ES_US Spanish (US) es-US Language.FR_FR French (France) fr-FR Language.FR_CA French (Canada) fr-CA Language.GU_IN Gujarati (India) gu-IN Language.HI_IN Hindi (India) hi-IN Language.ID_ID Indonesian id-ID Language.IT_IT Italian (Italy) it-IT Language.JA_JP Japanese (Japan) ja-JP Language.KN_IN Kannada (India) kn-IN Language.KO_KR Korean (Korea) ko-KR Language.ML_IN Malayalam (India) ml-IN Language.MR_IN Marathi (India) mr-IN Language.NL_NL Dutch (Netherlands) nl-NL Language.PL_PL Polish (Poland) pl-PL Language.PT_BR Portuguese (Brazil) pt-BR Language.RU_RU Russian (Russia) ru-RU Language.TA_IN Tamil (India) ta-IN Language.TE_IN Telugu (India) te-IN Language.TH_TH Thai (Thailand) th-TH Language.TR_TR Turkish (Turkey) tr-TR Language.VI_VN Vietnamese (Vietnam) vi-VN You can set the language using the language parameter: Copy Ask AI from pipecat.transcriptions.language import Language from pipecat.services.gemini_multimodal_live.gemini import ( GeminiMultimodalLiveLLMService, InputParams ) # Set language during initialization llm = GeminiMultimodalLiveLLMService( api_key = os.getenv( "GOOGLE_API_KEY" ), params = InputParams( language = Language. ES_ES ) # Spanish (Spain) ) ​ Next Steps ​ Examples Foundational Example Basic implementation showing core features and transcription Simple Chatbot A client/server example showing how to build a Pipecat JS or React client that connects to a Gemini Live Pipecat bot. ​ Learn More Check out our Gemini Multimodal Live Guide for detailed explanations and best practices. AWS Nova Sonic OpenAI Realtime Beta On this page Installation Basic Usage Configuration Constructor Parameters Input Parameters Methods Frame Types Input Frames Output Frames Function Calling Token Usage Tracking Language Support Next Steps Examples Learn More Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/s2s_gemini_14ef0843.txt b/s2s_gemini_14ef0843.txt
new file mode 100644
index 0000000000000000000000000000000000000000..7f1e8bb88879c424867abcd18d58e9541d3ec339
--- /dev/null
+++ b/s2s_gemini_14ef0843.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/s2s/gemini#param-api-key
+Title: Gemini Multimodal Live - Pipecat
+==================================================
+
+Gemini Multimodal Live - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Speech Gemini Multimodal Live Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech AWS Nova Sonic Gemini Multimodal Live OpenAI Realtime Beta Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline The GeminiMultimodalLiveLLMService enables natural, real-time conversations with Google’s Gemini model. It provides built-in audio transcription, voice activity detection, and context management for creating interactive AI experiences. It provides: Real-time Interaction Stream audio and video in real-time with low latency response times Speech Processing Built-in speech-to-text and text-to-speech capabilities with multiple voice options Voice Activity Detection Automatic detection of speech start/stop for natural conversations Context Management Intelligent handling of conversation history and system instructions Want to start building? Check out our Gemini Multimodal Live Guide . ​ Installation To use GeminiMultimodalLiveLLMService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[google]" You’ll need to set up your Google API key as an environment variable: GOOGLE_API_KEY . ​ Basic Usage Here’s a simple example of setting up a conversational AI bot with Gemini Multimodal Live: Copy Ask AI from pipecat.services.gemini_multimodal_live.gemini import ( GeminiMultimodalLiveLLMService, InputParams, GeminiMultimodalModalities ) llm = GeminiMultimodalLiveLLMService( api_key = os.getenv( "GOOGLE_API_KEY" ), voice_id = "Aoede" , # Voices: Aoede, Charon, Fenrir, Kore, Puck params = InputParams( temperature = 0.7 , # Set model input params language = Language. EN_US , # Set language (30+ languages supported) modalities = GeminiMultimodalModalities. AUDIO # Response modality ) ) ​ Configuration ​ Constructor Parameters ​ api_key str required Your Google API key ​ base_url str API endpoint URL ​ model str Gemini model to use (upgraded to new v1beta model) ​ voice_id str default: "Charon" Voice for text-to-speech (options: Aoede, Charon, Fenrir, Kore, Puck) Copy Ask AI llm = GeminiMultimodalLiveLLMService( api_key = os.getenv( "GOOGLE_API_KEY" ), voice_id = "Puck" , # Choose your preferred voice ) ​ system_instruction str High-level instructions that guide the model’s behavior Copy Ask AI llm = GeminiMultimodalLiveLLMService( api_key = os.getenv( "GOOGLE_API_KEY" ), system_instruction = "Talk like a pirate." , ) ​ start_audio_paused bool default: "False" Whether to start with audio input paused ​ start_video_paused bool default: "False" Whether to start with video input paused ​ tools Union[List[dict], ToolsSchema] Tools/functions available to the model ​ inference_on_context_initialization bool default: "True" Whether to generate a response when context is first set ​ Input Parameters ​ frequency_penalty float default: "None" Penalizes repeated tokens. Range: 0.0 to 2.0 ​ max_tokens int default: "4096" Maximum number of tokens to generate ​ modalities GeminiMultimodalModalities default: "AUDIO" Response modalities to include (options: AUDIO , TEXT ). ​ presence_penalty float default: "None" Penalizes tokens based on their presence in the text. Range: 0.0 to 2.0 ​ temperature float default: "None" Controls randomness in responses. Range: 0.0 to 2.0 ​ language Language default: "Language.EN_US" Language for generation. Over 30 languages are supported. ​ media_resolution GeminiMediaResolution default: "UNSPECIFIED" Controls image processing quality and token usage: LOW : Uses 64 tokens MEDIUM : Uses 256 tokens HIGH : Zoomed reframing with 256 tokens ​ vad GeminiVADParams Voice Activity Detection configuration: disabled : Toggle VAD on/off start_sensitivity : How quickly speech is detected (HIGH/LOW) end_sensitivity : How quickly turns end after pauses (HIGH/LOW) prefix_padding_ms : Milliseconds of audio to keep before speech silence_duration_ms : Milliseconds of silence to end a turn Copy Ask AI from pipecat.services.gemini_multimodal_live.events import ( StartSensitivity, EndSensitivity ) from pipecat.services.gemini_multimodal_live.gemini import ( GeminiVADParams, GeminiMediaResolution, ) llm = GeminiMultimodalLiveLLMService( api_key = os.getenv( "GOOGLE_API_KEY" ), params = InputParams( temperature = 0.7 , language = Language. ES , # Spanish language media_resolution = GeminiMediaResolution. HIGH , # Higher quality image processing vad = GeminiVADParams( start_sensitivity = StartSensitivity. HIGH , # Detect speech quickly end_sensitivity = EndSensitivity. LOW , # Allow longer pauses prefix_padding_ms = 300 , # Keep 300ms before speech silence_duration_ms = 1000 , # End turn after 1s silence ) ) ) ​ top_k int default: "None" Limits vocabulary to k most likely tokens. Minimum: 0 ​ top_p float default: "None" Cumulative probability cutoff for token selection. Range: 0.0 to 1.0 ​ context_window_compression ContextWindowCompressionParams Parameters for managing the context window: - enabled : Enable/disable compression (default: False) - trigger_tokens : Number of tokens that trigger compression (default: None, uses 80% of context window) Copy Ask AI from pipecat.services.gemini_multimodal_live.gemini import ( ContextWindowCompressionParams ) llm = GeminiMultimodalLiveLLMService( api_key = os.getenv( "GOOGLE_API_KEY" ), params = InputParams( top_p = 0.9 , # More focused token selection top_k = 40 , # Limit vocabulary options context_window_compression = ContextWindowCompressionParams( enabled = True , trigger_tokens = 8000 # Compress when reaching 8000 tokens ) ) ) ​ Methods ​ set_audio_input_paused(paused: bool) method Pause or unpause audio input processing ​ set_video_input_paused(paused: bool) method Pause or unpause video input processing ​ set_model_modalities(modalities: GeminiMultimodalModalities) method Change the response modality (TEXT or AUDIO) ​ set_language(language: Language) method Change the language for generation ​ set_context(context: OpenAILLMContext) method Set the conversation context explicitly ​ create_context_aggregator(context: OpenAILLMContext, user_params: LLMUserAggregatorParams, assistant_params: LLMAssistantAggregatorParams) method Create context aggregators for managing conversation state ​ Frame Types ​ Input Frames ​ InputAudioRawFrame Frame Raw audio data for speech input ​ InputImageRawFrame Frame Raw image data for visual input ​ StartInterruptionFrame Frame Signals start of user interruption ​ UserStartedSpeakingFrame Frame Signals user started speaking ​ UserStoppedSpeakingFrame Frame Signals user stopped speaking ​ OpenAILLMContextFrame Frame Contains conversation context ​ LLMMessagesAppendFrame Frame Adds messages to the conversation ​ LLMUpdateSettingsFrame Frame Updates LLM settings ​ LLMSetToolsFrame Frame Sets available tools for the LLM ​ Output Frames ​ TTSAudioRawFrame Frame Generated speech audio ​ TTSStartedFrame Frame Signals start of speech synthesis ​ TTSStoppedFrame Frame Signals end of speech synthesis ​ LLMTextFrame Frame Generated text responses from the LLM ​ TTSTextFrame Frame Text used for speech synthesis ​ TranscriptionFrame Frame Speech transcriptions from user audio ​ LLMFullResponseStartFrame Frame Signals the start of a complete LLM response ​ LLMFullResponseEndFrame Frame Signals the end of a complete LLM response ​ Function Calling This service supports function calling (also known as tool calling) which allows the LLM to request information from external services and APIs. For example, you can enable your bot to: Check current weather conditions Query databases Access external APIs Perform custom actions See the Function Calling guide for: Detailed implementation instructions Provider-specific function definitions Handler registration examples Control over function call behavior Complete usage examples ​ Token Usage Tracking Gemini Multimodal Live automatically tracks token usage metrics, providing: Prompt token counts Completion token counts Total token counts Detailed token breakdowns by modality (text, audio) These metrics can be used for monitoring usage, optimizing costs, and understanding model performance. ​ Language Support Gemini Multimodal Live supports the following languages: Language Code Description Gemini Code Language.AR Arabic ar-XA Language.BN_IN Bengali (India) bn-IN Language.CMN_CN Chinese (Mandarin) cmn-CN Language.DE_DE German (Germany) de-DE Language.EN_US English (US) en-US Language.EN_AU English (Australia) en-AU Language.EN_GB English (UK) en-GB Language.EN_IN English (India) en-IN Language.ES_ES Spanish (Spain) es-ES Language.ES_US Spanish (US) es-US Language.FR_FR French (France) fr-FR Language.FR_CA French (Canada) fr-CA Language.GU_IN Gujarati (India) gu-IN Language.HI_IN Hindi (India) hi-IN Language.ID_ID Indonesian id-ID Language.IT_IT Italian (Italy) it-IT Language.JA_JP Japanese (Japan) ja-JP Language.KN_IN Kannada (India) kn-IN Language.KO_KR Korean (Korea) ko-KR Language.ML_IN Malayalam (India) ml-IN Language.MR_IN Marathi (India) mr-IN Language.NL_NL Dutch (Netherlands) nl-NL Language.PL_PL Polish (Poland) pl-PL Language.PT_BR Portuguese (Brazil) pt-BR Language.RU_RU Russian (Russia) ru-RU Language.TA_IN Tamil (India) ta-IN Language.TE_IN Telugu (India) te-IN Language.TH_TH Thai (Thailand) th-TH Language.TR_TR Turkish (Turkey) tr-TR Language.VI_VN Vietnamese (Vietnam) vi-VN You can set the language using the language parameter: Copy Ask AI from pipecat.transcriptions.language import Language from pipecat.services.gemini_multimodal_live.gemini import ( GeminiMultimodalLiveLLMService, InputParams ) # Set language during initialization llm = GeminiMultimodalLiveLLMService( api_key = os.getenv( "GOOGLE_API_KEY" ), params = InputParams( language = Language. ES_ES ) # Spanish (Spain) ) ​ Next Steps ​ Examples Foundational Example Basic implementation showing core features and transcription Simple Chatbot A client/server example showing how to build a Pipecat JS or React client that connects to a Gemini Live Pipecat bot. ​ Learn More Check out our Gemini Multimodal Live Guide for detailed explanations and best practices. AWS Nova Sonic OpenAI Realtime Beta On this page Installation Basic Usage Configuration Constructor Parameters Input Parameters Methods Frame Types Input Frames Output Frames Function Calling Token Usage Tracking Language Support Next Steps Examples Learn More Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/s2s_gemini_15b357ac.txt b/s2s_gemini_15b357ac.txt
new file mode 100644
index 0000000000000000000000000000000000000000..b7d3db246267a5bc7957e7d004c3bb3986a484ea
--- /dev/null
+++ b/s2s_gemini_15b357ac.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/s2s/gemini#param-model
+Title: Gemini Multimodal Live - Pipecat
+==================================================
+
+Gemini Multimodal Live - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Speech Gemini Multimodal Live Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech AWS Nova Sonic Gemini Multimodal Live OpenAI Realtime Beta Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline The GeminiMultimodalLiveLLMService enables natural, real-time conversations with Google’s Gemini model. It provides built-in audio transcription, voice activity detection, and context management for creating interactive AI experiences. It provides: Real-time Interaction Stream audio and video in real-time with low latency response times Speech Processing Built-in speech-to-text and text-to-speech capabilities with multiple voice options Voice Activity Detection Automatic detection of speech start/stop for natural conversations Context Management Intelligent handling of conversation history and system instructions Want to start building? Check out our Gemini Multimodal Live Guide . ​ Installation To use GeminiMultimodalLiveLLMService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[google]" You’ll need to set up your Google API key as an environment variable: GOOGLE_API_KEY . ​ Basic Usage Here’s a simple example of setting up a conversational AI bot with Gemini Multimodal Live: Copy Ask AI from pipecat.services.gemini_multimodal_live.gemini import ( GeminiMultimodalLiveLLMService, InputParams, GeminiMultimodalModalities ) llm = GeminiMultimodalLiveLLMService( api_key = os.getenv( "GOOGLE_API_KEY" ), voice_id = "Aoede" , # Voices: Aoede, Charon, Fenrir, Kore, Puck params = InputParams( temperature = 0.7 , # Set model input params language = Language. EN_US , # Set language (30+ languages supported) modalities = GeminiMultimodalModalities. AUDIO # Response modality ) ) ​ Configuration ​ Constructor Parameters ​ api_key str required Your Google API key ​ base_url str API endpoint URL ​ model str Gemini model to use (upgraded to new v1beta model) ​ voice_id str default: "Charon" Voice for text-to-speech (options: Aoede, Charon, Fenrir, Kore, Puck) Copy Ask AI llm = GeminiMultimodalLiveLLMService( api_key = os.getenv( "GOOGLE_API_KEY" ), voice_id = "Puck" , # Choose your preferred voice ) ​ system_instruction str High-level instructions that guide the model’s behavior Copy Ask AI llm = GeminiMultimodalLiveLLMService( api_key = os.getenv( "GOOGLE_API_KEY" ), system_instruction = "Talk like a pirate." , ) ​ start_audio_paused bool default: "False" Whether to start with audio input paused ​ start_video_paused bool default: "False" Whether to start with video input paused ​ tools Union[List[dict], ToolsSchema] Tools/functions available to the model ​ inference_on_context_initialization bool default: "True" Whether to generate a response when context is first set ​ Input Parameters ​ frequency_penalty float default: "None" Penalizes repeated tokens. Range: 0.0 to 2.0 ​ max_tokens int default: "4096" Maximum number of tokens to generate ​ modalities GeminiMultimodalModalities default: "AUDIO" Response modalities to include (options: AUDIO , TEXT ). ​ presence_penalty float default: "None" Penalizes tokens based on their presence in the text. Range: 0.0 to 2.0 ​ temperature float default: "None" Controls randomness in responses. Range: 0.0 to 2.0 ​ language Language default: "Language.EN_US" Language for generation. Over 30 languages are supported. ​ media_resolution GeminiMediaResolution default: "UNSPECIFIED" Controls image processing quality and token usage: LOW : Uses 64 tokens MEDIUM : Uses 256 tokens HIGH : Zoomed reframing with 256 tokens ​ vad GeminiVADParams Voice Activity Detection configuration: disabled : Toggle VAD on/off start_sensitivity : How quickly speech is detected (HIGH/LOW) end_sensitivity : How quickly turns end after pauses (HIGH/LOW) prefix_padding_ms : Milliseconds of audio to keep before speech silence_duration_ms : Milliseconds of silence to end a turn Copy Ask AI from pipecat.services.gemini_multimodal_live.events import ( StartSensitivity, EndSensitivity ) from pipecat.services.gemini_multimodal_live.gemini import ( GeminiVADParams, GeminiMediaResolution, ) llm = GeminiMultimodalLiveLLMService( api_key = os.getenv( "GOOGLE_API_KEY" ), params = InputParams( temperature = 0.7 , language = Language. ES , # Spanish language media_resolution = GeminiMediaResolution. HIGH , # Higher quality image processing vad = GeminiVADParams( start_sensitivity = StartSensitivity. HIGH , # Detect speech quickly end_sensitivity = EndSensitivity. LOW , # Allow longer pauses prefix_padding_ms = 300 , # Keep 300ms before speech silence_duration_ms = 1000 , # End turn after 1s silence ) ) ) ​ top_k int default: "None" Limits vocabulary to k most likely tokens. Minimum: 0 ​ top_p float default: "None" Cumulative probability cutoff for token selection. Range: 0.0 to 1.0 ​ context_window_compression ContextWindowCompressionParams Parameters for managing the context window: - enabled : Enable/disable compression (default: False) - trigger_tokens : Number of tokens that trigger compression (default: None, uses 80% of context window) Copy Ask AI from pipecat.services.gemini_multimodal_live.gemini import ( ContextWindowCompressionParams ) llm = GeminiMultimodalLiveLLMService( api_key = os.getenv( "GOOGLE_API_KEY" ), params = InputParams( top_p = 0.9 , # More focused token selection top_k = 40 , # Limit vocabulary options context_window_compression = ContextWindowCompressionParams( enabled = True , trigger_tokens = 8000 # Compress when reaching 8000 tokens ) ) ) ​ Methods ​ set_audio_input_paused(paused: bool) method Pause or unpause audio input processing ​ set_video_input_paused(paused: bool) method Pause or unpause video input processing ​ set_model_modalities(modalities: GeminiMultimodalModalities) method Change the response modality (TEXT or AUDIO) ​ set_language(language: Language) method Change the language for generation ​ set_context(context: OpenAILLMContext) method Set the conversation context explicitly ​ create_context_aggregator(context: OpenAILLMContext, user_params: LLMUserAggregatorParams, assistant_params: LLMAssistantAggregatorParams) method Create context aggregators for managing conversation state ​ Frame Types ​ Input Frames ​ InputAudioRawFrame Frame Raw audio data for speech input ​ InputImageRawFrame Frame Raw image data for visual input ​ StartInterruptionFrame Frame Signals start of user interruption ​ UserStartedSpeakingFrame Frame Signals user started speaking ​ UserStoppedSpeakingFrame Frame Signals user stopped speaking ​ OpenAILLMContextFrame Frame Contains conversation context ​ LLMMessagesAppendFrame Frame Adds messages to the conversation ​ LLMUpdateSettingsFrame Frame Updates LLM settings ​ LLMSetToolsFrame Frame Sets available tools for the LLM ​ Output Frames ​ TTSAudioRawFrame Frame Generated speech audio ​ TTSStartedFrame Frame Signals start of speech synthesis ​ TTSStoppedFrame Frame Signals end of speech synthesis ​ LLMTextFrame Frame Generated text responses from the LLM ​ TTSTextFrame Frame Text used for speech synthesis ​ TranscriptionFrame Frame Speech transcriptions from user audio ​ LLMFullResponseStartFrame Frame Signals the start of a complete LLM response ​ LLMFullResponseEndFrame Frame Signals the end of a complete LLM response ​ Function Calling This service supports function calling (also known as tool calling) which allows the LLM to request information from external services and APIs. For example, you can enable your bot to: Check current weather conditions Query databases Access external APIs Perform custom actions See the Function Calling guide for: Detailed implementation instructions Provider-specific function definitions Handler registration examples Control over function call behavior Complete usage examples ​ Token Usage Tracking Gemini Multimodal Live automatically tracks token usage metrics, providing: Prompt token counts Completion token counts Total token counts Detailed token breakdowns by modality (text, audio) These metrics can be used for monitoring usage, optimizing costs, and understanding model performance. ​ Language Support Gemini Multimodal Live supports the following languages: Language Code Description Gemini Code Language.AR Arabic ar-XA Language.BN_IN Bengali (India) bn-IN Language.CMN_CN Chinese (Mandarin) cmn-CN Language.DE_DE German (Germany) de-DE Language.EN_US English (US) en-US Language.EN_AU English (Australia) en-AU Language.EN_GB English (UK) en-GB Language.EN_IN English (India) en-IN Language.ES_ES Spanish (Spain) es-ES Language.ES_US Spanish (US) es-US Language.FR_FR French (France) fr-FR Language.FR_CA French (Canada) fr-CA Language.GU_IN Gujarati (India) gu-IN Language.HI_IN Hindi (India) hi-IN Language.ID_ID Indonesian id-ID Language.IT_IT Italian (Italy) it-IT Language.JA_JP Japanese (Japan) ja-JP Language.KN_IN Kannada (India) kn-IN Language.KO_KR Korean (Korea) ko-KR Language.ML_IN Malayalam (India) ml-IN Language.MR_IN Marathi (India) mr-IN Language.NL_NL Dutch (Netherlands) nl-NL Language.PL_PL Polish (Poland) pl-PL Language.PT_BR Portuguese (Brazil) pt-BR Language.RU_RU Russian (Russia) ru-RU Language.TA_IN Tamil (India) ta-IN Language.TE_IN Telugu (India) te-IN Language.TH_TH Thai (Thailand) th-TH Language.TR_TR Turkish (Turkey) tr-TR Language.VI_VN Vietnamese (Vietnam) vi-VN You can set the language using the language parameter: Copy Ask AI from pipecat.transcriptions.language import Language from pipecat.services.gemini_multimodal_live.gemini import ( GeminiMultimodalLiveLLMService, InputParams ) # Set language during initialization llm = GeminiMultimodalLiveLLMService( api_key = os.getenv( "GOOGLE_API_KEY" ), params = InputParams( language = Language. ES_ES ) # Spanish (Spain) ) ​ Next Steps ​ Examples Foundational Example Basic implementation showing core features and transcription Simple Chatbot A client/server example showing how to build a Pipecat JS or React client that connects to a Gemini Live Pipecat bot. ​ Learn More Check out our Gemini Multimodal Live Guide for detailed explanations and best practices. AWS Nova Sonic OpenAI Realtime Beta On this page Installation Basic Usage Configuration Constructor Parameters Input Parameters Methods Frame Types Input Frames Output Frames Function Calling Token Usage Tracking Language Support Next Steps Examples Learn More Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/s2s_gemini_2ee14192.txt b/s2s_gemini_2ee14192.txt
new file mode 100644
index 0000000000000000000000000000000000000000..2ccfb63dd91cfc2675dfa4a0c76584425208d1f7
--- /dev/null
+++ b/s2s_gemini_2ee14192.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/s2s/gemini#param-voice-id
+Title: Gemini Multimodal Live - Pipecat
+==================================================
+
+Gemini Multimodal Live - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Speech Gemini Multimodal Live Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech AWS Nova Sonic Gemini Multimodal Live OpenAI Realtime Beta Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline The GeminiMultimodalLiveLLMService enables natural, real-time conversations with Google’s Gemini model. It provides built-in audio transcription, voice activity detection, and context management for creating interactive AI experiences. It provides: Real-time Interaction Stream audio and video in real-time with low latency response times Speech Processing Built-in speech-to-text and text-to-speech capabilities with multiple voice options Voice Activity Detection Automatic detection of speech start/stop for natural conversations Context Management Intelligent handling of conversation history and system instructions Want to start building? Check out our Gemini Multimodal Live Guide . ​ Installation To use GeminiMultimodalLiveLLMService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[google]" You’ll need to set up your Google API key as an environment variable: GOOGLE_API_KEY . ​ Basic Usage Here’s a simple example of setting up a conversational AI bot with Gemini Multimodal Live: Copy Ask AI from pipecat.services.gemini_multimodal_live.gemini import ( GeminiMultimodalLiveLLMService, InputParams, GeminiMultimodalModalities ) llm = GeminiMultimodalLiveLLMService( api_key = os.getenv( "GOOGLE_API_KEY" ), voice_id = "Aoede" , # Voices: Aoede, Charon, Fenrir, Kore, Puck params = InputParams( temperature = 0.7 , # Set model input params language = Language. EN_US , # Set language (30+ languages supported) modalities = GeminiMultimodalModalities. AUDIO # Response modality ) ) ​ Configuration ​ Constructor Parameters ​ api_key str required Your Google API key ​ base_url str API endpoint URL ​ model str Gemini model to use (upgraded to new v1beta model) ​ voice_id str default: "Charon" Voice for text-to-speech (options: Aoede, Charon, Fenrir, Kore, Puck) Copy Ask AI llm = GeminiMultimodalLiveLLMService( api_key = os.getenv( "GOOGLE_API_KEY" ), voice_id = "Puck" , # Choose your preferred voice ) ​ system_instruction str High-level instructions that guide the model’s behavior Copy Ask AI llm = GeminiMultimodalLiveLLMService( api_key = os.getenv( "GOOGLE_API_KEY" ), system_instruction = "Talk like a pirate." , ) ​ start_audio_paused bool default: "False" Whether to start with audio input paused ​ start_video_paused bool default: "False" Whether to start with video input paused ​ tools Union[List[dict], ToolsSchema] Tools/functions available to the model ​ inference_on_context_initialization bool default: "True" Whether to generate a response when context is first set ​ Input Parameters ​ frequency_penalty float default: "None" Penalizes repeated tokens. Range: 0.0 to 2.0 ​ max_tokens int default: "4096" Maximum number of tokens to generate ​ modalities GeminiMultimodalModalities default: "AUDIO" Response modalities to include (options: AUDIO , TEXT ). ​ presence_penalty float default: "None" Penalizes tokens based on their presence in the text. Range: 0.0 to 2.0 ​ temperature float default: "None" Controls randomness in responses. Range: 0.0 to 2.0 ​ language Language default: "Language.EN_US" Language for generation. Over 30 languages are supported. ​ media_resolution GeminiMediaResolution default: "UNSPECIFIED" Controls image processing quality and token usage: LOW : Uses 64 tokens MEDIUM : Uses 256 tokens HIGH : Zoomed reframing with 256 tokens ​ vad GeminiVADParams Voice Activity Detection configuration: disabled : Toggle VAD on/off start_sensitivity : How quickly speech is detected (HIGH/LOW) end_sensitivity : How quickly turns end after pauses (HIGH/LOW) prefix_padding_ms : Milliseconds of audio to keep before speech silence_duration_ms : Milliseconds of silence to end a turn Copy Ask AI from pipecat.services.gemini_multimodal_live.events import ( StartSensitivity, EndSensitivity ) from pipecat.services.gemini_multimodal_live.gemini import ( GeminiVADParams, GeminiMediaResolution, ) llm = GeminiMultimodalLiveLLMService( api_key = os.getenv( "GOOGLE_API_KEY" ), params = InputParams( temperature = 0.7 , language = Language. ES , # Spanish language media_resolution = GeminiMediaResolution. HIGH , # Higher quality image processing vad = GeminiVADParams( start_sensitivity = StartSensitivity. HIGH , # Detect speech quickly end_sensitivity = EndSensitivity. LOW , # Allow longer pauses prefix_padding_ms = 300 , # Keep 300ms before speech silence_duration_ms = 1000 , # End turn after 1s silence ) ) ) ​ top_k int default: "None" Limits vocabulary to k most likely tokens. Minimum: 0 ​ top_p float default: "None" Cumulative probability cutoff for token selection. Range: 0.0 to 1.0 ​ context_window_compression ContextWindowCompressionParams Parameters for managing the context window: - enabled : Enable/disable compression (default: False) - trigger_tokens : Number of tokens that trigger compression (default: None, uses 80% of context window) Copy Ask AI from pipecat.services.gemini_multimodal_live.gemini import ( ContextWindowCompressionParams ) llm = GeminiMultimodalLiveLLMService( api_key = os.getenv( "GOOGLE_API_KEY" ), params = InputParams( top_p = 0.9 , # More focused token selection top_k = 40 , # Limit vocabulary options context_window_compression = ContextWindowCompressionParams( enabled = True , trigger_tokens = 8000 # Compress when reaching 8000 tokens ) ) ) ​ Methods ​ set_audio_input_paused(paused: bool) method Pause or unpause audio input processing ​ set_video_input_paused(paused: bool) method Pause or unpause video input processing ​ set_model_modalities(modalities: GeminiMultimodalModalities) method Change the response modality (TEXT or AUDIO) ​ set_language(language: Language) method Change the language for generation ​ set_context(context: OpenAILLMContext) method Set the conversation context explicitly ​ create_context_aggregator(context: OpenAILLMContext, user_params: LLMUserAggregatorParams, assistant_params: LLMAssistantAggregatorParams) method Create context aggregators for managing conversation state ​ Frame Types ​ Input Frames ​ InputAudioRawFrame Frame Raw audio data for speech input ​ InputImageRawFrame Frame Raw image data for visual input ​ StartInterruptionFrame Frame Signals start of user interruption ​ UserStartedSpeakingFrame Frame Signals user started speaking ​ UserStoppedSpeakingFrame Frame Signals user stopped speaking ​ OpenAILLMContextFrame Frame Contains conversation context ​ LLMMessagesAppendFrame Frame Adds messages to the conversation ​ LLMUpdateSettingsFrame Frame Updates LLM settings ​ LLMSetToolsFrame Frame Sets available tools for the LLM ​ Output Frames ​ TTSAudioRawFrame Frame Generated speech audio ​ TTSStartedFrame Frame Signals start of speech synthesis ​ TTSStoppedFrame Frame Signals end of speech synthesis ​ LLMTextFrame Frame Generated text responses from the LLM ​ TTSTextFrame Frame Text used for speech synthesis ​ TranscriptionFrame Frame Speech transcriptions from user audio ​ LLMFullResponseStartFrame Frame Signals the start of a complete LLM response ​ LLMFullResponseEndFrame Frame Signals the end of a complete LLM response ​ Function Calling This service supports function calling (also known as tool calling) which allows the LLM to request information from external services and APIs. For example, you can enable your bot to: Check current weather conditions Query databases Access external APIs Perform custom actions See the Function Calling guide for: Detailed implementation instructions Provider-specific function definitions Handler registration examples Control over function call behavior Complete usage examples ​ Token Usage Tracking Gemini Multimodal Live automatically tracks token usage metrics, providing: Prompt token counts Completion token counts Total token counts Detailed token breakdowns by modality (text, audio) These metrics can be used for monitoring usage, optimizing costs, and understanding model performance. ​ Language Support Gemini Multimodal Live supports the following languages: Language Code Description Gemini Code Language.AR Arabic ar-XA Language.BN_IN Bengali (India) bn-IN Language.CMN_CN Chinese (Mandarin) cmn-CN Language.DE_DE German (Germany) de-DE Language.EN_US English (US) en-US Language.EN_AU English (Australia) en-AU Language.EN_GB English (UK) en-GB Language.EN_IN English (India) en-IN Language.ES_ES Spanish (Spain) es-ES Language.ES_US Spanish (US) es-US Language.FR_FR French (France) fr-FR Language.FR_CA French (Canada) fr-CA Language.GU_IN Gujarati (India) gu-IN Language.HI_IN Hindi (India) hi-IN Language.ID_ID Indonesian id-ID Language.IT_IT Italian (Italy) it-IT Language.JA_JP Japanese (Japan) ja-JP Language.KN_IN Kannada (India) kn-IN Language.KO_KR Korean (Korea) ko-KR Language.ML_IN Malayalam (India) ml-IN Language.MR_IN Marathi (India) mr-IN Language.NL_NL Dutch (Netherlands) nl-NL Language.PL_PL Polish (Poland) pl-PL Language.PT_BR Portuguese (Brazil) pt-BR Language.RU_RU Russian (Russia) ru-RU Language.TA_IN Tamil (India) ta-IN Language.TE_IN Telugu (India) te-IN Language.TH_TH Thai (Thailand) th-TH Language.TR_TR Turkish (Turkey) tr-TR Language.VI_VN Vietnamese (Vietnam) vi-VN You can set the language using the language parameter: Copy Ask AI from pipecat.transcriptions.language import Language from pipecat.services.gemini_multimodal_live.gemini import ( GeminiMultimodalLiveLLMService, InputParams ) # Set language during initialization llm = GeminiMultimodalLiveLLMService( api_key = os.getenv( "GOOGLE_API_KEY" ), params = InputParams( language = Language. ES_ES ) # Spanish (Spain) ) ​ Next Steps ​ Examples Foundational Example Basic implementation showing core features and transcription Simple Chatbot A client/server example showing how to build a Pipecat JS or React client that connects to a Gemini Live Pipecat bot. ​ Learn More Check out our Gemini Multimodal Live Guide for detailed explanations and best practices. AWS Nova Sonic OpenAI Realtime Beta On this page Installation Basic Usage Configuration Constructor Parameters Input Parameters Methods Frame Types Input Frames Output Frames Function Calling Token Usage Tracking Language Support Next Steps Examples Learn More Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/s2s_gemini_582efe6a.txt b/s2s_gemini_582efe6a.txt
new file mode 100644
index 0000000000000000000000000000000000000000..a640ce2c87e1952d1560829c3bca9abe8c3af54e
--- /dev/null
+++ b/s2s_gemini_582efe6a.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/s2s/gemini#token-usage-tracking
+Title: Gemini Multimodal Live - Pipecat
+==================================================
+
+Gemini Multimodal Live - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Speech Gemini Multimodal Live Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech AWS Nova Sonic Gemini Multimodal Live OpenAI Realtime Beta Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline The GeminiMultimodalLiveLLMService enables natural, real-time conversations with Google’s Gemini model. It provides built-in audio transcription, voice activity detection, and context management for creating interactive AI experiences. It provides: Real-time Interaction Stream audio and video in real-time with low latency response times Speech Processing Built-in speech-to-text and text-to-speech capabilities with multiple voice options Voice Activity Detection Automatic detection of speech start/stop for natural conversations Context Management Intelligent handling of conversation history and system instructions Want to start building? Check out our Gemini Multimodal Live Guide . ​ Installation To use GeminiMultimodalLiveLLMService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[google]" You’ll need to set up your Google API key as an environment variable: GOOGLE_API_KEY . ​ Basic Usage Here’s a simple example of setting up a conversational AI bot with Gemini Multimodal Live: Copy Ask AI from pipecat.services.gemini_multimodal_live.gemini import ( GeminiMultimodalLiveLLMService, InputParams, GeminiMultimodalModalities ) llm = GeminiMultimodalLiveLLMService( api_key = os.getenv( "GOOGLE_API_KEY" ), voice_id = "Aoede" , # Voices: Aoede, Charon, Fenrir, Kore, Puck params = InputParams( temperature = 0.7 , # Set model input params language = Language. EN_US , # Set language (30+ languages supported) modalities = GeminiMultimodalModalities. AUDIO # Response modality ) ) ​ Configuration ​ Constructor Parameters ​ api_key str required Your Google API key ​ base_url str API endpoint URL ​ model str Gemini model to use (upgraded to new v1beta model) ​ voice_id str default: "Charon" Voice for text-to-speech (options: Aoede, Charon, Fenrir, Kore, Puck) Copy Ask AI llm = GeminiMultimodalLiveLLMService( api_key = os.getenv( "GOOGLE_API_KEY" ), voice_id = "Puck" , # Choose your preferred voice ) ​ system_instruction str High-level instructions that guide the model’s behavior Copy Ask AI llm = GeminiMultimodalLiveLLMService( api_key = os.getenv( "GOOGLE_API_KEY" ), system_instruction = "Talk like a pirate." , ) ​ start_audio_paused bool default: "False" Whether to start with audio input paused ​ start_video_paused bool default: "False" Whether to start with video input paused ​ tools Union[List[dict], ToolsSchema] Tools/functions available to the model ​ inference_on_context_initialization bool default: "True" Whether to generate a response when context is first set ​ Input Parameters ​ frequency_penalty float default: "None" Penalizes repeated tokens. Range: 0.0 to 2.0 ​ max_tokens int default: "4096" Maximum number of tokens to generate ​ modalities GeminiMultimodalModalities default: "AUDIO" Response modalities to include (options: AUDIO , TEXT ). ​ presence_penalty float default: "None" Penalizes tokens based on their presence in the text. Range: 0.0 to 2.0 ​ temperature float default: "None" Controls randomness in responses. Range: 0.0 to 2.0 ​ language Language default: "Language.EN_US" Language for generation. Over 30 languages are supported. ​ media_resolution GeminiMediaResolution default: "UNSPECIFIED" Controls image processing quality and token usage: LOW : Uses 64 tokens MEDIUM : Uses 256 tokens HIGH : Zoomed reframing with 256 tokens ​ vad GeminiVADParams Voice Activity Detection configuration: disabled : Toggle VAD on/off start_sensitivity : How quickly speech is detected (HIGH/LOW) end_sensitivity : How quickly turns end after pauses (HIGH/LOW) prefix_padding_ms : Milliseconds of audio to keep before speech silence_duration_ms : Milliseconds of silence to end a turn Copy Ask AI from pipecat.services.gemini_multimodal_live.events import ( StartSensitivity, EndSensitivity ) from pipecat.services.gemini_multimodal_live.gemini import ( GeminiVADParams, GeminiMediaResolution, ) llm = GeminiMultimodalLiveLLMService( api_key = os.getenv( "GOOGLE_API_KEY" ), params = InputParams( temperature = 0.7 , language = Language. ES , # Spanish language media_resolution = GeminiMediaResolution. HIGH , # Higher quality image processing vad = GeminiVADParams( start_sensitivity = StartSensitivity. HIGH , # Detect speech quickly end_sensitivity = EndSensitivity. LOW , # Allow longer pauses prefix_padding_ms = 300 , # Keep 300ms before speech silence_duration_ms = 1000 , # End turn after 1s silence ) ) ) ​ top_k int default: "None" Limits vocabulary to k most likely tokens. Minimum: 0 ​ top_p float default: "None" Cumulative probability cutoff for token selection. Range: 0.0 to 1.0 ​ context_window_compression ContextWindowCompressionParams Parameters for managing the context window: - enabled : Enable/disable compression (default: False) - trigger_tokens : Number of tokens that trigger compression (default: None, uses 80% of context window) Copy Ask AI from pipecat.services.gemini_multimodal_live.gemini import ( ContextWindowCompressionParams ) llm = GeminiMultimodalLiveLLMService( api_key = os.getenv( "GOOGLE_API_KEY" ), params = InputParams( top_p = 0.9 , # More focused token selection top_k = 40 , # Limit vocabulary options context_window_compression = ContextWindowCompressionParams( enabled = True , trigger_tokens = 8000 # Compress when reaching 8000 tokens ) ) ) ​ Methods ​ set_audio_input_paused(paused: bool) method Pause or unpause audio input processing ​ set_video_input_paused(paused: bool) method Pause or unpause video input processing ​ set_model_modalities(modalities: GeminiMultimodalModalities) method Change the response modality (TEXT or AUDIO) ​ set_language(language: Language) method Change the language for generation ​ set_context(context: OpenAILLMContext) method Set the conversation context explicitly ​ create_context_aggregator(context: OpenAILLMContext, user_params: LLMUserAggregatorParams, assistant_params: LLMAssistantAggregatorParams) method Create context aggregators for managing conversation state ​ Frame Types ​ Input Frames ​ InputAudioRawFrame Frame Raw audio data for speech input ​ InputImageRawFrame Frame Raw image data for visual input ​ StartInterruptionFrame Frame Signals start of user interruption ​ UserStartedSpeakingFrame Frame Signals user started speaking ​ UserStoppedSpeakingFrame Frame Signals user stopped speaking ​ OpenAILLMContextFrame Frame Contains conversation context ​ LLMMessagesAppendFrame Frame Adds messages to the conversation ​ LLMUpdateSettingsFrame Frame Updates LLM settings ​ LLMSetToolsFrame Frame Sets available tools for the LLM ​ Output Frames ​ TTSAudioRawFrame Frame Generated speech audio ​ TTSStartedFrame Frame Signals start of speech synthesis ​ TTSStoppedFrame Frame Signals end of speech synthesis ​ LLMTextFrame Frame Generated text responses from the LLM ​ TTSTextFrame Frame Text used for speech synthesis ​ TranscriptionFrame Frame Speech transcriptions from user audio ​ LLMFullResponseStartFrame Frame Signals the start of a complete LLM response ​ LLMFullResponseEndFrame Frame Signals the end of a complete LLM response ​ Function Calling This service supports function calling (also known as tool calling) which allows the LLM to request information from external services and APIs. For example, you can enable your bot to: Check current weather conditions Query databases Access external APIs Perform custom actions See the Function Calling guide for: Detailed implementation instructions Provider-specific function definitions Handler registration examples Control over function call behavior Complete usage examples ​ Token Usage Tracking Gemini Multimodal Live automatically tracks token usage metrics, providing: Prompt token counts Completion token counts Total token counts Detailed token breakdowns by modality (text, audio) These metrics can be used for monitoring usage, optimizing costs, and understanding model performance. ​ Language Support Gemini Multimodal Live supports the following languages: Language Code Description Gemini Code Language.AR Arabic ar-XA Language.BN_IN Bengali (India) bn-IN Language.CMN_CN Chinese (Mandarin) cmn-CN Language.DE_DE German (Germany) de-DE Language.EN_US English (US) en-US Language.EN_AU English (Australia) en-AU Language.EN_GB English (UK) en-GB Language.EN_IN English (India) en-IN Language.ES_ES Spanish (Spain) es-ES Language.ES_US Spanish (US) es-US Language.FR_FR French (France) fr-FR Language.FR_CA French (Canada) fr-CA Language.GU_IN Gujarati (India) gu-IN Language.HI_IN Hindi (India) hi-IN Language.ID_ID Indonesian id-ID Language.IT_IT Italian (Italy) it-IT Language.JA_JP Japanese (Japan) ja-JP Language.KN_IN Kannada (India) kn-IN Language.KO_KR Korean (Korea) ko-KR Language.ML_IN Malayalam (India) ml-IN Language.MR_IN Marathi (India) mr-IN Language.NL_NL Dutch (Netherlands) nl-NL Language.PL_PL Polish (Poland) pl-PL Language.PT_BR Portuguese (Brazil) pt-BR Language.RU_RU Russian (Russia) ru-RU Language.TA_IN Tamil (India) ta-IN Language.TE_IN Telugu (India) te-IN Language.TH_TH Thai (Thailand) th-TH Language.TR_TR Turkish (Turkey) tr-TR Language.VI_VN Vietnamese (Vietnam) vi-VN You can set the language using the language parameter: Copy Ask AI from pipecat.transcriptions.language import Language from pipecat.services.gemini_multimodal_live.gemini import ( GeminiMultimodalLiveLLMService, InputParams ) # Set language during initialization llm = GeminiMultimodalLiveLLMService( api_key = os.getenv( "GOOGLE_API_KEY" ), params = InputParams( language = Language. ES_ES ) # Spanish (Spain) ) ​ Next Steps ​ Examples Foundational Example Basic implementation showing core features and transcription Simple Chatbot A client/server example showing how to build a Pipecat JS or React client that connects to a Gemini Live Pipecat bot. ​ Learn More Check out our Gemini Multimodal Live Guide for detailed explanations and best practices. AWS Nova Sonic OpenAI Realtime Beta On this page Installation Basic Usage Configuration Constructor Parameters Input Parameters Methods Frame Types Input Frames Output Frames Function Calling Token Usage Tracking Language Support Next Steps Examples Learn More Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/s2s_gemini_62b9848c.txt b/s2s_gemini_62b9848c.txt
new file mode 100644
index 0000000000000000000000000000000000000000..43bcf6d3a00c18e8d10df953c190568d28e022b1
--- /dev/null
+++ b/s2s_gemini_62b9848c.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/s2s/gemini#input-frames
+Title: Gemini Multimodal Live - Pipecat
+==================================================
+
+Gemini Multimodal Live - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Speech Gemini Multimodal Live Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech AWS Nova Sonic Gemini Multimodal Live OpenAI Realtime Beta Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline The GeminiMultimodalLiveLLMService enables natural, real-time conversations with Google’s Gemini model. It provides built-in audio transcription, voice activity detection, and context management for creating interactive AI experiences. It provides: Real-time Interaction Stream audio and video in real-time with low latency response times Speech Processing Built-in speech-to-text and text-to-speech capabilities with multiple voice options Voice Activity Detection Automatic detection of speech start/stop for natural conversations Context Management Intelligent handling of conversation history and system instructions Want to start building? Check out our Gemini Multimodal Live Guide . ​ Installation To use GeminiMultimodalLiveLLMService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[google]" You’ll need to set up your Google API key as an environment variable: GOOGLE_API_KEY . ​ Basic Usage Here’s a simple example of setting up a conversational AI bot with Gemini Multimodal Live: Copy Ask AI from pipecat.services.gemini_multimodal_live.gemini import ( GeminiMultimodalLiveLLMService, InputParams, GeminiMultimodalModalities ) llm = GeminiMultimodalLiveLLMService( api_key = os.getenv( "GOOGLE_API_KEY" ), voice_id = "Aoede" , # Voices: Aoede, Charon, Fenrir, Kore, Puck params = InputParams( temperature = 0.7 , # Set model input params language = Language. EN_US , # Set language (30+ languages supported) modalities = GeminiMultimodalModalities. AUDIO # Response modality ) ) ​ Configuration ​ Constructor Parameters ​ api_key str required Your Google API key ​ base_url str API endpoint URL ​ model str Gemini model to use (upgraded to new v1beta model) ​ voice_id str default: "Charon" Voice for text-to-speech (options: Aoede, Charon, Fenrir, Kore, Puck) Copy Ask AI llm = GeminiMultimodalLiveLLMService( api_key = os.getenv( "GOOGLE_API_KEY" ), voice_id = "Puck" , # Choose your preferred voice ) ​ system_instruction str High-level instructions that guide the model’s behavior Copy Ask AI llm = GeminiMultimodalLiveLLMService( api_key = os.getenv( "GOOGLE_API_KEY" ), system_instruction = "Talk like a pirate." , ) ​ start_audio_paused bool default: "False" Whether to start with audio input paused ​ start_video_paused bool default: "False" Whether to start with video input paused ​ tools Union[List[dict], ToolsSchema] Tools/functions available to the model ​ inference_on_context_initialization bool default: "True" Whether to generate a response when context is first set ​ Input Parameters ​ frequency_penalty float default: "None" Penalizes repeated tokens. Range: 0.0 to 2.0 ​ max_tokens int default: "4096" Maximum number of tokens to generate ​ modalities GeminiMultimodalModalities default: "AUDIO" Response modalities to include (options: AUDIO , TEXT ). ​ presence_penalty float default: "None" Penalizes tokens based on their presence in the text. Range: 0.0 to 2.0 ​ temperature float default: "None" Controls randomness in responses. Range: 0.0 to 2.0 ​ language Language default: "Language.EN_US" Language for generation. Over 30 languages are supported. ​ media_resolution GeminiMediaResolution default: "UNSPECIFIED" Controls image processing quality and token usage: LOW : Uses 64 tokens MEDIUM : Uses 256 tokens HIGH : Zoomed reframing with 256 tokens ​ vad GeminiVADParams Voice Activity Detection configuration: disabled : Toggle VAD on/off start_sensitivity : How quickly speech is detected (HIGH/LOW) end_sensitivity : How quickly turns end after pauses (HIGH/LOW) prefix_padding_ms : Milliseconds of audio to keep before speech silence_duration_ms : Milliseconds of silence to end a turn Copy Ask AI from pipecat.services.gemini_multimodal_live.events import ( StartSensitivity, EndSensitivity ) from pipecat.services.gemini_multimodal_live.gemini import ( GeminiVADParams, GeminiMediaResolution, ) llm = GeminiMultimodalLiveLLMService( api_key = os.getenv( "GOOGLE_API_KEY" ), params = InputParams( temperature = 0.7 , language = Language. ES , # Spanish language media_resolution = GeminiMediaResolution. HIGH , # Higher quality image processing vad = GeminiVADParams( start_sensitivity = StartSensitivity. HIGH , # Detect speech quickly end_sensitivity = EndSensitivity. LOW , # Allow longer pauses prefix_padding_ms = 300 , # Keep 300ms before speech silence_duration_ms = 1000 , # End turn after 1s silence ) ) ) ​ top_k int default: "None" Limits vocabulary to k most likely tokens. Minimum: 0 ​ top_p float default: "None" Cumulative probability cutoff for token selection. Range: 0.0 to 1.0 ​ context_window_compression ContextWindowCompressionParams Parameters for managing the context window: - enabled : Enable/disable compression (default: False) - trigger_tokens : Number of tokens that trigger compression (default: None, uses 80% of context window) Copy Ask AI from pipecat.services.gemini_multimodal_live.gemini import ( ContextWindowCompressionParams ) llm = GeminiMultimodalLiveLLMService( api_key = os.getenv( "GOOGLE_API_KEY" ), params = InputParams( top_p = 0.9 , # More focused token selection top_k = 40 , # Limit vocabulary options context_window_compression = ContextWindowCompressionParams( enabled = True , trigger_tokens = 8000 # Compress when reaching 8000 tokens ) ) ) ​ Methods ​ set_audio_input_paused(paused: bool) method Pause or unpause audio input processing ​ set_video_input_paused(paused: bool) method Pause or unpause video input processing ​ set_model_modalities(modalities: GeminiMultimodalModalities) method Change the response modality (TEXT or AUDIO) ​ set_language(language: Language) method Change the language for generation ​ set_context(context: OpenAILLMContext) method Set the conversation context explicitly ​ create_context_aggregator(context: OpenAILLMContext, user_params: LLMUserAggregatorParams, assistant_params: LLMAssistantAggregatorParams) method Create context aggregators for managing conversation state ​ Frame Types ​ Input Frames ​ InputAudioRawFrame Frame Raw audio data for speech input ​ InputImageRawFrame Frame Raw image data for visual input ​ StartInterruptionFrame Frame Signals start of user interruption ​ UserStartedSpeakingFrame Frame Signals user started speaking ​ UserStoppedSpeakingFrame Frame Signals user stopped speaking ​ OpenAILLMContextFrame Frame Contains conversation context ​ LLMMessagesAppendFrame Frame Adds messages to the conversation ​ LLMUpdateSettingsFrame Frame Updates LLM settings ​ LLMSetToolsFrame Frame Sets available tools for the LLM ​ Output Frames ​ TTSAudioRawFrame Frame Generated speech audio ​ TTSStartedFrame Frame Signals start of speech synthesis ​ TTSStoppedFrame Frame Signals end of speech synthesis ​ LLMTextFrame Frame Generated text responses from the LLM ​ TTSTextFrame Frame Text used for speech synthesis ​ TranscriptionFrame Frame Speech transcriptions from user audio ​ LLMFullResponseStartFrame Frame Signals the start of a complete LLM response ​ LLMFullResponseEndFrame Frame Signals the end of a complete LLM response ​ Function Calling This service supports function calling (also known as tool calling) which allows the LLM to request information from external services and APIs. For example, you can enable your bot to: Check current weather conditions Query databases Access external APIs Perform custom actions See the Function Calling guide for: Detailed implementation instructions Provider-specific function definitions Handler registration examples Control over function call behavior Complete usage examples ​ Token Usage Tracking Gemini Multimodal Live automatically tracks token usage metrics, providing: Prompt token counts Completion token counts Total token counts Detailed token breakdowns by modality (text, audio) These metrics can be used for monitoring usage, optimizing costs, and understanding model performance. ​ Language Support Gemini Multimodal Live supports the following languages: Language Code Description Gemini Code Language.AR Arabic ar-XA Language.BN_IN Bengali (India) bn-IN Language.CMN_CN Chinese (Mandarin) cmn-CN Language.DE_DE German (Germany) de-DE Language.EN_US English (US) en-US Language.EN_AU English (Australia) en-AU Language.EN_GB English (UK) en-GB Language.EN_IN English (India) en-IN Language.ES_ES Spanish (Spain) es-ES Language.ES_US Spanish (US) es-US Language.FR_FR French (France) fr-FR Language.FR_CA French (Canada) fr-CA Language.GU_IN Gujarati (India) gu-IN Language.HI_IN Hindi (India) hi-IN Language.ID_ID Indonesian id-ID Language.IT_IT Italian (Italy) it-IT Language.JA_JP Japanese (Japan) ja-JP Language.KN_IN Kannada (India) kn-IN Language.KO_KR Korean (Korea) ko-KR Language.ML_IN Malayalam (India) ml-IN Language.MR_IN Marathi (India) mr-IN Language.NL_NL Dutch (Netherlands) nl-NL Language.PL_PL Polish (Poland) pl-PL Language.PT_BR Portuguese (Brazil) pt-BR Language.RU_RU Russian (Russia) ru-RU Language.TA_IN Tamil (India) ta-IN Language.TE_IN Telugu (India) te-IN Language.TH_TH Thai (Thailand) th-TH Language.TR_TR Turkish (Turkey) tr-TR Language.VI_VN Vietnamese (Vietnam) vi-VN You can set the language using the language parameter: Copy Ask AI from pipecat.transcriptions.language import Language from pipecat.services.gemini_multimodal_live.gemini import ( GeminiMultimodalLiveLLMService, InputParams ) # Set language during initialization llm = GeminiMultimodalLiveLLMService( api_key = os.getenv( "GOOGLE_API_KEY" ), params = InputParams( language = Language. ES_ES ) # Spanish (Spain) ) ​ Next Steps ​ Examples Foundational Example Basic implementation showing core features and transcription Simple Chatbot A client/server example showing how to build a Pipecat JS or React client that connects to a Gemini Live Pipecat bot. ​ Learn More Check out our Gemini Multimodal Live Guide for detailed explanations and best practices. AWS Nova Sonic OpenAI Realtime Beta On this page Installation Basic Usage Configuration Constructor Parameters Input Parameters Methods Frame Types Input Frames Output Frames Function Calling Token Usage Tracking Language Support Next Steps Examples Learn More Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/s2s_gemini_66479790.txt b/s2s_gemini_66479790.txt
new file mode 100644
index 0000000000000000000000000000000000000000..7f99cdb6417f614406d3d41f028f4a4a449e4014
--- /dev/null
+++ b/s2s_gemini_66479790.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/s2s/gemini
+Title: Gemini Multimodal Live - Pipecat
+==================================================
+
+Gemini Multimodal Live - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Speech Gemini Multimodal Live Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech AWS Nova Sonic Gemini Multimodal Live OpenAI Realtime Beta Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline The GeminiMultimodalLiveLLMService enables natural, real-time conversations with Google’s Gemini model. It provides built-in audio transcription, voice activity detection, and context management for creating interactive AI experiences. It provides: Real-time Interaction Stream audio and video in real-time with low latency response times Speech Processing Built-in speech-to-text and text-to-speech capabilities with multiple voice options Voice Activity Detection Automatic detection of speech start/stop for natural conversations Context Management Intelligent handling of conversation history and system instructions Want to start building? Check out our Gemini Multimodal Live Guide . ​ Installation To use GeminiMultimodalLiveLLMService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[google]" You’ll need to set up your Google API key as an environment variable: GOOGLE_API_KEY . ​ Basic Usage Here’s a simple example of setting up a conversational AI bot with Gemini Multimodal Live: Copy Ask AI from pipecat.services.gemini_multimodal_live.gemini import ( GeminiMultimodalLiveLLMService, InputParams, GeminiMultimodalModalities ) llm = GeminiMultimodalLiveLLMService( api_key = os.getenv( "GOOGLE_API_KEY" ), voice_id = "Aoede" , # Voices: Aoede, Charon, Fenrir, Kore, Puck params = InputParams( temperature = 0.7 , # Set model input params language = Language. EN_US , # Set language (30+ languages supported) modalities = GeminiMultimodalModalities. AUDIO # Response modality ) ) ​ Configuration ​ Constructor Parameters ​ api_key str required Your Google API key ​ base_url str API endpoint URL ​ model str Gemini model to use (upgraded to new v1beta model) ​ voice_id str default: "Charon" Voice for text-to-speech (options: Aoede, Charon, Fenrir, Kore, Puck) Copy Ask AI llm = GeminiMultimodalLiveLLMService( api_key = os.getenv( "GOOGLE_API_KEY" ), voice_id = "Puck" , # Choose your preferred voice ) ​ system_instruction str High-level instructions that guide the model’s behavior Copy Ask AI llm = GeminiMultimodalLiveLLMService( api_key = os.getenv( "GOOGLE_API_KEY" ), system_instruction = "Talk like a pirate." , ) ​ start_audio_paused bool default: "False" Whether to start with audio input paused ​ start_video_paused bool default: "False" Whether to start with video input paused ​ tools Union[List[dict], ToolsSchema] Tools/functions available to the model ​ inference_on_context_initialization bool default: "True" Whether to generate a response when context is first set ​ Input Parameters ​ frequency_penalty float default: "None" Penalizes repeated tokens. Range: 0.0 to 2.0 ​ max_tokens int default: "4096" Maximum number of tokens to generate ​ modalities GeminiMultimodalModalities default: "AUDIO" Response modalities to include (options: AUDIO , TEXT ). ​ presence_penalty float default: "None" Penalizes tokens based on their presence in the text. Range: 0.0 to 2.0 ​ temperature float default: "None" Controls randomness in responses. Range: 0.0 to 2.0 ​ language Language default: "Language.EN_US" Language for generation. Over 30 languages are supported. ​ media_resolution GeminiMediaResolution default: "UNSPECIFIED" Controls image processing quality and token usage: LOW : Uses 64 tokens MEDIUM : Uses 256 tokens HIGH : Zoomed reframing with 256 tokens ​ vad GeminiVADParams Voice Activity Detection configuration: disabled : Toggle VAD on/off start_sensitivity : How quickly speech is detected (HIGH/LOW) end_sensitivity : How quickly turns end after pauses (HIGH/LOW) prefix_padding_ms : Milliseconds of audio to keep before speech silence_duration_ms : Milliseconds of silence to end a turn Copy Ask AI from pipecat.services.gemini_multimodal_live.events import ( StartSensitivity, EndSensitivity ) from pipecat.services.gemini_multimodal_live.gemini import ( GeminiVADParams, GeminiMediaResolution, ) llm = GeminiMultimodalLiveLLMService( api_key = os.getenv( "GOOGLE_API_KEY" ), params = InputParams( temperature = 0.7 , language = Language. ES , # Spanish language media_resolution = GeminiMediaResolution. HIGH , # Higher quality image processing vad = GeminiVADParams( start_sensitivity = StartSensitivity. HIGH , # Detect speech quickly end_sensitivity = EndSensitivity. LOW , # Allow longer pauses prefix_padding_ms = 300 , # Keep 300ms before speech silence_duration_ms = 1000 , # End turn after 1s silence ) ) ) ​ top_k int default: "None" Limits vocabulary to k most likely tokens. Minimum: 0 ​ top_p float default: "None" Cumulative probability cutoff for token selection. Range: 0.0 to 1.0 ​ context_window_compression ContextWindowCompressionParams Parameters for managing the context window: - enabled : Enable/disable compression (default: False) - trigger_tokens : Number of tokens that trigger compression (default: None, uses 80% of context window) Copy Ask AI from pipecat.services.gemini_multimodal_live.gemini import ( ContextWindowCompressionParams ) llm = GeminiMultimodalLiveLLMService( api_key = os.getenv( "GOOGLE_API_KEY" ), params = InputParams( top_p = 0.9 , # More focused token selection top_k = 40 , # Limit vocabulary options context_window_compression = ContextWindowCompressionParams( enabled = True , trigger_tokens = 8000 # Compress when reaching 8000 tokens ) ) ) ​ Methods ​ set_audio_input_paused(paused: bool) method Pause or unpause audio input processing ​ set_video_input_paused(paused: bool) method Pause or unpause video input processing ​ set_model_modalities(modalities: GeminiMultimodalModalities) method Change the response modality (TEXT or AUDIO) ​ set_language(language: Language) method Change the language for generation ​ set_context(context: OpenAILLMContext) method Set the conversation context explicitly ​ create_context_aggregator(context: OpenAILLMContext, user_params: LLMUserAggregatorParams, assistant_params: LLMAssistantAggregatorParams) method Create context aggregators for managing conversation state ​ Frame Types ​ Input Frames ​ InputAudioRawFrame Frame Raw audio data for speech input ​ InputImageRawFrame Frame Raw image data for visual input ​ StartInterruptionFrame Frame Signals start of user interruption ​ UserStartedSpeakingFrame Frame Signals user started speaking ​ UserStoppedSpeakingFrame Frame Signals user stopped speaking ​ OpenAILLMContextFrame Frame Contains conversation context ​ LLMMessagesAppendFrame Frame Adds messages to the conversation ​ LLMUpdateSettingsFrame Frame Updates LLM settings ​ LLMSetToolsFrame Frame Sets available tools for the LLM ​ Output Frames ​ TTSAudioRawFrame Frame Generated speech audio ​ TTSStartedFrame Frame Signals start of speech synthesis ​ TTSStoppedFrame Frame Signals end of speech synthesis ​ LLMTextFrame Frame Generated text responses from the LLM ​ TTSTextFrame Frame Text used for speech synthesis ​ TranscriptionFrame Frame Speech transcriptions from user audio ​ LLMFullResponseStartFrame Frame Signals the start of a complete LLM response ​ LLMFullResponseEndFrame Frame Signals the end of a complete LLM response ​ Function Calling This service supports function calling (also known as tool calling) which allows the LLM to request information from external services and APIs. For example, you can enable your bot to: Check current weather conditions Query databases Access external APIs Perform custom actions See the Function Calling guide for: Detailed implementation instructions Provider-specific function definitions Handler registration examples Control over function call behavior Complete usage examples ​ Token Usage Tracking Gemini Multimodal Live automatically tracks token usage metrics, providing: Prompt token counts Completion token counts Total token counts Detailed token breakdowns by modality (text, audio) These metrics can be used for monitoring usage, optimizing costs, and understanding model performance. ​ Language Support Gemini Multimodal Live supports the following languages: Language Code Description Gemini Code Language.AR Arabic ar-XA Language.BN_IN Bengali (India) bn-IN Language.CMN_CN Chinese (Mandarin) cmn-CN Language.DE_DE German (Germany) de-DE Language.EN_US English (US) en-US Language.EN_AU English (Australia) en-AU Language.EN_GB English (UK) en-GB Language.EN_IN English (India) en-IN Language.ES_ES Spanish (Spain) es-ES Language.ES_US Spanish (US) es-US Language.FR_FR French (France) fr-FR Language.FR_CA French (Canada) fr-CA Language.GU_IN Gujarati (India) gu-IN Language.HI_IN Hindi (India) hi-IN Language.ID_ID Indonesian id-ID Language.IT_IT Italian (Italy) it-IT Language.JA_JP Japanese (Japan) ja-JP Language.KN_IN Kannada (India) kn-IN Language.KO_KR Korean (Korea) ko-KR Language.ML_IN Malayalam (India) ml-IN Language.MR_IN Marathi (India) mr-IN Language.NL_NL Dutch (Netherlands) nl-NL Language.PL_PL Polish (Poland) pl-PL Language.PT_BR Portuguese (Brazil) pt-BR Language.RU_RU Russian (Russia) ru-RU Language.TA_IN Tamil (India) ta-IN Language.TE_IN Telugu (India) te-IN Language.TH_TH Thai (Thailand) th-TH Language.TR_TR Turkish (Turkey) tr-TR Language.VI_VN Vietnamese (Vietnam) vi-VN You can set the language using the language parameter: Copy Ask AI from pipecat.transcriptions.language import Language from pipecat.services.gemini_multimodal_live.gemini import ( GeminiMultimodalLiveLLMService, InputParams ) # Set language during initialization llm = GeminiMultimodalLiveLLMService( api_key = os.getenv( "GOOGLE_API_KEY" ), params = InputParams( language = Language. ES_ES ) # Spanish (Spain) ) ​ Next Steps ​ Examples Foundational Example Basic implementation showing core features and transcription Simple Chatbot A client/server example showing how to build a Pipecat JS or React client that connects to a Gemini Live Pipecat bot. ​ Learn More Check out our Gemini Multimodal Live Guide for detailed explanations and best practices. AWS Nova Sonic OpenAI Realtime Beta On this page Installation Basic Usage Configuration Constructor Parameters Input Parameters Methods Frame Types Input Frames Output Frames Function Calling Token Usage Tracking Language Support Next Steps Examples Learn More Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/s2s_gemini_6a1ae4a3.txt b/s2s_gemini_6a1ae4a3.txt
new file mode 100644
index 0000000000000000000000000000000000000000..ce1687e50a0f5c3d199874dc49e8684708feae23
--- /dev/null
+++ b/s2s_gemini_6a1ae4a3.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/s2s/gemini#param-user-stopped-speaking-frame
+Title: Gemini Multimodal Live - Pipecat
+==================================================
+
+Gemini Multimodal Live - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Speech Gemini Multimodal Live Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech AWS Nova Sonic Gemini Multimodal Live OpenAI Realtime Beta Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline The GeminiMultimodalLiveLLMService enables natural, real-time conversations with Google’s Gemini model. It provides built-in audio transcription, voice activity detection, and context management for creating interactive AI experiences. It provides: Real-time Interaction Stream audio and video in real-time with low latency response times Speech Processing Built-in speech-to-text and text-to-speech capabilities with multiple voice options Voice Activity Detection Automatic detection of speech start/stop for natural conversations Context Management Intelligent handling of conversation history and system instructions Want to start building? Check out our Gemini Multimodal Live Guide . ​ Installation To use GeminiMultimodalLiveLLMService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[google]" You’ll need to set up your Google API key as an environment variable: GOOGLE_API_KEY . ​ Basic Usage Here’s a simple example of setting up a conversational AI bot with Gemini Multimodal Live: Copy Ask AI from pipecat.services.gemini_multimodal_live.gemini import ( GeminiMultimodalLiveLLMService, InputParams, GeminiMultimodalModalities ) llm = GeminiMultimodalLiveLLMService( api_key = os.getenv( "GOOGLE_API_KEY" ), voice_id = "Aoede" , # Voices: Aoede, Charon, Fenrir, Kore, Puck params = InputParams( temperature = 0.7 , # Set model input params language = Language. EN_US , # Set language (30+ languages supported) modalities = GeminiMultimodalModalities. AUDIO # Response modality ) ) ​ Configuration ​ Constructor Parameters ​ api_key str required Your Google API key ​ base_url str API endpoint URL ​ model str Gemini model to use (upgraded to new v1beta model) ​ voice_id str default: "Charon" Voice for text-to-speech (options: Aoede, Charon, Fenrir, Kore, Puck) Copy Ask AI llm = GeminiMultimodalLiveLLMService( api_key = os.getenv( "GOOGLE_API_KEY" ), voice_id = "Puck" , # Choose your preferred voice ) ​ system_instruction str High-level instructions that guide the model’s behavior Copy Ask AI llm = GeminiMultimodalLiveLLMService( api_key = os.getenv( "GOOGLE_API_KEY" ), system_instruction = "Talk like a pirate." , ) ​ start_audio_paused bool default: "False" Whether to start with audio input paused ​ start_video_paused bool default: "False" Whether to start with video input paused ​ tools Union[List[dict], ToolsSchema] Tools/functions available to the model ​ inference_on_context_initialization bool default: "True" Whether to generate a response when context is first set ​ Input Parameters ​ frequency_penalty float default: "None" Penalizes repeated tokens. Range: 0.0 to 2.0 ​ max_tokens int default: "4096" Maximum number of tokens to generate ​ modalities GeminiMultimodalModalities default: "AUDIO" Response modalities to include (options: AUDIO , TEXT ). ​ presence_penalty float default: "None" Penalizes tokens based on their presence in the text. Range: 0.0 to 2.0 ​ temperature float default: "None" Controls randomness in responses. Range: 0.0 to 2.0 ​ language Language default: "Language.EN_US" Language for generation. Over 30 languages are supported. ​ media_resolution GeminiMediaResolution default: "UNSPECIFIED" Controls image processing quality and token usage: LOW : Uses 64 tokens MEDIUM : Uses 256 tokens HIGH : Zoomed reframing with 256 tokens ​ vad GeminiVADParams Voice Activity Detection configuration: disabled : Toggle VAD on/off start_sensitivity : How quickly speech is detected (HIGH/LOW) end_sensitivity : How quickly turns end after pauses (HIGH/LOW) prefix_padding_ms : Milliseconds of audio to keep before speech silence_duration_ms : Milliseconds of silence to end a turn Copy Ask AI from pipecat.services.gemini_multimodal_live.events import ( StartSensitivity, EndSensitivity ) from pipecat.services.gemini_multimodal_live.gemini import ( GeminiVADParams, GeminiMediaResolution, ) llm = GeminiMultimodalLiveLLMService( api_key = os.getenv( "GOOGLE_API_KEY" ), params = InputParams( temperature = 0.7 , language = Language. ES , # Spanish language media_resolution = GeminiMediaResolution. HIGH , # Higher quality image processing vad = GeminiVADParams( start_sensitivity = StartSensitivity. HIGH , # Detect speech quickly end_sensitivity = EndSensitivity. LOW , # Allow longer pauses prefix_padding_ms = 300 , # Keep 300ms before speech silence_duration_ms = 1000 , # End turn after 1s silence ) ) ) ​ top_k int default: "None" Limits vocabulary to k most likely tokens. Minimum: 0 ​ top_p float default: "None" Cumulative probability cutoff for token selection. Range: 0.0 to 1.0 ​ context_window_compression ContextWindowCompressionParams Parameters for managing the context window: - enabled : Enable/disable compression (default: False) - trigger_tokens : Number of tokens that trigger compression (default: None, uses 80% of context window) Copy Ask AI from pipecat.services.gemini_multimodal_live.gemini import ( ContextWindowCompressionParams ) llm = GeminiMultimodalLiveLLMService( api_key = os.getenv( "GOOGLE_API_KEY" ), params = InputParams( top_p = 0.9 , # More focused token selection top_k = 40 , # Limit vocabulary options context_window_compression = ContextWindowCompressionParams( enabled = True , trigger_tokens = 8000 # Compress when reaching 8000 tokens ) ) ) ​ Methods ​ set_audio_input_paused(paused: bool) method Pause or unpause audio input processing ​ set_video_input_paused(paused: bool) method Pause or unpause video input processing ​ set_model_modalities(modalities: GeminiMultimodalModalities) method Change the response modality (TEXT or AUDIO) ​ set_language(language: Language) method Change the language for generation ​ set_context(context: OpenAILLMContext) method Set the conversation context explicitly ​ create_context_aggregator(context: OpenAILLMContext, user_params: LLMUserAggregatorParams, assistant_params: LLMAssistantAggregatorParams) method Create context aggregators for managing conversation state ​ Frame Types ​ Input Frames ​ InputAudioRawFrame Frame Raw audio data for speech input ​ InputImageRawFrame Frame Raw image data for visual input ​ StartInterruptionFrame Frame Signals start of user interruption ​ UserStartedSpeakingFrame Frame Signals user started speaking ​ UserStoppedSpeakingFrame Frame Signals user stopped speaking ​ OpenAILLMContextFrame Frame Contains conversation context ​ LLMMessagesAppendFrame Frame Adds messages to the conversation ​ LLMUpdateSettingsFrame Frame Updates LLM settings ​ LLMSetToolsFrame Frame Sets available tools for the LLM ​ Output Frames ​ TTSAudioRawFrame Frame Generated speech audio ​ TTSStartedFrame Frame Signals start of speech synthesis ​ TTSStoppedFrame Frame Signals end of speech synthesis ​ LLMTextFrame Frame Generated text responses from the LLM ​ TTSTextFrame Frame Text used for speech synthesis ​ TranscriptionFrame Frame Speech transcriptions from user audio ​ LLMFullResponseStartFrame Frame Signals the start of a complete LLM response ​ LLMFullResponseEndFrame Frame Signals the end of a complete LLM response ​ Function Calling This service supports function calling (also known as tool calling) which allows the LLM to request information from external services and APIs. For example, you can enable your bot to: Check current weather conditions Query databases Access external APIs Perform custom actions See the Function Calling guide for: Detailed implementation instructions Provider-specific function definitions Handler registration examples Control over function call behavior Complete usage examples ​ Token Usage Tracking Gemini Multimodal Live automatically tracks token usage metrics, providing: Prompt token counts Completion token counts Total token counts Detailed token breakdowns by modality (text, audio) These metrics can be used for monitoring usage, optimizing costs, and understanding model performance. ​ Language Support Gemini Multimodal Live supports the following languages: Language Code Description Gemini Code Language.AR Arabic ar-XA Language.BN_IN Bengali (India) bn-IN Language.CMN_CN Chinese (Mandarin) cmn-CN Language.DE_DE German (Germany) de-DE Language.EN_US English (US) en-US Language.EN_AU English (Australia) en-AU Language.EN_GB English (UK) en-GB Language.EN_IN English (India) en-IN Language.ES_ES Spanish (Spain) es-ES Language.ES_US Spanish (US) es-US Language.FR_FR French (France) fr-FR Language.FR_CA French (Canada) fr-CA Language.GU_IN Gujarati (India) gu-IN Language.HI_IN Hindi (India) hi-IN Language.ID_ID Indonesian id-ID Language.IT_IT Italian (Italy) it-IT Language.JA_JP Japanese (Japan) ja-JP Language.KN_IN Kannada (India) kn-IN Language.KO_KR Korean (Korea) ko-KR Language.ML_IN Malayalam (India) ml-IN Language.MR_IN Marathi (India) mr-IN Language.NL_NL Dutch (Netherlands) nl-NL Language.PL_PL Polish (Poland) pl-PL Language.PT_BR Portuguese (Brazil) pt-BR Language.RU_RU Russian (Russia) ru-RU Language.TA_IN Tamil (India) ta-IN Language.TE_IN Telugu (India) te-IN Language.TH_TH Thai (Thailand) th-TH Language.TR_TR Turkish (Turkey) tr-TR Language.VI_VN Vietnamese (Vietnam) vi-VN You can set the language using the language parameter: Copy Ask AI from pipecat.transcriptions.language import Language from pipecat.services.gemini_multimodal_live.gemini import ( GeminiMultimodalLiveLLMService, InputParams ) # Set language during initialization llm = GeminiMultimodalLiveLLMService( api_key = os.getenv( "GOOGLE_API_KEY" ), params = InputParams( language = Language. ES_ES ) # Spanish (Spain) ) ​ Next Steps ​ Examples Foundational Example Basic implementation showing core features and transcription Simple Chatbot A client/server example showing how to build a Pipecat JS or React client that connects to a Gemini Live Pipecat bot. ​ Learn More Check out our Gemini Multimodal Live Guide for detailed explanations and best practices. AWS Nova Sonic OpenAI Realtime Beta On this page Installation Basic Usage Configuration Constructor Parameters Input Parameters Methods Frame Types Input Frames Output Frames Function Calling Token Usage Tracking Language Support Next Steps Examples Learn More Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/s2s_gemini_7df49d9e.txt b/s2s_gemini_7df49d9e.txt
new file mode 100644
index 0000000000000000000000000000000000000000..9ba6a09ab4d2ce852512b1b343bf0ffa730a6c36
--- /dev/null
+++ b/s2s_gemini_7df49d9e.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/s2s/gemini#param-frequency-penalty
+Title: Gemini Multimodal Live - Pipecat
+==================================================
+
+Gemini Multimodal Live - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Speech Gemini Multimodal Live Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech AWS Nova Sonic Gemini Multimodal Live OpenAI Realtime Beta Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline The GeminiMultimodalLiveLLMService enables natural, real-time conversations with Google’s Gemini model. It provides built-in audio transcription, voice activity detection, and context management for creating interactive AI experiences. It provides: Real-time Interaction Stream audio and video in real-time with low latency response times Speech Processing Built-in speech-to-text and text-to-speech capabilities with multiple voice options Voice Activity Detection Automatic detection of speech start/stop for natural conversations Context Management Intelligent handling of conversation history and system instructions Want to start building? Check out our Gemini Multimodal Live Guide . ​ Installation To use GeminiMultimodalLiveLLMService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[google]" You’ll need to set up your Google API key as an environment variable: GOOGLE_API_KEY . ​ Basic Usage Here’s a simple example of setting up a conversational AI bot with Gemini Multimodal Live: Copy Ask AI from pipecat.services.gemini_multimodal_live.gemini import ( GeminiMultimodalLiveLLMService, InputParams, GeminiMultimodalModalities ) llm = GeminiMultimodalLiveLLMService( api_key = os.getenv( "GOOGLE_API_KEY" ), voice_id = "Aoede" , # Voices: Aoede, Charon, Fenrir, Kore, Puck params = InputParams( temperature = 0.7 , # Set model input params language = Language. EN_US , # Set language (30+ languages supported) modalities = GeminiMultimodalModalities. AUDIO # Response modality ) ) ​ Configuration ​ Constructor Parameters ​ api_key str required Your Google API key ​ base_url str API endpoint URL ​ model str Gemini model to use (upgraded to new v1beta model) ​ voice_id str default: "Charon" Voice for text-to-speech (options: Aoede, Charon, Fenrir, Kore, Puck) Copy Ask AI llm = GeminiMultimodalLiveLLMService( api_key = os.getenv( "GOOGLE_API_KEY" ), voice_id = "Puck" , # Choose your preferred voice ) ​ system_instruction str High-level instructions that guide the model’s behavior Copy Ask AI llm = GeminiMultimodalLiveLLMService( api_key = os.getenv( "GOOGLE_API_KEY" ), system_instruction = "Talk like a pirate." , ) ​ start_audio_paused bool default: "False" Whether to start with audio input paused ​ start_video_paused bool default: "False" Whether to start with video input paused ​ tools Union[List[dict], ToolsSchema] Tools/functions available to the model ​ inference_on_context_initialization bool default: "True" Whether to generate a response when context is first set ​ Input Parameters ​ frequency_penalty float default: "None" Penalizes repeated tokens. Range: 0.0 to 2.0 ​ max_tokens int default: "4096" Maximum number of tokens to generate ​ modalities GeminiMultimodalModalities default: "AUDIO" Response modalities to include (options: AUDIO , TEXT ). ​ presence_penalty float default: "None" Penalizes tokens based on their presence in the text. Range: 0.0 to 2.0 ​ temperature float default: "None" Controls randomness in responses. Range: 0.0 to 2.0 ​ language Language default: "Language.EN_US" Language for generation. Over 30 languages are supported. ​ media_resolution GeminiMediaResolution default: "UNSPECIFIED" Controls image processing quality and token usage: LOW : Uses 64 tokens MEDIUM : Uses 256 tokens HIGH : Zoomed reframing with 256 tokens ​ vad GeminiVADParams Voice Activity Detection configuration: disabled : Toggle VAD on/off start_sensitivity : How quickly speech is detected (HIGH/LOW) end_sensitivity : How quickly turns end after pauses (HIGH/LOW) prefix_padding_ms : Milliseconds of audio to keep before speech silence_duration_ms : Milliseconds of silence to end a turn Copy Ask AI from pipecat.services.gemini_multimodal_live.events import ( StartSensitivity, EndSensitivity ) from pipecat.services.gemini_multimodal_live.gemini import ( GeminiVADParams, GeminiMediaResolution, ) llm = GeminiMultimodalLiveLLMService( api_key = os.getenv( "GOOGLE_API_KEY" ), params = InputParams( temperature = 0.7 , language = Language. ES , # Spanish language media_resolution = GeminiMediaResolution. HIGH , # Higher quality image processing vad = GeminiVADParams( start_sensitivity = StartSensitivity. HIGH , # Detect speech quickly end_sensitivity = EndSensitivity. LOW , # Allow longer pauses prefix_padding_ms = 300 , # Keep 300ms before speech silence_duration_ms = 1000 , # End turn after 1s silence ) ) ) ​ top_k int default: "None" Limits vocabulary to k most likely tokens. Minimum: 0 ​ top_p float default: "None" Cumulative probability cutoff for token selection. Range: 0.0 to 1.0 ​ context_window_compression ContextWindowCompressionParams Parameters for managing the context window: - enabled : Enable/disable compression (default: False) - trigger_tokens : Number of tokens that trigger compression (default: None, uses 80% of context window) Copy Ask AI from pipecat.services.gemini_multimodal_live.gemini import ( ContextWindowCompressionParams ) llm = GeminiMultimodalLiveLLMService( api_key = os.getenv( "GOOGLE_API_KEY" ), params = InputParams( top_p = 0.9 , # More focused token selection top_k = 40 , # Limit vocabulary options context_window_compression = ContextWindowCompressionParams( enabled = True , trigger_tokens = 8000 # Compress when reaching 8000 tokens ) ) ) ​ Methods ​ set_audio_input_paused(paused: bool) method Pause or unpause audio input processing ​ set_video_input_paused(paused: bool) method Pause or unpause video input processing ​ set_model_modalities(modalities: GeminiMultimodalModalities) method Change the response modality (TEXT or AUDIO) ​ set_language(language: Language) method Change the language for generation ​ set_context(context: OpenAILLMContext) method Set the conversation context explicitly ​ create_context_aggregator(context: OpenAILLMContext, user_params: LLMUserAggregatorParams, assistant_params: LLMAssistantAggregatorParams) method Create context aggregators for managing conversation state ​ Frame Types ​ Input Frames ​ InputAudioRawFrame Frame Raw audio data for speech input ​ InputImageRawFrame Frame Raw image data for visual input ​ StartInterruptionFrame Frame Signals start of user interruption ​ UserStartedSpeakingFrame Frame Signals user started speaking ​ UserStoppedSpeakingFrame Frame Signals user stopped speaking ​ OpenAILLMContextFrame Frame Contains conversation context ​ LLMMessagesAppendFrame Frame Adds messages to the conversation ​ LLMUpdateSettingsFrame Frame Updates LLM settings ​ LLMSetToolsFrame Frame Sets available tools for the LLM ​ Output Frames ​ TTSAudioRawFrame Frame Generated speech audio ​ TTSStartedFrame Frame Signals start of speech synthesis ​ TTSStoppedFrame Frame Signals end of speech synthesis ​ LLMTextFrame Frame Generated text responses from the LLM ​ TTSTextFrame Frame Text used for speech synthesis ​ TranscriptionFrame Frame Speech transcriptions from user audio ​ LLMFullResponseStartFrame Frame Signals the start of a complete LLM response ​ LLMFullResponseEndFrame Frame Signals the end of a complete LLM response ​ Function Calling This service supports function calling (also known as tool calling) which allows the LLM to request information from external services and APIs. For example, you can enable your bot to: Check current weather conditions Query databases Access external APIs Perform custom actions See the Function Calling guide for: Detailed implementation instructions Provider-specific function definitions Handler registration examples Control over function call behavior Complete usage examples ​ Token Usage Tracking Gemini Multimodal Live automatically tracks token usage metrics, providing: Prompt token counts Completion token counts Total token counts Detailed token breakdowns by modality (text, audio) These metrics can be used for monitoring usage, optimizing costs, and understanding model performance. ​ Language Support Gemini Multimodal Live supports the following languages: Language Code Description Gemini Code Language.AR Arabic ar-XA Language.BN_IN Bengali (India) bn-IN Language.CMN_CN Chinese (Mandarin) cmn-CN Language.DE_DE German (Germany) de-DE Language.EN_US English (US) en-US Language.EN_AU English (Australia) en-AU Language.EN_GB English (UK) en-GB Language.EN_IN English (India) en-IN Language.ES_ES Spanish (Spain) es-ES Language.ES_US Spanish (US) es-US Language.FR_FR French (France) fr-FR Language.FR_CA French (Canada) fr-CA Language.GU_IN Gujarati (India) gu-IN Language.HI_IN Hindi (India) hi-IN Language.ID_ID Indonesian id-ID Language.IT_IT Italian (Italy) it-IT Language.JA_JP Japanese (Japan) ja-JP Language.KN_IN Kannada (India) kn-IN Language.KO_KR Korean (Korea) ko-KR Language.ML_IN Malayalam (India) ml-IN Language.MR_IN Marathi (India) mr-IN Language.NL_NL Dutch (Netherlands) nl-NL Language.PL_PL Polish (Poland) pl-PL Language.PT_BR Portuguese (Brazil) pt-BR Language.RU_RU Russian (Russia) ru-RU Language.TA_IN Tamil (India) ta-IN Language.TE_IN Telugu (India) te-IN Language.TH_TH Thai (Thailand) th-TH Language.TR_TR Turkish (Turkey) tr-TR Language.VI_VN Vietnamese (Vietnam) vi-VN You can set the language using the language parameter: Copy Ask AI from pipecat.transcriptions.language import Language from pipecat.services.gemini_multimodal_live.gemini import ( GeminiMultimodalLiveLLMService, InputParams ) # Set language during initialization llm = GeminiMultimodalLiveLLMService( api_key = os.getenv( "GOOGLE_API_KEY" ), params = InputParams( language = Language. ES_ES ) # Spanish (Spain) ) ​ Next Steps ​ Examples Foundational Example Basic implementation showing core features and transcription Simple Chatbot A client/server example showing how to build a Pipecat JS or React client that connects to a Gemini Live Pipecat bot. ​ Learn More Check out our Gemini Multimodal Live Guide for detailed explanations and best practices. AWS Nova Sonic OpenAI Realtime Beta On this page Installation Basic Usage Configuration Constructor Parameters Input Parameters Methods Frame Types Input Frames Output Frames Function Calling Token Usage Tracking Language Support Next Steps Examples Learn More Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/s2s_gemini_a76b192a.txt b/s2s_gemini_a76b192a.txt
new file mode 100644
index 0000000000000000000000000000000000000000..ea9438686906fa42dc1894e8ff751dc38afc47d5
--- /dev/null
+++ b/s2s_gemini_a76b192a.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/s2s/gemini#param-vad
+Title: Gemini Multimodal Live - Pipecat
+==================================================
+
+Gemini Multimodal Live - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Speech Gemini Multimodal Live Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech AWS Nova Sonic Gemini Multimodal Live OpenAI Realtime Beta Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline The GeminiMultimodalLiveLLMService enables natural, real-time conversations with Google’s Gemini model. It provides built-in audio transcription, voice activity detection, and context management for creating interactive AI experiences. It provides: Real-time Interaction Stream audio and video in real-time with low latency response times Speech Processing Built-in speech-to-text and text-to-speech capabilities with multiple voice options Voice Activity Detection Automatic detection of speech start/stop for natural conversations Context Management Intelligent handling of conversation history and system instructions Want to start building? Check out our Gemini Multimodal Live Guide . ​ Installation To use GeminiMultimodalLiveLLMService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[google]" You’ll need to set up your Google API key as an environment variable: GOOGLE_API_KEY . ​ Basic Usage Here’s a simple example of setting up a conversational AI bot with Gemini Multimodal Live: Copy Ask AI from pipecat.services.gemini_multimodal_live.gemini import ( GeminiMultimodalLiveLLMService, InputParams, GeminiMultimodalModalities ) llm = GeminiMultimodalLiveLLMService( api_key = os.getenv( "GOOGLE_API_KEY" ), voice_id = "Aoede" , # Voices: Aoede, Charon, Fenrir, Kore, Puck params = InputParams( temperature = 0.7 , # Set model input params language = Language. EN_US , # Set language (30+ languages supported) modalities = GeminiMultimodalModalities. AUDIO # Response modality ) ) ​ Configuration ​ Constructor Parameters ​ api_key str required Your Google API key ​ base_url str API endpoint URL ​ model str Gemini model to use (upgraded to new v1beta model) ​ voice_id str default: "Charon" Voice for text-to-speech (options: Aoede, Charon, Fenrir, Kore, Puck) Copy Ask AI llm = GeminiMultimodalLiveLLMService( api_key = os.getenv( "GOOGLE_API_KEY" ), voice_id = "Puck" , # Choose your preferred voice ) ​ system_instruction str High-level instructions that guide the model’s behavior Copy Ask AI llm = GeminiMultimodalLiveLLMService( api_key = os.getenv( "GOOGLE_API_KEY" ), system_instruction = "Talk like a pirate." , ) ​ start_audio_paused bool default: "False" Whether to start with audio input paused ​ start_video_paused bool default: "False" Whether to start with video input paused ​ tools Union[List[dict], ToolsSchema] Tools/functions available to the model ​ inference_on_context_initialization bool default: "True" Whether to generate a response when context is first set ​ Input Parameters ​ frequency_penalty float default: "None" Penalizes repeated tokens. Range: 0.0 to 2.0 ​ max_tokens int default: "4096" Maximum number of tokens to generate ​ modalities GeminiMultimodalModalities default: "AUDIO" Response modalities to include (options: AUDIO , TEXT ). ​ presence_penalty float default: "None" Penalizes tokens based on their presence in the text. Range: 0.0 to 2.0 ​ temperature float default: "None" Controls randomness in responses. Range: 0.0 to 2.0 ​ language Language default: "Language.EN_US" Language for generation. Over 30 languages are supported. ​ media_resolution GeminiMediaResolution default: "UNSPECIFIED" Controls image processing quality and token usage: LOW : Uses 64 tokens MEDIUM : Uses 256 tokens HIGH : Zoomed reframing with 256 tokens ​ vad GeminiVADParams Voice Activity Detection configuration: disabled : Toggle VAD on/off start_sensitivity : How quickly speech is detected (HIGH/LOW) end_sensitivity : How quickly turns end after pauses (HIGH/LOW) prefix_padding_ms : Milliseconds of audio to keep before speech silence_duration_ms : Milliseconds of silence to end a turn Copy Ask AI from pipecat.services.gemini_multimodal_live.events import ( StartSensitivity, EndSensitivity ) from pipecat.services.gemini_multimodal_live.gemini import ( GeminiVADParams, GeminiMediaResolution, ) llm = GeminiMultimodalLiveLLMService( api_key = os.getenv( "GOOGLE_API_KEY" ), params = InputParams( temperature = 0.7 , language = Language. ES , # Spanish language media_resolution = GeminiMediaResolution. HIGH , # Higher quality image processing vad = GeminiVADParams( start_sensitivity = StartSensitivity. HIGH , # Detect speech quickly end_sensitivity = EndSensitivity. LOW , # Allow longer pauses prefix_padding_ms = 300 , # Keep 300ms before speech silence_duration_ms = 1000 , # End turn after 1s silence ) ) ) ​ top_k int default: "None" Limits vocabulary to k most likely tokens. Minimum: 0 ​ top_p float default: "None" Cumulative probability cutoff for token selection. Range: 0.0 to 1.0 ​ context_window_compression ContextWindowCompressionParams Parameters for managing the context window: - enabled : Enable/disable compression (default: False) - trigger_tokens : Number of tokens that trigger compression (default: None, uses 80% of context window) Copy Ask AI from pipecat.services.gemini_multimodal_live.gemini import ( ContextWindowCompressionParams ) llm = GeminiMultimodalLiveLLMService( api_key = os.getenv( "GOOGLE_API_KEY" ), params = InputParams( top_p = 0.9 , # More focused token selection top_k = 40 , # Limit vocabulary options context_window_compression = ContextWindowCompressionParams( enabled = True , trigger_tokens = 8000 # Compress when reaching 8000 tokens ) ) ) ​ Methods ​ set_audio_input_paused(paused: bool) method Pause or unpause audio input processing ​ set_video_input_paused(paused: bool) method Pause or unpause video input processing ​ set_model_modalities(modalities: GeminiMultimodalModalities) method Change the response modality (TEXT or AUDIO) ​ set_language(language: Language) method Change the language for generation ​ set_context(context: OpenAILLMContext) method Set the conversation context explicitly ​ create_context_aggregator(context: OpenAILLMContext, user_params: LLMUserAggregatorParams, assistant_params: LLMAssistantAggregatorParams) method Create context aggregators for managing conversation state ​ Frame Types ​ Input Frames ​ InputAudioRawFrame Frame Raw audio data for speech input ​ InputImageRawFrame Frame Raw image data for visual input ​ StartInterruptionFrame Frame Signals start of user interruption ​ UserStartedSpeakingFrame Frame Signals user started speaking ​ UserStoppedSpeakingFrame Frame Signals user stopped speaking ​ OpenAILLMContextFrame Frame Contains conversation context ​ LLMMessagesAppendFrame Frame Adds messages to the conversation ​ LLMUpdateSettingsFrame Frame Updates LLM settings ​ LLMSetToolsFrame Frame Sets available tools for the LLM ​ Output Frames ​ TTSAudioRawFrame Frame Generated speech audio ​ TTSStartedFrame Frame Signals start of speech synthesis ​ TTSStoppedFrame Frame Signals end of speech synthesis ​ LLMTextFrame Frame Generated text responses from the LLM ​ TTSTextFrame Frame Text used for speech synthesis ​ TranscriptionFrame Frame Speech transcriptions from user audio ​ LLMFullResponseStartFrame Frame Signals the start of a complete LLM response ​ LLMFullResponseEndFrame Frame Signals the end of a complete LLM response ​ Function Calling This service supports function calling (also known as tool calling) which allows the LLM to request information from external services and APIs. For example, you can enable your bot to: Check current weather conditions Query databases Access external APIs Perform custom actions See the Function Calling guide for: Detailed implementation instructions Provider-specific function definitions Handler registration examples Control over function call behavior Complete usage examples ​ Token Usage Tracking Gemini Multimodal Live automatically tracks token usage metrics, providing: Prompt token counts Completion token counts Total token counts Detailed token breakdowns by modality (text, audio) These metrics can be used for monitoring usage, optimizing costs, and understanding model performance. ​ Language Support Gemini Multimodal Live supports the following languages: Language Code Description Gemini Code Language.AR Arabic ar-XA Language.BN_IN Bengali (India) bn-IN Language.CMN_CN Chinese (Mandarin) cmn-CN Language.DE_DE German (Germany) de-DE Language.EN_US English (US) en-US Language.EN_AU English (Australia) en-AU Language.EN_GB English (UK) en-GB Language.EN_IN English (India) en-IN Language.ES_ES Spanish (Spain) es-ES Language.ES_US Spanish (US) es-US Language.FR_FR French (France) fr-FR Language.FR_CA French (Canada) fr-CA Language.GU_IN Gujarati (India) gu-IN Language.HI_IN Hindi (India) hi-IN Language.ID_ID Indonesian id-ID Language.IT_IT Italian (Italy) it-IT Language.JA_JP Japanese (Japan) ja-JP Language.KN_IN Kannada (India) kn-IN Language.KO_KR Korean (Korea) ko-KR Language.ML_IN Malayalam (India) ml-IN Language.MR_IN Marathi (India) mr-IN Language.NL_NL Dutch (Netherlands) nl-NL Language.PL_PL Polish (Poland) pl-PL Language.PT_BR Portuguese (Brazil) pt-BR Language.RU_RU Russian (Russia) ru-RU Language.TA_IN Tamil (India) ta-IN Language.TE_IN Telugu (India) te-IN Language.TH_TH Thai (Thailand) th-TH Language.TR_TR Turkish (Turkey) tr-TR Language.VI_VN Vietnamese (Vietnam) vi-VN You can set the language using the language parameter: Copy Ask AI from pipecat.transcriptions.language import Language from pipecat.services.gemini_multimodal_live.gemini import ( GeminiMultimodalLiveLLMService, InputParams ) # Set language during initialization llm = GeminiMultimodalLiveLLMService( api_key = os.getenv( "GOOGLE_API_KEY" ), params = InputParams( language = Language. ES_ES ) # Spanish (Spain) ) ​ Next Steps ​ Examples Foundational Example Basic implementation showing core features and transcription Simple Chatbot A client/server example showing how to build a Pipecat JS or React client that connects to a Gemini Live Pipecat bot. ​ Learn More Check out our Gemini Multimodal Live Guide for detailed explanations and best practices. AWS Nova Sonic OpenAI Realtime Beta On this page Installation Basic Usage Configuration Constructor Parameters Input Parameters Methods Frame Types Input Frames Output Frames Function Calling Token Usage Tracking Language Support Next Steps Examples Learn More Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/s2s_gemini_b6a8c59e.txt b/s2s_gemini_b6a8c59e.txt
new file mode 100644
index 0000000000000000000000000000000000000000..32b346de85846be8fef88f94332f5f217c928f42
--- /dev/null
+++ b/s2s_gemini_b6a8c59e.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/s2s/gemini#param-input-audio-raw-frame
+Title: Gemini Multimodal Live - Pipecat
+==================================================
+
+Gemini Multimodal Live - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Speech Gemini Multimodal Live Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech AWS Nova Sonic Gemini Multimodal Live OpenAI Realtime Beta Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline The GeminiMultimodalLiveLLMService enables natural, real-time conversations with Google’s Gemini model. It provides built-in audio transcription, voice activity detection, and context management for creating interactive AI experiences. It provides: Real-time Interaction Stream audio and video in real-time with low latency response times Speech Processing Built-in speech-to-text and text-to-speech capabilities with multiple voice options Voice Activity Detection Automatic detection of speech start/stop for natural conversations Context Management Intelligent handling of conversation history and system instructions Want to start building? Check out our Gemini Multimodal Live Guide . ​ Installation To use GeminiMultimodalLiveLLMService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[google]" You’ll need to set up your Google API key as an environment variable: GOOGLE_API_KEY . ​ Basic Usage Here’s a simple example of setting up a conversational AI bot with Gemini Multimodal Live: Copy Ask AI from pipecat.services.gemini_multimodal_live.gemini import ( GeminiMultimodalLiveLLMService, InputParams, GeminiMultimodalModalities ) llm = GeminiMultimodalLiveLLMService( api_key = os.getenv( "GOOGLE_API_KEY" ), voice_id = "Aoede" , # Voices: Aoede, Charon, Fenrir, Kore, Puck params = InputParams( temperature = 0.7 , # Set model input params language = Language. EN_US , # Set language (30+ languages supported) modalities = GeminiMultimodalModalities. AUDIO # Response modality ) ) ​ Configuration ​ Constructor Parameters ​ api_key str required Your Google API key ​ base_url str API endpoint URL ​ model str Gemini model to use (upgraded to new v1beta model) ​ voice_id str default: "Charon" Voice for text-to-speech (options: Aoede, Charon, Fenrir, Kore, Puck) Copy Ask AI llm = GeminiMultimodalLiveLLMService( api_key = os.getenv( "GOOGLE_API_KEY" ), voice_id = "Puck" , # Choose your preferred voice ) ​ system_instruction str High-level instructions that guide the model’s behavior Copy Ask AI llm = GeminiMultimodalLiveLLMService( api_key = os.getenv( "GOOGLE_API_KEY" ), system_instruction = "Talk like a pirate." , ) ​ start_audio_paused bool default: "False" Whether to start with audio input paused ​ start_video_paused bool default: "False" Whether to start with video input paused ​ tools Union[List[dict], ToolsSchema] Tools/functions available to the model ​ inference_on_context_initialization bool default: "True" Whether to generate a response when context is first set ​ Input Parameters ​ frequency_penalty float default: "None" Penalizes repeated tokens. Range: 0.0 to 2.0 ​ max_tokens int default: "4096" Maximum number of tokens to generate ​ modalities GeminiMultimodalModalities default: "AUDIO" Response modalities to include (options: AUDIO , TEXT ). ​ presence_penalty float default: "None" Penalizes tokens based on their presence in the text. Range: 0.0 to 2.0 ​ temperature float default: "None" Controls randomness in responses. Range: 0.0 to 2.0 ​ language Language default: "Language.EN_US" Language for generation. Over 30 languages are supported. ​ media_resolution GeminiMediaResolution default: "UNSPECIFIED" Controls image processing quality and token usage: LOW : Uses 64 tokens MEDIUM : Uses 256 tokens HIGH : Zoomed reframing with 256 tokens ​ vad GeminiVADParams Voice Activity Detection configuration: disabled : Toggle VAD on/off start_sensitivity : How quickly speech is detected (HIGH/LOW) end_sensitivity : How quickly turns end after pauses (HIGH/LOW) prefix_padding_ms : Milliseconds of audio to keep before speech silence_duration_ms : Milliseconds of silence to end a turn Copy Ask AI from pipecat.services.gemini_multimodal_live.events import ( StartSensitivity, EndSensitivity ) from pipecat.services.gemini_multimodal_live.gemini import ( GeminiVADParams, GeminiMediaResolution, ) llm = GeminiMultimodalLiveLLMService( api_key = os.getenv( "GOOGLE_API_KEY" ), params = InputParams( temperature = 0.7 , language = Language. ES , # Spanish language media_resolution = GeminiMediaResolution. HIGH , # Higher quality image processing vad = GeminiVADParams( start_sensitivity = StartSensitivity. HIGH , # Detect speech quickly end_sensitivity = EndSensitivity. LOW , # Allow longer pauses prefix_padding_ms = 300 , # Keep 300ms before speech silence_duration_ms = 1000 , # End turn after 1s silence ) ) ) ​ top_k int default: "None" Limits vocabulary to k most likely tokens. Minimum: 0 ​ top_p float default: "None" Cumulative probability cutoff for token selection. Range: 0.0 to 1.0 ​ context_window_compression ContextWindowCompressionParams Parameters for managing the context window: - enabled : Enable/disable compression (default: False) - trigger_tokens : Number of tokens that trigger compression (default: None, uses 80% of context window) Copy Ask AI from pipecat.services.gemini_multimodal_live.gemini import ( ContextWindowCompressionParams ) llm = GeminiMultimodalLiveLLMService( api_key = os.getenv( "GOOGLE_API_KEY" ), params = InputParams( top_p = 0.9 , # More focused token selection top_k = 40 , # Limit vocabulary options context_window_compression = ContextWindowCompressionParams( enabled = True , trigger_tokens = 8000 # Compress when reaching 8000 tokens ) ) ) ​ Methods ​ set_audio_input_paused(paused: bool) method Pause or unpause audio input processing ​ set_video_input_paused(paused: bool) method Pause or unpause video input processing ​ set_model_modalities(modalities: GeminiMultimodalModalities) method Change the response modality (TEXT or AUDIO) ​ set_language(language: Language) method Change the language for generation ​ set_context(context: OpenAILLMContext) method Set the conversation context explicitly ​ create_context_aggregator(context: OpenAILLMContext, user_params: LLMUserAggregatorParams, assistant_params: LLMAssistantAggregatorParams) method Create context aggregators for managing conversation state ​ Frame Types ​ Input Frames ​ InputAudioRawFrame Frame Raw audio data for speech input ​ InputImageRawFrame Frame Raw image data for visual input ​ StartInterruptionFrame Frame Signals start of user interruption ​ UserStartedSpeakingFrame Frame Signals user started speaking ​ UserStoppedSpeakingFrame Frame Signals user stopped speaking ​ OpenAILLMContextFrame Frame Contains conversation context ​ LLMMessagesAppendFrame Frame Adds messages to the conversation ​ LLMUpdateSettingsFrame Frame Updates LLM settings ​ LLMSetToolsFrame Frame Sets available tools for the LLM ​ Output Frames ​ TTSAudioRawFrame Frame Generated speech audio ​ TTSStartedFrame Frame Signals start of speech synthesis ​ TTSStoppedFrame Frame Signals end of speech synthesis ​ LLMTextFrame Frame Generated text responses from the LLM ​ TTSTextFrame Frame Text used for speech synthesis ​ TranscriptionFrame Frame Speech transcriptions from user audio ​ LLMFullResponseStartFrame Frame Signals the start of a complete LLM response ​ LLMFullResponseEndFrame Frame Signals the end of a complete LLM response ​ Function Calling This service supports function calling (also known as tool calling) which allows the LLM to request information from external services and APIs. For example, you can enable your bot to: Check current weather conditions Query databases Access external APIs Perform custom actions See the Function Calling guide for: Detailed implementation instructions Provider-specific function definitions Handler registration examples Control over function call behavior Complete usage examples ​ Token Usage Tracking Gemini Multimodal Live automatically tracks token usage metrics, providing: Prompt token counts Completion token counts Total token counts Detailed token breakdowns by modality (text, audio) These metrics can be used for monitoring usage, optimizing costs, and understanding model performance. ​ Language Support Gemini Multimodal Live supports the following languages: Language Code Description Gemini Code Language.AR Arabic ar-XA Language.BN_IN Bengali (India) bn-IN Language.CMN_CN Chinese (Mandarin) cmn-CN Language.DE_DE German (Germany) de-DE Language.EN_US English (US) en-US Language.EN_AU English (Australia) en-AU Language.EN_GB English (UK) en-GB Language.EN_IN English (India) en-IN Language.ES_ES Spanish (Spain) es-ES Language.ES_US Spanish (US) es-US Language.FR_FR French (France) fr-FR Language.FR_CA French (Canada) fr-CA Language.GU_IN Gujarati (India) gu-IN Language.HI_IN Hindi (India) hi-IN Language.ID_ID Indonesian id-ID Language.IT_IT Italian (Italy) it-IT Language.JA_JP Japanese (Japan) ja-JP Language.KN_IN Kannada (India) kn-IN Language.KO_KR Korean (Korea) ko-KR Language.ML_IN Malayalam (India) ml-IN Language.MR_IN Marathi (India) mr-IN Language.NL_NL Dutch (Netherlands) nl-NL Language.PL_PL Polish (Poland) pl-PL Language.PT_BR Portuguese (Brazil) pt-BR Language.RU_RU Russian (Russia) ru-RU Language.TA_IN Tamil (India) ta-IN Language.TE_IN Telugu (India) te-IN Language.TH_TH Thai (Thailand) th-TH Language.TR_TR Turkish (Turkey) tr-TR Language.VI_VN Vietnamese (Vietnam) vi-VN You can set the language using the language parameter: Copy Ask AI from pipecat.transcriptions.language import Language from pipecat.services.gemini_multimodal_live.gemini import ( GeminiMultimodalLiveLLMService, InputParams ) # Set language during initialization llm = GeminiMultimodalLiveLLMService( api_key = os.getenv( "GOOGLE_API_KEY" ), params = InputParams( language = Language. ES_ES ) # Spanish (Spain) ) ​ Next Steps ​ Examples Foundational Example Basic implementation showing core features and transcription Simple Chatbot A client/server example showing how to build a Pipecat JS or React client that connects to a Gemini Live Pipecat bot. ​ Learn More Check out our Gemini Multimodal Live Guide for detailed explanations and best practices. AWS Nova Sonic OpenAI Realtime Beta On this page Installation Basic Usage Configuration Constructor Parameters Input Parameters Methods Frame Types Input Frames Output Frames Function Calling Token Usage Tracking Language Support Next Steps Examples Learn More Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/s2s_gemini_c53cde02.txt b/s2s_gemini_c53cde02.txt
new file mode 100644
index 0000000000000000000000000000000000000000..4892ff13dbd80034ea8b3b8a4627bcd3d1647398
--- /dev/null
+++ b/s2s_gemini_c53cde02.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/s2s/gemini#param-context-window-compression
+Title: Gemini Multimodal Live - Pipecat
+==================================================
+
+Gemini Multimodal Live - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Speech Gemini Multimodal Live Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech AWS Nova Sonic Gemini Multimodal Live OpenAI Realtime Beta Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline The GeminiMultimodalLiveLLMService enables natural, real-time conversations with Google’s Gemini model. It provides built-in audio transcription, voice activity detection, and context management for creating interactive AI experiences. It provides: Real-time Interaction Stream audio and video in real-time with low latency response times Speech Processing Built-in speech-to-text and text-to-speech capabilities with multiple voice options Voice Activity Detection Automatic detection of speech start/stop for natural conversations Context Management Intelligent handling of conversation history and system instructions Want to start building? Check out our Gemini Multimodal Live Guide . ​ Installation To use GeminiMultimodalLiveLLMService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[google]" You’ll need to set up your Google API key as an environment variable: GOOGLE_API_KEY . ​ Basic Usage Here’s a simple example of setting up a conversational AI bot with Gemini Multimodal Live: Copy Ask AI from pipecat.services.gemini_multimodal_live.gemini import ( GeminiMultimodalLiveLLMService, InputParams, GeminiMultimodalModalities ) llm = GeminiMultimodalLiveLLMService( api_key = os.getenv( "GOOGLE_API_KEY" ), voice_id = "Aoede" , # Voices: Aoede, Charon, Fenrir, Kore, Puck params = InputParams( temperature = 0.7 , # Set model input params language = Language. EN_US , # Set language (30+ languages supported) modalities = GeminiMultimodalModalities. AUDIO # Response modality ) ) ​ Configuration ​ Constructor Parameters ​ api_key str required Your Google API key ​ base_url str API endpoint URL ​ model str Gemini model to use (upgraded to new v1beta model) ​ voice_id str default: "Charon" Voice for text-to-speech (options: Aoede, Charon, Fenrir, Kore, Puck) Copy Ask AI llm = GeminiMultimodalLiveLLMService( api_key = os.getenv( "GOOGLE_API_KEY" ), voice_id = "Puck" , # Choose your preferred voice ) ​ system_instruction str High-level instructions that guide the model’s behavior Copy Ask AI llm = GeminiMultimodalLiveLLMService( api_key = os.getenv( "GOOGLE_API_KEY" ), system_instruction = "Talk like a pirate." , ) ​ start_audio_paused bool default: "False" Whether to start with audio input paused ​ start_video_paused bool default: "False" Whether to start with video input paused ​ tools Union[List[dict], ToolsSchema] Tools/functions available to the model ​ inference_on_context_initialization bool default: "True" Whether to generate a response when context is first set ​ Input Parameters ​ frequency_penalty float default: "None" Penalizes repeated tokens. Range: 0.0 to 2.0 ​ max_tokens int default: "4096" Maximum number of tokens to generate ​ modalities GeminiMultimodalModalities default: "AUDIO" Response modalities to include (options: AUDIO , TEXT ). ​ presence_penalty float default: "None" Penalizes tokens based on their presence in the text. Range: 0.0 to 2.0 ​ temperature float default: "None" Controls randomness in responses. Range: 0.0 to 2.0 ​ language Language default: "Language.EN_US" Language for generation. Over 30 languages are supported. ​ media_resolution GeminiMediaResolution default: "UNSPECIFIED" Controls image processing quality and token usage: LOW : Uses 64 tokens MEDIUM : Uses 256 tokens HIGH : Zoomed reframing with 256 tokens ​ vad GeminiVADParams Voice Activity Detection configuration: disabled : Toggle VAD on/off start_sensitivity : How quickly speech is detected (HIGH/LOW) end_sensitivity : How quickly turns end after pauses (HIGH/LOW) prefix_padding_ms : Milliseconds of audio to keep before speech silence_duration_ms : Milliseconds of silence to end a turn Copy Ask AI from pipecat.services.gemini_multimodal_live.events import ( StartSensitivity, EndSensitivity ) from pipecat.services.gemini_multimodal_live.gemini import ( GeminiVADParams, GeminiMediaResolution, ) llm = GeminiMultimodalLiveLLMService( api_key = os.getenv( "GOOGLE_API_KEY" ), params = InputParams( temperature = 0.7 , language = Language. ES , # Spanish language media_resolution = GeminiMediaResolution. HIGH , # Higher quality image processing vad = GeminiVADParams( start_sensitivity = StartSensitivity. HIGH , # Detect speech quickly end_sensitivity = EndSensitivity. LOW , # Allow longer pauses prefix_padding_ms = 300 , # Keep 300ms before speech silence_duration_ms = 1000 , # End turn after 1s silence ) ) ) ​ top_k int default: "None" Limits vocabulary to k most likely tokens. Minimum: 0 ​ top_p float default: "None" Cumulative probability cutoff for token selection. Range: 0.0 to 1.0 ​ context_window_compression ContextWindowCompressionParams Parameters for managing the context window: - enabled : Enable/disable compression (default: False) - trigger_tokens : Number of tokens that trigger compression (default: None, uses 80% of context window) Copy Ask AI from pipecat.services.gemini_multimodal_live.gemini import ( ContextWindowCompressionParams ) llm = GeminiMultimodalLiveLLMService( api_key = os.getenv( "GOOGLE_API_KEY" ), params = InputParams( top_p = 0.9 , # More focused token selection top_k = 40 , # Limit vocabulary options context_window_compression = ContextWindowCompressionParams( enabled = True , trigger_tokens = 8000 # Compress when reaching 8000 tokens ) ) ) ​ Methods ​ set_audio_input_paused(paused: bool) method Pause or unpause audio input processing ​ set_video_input_paused(paused: bool) method Pause or unpause video input processing ​ set_model_modalities(modalities: GeminiMultimodalModalities) method Change the response modality (TEXT or AUDIO) ​ set_language(language: Language) method Change the language for generation ​ set_context(context: OpenAILLMContext) method Set the conversation context explicitly ​ create_context_aggregator(context: OpenAILLMContext, user_params: LLMUserAggregatorParams, assistant_params: LLMAssistantAggregatorParams) method Create context aggregators for managing conversation state ​ Frame Types ​ Input Frames ​ InputAudioRawFrame Frame Raw audio data for speech input ​ InputImageRawFrame Frame Raw image data for visual input ​ StartInterruptionFrame Frame Signals start of user interruption ​ UserStartedSpeakingFrame Frame Signals user started speaking ​ UserStoppedSpeakingFrame Frame Signals user stopped speaking ​ OpenAILLMContextFrame Frame Contains conversation context ​ LLMMessagesAppendFrame Frame Adds messages to the conversation ​ LLMUpdateSettingsFrame Frame Updates LLM settings ​ LLMSetToolsFrame Frame Sets available tools for the LLM ​ Output Frames ​ TTSAudioRawFrame Frame Generated speech audio ​ TTSStartedFrame Frame Signals start of speech synthesis ​ TTSStoppedFrame Frame Signals end of speech synthesis ​ LLMTextFrame Frame Generated text responses from the LLM ​ TTSTextFrame Frame Text used for speech synthesis ​ TranscriptionFrame Frame Speech transcriptions from user audio ​ LLMFullResponseStartFrame Frame Signals the start of a complete LLM response ​ LLMFullResponseEndFrame Frame Signals the end of a complete LLM response ​ Function Calling This service supports function calling (also known as tool calling) which allows the LLM to request information from external services and APIs. For example, you can enable your bot to: Check current weather conditions Query databases Access external APIs Perform custom actions See the Function Calling guide for: Detailed implementation instructions Provider-specific function definitions Handler registration examples Control over function call behavior Complete usage examples ​ Token Usage Tracking Gemini Multimodal Live automatically tracks token usage metrics, providing: Prompt token counts Completion token counts Total token counts Detailed token breakdowns by modality (text, audio) These metrics can be used for monitoring usage, optimizing costs, and understanding model performance. ​ Language Support Gemini Multimodal Live supports the following languages: Language Code Description Gemini Code Language.AR Arabic ar-XA Language.BN_IN Bengali (India) bn-IN Language.CMN_CN Chinese (Mandarin) cmn-CN Language.DE_DE German (Germany) de-DE Language.EN_US English (US) en-US Language.EN_AU English (Australia) en-AU Language.EN_GB English (UK) en-GB Language.EN_IN English (India) en-IN Language.ES_ES Spanish (Spain) es-ES Language.ES_US Spanish (US) es-US Language.FR_FR French (France) fr-FR Language.FR_CA French (Canada) fr-CA Language.GU_IN Gujarati (India) gu-IN Language.HI_IN Hindi (India) hi-IN Language.ID_ID Indonesian id-ID Language.IT_IT Italian (Italy) it-IT Language.JA_JP Japanese (Japan) ja-JP Language.KN_IN Kannada (India) kn-IN Language.KO_KR Korean (Korea) ko-KR Language.ML_IN Malayalam (India) ml-IN Language.MR_IN Marathi (India) mr-IN Language.NL_NL Dutch (Netherlands) nl-NL Language.PL_PL Polish (Poland) pl-PL Language.PT_BR Portuguese (Brazil) pt-BR Language.RU_RU Russian (Russia) ru-RU Language.TA_IN Tamil (India) ta-IN Language.TE_IN Telugu (India) te-IN Language.TH_TH Thai (Thailand) th-TH Language.TR_TR Turkish (Turkey) tr-TR Language.VI_VN Vietnamese (Vietnam) vi-VN You can set the language using the language parameter: Copy Ask AI from pipecat.transcriptions.language import Language from pipecat.services.gemini_multimodal_live.gemini import ( GeminiMultimodalLiveLLMService, InputParams ) # Set language during initialization llm = GeminiMultimodalLiveLLMService( api_key = os.getenv( "GOOGLE_API_KEY" ), params = InputParams( language = Language. ES_ES ) # Spanish (Spain) ) ​ Next Steps ​ Examples Foundational Example Basic implementation showing core features and transcription Simple Chatbot A client/server example showing how to build a Pipecat JS or React client that connects to a Gemini Live Pipecat bot. ​ Learn More Check out our Gemini Multimodal Live Guide for detailed explanations and best practices. AWS Nova Sonic OpenAI Realtime Beta On this page Installation Basic Usage Configuration Constructor Parameters Input Parameters Methods Frame Types Input Frames Output Frames Function Calling Token Usage Tracking Language Support Next Steps Examples Learn More Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/s2s_gemini_c89a084e.txt b/s2s_gemini_c89a084e.txt
new file mode 100644
index 0000000000000000000000000000000000000000..c9f15cb1d6ef7f08ba13ae0a1edaf538b3dc6ac5
--- /dev/null
+++ b/s2s_gemini_c89a084e.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/s2s/gemini#frame-types
+Title: Gemini Multimodal Live - Pipecat
+==================================================
+
+Gemini Multimodal Live - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Speech Gemini Multimodal Live Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech AWS Nova Sonic Gemini Multimodal Live OpenAI Realtime Beta Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline The GeminiMultimodalLiveLLMService enables natural, real-time conversations with Google’s Gemini model. It provides built-in audio transcription, voice activity detection, and context management for creating interactive AI experiences. It provides: Real-time Interaction Stream audio and video in real-time with low latency response times Speech Processing Built-in speech-to-text and text-to-speech capabilities with multiple voice options Voice Activity Detection Automatic detection of speech start/stop for natural conversations Context Management Intelligent handling of conversation history and system instructions Want to start building? Check out our Gemini Multimodal Live Guide . ​ Installation To use GeminiMultimodalLiveLLMService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[google]" You’ll need to set up your Google API key as an environment variable: GOOGLE_API_KEY . ​ Basic Usage Here’s a simple example of setting up a conversational AI bot with Gemini Multimodal Live: Copy Ask AI from pipecat.services.gemini_multimodal_live.gemini import ( GeminiMultimodalLiveLLMService, InputParams, GeminiMultimodalModalities ) llm = GeminiMultimodalLiveLLMService( api_key = os.getenv( "GOOGLE_API_KEY" ), voice_id = "Aoede" , # Voices: Aoede, Charon, Fenrir, Kore, Puck params = InputParams( temperature = 0.7 , # Set model input params language = Language. EN_US , # Set language (30+ languages supported) modalities = GeminiMultimodalModalities. AUDIO # Response modality ) ) ​ Configuration ​ Constructor Parameters ​ api_key str required Your Google API key ​ base_url str API endpoint URL ​ model str Gemini model to use (upgraded to new v1beta model) ​ voice_id str default: "Charon" Voice for text-to-speech (options: Aoede, Charon, Fenrir, Kore, Puck) Copy Ask AI llm = GeminiMultimodalLiveLLMService( api_key = os.getenv( "GOOGLE_API_KEY" ), voice_id = "Puck" , # Choose your preferred voice ) ​ system_instruction str High-level instructions that guide the model’s behavior Copy Ask AI llm = GeminiMultimodalLiveLLMService( api_key = os.getenv( "GOOGLE_API_KEY" ), system_instruction = "Talk like a pirate." , ) ​ start_audio_paused bool default: "False" Whether to start with audio input paused ​ start_video_paused bool default: "False" Whether to start with video input paused ​ tools Union[List[dict], ToolsSchema] Tools/functions available to the model ​ inference_on_context_initialization bool default: "True" Whether to generate a response when context is first set ​ Input Parameters ​ frequency_penalty float default: "None" Penalizes repeated tokens. Range: 0.0 to 2.0 ​ max_tokens int default: "4096" Maximum number of tokens to generate ​ modalities GeminiMultimodalModalities default: "AUDIO" Response modalities to include (options: AUDIO , TEXT ). ​ presence_penalty float default: "None" Penalizes tokens based on their presence in the text. Range: 0.0 to 2.0 ​ temperature float default: "None" Controls randomness in responses. Range: 0.0 to 2.0 ​ language Language default: "Language.EN_US" Language for generation. Over 30 languages are supported. ​ media_resolution GeminiMediaResolution default: "UNSPECIFIED" Controls image processing quality and token usage: LOW : Uses 64 tokens MEDIUM : Uses 256 tokens HIGH : Zoomed reframing with 256 tokens ​ vad GeminiVADParams Voice Activity Detection configuration: disabled : Toggle VAD on/off start_sensitivity : How quickly speech is detected (HIGH/LOW) end_sensitivity : How quickly turns end after pauses (HIGH/LOW) prefix_padding_ms : Milliseconds of audio to keep before speech silence_duration_ms : Milliseconds of silence to end a turn Copy Ask AI from pipecat.services.gemini_multimodal_live.events import ( StartSensitivity, EndSensitivity ) from pipecat.services.gemini_multimodal_live.gemini import ( GeminiVADParams, GeminiMediaResolution, ) llm = GeminiMultimodalLiveLLMService( api_key = os.getenv( "GOOGLE_API_KEY" ), params = InputParams( temperature = 0.7 , language = Language. ES , # Spanish language media_resolution = GeminiMediaResolution. HIGH , # Higher quality image processing vad = GeminiVADParams( start_sensitivity = StartSensitivity. HIGH , # Detect speech quickly end_sensitivity = EndSensitivity. LOW , # Allow longer pauses prefix_padding_ms = 300 , # Keep 300ms before speech silence_duration_ms = 1000 , # End turn after 1s silence ) ) ) ​ top_k int default: "None" Limits vocabulary to k most likely tokens. Minimum: 0 ​ top_p float default: "None" Cumulative probability cutoff for token selection. Range: 0.0 to 1.0 ​ context_window_compression ContextWindowCompressionParams Parameters for managing the context window: - enabled : Enable/disable compression (default: False) - trigger_tokens : Number of tokens that trigger compression (default: None, uses 80% of context window) Copy Ask AI from pipecat.services.gemini_multimodal_live.gemini import ( ContextWindowCompressionParams ) llm = GeminiMultimodalLiveLLMService( api_key = os.getenv( "GOOGLE_API_KEY" ), params = InputParams( top_p = 0.9 , # More focused token selection top_k = 40 , # Limit vocabulary options context_window_compression = ContextWindowCompressionParams( enabled = True , trigger_tokens = 8000 # Compress when reaching 8000 tokens ) ) ) ​ Methods ​ set_audio_input_paused(paused: bool) method Pause or unpause audio input processing ​ set_video_input_paused(paused: bool) method Pause or unpause video input processing ​ set_model_modalities(modalities: GeminiMultimodalModalities) method Change the response modality (TEXT or AUDIO) ​ set_language(language: Language) method Change the language for generation ​ set_context(context: OpenAILLMContext) method Set the conversation context explicitly ​ create_context_aggregator(context: OpenAILLMContext, user_params: LLMUserAggregatorParams, assistant_params: LLMAssistantAggregatorParams) method Create context aggregators for managing conversation state ​ Frame Types ​ Input Frames ​ InputAudioRawFrame Frame Raw audio data for speech input ​ InputImageRawFrame Frame Raw image data for visual input ​ StartInterruptionFrame Frame Signals start of user interruption ​ UserStartedSpeakingFrame Frame Signals user started speaking ​ UserStoppedSpeakingFrame Frame Signals user stopped speaking ​ OpenAILLMContextFrame Frame Contains conversation context ​ LLMMessagesAppendFrame Frame Adds messages to the conversation ​ LLMUpdateSettingsFrame Frame Updates LLM settings ​ LLMSetToolsFrame Frame Sets available tools for the LLM ​ Output Frames ​ TTSAudioRawFrame Frame Generated speech audio ​ TTSStartedFrame Frame Signals start of speech synthesis ​ TTSStoppedFrame Frame Signals end of speech synthesis ​ LLMTextFrame Frame Generated text responses from the LLM ​ TTSTextFrame Frame Text used for speech synthesis ​ TranscriptionFrame Frame Speech transcriptions from user audio ​ LLMFullResponseStartFrame Frame Signals the start of a complete LLM response ​ LLMFullResponseEndFrame Frame Signals the end of a complete LLM response ​ Function Calling This service supports function calling (also known as tool calling) which allows the LLM to request information from external services and APIs. For example, you can enable your bot to: Check current weather conditions Query databases Access external APIs Perform custom actions See the Function Calling guide for: Detailed implementation instructions Provider-specific function definitions Handler registration examples Control over function call behavior Complete usage examples ​ Token Usage Tracking Gemini Multimodal Live automatically tracks token usage metrics, providing: Prompt token counts Completion token counts Total token counts Detailed token breakdowns by modality (text, audio) These metrics can be used for monitoring usage, optimizing costs, and understanding model performance. ​ Language Support Gemini Multimodal Live supports the following languages: Language Code Description Gemini Code Language.AR Arabic ar-XA Language.BN_IN Bengali (India) bn-IN Language.CMN_CN Chinese (Mandarin) cmn-CN Language.DE_DE German (Germany) de-DE Language.EN_US English (US) en-US Language.EN_AU English (Australia) en-AU Language.EN_GB English (UK) en-GB Language.EN_IN English (India) en-IN Language.ES_ES Spanish (Spain) es-ES Language.ES_US Spanish (US) es-US Language.FR_FR French (France) fr-FR Language.FR_CA French (Canada) fr-CA Language.GU_IN Gujarati (India) gu-IN Language.HI_IN Hindi (India) hi-IN Language.ID_ID Indonesian id-ID Language.IT_IT Italian (Italy) it-IT Language.JA_JP Japanese (Japan) ja-JP Language.KN_IN Kannada (India) kn-IN Language.KO_KR Korean (Korea) ko-KR Language.ML_IN Malayalam (India) ml-IN Language.MR_IN Marathi (India) mr-IN Language.NL_NL Dutch (Netherlands) nl-NL Language.PL_PL Polish (Poland) pl-PL Language.PT_BR Portuguese (Brazil) pt-BR Language.RU_RU Russian (Russia) ru-RU Language.TA_IN Tamil (India) ta-IN Language.TE_IN Telugu (India) te-IN Language.TH_TH Thai (Thailand) th-TH Language.TR_TR Turkish (Turkey) tr-TR Language.VI_VN Vietnamese (Vietnam) vi-VN You can set the language using the language parameter: Copy Ask AI from pipecat.transcriptions.language import Language from pipecat.services.gemini_multimodal_live.gemini import ( GeminiMultimodalLiveLLMService, InputParams ) # Set language during initialization llm = GeminiMultimodalLiveLLMService( api_key = os.getenv( "GOOGLE_API_KEY" ), params = InputParams( language = Language. ES_ES ) # Spanish (Spain) ) ​ Next Steps ​ Examples Foundational Example Basic implementation showing core features and transcription Simple Chatbot A client/server example showing how to build a Pipecat JS or React client that connects to a Gemini Live Pipecat bot. ​ Learn More Check out our Gemini Multimodal Live Guide for detailed explanations and best practices. AWS Nova Sonic OpenAI Realtime Beta On this page Installation Basic Usage Configuration Constructor Parameters Input Parameters Methods Frame Types Input Frames Output Frames Function Calling Token Usage Tracking Language Support Next Steps Examples Learn More Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/s2s_gemini_e52257bf.txt b/s2s_gemini_e52257bf.txt
new file mode 100644
index 0000000000000000000000000000000000000000..aaa99b90007c2c65eb8cc993203d34d4b69656dc
--- /dev/null
+++ b/s2s_gemini_e52257bf.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/s2s/gemini#param-input-image-raw-frame
+Title: Gemini Multimodal Live - Pipecat
+==================================================
+
+Gemini Multimodal Live - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Speech Gemini Multimodal Live Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech AWS Nova Sonic Gemini Multimodal Live OpenAI Realtime Beta Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline The GeminiMultimodalLiveLLMService enables natural, real-time conversations with Google’s Gemini model. It provides built-in audio transcription, voice activity detection, and context management for creating interactive AI experiences. It provides: Real-time Interaction Stream audio and video in real-time with low latency response times Speech Processing Built-in speech-to-text and text-to-speech capabilities with multiple voice options Voice Activity Detection Automatic detection of speech start/stop for natural conversations Context Management Intelligent handling of conversation history and system instructions Want to start building? Check out our Gemini Multimodal Live Guide . ​ Installation To use GeminiMultimodalLiveLLMService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[google]" You’ll need to set up your Google API key as an environment variable: GOOGLE_API_KEY . ​ Basic Usage Here’s a simple example of setting up a conversational AI bot with Gemini Multimodal Live: Copy Ask AI from pipecat.services.gemini_multimodal_live.gemini import ( GeminiMultimodalLiveLLMService, InputParams, GeminiMultimodalModalities ) llm = GeminiMultimodalLiveLLMService( api_key = os.getenv( "GOOGLE_API_KEY" ), voice_id = "Aoede" , # Voices: Aoede, Charon, Fenrir, Kore, Puck params = InputParams( temperature = 0.7 , # Set model input params language = Language. EN_US , # Set language (30+ languages supported) modalities = GeminiMultimodalModalities. AUDIO # Response modality ) ) ​ Configuration ​ Constructor Parameters ​ api_key str required Your Google API key ​ base_url str API endpoint URL ​ model str Gemini model to use (upgraded to new v1beta model) ​ voice_id str default: "Charon" Voice for text-to-speech (options: Aoede, Charon, Fenrir, Kore, Puck) Copy Ask AI llm = GeminiMultimodalLiveLLMService( api_key = os.getenv( "GOOGLE_API_KEY" ), voice_id = "Puck" , # Choose your preferred voice ) ​ system_instruction str High-level instructions that guide the model’s behavior Copy Ask AI llm = GeminiMultimodalLiveLLMService( api_key = os.getenv( "GOOGLE_API_KEY" ), system_instruction = "Talk like a pirate." , ) ​ start_audio_paused bool default: "False" Whether to start with audio input paused ​ start_video_paused bool default: "False" Whether to start with video input paused ​ tools Union[List[dict], ToolsSchema] Tools/functions available to the model ​ inference_on_context_initialization bool default: "True" Whether to generate a response when context is first set ​ Input Parameters ​ frequency_penalty float default: "None" Penalizes repeated tokens. Range: 0.0 to 2.0 ​ max_tokens int default: "4096" Maximum number of tokens to generate ​ modalities GeminiMultimodalModalities default: "AUDIO" Response modalities to include (options: AUDIO , TEXT ). ​ presence_penalty float default: "None" Penalizes tokens based on their presence in the text. Range: 0.0 to 2.0 ​ temperature float default: "None" Controls randomness in responses. Range: 0.0 to 2.0 ​ language Language default: "Language.EN_US" Language for generation. Over 30 languages are supported. ​ media_resolution GeminiMediaResolution default: "UNSPECIFIED" Controls image processing quality and token usage: LOW : Uses 64 tokens MEDIUM : Uses 256 tokens HIGH : Zoomed reframing with 256 tokens ​ vad GeminiVADParams Voice Activity Detection configuration: disabled : Toggle VAD on/off start_sensitivity : How quickly speech is detected (HIGH/LOW) end_sensitivity : How quickly turns end after pauses (HIGH/LOW) prefix_padding_ms : Milliseconds of audio to keep before speech silence_duration_ms : Milliseconds of silence to end a turn Copy Ask AI from pipecat.services.gemini_multimodal_live.events import ( StartSensitivity, EndSensitivity ) from pipecat.services.gemini_multimodal_live.gemini import ( GeminiVADParams, GeminiMediaResolution, ) llm = GeminiMultimodalLiveLLMService( api_key = os.getenv( "GOOGLE_API_KEY" ), params = InputParams( temperature = 0.7 , language = Language. ES , # Spanish language media_resolution = GeminiMediaResolution. HIGH , # Higher quality image processing vad = GeminiVADParams( start_sensitivity = StartSensitivity. HIGH , # Detect speech quickly end_sensitivity = EndSensitivity. LOW , # Allow longer pauses prefix_padding_ms = 300 , # Keep 300ms before speech silence_duration_ms = 1000 , # End turn after 1s silence ) ) ) ​ top_k int default: "None" Limits vocabulary to k most likely tokens. Minimum: 0 ​ top_p float default: "None" Cumulative probability cutoff for token selection. Range: 0.0 to 1.0 ​ context_window_compression ContextWindowCompressionParams Parameters for managing the context window: - enabled : Enable/disable compression (default: False) - trigger_tokens : Number of tokens that trigger compression (default: None, uses 80% of context window) Copy Ask AI from pipecat.services.gemini_multimodal_live.gemini import ( ContextWindowCompressionParams ) llm = GeminiMultimodalLiveLLMService( api_key = os.getenv( "GOOGLE_API_KEY" ), params = InputParams( top_p = 0.9 , # More focused token selection top_k = 40 , # Limit vocabulary options context_window_compression = ContextWindowCompressionParams( enabled = True , trigger_tokens = 8000 # Compress when reaching 8000 tokens ) ) ) ​ Methods ​ set_audio_input_paused(paused: bool) method Pause or unpause audio input processing ​ set_video_input_paused(paused: bool) method Pause or unpause video input processing ​ set_model_modalities(modalities: GeminiMultimodalModalities) method Change the response modality (TEXT or AUDIO) ​ set_language(language: Language) method Change the language for generation ​ set_context(context: OpenAILLMContext) method Set the conversation context explicitly ​ create_context_aggregator(context: OpenAILLMContext, user_params: LLMUserAggregatorParams, assistant_params: LLMAssistantAggregatorParams) method Create context aggregators for managing conversation state ​ Frame Types ​ Input Frames ​ InputAudioRawFrame Frame Raw audio data for speech input ​ InputImageRawFrame Frame Raw image data for visual input ​ StartInterruptionFrame Frame Signals start of user interruption ​ UserStartedSpeakingFrame Frame Signals user started speaking ​ UserStoppedSpeakingFrame Frame Signals user stopped speaking ​ OpenAILLMContextFrame Frame Contains conversation context ​ LLMMessagesAppendFrame Frame Adds messages to the conversation ​ LLMUpdateSettingsFrame Frame Updates LLM settings ​ LLMSetToolsFrame Frame Sets available tools for the LLM ​ Output Frames ​ TTSAudioRawFrame Frame Generated speech audio ​ TTSStartedFrame Frame Signals start of speech synthesis ​ TTSStoppedFrame Frame Signals end of speech synthesis ​ LLMTextFrame Frame Generated text responses from the LLM ​ TTSTextFrame Frame Text used for speech synthesis ​ TranscriptionFrame Frame Speech transcriptions from user audio ​ LLMFullResponseStartFrame Frame Signals the start of a complete LLM response ​ LLMFullResponseEndFrame Frame Signals the end of a complete LLM response ​ Function Calling This service supports function calling (also known as tool calling) which allows the LLM to request information from external services and APIs. For example, you can enable your bot to: Check current weather conditions Query databases Access external APIs Perform custom actions See the Function Calling guide for: Detailed implementation instructions Provider-specific function definitions Handler registration examples Control over function call behavior Complete usage examples ​ Token Usage Tracking Gemini Multimodal Live automatically tracks token usage metrics, providing: Prompt token counts Completion token counts Total token counts Detailed token breakdowns by modality (text, audio) These metrics can be used for monitoring usage, optimizing costs, and understanding model performance. ​ Language Support Gemini Multimodal Live supports the following languages: Language Code Description Gemini Code Language.AR Arabic ar-XA Language.BN_IN Bengali (India) bn-IN Language.CMN_CN Chinese (Mandarin) cmn-CN Language.DE_DE German (Germany) de-DE Language.EN_US English (US) en-US Language.EN_AU English (Australia) en-AU Language.EN_GB English (UK) en-GB Language.EN_IN English (India) en-IN Language.ES_ES Spanish (Spain) es-ES Language.ES_US Spanish (US) es-US Language.FR_FR French (France) fr-FR Language.FR_CA French (Canada) fr-CA Language.GU_IN Gujarati (India) gu-IN Language.HI_IN Hindi (India) hi-IN Language.ID_ID Indonesian id-ID Language.IT_IT Italian (Italy) it-IT Language.JA_JP Japanese (Japan) ja-JP Language.KN_IN Kannada (India) kn-IN Language.KO_KR Korean (Korea) ko-KR Language.ML_IN Malayalam (India) ml-IN Language.MR_IN Marathi (India) mr-IN Language.NL_NL Dutch (Netherlands) nl-NL Language.PL_PL Polish (Poland) pl-PL Language.PT_BR Portuguese (Brazil) pt-BR Language.RU_RU Russian (Russia) ru-RU Language.TA_IN Tamil (India) ta-IN Language.TE_IN Telugu (India) te-IN Language.TH_TH Thai (Thailand) th-TH Language.TR_TR Turkish (Turkey) tr-TR Language.VI_VN Vietnamese (Vietnam) vi-VN You can set the language using the language parameter: Copy Ask AI from pipecat.transcriptions.language import Language from pipecat.services.gemini_multimodal_live.gemini import ( GeminiMultimodalLiveLLMService, InputParams ) # Set language during initialization llm = GeminiMultimodalLiveLLMService( api_key = os.getenv( "GOOGLE_API_KEY" ), params = InputParams( language = Language. ES_ES ) # Spanish (Spain) ) ​ Next Steps ​ Examples Foundational Example Basic implementation showing core features and transcription Simple Chatbot A client/server example showing how to build a Pipecat JS or React client that connects to a Gemini Live Pipecat bot. ​ Learn More Check out our Gemini Multimodal Live Guide for detailed explanations and best practices. AWS Nova Sonic OpenAI Realtime Beta On this page Installation Basic Usage Configuration Constructor Parameters Input Parameters Methods Frame Types Input Frames Output Frames Function Calling Token Usage Tracking Language Support Next Steps Examples Learn More Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/s2s_gemini_f48cee59.txt b/s2s_gemini_f48cee59.txt
new file mode 100644
index 0000000000000000000000000000000000000000..84b09c865ae9ca4bb5701c056aea3161f5463188
--- /dev/null
+++ b/s2s_gemini_f48cee59.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/s2s/gemini#function-calling
+Title: Gemini Multimodal Live - Pipecat
+==================================================
+
+Gemini Multimodal Live - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Speech Gemini Multimodal Live Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech AWS Nova Sonic Gemini Multimodal Live OpenAI Realtime Beta Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline The GeminiMultimodalLiveLLMService enables natural, real-time conversations with Google’s Gemini model. It provides built-in audio transcription, voice activity detection, and context management for creating interactive AI experiences. It provides: Real-time Interaction Stream audio and video in real-time with low latency response times Speech Processing Built-in speech-to-text and text-to-speech capabilities with multiple voice options Voice Activity Detection Automatic detection of speech start/stop for natural conversations Context Management Intelligent handling of conversation history and system instructions Want to start building? Check out our Gemini Multimodal Live Guide . ​ Installation To use GeminiMultimodalLiveLLMService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[google]" You’ll need to set up your Google API key as an environment variable: GOOGLE_API_KEY . ​ Basic Usage Here’s a simple example of setting up a conversational AI bot with Gemini Multimodal Live: Copy Ask AI from pipecat.services.gemini_multimodal_live.gemini import ( GeminiMultimodalLiveLLMService, InputParams, GeminiMultimodalModalities ) llm = GeminiMultimodalLiveLLMService( api_key = os.getenv( "GOOGLE_API_KEY" ), voice_id = "Aoede" , # Voices: Aoede, Charon, Fenrir, Kore, Puck params = InputParams( temperature = 0.7 , # Set model input params language = Language. EN_US , # Set language (30+ languages supported) modalities = GeminiMultimodalModalities. AUDIO # Response modality ) ) ​ Configuration ​ Constructor Parameters ​ api_key str required Your Google API key ​ base_url str API endpoint URL ​ model str Gemini model to use (upgraded to new v1beta model) ​ voice_id str default: "Charon" Voice for text-to-speech (options: Aoede, Charon, Fenrir, Kore, Puck) Copy Ask AI llm = GeminiMultimodalLiveLLMService( api_key = os.getenv( "GOOGLE_API_KEY" ), voice_id = "Puck" , # Choose your preferred voice ) ​ system_instruction str High-level instructions that guide the model’s behavior Copy Ask AI llm = GeminiMultimodalLiveLLMService( api_key = os.getenv( "GOOGLE_API_KEY" ), system_instruction = "Talk like a pirate." , ) ​ start_audio_paused bool default: "False" Whether to start with audio input paused ​ start_video_paused bool default: "False" Whether to start with video input paused ​ tools Union[List[dict], ToolsSchema] Tools/functions available to the model ​ inference_on_context_initialization bool default: "True" Whether to generate a response when context is first set ​ Input Parameters ​ frequency_penalty float default: "None" Penalizes repeated tokens. Range: 0.0 to 2.0 ​ max_tokens int default: "4096" Maximum number of tokens to generate ​ modalities GeminiMultimodalModalities default: "AUDIO" Response modalities to include (options: AUDIO , TEXT ). ​ presence_penalty float default: "None" Penalizes tokens based on their presence in the text. Range: 0.0 to 2.0 ​ temperature float default: "None" Controls randomness in responses. Range: 0.0 to 2.0 ​ language Language default: "Language.EN_US" Language for generation. Over 30 languages are supported. ​ media_resolution GeminiMediaResolution default: "UNSPECIFIED" Controls image processing quality and token usage: LOW : Uses 64 tokens MEDIUM : Uses 256 tokens HIGH : Zoomed reframing with 256 tokens ​ vad GeminiVADParams Voice Activity Detection configuration: disabled : Toggle VAD on/off start_sensitivity : How quickly speech is detected (HIGH/LOW) end_sensitivity : How quickly turns end after pauses (HIGH/LOW) prefix_padding_ms : Milliseconds of audio to keep before speech silence_duration_ms : Milliseconds of silence to end a turn Copy Ask AI from pipecat.services.gemini_multimodal_live.events import ( StartSensitivity, EndSensitivity ) from pipecat.services.gemini_multimodal_live.gemini import ( GeminiVADParams, GeminiMediaResolution, ) llm = GeminiMultimodalLiveLLMService( api_key = os.getenv( "GOOGLE_API_KEY" ), params = InputParams( temperature = 0.7 , language = Language. ES , # Spanish language media_resolution = GeminiMediaResolution. HIGH , # Higher quality image processing vad = GeminiVADParams( start_sensitivity = StartSensitivity. HIGH , # Detect speech quickly end_sensitivity = EndSensitivity. LOW , # Allow longer pauses prefix_padding_ms = 300 , # Keep 300ms before speech silence_duration_ms = 1000 , # End turn after 1s silence ) ) ) ​ top_k int default: "None" Limits vocabulary to k most likely tokens. Minimum: 0 ​ top_p float default: "None" Cumulative probability cutoff for token selection. Range: 0.0 to 1.0 ​ context_window_compression ContextWindowCompressionParams Parameters for managing the context window: - enabled : Enable/disable compression (default: False) - trigger_tokens : Number of tokens that trigger compression (default: None, uses 80% of context window) Copy Ask AI from pipecat.services.gemini_multimodal_live.gemini import ( ContextWindowCompressionParams ) llm = GeminiMultimodalLiveLLMService( api_key = os.getenv( "GOOGLE_API_KEY" ), params = InputParams( top_p = 0.9 , # More focused token selection top_k = 40 , # Limit vocabulary options context_window_compression = ContextWindowCompressionParams( enabled = True , trigger_tokens = 8000 # Compress when reaching 8000 tokens ) ) ) ​ Methods ​ set_audio_input_paused(paused: bool) method Pause or unpause audio input processing ​ set_video_input_paused(paused: bool) method Pause or unpause video input processing ​ set_model_modalities(modalities: GeminiMultimodalModalities) method Change the response modality (TEXT or AUDIO) ​ set_language(language: Language) method Change the language for generation ​ set_context(context: OpenAILLMContext) method Set the conversation context explicitly ​ create_context_aggregator(context: OpenAILLMContext, user_params: LLMUserAggregatorParams, assistant_params: LLMAssistantAggregatorParams) method Create context aggregators for managing conversation state ​ Frame Types ​ Input Frames ​ InputAudioRawFrame Frame Raw audio data for speech input ​ InputImageRawFrame Frame Raw image data for visual input ​ StartInterruptionFrame Frame Signals start of user interruption ​ UserStartedSpeakingFrame Frame Signals user started speaking ​ UserStoppedSpeakingFrame Frame Signals user stopped speaking ​ OpenAILLMContextFrame Frame Contains conversation context ​ LLMMessagesAppendFrame Frame Adds messages to the conversation ​ LLMUpdateSettingsFrame Frame Updates LLM settings ​ LLMSetToolsFrame Frame Sets available tools for the LLM ​ Output Frames ​ TTSAudioRawFrame Frame Generated speech audio ​ TTSStartedFrame Frame Signals start of speech synthesis ​ TTSStoppedFrame Frame Signals end of speech synthesis ​ LLMTextFrame Frame Generated text responses from the LLM ​ TTSTextFrame Frame Text used for speech synthesis ​ TranscriptionFrame Frame Speech transcriptions from user audio ​ LLMFullResponseStartFrame Frame Signals the start of a complete LLM response ​ LLMFullResponseEndFrame Frame Signals the end of a complete LLM response ​ Function Calling This service supports function calling (also known as tool calling) which allows the LLM to request information from external services and APIs. For example, you can enable your bot to: Check current weather conditions Query databases Access external APIs Perform custom actions See the Function Calling guide for: Detailed implementation instructions Provider-specific function definitions Handler registration examples Control over function call behavior Complete usage examples ​ Token Usage Tracking Gemini Multimodal Live automatically tracks token usage metrics, providing: Prompt token counts Completion token counts Total token counts Detailed token breakdowns by modality (text, audio) These metrics can be used for monitoring usage, optimizing costs, and understanding model performance. ​ Language Support Gemini Multimodal Live supports the following languages: Language Code Description Gemini Code Language.AR Arabic ar-XA Language.BN_IN Bengali (India) bn-IN Language.CMN_CN Chinese (Mandarin) cmn-CN Language.DE_DE German (Germany) de-DE Language.EN_US English (US) en-US Language.EN_AU English (Australia) en-AU Language.EN_GB English (UK) en-GB Language.EN_IN English (India) en-IN Language.ES_ES Spanish (Spain) es-ES Language.ES_US Spanish (US) es-US Language.FR_FR French (France) fr-FR Language.FR_CA French (Canada) fr-CA Language.GU_IN Gujarati (India) gu-IN Language.HI_IN Hindi (India) hi-IN Language.ID_ID Indonesian id-ID Language.IT_IT Italian (Italy) it-IT Language.JA_JP Japanese (Japan) ja-JP Language.KN_IN Kannada (India) kn-IN Language.KO_KR Korean (Korea) ko-KR Language.ML_IN Malayalam (India) ml-IN Language.MR_IN Marathi (India) mr-IN Language.NL_NL Dutch (Netherlands) nl-NL Language.PL_PL Polish (Poland) pl-PL Language.PT_BR Portuguese (Brazil) pt-BR Language.RU_RU Russian (Russia) ru-RU Language.TA_IN Tamil (India) ta-IN Language.TE_IN Telugu (India) te-IN Language.TH_TH Thai (Thailand) th-TH Language.TR_TR Turkish (Turkey) tr-TR Language.VI_VN Vietnamese (Vietnam) vi-VN You can set the language using the language parameter: Copy Ask AI from pipecat.transcriptions.language import Language from pipecat.services.gemini_multimodal_live.gemini import ( GeminiMultimodalLiveLLMService, InputParams ) # Set language during initialization llm = GeminiMultimodalLiveLLMService( api_key = os.getenv( "GOOGLE_API_KEY" ), params = InputParams( language = Language. ES_ES ) # Spanish (Spain) ) ​ Next Steps ​ Examples Foundational Example Basic implementation showing core features and transcription Simple Chatbot A client/server example showing how to build a Pipecat JS or React client that connects to a Gemini Live Pipecat bot. ​ Learn More Check out our Gemini Multimodal Live Guide for detailed explanations and best practices. AWS Nova Sonic OpenAI Realtime Beta On this page Installation Basic Usage Configuration Constructor Parameters Input Parameters Methods Frame Types Input Frames Output Frames Function Calling Token Usage Tracking Language Support Next Steps Examples Learn More Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/s2s_gemini_f54a3410.txt b/s2s_gemini_f54a3410.txt
new file mode 100644
index 0000000000000000000000000000000000000000..ffaf5ce88a963665bef9fd4a41da9ff8222885ab
--- /dev/null
+++ b/s2s_gemini_f54a3410.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/s2s/gemini#param-llm-text-frame
+Title: Gemini Multimodal Live - Pipecat
+==================================================
+
+Gemini Multimodal Live - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Speech Gemini Multimodal Live Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech AWS Nova Sonic Gemini Multimodal Live OpenAI Realtime Beta Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline The GeminiMultimodalLiveLLMService enables natural, real-time conversations with Google’s Gemini model. It provides built-in audio transcription, voice activity detection, and context management for creating interactive AI experiences. It provides: Real-time Interaction Stream audio and video in real-time with low latency response times Speech Processing Built-in speech-to-text and text-to-speech capabilities with multiple voice options Voice Activity Detection Automatic detection of speech start/stop for natural conversations Context Management Intelligent handling of conversation history and system instructions Want to start building? Check out our Gemini Multimodal Live Guide . ​ Installation To use GeminiMultimodalLiveLLMService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[google]" You’ll need to set up your Google API key as an environment variable: GOOGLE_API_KEY . ​ Basic Usage Here’s a simple example of setting up a conversational AI bot with Gemini Multimodal Live: Copy Ask AI from pipecat.services.gemini_multimodal_live.gemini import ( GeminiMultimodalLiveLLMService, InputParams, GeminiMultimodalModalities ) llm = GeminiMultimodalLiveLLMService( api_key = os.getenv( "GOOGLE_API_KEY" ), voice_id = "Aoede" , # Voices: Aoede, Charon, Fenrir, Kore, Puck params = InputParams( temperature = 0.7 , # Set model input params language = Language. EN_US , # Set language (30+ languages supported) modalities = GeminiMultimodalModalities. AUDIO # Response modality ) ) ​ Configuration ​ Constructor Parameters ​ api_key str required Your Google API key ​ base_url str API endpoint URL ​ model str Gemini model to use (upgraded to new v1beta model) ​ voice_id str default: "Charon" Voice for text-to-speech (options: Aoede, Charon, Fenrir, Kore, Puck) Copy Ask AI llm = GeminiMultimodalLiveLLMService( api_key = os.getenv( "GOOGLE_API_KEY" ), voice_id = "Puck" , # Choose your preferred voice ) ​ system_instruction str High-level instructions that guide the model’s behavior Copy Ask AI llm = GeminiMultimodalLiveLLMService( api_key = os.getenv( "GOOGLE_API_KEY" ), system_instruction = "Talk like a pirate." , ) ​ start_audio_paused bool default: "False" Whether to start with audio input paused ​ start_video_paused bool default: "False" Whether to start with video input paused ​ tools Union[List[dict], ToolsSchema] Tools/functions available to the model ​ inference_on_context_initialization bool default: "True" Whether to generate a response when context is first set ​ Input Parameters ​ frequency_penalty float default: "None" Penalizes repeated tokens. Range: 0.0 to 2.0 ​ max_tokens int default: "4096" Maximum number of tokens to generate ​ modalities GeminiMultimodalModalities default: "AUDIO" Response modalities to include (options: AUDIO , TEXT ). ​ presence_penalty float default: "None" Penalizes tokens based on their presence in the text. Range: 0.0 to 2.0 ​ temperature float default: "None" Controls randomness in responses. Range: 0.0 to 2.0 ​ language Language default: "Language.EN_US" Language for generation. Over 30 languages are supported. ​ media_resolution GeminiMediaResolution default: "UNSPECIFIED" Controls image processing quality and token usage: LOW : Uses 64 tokens MEDIUM : Uses 256 tokens HIGH : Zoomed reframing with 256 tokens ​ vad GeminiVADParams Voice Activity Detection configuration: disabled : Toggle VAD on/off start_sensitivity : How quickly speech is detected (HIGH/LOW) end_sensitivity : How quickly turns end after pauses (HIGH/LOW) prefix_padding_ms : Milliseconds of audio to keep before speech silence_duration_ms : Milliseconds of silence to end a turn Copy Ask AI from pipecat.services.gemini_multimodal_live.events import ( StartSensitivity, EndSensitivity ) from pipecat.services.gemini_multimodal_live.gemini import ( GeminiVADParams, GeminiMediaResolution, ) llm = GeminiMultimodalLiveLLMService( api_key = os.getenv( "GOOGLE_API_KEY" ), params = InputParams( temperature = 0.7 , language = Language. ES , # Spanish language media_resolution = GeminiMediaResolution. HIGH , # Higher quality image processing vad = GeminiVADParams( start_sensitivity = StartSensitivity. HIGH , # Detect speech quickly end_sensitivity = EndSensitivity. LOW , # Allow longer pauses prefix_padding_ms = 300 , # Keep 300ms before speech silence_duration_ms = 1000 , # End turn after 1s silence ) ) ) ​ top_k int default: "None" Limits vocabulary to k most likely tokens. Minimum: 0 ​ top_p float default: "None" Cumulative probability cutoff for token selection. Range: 0.0 to 1.0 ​ context_window_compression ContextWindowCompressionParams Parameters for managing the context window: - enabled : Enable/disable compression (default: False) - trigger_tokens : Number of tokens that trigger compression (default: None, uses 80% of context window) Copy Ask AI from pipecat.services.gemini_multimodal_live.gemini import ( ContextWindowCompressionParams ) llm = GeminiMultimodalLiveLLMService( api_key = os.getenv( "GOOGLE_API_KEY" ), params = InputParams( top_p = 0.9 , # More focused token selection top_k = 40 , # Limit vocabulary options context_window_compression = ContextWindowCompressionParams( enabled = True , trigger_tokens = 8000 # Compress when reaching 8000 tokens ) ) ) ​ Methods ​ set_audio_input_paused(paused: bool) method Pause or unpause audio input processing ​ set_video_input_paused(paused: bool) method Pause or unpause video input processing ​ set_model_modalities(modalities: GeminiMultimodalModalities) method Change the response modality (TEXT or AUDIO) ​ set_language(language: Language) method Change the language for generation ​ set_context(context: OpenAILLMContext) method Set the conversation context explicitly ​ create_context_aggregator(context: OpenAILLMContext, user_params: LLMUserAggregatorParams, assistant_params: LLMAssistantAggregatorParams) method Create context aggregators for managing conversation state ​ Frame Types ​ Input Frames ​ InputAudioRawFrame Frame Raw audio data for speech input ​ InputImageRawFrame Frame Raw image data for visual input ​ StartInterruptionFrame Frame Signals start of user interruption ​ UserStartedSpeakingFrame Frame Signals user started speaking ​ UserStoppedSpeakingFrame Frame Signals user stopped speaking ​ OpenAILLMContextFrame Frame Contains conversation context ​ LLMMessagesAppendFrame Frame Adds messages to the conversation ​ LLMUpdateSettingsFrame Frame Updates LLM settings ​ LLMSetToolsFrame Frame Sets available tools for the LLM ​ Output Frames ​ TTSAudioRawFrame Frame Generated speech audio ​ TTSStartedFrame Frame Signals start of speech synthesis ​ TTSStoppedFrame Frame Signals end of speech synthesis ​ LLMTextFrame Frame Generated text responses from the LLM ​ TTSTextFrame Frame Text used for speech synthesis ​ TranscriptionFrame Frame Speech transcriptions from user audio ​ LLMFullResponseStartFrame Frame Signals the start of a complete LLM response ​ LLMFullResponseEndFrame Frame Signals the end of a complete LLM response ​ Function Calling This service supports function calling (also known as tool calling) which allows the LLM to request information from external services and APIs. For example, you can enable your bot to: Check current weather conditions Query databases Access external APIs Perform custom actions See the Function Calling guide for: Detailed implementation instructions Provider-specific function definitions Handler registration examples Control over function call behavior Complete usage examples ​ Token Usage Tracking Gemini Multimodal Live automatically tracks token usage metrics, providing: Prompt token counts Completion token counts Total token counts Detailed token breakdowns by modality (text, audio) These metrics can be used for monitoring usage, optimizing costs, and understanding model performance. ​ Language Support Gemini Multimodal Live supports the following languages: Language Code Description Gemini Code Language.AR Arabic ar-XA Language.BN_IN Bengali (India) bn-IN Language.CMN_CN Chinese (Mandarin) cmn-CN Language.DE_DE German (Germany) de-DE Language.EN_US English (US) en-US Language.EN_AU English (Australia) en-AU Language.EN_GB English (UK) en-GB Language.EN_IN English (India) en-IN Language.ES_ES Spanish (Spain) es-ES Language.ES_US Spanish (US) es-US Language.FR_FR French (France) fr-FR Language.FR_CA French (Canada) fr-CA Language.GU_IN Gujarati (India) gu-IN Language.HI_IN Hindi (India) hi-IN Language.ID_ID Indonesian id-ID Language.IT_IT Italian (Italy) it-IT Language.JA_JP Japanese (Japan) ja-JP Language.KN_IN Kannada (India) kn-IN Language.KO_KR Korean (Korea) ko-KR Language.ML_IN Malayalam (India) ml-IN Language.MR_IN Marathi (India) mr-IN Language.NL_NL Dutch (Netherlands) nl-NL Language.PL_PL Polish (Poland) pl-PL Language.PT_BR Portuguese (Brazil) pt-BR Language.RU_RU Russian (Russia) ru-RU Language.TA_IN Tamil (India) ta-IN Language.TE_IN Telugu (India) te-IN Language.TH_TH Thai (Thailand) th-TH Language.TR_TR Turkish (Turkey) tr-TR Language.VI_VN Vietnamese (Vietnam) vi-VN You can set the language using the language parameter: Copy Ask AI from pipecat.transcriptions.language import Language from pipecat.services.gemini_multimodal_live.gemini import ( GeminiMultimodalLiveLLMService, InputParams ) # Set language during initialization llm = GeminiMultimodalLiveLLMService( api_key = os.getenv( "GOOGLE_API_KEY" ), params = InputParams( language = Language. ES_ES ) # Spanish (Spain) ) ​ Next Steps ​ Examples Foundational Example Basic implementation showing core features and transcription Simple Chatbot A client/server example showing how to build a Pipecat JS or React client that connects to a Gemini Live Pipecat bot. ​ Learn More Check out our Gemini Multimodal Live Guide for detailed explanations and best practices. AWS Nova Sonic OpenAI Realtime Beta On this page Installation Basic Usage Configuration Constructor Parameters Input Parameters Methods Frame Types Input Frames Output Frames Function Calling Token Usage Tracking Language Support Next Steps Examples Learn More Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/s2s_gemini_f7996087.txt b/s2s_gemini_f7996087.txt
new file mode 100644
index 0000000000000000000000000000000000000000..f6a15d01bee6ad662bfac3e8779d260d549c8b45
--- /dev/null
+++ b/s2s_gemini_f7996087.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/s2s/gemini#methods
+Title: Gemini Multimodal Live - Pipecat
+==================================================
+
+Gemini Multimodal Live - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Speech Gemini Multimodal Live Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech AWS Nova Sonic Gemini Multimodal Live OpenAI Realtime Beta Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline The GeminiMultimodalLiveLLMService enables natural, real-time conversations with Google’s Gemini model. It provides built-in audio transcription, voice activity detection, and context management for creating interactive AI experiences. It provides: Real-time Interaction Stream audio and video in real-time with low latency response times Speech Processing Built-in speech-to-text and text-to-speech capabilities with multiple voice options Voice Activity Detection Automatic detection of speech start/stop for natural conversations Context Management Intelligent handling of conversation history and system instructions Want to start building? Check out our Gemini Multimodal Live Guide . ​ Installation To use GeminiMultimodalLiveLLMService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[google]" You’ll need to set up your Google API key as an environment variable: GOOGLE_API_KEY . ​ Basic Usage Here’s a simple example of setting up a conversational AI bot with Gemini Multimodal Live: Copy Ask AI from pipecat.services.gemini_multimodal_live.gemini import ( GeminiMultimodalLiveLLMService, InputParams, GeminiMultimodalModalities ) llm = GeminiMultimodalLiveLLMService( api_key = os.getenv( "GOOGLE_API_KEY" ), voice_id = "Aoede" , # Voices: Aoede, Charon, Fenrir, Kore, Puck params = InputParams( temperature = 0.7 , # Set model input params language = Language. EN_US , # Set language (30+ languages supported) modalities = GeminiMultimodalModalities. AUDIO # Response modality ) ) ​ Configuration ​ Constructor Parameters ​ api_key str required Your Google API key ​ base_url str API endpoint URL ​ model str Gemini model to use (upgraded to new v1beta model) ​ voice_id str default: "Charon" Voice for text-to-speech (options: Aoede, Charon, Fenrir, Kore, Puck) Copy Ask AI llm = GeminiMultimodalLiveLLMService( api_key = os.getenv( "GOOGLE_API_KEY" ), voice_id = "Puck" , # Choose your preferred voice ) ​ system_instruction str High-level instructions that guide the model’s behavior Copy Ask AI llm = GeminiMultimodalLiveLLMService( api_key = os.getenv( "GOOGLE_API_KEY" ), system_instruction = "Talk like a pirate." , ) ​ start_audio_paused bool default: "False" Whether to start with audio input paused ​ start_video_paused bool default: "False" Whether to start with video input paused ​ tools Union[List[dict], ToolsSchema] Tools/functions available to the model ​ inference_on_context_initialization bool default: "True" Whether to generate a response when context is first set ​ Input Parameters ​ frequency_penalty float default: "None" Penalizes repeated tokens. Range: 0.0 to 2.0 ​ max_tokens int default: "4096" Maximum number of tokens to generate ​ modalities GeminiMultimodalModalities default: "AUDIO" Response modalities to include (options: AUDIO , TEXT ). ​ presence_penalty float default: "None" Penalizes tokens based on their presence in the text. Range: 0.0 to 2.0 ​ temperature float default: "None" Controls randomness in responses. Range: 0.0 to 2.0 ​ language Language default: "Language.EN_US" Language for generation. Over 30 languages are supported. ​ media_resolution GeminiMediaResolution default: "UNSPECIFIED" Controls image processing quality and token usage: LOW : Uses 64 tokens MEDIUM : Uses 256 tokens HIGH : Zoomed reframing with 256 tokens ​ vad GeminiVADParams Voice Activity Detection configuration: disabled : Toggle VAD on/off start_sensitivity : How quickly speech is detected (HIGH/LOW) end_sensitivity : How quickly turns end after pauses (HIGH/LOW) prefix_padding_ms : Milliseconds of audio to keep before speech silence_duration_ms : Milliseconds of silence to end a turn Copy Ask AI from pipecat.services.gemini_multimodal_live.events import ( StartSensitivity, EndSensitivity ) from pipecat.services.gemini_multimodal_live.gemini import ( GeminiVADParams, GeminiMediaResolution, ) llm = GeminiMultimodalLiveLLMService( api_key = os.getenv( "GOOGLE_API_KEY" ), params = InputParams( temperature = 0.7 , language = Language. ES , # Spanish language media_resolution = GeminiMediaResolution. HIGH , # Higher quality image processing vad = GeminiVADParams( start_sensitivity = StartSensitivity. HIGH , # Detect speech quickly end_sensitivity = EndSensitivity. LOW , # Allow longer pauses prefix_padding_ms = 300 , # Keep 300ms before speech silence_duration_ms = 1000 , # End turn after 1s silence ) ) ) ​ top_k int default: "None" Limits vocabulary to k most likely tokens. Minimum: 0 ​ top_p float default: "None" Cumulative probability cutoff for token selection. Range: 0.0 to 1.0 ​ context_window_compression ContextWindowCompressionParams Parameters for managing the context window: - enabled : Enable/disable compression (default: False) - trigger_tokens : Number of tokens that trigger compression (default: None, uses 80% of context window) Copy Ask AI from pipecat.services.gemini_multimodal_live.gemini import ( ContextWindowCompressionParams ) llm = GeminiMultimodalLiveLLMService( api_key = os.getenv( "GOOGLE_API_KEY" ), params = InputParams( top_p = 0.9 , # More focused token selection top_k = 40 , # Limit vocabulary options context_window_compression = ContextWindowCompressionParams( enabled = True , trigger_tokens = 8000 # Compress when reaching 8000 tokens ) ) ) ​ Methods ​ set_audio_input_paused(paused: bool) method Pause or unpause audio input processing ​ set_video_input_paused(paused: bool) method Pause or unpause video input processing ​ set_model_modalities(modalities: GeminiMultimodalModalities) method Change the response modality (TEXT or AUDIO) ​ set_language(language: Language) method Change the language for generation ​ set_context(context: OpenAILLMContext) method Set the conversation context explicitly ​ create_context_aggregator(context: OpenAILLMContext, user_params: LLMUserAggregatorParams, assistant_params: LLMAssistantAggregatorParams) method Create context aggregators for managing conversation state ​ Frame Types ​ Input Frames ​ InputAudioRawFrame Frame Raw audio data for speech input ​ InputImageRawFrame Frame Raw image data for visual input ​ StartInterruptionFrame Frame Signals start of user interruption ​ UserStartedSpeakingFrame Frame Signals user started speaking ​ UserStoppedSpeakingFrame Frame Signals user stopped speaking ​ OpenAILLMContextFrame Frame Contains conversation context ​ LLMMessagesAppendFrame Frame Adds messages to the conversation ​ LLMUpdateSettingsFrame Frame Updates LLM settings ​ LLMSetToolsFrame Frame Sets available tools for the LLM ​ Output Frames ​ TTSAudioRawFrame Frame Generated speech audio ​ TTSStartedFrame Frame Signals start of speech synthesis ​ TTSStoppedFrame Frame Signals end of speech synthesis ​ LLMTextFrame Frame Generated text responses from the LLM ​ TTSTextFrame Frame Text used for speech synthesis ​ TranscriptionFrame Frame Speech transcriptions from user audio ​ LLMFullResponseStartFrame Frame Signals the start of a complete LLM response ​ LLMFullResponseEndFrame Frame Signals the end of a complete LLM response ​ Function Calling This service supports function calling (also known as tool calling) which allows the LLM to request information from external services and APIs. For example, you can enable your bot to: Check current weather conditions Query databases Access external APIs Perform custom actions See the Function Calling guide for: Detailed implementation instructions Provider-specific function definitions Handler registration examples Control over function call behavior Complete usage examples ​ Token Usage Tracking Gemini Multimodal Live automatically tracks token usage metrics, providing: Prompt token counts Completion token counts Total token counts Detailed token breakdowns by modality (text, audio) These metrics can be used for monitoring usage, optimizing costs, and understanding model performance. ​ Language Support Gemini Multimodal Live supports the following languages: Language Code Description Gemini Code Language.AR Arabic ar-XA Language.BN_IN Bengali (India) bn-IN Language.CMN_CN Chinese (Mandarin) cmn-CN Language.DE_DE German (Germany) de-DE Language.EN_US English (US) en-US Language.EN_AU English (Australia) en-AU Language.EN_GB English (UK) en-GB Language.EN_IN English (India) en-IN Language.ES_ES Spanish (Spain) es-ES Language.ES_US Spanish (US) es-US Language.FR_FR French (France) fr-FR Language.FR_CA French (Canada) fr-CA Language.GU_IN Gujarati (India) gu-IN Language.HI_IN Hindi (India) hi-IN Language.ID_ID Indonesian id-ID Language.IT_IT Italian (Italy) it-IT Language.JA_JP Japanese (Japan) ja-JP Language.KN_IN Kannada (India) kn-IN Language.KO_KR Korean (Korea) ko-KR Language.ML_IN Malayalam (India) ml-IN Language.MR_IN Marathi (India) mr-IN Language.NL_NL Dutch (Netherlands) nl-NL Language.PL_PL Polish (Poland) pl-PL Language.PT_BR Portuguese (Brazil) pt-BR Language.RU_RU Russian (Russia) ru-RU Language.TA_IN Tamil (India) ta-IN Language.TE_IN Telugu (India) te-IN Language.TH_TH Thai (Thailand) th-TH Language.TR_TR Turkish (Turkey) tr-TR Language.VI_VN Vietnamese (Vietnam) vi-VN You can set the language using the language parameter: Copy Ask AI from pipecat.transcriptions.language import Language from pipecat.services.gemini_multimodal_live.gemini import ( GeminiMultimodalLiveLLMService, InputParams ) # Set language during initialization llm = GeminiMultimodalLiveLLMService( api_key = os.getenv( "GOOGLE_API_KEY" ), params = InputParams( language = Language. ES_ES ) # Spanish (Spain) ) ​ Next Steps ​ Examples Foundational Example Basic implementation showing core features and transcription Simple Chatbot A client/server example showing how to build a Pipecat JS or React client that connects to a Gemini Live Pipecat bot. ​ Learn More Check out our Gemini Multimodal Live Guide for detailed explanations and best practices. AWS Nova Sonic OpenAI Realtime Beta On this page Installation Basic Usage Configuration Constructor Parameters Input Parameters Methods Frame Types Input Frames Output Frames Function Calling Token Usage Tracking Language Support Next Steps Examples Learn More Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/s2s_openai_14bef24b.txt b/s2s_openai_14bef24b.txt
new file mode 100644
index 0000000000000000000000000000000000000000..feff4caa14847c50f6781fd41c6837e2fee71e02
--- /dev/null
+++ b/s2s_openai_14bef24b.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/s2s/openai#turn-detection
+Title: OpenAI Realtime Beta - Pipecat
+==================================================
+
+OpenAI Realtime Beta - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Speech OpenAI Realtime Beta Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech AWS Nova Sonic Gemini Multimodal Live OpenAI Realtime Beta Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline OpenAIRealtimeBetaLLMService provides real-time, multimodal conversation capabilities using OpenAI’s Realtime Beta API. It supports speech-to-speech interactions with integrated LLM processing, function calling, and advanced conversation management. Real-time Interaction Stream audio in real-time with minimal latency response times Speech Processing Built-in speech-to-text and text-to-speech capabilities with voice options Advanced Turn Detection Multiple voice activity detection options including semantic turn detection Powerful Function Calling Seamless support for calling external functions and APIs ​ Installation To use OpenAIRealtimeBetaLLMService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[openai]" You’ll also need to set up your OpenAI API key as an environment variable: OPENAI_API_KEY . ​ Configuration ​ Constructor Parameters ​ api_key str required Your OpenAI API key ​ model str default: "gpt-4o-realtime-preview-2025-06-03" The speech-to-speech model used for processing ​ base_url str default: "wss://api.openai.com/v1/realtime" WebSocket endpoint URL ​ session_properties SessionProperties Configuration for the realtime session ​ start_audio_paused bool default: "False" Whether to start with audio input paused ​ send_transcription_frames bool default: "True" Whether to emit transcription frames ​ Session Properties The SessionProperties object configures the behavior of the realtime session: ​ modalities List[Literal['text', 'audio']] The modalities to enable (default includes both text and audio) ​ instructions str System instructions that guide the model’s behavior Copy Ask AI service = OpenAIRealtimeBetaLLMService( api_key = os.getenv( "OPENAI_API_KEY" ), session_properties = SessionProperties( instructions = "You are a helpful assistant. Be concise and friendly." ) ) ​ voice str Voice ID for text-to-speech (options: alloy, echo, fable, onyx, nova, shimmer) ​ input_audio_format Literal['pcm16', 'g711_ulaw', 'g711_alaw'] Format of the input audio ​ output_audio_format Literal['pcm16', 'g711_ulaw', 'g711_alaw'] Format of the output audio ​ input_audio_transcription InputAudioTranscription Configuration for audio transcription Copy Ask AI from pipecat.services.openai_realtime_beta.events import InputAudioTranscription service = OpenAIRealtimeBetaLLMService( api_key = os.getenv( "OPENAI_API_KEY" ), session_properties = SessionProperties( input_audio_transcription = InputAudioTranscription( model = "gpt-4o-transcribe" , language = "en" , prompt = "This is a technical conversation about programming" ) ) ) ​ input_audio_noise_reduction InputAudioNoiseReduction Configuration for audio noise reduction ​ turn_detection Union[TurnDetection, SemanticTurnDetection, bool] Configuration for turn detection (set to False to disable) ​ tools List[Dict] List of function definitions for tool/function calling ​ tool_choice Literal['auto', 'none', 'required'] Controls when the model calls functions ​ temperature float Controls randomness in responses (0.0 to 2.0) ​ max_response_output_tokens Union[int, Literal['inf']] Maximum number of tokens to generate ​ Input Frames ​ Audio Input ​ InputAudioRawFrame Frame Raw audio data for speech input ​ Control Input ​ StartInterruptionFrame Frame Signals start of user interruption ​ UserStartedSpeakingFrame Frame Signals user started speaking ​ UserStoppedSpeakingFrame Frame Signals user stopped speaking ​ Context Input ​ OpenAILLMContextFrame Frame Contains conversation context ​ LLMMessagesAppendFrame Frame Appends messages to conversation ​ Output Frames ​ Audio Output ​ TTSAudioRawFrame Frame Generated speech audio ​ Control Output ​ TTSStartedFrame Frame Signals start of speech synthesis ​ TTSStoppedFrame Frame Signals end of speech synthesis ​ Text Output ​ TextFrame Frame Generated text responses ​ TranscriptionFrame Frame Speech transcriptions ​ Events ​ on_conversation_item_created event Emitted when a conversation item on the server is created. Handler receives: item_id: str item: ConversationItem ​ on_conversation_item_updated event Emitted when a conversation item on the server is updated. Handler receives: item_id: str item: Optional[ConversationItem] (may not exist for some updates) ​ Methods ​ retrieve_conversation_item method Retrieves a conversation item’s details from the server. Copy Ask AI async def retrieve_conversation_item ( self , item_id : str ) -> ConversationItem ​ Usage Example Copy Ask AI from pipecat.services.openai_realtime_beta import OpenAIRealtimeBetaLLMService from pipecat.services.openai_realtime_beta.events import SessionProperties, TurnDetection # Configure service service = OpenAIRealtimeBetaLLMService( api_key = "your-api-key" , session_properties = SessionProperties( modalities = [ "audio" , "text" ], voice = "alloy" , turn_detection = TurnDetection( threshold = 0.5 , silence_duration_ms = 800 ), temperature = 0.7 ) ) # Use in pipeline pipeline = Pipeline([ audio_input, # Produces InputAudioRawFrame service, # Processes speech/generates responses audio_output # Handles TTSAudioRawFrame ]) ​ Function Calling The service supports function calling with automatic response handling: Copy Ask AI from pipecat.adapters.schemas.function_schema import FunctionSchema from pipecat.adapters.schemas.tools_schema import ToolsSchema from pipecat.services.openai_realtime_beta import SessionProperties # Define weather function using standardized schema weather_function = FunctionSchema( name = "get_weather" , description = "Get weather information" , properties = { "location" : { "type" : "string" } }, required = [ "location" ] ) # Create tools schema tools = ToolsSchema( standard_tools = [weather_function]) # Configure service with tools llm = OpenAIRealtimeBetaLLMService( api_key = "your-api-key" , session_properties = SessionProperties( tools = tools, tool_choice = "auto" ) ) llm.register_function( "get_weather" , fetch_weather_from_api) See the Function Calling guide for: Detailed implementation instructions Provider-specific function definitions Handler registration examples Control over function call behavior Complete usage examples ​ Frame Flow ​ Metrics Support The service collects comprehensive metrics: Token usage (prompt and completion) Processing duration Time to First Byte (TTFB) Audio processing metrics Function call metrics ​ Advanced Features ​ Turn Detection Copy Ask AI # Server-side basic VAD turn_detection = TurnDetection( type = "server_vad" , threshold = 0.5 , prefix_padding_ms = 300 , silence_duration_ms = 800 ) # Server-side semantic VAD turn_detection = SemanticTurnDetection( type = "semantic_vad" , eagerness = "auto" , # default. could also be "low" | "medium" | "high" create_response = True # default interrupt_response = True # default ) # Disable turn detection turn_detection = False ​ Context Management Copy Ask AI # Create context context = OpenAIRealtimeLLMContext( messages = [], tools = [], system = "You are a helpful assistant" ) # Create aggregators aggregators = service.create_context_aggregator(context) ​ Foundational Examples OpenAI Realtime Beta Example Basic implementation showing core realtime features including audio streaming, turn detection, and function calling. ​ Notes Supports real-time speech-to-speech conversation Handles interruptions and turn-taking Manages WebSocket connection lifecycle Provides function calling capabilities Supports conversation context management Includes comprehensive error handling Manages audio streaming and processing Handles both text and audio modalities Gemini Multimodal Live fal On this page Installation Configuration Constructor Parameters Session Properties Input Frames Audio Input Control Input Context Input Output Frames Audio Output Control Output Text Output Events Methods Usage Example Function Calling Frame Flow Metrics Support Advanced Features Turn Detection Context Management Foundational Examples Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/s2s_openai_185c1d0d.txt b/s2s_openai_185c1d0d.txt
new file mode 100644
index 0000000000000000000000000000000000000000..fef522b1dd914e4a82174a4586136d5c07cb5d11
--- /dev/null
+++ b/s2s_openai_185c1d0d.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/s2s/openai#param-tts-started-frame
+Title: OpenAI Realtime Beta - Pipecat
+==================================================
+
+OpenAI Realtime Beta - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Speech OpenAI Realtime Beta Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech AWS Nova Sonic Gemini Multimodal Live OpenAI Realtime Beta Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline OpenAIRealtimeBetaLLMService provides real-time, multimodal conversation capabilities using OpenAI’s Realtime Beta API. It supports speech-to-speech interactions with integrated LLM processing, function calling, and advanced conversation management. Real-time Interaction Stream audio in real-time with minimal latency response times Speech Processing Built-in speech-to-text and text-to-speech capabilities with voice options Advanced Turn Detection Multiple voice activity detection options including semantic turn detection Powerful Function Calling Seamless support for calling external functions and APIs ​ Installation To use OpenAIRealtimeBetaLLMService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[openai]" You’ll also need to set up your OpenAI API key as an environment variable: OPENAI_API_KEY . ​ Configuration ​ Constructor Parameters ​ api_key str required Your OpenAI API key ​ model str default: "gpt-4o-realtime-preview-2025-06-03" The speech-to-speech model used for processing ​ base_url str default: "wss://api.openai.com/v1/realtime" WebSocket endpoint URL ​ session_properties SessionProperties Configuration for the realtime session ​ start_audio_paused bool default: "False" Whether to start with audio input paused ​ send_transcription_frames bool default: "True" Whether to emit transcription frames ​ Session Properties The SessionProperties object configures the behavior of the realtime session: ​ modalities List[Literal['text', 'audio']] The modalities to enable (default includes both text and audio) ​ instructions str System instructions that guide the model’s behavior Copy Ask AI service = OpenAIRealtimeBetaLLMService( api_key = os.getenv( "OPENAI_API_KEY" ), session_properties = SessionProperties( instructions = "You are a helpful assistant. Be concise and friendly." ) ) ​ voice str Voice ID for text-to-speech (options: alloy, echo, fable, onyx, nova, shimmer) ​ input_audio_format Literal['pcm16', 'g711_ulaw', 'g711_alaw'] Format of the input audio ​ output_audio_format Literal['pcm16', 'g711_ulaw', 'g711_alaw'] Format of the output audio ​ input_audio_transcription InputAudioTranscription Configuration for audio transcription Copy Ask AI from pipecat.services.openai_realtime_beta.events import InputAudioTranscription service = OpenAIRealtimeBetaLLMService( api_key = os.getenv( "OPENAI_API_KEY" ), session_properties = SessionProperties( input_audio_transcription = InputAudioTranscription( model = "gpt-4o-transcribe" , language = "en" , prompt = "This is a technical conversation about programming" ) ) ) ​ input_audio_noise_reduction InputAudioNoiseReduction Configuration for audio noise reduction ​ turn_detection Union[TurnDetection, SemanticTurnDetection, bool] Configuration for turn detection (set to False to disable) ​ tools List[Dict] List of function definitions for tool/function calling ​ tool_choice Literal['auto', 'none', 'required'] Controls when the model calls functions ​ temperature float Controls randomness in responses (0.0 to 2.0) ​ max_response_output_tokens Union[int, Literal['inf']] Maximum number of tokens to generate ​ Input Frames ​ Audio Input ​ InputAudioRawFrame Frame Raw audio data for speech input ​ Control Input ​ StartInterruptionFrame Frame Signals start of user interruption ​ UserStartedSpeakingFrame Frame Signals user started speaking ​ UserStoppedSpeakingFrame Frame Signals user stopped speaking ​ Context Input ​ OpenAILLMContextFrame Frame Contains conversation context ​ LLMMessagesAppendFrame Frame Appends messages to conversation ​ Output Frames ​ Audio Output ​ TTSAudioRawFrame Frame Generated speech audio ​ Control Output ​ TTSStartedFrame Frame Signals start of speech synthesis ​ TTSStoppedFrame Frame Signals end of speech synthesis ​ Text Output ​ TextFrame Frame Generated text responses ​ TranscriptionFrame Frame Speech transcriptions ​ Events ​ on_conversation_item_created event Emitted when a conversation item on the server is created. Handler receives: item_id: str item: ConversationItem ​ on_conversation_item_updated event Emitted when a conversation item on the server is updated. Handler receives: item_id: str item: Optional[ConversationItem] (may not exist for some updates) ​ Methods ​ retrieve_conversation_item method Retrieves a conversation item’s details from the server. Copy Ask AI async def retrieve_conversation_item ( self , item_id : str ) -> ConversationItem ​ Usage Example Copy Ask AI from pipecat.services.openai_realtime_beta import OpenAIRealtimeBetaLLMService from pipecat.services.openai_realtime_beta.events import SessionProperties, TurnDetection # Configure service service = OpenAIRealtimeBetaLLMService( api_key = "your-api-key" , session_properties = SessionProperties( modalities = [ "audio" , "text" ], voice = "alloy" , turn_detection = TurnDetection( threshold = 0.5 , silence_duration_ms = 800 ), temperature = 0.7 ) ) # Use in pipeline pipeline = Pipeline([ audio_input, # Produces InputAudioRawFrame service, # Processes speech/generates responses audio_output # Handles TTSAudioRawFrame ]) ​ Function Calling The service supports function calling with automatic response handling: Copy Ask AI from pipecat.adapters.schemas.function_schema import FunctionSchema from pipecat.adapters.schemas.tools_schema import ToolsSchema from pipecat.services.openai_realtime_beta import SessionProperties # Define weather function using standardized schema weather_function = FunctionSchema( name = "get_weather" , description = "Get weather information" , properties = { "location" : { "type" : "string" } }, required = [ "location" ] ) # Create tools schema tools = ToolsSchema( standard_tools = [weather_function]) # Configure service with tools llm = OpenAIRealtimeBetaLLMService( api_key = "your-api-key" , session_properties = SessionProperties( tools = tools, tool_choice = "auto" ) ) llm.register_function( "get_weather" , fetch_weather_from_api) See the Function Calling guide for: Detailed implementation instructions Provider-specific function definitions Handler registration examples Control over function call behavior Complete usage examples ​ Frame Flow ​ Metrics Support The service collects comprehensive metrics: Token usage (prompt and completion) Processing duration Time to First Byte (TTFB) Audio processing metrics Function call metrics ​ Advanced Features ​ Turn Detection Copy Ask AI # Server-side basic VAD turn_detection = TurnDetection( type = "server_vad" , threshold = 0.5 , prefix_padding_ms = 300 , silence_duration_ms = 800 ) # Server-side semantic VAD turn_detection = SemanticTurnDetection( type = "semantic_vad" , eagerness = "auto" , # default. could also be "low" | "medium" | "high" create_response = True # default interrupt_response = True # default ) # Disable turn detection turn_detection = False ​ Context Management Copy Ask AI # Create context context = OpenAIRealtimeLLMContext( messages = [], tools = [], system = "You are a helpful assistant" ) # Create aggregators aggregators = service.create_context_aggregator(context) ​ Foundational Examples OpenAI Realtime Beta Example Basic implementation showing core realtime features including audio streaming, turn detection, and function calling. ​ Notes Supports real-time speech-to-speech conversation Handles interruptions and turn-taking Manages WebSocket connection lifecycle Provides function calling capabilities Supports conversation context management Includes comprehensive error handling Manages audio streaming and processing Handles both text and audio modalities Gemini Multimodal Live fal On this page Installation Configuration Constructor Parameters Session Properties Input Frames Audio Input Control Input Context Input Output Frames Audio Output Control Output Text Output Events Methods Usage Example Function Calling Frame Flow Metrics Support Advanced Features Turn Detection Context Management Foundational Examples Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/s2s_openai_1dd2843f.txt b/s2s_openai_1dd2843f.txt
new file mode 100644
index 0000000000000000000000000000000000000000..ded7bcbe29f938d61510feb6c5b8918ea4485ec3
--- /dev/null
+++ b/s2s_openai_1dd2843f.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/s2s/openai#param-base-url
+Title: OpenAI Realtime Beta - Pipecat
+==================================================
+
+OpenAI Realtime Beta - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Speech OpenAI Realtime Beta Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech AWS Nova Sonic Gemini Multimodal Live OpenAI Realtime Beta Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline OpenAIRealtimeBetaLLMService provides real-time, multimodal conversation capabilities using OpenAI’s Realtime Beta API. It supports speech-to-speech interactions with integrated LLM processing, function calling, and advanced conversation management. Real-time Interaction Stream audio in real-time with minimal latency response times Speech Processing Built-in speech-to-text and text-to-speech capabilities with voice options Advanced Turn Detection Multiple voice activity detection options including semantic turn detection Powerful Function Calling Seamless support for calling external functions and APIs ​ Installation To use OpenAIRealtimeBetaLLMService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[openai]" You’ll also need to set up your OpenAI API key as an environment variable: OPENAI_API_KEY . ​ Configuration ​ Constructor Parameters ​ api_key str required Your OpenAI API key ​ model str default: "gpt-4o-realtime-preview-2025-06-03" The speech-to-speech model used for processing ​ base_url str default: "wss://api.openai.com/v1/realtime" WebSocket endpoint URL ​ session_properties SessionProperties Configuration for the realtime session ​ start_audio_paused bool default: "False" Whether to start with audio input paused ​ send_transcription_frames bool default: "True" Whether to emit transcription frames ​ Session Properties The SessionProperties object configures the behavior of the realtime session: ​ modalities List[Literal['text', 'audio']] The modalities to enable (default includes both text and audio) ​ instructions str System instructions that guide the model’s behavior Copy Ask AI service = OpenAIRealtimeBetaLLMService( api_key = os.getenv( "OPENAI_API_KEY" ), session_properties = SessionProperties( instructions = "You are a helpful assistant. Be concise and friendly." ) ) ​ voice str Voice ID for text-to-speech (options: alloy, echo, fable, onyx, nova, shimmer) ​ input_audio_format Literal['pcm16', 'g711_ulaw', 'g711_alaw'] Format of the input audio ​ output_audio_format Literal['pcm16', 'g711_ulaw', 'g711_alaw'] Format of the output audio ​ input_audio_transcription InputAudioTranscription Configuration for audio transcription Copy Ask AI from pipecat.services.openai_realtime_beta.events import InputAudioTranscription service = OpenAIRealtimeBetaLLMService( api_key = os.getenv( "OPENAI_API_KEY" ), session_properties = SessionProperties( input_audio_transcription = InputAudioTranscription( model = "gpt-4o-transcribe" , language = "en" , prompt = "This is a technical conversation about programming" ) ) ) ​ input_audio_noise_reduction InputAudioNoiseReduction Configuration for audio noise reduction ​ turn_detection Union[TurnDetection, SemanticTurnDetection, bool] Configuration for turn detection (set to False to disable) ​ tools List[Dict] List of function definitions for tool/function calling ​ tool_choice Literal['auto', 'none', 'required'] Controls when the model calls functions ​ temperature float Controls randomness in responses (0.0 to 2.0) ​ max_response_output_tokens Union[int, Literal['inf']] Maximum number of tokens to generate ​ Input Frames ​ Audio Input ​ InputAudioRawFrame Frame Raw audio data for speech input ​ Control Input ​ StartInterruptionFrame Frame Signals start of user interruption ​ UserStartedSpeakingFrame Frame Signals user started speaking ​ UserStoppedSpeakingFrame Frame Signals user stopped speaking ​ Context Input ​ OpenAILLMContextFrame Frame Contains conversation context ​ LLMMessagesAppendFrame Frame Appends messages to conversation ​ Output Frames ​ Audio Output ​ TTSAudioRawFrame Frame Generated speech audio ​ Control Output ​ TTSStartedFrame Frame Signals start of speech synthesis ​ TTSStoppedFrame Frame Signals end of speech synthesis ​ Text Output ​ TextFrame Frame Generated text responses ​ TranscriptionFrame Frame Speech transcriptions ​ Events ​ on_conversation_item_created event Emitted when a conversation item on the server is created. Handler receives: item_id: str item: ConversationItem ​ on_conversation_item_updated event Emitted when a conversation item on the server is updated. Handler receives: item_id: str item: Optional[ConversationItem] (may not exist for some updates) ​ Methods ​ retrieve_conversation_item method Retrieves a conversation item’s details from the server. Copy Ask AI async def retrieve_conversation_item ( self , item_id : str ) -> ConversationItem ​ Usage Example Copy Ask AI from pipecat.services.openai_realtime_beta import OpenAIRealtimeBetaLLMService from pipecat.services.openai_realtime_beta.events import SessionProperties, TurnDetection # Configure service service = OpenAIRealtimeBetaLLMService( api_key = "your-api-key" , session_properties = SessionProperties( modalities = [ "audio" , "text" ], voice = "alloy" , turn_detection = TurnDetection( threshold = 0.5 , silence_duration_ms = 800 ), temperature = 0.7 ) ) # Use in pipeline pipeline = Pipeline([ audio_input, # Produces InputAudioRawFrame service, # Processes speech/generates responses audio_output # Handles TTSAudioRawFrame ]) ​ Function Calling The service supports function calling with automatic response handling: Copy Ask AI from pipecat.adapters.schemas.function_schema import FunctionSchema from pipecat.adapters.schemas.tools_schema import ToolsSchema from pipecat.services.openai_realtime_beta import SessionProperties # Define weather function using standardized schema weather_function = FunctionSchema( name = "get_weather" , description = "Get weather information" , properties = { "location" : { "type" : "string" } }, required = [ "location" ] ) # Create tools schema tools = ToolsSchema( standard_tools = [weather_function]) # Configure service with tools llm = OpenAIRealtimeBetaLLMService( api_key = "your-api-key" , session_properties = SessionProperties( tools = tools, tool_choice = "auto" ) ) llm.register_function( "get_weather" , fetch_weather_from_api) See the Function Calling guide for: Detailed implementation instructions Provider-specific function definitions Handler registration examples Control over function call behavior Complete usage examples ​ Frame Flow ​ Metrics Support The service collects comprehensive metrics: Token usage (prompt and completion) Processing duration Time to First Byte (TTFB) Audio processing metrics Function call metrics ​ Advanced Features ​ Turn Detection Copy Ask AI # Server-side basic VAD turn_detection = TurnDetection( type = "server_vad" , threshold = 0.5 , prefix_padding_ms = 300 , silence_duration_ms = 800 ) # Server-side semantic VAD turn_detection = SemanticTurnDetection( type = "semantic_vad" , eagerness = "auto" , # default. could also be "low" | "medium" | "high" create_response = True # default interrupt_response = True # default ) # Disable turn detection turn_detection = False ​ Context Management Copy Ask AI # Create context context = OpenAIRealtimeLLMContext( messages = [], tools = [], system = "You are a helpful assistant" ) # Create aggregators aggregators = service.create_context_aggregator(context) ​ Foundational Examples OpenAI Realtime Beta Example Basic implementation showing core realtime features including audio streaming, turn detection, and function calling. ​ Notes Supports real-time speech-to-speech conversation Handles interruptions and turn-taking Manages WebSocket connection lifecycle Provides function calling capabilities Supports conversation context management Includes comprehensive error handling Manages audio streaming and processing Handles both text and audio modalities Gemini Multimodal Live fal On this page Installation Configuration Constructor Parameters Session Properties Input Frames Audio Input Control Input Context Input Output Frames Audio Output Control Output Text Output Events Methods Usage Example Function Calling Frame Flow Metrics Support Advanced Features Turn Detection Context Management Foundational Examples Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/s2s_openai_25458d67.txt b/s2s_openai_25458d67.txt
new file mode 100644
index 0000000000000000000000000000000000000000..0e05e949c186ac6f0dc420a2c51c7055a1db2357
--- /dev/null
+++ b/s2s_openai_25458d67.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/s2s/openai#configuration
+Title: OpenAI Realtime Beta - Pipecat
+==================================================
+
+OpenAI Realtime Beta - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Speech OpenAI Realtime Beta Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech AWS Nova Sonic Gemini Multimodal Live OpenAI Realtime Beta Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline OpenAIRealtimeBetaLLMService provides real-time, multimodal conversation capabilities using OpenAI’s Realtime Beta API. It supports speech-to-speech interactions with integrated LLM processing, function calling, and advanced conversation management. Real-time Interaction Stream audio in real-time with minimal latency response times Speech Processing Built-in speech-to-text and text-to-speech capabilities with voice options Advanced Turn Detection Multiple voice activity detection options including semantic turn detection Powerful Function Calling Seamless support for calling external functions and APIs ​ Installation To use OpenAIRealtimeBetaLLMService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[openai]" You’ll also need to set up your OpenAI API key as an environment variable: OPENAI_API_KEY . ​ Configuration ​ Constructor Parameters ​ api_key str required Your OpenAI API key ​ model str default: "gpt-4o-realtime-preview-2025-06-03" The speech-to-speech model used for processing ​ base_url str default: "wss://api.openai.com/v1/realtime" WebSocket endpoint URL ​ session_properties SessionProperties Configuration for the realtime session ​ start_audio_paused bool default: "False" Whether to start with audio input paused ​ send_transcription_frames bool default: "True" Whether to emit transcription frames ​ Session Properties The SessionProperties object configures the behavior of the realtime session: ​ modalities List[Literal['text', 'audio']] The modalities to enable (default includes both text and audio) ​ instructions str System instructions that guide the model’s behavior Copy Ask AI service = OpenAIRealtimeBetaLLMService( api_key = os.getenv( "OPENAI_API_KEY" ), session_properties = SessionProperties( instructions = "You are a helpful assistant. Be concise and friendly." ) ) ​ voice str Voice ID for text-to-speech (options: alloy, echo, fable, onyx, nova, shimmer) ​ input_audio_format Literal['pcm16', 'g711_ulaw', 'g711_alaw'] Format of the input audio ​ output_audio_format Literal['pcm16', 'g711_ulaw', 'g711_alaw'] Format of the output audio ​ input_audio_transcription InputAudioTranscription Configuration for audio transcription Copy Ask AI from pipecat.services.openai_realtime_beta.events import InputAudioTranscription service = OpenAIRealtimeBetaLLMService( api_key = os.getenv( "OPENAI_API_KEY" ), session_properties = SessionProperties( input_audio_transcription = InputAudioTranscription( model = "gpt-4o-transcribe" , language = "en" , prompt = "This is a technical conversation about programming" ) ) ) ​ input_audio_noise_reduction InputAudioNoiseReduction Configuration for audio noise reduction ​ turn_detection Union[TurnDetection, SemanticTurnDetection, bool] Configuration for turn detection (set to False to disable) ​ tools List[Dict] List of function definitions for tool/function calling ​ tool_choice Literal['auto', 'none', 'required'] Controls when the model calls functions ​ temperature float Controls randomness in responses (0.0 to 2.0) ​ max_response_output_tokens Union[int, Literal['inf']] Maximum number of tokens to generate ​ Input Frames ​ Audio Input ​ InputAudioRawFrame Frame Raw audio data for speech input ​ Control Input ​ StartInterruptionFrame Frame Signals start of user interruption ​ UserStartedSpeakingFrame Frame Signals user started speaking ​ UserStoppedSpeakingFrame Frame Signals user stopped speaking ​ Context Input ​ OpenAILLMContextFrame Frame Contains conversation context ​ LLMMessagesAppendFrame Frame Appends messages to conversation ​ Output Frames ​ Audio Output ​ TTSAudioRawFrame Frame Generated speech audio ​ Control Output ​ TTSStartedFrame Frame Signals start of speech synthesis ​ TTSStoppedFrame Frame Signals end of speech synthesis ​ Text Output ​ TextFrame Frame Generated text responses ​ TranscriptionFrame Frame Speech transcriptions ​ Events ​ on_conversation_item_created event Emitted when a conversation item on the server is created. Handler receives: item_id: str item: ConversationItem ​ on_conversation_item_updated event Emitted when a conversation item on the server is updated. Handler receives: item_id: str item: Optional[ConversationItem] (may not exist for some updates) ​ Methods ​ retrieve_conversation_item method Retrieves a conversation item’s details from the server. Copy Ask AI async def retrieve_conversation_item ( self , item_id : str ) -> ConversationItem ​ Usage Example Copy Ask AI from pipecat.services.openai_realtime_beta import OpenAIRealtimeBetaLLMService from pipecat.services.openai_realtime_beta.events import SessionProperties, TurnDetection # Configure service service = OpenAIRealtimeBetaLLMService( api_key = "your-api-key" , session_properties = SessionProperties( modalities = [ "audio" , "text" ], voice = "alloy" , turn_detection = TurnDetection( threshold = 0.5 , silence_duration_ms = 800 ), temperature = 0.7 ) ) # Use in pipeline pipeline = Pipeline([ audio_input, # Produces InputAudioRawFrame service, # Processes speech/generates responses audio_output # Handles TTSAudioRawFrame ]) ​ Function Calling The service supports function calling with automatic response handling: Copy Ask AI from pipecat.adapters.schemas.function_schema import FunctionSchema from pipecat.adapters.schemas.tools_schema import ToolsSchema from pipecat.services.openai_realtime_beta import SessionProperties # Define weather function using standardized schema weather_function = FunctionSchema( name = "get_weather" , description = "Get weather information" , properties = { "location" : { "type" : "string" } }, required = [ "location" ] ) # Create tools schema tools = ToolsSchema( standard_tools = [weather_function]) # Configure service with tools llm = OpenAIRealtimeBetaLLMService( api_key = "your-api-key" , session_properties = SessionProperties( tools = tools, tool_choice = "auto" ) ) llm.register_function( "get_weather" , fetch_weather_from_api) See the Function Calling guide for: Detailed implementation instructions Provider-specific function definitions Handler registration examples Control over function call behavior Complete usage examples ​ Frame Flow ​ Metrics Support The service collects comprehensive metrics: Token usage (prompt and completion) Processing duration Time to First Byte (TTFB) Audio processing metrics Function call metrics ​ Advanced Features ​ Turn Detection Copy Ask AI # Server-side basic VAD turn_detection = TurnDetection( type = "server_vad" , threshold = 0.5 , prefix_padding_ms = 300 , silence_duration_ms = 800 ) # Server-side semantic VAD turn_detection = SemanticTurnDetection( type = "semantic_vad" , eagerness = "auto" , # default. could also be "low" | "medium" | "high" create_response = True # default interrupt_response = True # default ) # Disable turn detection turn_detection = False ​ Context Management Copy Ask AI # Create context context = OpenAIRealtimeLLMContext( messages = [], tools = [], system = "You are a helpful assistant" ) # Create aggregators aggregators = service.create_context_aggregator(context) ​ Foundational Examples OpenAI Realtime Beta Example Basic implementation showing core realtime features including audio streaming, turn detection, and function calling. ​ Notes Supports real-time speech-to-speech conversation Handles interruptions and turn-taking Manages WebSocket connection lifecycle Provides function calling capabilities Supports conversation context management Includes comprehensive error handling Manages audio streaming and processing Handles both text and audio modalities Gemini Multimodal Live fal On this page Installation Configuration Constructor Parameters Session Properties Input Frames Audio Input Control Input Context Input Output Frames Audio Output Control Output Text Output Events Methods Usage Example Function Calling Frame Flow Metrics Support Advanced Features Turn Detection Context Management Foundational Examples Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/s2s_openai_2a33d57a.txt b/s2s_openai_2a33d57a.txt
new file mode 100644
index 0000000000000000000000000000000000000000..0e32d734e77bb1e4907086561429d82318e4185e
--- /dev/null
+++ b/s2s_openai_2a33d57a.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/s2s/openai#param-input-audio-raw-frame
+Title: OpenAI Realtime Beta - Pipecat
+==================================================
+
+OpenAI Realtime Beta - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Speech OpenAI Realtime Beta Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech AWS Nova Sonic Gemini Multimodal Live OpenAI Realtime Beta Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline OpenAIRealtimeBetaLLMService provides real-time, multimodal conversation capabilities using OpenAI’s Realtime Beta API. It supports speech-to-speech interactions with integrated LLM processing, function calling, and advanced conversation management. Real-time Interaction Stream audio in real-time with minimal latency response times Speech Processing Built-in speech-to-text and text-to-speech capabilities with voice options Advanced Turn Detection Multiple voice activity detection options including semantic turn detection Powerful Function Calling Seamless support for calling external functions and APIs ​ Installation To use OpenAIRealtimeBetaLLMService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[openai]" You’ll also need to set up your OpenAI API key as an environment variable: OPENAI_API_KEY . ​ Configuration ​ Constructor Parameters ​ api_key str required Your OpenAI API key ​ model str default: "gpt-4o-realtime-preview-2025-06-03" The speech-to-speech model used for processing ​ base_url str default: "wss://api.openai.com/v1/realtime" WebSocket endpoint URL ​ session_properties SessionProperties Configuration for the realtime session ​ start_audio_paused bool default: "False" Whether to start with audio input paused ​ send_transcription_frames bool default: "True" Whether to emit transcription frames ​ Session Properties The SessionProperties object configures the behavior of the realtime session: ​ modalities List[Literal['text', 'audio']] The modalities to enable (default includes both text and audio) ​ instructions str System instructions that guide the model’s behavior Copy Ask AI service = OpenAIRealtimeBetaLLMService( api_key = os.getenv( "OPENAI_API_KEY" ), session_properties = SessionProperties( instructions = "You are a helpful assistant. Be concise and friendly." ) ) ​ voice str Voice ID for text-to-speech (options: alloy, echo, fable, onyx, nova, shimmer) ​ input_audio_format Literal['pcm16', 'g711_ulaw', 'g711_alaw'] Format of the input audio ​ output_audio_format Literal['pcm16', 'g711_ulaw', 'g711_alaw'] Format of the output audio ​ input_audio_transcription InputAudioTranscription Configuration for audio transcription Copy Ask AI from pipecat.services.openai_realtime_beta.events import InputAudioTranscription service = OpenAIRealtimeBetaLLMService( api_key = os.getenv( "OPENAI_API_KEY" ), session_properties = SessionProperties( input_audio_transcription = InputAudioTranscription( model = "gpt-4o-transcribe" , language = "en" , prompt = "This is a technical conversation about programming" ) ) ) ​ input_audio_noise_reduction InputAudioNoiseReduction Configuration for audio noise reduction ​ turn_detection Union[TurnDetection, SemanticTurnDetection, bool] Configuration for turn detection (set to False to disable) ​ tools List[Dict] List of function definitions for tool/function calling ​ tool_choice Literal['auto', 'none', 'required'] Controls when the model calls functions ​ temperature float Controls randomness in responses (0.0 to 2.0) ​ max_response_output_tokens Union[int, Literal['inf']] Maximum number of tokens to generate ​ Input Frames ​ Audio Input ​ InputAudioRawFrame Frame Raw audio data for speech input ​ Control Input ​ StartInterruptionFrame Frame Signals start of user interruption ​ UserStartedSpeakingFrame Frame Signals user started speaking ​ UserStoppedSpeakingFrame Frame Signals user stopped speaking ​ Context Input ​ OpenAILLMContextFrame Frame Contains conversation context ​ LLMMessagesAppendFrame Frame Appends messages to conversation ​ Output Frames ​ Audio Output ​ TTSAudioRawFrame Frame Generated speech audio ​ Control Output ​ TTSStartedFrame Frame Signals start of speech synthesis ​ TTSStoppedFrame Frame Signals end of speech synthesis ​ Text Output ​ TextFrame Frame Generated text responses ​ TranscriptionFrame Frame Speech transcriptions ​ Events ​ on_conversation_item_created event Emitted when a conversation item on the server is created. Handler receives: item_id: str item: ConversationItem ​ on_conversation_item_updated event Emitted when a conversation item on the server is updated. Handler receives: item_id: str item: Optional[ConversationItem] (may not exist for some updates) ​ Methods ​ retrieve_conversation_item method Retrieves a conversation item’s details from the server. Copy Ask AI async def retrieve_conversation_item ( self , item_id : str ) -> ConversationItem ​ Usage Example Copy Ask AI from pipecat.services.openai_realtime_beta import OpenAIRealtimeBetaLLMService from pipecat.services.openai_realtime_beta.events import SessionProperties, TurnDetection # Configure service service = OpenAIRealtimeBetaLLMService( api_key = "your-api-key" , session_properties = SessionProperties( modalities = [ "audio" , "text" ], voice = "alloy" , turn_detection = TurnDetection( threshold = 0.5 , silence_duration_ms = 800 ), temperature = 0.7 ) ) # Use in pipeline pipeline = Pipeline([ audio_input, # Produces InputAudioRawFrame service, # Processes speech/generates responses audio_output # Handles TTSAudioRawFrame ]) ​ Function Calling The service supports function calling with automatic response handling: Copy Ask AI from pipecat.adapters.schemas.function_schema import FunctionSchema from pipecat.adapters.schemas.tools_schema import ToolsSchema from pipecat.services.openai_realtime_beta import SessionProperties # Define weather function using standardized schema weather_function = FunctionSchema( name = "get_weather" , description = "Get weather information" , properties = { "location" : { "type" : "string" } }, required = [ "location" ] ) # Create tools schema tools = ToolsSchema( standard_tools = [weather_function]) # Configure service with tools llm = OpenAIRealtimeBetaLLMService( api_key = "your-api-key" , session_properties = SessionProperties( tools = tools, tool_choice = "auto" ) ) llm.register_function( "get_weather" , fetch_weather_from_api) See the Function Calling guide for: Detailed implementation instructions Provider-specific function definitions Handler registration examples Control over function call behavior Complete usage examples ​ Frame Flow ​ Metrics Support The service collects comprehensive metrics: Token usage (prompt and completion) Processing duration Time to First Byte (TTFB) Audio processing metrics Function call metrics ​ Advanced Features ​ Turn Detection Copy Ask AI # Server-side basic VAD turn_detection = TurnDetection( type = "server_vad" , threshold = 0.5 , prefix_padding_ms = 300 , silence_duration_ms = 800 ) # Server-side semantic VAD turn_detection = SemanticTurnDetection( type = "semantic_vad" , eagerness = "auto" , # default. could also be "low" | "medium" | "high" create_response = True # default interrupt_response = True # default ) # Disable turn detection turn_detection = False ​ Context Management Copy Ask AI # Create context context = OpenAIRealtimeLLMContext( messages = [], tools = [], system = "You are a helpful assistant" ) # Create aggregators aggregators = service.create_context_aggregator(context) ​ Foundational Examples OpenAI Realtime Beta Example Basic implementation showing core realtime features including audio streaming, turn detection, and function calling. ​ Notes Supports real-time speech-to-speech conversation Handles interruptions and turn-taking Manages WebSocket connection lifecycle Provides function calling capabilities Supports conversation context management Includes comprehensive error handling Manages audio streaming and processing Handles both text and audio modalities Gemini Multimodal Live fal On this page Installation Configuration Constructor Parameters Session Properties Input Frames Audio Input Control Input Context Input Output Frames Audio Output Control Output Text Output Events Methods Usage Example Function Calling Frame Flow Metrics Support Advanced Features Turn Detection Context Management Foundational Examples Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/s2s_openai_3a60b005.txt b/s2s_openai_3a60b005.txt
new file mode 100644
index 0000000000000000000000000000000000000000..f2be0a90dca592c0a63a40635705a1e7adfa9df5
--- /dev/null
+++ b/s2s_openai_3a60b005.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/s2s/openai#methods
+Title: OpenAI Realtime Beta - Pipecat
+==================================================
+
+OpenAI Realtime Beta - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Speech OpenAI Realtime Beta Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech AWS Nova Sonic Gemini Multimodal Live OpenAI Realtime Beta Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline OpenAIRealtimeBetaLLMService provides real-time, multimodal conversation capabilities using OpenAI’s Realtime Beta API. It supports speech-to-speech interactions with integrated LLM processing, function calling, and advanced conversation management. Real-time Interaction Stream audio in real-time with minimal latency response times Speech Processing Built-in speech-to-text and text-to-speech capabilities with voice options Advanced Turn Detection Multiple voice activity detection options including semantic turn detection Powerful Function Calling Seamless support for calling external functions and APIs ​ Installation To use OpenAIRealtimeBetaLLMService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[openai]" You’ll also need to set up your OpenAI API key as an environment variable: OPENAI_API_KEY . ​ Configuration ​ Constructor Parameters ​ api_key str required Your OpenAI API key ​ model str default: "gpt-4o-realtime-preview-2025-06-03" The speech-to-speech model used for processing ​ base_url str default: "wss://api.openai.com/v1/realtime" WebSocket endpoint URL ​ session_properties SessionProperties Configuration for the realtime session ​ start_audio_paused bool default: "False" Whether to start with audio input paused ​ send_transcription_frames bool default: "True" Whether to emit transcription frames ​ Session Properties The SessionProperties object configures the behavior of the realtime session: ​ modalities List[Literal['text', 'audio']] The modalities to enable (default includes both text and audio) ​ instructions str System instructions that guide the model’s behavior Copy Ask AI service = OpenAIRealtimeBetaLLMService( api_key = os.getenv( "OPENAI_API_KEY" ), session_properties = SessionProperties( instructions = "You are a helpful assistant. Be concise and friendly." ) ) ​ voice str Voice ID for text-to-speech (options: alloy, echo, fable, onyx, nova, shimmer) ​ input_audio_format Literal['pcm16', 'g711_ulaw', 'g711_alaw'] Format of the input audio ​ output_audio_format Literal['pcm16', 'g711_ulaw', 'g711_alaw'] Format of the output audio ​ input_audio_transcription InputAudioTranscription Configuration for audio transcription Copy Ask AI from pipecat.services.openai_realtime_beta.events import InputAudioTranscription service = OpenAIRealtimeBetaLLMService( api_key = os.getenv( "OPENAI_API_KEY" ), session_properties = SessionProperties( input_audio_transcription = InputAudioTranscription( model = "gpt-4o-transcribe" , language = "en" , prompt = "This is a technical conversation about programming" ) ) ) ​ input_audio_noise_reduction InputAudioNoiseReduction Configuration for audio noise reduction ​ turn_detection Union[TurnDetection, SemanticTurnDetection, bool] Configuration for turn detection (set to False to disable) ​ tools List[Dict] List of function definitions for tool/function calling ​ tool_choice Literal['auto', 'none', 'required'] Controls when the model calls functions ​ temperature float Controls randomness in responses (0.0 to 2.0) ​ max_response_output_tokens Union[int, Literal['inf']] Maximum number of tokens to generate ​ Input Frames ​ Audio Input ​ InputAudioRawFrame Frame Raw audio data for speech input ​ Control Input ​ StartInterruptionFrame Frame Signals start of user interruption ​ UserStartedSpeakingFrame Frame Signals user started speaking ​ UserStoppedSpeakingFrame Frame Signals user stopped speaking ​ Context Input ​ OpenAILLMContextFrame Frame Contains conversation context ​ LLMMessagesAppendFrame Frame Appends messages to conversation ​ Output Frames ​ Audio Output ​ TTSAudioRawFrame Frame Generated speech audio ​ Control Output ​ TTSStartedFrame Frame Signals start of speech synthesis ​ TTSStoppedFrame Frame Signals end of speech synthesis ​ Text Output ​ TextFrame Frame Generated text responses ​ TranscriptionFrame Frame Speech transcriptions ​ Events ​ on_conversation_item_created event Emitted when a conversation item on the server is created. Handler receives: item_id: str item: ConversationItem ​ on_conversation_item_updated event Emitted when a conversation item on the server is updated. Handler receives: item_id: str item: Optional[ConversationItem] (may not exist for some updates) ​ Methods ​ retrieve_conversation_item method Retrieves a conversation item’s details from the server. Copy Ask AI async def retrieve_conversation_item ( self , item_id : str ) -> ConversationItem ​ Usage Example Copy Ask AI from pipecat.services.openai_realtime_beta import OpenAIRealtimeBetaLLMService from pipecat.services.openai_realtime_beta.events import SessionProperties, TurnDetection # Configure service service = OpenAIRealtimeBetaLLMService( api_key = "your-api-key" , session_properties = SessionProperties( modalities = [ "audio" , "text" ], voice = "alloy" , turn_detection = TurnDetection( threshold = 0.5 , silence_duration_ms = 800 ), temperature = 0.7 ) ) # Use in pipeline pipeline = Pipeline([ audio_input, # Produces InputAudioRawFrame service, # Processes speech/generates responses audio_output # Handles TTSAudioRawFrame ]) ​ Function Calling The service supports function calling with automatic response handling: Copy Ask AI from pipecat.adapters.schemas.function_schema import FunctionSchema from pipecat.adapters.schemas.tools_schema import ToolsSchema from pipecat.services.openai_realtime_beta import SessionProperties # Define weather function using standardized schema weather_function = FunctionSchema( name = "get_weather" , description = "Get weather information" , properties = { "location" : { "type" : "string" } }, required = [ "location" ] ) # Create tools schema tools = ToolsSchema( standard_tools = [weather_function]) # Configure service with tools llm = OpenAIRealtimeBetaLLMService( api_key = "your-api-key" , session_properties = SessionProperties( tools = tools, tool_choice = "auto" ) ) llm.register_function( "get_weather" , fetch_weather_from_api) See the Function Calling guide for: Detailed implementation instructions Provider-specific function definitions Handler registration examples Control over function call behavior Complete usage examples ​ Frame Flow ​ Metrics Support The service collects comprehensive metrics: Token usage (prompt and completion) Processing duration Time to First Byte (TTFB) Audio processing metrics Function call metrics ​ Advanced Features ​ Turn Detection Copy Ask AI # Server-side basic VAD turn_detection = TurnDetection( type = "server_vad" , threshold = 0.5 , prefix_padding_ms = 300 , silence_duration_ms = 800 ) # Server-side semantic VAD turn_detection = SemanticTurnDetection( type = "semantic_vad" , eagerness = "auto" , # default. could also be "low" | "medium" | "high" create_response = True # default interrupt_response = True # default ) # Disable turn detection turn_detection = False ​ Context Management Copy Ask AI # Create context context = OpenAIRealtimeLLMContext( messages = [], tools = [], system = "You are a helpful assistant" ) # Create aggregators aggregators = service.create_context_aggregator(context) ​ Foundational Examples OpenAI Realtime Beta Example Basic implementation showing core realtime features including audio streaming, turn detection, and function calling. ​ Notes Supports real-time speech-to-speech conversation Handles interruptions and turn-taking Manages WebSocket connection lifecycle Provides function calling capabilities Supports conversation context management Includes comprehensive error handling Manages audio streaming and processing Handles both text and audio modalities Gemini Multimodal Live fal On this page Installation Configuration Constructor Parameters Session Properties Input Frames Audio Input Control Input Context Input Output Frames Audio Output Control Output Text Output Events Methods Usage Example Function Calling Frame Flow Metrics Support Advanced Features Turn Detection Context Management Foundational Examples Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/s2s_openai_58b60c5b.txt b/s2s_openai_58b60c5b.txt
new file mode 100644
index 0000000000000000000000000000000000000000..00082f9bb7cba79ce8fe72517ae6aa7603da0578
--- /dev/null
+++ b/s2s_openai_58b60c5b.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/s2s/openai#param-voice
+Title: OpenAI Realtime Beta - Pipecat
+==================================================
+
+OpenAI Realtime Beta - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Speech OpenAI Realtime Beta Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech AWS Nova Sonic Gemini Multimodal Live OpenAI Realtime Beta Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline OpenAIRealtimeBetaLLMService provides real-time, multimodal conversation capabilities using OpenAI’s Realtime Beta API. It supports speech-to-speech interactions with integrated LLM processing, function calling, and advanced conversation management. Real-time Interaction Stream audio in real-time with minimal latency response times Speech Processing Built-in speech-to-text and text-to-speech capabilities with voice options Advanced Turn Detection Multiple voice activity detection options including semantic turn detection Powerful Function Calling Seamless support for calling external functions and APIs ​ Installation To use OpenAIRealtimeBetaLLMService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[openai]" You’ll also need to set up your OpenAI API key as an environment variable: OPENAI_API_KEY . ​ Configuration ​ Constructor Parameters ​ api_key str required Your OpenAI API key ​ model str default: "gpt-4o-realtime-preview-2025-06-03" The speech-to-speech model used for processing ​ base_url str default: "wss://api.openai.com/v1/realtime" WebSocket endpoint URL ​ session_properties SessionProperties Configuration for the realtime session ​ start_audio_paused bool default: "False" Whether to start with audio input paused ​ send_transcription_frames bool default: "True" Whether to emit transcription frames ​ Session Properties The SessionProperties object configures the behavior of the realtime session: ​ modalities List[Literal['text', 'audio']] The modalities to enable (default includes both text and audio) ​ instructions str System instructions that guide the model’s behavior Copy Ask AI service = OpenAIRealtimeBetaLLMService( api_key = os.getenv( "OPENAI_API_KEY" ), session_properties = SessionProperties( instructions = "You are a helpful assistant. Be concise and friendly." ) ) ​ voice str Voice ID for text-to-speech (options: alloy, echo, fable, onyx, nova, shimmer) ​ input_audio_format Literal['pcm16', 'g711_ulaw', 'g711_alaw'] Format of the input audio ​ output_audio_format Literal['pcm16', 'g711_ulaw', 'g711_alaw'] Format of the output audio ​ input_audio_transcription InputAudioTranscription Configuration for audio transcription Copy Ask AI from pipecat.services.openai_realtime_beta.events import InputAudioTranscription service = OpenAIRealtimeBetaLLMService( api_key = os.getenv( "OPENAI_API_KEY" ), session_properties = SessionProperties( input_audio_transcription = InputAudioTranscription( model = "gpt-4o-transcribe" , language = "en" , prompt = "This is a technical conversation about programming" ) ) ) ​ input_audio_noise_reduction InputAudioNoiseReduction Configuration for audio noise reduction ​ turn_detection Union[TurnDetection, SemanticTurnDetection, bool] Configuration for turn detection (set to False to disable) ​ tools List[Dict] List of function definitions for tool/function calling ​ tool_choice Literal['auto', 'none', 'required'] Controls when the model calls functions ​ temperature float Controls randomness in responses (0.0 to 2.0) ​ max_response_output_tokens Union[int, Literal['inf']] Maximum number of tokens to generate ​ Input Frames ​ Audio Input ​ InputAudioRawFrame Frame Raw audio data for speech input ​ Control Input ​ StartInterruptionFrame Frame Signals start of user interruption ​ UserStartedSpeakingFrame Frame Signals user started speaking ​ UserStoppedSpeakingFrame Frame Signals user stopped speaking ​ Context Input ​ OpenAILLMContextFrame Frame Contains conversation context ​ LLMMessagesAppendFrame Frame Appends messages to conversation ​ Output Frames ​ Audio Output ​ TTSAudioRawFrame Frame Generated speech audio ​ Control Output ​ TTSStartedFrame Frame Signals start of speech synthesis ​ TTSStoppedFrame Frame Signals end of speech synthesis ​ Text Output ​ TextFrame Frame Generated text responses ​ TranscriptionFrame Frame Speech transcriptions ​ Events ​ on_conversation_item_created event Emitted when a conversation item on the server is created. Handler receives: item_id: str item: ConversationItem ​ on_conversation_item_updated event Emitted when a conversation item on the server is updated. Handler receives: item_id: str item: Optional[ConversationItem] (may not exist for some updates) ​ Methods ​ retrieve_conversation_item method Retrieves a conversation item’s details from the server. Copy Ask AI async def retrieve_conversation_item ( self , item_id : str ) -> ConversationItem ​ Usage Example Copy Ask AI from pipecat.services.openai_realtime_beta import OpenAIRealtimeBetaLLMService from pipecat.services.openai_realtime_beta.events import SessionProperties, TurnDetection # Configure service service = OpenAIRealtimeBetaLLMService( api_key = "your-api-key" , session_properties = SessionProperties( modalities = [ "audio" , "text" ], voice = "alloy" , turn_detection = TurnDetection( threshold = 0.5 , silence_duration_ms = 800 ), temperature = 0.7 ) ) # Use in pipeline pipeline = Pipeline([ audio_input, # Produces InputAudioRawFrame service, # Processes speech/generates responses audio_output # Handles TTSAudioRawFrame ]) ​ Function Calling The service supports function calling with automatic response handling: Copy Ask AI from pipecat.adapters.schemas.function_schema import FunctionSchema from pipecat.adapters.schemas.tools_schema import ToolsSchema from pipecat.services.openai_realtime_beta import SessionProperties # Define weather function using standardized schema weather_function = FunctionSchema( name = "get_weather" , description = "Get weather information" , properties = { "location" : { "type" : "string" } }, required = [ "location" ] ) # Create tools schema tools = ToolsSchema( standard_tools = [weather_function]) # Configure service with tools llm = OpenAIRealtimeBetaLLMService( api_key = "your-api-key" , session_properties = SessionProperties( tools = tools, tool_choice = "auto" ) ) llm.register_function( "get_weather" , fetch_weather_from_api) See the Function Calling guide for: Detailed implementation instructions Provider-specific function definitions Handler registration examples Control over function call behavior Complete usage examples ​ Frame Flow ​ Metrics Support The service collects comprehensive metrics: Token usage (prompt and completion) Processing duration Time to First Byte (TTFB) Audio processing metrics Function call metrics ​ Advanced Features ​ Turn Detection Copy Ask AI # Server-side basic VAD turn_detection = TurnDetection( type = "server_vad" , threshold = 0.5 , prefix_padding_ms = 300 , silence_duration_ms = 800 ) # Server-side semantic VAD turn_detection = SemanticTurnDetection( type = "semantic_vad" , eagerness = "auto" , # default. could also be "low" | "medium" | "high" create_response = True # default interrupt_response = True # default ) # Disable turn detection turn_detection = False ​ Context Management Copy Ask AI # Create context context = OpenAIRealtimeLLMContext( messages = [], tools = [], system = "You are a helpful assistant" ) # Create aggregators aggregators = service.create_context_aggregator(context) ​ Foundational Examples OpenAI Realtime Beta Example Basic implementation showing core realtime features including audio streaming, turn detection, and function calling. ​ Notes Supports real-time speech-to-speech conversation Handles interruptions and turn-taking Manages WebSocket connection lifecycle Provides function calling capabilities Supports conversation context management Includes comprehensive error handling Manages audio streaming and processing Handles both text and audio modalities Gemini Multimodal Live fal On this page Installation Configuration Constructor Parameters Session Properties Input Frames Audio Input Control Input Context Input Output Frames Audio Output Control Output Text Output Events Methods Usage Example Function Calling Frame Flow Metrics Support Advanced Features Turn Detection Context Management Foundational Examples Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/s2s_openai_74514852.txt b/s2s_openai_74514852.txt
new file mode 100644
index 0000000000000000000000000000000000000000..40acaee41dd3ccf2a9bcd9337b78801e8d0e85ed
--- /dev/null
+++ b/s2s_openai_74514852.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/s2s/openai#param-start-interruption-frame
+Title: OpenAI Realtime Beta - Pipecat
+==================================================
+
+OpenAI Realtime Beta - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Speech OpenAI Realtime Beta Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech AWS Nova Sonic Gemini Multimodal Live OpenAI Realtime Beta Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline OpenAIRealtimeBetaLLMService provides real-time, multimodal conversation capabilities using OpenAI’s Realtime Beta API. It supports speech-to-speech interactions with integrated LLM processing, function calling, and advanced conversation management. Real-time Interaction Stream audio in real-time with minimal latency response times Speech Processing Built-in speech-to-text and text-to-speech capabilities with voice options Advanced Turn Detection Multiple voice activity detection options including semantic turn detection Powerful Function Calling Seamless support for calling external functions and APIs ​ Installation To use OpenAIRealtimeBetaLLMService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[openai]" You’ll also need to set up your OpenAI API key as an environment variable: OPENAI_API_KEY . ​ Configuration ​ Constructor Parameters ​ api_key str required Your OpenAI API key ​ model str default: "gpt-4o-realtime-preview-2025-06-03" The speech-to-speech model used for processing ​ base_url str default: "wss://api.openai.com/v1/realtime" WebSocket endpoint URL ​ session_properties SessionProperties Configuration for the realtime session ​ start_audio_paused bool default: "False" Whether to start with audio input paused ​ send_transcription_frames bool default: "True" Whether to emit transcription frames ​ Session Properties The SessionProperties object configures the behavior of the realtime session: ​ modalities List[Literal['text', 'audio']] The modalities to enable (default includes both text and audio) ​ instructions str System instructions that guide the model’s behavior Copy Ask AI service = OpenAIRealtimeBetaLLMService( api_key = os.getenv( "OPENAI_API_KEY" ), session_properties = SessionProperties( instructions = "You are a helpful assistant. Be concise and friendly." ) ) ​ voice str Voice ID for text-to-speech (options: alloy, echo, fable, onyx, nova, shimmer) ​ input_audio_format Literal['pcm16', 'g711_ulaw', 'g711_alaw'] Format of the input audio ​ output_audio_format Literal['pcm16', 'g711_ulaw', 'g711_alaw'] Format of the output audio ​ input_audio_transcription InputAudioTranscription Configuration for audio transcription Copy Ask AI from pipecat.services.openai_realtime_beta.events import InputAudioTranscription service = OpenAIRealtimeBetaLLMService( api_key = os.getenv( "OPENAI_API_KEY" ), session_properties = SessionProperties( input_audio_transcription = InputAudioTranscription( model = "gpt-4o-transcribe" , language = "en" , prompt = "This is a technical conversation about programming" ) ) ) ​ input_audio_noise_reduction InputAudioNoiseReduction Configuration for audio noise reduction ​ turn_detection Union[TurnDetection, SemanticTurnDetection, bool] Configuration for turn detection (set to False to disable) ​ tools List[Dict] List of function definitions for tool/function calling ​ tool_choice Literal['auto', 'none', 'required'] Controls when the model calls functions ​ temperature float Controls randomness in responses (0.0 to 2.0) ​ max_response_output_tokens Union[int, Literal['inf']] Maximum number of tokens to generate ​ Input Frames ​ Audio Input ​ InputAudioRawFrame Frame Raw audio data for speech input ​ Control Input ​ StartInterruptionFrame Frame Signals start of user interruption ​ UserStartedSpeakingFrame Frame Signals user started speaking ​ UserStoppedSpeakingFrame Frame Signals user stopped speaking ​ Context Input ​ OpenAILLMContextFrame Frame Contains conversation context ​ LLMMessagesAppendFrame Frame Appends messages to conversation ​ Output Frames ​ Audio Output ​ TTSAudioRawFrame Frame Generated speech audio ​ Control Output ​ TTSStartedFrame Frame Signals start of speech synthesis ​ TTSStoppedFrame Frame Signals end of speech synthesis ​ Text Output ​ TextFrame Frame Generated text responses ​ TranscriptionFrame Frame Speech transcriptions ​ Events ​ on_conversation_item_created event Emitted when a conversation item on the server is created. Handler receives: item_id: str item: ConversationItem ​ on_conversation_item_updated event Emitted when a conversation item on the server is updated. Handler receives: item_id: str item: Optional[ConversationItem] (may not exist for some updates) ​ Methods ​ retrieve_conversation_item method Retrieves a conversation item’s details from the server. Copy Ask AI async def retrieve_conversation_item ( self , item_id : str ) -> ConversationItem ​ Usage Example Copy Ask AI from pipecat.services.openai_realtime_beta import OpenAIRealtimeBetaLLMService from pipecat.services.openai_realtime_beta.events import SessionProperties, TurnDetection # Configure service service = OpenAIRealtimeBetaLLMService( api_key = "your-api-key" , session_properties = SessionProperties( modalities = [ "audio" , "text" ], voice = "alloy" , turn_detection = TurnDetection( threshold = 0.5 , silence_duration_ms = 800 ), temperature = 0.7 ) ) # Use in pipeline pipeline = Pipeline([ audio_input, # Produces InputAudioRawFrame service, # Processes speech/generates responses audio_output # Handles TTSAudioRawFrame ]) ​ Function Calling The service supports function calling with automatic response handling: Copy Ask AI from pipecat.adapters.schemas.function_schema import FunctionSchema from pipecat.adapters.schemas.tools_schema import ToolsSchema from pipecat.services.openai_realtime_beta import SessionProperties # Define weather function using standardized schema weather_function = FunctionSchema( name = "get_weather" , description = "Get weather information" , properties = { "location" : { "type" : "string" } }, required = [ "location" ] ) # Create tools schema tools = ToolsSchema( standard_tools = [weather_function]) # Configure service with tools llm = OpenAIRealtimeBetaLLMService( api_key = "your-api-key" , session_properties = SessionProperties( tools = tools, tool_choice = "auto" ) ) llm.register_function( "get_weather" , fetch_weather_from_api) See the Function Calling guide for: Detailed implementation instructions Provider-specific function definitions Handler registration examples Control over function call behavior Complete usage examples ​ Frame Flow ​ Metrics Support The service collects comprehensive metrics: Token usage (prompt and completion) Processing duration Time to First Byte (TTFB) Audio processing metrics Function call metrics ​ Advanced Features ​ Turn Detection Copy Ask AI # Server-side basic VAD turn_detection = TurnDetection( type = "server_vad" , threshold = 0.5 , prefix_padding_ms = 300 , silence_duration_ms = 800 ) # Server-side semantic VAD turn_detection = SemanticTurnDetection( type = "semantic_vad" , eagerness = "auto" , # default. could also be "low" | "medium" | "high" create_response = True # default interrupt_response = True # default ) # Disable turn detection turn_detection = False ​ Context Management Copy Ask AI # Create context context = OpenAIRealtimeLLMContext( messages = [], tools = [], system = "You are a helpful assistant" ) # Create aggregators aggregators = service.create_context_aggregator(context) ​ Foundational Examples OpenAI Realtime Beta Example Basic implementation showing core realtime features including audio streaming, turn detection, and function calling. ​ Notes Supports real-time speech-to-speech conversation Handles interruptions and turn-taking Manages WebSocket connection lifecycle Provides function calling capabilities Supports conversation context management Includes comprehensive error handling Manages audio streaming and processing Handles both text and audio modalities Gemini Multimodal Live fal On this page Installation Configuration Constructor Parameters Session Properties Input Frames Audio Input Control Input Context Input Output Frames Audio Output Control Output Text Output Events Methods Usage Example Function Calling Frame Flow Metrics Support Advanced Features Turn Detection Context Management Foundational Examples Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/s2s_openai_7c449558.txt b/s2s_openai_7c449558.txt
new file mode 100644
index 0000000000000000000000000000000000000000..498422f0ec75d32312caed8b6aac27624e70306b
--- /dev/null
+++ b/s2s_openai_7c449558.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/s2s/openai#param-llm-messages-append-frame
+Title: OpenAI Realtime Beta - Pipecat
+==================================================
+
+OpenAI Realtime Beta - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Speech OpenAI Realtime Beta Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech AWS Nova Sonic Gemini Multimodal Live OpenAI Realtime Beta Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline OpenAIRealtimeBetaLLMService provides real-time, multimodal conversation capabilities using OpenAI’s Realtime Beta API. It supports speech-to-speech interactions with integrated LLM processing, function calling, and advanced conversation management. Real-time Interaction Stream audio in real-time with minimal latency response times Speech Processing Built-in speech-to-text and text-to-speech capabilities with voice options Advanced Turn Detection Multiple voice activity detection options including semantic turn detection Powerful Function Calling Seamless support for calling external functions and APIs ​ Installation To use OpenAIRealtimeBetaLLMService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[openai]" You’ll also need to set up your OpenAI API key as an environment variable: OPENAI_API_KEY . ​ Configuration ​ Constructor Parameters ​ api_key str required Your OpenAI API key ​ model str default: "gpt-4o-realtime-preview-2025-06-03" The speech-to-speech model used for processing ​ base_url str default: "wss://api.openai.com/v1/realtime" WebSocket endpoint URL ​ session_properties SessionProperties Configuration for the realtime session ​ start_audio_paused bool default: "False" Whether to start with audio input paused ​ send_transcription_frames bool default: "True" Whether to emit transcription frames ​ Session Properties The SessionProperties object configures the behavior of the realtime session: ​ modalities List[Literal['text', 'audio']] The modalities to enable (default includes both text and audio) ​ instructions str System instructions that guide the model’s behavior Copy Ask AI service = OpenAIRealtimeBetaLLMService( api_key = os.getenv( "OPENAI_API_KEY" ), session_properties = SessionProperties( instructions = "You are a helpful assistant. Be concise and friendly." ) ) ​ voice str Voice ID for text-to-speech (options: alloy, echo, fable, onyx, nova, shimmer) ​ input_audio_format Literal['pcm16', 'g711_ulaw', 'g711_alaw'] Format of the input audio ​ output_audio_format Literal['pcm16', 'g711_ulaw', 'g711_alaw'] Format of the output audio ​ input_audio_transcription InputAudioTranscription Configuration for audio transcription Copy Ask AI from pipecat.services.openai_realtime_beta.events import InputAudioTranscription service = OpenAIRealtimeBetaLLMService( api_key = os.getenv( "OPENAI_API_KEY" ), session_properties = SessionProperties( input_audio_transcription = InputAudioTranscription( model = "gpt-4o-transcribe" , language = "en" , prompt = "This is a technical conversation about programming" ) ) ) ​ input_audio_noise_reduction InputAudioNoiseReduction Configuration for audio noise reduction ​ turn_detection Union[TurnDetection, SemanticTurnDetection, bool] Configuration for turn detection (set to False to disable) ​ tools List[Dict] List of function definitions for tool/function calling ​ tool_choice Literal['auto', 'none', 'required'] Controls when the model calls functions ​ temperature float Controls randomness in responses (0.0 to 2.0) ​ max_response_output_tokens Union[int, Literal['inf']] Maximum number of tokens to generate ​ Input Frames ​ Audio Input ​ InputAudioRawFrame Frame Raw audio data for speech input ​ Control Input ​ StartInterruptionFrame Frame Signals start of user interruption ​ UserStartedSpeakingFrame Frame Signals user started speaking ​ UserStoppedSpeakingFrame Frame Signals user stopped speaking ​ Context Input ​ OpenAILLMContextFrame Frame Contains conversation context ​ LLMMessagesAppendFrame Frame Appends messages to conversation ​ Output Frames ​ Audio Output ​ TTSAudioRawFrame Frame Generated speech audio ​ Control Output ​ TTSStartedFrame Frame Signals start of speech synthesis ​ TTSStoppedFrame Frame Signals end of speech synthesis ​ Text Output ​ TextFrame Frame Generated text responses ​ TranscriptionFrame Frame Speech transcriptions ​ Events ​ on_conversation_item_created event Emitted when a conversation item on the server is created. Handler receives: item_id: str item: ConversationItem ​ on_conversation_item_updated event Emitted when a conversation item on the server is updated. Handler receives: item_id: str item: Optional[ConversationItem] (may not exist for some updates) ​ Methods ​ retrieve_conversation_item method Retrieves a conversation item’s details from the server. Copy Ask AI async def retrieve_conversation_item ( self , item_id : str ) -> ConversationItem ​ Usage Example Copy Ask AI from pipecat.services.openai_realtime_beta import OpenAIRealtimeBetaLLMService from pipecat.services.openai_realtime_beta.events import SessionProperties, TurnDetection # Configure service service = OpenAIRealtimeBetaLLMService( api_key = "your-api-key" , session_properties = SessionProperties( modalities = [ "audio" , "text" ], voice = "alloy" , turn_detection = TurnDetection( threshold = 0.5 , silence_duration_ms = 800 ), temperature = 0.7 ) ) # Use in pipeline pipeline = Pipeline([ audio_input, # Produces InputAudioRawFrame service, # Processes speech/generates responses audio_output # Handles TTSAudioRawFrame ]) ​ Function Calling The service supports function calling with automatic response handling: Copy Ask AI from pipecat.adapters.schemas.function_schema import FunctionSchema from pipecat.adapters.schemas.tools_schema import ToolsSchema from pipecat.services.openai_realtime_beta import SessionProperties # Define weather function using standardized schema weather_function = FunctionSchema( name = "get_weather" , description = "Get weather information" , properties = { "location" : { "type" : "string" } }, required = [ "location" ] ) # Create tools schema tools = ToolsSchema( standard_tools = [weather_function]) # Configure service with tools llm = OpenAIRealtimeBetaLLMService( api_key = "your-api-key" , session_properties = SessionProperties( tools = tools, tool_choice = "auto" ) ) llm.register_function( "get_weather" , fetch_weather_from_api) See the Function Calling guide for: Detailed implementation instructions Provider-specific function definitions Handler registration examples Control over function call behavior Complete usage examples ​ Frame Flow ​ Metrics Support The service collects comprehensive metrics: Token usage (prompt and completion) Processing duration Time to First Byte (TTFB) Audio processing metrics Function call metrics ​ Advanced Features ​ Turn Detection Copy Ask AI # Server-side basic VAD turn_detection = TurnDetection( type = "server_vad" , threshold = 0.5 , prefix_padding_ms = 300 , silence_duration_ms = 800 ) # Server-side semantic VAD turn_detection = SemanticTurnDetection( type = "semantic_vad" , eagerness = "auto" , # default. could also be "low" | "medium" | "high" create_response = True # default interrupt_response = True # default ) # Disable turn detection turn_detection = False ​ Context Management Copy Ask AI # Create context context = OpenAIRealtimeLLMContext( messages = [], tools = [], system = "You are a helpful assistant" ) # Create aggregators aggregators = service.create_context_aggregator(context) ​ Foundational Examples OpenAI Realtime Beta Example Basic implementation showing core realtime features including audio streaming, turn detection, and function calling. ​ Notes Supports real-time speech-to-speech conversation Handles interruptions and turn-taking Manages WebSocket connection lifecycle Provides function calling capabilities Supports conversation context management Includes comprehensive error handling Manages audio streaming and processing Handles both text and audio modalities Gemini Multimodal Live fal On this page Installation Configuration Constructor Parameters Session Properties Input Frames Audio Input Control Input Context Input Output Frames Audio Output Control Output Text Output Events Methods Usage Example Function Calling Frame Flow Metrics Support Advanced Features Turn Detection Context Management Foundational Examples Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/s2s_openai_ac565c7e.txt b/s2s_openai_ac565c7e.txt
new file mode 100644
index 0000000000000000000000000000000000000000..b7f9a9398b21a1fe0ec97267ed5ea85c48f2d016
--- /dev/null
+++ b/s2s_openai_ac565c7e.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/s2s/openai#notes
+Title: OpenAI Realtime Beta - Pipecat
+==================================================
+
+OpenAI Realtime Beta - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Speech OpenAI Realtime Beta Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech AWS Nova Sonic Gemini Multimodal Live OpenAI Realtime Beta Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline OpenAIRealtimeBetaLLMService provides real-time, multimodal conversation capabilities using OpenAI’s Realtime Beta API. It supports speech-to-speech interactions with integrated LLM processing, function calling, and advanced conversation management. Real-time Interaction Stream audio in real-time with minimal latency response times Speech Processing Built-in speech-to-text and text-to-speech capabilities with voice options Advanced Turn Detection Multiple voice activity detection options including semantic turn detection Powerful Function Calling Seamless support for calling external functions and APIs ​ Installation To use OpenAIRealtimeBetaLLMService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[openai]" You’ll also need to set up your OpenAI API key as an environment variable: OPENAI_API_KEY . ​ Configuration ​ Constructor Parameters ​ api_key str required Your OpenAI API key ​ model str default: "gpt-4o-realtime-preview-2025-06-03" The speech-to-speech model used for processing ​ base_url str default: "wss://api.openai.com/v1/realtime" WebSocket endpoint URL ​ session_properties SessionProperties Configuration for the realtime session ​ start_audio_paused bool default: "False" Whether to start with audio input paused ​ send_transcription_frames bool default: "True" Whether to emit transcription frames ​ Session Properties The SessionProperties object configures the behavior of the realtime session: ​ modalities List[Literal['text', 'audio']] The modalities to enable (default includes both text and audio) ​ instructions str System instructions that guide the model’s behavior Copy Ask AI service = OpenAIRealtimeBetaLLMService( api_key = os.getenv( "OPENAI_API_KEY" ), session_properties = SessionProperties( instructions = "You are a helpful assistant. Be concise and friendly." ) ) ​ voice str Voice ID for text-to-speech (options: alloy, echo, fable, onyx, nova, shimmer) ​ input_audio_format Literal['pcm16', 'g711_ulaw', 'g711_alaw'] Format of the input audio ​ output_audio_format Literal['pcm16', 'g711_ulaw', 'g711_alaw'] Format of the output audio ​ input_audio_transcription InputAudioTranscription Configuration for audio transcription Copy Ask AI from pipecat.services.openai_realtime_beta.events import InputAudioTranscription service = OpenAIRealtimeBetaLLMService( api_key = os.getenv( "OPENAI_API_KEY" ), session_properties = SessionProperties( input_audio_transcription = InputAudioTranscription( model = "gpt-4o-transcribe" , language = "en" , prompt = "This is a technical conversation about programming" ) ) ) ​ input_audio_noise_reduction InputAudioNoiseReduction Configuration for audio noise reduction ​ turn_detection Union[TurnDetection, SemanticTurnDetection, bool] Configuration for turn detection (set to False to disable) ​ tools List[Dict] List of function definitions for tool/function calling ​ tool_choice Literal['auto', 'none', 'required'] Controls when the model calls functions ​ temperature float Controls randomness in responses (0.0 to 2.0) ​ max_response_output_tokens Union[int, Literal['inf']] Maximum number of tokens to generate ​ Input Frames ​ Audio Input ​ InputAudioRawFrame Frame Raw audio data for speech input ​ Control Input ​ StartInterruptionFrame Frame Signals start of user interruption ​ UserStartedSpeakingFrame Frame Signals user started speaking ​ UserStoppedSpeakingFrame Frame Signals user stopped speaking ​ Context Input ​ OpenAILLMContextFrame Frame Contains conversation context ​ LLMMessagesAppendFrame Frame Appends messages to conversation ​ Output Frames ​ Audio Output ​ TTSAudioRawFrame Frame Generated speech audio ​ Control Output ​ TTSStartedFrame Frame Signals start of speech synthesis ​ TTSStoppedFrame Frame Signals end of speech synthesis ​ Text Output ​ TextFrame Frame Generated text responses ​ TranscriptionFrame Frame Speech transcriptions ​ Events ​ on_conversation_item_created event Emitted when a conversation item on the server is created. Handler receives: item_id: str item: ConversationItem ​ on_conversation_item_updated event Emitted when a conversation item on the server is updated. Handler receives: item_id: str item: Optional[ConversationItem] (may not exist for some updates) ​ Methods ​ retrieve_conversation_item method Retrieves a conversation item’s details from the server. Copy Ask AI async def retrieve_conversation_item ( self , item_id : str ) -> ConversationItem ​ Usage Example Copy Ask AI from pipecat.services.openai_realtime_beta import OpenAIRealtimeBetaLLMService from pipecat.services.openai_realtime_beta.events import SessionProperties, TurnDetection # Configure service service = OpenAIRealtimeBetaLLMService( api_key = "your-api-key" , session_properties = SessionProperties( modalities = [ "audio" , "text" ], voice = "alloy" , turn_detection = TurnDetection( threshold = 0.5 , silence_duration_ms = 800 ), temperature = 0.7 ) ) # Use in pipeline pipeline = Pipeline([ audio_input, # Produces InputAudioRawFrame service, # Processes speech/generates responses audio_output # Handles TTSAudioRawFrame ]) ​ Function Calling The service supports function calling with automatic response handling: Copy Ask AI from pipecat.adapters.schemas.function_schema import FunctionSchema from pipecat.adapters.schemas.tools_schema import ToolsSchema from pipecat.services.openai_realtime_beta import SessionProperties # Define weather function using standardized schema weather_function = FunctionSchema( name = "get_weather" , description = "Get weather information" , properties = { "location" : { "type" : "string" } }, required = [ "location" ] ) # Create tools schema tools = ToolsSchema( standard_tools = [weather_function]) # Configure service with tools llm = OpenAIRealtimeBetaLLMService( api_key = "your-api-key" , session_properties = SessionProperties( tools = tools, tool_choice = "auto" ) ) llm.register_function( "get_weather" , fetch_weather_from_api) See the Function Calling guide for: Detailed implementation instructions Provider-specific function definitions Handler registration examples Control over function call behavior Complete usage examples ​ Frame Flow ​ Metrics Support The service collects comprehensive metrics: Token usage (prompt and completion) Processing duration Time to First Byte (TTFB) Audio processing metrics Function call metrics ​ Advanced Features ​ Turn Detection Copy Ask AI # Server-side basic VAD turn_detection = TurnDetection( type = "server_vad" , threshold = 0.5 , prefix_padding_ms = 300 , silence_duration_ms = 800 ) # Server-side semantic VAD turn_detection = SemanticTurnDetection( type = "semantic_vad" , eagerness = "auto" , # default. could also be "low" | "medium" | "high" create_response = True # default interrupt_response = True # default ) # Disable turn detection turn_detection = False ​ Context Management Copy Ask AI # Create context context = OpenAIRealtimeLLMContext( messages = [], tools = [], system = "You are a helpful assistant" ) # Create aggregators aggregators = service.create_context_aggregator(context) ​ Foundational Examples OpenAI Realtime Beta Example Basic implementation showing core realtime features including audio streaming, turn detection, and function calling. ​ Notes Supports real-time speech-to-speech conversation Handles interruptions and turn-taking Manages WebSocket connection lifecycle Provides function calling capabilities Supports conversation context management Includes comprehensive error handling Manages audio streaming and processing Handles both text and audio modalities Gemini Multimodal Live fal On this page Installation Configuration Constructor Parameters Session Properties Input Frames Audio Input Control Input Context Input Output Frames Audio Output Control Output Text Output Events Methods Usage Example Function Calling Frame Flow Metrics Support Advanced Features Turn Detection Context Management Foundational Examples Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/s2s_openai_ba3940f6.txt b/s2s_openai_ba3940f6.txt
new file mode 100644
index 0000000000000000000000000000000000000000..2e3f68eef1c07e5feb460cff75f415d4b919b360
--- /dev/null
+++ b/s2s_openai_ba3940f6.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/s2s/openai#param-tts-stopped-frame
+Title: OpenAI Realtime Beta - Pipecat
+==================================================
+
+OpenAI Realtime Beta - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Speech OpenAI Realtime Beta Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech AWS Nova Sonic Gemini Multimodal Live OpenAI Realtime Beta Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline OpenAIRealtimeBetaLLMService provides real-time, multimodal conversation capabilities using OpenAI’s Realtime Beta API. It supports speech-to-speech interactions with integrated LLM processing, function calling, and advanced conversation management. Real-time Interaction Stream audio in real-time with minimal latency response times Speech Processing Built-in speech-to-text and text-to-speech capabilities with voice options Advanced Turn Detection Multiple voice activity detection options including semantic turn detection Powerful Function Calling Seamless support for calling external functions and APIs ​ Installation To use OpenAIRealtimeBetaLLMService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[openai]" You’ll also need to set up your OpenAI API key as an environment variable: OPENAI_API_KEY . ​ Configuration ​ Constructor Parameters ​ api_key str required Your OpenAI API key ​ model str default: "gpt-4o-realtime-preview-2025-06-03" The speech-to-speech model used for processing ​ base_url str default: "wss://api.openai.com/v1/realtime" WebSocket endpoint URL ​ session_properties SessionProperties Configuration for the realtime session ​ start_audio_paused bool default: "False" Whether to start with audio input paused ​ send_transcription_frames bool default: "True" Whether to emit transcription frames ​ Session Properties The SessionProperties object configures the behavior of the realtime session: ​ modalities List[Literal['text', 'audio']] The modalities to enable (default includes both text and audio) ​ instructions str System instructions that guide the model’s behavior Copy Ask AI service = OpenAIRealtimeBetaLLMService( api_key = os.getenv( "OPENAI_API_KEY" ), session_properties = SessionProperties( instructions = "You are a helpful assistant. Be concise and friendly." ) ) ​ voice str Voice ID for text-to-speech (options: alloy, echo, fable, onyx, nova, shimmer) ​ input_audio_format Literal['pcm16', 'g711_ulaw', 'g711_alaw'] Format of the input audio ​ output_audio_format Literal['pcm16', 'g711_ulaw', 'g711_alaw'] Format of the output audio ​ input_audio_transcription InputAudioTranscription Configuration for audio transcription Copy Ask AI from pipecat.services.openai_realtime_beta.events import InputAudioTranscription service = OpenAIRealtimeBetaLLMService( api_key = os.getenv( "OPENAI_API_KEY" ), session_properties = SessionProperties( input_audio_transcription = InputAudioTranscription( model = "gpt-4o-transcribe" , language = "en" , prompt = "This is a technical conversation about programming" ) ) ) ​ input_audio_noise_reduction InputAudioNoiseReduction Configuration for audio noise reduction ​ turn_detection Union[TurnDetection, SemanticTurnDetection, bool] Configuration for turn detection (set to False to disable) ​ tools List[Dict] List of function definitions for tool/function calling ​ tool_choice Literal['auto', 'none', 'required'] Controls when the model calls functions ​ temperature float Controls randomness in responses (0.0 to 2.0) ​ max_response_output_tokens Union[int, Literal['inf']] Maximum number of tokens to generate ​ Input Frames ​ Audio Input ​ InputAudioRawFrame Frame Raw audio data for speech input ​ Control Input ​ StartInterruptionFrame Frame Signals start of user interruption ​ UserStartedSpeakingFrame Frame Signals user started speaking ​ UserStoppedSpeakingFrame Frame Signals user stopped speaking ​ Context Input ​ OpenAILLMContextFrame Frame Contains conversation context ​ LLMMessagesAppendFrame Frame Appends messages to conversation ​ Output Frames ​ Audio Output ​ TTSAudioRawFrame Frame Generated speech audio ​ Control Output ​ TTSStartedFrame Frame Signals start of speech synthesis ​ TTSStoppedFrame Frame Signals end of speech synthesis ​ Text Output ​ TextFrame Frame Generated text responses ​ TranscriptionFrame Frame Speech transcriptions ​ Events ​ on_conversation_item_created event Emitted when a conversation item on the server is created. Handler receives: item_id: str item: ConversationItem ​ on_conversation_item_updated event Emitted when a conversation item on the server is updated. Handler receives: item_id: str item: Optional[ConversationItem] (may not exist for some updates) ​ Methods ​ retrieve_conversation_item method Retrieves a conversation item’s details from the server. Copy Ask AI async def retrieve_conversation_item ( self , item_id : str ) -> ConversationItem ​ Usage Example Copy Ask AI from pipecat.services.openai_realtime_beta import OpenAIRealtimeBetaLLMService from pipecat.services.openai_realtime_beta.events import SessionProperties, TurnDetection # Configure service service = OpenAIRealtimeBetaLLMService( api_key = "your-api-key" , session_properties = SessionProperties( modalities = [ "audio" , "text" ], voice = "alloy" , turn_detection = TurnDetection( threshold = 0.5 , silence_duration_ms = 800 ), temperature = 0.7 ) ) # Use in pipeline pipeline = Pipeline([ audio_input, # Produces InputAudioRawFrame service, # Processes speech/generates responses audio_output # Handles TTSAudioRawFrame ]) ​ Function Calling The service supports function calling with automatic response handling: Copy Ask AI from pipecat.adapters.schemas.function_schema import FunctionSchema from pipecat.adapters.schemas.tools_schema import ToolsSchema from pipecat.services.openai_realtime_beta import SessionProperties # Define weather function using standardized schema weather_function = FunctionSchema( name = "get_weather" , description = "Get weather information" , properties = { "location" : { "type" : "string" } }, required = [ "location" ] ) # Create tools schema tools = ToolsSchema( standard_tools = [weather_function]) # Configure service with tools llm = OpenAIRealtimeBetaLLMService( api_key = "your-api-key" , session_properties = SessionProperties( tools = tools, tool_choice = "auto" ) ) llm.register_function( "get_weather" , fetch_weather_from_api) See the Function Calling guide for: Detailed implementation instructions Provider-specific function definitions Handler registration examples Control over function call behavior Complete usage examples ​ Frame Flow ​ Metrics Support The service collects comprehensive metrics: Token usage (prompt and completion) Processing duration Time to First Byte (TTFB) Audio processing metrics Function call metrics ​ Advanced Features ​ Turn Detection Copy Ask AI # Server-side basic VAD turn_detection = TurnDetection( type = "server_vad" , threshold = 0.5 , prefix_padding_ms = 300 , silence_duration_ms = 800 ) # Server-side semantic VAD turn_detection = SemanticTurnDetection( type = "semantic_vad" , eagerness = "auto" , # default. could also be "low" | "medium" | "high" create_response = True # default interrupt_response = True # default ) # Disable turn detection turn_detection = False ​ Context Management Copy Ask AI # Create context context = OpenAIRealtimeLLMContext( messages = [], tools = [], system = "You are a helpful assistant" ) # Create aggregators aggregators = service.create_context_aggregator(context) ​ Foundational Examples OpenAI Realtime Beta Example Basic implementation showing core realtime features including audio streaming, turn detection, and function calling. ​ Notes Supports real-time speech-to-speech conversation Handles interruptions and turn-taking Manages WebSocket connection lifecycle Provides function calling capabilities Supports conversation context management Includes comprehensive error handling Manages audio streaming and processing Handles both text and audio modalities Gemini Multimodal Live fal On this page Installation Configuration Constructor Parameters Session Properties Input Frames Audio Input Control Input Context Input Output Frames Audio Output Control Output Text Output Events Methods Usage Example Function Calling Frame Flow Metrics Support Advanced Features Turn Detection Context Management Foundational Examples Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/s2s_openai_c4c2aeb9.txt b/s2s_openai_c4c2aeb9.txt
new file mode 100644
index 0000000000000000000000000000000000000000..84451ccbe411c89bd1d3dcf2528bebfc8039609e
--- /dev/null
+++ b/s2s_openai_c4c2aeb9.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/s2s/openai#function-calling
+Title: OpenAI Realtime Beta - Pipecat
+==================================================
+
+OpenAI Realtime Beta - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Speech OpenAI Realtime Beta Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech AWS Nova Sonic Gemini Multimodal Live OpenAI Realtime Beta Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline OpenAIRealtimeBetaLLMService provides real-time, multimodal conversation capabilities using OpenAI’s Realtime Beta API. It supports speech-to-speech interactions with integrated LLM processing, function calling, and advanced conversation management. Real-time Interaction Stream audio in real-time with minimal latency response times Speech Processing Built-in speech-to-text and text-to-speech capabilities with voice options Advanced Turn Detection Multiple voice activity detection options including semantic turn detection Powerful Function Calling Seamless support for calling external functions and APIs ​ Installation To use OpenAIRealtimeBetaLLMService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[openai]" You’ll also need to set up your OpenAI API key as an environment variable: OPENAI_API_KEY . ​ Configuration ​ Constructor Parameters ​ api_key str required Your OpenAI API key ​ model str default: "gpt-4o-realtime-preview-2025-06-03" The speech-to-speech model used for processing ​ base_url str default: "wss://api.openai.com/v1/realtime" WebSocket endpoint URL ​ session_properties SessionProperties Configuration for the realtime session ​ start_audio_paused bool default: "False" Whether to start with audio input paused ​ send_transcription_frames bool default: "True" Whether to emit transcription frames ​ Session Properties The SessionProperties object configures the behavior of the realtime session: ​ modalities List[Literal['text', 'audio']] The modalities to enable (default includes both text and audio) ​ instructions str System instructions that guide the model’s behavior Copy Ask AI service = OpenAIRealtimeBetaLLMService( api_key = os.getenv( "OPENAI_API_KEY" ), session_properties = SessionProperties( instructions = "You are a helpful assistant. Be concise and friendly." ) ) ​ voice str Voice ID for text-to-speech (options: alloy, echo, fable, onyx, nova, shimmer) ​ input_audio_format Literal['pcm16', 'g711_ulaw', 'g711_alaw'] Format of the input audio ​ output_audio_format Literal['pcm16', 'g711_ulaw', 'g711_alaw'] Format of the output audio ​ input_audio_transcription InputAudioTranscription Configuration for audio transcription Copy Ask AI from pipecat.services.openai_realtime_beta.events import InputAudioTranscription service = OpenAIRealtimeBetaLLMService( api_key = os.getenv( "OPENAI_API_KEY" ), session_properties = SessionProperties( input_audio_transcription = InputAudioTranscription( model = "gpt-4o-transcribe" , language = "en" , prompt = "This is a technical conversation about programming" ) ) ) ​ input_audio_noise_reduction InputAudioNoiseReduction Configuration for audio noise reduction ​ turn_detection Union[TurnDetection, SemanticTurnDetection, bool] Configuration for turn detection (set to False to disable) ​ tools List[Dict] List of function definitions for tool/function calling ​ tool_choice Literal['auto', 'none', 'required'] Controls when the model calls functions ​ temperature float Controls randomness in responses (0.0 to 2.0) ​ max_response_output_tokens Union[int, Literal['inf']] Maximum number of tokens to generate ​ Input Frames ​ Audio Input ​ InputAudioRawFrame Frame Raw audio data for speech input ​ Control Input ​ StartInterruptionFrame Frame Signals start of user interruption ​ UserStartedSpeakingFrame Frame Signals user started speaking ​ UserStoppedSpeakingFrame Frame Signals user stopped speaking ​ Context Input ​ OpenAILLMContextFrame Frame Contains conversation context ​ LLMMessagesAppendFrame Frame Appends messages to conversation ​ Output Frames ​ Audio Output ​ TTSAudioRawFrame Frame Generated speech audio ​ Control Output ​ TTSStartedFrame Frame Signals start of speech synthesis ​ TTSStoppedFrame Frame Signals end of speech synthesis ​ Text Output ​ TextFrame Frame Generated text responses ​ TranscriptionFrame Frame Speech transcriptions ​ Events ​ on_conversation_item_created event Emitted when a conversation item on the server is created. Handler receives: item_id: str item: ConversationItem ​ on_conversation_item_updated event Emitted when a conversation item on the server is updated. Handler receives: item_id: str item: Optional[ConversationItem] (may not exist for some updates) ​ Methods ​ retrieve_conversation_item method Retrieves a conversation item’s details from the server. Copy Ask AI async def retrieve_conversation_item ( self , item_id : str ) -> ConversationItem ​ Usage Example Copy Ask AI from pipecat.services.openai_realtime_beta import OpenAIRealtimeBetaLLMService from pipecat.services.openai_realtime_beta.events import SessionProperties, TurnDetection # Configure service service = OpenAIRealtimeBetaLLMService( api_key = "your-api-key" , session_properties = SessionProperties( modalities = [ "audio" , "text" ], voice = "alloy" , turn_detection = TurnDetection( threshold = 0.5 , silence_duration_ms = 800 ), temperature = 0.7 ) ) # Use in pipeline pipeline = Pipeline([ audio_input, # Produces InputAudioRawFrame service, # Processes speech/generates responses audio_output # Handles TTSAudioRawFrame ]) ​ Function Calling The service supports function calling with automatic response handling: Copy Ask AI from pipecat.adapters.schemas.function_schema import FunctionSchema from pipecat.adapters.schemas.tools_schema import ToolsSchema from pipecat.services.openai_realtime_beta import SessionProperties # Define weather function using standardized schema weather_function = FunctionSchema( name = "get_weather" , description = "Get weather information" , properties = { "location" : { "type" : "string" } }, required = [ "location" ] ) # Create tools schema tools = ToolsSchema( standard_tools = [weather_function]) # Configure service with tools llm = OpenAIRealtimeBetaLLMService( api_key = "your-api-key" , session_properties = SessionProperties( tools = tools, tool_choice = "auto" ) ) llm.register_function( "get_weather" , fetch_weather_from_api) See the Function Calling guide for: Detailed implementation instructions Provider-specific function definitions Handler registration examples Control over function call behavior Complete usage examples ​ Frame Flow ​ Metrics Support The service collects comprehensive metrics: Token usage (prompt and completion) Processing duration Time to First Byte (TTFB) Audio processing metrics Function call metrics ​ Advanced Features ​ Turn Detection Copy Ask AI # Server-side basic VAD turn_detection = TurnDetection( type = "server_vad" , threshold = 0.5 , prefix_padding_ms = 300 , silence_duration_ms = 800 ) # Server-side semantic VAD turn_detection = SemanticTurnDetection( type = "semantic_vad" , eagerness = "auto" , # default. could also be "low" | "medium" | "high" create_response = True # default interrupt_response = True # default ) # Disable turn detection turn_detection = False ​ Context Management Copy Ask AI # Create context context = OpenAIRealtimeLLMContext( messages = [], tools = [], system = "You are a helpful assistant" ) # Create aggregators aggregators = service.create_context_aggregator(context) ​ Foundational Examples OpenAI Realtime Beta Example Basic implementation showing core realtime features including audio streaming, turn detection, and function calling. ​ Notes Supports real-time speech-to-speech conversation Handles interruptions and turn-taking Manages WebSocket connection lifecycle Provides function calling capabilities Supports conversation context management Includes comprehensive error handling Manages audio streaming and processing Handles both text and audio modalities Gemini Multimodal Live fal On this page Installation Configuration Constructor Parameters Session Properties Input Frames Audio Input Control Input Context Input Output Frames Audio Output Control Output Text Output Events Methods Usage Example Function Calling Frame Flow Metrics Support Advanced Features Turn Detection Context Management Foundational Examples Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/s2s_openai_ca9d8bee.txt b/s2s_openai_ca9d8bee.txt
new file mode 100644
index 0000000000000000000000000000000000000000..67d0b05b48e0ce999111434438610c820dfefdb3
--- /dev/null
+++ b/s2s_openai_ca9d8bee.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/s2s/openai#audio-input
+Title: OpenAI Realtime Beta - Pipecat
+==================================================
+
+OpenAI Realtime Beta - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Speech OpenAI Realtime Beta Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech AWS Nova Sonic Gemini Multimodal Live OpenAI Realtime Beta Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline OpenAIRealtimeBetaLLMService provides real-time, multimodal conversation capabilities using OpenAI’s Realtime Beta API. It supports speech-to-speech interactions with integrated LLM processing, function calling, and advanced conversation management. Real-time Interaction Stream audio in real-time with minimal latency response times Speech Processing Built-in speech-to-text and text-to-speech capabilities with voice options Advanced Turn Detection Multiple voice activity detection options including semantic turn detection Powerful Function Calling Seamless support for calling external functions and APIs ​ Installation To use OpenAIRealtimeBetaLLMService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[openai]" You’ll also need to set up your OpenAI API key as an environment variable: OPENAI_API_KEY . ​ Configuration ​ Constructor Parameters ​ api_key str required Your OpenAI API key ​ model str default: "gpt-4o-realtime-preview-2025-06-03" The speech-to-speech model used for processing ​ base_url str default: "wss://api.openai.com/v1/realtime" WebSocket endpoint URL ​ session_properties SessionProperties Configuration for the realtime session ​ start_audio_paused bool default: "False" Whether to start with audio input paused ​ send_transcription_frames bool default: "True" Whether to emit transcription frames ​ Session Properties The SessionProperties object configures the behavior of the realtime session: ​ modalities List[Literal['text', 'audio']] The modalities to enable (default includes both text and audio) ​ instructions str System instructions that guide the model’s behavior Copy Ask AI service = OpenAIRealtimeBetaLLMService( api_key = os.getenv( "OPENAI_API_KEY" ), session_properties = SessionProperties( instructions = "You are a helpful assistant. Be concise and friendly." ) ) ​ voice str Voice ID for text-to-speech (options: alloy, echo, fable, onyx, nova, shimmer) ​ input_audio_format Literal['pcm16', 'g711_ulaw', 'g711_alaw'] Format of the input audio ​ output_audio_format Literal['pcm16', 'g711_ulaw', 'g711_alaw'] Format of the output audio ​ input_audio_transcription InputAudioTranscription Configuration for audio transcription Copy Ask AI from pipecat.services.openai_realtime_beta.events import InputAudioTranscription service = OpenAIRealtimeBetaLLMService( api_key = os.getenv( "OPENAI_API_KEY" ), session_properties = SessionProperties( input_audio_transcription = InputAudioTranscription( model = "gpt-4o-transcribe" , language = "en" , prompt = "This is a technical conversation about programming" ) ) ) ​ input_audio_noise_reduction InputAudioNoiseReduction Configuration for audio noise reduction ​ turn_detection Union[TurnDetection, SemanticTurnDetection, bool] Configuration for turn detection (set to False to disable) ​ tools List[Dict] List of function definitions for tool/function calling ​ tool_choice Literal['auto', 'none', 'required'] Controls when the model calls functions ​ temperature float Controls randomness in responses (0.0 to 2.0) ​ max_response_output_tokens Union[int, Literal['inf']] Maximum number of tokens to generate ​ Input Frames ​ Audio Input ​ InputAudioRawFrame Frame Raw audio data for speech input ​ Control Input ​ StartInterruptionFrame Frame Signals start of user interruption ​ UserStartedSpeakingFrame Frame Signals user started speaking ​ UserStoppedSpeakingFrame Frame Signals user stopped speaking ​ Context Input ​ OpenAILLMContextFrame Frame Contains conversation context ​ LLMMessagesAppendFrame Frame Appends messages to conversation ​ Output Frames ​ Audio Output ​ TTSAudioRawFrame Frame Generated speech audio ​ Control Output ​ TTSStartedFrame Frame Signals start of speech synthesis ​ TTSStoppedFrame Frame Signals end of speech synthesis ​ Text Output ​ TextFrame Frame Generated text responses ​ TranscriptionFrame Frame Speech transcriptions ​ Events ​ on_conversation_item_created event Emitted when a conversation item on the server is created. Handler receives: item_id: str item: ConversationItem ​ on_conversation_item_updated event Emitted when a conversation item on the server is updated. Handler receives: item_id: str item: Optional[ConversationItem] (may not exist for some updates) ​ Methods ​ retrieve_conversation_item method Retrieves a conversation item’s details from the server. Copy Ask AI async def retrieve_conversation_item ( self , item_id : str ) -> ConversationItem ​ Usage Example Copy Ask AI from pipecat.services.openai_realtime_beta import OpenAIRealtimeBetaLLMService from pipecat.services.openai_realtime_beta.events import SessionProperties, TurnDetection # Configure service service = OpenAIRealtimeBetaLLMService( api_key = "your-api-key" , session_properties = SessionProperties( modalities = [ "audio" , "text" ], voice = "alloy" , turn_detection = TurnDetection( threshold = 0.5 , silence_duration_ms = 800 ), temperature = 0.7 ) ) # Use in pipeline pipeline = Pipeline([ audio_input, # Produces InputAudioRawFrame service, # Processes speech/generates responses audio_output # Handles TTSAudioRawFrame ]) ​ Function Calling The service supports function calling with automatic response handling: Copy Ask AI from pipecat.adapters.schemas.function_schema import FunctionSchema from pipecat.adapters.schemas.tools_schema import ToolsSchema from pipecat.services.openai_realtime_beta import SessionProperties # Define weather function using standardized schema weather_function = FunctionSchema( name = "get_weather" , description = "Get weather information" , properties = { "location" : { "type" : "string" } }, required = [ "location" ] ) # Create tools schema tools = ToolsSchema( standard_tools = [weather_function]) # Configure service with tools llm = OpenAIRealtimeBetaLLMService( api_key = "your-api-key" , session_properties = SessionProperties( tools = tools, tool_choice = "auto" ) ) llm.register_function( "get_weather" , fetch_weather_from_api) See the Function Calling guide for: Detailed implementation instructions Provider-specific function definitions Handler registration examples Control over function call behavior Complete usage examples ​ Frame Flow ​ Metrics Support The service collects comprehensive metrics: Token usage (prompt and completion) Processing duration Time to First Byte (TTFB) Audio processing metrics Function call metrics ​ Advanced Features ​ Turn Detection Copy Ask AI # Server-side basic VAD turn_detection = TurnDetection( type = "server_vad" , threshold = 0.5 , prefix_padding_ms = 300 , silence_duration_ms = 800 ) # Server-side semantic VAD turn_detection = SemanticTurnDetection( type = "semantic_vad" , eagerness = "auto" , # default. could also be "low" | "medium" | "high" create_response = True # default interrupt_response = True # default ) # Disable turn detection turn_detection = False ​ Context Management Copy Ask AI # Create context context = OpenAIRealtimeLLMContext( messages = [], tools = [], system = "You are a helpful assistant" ) # Create aggregators aggregators = service.create_context_aggregator(context) ​ Foundational Examples OpenAI Realtime Beta Example Basic implementation showing core realtime features including audio streaming, turn detection, and function calling. ​ Notes Supports real-time speech-to-speech conversation Handles interruptions and turn-taking Manages WebSocket connection lifecycle Provides function calling capabilities Supports conversation context management Includes comprehensive error handling Manages audio streaming and processing Handles both text and audio modalities Gemini Multimodal Live fal On this page Installation Configuration Constructor Parameters Session Properties Input Frames Audio Input Control Input Context Input Output Frames Audio Output Control Output Text Output Events Methods Usage Example Function Calling Frame Flow Metrics Support Advanced Features Turn Detection Context Management Foundational Examples Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/s2s_openai_d85d0f2c.txt b/s2s_openai_d85d0f2c.txt
new file mode 100644
index 0000000000000000000000000000000000000000..0dd4345ac95a81b4e87b29da23cd7c7314999c0c
--- /dev/null
+++ b/s2s_openai_d85d0f2c.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/s2s/openai#param-tools
+Title: OpenAI Realtime Beta - Pipecat
+==================================================
+
+OpenAI Realtime Beta - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Speech OpenAI Realtime Beta Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech AWS Nova Sonic Gemini Multimodal Live OpenAI Realtime Beta Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline OpenAIRealtimeBetaLLMService provides real-time, multimodal conversation capabilities using OpenAI’s Realtime Beta API. It supports speech-to-speech interactions with integrated LLM processing, function calling, and advanced conversation management. Real-time Interaction Stream audio in real-time with minimal latency response times Speech Processing Built-in speech-to-text and text-to-speech capabilities with voice options Advanced Turn Detection Multiple voice activity detection options including semantic turn detection Powerful Function Calling Seamless support for calling external functions and APIs ​ Installation To use OpenAIRealtimeBetaLLMService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[openai]" You’ll also need to set up your OpenAI API key as an environment variable: OPENAI_API_KEY . ​ Configuration ​ Constructor Parameters ​ api_key str required Your OpenAI API key ​ model str default: "gpt-4o-realtime-preview-2025-06-03" The speech-to-speech model used for processing ​ base_url str default: "wss://api.openai.com/v1/realtime" WebSocket endpoint URL ​ session_properties SessionProperties Configuration for the realtime session ​ start_audio_paused bool default: "False" Whether to start with audio input paused ​ send_transcription_frames bool default: "True" Whether to emit transcription frames ​ Session Properties The SessionProperties object configures the behavior of the realtime session: ​ modalities List[Literal['text', 'audio']] The modalities to enable (default includes both text and audio) ​ instructions str System instructions that guide the model’s behavior Copy Ask AI service = OpenAIRealtimeBetaLLMService( api_key = os.getenv( "OPENAI_API_KEY" ), session_properties = SessionProperties( instructions = "You are a helpful assistant. Be concise and friendly." ) ) ​ voice str Voice ID for text-to-speech (options: alloy, echo, fable, onyx, nova, shimmer) ​ input_audio_format Literal['pcm16', 'g711_ulaw', 'g711_alaw'] Format of the input audio ​ output_audio_format Literal['pcm16', 'g711_ulaw', 'g711_alaw'] Format of the output audio ​ input_audio_transcription InputAudioTranscription Configuration for audio transcription Copy Ask AI from pipecat.services.openai_realtime_beta.events import InputAudioTranscription service = OpenAIRealtimeBetaLLMService( api_key = os.getenv( "OPENAI_API_KEY" ), session_properties = SessionProperties( input_audio_transcription = InputAudioTranscription( model = "gpt-4o-transcribe" , language = "en" , prompt = "This is a technical conversation about programming" ) ) ) ​ input_audio_noise_reduction InputAudioNoiseReduction Configuration for audio noise reduction ​ turn_detection Union[TurnDetection, SemanticTurnDetection, bool] Configuration for turn detection (set to False to disable) ​ tools List[Dict] List of function definitions for tool/function calling ​ tool_choice Literal['auto', 'none', 'required'] Controls when the model calls functions ​ temperature float Controls randomness in responses (0.0 to 2.0) ​ max_response_output_tokens Union[int, Literal['inf']] Maximum number of tokens to generate ​ Input Frames ​ Audio Input ​ InputAudioRawFrame Frame Raw audio data for speech input ​ Control Input ​ StartInterruptionFrame Frame Signals start of user interruption ​ UserStartedSpeakingFrame Frame Signals user started speaking ​ UserStoppedSpeakingFrame Frame Signals user stopped speaking ​ Context Input ​ OpenAILLMContextFrame Frame Contains conversation context ​ LLMMessagesAppendFrame Frame Appends messages to conversation ​ Output Frames ​ Audio Output ​ TTSAudioRawFrame Frame Generated speech audio ​ Control Output ​ TTSStartedFrame Frame Signals start of speech synthesis ​ TTSStoppedFrame Frame Signals end of speech synthesis ​ Text Output ​ TextFrame Frame Generated text responses ​ TranscriptionFrame Frame Speech transcriptions ​ Events ​ on_conversation_item_created event Emitted when a conversation item on the server is created. Handler receives: item_id: str item: ConversationItem ​ on_conversation_item_updated event Emitted when a conversation item on the server is updated. Handler receives: item_id: str item: Optional[ConversationItem] (may not exist for some updates) ​ Methods ​ retrieve_conversation_item method Retrieves a conversation item’s details from the server. Copy Ask AI async def retrieve_conversation_item ( self , item_id : str ) -> ConversationItem ​ Usage Example Copy Ask AI from pipecat.services.openai_realtime_beta import OpenAIRealtimeBetaLLMService from pipecat.services.openai_realtime_beta.events import SessionProperties, TurnDetection # Configure service service = OpenAIRealtimeBetaLLMService( api_key = "your-api-key" , session_properties = SessionProperties( modalities = [ "audio" , "text" ], voice = "alloy" , turn_detection = TurnDetection( threshold = 0.5 , silence_duration_ms = 800 ), temperature = 0.7 ) ) # Use in pipeline pipeline = Pipeline([ audio_input, # Produces InputAudioRawFrame service, # Processes speech/generates responses audio_output # Handles TTSAudioRawFrame ]) ​ Function Calling The service supports function calling with automatic response handling: Copy Ask AI from pipecat.adapters.schemas.function_schema import FunctionSchema from pipecat.adapters.schemas.tools_schema import ToolsSchema from pipecat.services.openai_realtime_beta import SessionProperties # Define weather function using standardized schema weather_function = FunctionSchema( name = "get_weather" , description = "Get weather information" , properties = { "location" : { "type" : "string" } }, required = [ "location" ] ) # Create tools schema tools = ToolsSchema( standard_tools = [weather_function]) # Configure service with tools llm = OpenAIRealtimeBetaLLMService( api_key = "your-api-key" , session_properties = SessionProperties( tools = tools, tool_choice = "auto" ) ) llm.register_function( "get_weather" , fetch_weather_from_api) See the Function Calling guide for: Detailed implementation instructions Provider-specific function definitions Handler registration examples Control over function call behavior Complete usage examples ​ Frame Flow ​ Metrics Support The service collects comprehensive metrics: Token usage (prompt and completion) Processing duration Time to First Byte (TTFB) Audio processing metrics Function call metrics ​ Advanced Features ​ Turn Detection Copy Ask AI # Server-side basic VAD turn_detection = TurnDetection( type = "server_vad" , threshold = 0.5 , prefix_padding_ms = 300 , silence_duration_ms = 800 ) # Server-side semantic VAD turn_detection = SemanticTurnDetection( type = "semantic_vad" , eagerness = "auto" , # default. could also be "low" | "medium" | "high" create_response = True # default interrupt_response = True # default ) # Disable turn detection turn_detection = False ​ Context Management Copy Ask AI # Create context context = OpenAIRealtimeLLMContext( messages = [], tools = [], system = "You are a helpful assistant" ) # Create aggregators aggregators = service.create_context_aggregator(context) ​ Foundational Examples OpenAI Realtime Beta Example Basic implementation showing core realtime features including audio streaming, turn detection, and function calling. ​ Notes Supports real-time speech-to-speech conversation Handles interruptions and turn-taking Manages WebSocket connection lifecycle Provides function calling capabilities Supports conversation context management Includes comprehensive error handling Manages audio streaming and processing Handles both text and audio modalities Gemini Multimodal Live fal On this page Installation Configuration Constructor Parameters Session Properties Input Frames Audio Input Control Input Context Input Output Frames Audio Output Control Output Text Output Events Methods Usage Example Function Calling Frame Flow Metrics Support Advanced Features Turn Detection Context Management Foundational Examples Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/s2s_openai_e03ddc19.txt b/s2s_openai_e03ddc19.txt
new file mode 100644
index 0000000000000000000000000000000000000000..d4c16f378382a583e840d76b34faeb01d69cf62f
--- /dev/null
+++ b/s2s_openai_e03ddc19.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/s2s/openai#param-temperature
+Title: OpenAI Realtime Beta - Pipecat
+==================================================
+
+OpenAI Realtime Beta - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Speech OpenAI Realtime Beta Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech AWS Nova Sonic Gemini Multimodal Live OpenAI Realtime Beta Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline OpenAIRealtimeBetaLLMService provides real-time, multimodal conversation capabilities using OpenAI’s Realtime Beta API. It supports speech-to-speech interactions with integrated LLM processing, function calling, and advanced conversation management. Real-time Interaction Stream audio in real-time with minimal latency response times Speech Processing Built-in speech-to-text and text-to-speech capabilities with voice options Advanced Turn Detection Multiple voice activity detection options including semantic turn detection Powerful Function Calling Seamless support for calling external functions and APIs ​ Installation To use OpenAIRealtimeBetaLLMService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[openai]" You’ll also need to set up your OpenAI API key as an environment variable: OPENAI_API_KEY . ​ Configuration ​ Constructor Parameters ​ api_key str required Your OpenAI API key ​ model str default: "gpt-4o-realtime-preview-2025-06-03" The speech-to-speech model used for processing ​ base_url str default: "wss://api.openai.com/v1/realtime" WebSocket endpoint URL ​ session_properties SessionProperties Configuration for the realtime session ​ start_audio_paused bool default: "False" Whether to start with audio input paused ​ send_transcription_frames bool default: "True" Whether to emit transcription frames ​ Session Properties The SessionProperties object configures the behavior of the realtime session: ​ modalities List[Literal['text', 'audio']] The modalities to enable (default includes both text and audio) ​ instructions str System instructions that guide the model’s behavior Copy Ask AI service = OpenAIRealtimeBetaLLMService( api_key = os.getenv( "OPENAI_API_KEY" ), session_properties = SessionProperties( instructions = "You are a helpful assistant. Be concise and friendly." ) ) ​ voice str Voice ID for text-to-speech (options: alloy, echo, fable, onyx, nova, shimmer) ​ input_audio_format Literal['pcm16', 'g711_ulaw', 'g711_alaw'] Format of the input audio ​ output_audio_format Literal['pcm16', 'g711_ulaw', 'g711_alaw'] Format of the output audio ​ input_audio_transcription InputAudioTranscription Configuration for audio transcription Copy Ask AI from pipecat.services.openai_realtime_beta.events import InputAudioTranscription service = OpenAIRealtimeBetaLLMService( api_key = os.getenv( "OPENAI_API_KEY" ), session_properties = SessionProperties( input_audio_transcription = InputAudioTranscription( model = "gpt-4o-transcribe" , language = "en" , prompt = "This is a technical conversation about programming" ) ) ) ​ input_audio_noise_reduction InputAudioNoiseReduction Configuration for audio noise reduction ​ turn_detection Union[TurnDetection, SemanticTurnDetection, bool] Configuration for turn detection (set to False to disable) ​ tools List[Dict] List of function definitions for tool/function calling ​ tool_choice Literal['auto', 'none', 'required'] Controls when the model calls functions ​ temperature float Controls randomness in responses (0.0 to 2.0) ​ max_response_output_tokens Union[int, Literal['inf']] Maximum number of tokens to generate ​ Input Frames ​ Audio Input ​ InputAudioRawFrame Frame Raw audio data for speech input ​ Control Input ​ StartInterruptionFrame Frame Signals start of user interruption ​ UserStartedSpeakingFrame Frame Signals user started speaking ​ UserStoppedSpeakingFrame Frame Signals user stopped speaking ​ Context Input ​ OpenAILLMContextFrame Frame Contains conversation context ​ LLMMessagesAppendFrame Frame Appends messages to conversation ​ Output Frames ​ Audio Output ​ TTSAudioRawFrame Frame Generated speech audio ​ Control Output ​ TTSStartedFrame Frame Signals start of speech synthesis ​ TTSStoppedFrame Frame Signals end of speech synthesis ​ Text Output ​ TextFrame Frame Generated text responses ​ TranscriptionFrame Frame Speech transcriptions ​ Events ​ on_conversation_item_created event Emitted when a conversation item on the server is created. Handler receives: item_id: str item: ConversationItem ​ on_conversation_item_updated event Emitted when a conversation item on the server is updated. Handler receives: item_id: str item: Optional[ConversationItem] (may not exist for some updates) ​ Methods ​ retrieve_conversation_item method Retrieves a conversation item’s details from the server. Copy Ask AI async def retrieve_conversation_item ( self , item_id : str ) -> ConversationItem ​ Usage Example Copy Ask AI from pipecat.services.openai_realtime_beta import OpenAIRealtimeBetaLLMService from pipecat.services.openai_realtime_beta.events import SessionProperties, TurnDetection # Configure service service = OpenAIRealtimeBetaLLMService( api_key = "your-api-key" , session_properties = SessionProperties( modalities = [ "audio" , "text" ], voice = "alloy" , turn_detection = TurnDetection( threshold = 0.5 , silence_duration_ms = 800 ), temperature = 0.7 ) ) # Use in pipeline pipeline = Pipeline([ audio_input, # Produces InputAudioRawFrame service, # Processes speech/generates responses audio_output # Handles TTSAudioRawFrame ]) ​ Function Calling The service supports function calling with automatic response handling: Copy Ask AI from pipecat.adapters.schemas.function_schema import FunctionSchema from pipecat.adapters.schemas.tools_schema import ToolsSchema from pipecat.services.openai_realtime_beta import SessionProperties # Define weather function using standardized schema weather_function = FunctionSchema( name = "get_weather" , description = "Get weather information" , properties = { "location" : { "type" : "string" } }, required = [ "location" ] ) # Create tools schema tools = ToolsSchema( standard_tools = [weather_function]) # Configure service with tools llm = OpenAIRealtimeBetaLLMService( api_key = "your-api-key" , session_properties = SessionProperties( tools = tools, tool_choice = "auto" ) ) llm.register_function( "get_weather" , fetch_weather_from_api) See the Function Calling guide for: Detailed implementation instructions Provider-specific function definitions Handler registration examples Control over function call behavior Complete usage examples ​ Frame Flow ​ Metrics Support The service collects comprehensive metrics: Token usage (prompt and completion) Processing duration Time to First Byte (TTFB) Audio processing metrics Function call metrics ​ Advanced Features ​ Turn Detection Copy Ask AI # Server-side basic VAD turn_detection = TurnDetection( type = "server_vad" , threshold = 0.5 , prefix_padding_ms = 300 , silence_duration_ms = 800 ) # Server-side semantic VAD turn_detection = SemanticTurnDetection( type = "semantic_vad" , eagerness = "auto" , # default. could also be "low" | "medium" | "high" create_response = True # default interrupt_response = True # default ) # Disable turn detection turn_detection = False ​ Context Management Copy Ask AI # Create context context = OpenAIRealtimeLLMContext( messages = [], tools = [], system = "You are a helpful assistant" ) # Create aggregators aggregators = service.create_context_aggregator(context) ​ Foundational Examples OpenAI Realtime Beta Example Basic implementation showing core realtime features including audio streaming, turn detection, and function calling. ​ Notes Supports real-time speech-to-speech conversation Handles interruptions and turn-taking Manages WebSocket connection lifecycle Provides function calling capabilities Supports conversation context management Includes comprehensive error handling Manages audio streaming and processing Handles both text and audio modalities Gemini Multimodal Live fal On this page Installation Configuration Constructor Parameters Session Properties Input Frames Audio Input Control Input Context Input Output Frames Audio Output Control Output Text Output Events Methods Usage Example Function Calling Frame Flow Metrics Support Advanced Features Turn Detection Context Management Foundational Examples Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/serializers_exotel_0476019d.txt b/serializers_exotel_0476019d.txt
new file mode 100644
index 0000000000000000000000000000000000000000..0ccd4f4f1e38180faa4f8a0c5d11392c386667d1
--- /dev/null
+++ b/serializers_exotel_0476019d.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/serializers/exotel#param-sample-rate
+Title: ExotelFrameSerializer - Pipecat
+==================================================
+
+ExotelFrameSerializer - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Serializers ExotelFrameSerializer Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Frame Serializer Overview ExotelFrameSerializer PlivoFrameSerializer TwilioFrameSerializer TelnyxFrameSerializer Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview ExotelFrameSerializer enables integration with Exotel’s WebSocket media streaming protocol, allowing your Pipecat application to handle phone calls via Exotel’s voice services. ​ Features Bidirectional audio conversion between Pipecat and Exotel DTMF (touch-tone) event handling ​ Installation The ExotelFrameSerializer does not require any additional dependencies beyond the core Pipecat library. ​ Configuration ​ Constructor Parameters ​ stream_id str required The Stream ID for Exotel ​ call_sid Optional[str] default: "None" The associated Exotel Call SID. ​ params InputParams default: "InputParams()" Configuration parameters ​ InputParams Configuration ​ exotel_sample_rate int default: "8000" Sample rate used by Exotel (typically 8kHz) ​ sample_rate int | None default: "None" Optional override for pipeline input sample rate ​ Basic Usage Copy Ask AI from pipecat.serializers.exotel import ExotelFrameSerializer from pipecat.transports.network.fastapi_websocket import ( FastAPIWebsocketTransport, FastAPIWebsocketParams ) # Extract required values from Exotel WebSocket connection stream_id = call_data[ "stream_id" ] call_sid = call_data[ "start" ][ "call_sid" ] # Create serializer serializer = ExotelFrameSerializer( stream_id = stream_id, call_sid, ) # Use with FastAPIWebsocketTransport transport = FastAPIWebsocketTransport( websocket = websocket, params = FastAPIWebsocketParams( audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer(), serializer = serializer, ) ) ​ Server Code Example Here’s a complete example of handling a Exotel WebSocket connection: Copy Ask AI from fastapi import FastAPI, WebSocket from pipecat.serializers.exotel import ExotelFrameSerializer import json import os app = FastAPI() @app.websocket ( "/ws" ) async def websocket_endpoint ( websocket : WebSocket): await websocket.accept() # Read initial messages from Exotel start_data = websocket.iter_text() await start_data. __anext__ () # Skip first message # Parse the second message to get call details call_data = json.loads( await start_data. __anext__ ()) # Extract Exotel-specific IDs and encoding stream_id = call_data[ "stream_id" ] call_sid = call_data[ "start" ][ "call_sid" ] # Create serializer with API key for auto hang-up serializer = ExotelFrameSerializer( stream_id = stream_id, call_sid = call_sid, ) # Continue with transport and pipeline setup... Frame Serializer Overview PlivoFrameSerializer On this page Overview Features Installation Configuration Constructor Parameters InputParams Configuration Basic Usage Server Code Example Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/serializers_exotel_09950dc8.txt b/serializers_exotel_09950dc8.txt
new file mode 100644
index 0000000000000000000000000000000000000000..1506de1b29953a74bdce6156392dd088048a281f
--- /dev/null
+++ b/serializers_exotel_09950dc8.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/serializers/exotel#param-call-sid
+Title: ExotelFrameSerializer - Pipecat
+==================================================
+
+ExotelFrameSerializer - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Serializers ExotelFrameSerializer Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Frame Serializer Overview ExotelFrameSerializer PlivoFrameSerializer TwilioFrameSerializer TelnyxFrameSerializer Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview ExotelFrameSerializer enables integration with Exotel’s WebSocket media streaming protocol, allowing your Pipecat application to handle phone calls via Exotel’s voice services. ​ Features Bidirectional audio conversion between Pipecat and Exotel DTMF (touch-tone) event handling ​ Installation The ExotelFrameSerializer does not require any additional dependencies beyond the core Pipecat library. ​ Configuration ​ Constructor Parameters ​ stream_id str required The Stream ID for Exotel ​ call_sid Optional[str] default: "None" The associated Exotel Call SID. ​ params InputParams default: "InputParams()" Configuration parameters ​ InputParams Configuration ​ exotel_sample_rate int default: "8000" Sample rate used by Exotel (typically 8kHz) ​ sample_rate int | None default: "None" Optional override for pipeline input sample rate ​ Basic Usage Copy Ask AI from pipecat.serializers.exotel import ExotelFrameSerializer from pipecat.transports.network.fastapi_websocket import ( FastAPIWebsocketTransport, FastAPIWebsocketParams ) # Extract required values from Exotel WebSocket connection stream_id = call_data[ "stream_id" ] call_sid = call_data[ "start" ][ "call_sid" ] # Create serializer serializer = ExotelFrameSerializer( stream_id = stream_id, call_sid, ) # Use with FastAPIWebsocketTransport transport = FastAPIWebsocketTransport( websocket = websocket, params = FastAPIWebsocketParams( audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer(), serializer = serializer, ) ) ​ Server Code Example Here’s a complete example of handling a Exotel WebSocket connection: Copy Ask AI from fastapi import FastAPI, WebSocket from pipecat.serializers.exotel import ExotelFrameSerializer import json import os app = FastAPI() @app.websocket ( "/ws" ) async def websocket_endpoint ( websocket : WebSocket): await websocket.accept() # Read initial messages from Exotel start_data = websocket.iter_text() await start_data. __anext__ () # Skip first message # Parse the second message to get call details call_data = json.loads( await start_data. __anext__ ()) # Extract Exotel-specific IDs and encoding stream_id = call_data[ "stream_id" ] call_sid = call_data[ "start" ][ "call_sid" ] # Create serializer with API key for auto hang-up serializer = ExotelFrameSerializer( stream_id = stream_id, call_sid = call_sid, ) # Continue with transport and pipeline setup... Frame Serializer Overview PlivoFrameSerializer On this page Overview Features Installation Configuration Constructor Parameters InputParams Configuration Basic Usage Server Code Example Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/serializers_exotel_0fd9f81f.txt b/serializers_exotel_0fd9f81f.txt
new file mode 100644
index 0000000000000000000000000000000000000000..461a319fc05bae2ef6e0153b116684a6eeac8265
--- /dev/null
+++ b/serializers_exotel_0fd9f81f.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/serializers/exotel
+Title: ExotelFrameSerializer - Pipecat
+==================================================
+
+ExotelFrameSerializer - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Serializers ExotelFrameSerializer Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Frame Serializer Overview ExotelFrameSerializer PlivoFrameSerializer TwilioFrameSerializer TelnyxFrameSerializer Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview ExotelFrameSerializer enables integration with Exotel’s WebSocket media streaming protocol, allowing your Pipecat application to handle phone calls via Exotel’s voice services. ​ Features Bidirectional audio conversion between Pipecat and Exotel DTMF (touch-tone) event handling ​ Installation The ExotelFrameSerializer does not require any additional dependencies beyond the core Pipecat library. ​ Configuration ​ Constructor Parameters ​ stream_id str required The Stream ID for Exotel ​ call_sid Optional[str] default: "None" The associated Exotel Call SID. ​ params InputParams default: "InputParams()" Configuration parameters ​ InputParams Configuration ​ exotel_sample_rate int default: "8000" Sample rate used by Exotel (typically 8kHz) ​ sample_rate int | None default: "None" Optional override for pipeline input sample rate ​ Basic Usage Copy Ask AI from pipecat.serializers.exotel import ExotelFrameSerializer from pipecat.transports.network.fastapi_websocket import ( FastAPIWebsocketTransport, FastAPIWebsocketParams ) # Extract required values from Exotel WebSocket connection stream_id = call_data[ "stream_id" ] call_sid = call_data[ "start" ][ "call_sid" ] # Create serializer serializer = ExotelFrameSerializer( stream_id = stream_id, call_sid, ) # Use with FastAPIWebsocketTransport transport = FastAPIWebsocketTransport( websocket = websocket, params = FastAPIWebsocketParams( audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer(), serializer = serializer, ) ) ​ Server Code Example Here’s a complete example of handling a Exotel WebSocket connection: Copy Ask AI from fastapi import FastAPI, WebSocket from pipecat.serializers.exotel import ExotelFrameSerializer import json import os app = FastAPI() @app.websocket ( "/ws" ) async def websocket_endpoint ( websocket : WebSocket): await websocket.accept() # Read initial messages from Exotel start_data = websocket.iter_text() await start_data. __anext__ () # Skip first message # Parse the second message to get call details call_data = json.loads( await start_data. __anext__ ()) # Extract Exotel-specific IDs and encoding stream_id = call_data[ "stream_id" ] call_sid = call_data[ "start" ][ "call_sid" ] # Create serializer with API key for auto hang-up serializer = ExotelFrameSerializer( stream_id = stream_id, call_sid = call_sid, ) # Continue with transport and pipeline setup... Frame Serializer Overview PlivoFrameSerializer On this page Overview Features Installation Configuration Constructor Parameters InputParams Configuration Basic Usage Server Code Example Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/serializers_exotel_707f5e2b.txt b/serializers_exotel_707f5e2b.txt
new file mode 100644
index 0000000000000000000000000000000000000000..c19201a0f11f9e0a2b76ede2de854f29592cf404
--- /dev/null
+++ b/serializers_exotel_707f5e2b.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/serializers/exotel#param-exotel-sample-rate
+Title: ExotelFrameSerializer - Pipecat
+==================================================
+
+ExotelFrameSerializer - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Serializers ExotelFrameSerializer Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Frame Serializer Overview ExotelFrameSerializer PlivoFrameSerializer TwilioFrameSerializer TelnyxFrameSerializer Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview ExotelFrameSerializer enables integration with Exotel’s WebSocket media streaming protocol, allowing your Pipecat application to handle phone calls via Exotel’s voice services. ​ Features Bidirectional audio conversion between Pipecat and Exotel DTMF (touch-tone) event handling ​ Installation The ExotelFrameSerializer does not require any additional dependencies beyond the core Pipecat library. ​ Configuration ​ Constructor Parameters ​ stream_id str required The Stream ID for Exotel ​ call_sid Optional[str] default: "None" The associated Exotel Call SID. ​ params InputParams default: "InputParams()" Configuration parameters ​ InputParams Configuration ​ exotel_sample_rate int default: "8000" Sample rate used by Exotel (typically 8kHz) ​ sample_rate int | None default: "None" Optional override for pipeline input sample rate ​ Basic Usage Copy Ask AI from pipecat.serializers.exotel import ExotelFrameSerializer from pipecat.transports.network.fastapi_websocket import ( FastAPIWebsocketTransport, FastAPIWebsocketParams ) # Extract required values from Exotel WebSocket connection stream_id = call_data[ "stream_id" ] call_sid = call_data[ "start" ][ "call_sid" ] # Create serializer serializer = ExotelFrameSerializer( stream_id = stream_id, call_sid, ) # Use with FastAPIWebsocketTransport transport = FastAPIWebsocketTransport( websocket = websocket, params = FastAPIWebsocketParams( audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer(), serializer = serializer, ) ) ​ Server Code Example Here’s a complete example of handling a Exotel WebSocket connection: Copy Ask AI from fastapi import FastAPI, WebSocket from pipecat.serializers.exotel import ExotelFrameSerializer import json import os app = FastAPI() @app.websocket ( "/ws" ) async def websocket_endpoint ( websocket : WebSocket): await websocket.accept() # Read initial messages from Exotel start_data = websocket.iter_text() await start_data. __anext__ () # Skip first message # Parse the second message to get call details call_data = json.loads( await start_data. __anext__ ()) # Extract Exotel-specific IDs and encoding stream_id = call_data[ "stream_id" ] call_sid = call_data[ "start" ][ "call_sid" ] # Create serializer with API key for auto hang-up serializer = ExotelFrameSerializer( stream_id = stream_id, call_sid = call_sid, ) # Continue with transport and pipeline setup... Frame Serializer Overview PlivoFrameSerializer On this page Overview Features Installation Configuration Constructor Parameters InputParams Configuration Basic Usage Server Code Example Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/serializers_exotel_bc0065dd.txt b/serializers_exotel_bc0065dd.txt
new file mode 100644
index 0000000000000000000000000000000000000000..5690d0c7c0a039b914cfd28655c45109d72f7923
--- /dev/null
+++ b/serializers_exotel_bc0065dd.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/serializers/exotel#param-stream-id
+Title: ExotelFrameSerializer - Pipecat
+==================================================
+
+ExotelFrameSerializer - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Serializers ExotelFrameSerializer Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Frame Serializer Overview ExotelFrameSerializer PlivoFrameSerializer TwilioFrameSerializer TelnyxFrameSerializer Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview ExotelFrameSerializer enables integration with Exotel’s WebSocket media streaming protocol, allowing your Pipecat application to handle phone calls via Exotel’s voice services. ​ Features Bidirectional audio conversion between Pipecat and Exotel DTMF (touch-tone) event handling ​ Installation The ExotelFrameSerializer does not require any additional dependencies beyond the core Pipecat library. ​ Configuration ​ Constructor Parameters ​ stream_id str required The Stream ID for Exotel ​ call_sid Optional[str] default: "None" The associated Exotel Call SID. ​ params InputParams default: "InputParams()" Configuration parameters ​ InputParams Configuration ​ exotel_sample_rate int default: "8000" Sample rate used by Exotel (typically 8kHz) ​ sample_rate int | None default: "None" Optional override for pipeline input sample rate ​ Basic Usage Copy Ask AI from pipecat.serializers.exotel import ExotelFrameSerializer from pipecat.transports.network.fastapi_websocket import ( FastAPIWebsocketTransport, FastAPIWebsocketParams ) # Extract required values from Exotel WebSocket connection stream_id = call_data[ "stream_id" ] call_sid = call_data[ "start" ][ "call_sid" ] # Create serializer serializer = ExotelFrameSerializer( stream_id = stream_id, call_sid, ) # Use with FastAPIWebsocketTransport transport = FastAPIWebsocketTransport( websocket = websocket, params = FastAPIWebsocketParams( audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer(), serializer = serializer, ) ) ​ Server Code Example Here’s a complete example of handling a Exotel WebSocket connection: Copy Ask AI from fastapi import FastAPI, WebSocket from pipecat.serializers.exotel import ExotelFrameSerializer import json import os app = FastAPI() @app.websocket ( "/ws" ) async def websocket_endpoint ( websocket : WebSocket): await websocket.accept() # Read initial messages from Exotel start_data = websocket.iter_text() await start_data. __anext__ () # Skip first message # Parse the second message to get call details call_data = json.loads( await start_data. __anext__ ()) # Extract Exotel-specific IDs and encoding stream_id = call_data[ "stream_id" ] call_sid = call_data[ "start" ][ "call_sid" ] # Create serializer with API key for auto hang-up serializer = ExotelFrameSerializer( stream_id = stream_id, call_sid = call_sid, ) # Continue with transport and pipeline setup... Frame Serializer Overview PlivoFrameSerializer On this page Overview Features Installation Configuration Constructor Parameters InputParams Configuration Basic Usage Server Code Example Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/serializers_exotel_fae956ed.txt b/serializers_exotel_fae956ed.txt
new file mode 100644
index 0000000000000000000000000000000000000000..13763e6b214b7766b2f18a55332c9d6abd21951a
--- /dev/null
+++ b/serializers_exotel_fae956ed.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/serializers/exotel#installation
+Title: ExotelFrameSerializer - Pipecat
+==================================================
+
+ExotelFrameSerializer - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Serializers ExotelFrameSerializer Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Frame Serializer Overview ExotelFrameSerializer PlivoFrameSerializer TwilioFrameSerializer TelnyxFrameSerializer Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview ExotelFrameSerializer enables integration with Exotel’s WebSocket media streaming protocol, allowing your Pipecat application to handle phone calls via Exotel’s voice services. ​ Features Bidirectional audio conversion between Pipecat and Exotel DTMF (touch-tone) event handling ​ Installation The ExotelFrameSerializer does not require any additional dependencies beyond the core Pipecat library. ​ Configuration ​ Constructor Parameters ​ stream_id str required The Stream ID for Exotel ​ call_sid Optional[str] default: "None" The associated Exotel Call SID. ​ params InputParams default: "InputParams()" Configuration parameters ​ InputParams Configuration ​ exotel_sample_rate int default: "8000" Sample rate used by Exotel (typically 8kHz) ​ sample_rate int | None default: "None" Optional override for pipeline input sample rate ​ Basic Usage Copy Ask AI from pipecat.serializers.exotel import ExotelFrameSerializer from pipecat.transports.network.fastapi_websocket import ( FastAPIWebsocketTransport, FastAPIWebsocketParams ) # Extract required values from Exotel WebSocket connection stream_id = call_data[ "stream_id" ] call_sid = call_data[ "start" ][ "call_sid" ] # Create serializer serializer = ExotelFrameSerializer( stream_id = stream_id, call_sid, ) # Use with FastAPIWebsocketTransport transport = FastAPIWebsocketTransport( websocket = websocket, params = FastAPIWebsocketParams( audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer(), serializer = serializer, ) ) ​ Server Code Example Here’s a complete example of handling a Exotel WebSocket connection: Copy Ask AI from fastapi import FastAPI, WebSocket from pipecat.serializers.exotel import ExotelFrameSerializer import json import os app = FastAPI() @app.websocket ( "/ws" ) async def websocket_endpoint ( websocket : WebSocket): await websocket.accept() # Read initial messages from Exotel start_data = websocket.iter_text() await start_data. __anext__ () # Skip first message # Parse the second message to get call details call_data = json.loads( await start_data. __anext__ ()) # Extract Exotel-specific IDs and encoding stream_id = call_data[ "stream_id" ] call_sid = call_data[ "start" ][ "call_sid" ] # Create serializer with API key for auto hang-up serializer = ExotelFrameSerializer( stream_id = stream_id, call_sid = call_sid, ) # Continue with transport and pipeline setup... Frame Serializer Overview PlivoFrameSerializer On this page Overview Features Installation Configuration Constructor Parameters InputParams Configuration Basic Usage Server Code Example Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/serializers_introduction_1a37e675.txt b/serializers_introduction_1a37e675.txt
new file mode 100644
index 0000000000000000000000000000000000000000..e0a3c9b9dd93f1b282ac198eb528c1388440309d
--- /dev/null
+++ b/serializers_introduction_1a37e675.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/serializers/introduction#core-responsibilities
+Title: Frame Serializer Overview - Pipecat
+==================================================
+
+Frame Serializer Overview - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Serializers Frame Serializer Overview Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Frame Serializer Overview ExotelFrameSerializer PlivoFrameSerializer TwilioFrameSerializer TelnyxFrameSerializer Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview Frame serializers are components that convert between Pipecat’s internal frame format and external protocols or formats. They’re essential when integrating with third-party services or APIs that have their own message formats. ​ Core Responsibilities Serializers handle: Serialization : Converting Pipecat frames to external formats or protocols Deserialization : Converting external messages to Pipecat frames Protocol-specific behaviors : Managing unique aspects of each integration ​ Available Serializers Pipecat includes serializers for popular voice and communications platforms: Exotel Serializer For integrating with Exotel WebSocket media streaming Plivo Serializer For integrating with Telnyx WebSocket media streaming Telnyx Serializer For integrating with Telnyx WebSocket media streaming Twilio Serializer For integrating with Twilio Media Streams WebSocket protocol ​ Custom Serializers You can create custom serializers by implementing the FrameSerializer base class: Copy Ask AI from pipecat.serializers.base_serializer import FrameSerializer, FrameSerializerType from pipecat.frames.frames import Frame, StartFrame class MyCustomSerializer ( FrameSerializer ): @ property def type ( self ) -> FrameSerializerType: return FrameSerializerType. TEXT # or BINARY async def setup ( self , frame : StartFrame): # Initialize with pipeline configuration pass async def serialize ( self , frame : Frame) -> str | bytes | None : # Convert Pipecat frame to external format pass async def deserialize ( self , data : str | bytes ) -> Frame | None : # Convert external data to Pipecat frame pass WebSocket Server ExotelFrameSerializer On this page Overview Core Responsibilities Available Serializers Custom Serializers Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/serializers_introduction_b9a45478.txt b/serializers_introduction_b9a45478.txt
new file mode 100644
index 0000000000000000000000000000000000000000..a514a3a68b983434195db9cba82338a67889ec8a
--- /dev/null
+++ b/serializers_introduction_b9a45478.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/serializers/introduction
+Title: Frame Serializer Overview - Pipecat
+==================================================
+
+Frame Serializer Overview - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Serializers Frame Serializer Overview Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Frame Serializer Overview ExotelFrameSerializer PlivoFrameSerializer TwilioFrameSerializer TelnyxFrameSerializer Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview Frame serializers are components that convert between Pipecat’s internal frame format and external protocols or formats. They’re essential when integrating with third-party services or APIs that have their own message formats. ​ Core Responsibilities Serializers handle: Serialization : Converting Pipecat frames to external formats or protocols Deserialization : Converting external messages to Pipecat frames Protocol-specific behaviors : Managing unique aspects of each integration ​ Available Serializers Pipecat includes serializers for popular voice and communications platforms: Exotel Serializer For integrating with Exotel WebSocket media streaming Plivo Serializer For integrating with Telnyx WebSocket media streaming Telnyx Serializer For integrating with Telnyx WebSocket media streaming Twilio Serializer For integrating with Twilio Media Streams WebSocket protocol ​ Custom Serializers You can create custom serializers by implementing the FrameSerializer base class: Copy Ask AI from pipecat.serializers.base_serializer import FrameSerializer, FrameSerializerType from pipecat.frames.frames import Frame, StartFrame class MyCustomSerializer ( FrameSerializer ): @ property def type ( self ) -> FrameSerializerType: return FrameSerializerType. TEXT # or BINARY async def setup ( self , frame : StartFrame): # Initialize with pipeline configuration pass async def serialize ( self , frame : Frame) -> str | bytes | None : # Convert Pipecat frame to external format pass async def deserialize ( self , data : str | bytes ) -> Frame | None : # Convert external data to Pipecat frame pass WebSocket Server ExotelFrameSerializer On this page Overview Core Responsibilities Available Serializers Custom Serializers Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/serializers_plivo_804b26bf.txt b/serializers_plivo_804b26bf.txt
new file mode 100644
index 0000000000000000000000000000000000000000..3767e9585983816894f59b4e389f577579e2b0bf
--- /dev/null
+++ b/serializers_plivo_804b26bf.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/serializers/plivo#inputparams-configuration
+Title: PlivoFrameSerializer - Pipecat
+==================================================
+
+PlivoFrameSerializer - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Serializers PlivoFrameSerializer Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Frame Serializer Overview ExotelFrameSerializer PlivoFrameSerializer TwilioFrameSerializer TelnyxFrameSerializer Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview PlivoFrameSerializer enables integration with Plivo’s Audio Streaming WebSocket protocol, allowing your Pipecat application to handle phone calls via Plivo’s voice services. ​ Features Bidirectional audio conversion between Pipecat and Plivo DTMF (touch-tone) event handling Automatic call termination via Plivo’s REST API μ-law audio encoding/decoding ​ Installation The PlivoFrameSerializer does not require any additional dependencies beyond the core Pipecat library. ​ Configuration ​ Constructor Parameters ​ stream_id str required The Plivo Stream ID ​ call_id Optional[str] default: "None" The associated Plivo Call ID (required for auto hang-up) ​ auth_id Optional[str] default: "None" Plivo auth ID (required for auto hang-up) ​ auth_token Optional[str] default: "None" Plivo auth token (required for auto hang-up) ​ params InputParams default: "InputParams()" Configuration parameters ​ InputParams Configuration ​ plivo_sample_rate int default: "8000" Sample rate used by Plivo (typically 8kHz) ​ sample_rate int | None default: "None" Optional override for pipeline input sample rate ​ auto_hang_up bool default: "True" Whether to automatically terminate call on EndFrame ​ Basic Usage Copy Ask AI from pipecat.serializers.plivo import PlivoFrameSerializer from pipecat.transports.network.fastapi_websocket import ( FastAPIWebsocketTransport, FastAPIWebsocketParams ) # Extract required values from Plivo WebSocket connection stream_id = start_message[ "start" ][ "streamId" ] call_id = start_message[ "start" ][ "callId" ] # Create serializer serializer = PlivoFrameSerializer( stream_id = stream_id, call_id = call_id, auth_id = "your_plivo_auth_id" , auth_token = "your_plivo_auth_token" ) # Use with FastAPIWebsocketTransport transport = FastAPIWebsocketTransport( websocket = websocket, params = FastAPIWebsocketParams( audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer(), serializer = serializer, ) ) ​ Hang-up Functionality When auto_hang_up is enabled, the serializer will automatically hang up the Plivo call when an EndFrame or CancelFrame is processed, using Plivo’s REST API: Copy Ask AI # Properly configured with hang-up support serializer = PlivoFrameSerializer( stream_id = stream_id, call_id = call_id, # Required for auto hang-up auth_id = os.getenv( "PLIVO_AUTH_ID" ), # Required for auto hang-up auth_token = os.getenv( "PLIVO_AUTH_TOKEN" ), # Required for auto hang-up ) ​ Server Code Example Here’s a complete example of handling a Plivo WebSocket connection: Copy Ask AI from fastapi import FastAPI, WebSocket from pipecat.serializers.plivo import PlivoFrameSerializer import json import os app = FastAPI() @app.websocket ( "/ws" ) async def websocket_endpoint ( websocket : WebSocket): await websocket.accept() # Read the start message from Plivo start_data = websocket.iter_text() start_message = json.loads( await start_data. __anext__ ()) # Extract Plivo-specific IDs from the start event start_info = start_message.get( "start" , {}) stream_id = start_info.get( "streamId" ) call_id = start_info.get( "callId" ) # Create serializer with authentication for auto hang-up serializer = PlivoFrameSerializer( stream_id = stream_id, call_id = call_id, auth_id = os.getenv( "PLIVO_AUTH_ID" ), auth_token = os.getenv( "PLIVO_AUTH_TOKEN" ), ) # Continue with transport and pipeline setup... ​ Plivo XML Configuration To enable audio streaming with Plivo, you’ll need to configure your Plivo application to return appropriate XML: Copy Ask AI <? xml version = "1.0" encoding = "UTF-8" ?> < Response > < Stream keepCallAlive = "true" bidirectional = "true" contentType = "audio/x-mulaw;rate=8000" > wss://your-websocket-url/ws </ Stream > </ Response > The bidirectional="true" attribute is required for two-way audio communication, and keepCallAlive="true" prevents the call from being disconnected after XML execution. ​ Key Differences from Twilio Stream Identifier : Plivo uses streamId instead of streamSid Call Identifier : Plivo uses callId instead of callSid XML Structure : Plivo uses <Stream> element directly instead of <Connect><Stream> Authentication : Plivo uses Auth ID and Auth Token instead of Account SID and Auth Token See the Plivo Chatbot example for a complete implementation. ExotelFrameSerializer TwilioFrameSerializer On this page Overview Features Installation Configuration Constructor Parameters InputParams Configuration Basic Usage Hang-up Functionality Server Code Example Plivo XML Configuration Key Differences from Twilio Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/serializers_telnyx_668c3790.txt b/serializers_telnyx_668c3790.txt
new file mode 100644
index 0000000000000000000000000000000000000000..84dc1bdbb0a8550743521338c3b8f2418965e621
--- /dev/null
+++ b/serializers_telnyx_668c3790.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/serializers/telnyx#constructor-parameters
+Title: TelnyxFrameSerializer - Pipecat
+==================================================
+
+TelnyxFrameSerializer - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Serializers TelnyxFrameSerializer Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Frame Serializer Overview ExotelFrameSerializer PlivoFrameSerializer TwilioFrameSerializer TelnyxFrameSerializer Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview TelnyxFrameSerializer enables integration with Telnyx’s WebSocket media streaming protocol, allowing your Pipecat application to handle phone calls via Telnyx’s voice services. ​ Features Bidirectional audio conversion between Pipecat and Telnyx DTMF (touch-tone) event handling Automatic call termination via Telnyx’s REST API Support for multiple audio encodings (PCMU, PCMA) ​ Installation The TelnyxFrameSerializer does not require any additional dependencies beyond the core Pipecat library. ​ Configuration ​ Constructor Parameters ​ stream_id str required The Stream ID for Telnyx ​ outbound_encoding str required The encoding type for outbound audio (e.g., “PCMU”, “PCMA”) ​ inbound_encoding str required The encoding type for inbound audio (e.g., “PCMU”, “PCMA”) ​ call_control_id Optional[str] default: "None" The Call Control ID for the Telnyx call (required for auto hang-up) ​ api_key Optional[str] default: "None" Your Telnyx API key (required for auto hang-up) ​ params InputParams default: "InputParams()" Configuration parameters ​ InputParams Configuration ​ telnyx_sample_rate int default: "8000" Sample rate used by Telnyx (typically 8kHz) ​ sample_rate int | None default: "None" Optional override for pipeline input sample rate ​ inbound_encoding str default: "PCMU" Audio encoding for data sent to Telnyx ​ outbound_encoding str default: "PCMU" Audio encoding for data received from Telnyx ​ auto_hang_up bool default: "True" Whether to automatically terminate call on EndFrame ​ Basic Usage Copy Ask AI from pipecat.serializers.telnyx import TelnyxFrameSerializer from pipecat.transports.network.fastapi_websocket import ( FastAPIWebsocketTransport, FastAPIWebsocketParams ) # Extract required values from Telnyx WebSocket connection stream_id = call_data[ "stream_id" ] call_control_id = call_data[ "start" ][ "call_control_id" ] outbound_encoding = call_data[ "start" ][ "media_format" ][ "encoding" ] # Create serializer serializer = TelnyxFrameSerializer( stream_id = stream_id, outbound_encoding = outbound_encoding, inbound_encoding = "PCMU" , call_control_id = call_control_id, api_key = os.getenv( "TELNYX_API_KEY" ) ) # Use with FastAPIWebsocketTransport transport = FastAPIWebsocketTransport( websocket = websocket, params = FastAPIWebsocketParams( audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer(), serializer = serializer, ) ) ​ Hang-up Functionality When auto_hang_up is enabled, the serializer will automatically hang up the Telnyx call when an EndFrame or CancelFrame is processed, using Telnyx’s REST API: Copy Ask AI # Properly configured with hang-up support serializer = TelnyxFrameSerializer( stream_id = stream_id, outbound_encoding = outbound_encoding, inbound_encoding = "PCMU" , call_control_id = call_control_id, # Required for auto hang-up api_key = os.getenv( "TELNYX_API_KEY" ) # Required for auto hang-up ) ​ Server Code Example Here’s a complete example of handling a Telnyx WebSocket connection: Copy Ask AI from fastapi import FastAPI, WebSocket from pipecat.serializers.telnyx import TelnyxFrameSerializer import json import os app = FastAPI() @app.websocket ( "/ws" ) async def websocket_endpoint ( websocket : WebSocket): await websocket.accept() # Read initial messages from Telnyx start_data = websocket.iter_text() await start_data. __anext__ () # Skip first message # Parse the second message to get call details call_data = json.loads( await start_data. __anext__ ()) # Extract Telnyx-specific IDs and encoding stream_id = call_data[ "stream_id" ] call_control_id = call_data[ "start" ][ "call_control_id" ] outbound_encoding = call_data[ "start" ][ "media_format" ][ "encoding" ] # Create serializer with API key for auto hang-up serializer = TelnyxFrameSerializer( stream_id = stream_id, outbound_encoding = outbound_encoding, inbound_encoding = "PCMU" , call_control_id = call_control_id, api_key = os.getenv( "TELNYX_API_KEY" ), ) # Continue with transport and pipeline setup... See the Telnyx Chatbot example for a complete implementation. TwilioFrameSerializer AssemblyAI On this page Overview Features Installation Configuration Constructor Parameters InputParams Configuration Basic Usage Hang-up Functionality Server Code Example Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/serializers_telnyx_79490174.txt b/serializers_telnyx_79490174.txt
new file mode 100644
index 0000000000000000000000000000000000000000..371b1a6649ffc3d5b5cf21caa67b3f0ae27096b4
--- /dev/null
+++ b/serializers_telnyx_79490174.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/serializers/telnyx#installation
+Title: TelnyxFrameSerializer - Pipecat
+==================================================
+
+TelnyxFrameSerializer - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Serializers TelnyxFrameSerializer Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Frame Serializer Overview ExotelFrameSerializer PlivoFrameSerializer TwilioFrameSerializer TelnyxFrameSerializer Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview TelnyxFrameSerializer enables integration with Telnyx’s WebSocket media streaming protocol, allowing your Pipecat application to handle phone calls via Telnyx’s voice services. ​ Features Bidirectional audio conversion between Pipecat and Telnyx DTMF (touch-tone) event handling Automatic call termination via Telnyx’s REST API Support for multiple audio encodings (PCMU, PCMA) ​ Installation The TelnyxFrameSerializer does not require any additional dependencies beyond the core Pipecat library. ​ Configuration ​ Constructor Parameters ​ stream_id str required The Stream ID for Telnyx ​ outbound_encoding str required The encoding type for outbound audio (e.g., “PCMU”, “PCMA”) ​ inbound_encoding str required The encoding type for inbound audio (e.g., “PCMU”, “PCMA”) ​ call_control_id Optional[str] default: "None" The Call Control ID for the Telnyx call (required for auto hang-up) ​ api_key Optional[str] default: "None" Your Telnyx API key (required for auto hang-up) ​ params InputParams default: "InputParams()" Configuration parameters ​ InputParams Configuration ​ telnyx_sample_rate int default: "8000" Sample rate used by Telnyx (typically 8kHz) ​ sample_rate int | None default: "None" Optional override for pipeline input sample rate ​ inbound_encoding str default: "PCMU" Audio encoding for data sent to Telnyx ​ outbound_encoding str default: "PCMU" Audio encoding for data received from Telnyx ​ auto_hang_up bool default: "True" Whether to automatically terminate call on EndFrame ​ Basic Usage Copy Ask AI from pipecat.serializers.telnyx import TelnyxFrameSerializer from pipecat.transports.network.fastapi_websocket import ( FastAPIWebsocketTransport, FastAPIWebsocketParams ) # Extract required values from Telnyx WebSocket connection stream_id = call_data[ "stream_id" ] call_control_id = call_data[ "start" ][ "call_control_id" ] outbound_encoding = call_data[ "start" ][ "media_format" ][ "encoding" ] # Create serializer serializer = TelnyxFrameSerializer( stream_id = stream_id, outbound_encoding = outbound_encoding, inbound_encoding = "PCMU" , call_control_id = call_control_id, api_key = os.getenv( "TELNYX_API_KEY" ) ) # Use with FastAPIWebsocketTransport transport = FastAPIWebsocketTransport( websocket = websocket, params = FastAPIWebsocketParams( audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer(), serializer = serializer, ) ) ​ Hang-up Functionality When auto_hang_up is enabled, the serializer will automatically hang up the Telnyx call when an EndFrame or CancelFrame is processed, using Telnyx’s REST API: Copy Ask AI # Properly configured with hang-up support serializer = TelnyxFrameSerializer( stream_id = stream_id, outbound_encoding = outbound_encoding, inbound_encoding = "PCMU" , call_control_id = call_control_id, # Required for auto hang-up api_key = os.getenv( "TELNYX_API_KEY" ) # Required for auto hang-up ) ​ Server Code Example Here’s a complete example of handling a Telnyx WebSocket connection: Copy Ask AI from fastapi import FastAPI, WebSocket from pipecat.serializers.telnyx import TelnyxFrameSerializer import json import os app = FastAPI() @app.websocket ( "/ws" ) async def websocket_endpoint ( websocket : WebSocket): await websocket.accept() # Read initial messages from Telnyx start_data = websocket.iter_text() await start_data. __anext__ () # Skip first message # Parse the second message to get call details call_data = json.loads( await start_data. __anext__ ()) # Extract Telnyx-specific IDs and encoding stream_id = call_data[ "stream_id" ] call_control_id = call_data[ "start" ][ "call_control_id" ] outbound_encoding = call_data[ "start" ][ "media_format" ][ "encoding" ] # Create serializer with API key for auto hang-up serializer = TelnyxFrameSerializer( stream_id = stream_id, outbound_encoding = outbound_encoding, inbound_encoding = "PCMU" , call_control_id = call_control_id, api_key = os.getenv( "TELNYX_API_KEY" ), ) # Continue with transport and pipeline setup... See the Telnyx Chatbot example for a complete implementation. TwilioFrameSerializer AssemblyAI On this page Overview Features Installation Configuration Constructor Parameters InputParams Configuration Basic Usage Hang-up Functionality Server Code Example Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/serializers_telnyx_8d687e7c.txt b/serializers_telnyx_8d687e7c.txt
new file mode 100644
index 0000000000000000000000000000000000000000..840e95e8ad182282f1f353af9e239639479403c0
--- /dev/null
+++ b/serializers_telnyx_8d687e7c.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/serializers/telnyx#param-api-key
+Title: TelnyxFrameSerializer - Pipecat
+==================================================
+
+TelnyxFrameSerializer - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Serializers TelnyxFrameSerializer Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Frame Serializer Overview ExotelFrameSerializer PlivoFrameSerializer TwilioFrameSerializer TelnyxFrameSerializer Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview TelnyxFrameSerializer enables integration with Telnyx’s WebSocket media streaming protocol, allowing your Pipecat application to handle phone calls via Telnyx’s voice services. ​ Features Bidirectional audio conversion between Pipecat and Telnyx DTMF (touch-tone) event handling Automatic call termination via Telnyx’s REST API Support for multiple audio encodings (PCMU, PCMA) ​ Installation The TelnyxFrameSerializer does not require any additional dependencies beyond the core Pipecat library. ​ Configuration ​ Constructor Parameters ​ stream_id str required The Stream ID for Telnyx ​ outbound_encoding str required The encoding type for outbound audio (e.g., “PCMU”, “PCMA”) ​ inbound_encoding str required The encoding type for inbound audio (e.g., “PCMU”, “PCMA”) ​ call_control_id Optional[str] default: "None" The Call Control ID for the Telnyx call (required for auto hang-up) ​ api_key Optional[str] default: "None" Your Telnyx API key (required for auto hang-up) ​ params InputParams default: "InputParams()" Configuration parameters ​ InputParams Configuration ​ telnyx_sample_rate int default: "8000" Sample rate used by Telnyx (typically 8kHz) ​ sample_rate int | None default: "None" Optional override for pipeline input sample rate ​ inbound_encoding str default: "PCMU" Audio encoding for data sent to Telnyx ​ outbound_encoding str default: "PCMU" Audio encoding for data received from Telnyx ​ auto_hang_up bool default: "True" Whether to automatically terminate call on EndFrame ​ Basic Usage Copy Ask AI from pipecat.serializers.telnyx import TelnyxFrameSerializer from pipecat.transports.network.fastapi_websocket import ( FastAPIWebsocketTransport, FastAPIWebsocketParams ) # Extract required values from Telnyx WebSocket connection stream_id = call_data[ "stream_id" ] call_control_id = call_data[ "start" ][ "call_control_id" ] outbound_encoding = call_data[ "start" ][ "media_format" ][ "encoding" ] # Create serializer serializer = TelnyxFrameSerializer( stream_id = stream_id, outbound_encoding = outbound_encoding, inbound_encoding = "PCMU" , call_control_id = call_control_id, api_key = os.getenv( "TELNYX_API_KEY" ) ) # Use with FastAPIWebsocketTransport transport = FastAPIWebsocketTransport( websocket = websocket, params = FastAPIWebsocketParams( audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer(), serializer = serializer, ) ) ​ Hang-up Functionality When auto_hang_up is enabled, the serializer will automatically hang up the Telnyx call when an EndFrame or CancelFrame is processed, using Telnyx’s REST API: Copy Ask AI # Properly configured with hang-up support serializer = TelnyxFrameSerializer( stream_id = stream_id, outbound_encoding = outbound_encoding, inbound_encoding = "PCMU" , call_control_id = call_control_id, # Required for auto hang-up api_key = os.getenv( "TELNYX_API_KEY" ) # Required for auto hang-up ) ​ Server Code Example Here’s a complete example of handling a Telnyx WebSocket connection: Copy Ask AI from fastapi import FastAPI, WebSocket from pipecat.serializers.telnyx import TelnyxFrameSerializer import json import os app = FastAPI() @app.websocket ( "/ws" ) async def websocket_endpoint ( websocket : WebSocket): await websocket.accept() # Read initial messages from Telnyx start_data = websocket.iter_text() await start_data. __anext__ () # Skip first message # Parse the second message to get call details call_data = json.loads( await start_data. __anext__ ()) # Extract Telnyx-specific IDs and encoding stream_id = call_data[ "stream_id" ] call_control_id = call_data[ "start" ][ "call_control_id" ] outbound_encoding = call_data[ "start" ][ "media_format" ][ "encoding" ] # Create serializer with API key for auto hang-up serializer = TelnyxFrameSerializer( stream_id = stream_id, outbound_encoding = outbound_encoding, inbound_encoding = "PCMU" , call_control_id = call_control_id, api_key = os.getenv( "TELNYX_API_KEY" ), ) # Continue with transport and pipeline setup... See the Telnyx Chatbot example for a complete implementation. TwilioFrameSerializer AssemblyAI On this page Overview Features Installation Configuration Constructor Parameters InputParams Configuration Basic Usage Hang-up Functionality Server Code Example Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/serializers_telnyx_f0ad1487.txt b/serializers_telnyx_f0ad1487.txt
new file mode 100644
index 0000000000000000000000000000000000000000..f3548dcd722df3e2b935ea2c7cf0285895f2add5
--- /dev/null
+++ b/serializers_telnyx_f0ad1487.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/serializers/telnyx#param-inbound-encoding
+Title: TelnyxFrameSerializer - Pipecat
+==================================================
+
+TelnyxFrameSerializer - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Serializers TelnyxFrameSerializer Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Frame Serializer Overview ExotelFrameSerializer PlivoFrameSerializer TwilioFrameSerializer TelnyxFrameSerializer Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview TelnyxFrameSerializer enables integration with Telnyx’s WebSocket media streaming protocol, allowing your Pipecat application to handle phone calls via Telnyx’s voice services. ​ Features Bidirectional audio conversion between Pipecat and Telnyx DTMF (touch-tone) event handling Automatic call termination via Telnyx’s REST API Support for multiple audio encodings (PCMU, PCMA) ​ Installation The TelnyxFrameSerializer does not require any additional dependencies beyond the core Pipecat library. ​ Configuration ​ Constructor Parameters ​ stream_id str required The Stream ID for Telnyx ​ outbound_encoding str required The encoding type for outbound audio (e.g., “PCMU”, “PCMA”) ​ inbound_encoding str required The encoding type for inbound audio (e.g., “PCMU”, “PCMA”) ​ call_control_id Optional[str] default: "None" The Call Control ID for the Telnyx call (required for auto hang-up) ​ api_key Optional[str] default: "None" Your Telnyx API key (required for auto hang-up) ​ params InputParams default: "InputParams()" Configuration parameters ​ InputParams Configuration ​ telnyx_sample_rate int default: "8000" Sample rate used by Telnyx (typically 8kHz) ​ sample_rate int | None default: "None" Optional override for pipeline input sample rate ​ inbound_encoding str default: "PCMU" Audio encoding for data sent to Telnyx ​ outbound_encoding str default: "PCMU" Audio encoding for data received from Telnyx ​ auto_hang_up bool default: "True" Whether to automatically terminate call on EndFrame ​ Basic Usage Copy Ask AI from pipecat.serializers.telnyx import TelnyxFrameSerializer from pipecat.transports.network.fastapi_websocket import ( FastAPIWebsocketTransport, FastAPIWebsocketParams ) # Extract required values from Telnyx WebSocket connection stream_id = call_data[ "stream_id" ] call_control_id = call_data[ "start" ][ "call_control_id" ] outbound_encoding = call_data[ "start" ][ "media_format" ][ "encoding" ] # Create serializer serializer = TelnyxFrameSerializer( stream_id = stream_id, outbound_encoding = outbound_encoding, inbound_encoding = "PCMU" , call_control_id = call_control_id, api_key = os.getenv( "TELNYX_API_KEY" ) ) # Use with FastAPIWebsocketTransport transport = FastAPIWebsocketTransport( websocket = websocket, params = FastAPIWebsocketParams( audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer(), serializer = serializer, ) ) ​ Hang-up Functionality When auto_hang_up is enabled, the serializer will automatically hang up the Telnyx call when an EndFrame or CancelFrame is processed, using Telnyx’s REST API: Copy Ask AI # Properly configured with hang-up support serializer = TelnyxFrameSerializer( stream_id = stream_id, outbound_encoding = outbound_encoding, inbound_encoding = "PCMU" , call_control_id = call_control_id, # Required for auto hang-up api_key = os.getenv( "TELNYX_API_KEY" ) # Required for auto hang-up ) ​ Server Code Example Here’s a complete example of handling a Telnyx WebSocket connection: Copy Ask AI from fastapi import FastAPI, WebSocket from pipecat.serializers.telnyx import TelnyxFrameSerializer import json import os app = FastAPI() @app.websocket ( "/ws" ) async def websocket_endpoint ( websocket : WebSocket): await websocket.accept() # Read initial messages from Telnyx start_data = websocket.iter_text() await start_data. __anext__ () # Skip first message # Parse the second message to get call details call_data = json.loads( await start_data. __anext__ ()) # Extract Telnyx-specific IDs and encoding stream_id = call_data[ "stream_id" ] call_control_id = call_data[ "start" ][ "call_control_id" ] outbound_encoding = call_data[ "start" ][ "media_format" ][ "encoding" ] # Create serializer with API key for auto hang-up serializer = TelnyxFrameSerializer( stream_id = stream_id, outbound_encoding = outbound_encoding, inbound_encoding = "PCMU" , call_control_id = call_control_id, api_key = os.getenv( "TELNYX_API_KEY" ), ) # Continue with transport and pipeline setup... See the Telnyx Chatbot example for a complete implementation. TwilioFrameSerializer AssemblyAI On this page Overview Features Installation Configuration Constructor Parameters InputParams Configuration Basic Usage Hang-up Functionality Server Code Example Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/serializers_twilio_0376f645.txt b/serializers_twilio_0376f645.txt
new file mode 100644
index 0000000000000000000000000000000000000000..f9916d7c096934730f782b84c9c763ab18ddc56b
--- /dev/null
+++ b/serializers_twilio_0376f645.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/serializers/twilio#overview
+Title: TwilioFrameSerializer - Pipecat
+==================================================
+
+TwilioFrameSerializer - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Serializers TwilioFrameSerializer Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Frame Serializer Overview ExotelFrameSerializer PlivoFrameSerializer TwilioFrameSerializer TelnyxFrameSerializer Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview TwilioFrameSerializer enables integration with Twilio’s Media Streams WebSocket protocol, allowing your Pipecat application to handle phone calls via Twilio’s voice services. ​ Features Bidirectional audio conversion between Pipecat and Twilio DTMF (touch-tone) event handling Automatic call termination via Twilio’s REST API μ-law audio encoding/decoding ​ Installation The TwilioFrameSerializer does not require any additional dependencies beyond the core Pipecat library. ​ Configuration ​ Constructor Parameters ​ stream_sid str required The Twilio Media Stream SID ​ call_sid Optional[str] default: "None" The associated Twilio Call SID (required for auto hang-up) ​ account_sid Optional[str] default: "None" Twilio account SID (required for auto hang-up) ​ auth_token Optional[str] default: "None" Twilio auth token (required for auto hang-up) ​ params InputParams default: "InputParams()" Configuration parameters ​ InputParams Configuration ​ twilio_sample_rate int default: "8000" Sample rate used by Twilio (typically 8kHz) ​ sample_rate int | None default: "None" Optional override for pipeline input sample rate ​ auto_hang_up bool default: "True" Whether to automatically terminate call on EndFrame ​ Basic Usage Copy Ask AI from pipecat.serializers.twilio import TwilioFrameSerializer from pipecat.transports.network.fastapi_websocket import ( FastAPIWebsocketTransport, FastAPIWebsocketParams ) # Extract required values from Twilio WebSocket connection stream_sid = call_data[ "start" ][ "streamSid" ] call_sid = call_data[ "start" ][ "callSid" ] # Create serializer serializer = TwilioFrameSerializer( stream_sid = stream_sid, call_sid = call_sid, account_sid = "your_twilio_account_sid" , auth_token = "your_twilio_auth_token" ) # Use with FastAPIWebsocketTransport transport = FastAPIWebsocketTransport( websocket = websocket, params = FastAPIWebsocketParams( audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer(), serializer = serializer, ) ) ​ Hang-up Functionality When auto_hang_up is enabled, the serializer will automatically hang up the Twilio call when an EndFrame or CancelFrame is processed, using Twilio’s REST API: Copy Ask AI # Properly configured with hang-up support serializer = TwilioFrameSerializer( stream_sid = stream_sid, call_sid = call_sid, # Required for auto hang-up account_sid = os.getenv( "TWILIO_ACCOUNT_SID" ), # Required for auto hang-up auth_token = os.getenv( "TWILIO_AUTH_TOKEN" ), # Required for auto hang-up ) ​ Server Code Example Here’s a complete example of handling a Twilio WebSocket connection: Copy Ask AI from fastapi import FastAPI, WebSocket from pipecat.serializers.twilio import TwilioFrameSerializer import json import os app = FastAPI() @app.websocket ( "/ws" ) async def websocket_endpoint ( websocket : WebSocket): await websocket.accept() # Read initial messages from Twilio start_data = websocket.iter_text() await start_data. __anext__ () # Skip first message # Parse the second message to get call details call_data = json.loads( await start_data. __anext__ ()) # Extract Twilio-specific IDs stream_sid = call_data[ "start" ][ "streamSid" ] call_sid = call_data[ "start" ][ "callSid" ] # Create serializer with authentication for auto hang-up serializer = TwilioFrameSerializer( stream_sid = stream_sid, call_sid = call_sid, account_sid = os.getenv( "TWILIO_ACCOUNT_SID" ), auth_token = os.getenv( "TWILIO_AUTH_TOKEN" ), ) # Continue with transport and pipeline setup... See the Twilio Chatbot example for a complete implementation. PlivoFrameSerializer TelnyxFrameSerializer On this page Overview Features Installation Configuration Constructor Parameters InputParams Configuration Basic Usage Hang-up Functionality Server Code Example Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/serializers_twilio_3b9f53e1.txt b/serializers_twilio_3b9f53e1.txt
new file mode 100644
index 0000000000000000000000000000000000000000..3e19f35954c6de14dc4912b69882b94f846d78fd
--- /dev/null
+++ b/serializers_twilio_3b9f53e1.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/serializers/twilio#param-call-sid
+Title: TwilioFrameSerializer - Pipecat
+==================================================
+
+TwilioFrameSerializer - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Serializers TwilioFrameSerializer Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Frame Serializer Overview ExotelFrameSerializer PlivoFrameSerializer TwilioFrameSerializer TelnyxFrameSerializer Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview TwilioFrameSerializer enables integration with Twilio’s Media Streams WebSocket protocol, allowing your Pipecat application to handle phone calls via Twilio’s voice services. ​ Features Bidirectional audio conversion between Pipecat and Twilio DTMF (touch-tone) event handling Automatic call termination via Twilio’s REST API μ-law audio encoding/decoding ​ Installation The TwilioFrameSerializer does not require any additional dependencies beyond the core Pipecat library. ​ Configuration ​ Constructor Parameters ​ stream_sid str required The Twilio Media Stream SID ​ call_sid Optional[str] default: "None" The associated Twilio Call SID (required for auto hang-up) ​ account_sid Optional[str] default: "None" Twilio account SID (required for auto hang-up) ​ auth_token Optional[str] default: "None" Twilio auth token (required for auto hang-up) ​ params InputParams default: "InputParams()" Configuration parameters ​ InputParams Configuration ​ twilio_sample_rate int default: "8000" Sample rate used by Twilio (typically 8kHz) ​ sample_rate int | None default: "None" Optional override for pipeline input sample rate ​ auto_hang_up bool default: "True" Whether to automatically terminate call on EndFrame ​ Basic Usage Copy Ask AI from pipecat.serializers.twilio import TwilioFrameSerializer from pipecat.transports.network.fastapi_websocket import ( FastAPIWebsocketTransport, FastAPIWebsocketParams ) # Extract required values from Twilio WebSocket connection stream_sid = call_data[ "start" ][ "streamSid" ] call_sid = call_data[ "start" ][ "callSid" ] # Create serializer serializer = TwilioFrameSerializer( stream_sid = stream_sid, call_sid = call_sid, account_sid = "your_twilio_account_sid" , auth_token = "your_twilio_auth_token" ) # Use with FastAPIWebsocketTransport transport = FastAPIWebsocketTransport( websocket = websocket, params = FastAPIWebsocketParams( audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer(), serializer = serializer, ) ) ​ Hang-up Functionality When auto_hang_up is enabled, the serializer will automatically hang up the Twilio call when an EndFrame or CancelFrame is processed, using Twilio’s REST API: Copy Ask AI # Properly configured with hang-up support serializer = TwilioFrameSerializer( stream_sid = stream_sid, call_sid = call_sid, # Required for auto hang-up account_sid = os.getenv( "TWILIO_ACCOUNT_SID" ), # Required for auto hang-up auth_token = os.getenv( "TWILIO_AUTH_TOKEN" ), # Required for auto hang-up ) ​ Server Code Example Here’s a complete example of handling a Twilio WebSocket connection: Copy Ask AI from fastapi import FastAPI, WebSocket from pipecat.serializers.twilio import TwilioFrameSerializer import json import os app = FastAPI() @app.websocket ( "/ws" ) async def websocket_endpoint ( websocket : WebSocket): await websocket.accept() # Read initial messages from Twilio start_data = websocket.iter_text() await start_data. __anext__ () # Skip first message # Parse the second message to get call details call_data = json.loads( await start_data. __anext__ ()) # Extract Twilio-specific IDs stream_sid = call_data[ "start" ][ "streamSid" ] call_sid = call_data[ "start" ][ "callSid" ] # Create serializer with authentication for auto hang-up serializer = TwilioFrameSerializer( stream_sid = stream_sid, call_sid = call_sid, account_sid = os.getenv( "TWILIO_ACCOUNT_SID" ), auth_token = os.getenv( "TWILIO_AUTH_TOKEN" ), ) # Continue with transport and pipeline setup... See the Twilio Chatbot example for a complete implementation. PlivoFrameSerializer TelnyxFrameSerializer On this page Overview Features Installation Configuration Constructor Parameters InputParams Configuration Basic Usage Hang-up Functionality Server Code Example Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/serializers_twilio_4f049ab2.txt b/serializers_twilio_4f049ab2.txt
new file mode 100644
index 0000000000000000000000000000000000000000..b0fe8db8fcb8621b6167e3c8a6aa4dbad6c95828
--- /dev/null
+++ b/serializers_twilio_4f049ab2.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/serializers/twilio#server-code-example
+Title: TwilioFrameSerializer - Pipecat
+==================================================
+
+TwilioFrameSerializer - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Serializers TwilioFrameSerializer Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Frame Serializer Overview ExotelFrameSerializer PlivoFrameSerializer TwilioFrameSerializer TelnyxFrameSerializer Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview TwilioFrameSerializer enables integration with Twilio’s Media Streams WebSocket protocol, allowing your Pipecat application to handle phone calls via Twilio’s voice services. ​ Features Bidirectional audio conversion between Pipecat and Twilio DTMF (touch-tone) event handling Automatic call termination via Twilio’s REST API μ-law audio encoding/decoding ​ Installation The TwilioFrameSerializer does not require any additional dependencies beyond the core Pipecat library. ​ Configuration ​ Constructor Parameters ​ stream_sid str required The Twilio Media Stream SID ​ call_sid Optional[str] default: "None" The associated Twilio Call SID (required for auto hang-up) ​ account_sid Optional[str] default: "None" Twilio account SID (required for auto hang-up) ​ auth_token Optional[str] default: "None" Twilio auth token (required for auto hang-up) ​ params InputParams default: "InputParams()" Configuration parameters ​ InputParams Configuration ​ twilio_sample_rate int default: "8000" Sample rate used by Twilio (typically 8kHz) ​ sample_rate int | None default: "None" Optional override for pipeline input sample rate ​ auto_hang_up bool default: "True" Whether to automatically terminate call on EndFrame ​ Basic Usage Copy Ask AI from pipecat.serializers.twilio import TwilioFrameSerializer from pipecat.transports.network.fastapi_websocket import ( FastAPIWebsocketTransport, FastAPIWebsocketParams ) # Extract required values from Twilio WebSocket connection stream_sid = call_data[ "start" ][ "streamSid" ] call_sid = call_data[ "start" ][ "callSid" ] # Create serializer serializer = TwilioFrameSerializer( stream_sid = stream_sid, call_sid = call_sid, account_sid = "your_twilio_account_sid" , auth_token = "your_twilio_auth_token" ) # Use with FastAPIWebsocketTransport transport = FastAPIWebsocketTransport( websocket = websocket, params = FastAPIWebsocketParams( audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer(), serializer = serializer, ) ) ​ Hang-up Functionality When auto_hang_up is enabled, the serializer will automatically hang up the Twilio call when an EndFrame or CancelFrame is processed, using Twilio’s REST API: Copy Ask AI # Properly configured with hang-up support serializer = TwilioFrameSerializer( stream_sid = stream_sid, call_sid = call_sid, # Required for auto hang-up account_sid = os.getenv( "TWILIO_ACCOUNT_SID" ), # Required for auto hang-up auth_token = os.getenv( "TWILIO_AUTH_TOKEN" ), # Required for auto hang-up ) ​ Server Code Example Here’s a complete example of handling a Twilio WebSocket connection: Copy Ask AI from fastapi import FastAPI, WebSocket from pipecat.serializers.twilio import TwilioFrameSerializer import json import os app = FastAPI() @app.websocket ( "/ws" ) async def websocket_endpoint ( websocket : WebSocket): await websocket.accept() # Read initial messages from Twilio start_data = websocket.iter_text() await start_data. __anext__ () # Skip first message # Parse the second message to get call details call_data = json.loads( await start_data. __anext__ ()) # Extract Twilio-specific IDs stream_sid = call_data[ "start" ][ "streamSid" ] call_sid = call_data[ "start" ][ "callSid" ] # Create serializer with authentication for auto hang-up serializer = TwilioFrameSerializer( stream_sid = stream_sid, call_sid = call_sid, account_sid = os.getenv( "TWILIO_ACCOUNT_SID" ), auth_token = os.getenv( "TWILIO_AUTH_TOKEN" ), ) # Continue with transport and pipeline setup... See the Twilio Chatbot example for a complete implementation. PlivoFrameSerializer TelnyxFrameSerializer On this page Overview Features Installation Configuration Constructor Parameters InputParams Configuration Basic Usage Hang-up Functionality Server Code Example Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/serializers_twilio_8ee62046.txt b/serializers_twilio_8ee62046.txt
new file mode 100644
index 0000000000000000000000000000000000000000..ca47d7ccacd750d7b6f4470ade36df959eb0df1b
--- /dev/null
+++ b/serializers_twilio_8ee62046.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/serializers/twilio#param-stream-sid
+Title: TwilioFrameSerializer - Pipecat
+==================================================
+
+TwilioFrameSerializer - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Serializers TwilioFrameSerializer Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Frame Serializer Overview ExotelFrameSerializer PlivoFrameSerializer TwilioFrameSerializer TelnyxFrameSerializer Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview TwilioFrameSerializer enables integration with Twilio’s Media Streams WebSocket protocol, allowing your Pipecat application to handle phone calls via Twilio’s voice services. ​ Features Bidirectional audio conversion between Pipecat and Twilio DTMF (touch-tone) event handling Automatic call termination via Twilio’s REST API μ-law audio encoding/decoding ​ Installation The TwilioFrameSerializer does not require any additional dependencies beyond the core Pipecat library. ​ Configuration ​ Constructor Parameters ​ stream_sid str required The Twilio Media Stream SID ​ call_sid Optional[str] default: "None" The associated Twilio Call SID (required for auto hang-up) ​ account_sid Optional[str] default: "None" Twilio account SID (required for auto hang-up) ​ auth_token Optional[str] default: "None" Twilio auth token (required for auto hang-up) ​ params InputParams default: "InputParams()" Configuration parameters ​ InputParams Configuration ​ twilio_sample_rate int default: "8000" Sample rate used by Twilio (typically 8kHz) ​ sample_rate int | None default: "None" Optional override for pipeline input sample rate ​ auto_hang_up bool default: "True" Whether to automatically terminate call on EndFrame ​ Basic Usage Copy Ask AI from pipecat.serializers.twilio import TwilioFrameSerializer from pipecat.transports.network.fastapi_websocket import ( FastAPIWebsocketTransport, FastAPIWebsocketParams ) # Extract required values from Twilio WebSocket connection stream_sid = call_data[ "start" ][ "streamSid" ] call_sid = call_data[ "start" ][ "callSid" ] # Create serializer serializer = TwilioFrameSerializer( stream_sid = stream_sid, call_sid = call_sid, account_sid = "your_twilio_account_sid" , auth_token = "your_twilio_auth_token" ) # Use with FastAPIWebsocketTransport transport = FastAPIWebsocketTransport( websocket = websocket, params = FastAPIWebsocketParams( audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer(), serializer = serializer, ) ) ​ Hang-up Functionality When auto_hang_up is enabled, the serializer will automatically hang up the Twilio call when an EndFrame or CancelFrame is processed, using Twilio’s REST API: Copy Ask AI # Properly configured with hang-up support serializer = TwilioFrameSerializer( stream_sid = stream_sid, call_sid = call_sid, # Required for auto hang-up account_sid = os.getenv( "TWILIO_ACCOUNT_SID" ), # Required for auto hang-up auth_token = os.getenv( "TWILIO_AUTH_TOKEN" ), # Required for auto hang-up ) ​ Server Code Example Here’s a complete example of handling a Twilio WebSocket connection: Copy Ask AI from fastapi import FastAPI, WebSocket from pipecat.serializers.twilio import TwilioFrameSerializer import json import os app = FastAPI() @app.websocket ( "/ws" ) async def websocket_endpoint ( websocket : WebSocket): await websocket.accept() # Read initial messages from Twilio start_data = websocket.iter_text() await start_data. __anext__ () # Skip first message # Parse the second message to get call details call_data = json.loads( await start_data. __anext__ ()) # Extract Twilio-specific IDs stream_sid = call_data[ "start" ][ "streamSid" ] call_sid = call_data[ "start" ][ "callSid" ] # Create serializer with authentication for auto hang-up serializer = TwilioFrameSerializer( stream_sid = stream_sid, call_sid = call_sid, account_sid = os.getenv( "TWILIO_ACCOUNT_SID" ), auth_token = os.getenv( "TWILIO_AUTH_TOKEN" ), ) # Continue with transport and pipeline setup... See the Twilio Chatbot example for a complete implementation. PlivoFrameSerializer TelnyxFrameSerializer On this page Overview Features Installation Configuration Constructor Parameters InputParams Configuration Basic Usage Hang-up Functionality Server Code Example Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/serializers_twilio_8ffc60fc.txt b/serializers_twilio_8ffc60fc.txt
new file mode 100644
index 0000000000000000000000000000000000000000..8e291eefa2e0e837581a7051d845462883b6087b
--- /dev/null
+++ b/serializers_twilio_8ffc60fc.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/serializers/twilio#constructor-parameters
+Title: TwilioFrameSerializer - Pipecat
+==================================================
+
+TwilioFrameSerializer - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Serializers TwilioFrameSerializer Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Frame Serializer Overview ExotelFrameSerializer PlivoFrameSerializer TwilioFrameSerializer TelnyxFrameSerializer Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview TwilioFrameSerializer enables integration with Twilio’s Media Streams WebSocket protocol, allowing your Pipecat application to handle phone calls via Twilio’s voice services. ​ Features Bidirectional audio conversion between Pipecat and Twilio DTMF (touch-tone) event handling Automatic call termination via Twilio’s REST API μ-law audio encoding/decoding ​ Installation The TwilioFrameSerializer does not require any additional dependencies beyond the core Pipecat library. ​ Configuration ​ Constructor Parameters ​ stream_sid str required The Twilio Media Stream SID ​ call_sid Optional[str] default: "None" The associated Twilio Call SID (required for auto hang-up) ​ account_sid Optional[str] default: "None" Twilio account SID (required for auto hang-up) ​ auth_token Optional[str] default: "None" Twilio auth token (required for auto hang-up) ​ params InputParams default: "InputParams()" Configuration parameters ​ InputParams Configuration ​ twilio_sample_rate int default: "8000" Sample rate used by Twilio (typically 8kHz) ​ sample_rate int | None default: "None" Optional override for pipeline input sample rate ​ auto_hang_up bool default: "True" Whether to automatically terminate call on EndFrame ​ Basic Usage Copy Ask AI from pipecat.serializers.twilio import TwilioFrameSerializer from pipecat.transports.network.fastapi_websocket import ( FastAPIWebsocketTransport, FastAPIWebsocketParams ) # Extract required values from Twilio WebSocket connection stream_sid = call_data[ "start" ][ "streamSid" ] call_sid = call_data[ "start" ][ "callSid" ] # Create serializer serializer = TwilioFrameSerializer( stream_sid = stream_sid, call_sid = call_sid, account_sid = "your_twilio_account_sid" , auth_token = "your_twilio_auth_token" ) # Use with FastAPIWebsocketTransport transport = FastAPIWebsocketTransport( websocket = websocket, params = FastAPIWebsocketParams( audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer(), serializer = serializer, ) ) ​ Hang-up Functionality When auto_hang_up is enabled, the serializer will automatically hang up the Twilio call when an EndFrame or CancelFrame is processed, using Twilio’s REST API: Copy Ask AI # Properly configured with hang-up support serializer = TwilioFrameSerializer( stream_sid = stream_sid, call_sid = call_sid, # Required for auto hang-up account_sid = os.getenv( "TWILIO_ACCOUNT_SID" ), # Required for auto hang-up auth_token = os.getenv( "TWILIO_AUTH_TOKEN" ), # Required for auto hang-up ) ​ Server Code Example Here’s a complete example of handling a Twilio WebSocket connection: Copy Ask AI from fastapi import FastAPI, WebSocket from pipecat.serializers.twilio import TwilioFrameSerializer import json import os app = FastAPI() @app.websocket ( "/ws" ) async def websocket_endpoint ( websocket : WebSocket): await websocket.accept() # Read initial messages from Twilio start_data = websocket.iter_text() await start_data. __anext__ () # Skip first message # Parse the second message to get call details call_data = json.loads( await start_data. __anext__ ()) # Extract Twilio-specific IDs stream_sid = call_data[ "start" ][ "streamSid" ] call_sid = call_data[ "start" ][ "callSid" ] # Create serializer with authentication for auto hang-up serializer = TwilioFrameSerializer( stream_sid = stream_sid, call_sid = call_sid, account_sid = os.getenv( "TWILIO_ACCOUNT_SID" ), auth_token = os.getenv( "TWILIO_AUTH_TOKEN" ), ) # Continue with transport and pipeline setup... See the Twilio Chatbot example for a complete implementation. PlivoFrameSerializer TelnyxFrameSerializer On this page Overview Features Installation Configuration Constructor Parameters InputParams Configuration Basic Usage Hang-up Functionality Server Code Example Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/serializers_twilio_ac0fbf26.txt b/serializers_twilio_ac0fbf26.txt
new file mode 100644
index 0000000000000000000000000000000000000000..17b84a1e53bdfee618f4a3204b7e7fc2c99d74f2
--- /dev/null
+++ b/serializers_twilio_ac0fbf26.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/serializers/twilio#param-auth-token
+Title: TwilioFrameSerializer - Pipecat
+==================================================
+
+TwilioFrameSerializer - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Serializers TwilioFrameSerializer Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Frame Serializer Overview ExotelFrameSerializer PlivoFrameSerializer TwilioFrameSerializer TelnyxFrameSerializer Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview TwilioFrameSerializer enables integration with Twilio’s Media Streams WebSocket protocol, allowing your Pipecat application to handle phone calls via Twilio’s voice services. ​ Features Bidirectional audio conversion between Pipecat and Twilio DTMF (touch-tone) event handling Automatic call termination via Twilio’s REST API μ-law audio encoding/decoding ​ Installation The TwilioFrameSerializer does not require any additional dependencies beyond the core Pipecat library. ​ Configuration ​ Constructor Parameters ​ stream_sid str required The Twilio Media Stream SID ​ call_sid Optional[str] default: "None" The associated Twilio Call SID (required for auto hang-up) ​ account_sid Optional[str] default: "None" Twilio account SID (required for auto hang-up) ​ auth_token Optional[str] default: "None" Twilio auth token (required for auto hang-up) ​ params InputParams default: "InputParams()" Configuration parameters ​ InputParams Configuration ​ twilio_sample_rate int default: "8000" Sample rate used by Twilio (typically 8kHz) ​ sample_rate int | None default: "None" Optional override for pipeline input sample rate ​ auto_hang_up bool default: "True" Whether to automatically terminate call on EndFrame ​ Basic Usage Copy Ask AI from pipecat.serializers.twilio import TwilioFrameSerializer from pipecat.transports.network.fastapi_websocket import ( FastAPIWebsocketTransport, FastAPIWebsocketParams ) # Extract required values from Twilio WebSocket connection stream_sid = call_data[ "start" ][ "streamSid" ] call_sid = call_data[ "start" ][ "callSid" ] # Create serializer serializer = TwilioFrameSerializer( stream_sid = stream_sid, call_sid = call_sid, account_sid = "your_twilio_account_sid" , auth_token = "your_twilio_auth_token" ) # Use with FastAPIWebsocketTransport transport = FastAPIWebsocketTransport( websocket = websocket, params = FastAPIWebsocketParams( audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer(), serializer = serializer, ) ) ​ Hang-up Functionality When auto_hang_up is enabled, the serializer will automatically hang up the Twilio call when an EndFrame or CancelFrame is processed, using Twilio’s REST API: Copy Ask AI # Properly configured with hang-up support serializer = TwilioFrameSerializer( stream_sid = stream_sid, call_sid = call_sid, # Required for auto hang-up account_sid = os.getenv( "TWILIO_ACCOUNT_SID" ), # Required for auto hang-up auth_token = os.getenv( "TWILIO_AUTH_TOKEN" ), # Required for auto hang-up ) ​ Server Code Example Here’s a complete example of handling a Twilio WebSocket connection: Copy Ask AI from fastapi import FastAPI, WebSocket from pipecat.serializers.twilio import TwilioFrameSerializer import json import os app = FastAPI() @app.websocket ( "/ws" ) async def websocket_endpoint ( websocket : WebSocket): await websocket.accept() # Read initial messages from Twilio start_data = websocket.iter_text() await start_data. __anext__ () # Skip first message # Parse the second message to get call details call_data = json.loads( await start_data. __anext__ ()) # Extract Twilio-specific IDs stream_sid = call_data[ "start" ][ "streamSid" ] call_sid = call_data[ "start" ][ "callSid" ] # Create serializer with authentication for auto hang-up serializer = TwilioFrameSerializer( stream_sid = stream_sid, call_sid = call_sid, account_sid = os.getenv( "TWILIO_ACCOUNT_SID" ), auth_token = os.getenv( "TWILIO_AUTH_TOKEN" ), ) # Continue with transport and pipeline setup... See the Twilio Chatbot example for a complete implementation. PlivoFrameSerializer TelnyxFrameSerializer On this page Overview Features Installation Configuration Constructor Parameters InputParams Configuration Basic Usage Hang-up Functionality Server Code Example Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/serializers_twilio_b0d0c9f6.txt b/serializers_twilio_b0d0c9f6.txt
new file mode 100644
index 0000000000000000000000000000000000000000..c6e4c44cfa3dec7b81e7c6507a8b62be12173370
--- /dev/null
+++ b/serializers_twilio_b0d0c9f6.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/serializers/twilio
+Title: TwilioFrameSerializer - Pipecat
+==================================================
+
+TwilioFrameSerializer - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Serializers TwilioFrameSerializer Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Frame Serializer Overview ExotelFrameSerializer PlivoFrameSerializer TwilioFrameSerializer TelnyxFrameSerializer Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview TwilioFrameSerializer enables integration with Twilio’s Media Streams WebSocket protocol, allowing your Pipecat application to handle phone calls via Twilio’s voice services. ​ Features Bidirectional audio conversion between Pipecat and Twilio DTMF (touch-tone) event handling Automatic call termination via Twilio’s REST API μ-law audio encoding/decoding ​ Installation The TwilioFrameSerializer does not require any additional dependencies beyond the core Pipecat library. ​ Configuration ​ Constructor Parameters ​ stream_sid str required The Twilio Media Stream SID ​ call_sid Optional[str] default: "None" The associated Twilio Call SID (required for auto hang-up) ​ account_sid Optional[str] default: "None" Twilio account SID (required for auto hang-up) ​ auth_token Optional[str] default: "None" Twilio auth token (required for auto hang-up) ​ params InputParams default: "InputParams()" Configuration parameters ​ InputParams Configuration ​ twilio_sample_rate int default: "8000" Sample rate used by Twilio (typically 8kHz) ​ sample_rate int | None default: "None" Optional override for pipeline input sample rate ​ auto_hang_up bool default: "True" Whether to automatically terminate call on EndFrame ​ Basic Usage Copy Ask AI from pipecat.serializers.twilio import TwilioFrameSerializer from pipecat.transports.network.fastapi_websocket import ( FastAPIWebsocketTransport, FastAPIWebsocketParams ) # Extract required values from Twilio WebSocket connection stream_sid = call_data[ "start" ][ "streamSid" ] call_sid = call_data[ "start" ][ "callSid" ] # Create serializer serializer = TwilioFrameSerializer( stream_sid = stream_sid, call_sid = call_sid, account_sid = "your_twilio_account_sid" , auth_token = "your_twilio_auth_token" ) # Use with FastAPIWebsocketTransport transport = FastAPIWebsocketTransport( websocket = websocket, params = FastAPIWebsocketParams( audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer(), serializer = serializer, ) ) ​ Hang-up Functionality When auto_hang_up is enabled, the serializer will automatically hang up the Twilio call when an EndFrame or CancelFrame is processed, using Twilio’s REST API: Copy Ask AI # Properly configured with hang-up support serializer = TwilioFrameSerializer( stream_sid = stream_sid, call_sid = call_sid, # Required for auto hang-up account_sid = os.getenv( "TWILIO_ACCOUNT_SID" ), # Required for auto hang-up auth_token = os.getenv( "TWILIO_AUTH_TOKEN" ), # Required for auto hang-up ) ​ Server Code Example Here’s a complete example of handling a Twilio WebSocket connection: Copy Ask AI from fastapi import FastAPI, WebSocket from pipecat.serializers.twilio import TwilioFrameSerializer import json import os app = FastAPI() @app.websocket ( "/ws" ) async def websocket_endpoint ( websocket : WebSocket): await websocket.accept() # Read initial messages from Twilio start_data = websocket.iter_text() await start_data. __anext__ () # Skip first message # Parse the second message to get call details call_data = json.loads( await start_data. __anext__ ()) # Extract Twilio-specific IDs stream_sid = call_data[ "start" ][ "streamSid" ] call_sid = call_data[ "start" ][ "callSid" ] # Create serializer with authentication for auto hang-up serializer = TwilioFrameSerializer( stream_sid = stream_sid, call_sid = call_sid, account_sid = os.getenv( "TWILIO_ACCOUNT_SID" ), auth_token = os.getenv( "TWILIO_AUTH_TOKEN" ), ) # Continue with transport and pipeline setup... See the Twilio Chatbot example for a complete implementation. PlivoFrameSerializer TelnyxFrameSerializer On this page Overview Features Installation Configuration Constructor Parameters InputParams Configuration Basic Usage Hang-up Functionality Server Code Example Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/serializers_twilio_edccd8d6.txt b/serializers_twilio_edccd8d6.txt
new file mode 100644
index 0000000000000000000000000000000000000000..8b02045586e6ff33ec671b25d7868e18b1875385
--- /dev/null
+++ b/serializers_twilio_edccd8d6.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/serializers/twilio#installation
+Title: TwilioFrameSerializer - Pipecat
+==================================================
+
+TwilioFrameSerializer - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Serializers TwilioFrameSerializer Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Frame Serializer Overview ExotelFrameSerializer PlivoFrameSerializer TwilioFrameSerializer TelnyxFrameSerializer Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview TwilioFrameSerializer enables integration with Twilio’s Media Streams WebSocket protocol, allowing your Pipecat application to handle phone calls via Twilio’s voice services. ​ Features Bidirectional audio conversion between Pipecat and Twilio DTMF (touch-tone) event handling Automatic call termination via Twilio’s REST API μ-law audio encoding/decoding ​ Installation The TwilioFrameSerializer does not require any additional dependencies beyond the core Pipecat library. ​ Configuration ​ Constructor Parameters ​ stream_sid str required The Twilio Media Stream SID ​ call_sid Optional[str] default: "None" The associated Twilio Call SID (required for auto hang-up) ​ account_sid Optional[str] default: "None" Twilio account SID (required for auto hang-up) ​ auth_token Optional[str] default: "None" Twilio auth token (required for auto hang-up) ​ params InputParams default: "InputParams()" Configuration parameters ​ InputParams Configuration ​ twilio_sample_rate int default: "8000" Sample rate used by Twilio (typically 8kHz) ​ sample_rate int | None default: "None" Optional override for pipeline input sample rate ​ auto_hang_up bool default: "True" Whether to automatically terminate call on EndFrame ​ Basic Usage Copy Ask AI from pipecat.serializers.twilio import TwilioFrameSerializer from pipecat.transports.network.fastapi_websocket import ( FastAPIWebsocketTransport, FastAPIWebsocketParams ) # Extract required values from Twilio WebSocket connection stream_sid = call_data[ "start" ][ "streamSid" ] call_sid = call_data[ "start" ][ "callSid" ] # Create serializer serializer = TwilioFrameSerializer( stream_sid = stream_sid, call_sid = call_sid, account_sid = "your_twilio_account_sid" , auth_token = "your_twilio_auth_token" ) # Use with FastAPIWebsocketTransport transport = FastAPIWebsocketTransport( websocket = websocket, params = FastAPIWebsocketParams( audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer(), serializer = serializer, ) ) ​ Hang-up Functionality When auto_hang_up is enabled, the serializer will automatically hang up the Twilio call when an EndFrame or CancelFrame is processed, using Twilio’s REST API: Copy Ask AI # Properly configured with hang-up support serializer = TwilioFrameSerializer( stream_sid = stream_sid, call_sid = call_sid, # Required for auto hang-up account_sid = os.getenv( "TWILIO_ACCOUNT_SID" ), # Required for auto hang-up auth_token = os.getenv( "TWILIO_AUTH_TOKEN" ), # Required for auto hang-up ) ​ Server Code Example Here’s a complete example of handling a Twilio WebSocket connection: Copy Ask AI from fastapi import FastAPI, WebSocket from pipecat.serializers.twilio import TwilioFrameSerializer import json import os app = FastAPI() @app.websocket ( "/ws" ) async def websocket_endpoint ( websocket : WebSocket): await websocket.accept() # Read initial messages from Twilio start_data = websocket.iter_text() await start_data. __anext__ () # Skip first message # Parse the second message to get call details call_data = json.loads( await start_data. __anext__ ()) # Extract Twilio-specific IDs stream_sid = call_data[ "start" ][ "streamSid" ] call_sid = call_data[ "start" ][ "callSid" ] # Create serializer with authentication for auto hang-up serializer = TwilioFrameSerializer( stream_sid = stream_sid, call_sid = call_sid, account_sid = os.getenv( "TWILIO_ACCOUNT_SID" ), auth_token = os.getenv( "TWILIO_AUTH_TOKEN" ), ) # Continue with transport and pipeline setup... See the Twilio Chatbot example for a complete implementation. PlivoFrameSerializer TelnyxFrameSerializer On this page Overview Features Installation Configuration Constructor Parameters InputParams Configuration Basic Usage Hang-up Functionality Server Code Example Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/server_frameworks_19c8cd59.txt b/server_frameworks_19c8cd59.txt
new file mode 100644
index 0000000000000000000000000000000000000000..af08e0a75aa3e92660fdfcceb9356a860eabc87d
--- /dev/null
+++ b/server_frameworks_19c8cd59.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/frameworks#client-integration
+Title: RTVI (Real-Time Voice Interaction) - Pipecat
+==================================================
+
+RTVI (Real-Time Voice Interaction) - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation RTVI RTVI (Real-Time Voice Interaction) Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Overview RTVIProcessor RTVI Observer Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline Pipecat’s RTVI (Real-Time Voice Interaction) protocol provides a standardized communication layer between clients and servers for building real-time voice and multimodal applications. It handles the synchronization of user and bot interactions, transcriptions, LLM processing, and text-to-speech delivery. Speaking States Track when users and bots start/stop speaking for natural turn-taking Transcription Handle real-time transcriptions from both users and bots LLM Processing Manage LLM responses and function calls with proper client notifications TTS Management Control text-to-speech state and audio delivery ​ Architecture RTVI operates with two primary components: RTVIProcessor - A frame processor residing in the pipeline that serves as the entry point for sending and receiving messages to/from the client. RTVIObserver - An observer that monitors pipeline events and translates them into client-compatible messages, handling: Speaking state changes Transcription updates LLM responses TTS events Performance metrics ​ Basic Example Here’s how to set up RTVI in your Pipecat application: Copy Ask AI from pipecat.processors.frameworks.rtvi import RTVIConfig, RTVIObserver, RTVIProcessor # Create the RTVI processor rtvi = RTVIProcessor( config = RTVIConfig( config = [])) # Include the RTVIProcessor in your pipeline pipeline = Pipeline( [ transport.input(), rtvi, stt, context_aggregator.user(), llm, tts, transport.output(), context_aggregator.assistant(), ] ) # Add the RTVIObserver to your pipeline task task = PipelineTask( pipeline, params = PipelineParams( allow_interruptions = True , enable_metrics = True , ), observers = [RTVIObserver(rtvi)], ) # Handle client connection @rtvi.event_handler ( "on_client_ready" ) async def on_client_ready ( rtvi ): # Signal bot is ready to receive messages await rtvi.set_bot_ready() # Initialize the conversation await task.queue_frames([context_aggregator.user().get_context_frame()]) # Handle participant disconnection @transport.event_handler ( "on_participant_left" ) async def on_participant_left ( transport , participant , reason ): await task.cancel() # Run the pipeline runner = PipelineRunner() await runner.run(task) ​ Protocol Flow Client connects and sends a client-ready message Server responds with bot-ready and initial configuration Client and server exchange real-time events: Speaking state changes ( user/bot-started/stopped-speaking ) Transcriptions ( user/bot-transcription ) LLM processing ( bot-llm-text , llm-function-call ) TTS events ( bot-tts-text , bot-tts-audio ) ​ Key Components RTVIProcessor Configure and manage RTVI services, actions, and client communication RTVIObserver Translate internal pipeline events to standardized client messages ​ Client Integration RTVI is implemented in Pipecat client SDKs, providing a high-level API to interact with the protocol. Visit the Pipecat Client SDKs documentation: Client SDKs Learn how to implement RTVI on the client-side with our JavaScript, React, and mobile SDKs Interruption Strategies RTVIProcessor On this page Architecture Basic Example Protocol Flow Key Components Client Integration Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/server_frameworks_c4262d62.txt b/server_frameworks_c4262d62.txt
new file mode 100644
index 0000000000000000000000000000000000000000..a150534fc87be4f74e59c7cfdd87d1441bed7eb8
--- /dev/null
+++ b/server_frameworks_c4262d62.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/frameworks#architecture
+Title: RTVI (Real-Time Voice Interaction) - Pipecat
+==================================================
+
+RTVI (Real-Time Voice Interaction) - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation RTVI RTVI (Real-Time Voice Interaction) Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Overview RTVIProcessor RTVI Observer Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline Pipecat’s RTVI (Real-Time Voice Interaction) protocol provides a standardized communication layer between clients and servers for building real-time voice and multimodal applications. It handles the synchronization of user and bot interactions, transcriptions, LLM processing, and text-to-speech delivery. Speaking States Track when users and bots start/stop speaking for natural turn-taking Transcription Handle real-time transcriptions from both users and bots LLM Processing Manage LLM responses and function calls with proper client notifications TTS Management Control text-to-speech state and audio delivery ​ Architecture RTVI operates with two primary components: RTVIProcessor - A frame processor residing in the pipeline that serves as the entry point for sending and receiving messages to/from the client. RTVIObserver - An observer that monitors pipeline events and translates them into client-compatible messages, handling: Speaking state changes Transcription updates LLM responses TTS events Performance metrics ​ Basic Example Here’s how to set up RTVI in your Pipecat application: Copy Ask AI from pipecat.processors.frameworks.rtvi import RTVIConfig, RTVIObserver, RTVIProcessor # Create the RTVI processor rtvi = RTVIProcessor( config = RTVIConfig( config = [])) # Include the RTVIProcessor in your pipeline pipeline = Pipeline( [ transport.input(), rtvi, stt, context_aggregator.user(), llm, tts, transport.output(), context_aggregator.assistant(), ] ) # Add the RTVIObserver to your pipeline task task = PipelineTask( pipeline, params = PipelineParams( allow_interruptions = True , enable_metrics = True , ), observers = [RTVIObserver(rtvi)], ) # Handle client connection @rtvi.event_handler ( "on_client_ready" ) async def on_client_ready ( rtvi ): # Signal bot is ready to receive messages await rtvi.set_bot_ready() # Initialize the conversation await task.queue_frames([context_aggregator.user().get_context_frame()]) # Handle participant disconnection @transport.event_handler ( "on_participant_left" ) async def on_participant_left ( transport , participant , reason ): await task.cancel() # Run the pipeline runner = PipelineRunner() await runner.run(task) ​ Protocol Flow Client connects and sends a client-ready message Server responds with bot-ready and initial configuration Client and server exchange real-time events: Speaking state changes ( user/bot-started/stopped-speaking ) Transcriptions ( user/bot-transcription ) LLM processing ( bot-llm-text , llm-function-call ) TTS events ( bot-tts-text , bot-tts-audio ) ​ Key Components RTVIProcessor Configure and manage RTVI services, actions, and client communication RTVIObserver Translate internal pipeline events to standardized client messages ​ Client Integration RTVI is implemented in Pipecat client SDKs, providing a high-level API to interact with the protocol. Visit the Pipecat Client SDKs documentation: Client SDKs Learn how to implement RTVI on the client-side with our JavaScript, React, and mobile SDKs Interruption Strategies RTVIProcessor On this page Architecture Basic Example Protocol Flow Key Components Client Integration Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/server_frameworks_c821fa55.txt b/server_frameworks_c821fa55.txt
new file mode 100644
index 0000000000000000000000000000000000000000..c10c03a5a9057cd32523c73b41870aae51e55537
--- /dev/null
+++ b/server_frameworks_c821fa55.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/frameworks#protocol-flow
+Title: RTVI (Real-Time Voice Interaction) - Pipecat
+==================================================
+
+RTVI (Real-Time Voice Interaction) - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation RTVI RTVI (Real-Time Voice Interaction) Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Overview RTVIProcessor RTVI Observer Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline Pipecat’s RTVI (Real-Time Voice Interaction) protocol provides a standardized communication layer between clients and servers for building real-time voice and multimodal applications. It handles the synchronization of user and bot interactions, transcriptions, LLM processing, and text-to-speech delivery. Speaking States Track when users and bots start/stop speaking for natural turn-taking Transcription Handle real-time transcriptions from both users and bots LLM Processing Manage LLM responses and function calls with proper client notifications TTS Management Control text-to-speech state and audio delivery ​ Architecture RTVI operates with two primary components: RTVIProcessor - A frame processor residing in the pipeline that serves as the entry point for sending and receiving messages to/from the client. RTVIObserver - An observer that monitors pipeline events and translates them into client-compatible messages, handling: Speaking state changes Transcription updates LLM responses TTS events Performance metrics ​ Basic Example Here’s how to set up RTVI in your Pipecat application: Copy Ask AI from pipecat.processors.frameworks.rtvi import RTVIConfig, RTVIObserver, RTVIProcessor # Create the RTVI processor rtvi = RTVIProcessor( config = RTVIConfig( config = [])) # Include the RTVIProcessor in your pipeline pipeline = Pipeline( [ transport.input(), rtvi, stt, context_aggregator.user(), llm, tts, transport.output(), context_aggregator.assistant(), ] ) # Add the RTVIObserver to your pipeline task task = PipelineTask( pipeline, params = PipelineParams( allow_interruptions = True , enable_metrics = True , ), observers = [RTVIObserver(rtvi)], ) # Handle client connection @rtvi.event_handler ( "on_client_ready" ) async def on_client_ready ( rtvi ): # Signal bot is ready to receive messages await rtvi.set_bot_ready() # Initialize the conversation await task.queue_frames([context_aggregator.user().get_context_frame()]) # Handle participant disconnection @transport.event_handler ( "on_participant_left" ) async def on_participant_left ( transport , participant , reason ): await task.cancel() # Run the pipeline runner = PipelineRunner() await runner.run(task) ​ Protocol Flow Client connects and sends a client-ready message Server responds with bot-ready and initial configuration Client and server exchange real-time events: Speaking state changes ( user/bot-started/stopped-speaking ) Transcriptions ( user/bot-transcription ) LLM processing ( bot-llm-text , llm-function-call ) TTS events ( bot-tts-text , bot-tts-audio ) ​ Key Components RTVIProcessor Configure and manage RTVI services, actions, and client communication RTVIObserver Translate internal pipeline events to standardized client messages ​ Client Integration RTVI is implemented in Pipecat client SDKs, providing a high-level API to interact with the protocol. Visit the Pipecat Client SDKs documentation: Client SDKs Learn how to implement RTVI on the client-side with our JavaScript, React, and mobile SDKs Interruption Strategies RTVIProcessor On this page Architecture Basic Example Protocol Flow Key Components Client Integration Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/server_introduction_84e97b53.txt b/server_introduction_84e97b53.txt
new file mode 100644
index 0000000000000000000000000000000000000000..b5df238e7fba610af1825e3e9e0d8914fd917671
--- /dev/null
+++ b/server_introduction_84e97b53.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/introduction#example-usage
+Title: Server API Reference - Pipecat
+==================================================
+
+Server API Reference - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Server API Reference Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview Pipecat’s server-side functionality is organized into three main categories: Services AI service integrations for speech, language, vision, and more Frameworks User RTVI for client/server communication or Pipecat Flows for building structured conversations Utilities Helper functions and tools for audio, filtering, and flows ​ Service Categories Pipecat integrates with various AI services across different categories: Transport WebRTC and WebSocket implementations for real-time communication Speech Processing Speech-to-text, text-to-speech, and speech-to-speech services Language Models Integration with various LLM providers Vision & Media Image generation, video processing, and computer vision ​ Getting Started Browse our Supported Services to see available integrations Install required dependencies for your chosen services Reference individual service docs for detailed configuration options ​ Example Usage Copy Ask AI from pipecat.audio.vad.silero import SileroVADAnalyzer from pipecat.pipeline import Pipeline from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext from pipecat.services.cartesia.tts import CartesiaTTSService from pipecat.services.deepgram.stt import DeepgramSTTService from pipecat.services.openai.llm import OpenAILLMService from pipecat.transports.services.daily import DailyParams, DailyTransport transport = DailyTransport( room_url, token, "Respond bot" , DailyParams( audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer(), ), ) # Configure services stt = DeepgramSTTService( api_key = KEY ) llm = OpenAILLMService( api_key = KEY , model = "gpt-4o" ) tts = CartesiaTTSService( api_key = KEY , voice_id = ID ) # Create context and aggregators context = OpenAILLMContext( messages = [{ "role" : "system" , "content" : "You are a helpful assistant." }] ) context_aggregator = llm.create_context_aggregator(context) # Create pipeline pipeline = Pipeline([ transport.input(), stt, context_aggregator.user(), llm, tts, transport.output(), context_aggregator,assistant() ]) Reference docs On this page Overview Service Categories Getting Started Example Usage Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/server_introduction_a53203e6.txt b/server_introduction_a53203e6.txt
new file mode 100644
index 0000000000000000000000000000000000000000..9add58b70ce38a43ac632f56e69784c33c31e49a
--- /dev/null
+++ b/server_introduction_a53203e6.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/introduction#getting-started
+Title: Server API Reference - Pipecat
+==================================================
+
+Server API Reference - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Server API Reference Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview Pipecat’s server-side functionality is organized into three main categories: Services AI service integrations for speech, language, vision, and more Frameworks User RTVI for client/server communication or Pipecat Flows for building structured conversations Utilities Helper functions and tools for audio, filtering, and flows ​ Service Categories Pipecat integrates with various AI services across different categories: Transport WebRTC and WebSocket implementations for real-time communication Speech Processing Speech-to-text, text-to-speech, and speech-to-speech services Language Models Integration with various LLM providers Vision & Media Image generation, video processing, and computer vision ​ Getting Started Browse our Supported Services to see available integrations Install required dependencies for your chosen services Reference individual service docs for detailed configuration options ​ Example Usage Copy Ask AI from pipecat.audio.vad.silero import SileroVADAnalyzer from pipecat.pipeline import Pipeline from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext from pipecat.services.cartesia.tts import CartesiaTTSService from pipecat.services.deepgram.stt import DeepgramSTTService from pipecat.services.openai.llm import OpenAILLMService from pipecat.transports.services.daily import DailyParams, DailyTransport transport = DailyTransport( room_url, token, "Respond bot" , DailyParams( audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer(), ), ) # Configure services stt = DeepgramSTTService( api_key = KEY ) llm = OpenAILLMService( api_key = KEY , model = "gpt-4o" ) tts = CartesiaTTSService( api_key = KEY , voice_id = ID ) # Create context and aggregators context = OpenAILLMContext( messages = [{ "role" : "system" , "content" : "You are a helpful assistant." }] ) context_aggregator = llm.create_context_aggregator(context) # Create pipeline pipeline = Pipeline([ transport.input(), stt, context_aggregator.user(), llm, tts, transport.output(), context_aggregator,assistant() ]) Reference docs On this page Overview Service Categories Getting Started Example Usage Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/server_introduction_c3d30d59.txt b/server_introduction_c3d30d59.txt
new file mode 100644
index 0000000000000000000000000000000000000000..e52329ae49be0a0c7534c96763d57ac944d438ae
--- /dev/null
+++ b/server_introduction_c3d30d59.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/introduction#overview
+Title: Server API Reference - Pipecat
+==================================================
+
+Server API Reference - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Server API Reference Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview Pipecat’s server-side functionality is organized into three main categories: Services AI service integrations for speech, language, vision, and more Frameworks User RTVI for client/server communication or Pipecat Flows for building structured conversations Utilities Helper functions and tools for audio, filtering, and flows ​ Service Categories Pipecat integrates with various AI services across different categories: Transport WebRTC and WebSocket implementations for real-time communication Speech Processing Speech-to-text, text-to-speech, and speech-to-speech services Language Models Integration with various LLM providers Vision & Media Image generation, video processing, and computer vision ​ Getting Started Browse our Supported Services to see available integrations Install required dependencies for your chosen services Reference individual service docs for detailed configuration options ​ Example Usage Copy Ask AI from pipecat.audio.vad.silero import SileroVADAnalyzer from pipecat.pipeline import Pipeline from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext from pipecat.services.cartesia.tts import CartesiaTTSService from pipecat.services.deepgram.stt import DeepgramSTTService from pipecat.services.openai.llm import OpenAILLMService from pipecat.transports.services.daily import DailyParams, DailyTransport transport = DailyTransport( room_url, token, "Respond bot" , DailyParams( audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer(), ), ) # Configure services stt = DeepgramSTTService( api_key = KEY ) llm = OpenAILLMService( api_key = KEY , model = "gpt-4o" ) tts = CartesiaTTSService( api_key = KEY , voice_id = ID ) # Create context and aggregators context = OpenAILLMContext( messages = [{ "role" : "system" , "content" : "You are a helpful assistant." }] ) context_aggregator = llm.create_context_aggregator(context) # Create pipeline pipeline = Pipeline([ transport.input(), stt, context_aggregator.user(), llm, tts, transport.output(), context_aggregator,assistant() ]) Reference docs On this page Overview Service Categories Getting Started Example Usage Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/server_utilities_0a5d7ed1.txt b/server_utilities_0a5d7ed1.txt
new file mode 100644
index 0000000000000000000000000000000000000000..cb749c9de6e863c1100925dedec0aee19dd03e22
--- /dev/null
+++ b/server_utilities_0a5d7ed1.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities#constructor-parameters
+Title: Producer & Consumer Processors - Pipecat
+==================================================
+
+Producer & Consumer Processors - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Advanced Frame Processors Producer & Consumer Processors Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Producer & Consumer Processors UserIdleProcessor Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview The Producer and Consumer processors work as a pair to route frames between different parts of a pipeline, particularly useful when working with ParallelPipeline . They allow you to selectively capture frames from one pipeline branch and inject them into another. ​ ProducerProcessor ProducerProcessor examines frames flowing through the pipeline, applies a filter to decide which frames to share, and optionally transforms these frames before sending them to connected consumers. ​ Constructor Parameters ​ filter Callable[[Frame], Awaitable[bool]] required An async function that determines which frames should be sent to consumers. Should return True for frames to be shared. ​ transformer Callable[[Frame], Awaitable[Frame]] default: "identity_transformer" Optional async function that transforms frames before sending to consumers. By default, passes frames unchanged. ​ passthrough bool default: "True" When True , passes all frames through the normal pipeline flow. When False , only passes through frames that don’t match the filter. ​ ConsumerProcessor ConsumerProcessor receives frames from a ProducerProcessor and injects them into its pipeline branch. ​ Constructor Parameters ​ producer ProducerProcessor required The producer processor that will send frames to this consumer. ​ transformer Callable[[Frame], Awaitable[Frame]] default: "identity_transformer" Optional async function that transforms frames before injecting them into the pipeline. ​ direction FrameDirection default: "FrameDirection.DOWNSTREAM" The direction in which to push received frames. Usually DOWNSTREAM to send frames forward in the pipeline. ​ Usage Examples ​ Basic Usage: Moving TTS Audio Between Branches Copy Ask AI # Create a producer that captures TTS audio frames async def is_tts_audio ( frame : Frame) -> bool : return isinstance (frame, TTSAudioRawFrame) # Define an async transformer function async def tts_to_input_audio_transformer ( frame : Frame) -> Frame: if isinstance (frame, TTSAudioRawFrame): # Convert TTS audio to input audio format return InputAudioRawFrame( audio = frame.audio, sample_rate = frame.sample_rate, num_channels = frame.num_channels ) return frame producer = ProducerProcessor( filter = is_tts_audio, transformer = tts_to_input_audio_transformer passthrough = True # Keep these frames in original pipeline ) # Create a consumer to receive the frames consumer = ConsumerProcessor( producer = producer, direction = FrameDirection. DOWNSTREAM ) # Use in a ParallelPipeline pipeline = Pipeline([ transport.input(), ParallelPipeline( # Branch 1: LLM for bot responses [ llm, tts, producer, # Capture TTS audio here ], # Branch 2: Audio processing branch [ consumer, # Receive TTS audio here llm, # Speech-to-Speech LLM (audio in) ] ), transport.output(), ]) Sentry Metrics UserIdleProcessor On this page Overview ProducerProcessor Constructor Parameters ConsumerProcessor Constructor Parameters Usage Examples Basic Usage: Moving TTS Audio Between Branches Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/server_utilities_489c8b9f.txt b/server_utilities_489c8b9f.txt
new file mode 100644
index 0000000000000000000000000000000000000000..87bc9b182f6bd99802939cec5480785daee55304
--- /dev/null
+++ b/server_utilities_489c8b9f.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities#consumerprocessor
+Title: Producer & Consumer Processors - Pipecat
+==================================================
+
+Producer & Consumer Processors - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Advanced Frame Processors Producer & Consumer Processors Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Producer & Consumer Processors UserIdleProcessor Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview The Producer and Consumer processors work as a pair to route frames between different parts of a pipeline, particularly useful when working with ParallelPipeline . They allow you to selectively capture frames from one pipeline branch and inject them into another. ​ ProducerProcessor ProducerProcessor examines frames flowing through the pipeline, applies a filter to decide which frames to share, and optionally transforms these frames before sending them to connected consumers. ​ Constructor Parameters ​ filter Callable[[Frame], Awaitable[bool]] required An async function that determines which frames should be sent to consumers. Should return True for frames to be shared. ​ transformer Callable[[Frame], Awaitable[Frame]] default: "identity_transformer" Optional async function that transforms frames before sending to consumers. By default, passes frames unchanged. ​ passthrough bool default: "True" When True , passes all frames through the normal pipeline flow. When False , only passes through frames that don’t match the filter. ​ ConsumerProcessor ConsumerProcessor receives frames from a ProducerProcessor and injects them into its pipeline branch. ​ Constructor Parameters ​ producer ProducerProcessor required The producer processor that will send frames to this consumer. ​ transformer Callable[[Frame], Awaitable[Frame]] default: "identity_transformer" Optional async function that transforms frames before injecting them into the pipeline. ​ direction FrameDirection default: "FrameDirection.DOWNSTREAM" The direction in which to push received frames. Usually DOWNSTREAM to send frames forward in the pipeline. ​ Usage Examples ​ Basic Usage: Moving TTS Audio Between Branches Copy Ask AI # Create a producer that captures TTS audio frames async def is_tts_audio ( frame : Frame) -> bool : return isinstance (frame, TTSAudioRawFrame) # Define an async transformer function async def tts_to_input_audio_transformer ( frame : Frame) -> Frame: if isinstance (frame, TTSAudioRawFrame): # Convert TTS audio to input audio format return InputAudioRawFrame( audio = frame.audio, sample_rate = frame.sample_rate, num_channels = frame.num_channels ) return frame producer = ProducerProcessor( filter = is_tts_audio, transformer = tts_to_input_audio_transformer passthrough = True # Keep these frames in original pipeline ) # Create a consumer to receive the frames consumer = ConsumerProcessor( producer = producer, direction = FrameDirection. DOWNSTREAM ) # Use in a ParallelPipeline pipeline = Pipeline([ transport.input(), ParallelPipeline( # Branch 1: LLM for bot responses [ llm, tts, producer, # Capture TTS audio here ], # Branch 2: Audio processing branch [ consumer, # Receive TTS audio here llm, # Speech-to-Speech LLM (audio in) ] ), transport.output(), ]) Sentry Metrics UserIdleProcessor On this page Overview ProducerProcessor Constructor Parameters ConsumerProcessor Constructor Parameters Usage Examples Basic Usage: Moving TTS Audio Between Branches Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/server_utilities_5b34307b.txt b/server_utilities_5b34307b.txt
new file mode 100644
index 0000000000000000000000000000000000000000..54cba301bdc7e29373c87962e384416b52d2580f
--- /dev/null
+++ b/server_utilities_5b34307b.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities#usage-examples
+Title: Producer & Consumer Processors - Pipecat
+==================================================
+
+Producer & Consumer Processors - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Advanced Frame Processors Producer & Consumer Processors Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Producer & Consumer Processors UserIdleProcessor Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview The Producer and Consumer processors work as a pair to route frames between different parts of a pipeline, particularly useful when working with ParallelPipeline . They allow you to selectively capture frames from one pipeline branch and inject them into another. ​ ProducerProcessor ProducerProcessor examines frames flowing through the pipeline, applies a filter to decide which frames to share, and optionally transforms these frames before sending them to connected consumers. ​ Constructor Parameters ​ filter Callable[[Frame], Awaitable[bool]] required An async function that determines which frames should be sent to consumers. Should return True for frames to be shared. ​ transformer Callable[[Frame], Awaitable[Frame]] default: "identity_transformer" Optional async function that transforms frames before sending to consumers. By default, passes frames unchanged. ​ passthrough bool default: "True" When True , passes all frames through the normal pipeline flow. When False , only passes through frames that don’t match the filter. ​ ConsumerProcessor ConsumerProcessor receives frames from a ProducerProcessor and injects them into its pipeline branch. ​ Constructor Parameters ​ producer ProducerProcessor required The producer processor that will send frames to this consumer. ​ transformer Callable[[Frame], Awaitable[Frame]] default: "identity_transformer" Optional async function that transforms frames before injecting them into the pipeline. ​ direction FrameDirection default: "FrameDirection.DOWNSTREAM" The direction in which to push received frames. Usually DOWNSTREAM to send frames forward in the pipeline. ​ Usage Examples ​ Basic Usage: Moving TTS Audio Between Branches Copy Ask AI # Create a producer that captures TTS audio frames async def is_tts_audio ( frame : Frame) -> bool : return isinstance (frame, TTSAudioRawFrame) # Define an async transformer function async def tts_to_input_audio_transformer ( frame : Frame) -> Frame: if isinstance (frame, TTSAudioRawFrame): # Convert TTS audio to input audio format return InputAudioRawFrame( audio = frame.audio, sample_rate = frame.sample_rate, num_channels = frame.num_channels ) return frame producer = ProducerProcessor( filter = is_tts_audio, transformer = tts_to_input_audio_transformer passthrough = True # Keep these frames in original pipeline ) # Create a consumer to receive the frames consumer = ConsumerProcessor( producer = producer, direction = FrameDirection. DOWNSTREAM ) # Use in a ParallelPipeline pipeline = Pipeline([ transport.input(), ParallelPipeline( # Branch 1: LLM for bot responses [ llm, tts, producer, # Capture TTS audio here ], # Branch 2: Audio processing branch [ consumer, # Receive TTS audio here llm, # Speech-to-Speech LLM (audio in) ] ), transport.output(), ]) Sentry Metrics UserIdleProcessor On this page Overview ProducerProcessor Constructor Parameters ConsumerProcessor Constructor Parameters Usage Examples Basic Usage: Moving TTS Audio Between Branches Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/server_utilities_5dfa75a0.txt b/server_utilities_5dfa75a0.txt
new file mode 100644
index 0000000000000000000000000000000000000000..cb274afa1911a1cfb49065ec9c9f9144912b10ce
--- /dev/null
+++ b/server_utilities_5dfa75a0.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities#constructor-parameters-2
+Title: Producer & Consumer Processors - Pipecat
+==================================================
+
+Producer & Consumer Processors - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Advanced Frame Processors Producer & Consumer Processors Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Producer & Consumer Processors UserIdleProcessor Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview The Producer and Consumer processors work as a pair to route frames between different parts of a pipeline, particularly useful when working with ParallelPipeline . They allow you to selectively capture frames from one pipeline branch and inject them into another. ​ ProducerProcessor ProducerProcessor examines frames flowing through the pipeline, applies a filter to decide which frames to share, and optionally transforms these frames before sending them to connected consumers. ​ Constructor Parameters ​ filter Callable[[Frame], Awaitable[bool]] required An async function that determines which frames should be sent to consumers. Should return True for frames to be shared. ​ transformer Callable[[Frame], Awaitable[Frame]] default: "identity_transformer" Optional async function that transforms frames before sending to consumers. By default, passes frames unchanged. ​ passthrough bool default: "True" When True , passes all frames through the normal pipeline flow. When False , only passes through frames that don’t match the filter. ​ ConsumerProcessor ConsumerProcessor receives frames from a ProducerProcessor and injects them into its pipeline branch. ​ Constructor Parameters ​ producer ProducerProcessor required The producer processor that will send frames to this consumer. ​ transformer Callable[[Frame], Awaitable[Frame]] default: "identity_transformer" Optional async function that transforms frames before injecting them into the pipeline. ​ direction FrameDirection default: "FrameDirection.DOWNSTREAM" The direction in which to push received frames. Usually DOWNSTREAM to send frames forward in the pipeline. ​ Usage Examples ​ Basic Usage: Moving TTS Audio Between Branches Copy Ask AI # Create a producer that captures TTS audio frames async def is_tts_audio ( frame : Frame) -> bool : return isinstance (frame, TTSAudioRawFrame) # Define an async transformer function async def tts_to_input_audio_transformer ( frame : Frame) -> Frame: if isinstance (frame, TTSAudioRawFrame): # Convert TTS audio to input audio format return InputAudioRawFrame( audio = frame.audio, sample_rate = frame.sample_rate, num_channels = frame.num_channels ) return frame producer = ProducerProcessor( filter = is_tts_audio, transformer = tts_to_input_audio_transformer passthrough = True # Keep these frames in original pipeline ) # Create a consumer to receive the frames consumer = ConsumerProcessor( producer = producer, direction = FrameDirection. DOWNSTREAM ) # Use in a ParallelPipeline pipeline = Pipeline([ transport.input(), ParallelPipeline( # Branch 1: LLM for bot responses [ llm, tts, producer, # Capture TTS audio here ], # Branch 2: Audio processing branch [ consumer, # Receive TTS audio here llm, # Speech-to-Speech LLM (audio in) ] ), transport.output(), ]) Sentry Metrics UserIdleProcessor On this page Overview ProducerProcessor Constructor Parameters ConsumerProcessor Constructor Parameters Usage Examples Basic Usage: Moving TTS Audio Between Branches Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/services_supported-services_12dbab9c.txt b/services_supported-services_12dbab9c.txt
new file mode 100644
index 0000000000000000000000000000000000000000..af2945719ba98daf78dac9d0747c142a587d1634
--- /dev/null
+++ b/services_supported-services_12dbab9c.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/supported-services#speech-to-speech
+Title: Supported Services - Pipecat
+==================================================
+
+Supported Services - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Services Supported Services Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Transports Transports exchange audio and video streams between the user and bot. Service Setup Daily pip install "pipecat-ai[daily]" SmallWebRTCTransport pip install "pipecat-ai[webrtc]" FastAPI WebSocket pip install "pipecat-ai[websocket]" WebSocket Server pip install "pipecat-ai[websocket]" ​ Serializers Serializers convert between frames and media streams, enabling real-time communication over a websocket. Service Setup Exotel No dependencies required Plivo No dependencies required Telnyx No dependencies required Twilio No dependencies required ​ Speech-to-Text Speech-to-Text services receive and audio input and output transcriptions. Service Setup AssemblyAI pip install "pipecat-ai[assemblyai]" AWS Transcribe pip install "pipecat-ai[aws]" Azure pip install "pipecat-ai[azure]" Cartesia pip install "pipecat-ai[cartesia]" Deepgram pip install "pipecat-ai[deepgram]" Fal Wizper pip install "pipecat-ai[fal]" Gladia pip install "pipecat-ai[gladia]" Google pip install "pipecat-ai[google]" Groq (Whisper) pip install "pipecat-ai[groq]" NVIDIA Riva pip install "pipecat-ai[riva]" OpenAI (Whisper) pip install "pipecat-ai[openai]" SambaNova (Whisper) pip install "pipecat-ai[sambanova]" Speechmatics pip install "pipecat-ai[speechmatics]" Ultravox pip install "pipecat-ai[ultravox]" Whisper pip install "pipecat-ai[whisper]" ​ Large Language Models LLMs receive text or audio based input and output a streaming text response. Service Setup Anthropic pip install "pipecat-ai[anthropic]" AWS Bedrock pip install "pipecat-ai[aws]" Azure pip install "pipecat-ai[azure]" Cerebras pip install "pipecat-ai[cerebras]" DeepSeek pip install "pipecat-ai[deepseek]" Fireworks AI pip install "pipecat-ai[fireworks]" Google Gemini pip install "pipecat-ai[google]" Google Vertex AI pip install "pipecat-ai[google]" Grok pip install "pipecat-ai[grok]" Groq pip install "pipecat-ai[groq]" NVIDIA NIM pip install "pipecat-ai[nim]" Ollama pip install "pipecat-ai[ollama]" OpenAI pip install "pipecat-ai[openai]" OpenPipe pip install "pipecat-ai[openpipe]" OpenRouter pip install "pipecat-ai[openrouter]" Perplexity pip install "pipecat-ai[perplexity]" Qwen pip install "pipecat-ai[qwen]" SambaNova pip install "pipecat-ai[sambanova]" Together AI pip install "pipecat-ai[together]" ​ Text-to-Speech Text-to-Speech services receive text input and output audio streams or chunks. Service Setup AWS Polly pip install "pipecat-ai[aws]" Azure pip install "pipecat-ai[azure]" Cartesia pip install "pipecat-ai[cartesia]" Deepgram pip install "pipecat-ai[deepgram]" ElevenLabs pip install "pipecat-ai[elevenlabs]" Fish pip install "pipecat-ai[fish]" Google pip install "pipecat-ai[google]" Groq pip install "pipecat-ai[groq]" LMNT pip install "pipecat-ai[lmnt]" MiniMax No dependencies required Neuphonic pip install "pipecat-ai[neuphonic]" NVIDIA Riva pip install "pipecat-ai[riva]" OpenAI pip install "pipecat-ai[openai]" Piper No dependencies required PlayHT pip install "pipecat-ai[playht]" Rime pip install "pipecat-ai[rime]" Sarvam No dependencies required XTTS pip install "pipecat-ai[xtts]" ​ Speech-to-Speech Speech-to-Speech services are multi-modal LLM services that take in audio, video, or text and output audio or text. Service Setup AWS Nova Sonic pip install "pipecat-ai[aws-nova-sonic]" Gemini Multimodal Live pip install "pipecat-ai[google]" OpenAI Realtime pip install "pipecat-ai[openai]" ​ Image Generation Image generation services receive text inputs and output images. Service Setup fal pip install "pipecat-ai[fal]" Google Imagen pip install "pipecat-ai[google]" OpenAI pip install "pipecat-ai[openai]" ​ Video Video services enable you to build an avatar where audio and video are synchronized. Service Setup Simli pip install "pipecat-ai[simli]" Tavus pip install "pipecat-ai[tavus]" ​ Memory Memory services can be used to store and retrieve conversations. Service Setup mem0 pip install "pipecat-ai[mem0]" ​ Vision Vision services receive a streaming video input and output text describing the video input. Service Setup Moondream pip install "pipecat-ai[moondream]" ​ Analytics & Monitoring Analytics services help you better understand how your service operates. Service Setup Sentry pip install "pipecat-ai[sentry]" Reference docs Daily WebRTC On this page Transports Serializers Speech-to-Text Large Language Models Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/services_supported-services_d8fc899b.txt b/services_supported-services_d8fc899b.txt
new file mode 100644
index 0000000000000000000000000000000000000000..90e945e2da742a2deff79a084b069f6243e7679c
--- /dev/null
+++ b/services_supported-services_d8fc899b.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/supported-services#text-to-speech
+Title: Supported Services - Pipecat
+==================================================
+
+Supported Services - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Services Supported Services Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Transports Transports exchange audio and video streams between the user and bot. Service Setup Daily pip install "pipecat-ai[daily]" SmallWebRTCTransport pip install "pipecat-ai[webrtc]" FastAPI WebSocket pip install "pipecat-ai[websocket]" WebSocket Server pip install "pipecat-ai[websocket]" ​ Serializers Serializers convert between frames and media streams, enabling real-time communication over a websocket. Service Setup Exotel No dependencies required Plivo No dependencies required Telnyx No dependencies required Twilio No dependencies required ​ Speech-to-Text Speech-to-Text services receive and audio input and output transcriptions. Service Setup AssemblyAI pip install "pipecat-ai[assemblyai]" AWS Transcribe pip install "pipecat-ai[aws]" Azure pip install "pipecat-ai[azure]" Cartesia pip install "pipecat-ai[cartesia]" Deepgram pip install "pipecat-ai[deepgram]" Fal Wizper pip install "pipecat-ai[fal]" Gladia pip install "pipecat-ai[gladia]" Google pip install "pipecat-ai[google]" Groq (Whisper) pip install "pipecat-ai[groq]" NVIDIA Riva pip install "pipecat-ai[riva]" OpenAI (Whisper) pip install "pipecat-ai[openai]" SambaNova (Whisper) pip install "pipecat-ai[sambanova]" Speechmatics pip install "pipecat-ai[speechmatics]" Ultravox pip install "pipecat-ai[ultravox]" Whisper pip install "pipecat-ai[whisper]" ​ Large Language Models LLMs receive text or audio based input and output a streaming text response. Service Setup Anthropic pip install "pipecat-ai[anthropic]" AWS Bedrock pip install "pipecat-ai[aws]" Azure pip install "pipecat-ai[azure]" Cerebras pip install "pipecat-ai[cerebras]" DeepSeek pip install "pipecat-ai[deepseek]" Fireworks AI pip install "pipecat-ai[fireworks]" Google Gemini pip install "pipecat-ai[google]" Google Vertex AI pip install "pipecat-ai[google]" Grok pip install "pipecat-ai[grok]" Groq pip install "pipecat-ai[groq]" NVIDIA NIM pip install "pipecat-ai[nim]" Ollama pip install "pipecat-ai[ollama]" OpenAI pip install "pipecat-ai[openai]" OpenPipe pip install "pipecat-ai[openpipe]" OpenRouter pip install "pipecat-ai[openrouter]" Perplexity pip install "pipecat-ai[perplexity]" Qwen pip install "pipecat-ai[qwen]" SambaNova pip install "pipecat-ai[sambanova]" Together AI pip install "pipecat-ai[together]" ​ Text-to-Speech Text-to-Speech services receive text input and output audio streams or chunks. Service Setup AWS Polly pip install "pipecat-ai[aws]" Azure pip install "pipecat-ai[azure]" Cartesia pip install "pipecat-ai[cartesia]" Deepgram pip install "pipecat-ai[deepgram]" ElevenLabs pip install "pipecat-ai[elevenlabs]" Fish pip install "pipecat-ai[fish]" Google pip install "pipecat-ai[google]" Groq pip install "pipecat-ai[groq]" LMNT pip install "pipecat-ai[lmnt]" MiniMax No dependencies required Neuphonic pip install "pipecat-ai[neuphonic]" NVIDIA Riva pip install "pipecat-ai[riva]" OpenAI pip install "pipecat-ai[openai]" Piper No dependencies required PlayHT pip install "pipecat-ai[playht]" Rime pip install "pipecat-ai[rime]" Sarvam No dependencies required XTTS pip install "pipecat-ai[xtts]" ​ Speech-to-Speech Speech-to-Speech services are multi-modal LLM services that take in audio, video, or text and output audio or text. Service Setup AWS Nova Sonic pip install "pipecat-ai[aws-nova-sonic]" Gemini Multimodal Live pip install "pipecat-ai[google]" OpenAI Realtime pip install "pipecat-ai[openai]" ​ Image Generation Image generation services receive text inputs and output images. Service Setup fal pip install "pipecat-ai[fal]" Google Imagen pip install "pipecat-ai[google]" OpenAI pip install "pipecat-ai[openai]" ​ Video Video services enable you to build an avatar where audio and video are synchronized. Service Setup Simli pip install "pipecat-ai[simli]" Tavus pip install "pipecat-ai[tavus]" ​ Memory Memory services can be used to store and retrieve conversations. Service Setup mem0 pip install "pipecat-ai[mem0]" ​ Vision Vision services receive a streaming video input and output text describing the video input. Service Setup Moondream pip install "pipecat-ai[moondream]" ​ Analytics & Monitoring Analytics services help you better understand how your service operates. Service Setup Sentry pip install "pipecat-ai[sentry]" Reference docs Daily WebRTC On this page Transports Serializers Speech-to-Text Large Language Models Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/smart-turn_fal-smart-turn_0bb5c895.txt b/smart-turn_fal-smart-turn_0bb5c895.txt
new file mode 100644
index 0000000000000000000000000000000000000000..0ab12d3d4ab23ec5321f4965ccccfdd700496078
--- /dev/null
+++ b/smart-turn_fal-smart-turn_0bb5c895.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/smart-turn/fal-smart-turn#notes
+Title: Fal Smart Turn - Pipecat
+==================================================
+
+Fal Smart Turn - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Smart Turn Detection Fal Smart Turn Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Smart Turn Overview Fal Smart Turn Local CoreML Smart Turn Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview FalSmartTurnAnalyzer provides an easy way to use Smart Turn detection via Fal.ai’s cloud infrastructure. This implementation requires minimal setup - just an API key - and offers scalable inference without having to manage your own servers. ​ Installation Copy Ask AI pip install "pipecat-ai[remote-smart-turn]" ​ Requirements A Fal.ai account and API key (get one at Fal.ai ) Internet connectivity for making API calls ​ Configuration ​ Constructor Parameters ​ api_key Optional[str] default: "None" Your Fal.ai API key for authentication (required unless using a custom deployment) ​ url str default: "https://fal.run/fal-ai/smart-turn/raw" URL endpoint for the Smart Turn API (defaults to the official Fal deployment) ​ aiohttp_session aiohttp.ClientSession required An aiohttp client session for making HTTP requests ​ sample_rate Optional[int] default: "None" Audio sample rate (will be set by the transport if not provided) ​ params SmartTurnParams default: "SmartTurnParams()" Configuration parameters for turn detection. See SmartTurnParams for details. ​ Example Copy Ask AI import os import aiohttp from pipecat.audio.turn.smart_turn.fal_smart_turn import FalSmartTurnAnalyzer from pipecat.audio.vad.silero import SileroVADAnalyzer from pipecat.audio.vad.vad_analyzer import VADParams from pipecat.transports.base_transport import TransportParams async def setup_transport (): async with aiohttp.ClientSession() as session: transport = SmallWebRTCTransport( webrtc_connection = webrtc_connection, params = TransportParams( audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer( params = VADParams( stop_secs = 0.2 )), turn_analyzer = FalSmartTurnAnalyzer( api_key = os.getenv( "FAL_SMART_TURN_API_KEY" ), aiohttp_session = session ), ), ) # Continue with pipeline setup... ​ Custom Deployment You can also deploy the Smart Turn model yourself on Fal.ai and point to your custom deployment: Copy Ask AI turn_analyzer = FalSmartTurnAnalyzer( url = "https://fal.run/your-username/your-deployment/raw" , api_key = os.getenv( "FAL_API_KEY" ), aiohttp_session = session ) ​ Performance Considerations Latency : While Fal provides global infrastructure, there will be network latency compared to local inference Reliability : Depends on network connectivity and Fal.ai service availability Scalability : Handles scaling automatically based on your usage ​ Notes Fal handles the model hosting, scaling, and infrastructure management The session timeout is controlled by the stop_secs parameter For high-throughput applications, consider deploying your own inference service Smart Turn Overview Local CoreML Smart Turn On this page Overview Installation Requirements Configuration Constructor Parameters Example Custom Deployment Performance Considerations Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/smart-turn_fal-smart-turn_191f175a.txt b/smart-turn_fal-smart-turn_191f175a.txt
new file mode 100644
index 0000000000000000000000000000000000000000..3e34f507ac248a894b89a5dfba56b2a18d2cd1fd
--- /dev/null
+++ b/smart-turn_fal-smart-turn_191f175a.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/smart-turn/fal-smart-turn#param-aiohttp-session
+Title: Fal Smart Turn - Pipecat
+==================================================
+
+Fal Smart Turn - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Smart Turn Detection Fal Smart Turn Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Smart Turn Overview Fal Smart Turn Local CoreML Smart Turn Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview FalSmartTurnAnalyzer provides an easy way to use Smart Turn detection via Fal.ai’s cloud infrastructure. This implementation requires minimal setup - just an API key - and offers scalable inference without having to manage your own servers. ​ Installation Copy Ask AI pip install "pipecat-ai[remote-smart-turn]" ​ Requirements A Fal.ai account and API key (get one at Fal.ai ) Internet connectivity for making API calls ​ Configuration ​ Constructor Parameters ​ api_key Optional[str] default: "None" Your Fal.ai API key for authentication (required unless using a custom deployment) ​ url str default: "https://fal.run/fal-ai/smart-turn/raw" URL endpoint for the Smart Turn API (defaults to the official Fal deployment) ​ aiohttp_session aiohttp.ClientSession required An aiohttp client session for making HTTP requests ​ sample_rate Optional[int] default: "None" Audio sample rate (will be set by the transport if not provided) ​ params SmartTurnParams default: "SmartTurnParams()" Configuration parameters for turn detection. See SmartTurnParams for details. ​ Example Copy Ask AI import os import aiohttp from pipecat.audio.turn.smart_turn.fal_smart_turn import FalSmartTurnAnalyzer from pipecat.audio.vad.silero import SileroVADAnalyzer from pipecat.audio.vad.vad_analyzer import VADParams from pipecat.transports.base_transport import TransportParams async def setup_transport (): async with aiohttp.ClientSession() as session: transport = SmallWebRTCTransport( webrtc_connection = webrtc_connection, params = TransportParams( audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer( params = VADParams( stop_secs = 0.2 )), turn_analyzer = FalSmartTurnAnalyzer( api_key = os.getenv( "FAL_SMART_TURN_API_KEY" ), aiohttp_session = session ), ), ) # Continue with pipeline setup... ​ Custom Deployment You can also deploy the Smart Turn model yourself on Fal.ai and point to your custom deployment: Copy Ask AI turn_analyzer = FalSmartTurnAnalyzer( url = "https://fal.run/your-username/your-deployment/raw" , api_key = os.getenv( "FAL_API_KEY" ), aiohttp_session = session ) ​ Performance Considerations Latency : While Fal provides global infrastructure, there will be network latency compared to local inference Reliability : Depends on network connectivity and Fal.ai service availability Scalability : Handles scaling automatically based on your usage ​ Notes Fal handles the model hosting, scaling, and infrastructure management The session timeout is controlled by the stop_secs parameter For high-throughput applications, consider deploying your own inference service Smart Turn Overview Local CoreML Smart Turn On this page Overview Installation Requirements Configuration Constructor Parameters Example Custom Deployment Performance Considerations Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/smart-turn_fal-smart-turn_2aeb1dcf.txt b/smart-turn_fal-smart-turn_2aeb1dcf.txt
new file mode 100644
index 0000000000000000000000000000000000000000..20b72c2d0d6ae5e3909db9c2d460b882abe6eb44
--- /dev/null
+++ b/smart-turn_fal-smart-turn_2aeb1dcf.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/smart-turn/fal-smart-turn#overview
+Title: Fal Smart Turn - Pipecat
+==================================================
+
+Fal Smart Turn - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Smart Turn Detection Fal Smart Turn Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Smart Turn Overview Fal Smart Turn Local CoreML Smart Turn Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview FalSmartTurnAnalyzer provides an easy way to use Smart Turn detection via Fal.ai’s cloud infrastructure. This implementation requires minimal setup - just an API key - and offers scalable inference without having to manage your own servers. ​ Installation Copy Ask AI pip install "pipecat-ai[remote-smart-turn]" ​ Requirements A Fal.ai account and API key (get one at Fal.ai ) Internet connectivity for making API calls ​ Configuration ​ Constructor Parameters ​ api_key Optional[str] default: "None" Your Fal.ai API key for authentication (required unless using a custom deployment) ​ url str default: "https://fal.run/fal-ai/smart-turn/raw" URL endpoint for the Smart Turn API (defaults to the official Fal deployment) ​ aiohttp_session aiohttp.ClientSession required An aiohttp client session for making HTTP requests ​ sample_rate Optional[int] default: "None" Audio sample rate (will be set by the transport if not provided) ​ params SmartTurnParams default: "SmartTurnParams()" Configuration parameters for turn detection. See SmartTurnParams for details. ​ Example Copy Ask AI import os import aiohttp from pipecat.audio.turn.smart_turn.fal_smart_turn import FalSmartTurnAnalyzer from pipecat.audio.vad.silero import SileroVADAnalyzer from pipecat.audio.vad.vad_analyzer import VADParams from pipecat.transports.base_transport import TransportParams async def setup_transport (): async with aiohttp.ClientSession() as session: transport = SmallWebRTCTransport( webrtc_connection = webrtc_connection, params = TransportParams( audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer( params = VADParams( stop_secs = 0.2 )), turn_analyzer = FalSmartTurnAnalyzer( api_key = os.getenv( "FAL_SMART_TURN_API_KEY" ), aiohttp_session = session ), ), ) # Continue with pipeline setup... ​ Custom Deployment You can also deploy the Smart Turn model yourself on Fal.ai and point to your custom deployment: Copy Ask AI turn_analyzer = FalSmartTurnAnalyzer( url = "https://fal.run/your-username/your-deployment/raw" , api_key = os.getenv( "FAL_API_KEY" ), aiohttp_session = session ) ​ Performance Considerations Latency : While Fal provides global infrastructure, there will be network latency compared to local inference Reliability : Depends on network connectivity and Fal.ai service availability Scalability : Handles scaling automatically based on your usage ​ Notes Fal handles the model hosting, scaling, and infrastructure management The session timeout is controlled by the stop_secs parameter For high-throughput applications, consider deploying your own inference service Smart Turn Overview Local CoreML Smart Turn On this page Overview Installation Requirements Configuration Constructor Parameters Example Custom Deployment Performance Considerations Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/smart-turn_fal-smart-turn_3f807c11.txt b/smart-turn_fal-smart-turn_3f807c11.txt
new file mode 100644
index 0000000000000000000000000000000000000000..6e6d5a3ab65d82a1329585df126d14dae5de11bc
--- /dev/null
+++ b/smart-turn_fal-smart-turn_3f807c11.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/smart-turn/fal-smart-turn#requirements
+Title: Fal Smart Turn - Pipecat
+==================================================
+
+Fal Smart Turn - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Smart Turn Detection Fal Smart Turn Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Smart Turn Overview Fal Smart Turn Local CoreML Smart Turn Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview FalSmartTurnAnalyzer provides an easy way to use Smart Turn detection via Fal.ai’s cloud infrastructure. This implementation requires minimal setup - just an API key - and offers scalable inference without having to manage your own servers. ​ Installation Copy Ask AI pip install "pipecat-ai[remote-smart-turn]" ​ Requirements A Fal.ai account and API key (get one at Fal.ai ) Internet connectivity for making API calls ​ Configuration ​ Constructor Parameters ​ api_key Optional[str] default: "None" Your Fal.ai API key for authentication (required unless using a custom deployment) ​ url str default: "https://fal.run/fal-ai/smart-turn/raw" URL endpoint for the Smart Turn API (defaults to the official Fal deployment) ​ aiohttp_session aiohttp.ClientSession required An aiohttp client session for making HTTP requests ​ sample_rate Optional[int] default: "None" Audio sample rate (will be set by the transport if not provided) ​ params SmartTurnParams default: "SmartTurnParams()" Configuration parameters for turn detection. See SmartTurnParams for details. ​ Example Copy Ask AI import os import aiohttp from pipecat.audio.turn.smart_turn.fal_smart_turn import FalSmartTurnAnalyzer from pipecat.audio.vad.silero import SileroVADAnalyzer from pipecat.audio.vad.vad_analyzer import VADParams from pipecat.transports.base_transport import TransportParams async def setup_transport (): async with aiohttp.ClientSession() as session: transport = SmallWebRTCTransport( webrtc_connection = webrtc_connection, params = TransportParams( audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer( params = VADParams( stop_secs = 0.2 )), turn_analyzer = FalSmartTurnAnalyzer( api_key = os.getenv( "FAL_SMART_TURN_API_KEY" ), aiohttp_session = session ), ), ) # Continue with pipeline setup... ​ Custom Deployment You can also deploy the Smart Turn model yourself on Fal.ai and point to your custom deployment: Copy Ask AI turn_analyzer = FalSmartTurnAnalyzer( url = "https://fal.run/your-username/your-deployment/raw" , api_key = os.getenv( "FAL_API_KEY" ), aiohttp_session = session ) ​ Performance Considerations Latency : While Fal provides global infrastructure, there will be network latency compared to local inference Reliability : Depends on network connectivity and Fal.ai service availability Scalability : Handles scaling automatically based on your usage ​ Notes Fal handles the model hosting, scaling, and infrastructure management The session timeout is controlled by the stop_secs parameter For high-throughput applications, consider deploying your own inference service Smart Turn Overview Local CoreML Smart Turn On this page Overview Installation Requirements Configuration Constructor Parameters Example Custom Deployment Performance Considerations Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/smart-turn_fal-smart-turn_6f9acb09.txt b/smart-turn_fal-smart-turn_6f9acb09.txt
new file mode 100644
index 0000000000000000000000000000000000000000..5ec0b8c1e382dd3bbd8d4cfec943bfbc31c4dd07
--- /dev/null
+++ b/smart-turn_fal-smart-turn_6f9acb09.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/smart-turn/fal-smart-turn#example
+Title: Fal Smart Turn - Pipecat
+==================================================
+
+Fal Smart Turn - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Smart Turn Detection Fal Smart Turn Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Smart Turn Overview Fal Smart Turn Local CoreML Smart Turn Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview FalSmartTurnAnalyzer provides an easy way to use Smart Turn detection via Fal.ai’s cloud infrastructure. This implementation requires minimal setup - just an API key - and offers scalable inference without having to manage your own servers. ​ Installation Copy Ask AI pip install "pipecat-ai[remote-smart-turn]" ​ Requirements A Fal.ai account and API key (get one at Fal.ai ) Internet connectivity for making API calls ​ Configuration ​ Constructor Parameters ​ api_key Optional[str] default: "None" Your Fal.ai API key for authentication (required unless using a custom deployment) ​ url str default: "https://fal.run/fal-ai/smart-turn/raw" URL endpoint for the Smart Turn API (defaults to the official Fal deployment) ​ aiohttp_session aiohttp.ClientSession required An aiohttp client session for making HTTP requests ​ sample_rate Optional[int] default: "None" Audio sample rate (will be set by the transport if not provided) ​ params SmartTurnParams default: "SmartTurnParams()" Configuration parameters for turn detection. See SmartTurnParams for details. ​ Example Copy Ask AI import os import aiohttp from pipecat.audio.turn.smart_turn.fal_smart_turn import FalSmartTurnAnalyzer from pipecat.audio.vad.silero import SileroVADAnalyzer from pipecat.audio.vad.vad_analyzer import VADParams from pipecat.transports.base_transport import TransportParams async def setup_transport (): async with aiohttp.ClientSession() as session: transport = SmallWebRTCTransport( webrtc_connection = webrtc_connection, params = TransportParams( audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer( params = VADParams( stop_secs = 0.2 )), turn_analyzer = FalSmartTurnAnalyzer( api_key = os.getenv( "FAL_SMART_TURN_API_KEY" ), aiohttp_session = session ), ), ) # Continue with pipeline setup... ​ Custom Deployment You can also deploy the Smart Turn model yourself on Fal.ai and point to your custom deployment: Copy Ask AI turn_analyzer = FalSmartTurnAnalyzer( url = "https://fal.run/your-username/your-deployment/raw" , api_key = os.getenv( "FAL_API_KEY" ), aiohttp_session = session ) ​ Performance Considerations Latency : While Fal provides global infrastructure, there will be network latency compared to local inference Reliability : Depends on network connectivity and Fal.ai service availability Scalability : Handles scaling automatically based on your usage ​ Notes Fal handles the model hosting, scaling, and infrastructure management The session timeout is controlled by the stop_secs parameter For high-throughput applications, consider deploying your own inference service Smart Turn Overview Local CoreML Smart Turn On this page Overview Installation Requirements Configuration Constructor Parameters Example Custom Deployment Performance Considerations Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/smart-turn_fal-smart-turn_8d3ebd99.txt b/smart-turn_fal-smart-turn_8d3ebd99.txt
new file mode 100644
index 0000000000000000000000000000000000000000..564c82c7d086d3c7ab5b10f4926b99f530648ec7
--- /dev/null
+++ b/smart-turn_fal-smart-turn_8d3ebd99.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/smart-turn/fal-smart-turn#param-api-key
+Title: Fal Smart Turn - Pipecat
+==================================================
+
+Fal Smart Turn - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Smart Turn Detection Fal Smart Turn Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Smart Turn Overview Fal Smart Turn Local CoreML Smart Turn Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview FalSmartTurnAnalyzer provides an easy way to use Smart Turn detection via Fal.ai’s cloud infrastructure. This implementation requires minimal setup - just an API key - and offers scalable inference without having to manage your own servers. ​ Installation Copy Ask AI pip install "pipecat-ai[remote-smart-turn]" ​ Requirements A Fal.ai account and API key (get one at Fal.ai ) Internet connectivity for making API calls ​ Configuration ​ Constructor Parameters ​ api_key Optional[str] default: "None" Your Fal.ai API key for authentication (required unless using a custom deployment) ​ url str default: "https://fal.run/fal-ai/smart-turn/raw" URL endpoint for the Smart Turn API (defaults to the official Fal deployment) ​ aiohttp_session aiohttp.ClientSession required An aiohttp client session for making HTTP requests ​ sample_rate Optional[int] default: "None" Audio sample rate (will be set by the transport if not provided) ​ params SmartTurnParams default: "SmartTurnParams()" Configuration parameters for turn detection. See SmartTurnParams for details. ​ Example Copy Ask AI import os import aiohttp from pipecat.audio.turn.smart_turn.fal_smart_turn import FalSmartTurnAnalyzer from pipecat.audio.vad.silero import SileroVADAnalyzer from pipecat.audio.vad.vad_analyzer import VADParams from pipecat.transports.base_transport import TransportParams async def setup_transport (): async with aiohttp.ClientSession() as session: transport = SmallWebRTCTransport( webrtc_connection = webrtc_connection, params = TransportParams( audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer( params = VADParams( stop_secs = 0.2 )), turn_analyzer = FalSmartTurnAnalyzer( api_key = os.getenv( "FAL_SMART_TURN_API_KEY" ), aiohttp_session = session ), ), ) # Continue with pipeline setup... ​ Custom Deployment You can also deploy the Smart Turn model yourself on Fal.ai and point to your custom deployment: Copy Ask AI turn_analyzer = FalSmartTurnAnalyzer( url = "https://fal.run/your-username/your-deployment/raw" , api_key = os.getenv( "FAL_API_KEY" ), aiohttp_session = session ) ​ Performance Considerations Latency : While Fal provides global infrastructure, there will be network latency compared to local inference Reliability : Depends on network connectivity and Fal.ai service availability Scalability : Handles scaling automatically based on your usage ​ Notes Fal handles the model hosting, scaling, and infrastructure management The session timeout is controlled by the stop_secs parameter For high-throughput applications, consider deploying your own inference service Smart Turn Overview Local CoreML Smart Turn On this page Overview Installation Requirements Configuration Constructor Parameters Example Custom Deployment Performance Considerations Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/smart-turn_fal-smart-turn_b3619e69.txt b/smart-turn_fal-smart-turn_b3619e69.txt
new file mode 100644
index 0000000000000000000000000000000000000000..def37a9b9f092bd1b5cf1f81b3de44859df72c40
--- /dev/null
+++ b/smart-turn_fal-smart-turn_b3619e69.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/smart-turn/fal-smart-turn#performance-considerations
+Title: Fal Smart Turn - Pipecat
+==================================================
+
+Fal Smart Turn - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Smart Turn Detection Fal Smart Turn Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Smart Turn Overview Fal Smart Turn Local CoreML Smart Turn Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview FalSmartTurnAnalyzer provides an easy way to use Smart Turn detection via Fal.ai’s cloud infrastructure. This implementation requires minimal setup - just an API key - and offers scalable inference without having to manage your own servers. ​ Installation Copy Ask AI pip install "pipecat-ai[remote-smart-turn]" ​ Requirements A Fal.ai account and API key (get one at Fal.ai ) Internet connectivity for making API calls ​ Configuration ​ Constructor Parameters ​ api_key Optional[str] default: "None" Your Fal.ai API key for authentication (required unless using a custom deployment) ​ url str default: "https://fal.run/fal-ai/smart-turn/raw" URL endpoint for the Smart Turn API (defaults to the official Fal deployment) ​ aiohttp_session aiohttp.ClientSession required An aiohttp client session for making HTTP requests ​ sample_rate Optional[int] default: "None" Audio sample rate (will be set by the transport if not provided) ​ params SmartTurnParams default: "SmartTurnParams()" Configuration parameters for turn detection. See SmartTurnParams for details. ​ Example Copy Ask AI import os import aiohttp from pipecat.audio.turn.smart_turn.fal_smart_turn import FalSmartTurnAnalyzer from pipecat.audio.vad.silero import SileroVADAnalyzer from pipecat.audio.vad.vad_analyzer import VADParams from pipecat.transports.base_transport import TransportParams async def setup_transport (): async with aiohttp.ClientSession() as session: transport = SmallWebRTCTransport( webrtc_connection = webrtc_connection, params = TransportParams( audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer( params = VADParams( stop_secs = 0.2 )), turn_analyzer = FalSmartTurnAnalyzer( api_key = os.getenv( "FAL_SMART_TURN_API_KEY" ), aiohttp_session = session ), ), ) # Continue with pipeline setup... ​ Custom Deployment You can also deploy the Smart Turn model yourself on Fal.ai and point to your custom deployment: Copy Ask AI turn_analyzer = FalSmartTurnAnalyzer( url = "https://fal.run/your-username/your-deployment/raw" , api_key = os.getenv( "FAL_API_KEY" ), aiohttp_session = session ) ​ Performance Considerations Latency : While Fal provides global infrastructure, there will be network latency compared to local inference Reliability : Depends on network connectivity and Fal.ai service availability Scalability : Handles scaling automatically based on your usage ​ Notes Fal handles the model hosting, scaling, and infrastructure management The session timeout is controlled by the stop_secs parameter For high-throughput applications, consider deploying your own inference service Smart Turn Overview Local CoreML Smart Turn On this page Overview Installation Requirements Configuration Constructor Parameters Example Custom Deployment Performance Considerations Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/smart-turn_fal-smart-turn_c18eb663.txt b/smart-turn_fal-smart-turn_c18eb663.txt
new file mode 100644
index 0000000000000000000000000000000000000000..2b262dd9c31c69b440362ff3de95e7c95b0b5ec1
--- /dev/null
+++ b/smart-turn_fal-smart-turn_c18eb663.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/smart-turn/fal-smart-turn#param-url
+Title: Fal Smart Turn - Pipecat
+==================================================
+
+Fal Smart Turn - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Smart Turn Detection Fal Smart Turn Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Smart Turn Overview Fal Smart Turn Local CoreML Smart Turn Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview FalSmartTurnAnalyzer provides an easy way to use Smart Turn detection via Fal.ai’s cloud infrastructure. This implementation requires minimal setup - just an API key - and offers scalable inference without having to manage your own servers. ​ Installation Copy Ask AI pip install "pipecat-ai[remote-smart-turn]" ​ Requirements A Fal.ai account and API key (get one at Fal.ai ) Internet connectivity for making API calls ​ Configuration ​ Constructor Parameters ​ api_key Optional[str] default: "None" Your Fal.ai API key for authentication (required unless using a custom deployment) ​ url str default: "https://fal.run/fal-ai/smart-turn/raw" URL endpoint for the Smart Turn API (defaults to the official Fal deployment) ​ aiohttp_session aiohttp.ClientSession required An aiohttp client session for making HTTP requests ​ sample_rate Optional[int] default: "None" Audio sample rate (will be set by the transport if not provided) ​ params SmartTurnParams default: "SmartTurnParams()" Configuration parameters for turn detection. See SmartTurnParams for details. ​ Example Copy Ask AI import os import aiohttp from pipecat.audio.turn.smart_turn.fal_smart_turn import FalSmartTurnAnalyzer from pipecat.audio.vad.silero import SileroVADAnalyzer from pipecat.audio.vad.vad_analyzer import VADParams from pipecat.transports.base_transport import TransportParams async def setup_transport (): async with aiohttp.ClientSession() as session: transport = SmallWebRTCTransport( webrtc_connection = webrtc_connection, params = TransportParams( audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer( params = VADParams( stop_secs = 0.2 )), turn_analyzer = FalSmartTurnAnalyzer( api_key = os.getenv( "FAL_SMART_TURN_API_KEY" ), aiohttp_session = session ), ), ) # Continue with pipeline setup... ​ Custom Deployment You can also deploy the Smart Turn model yourself on Fal.ai and point to your custom deployment: Copy Ask AI turn_analyzer = FalSmartTurnAnalyzer( url = "https://fal.run/your-username/your-deployment/raw" , api_key = os.getenv( "FAL_API_KEY" ), aiohttp_session = session ) ​ Performance Considerations Latency : While Fal provides global infrastructure, there will be network latency compared to local inference Reliability : Depends on network connectivity and Fal.ai service availability Scalability : Handles scaling automatically based on your usage ​ Notes Fal handles the model hosting, scaling, and infrastructure management The session timeout is controlled by the stop_secs parameter For high-throughput applications, consider deploying your own inference service Smart Turn Overview Local CoreML Smart Turn On this page Overview Installation Requirements Configuration Constructor Parameters Example Custom Deployment Performance Considerations Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/smart-turn_fal-smart-turn_f1e07606.txt b/smart-turn_fal-smart-turn_f1e07606.txt
new file mode 100644
index 0000000000000000000000000000000000000000..e20c45eec1abf88ce16a39c81c831dc6a42f870d
--- /dev/null
+++ b/smart-turn_fal-smart-turn_f1e07606.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/smart-turn/fal-smart-turn
+Title: Fal Smart Turn - Pipecat
+==================================================
+
+Fal Smart Turn - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Smart Turn Detection Fal Smart Turn Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Smart Turn Overview Fal Smart Turn Local CoreML Smart Turn Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview FalSmartTurnAnalyzer provides an easy way to use Smart Turn detection via Fal.ai’s cloud infrastructure. This implementation requires minimal setup - just an API key - and offers scalable inference without having to manage your own servers. ​ Installation Copy Ask AI pip install "pipecat-ai[remote-smart-turn]" ​ Requirements A Fal.ai account and API key (get one at Fal.ai ) Internet connectivity for making API calls ​ Configuration ​ Constructor Parameters ​ api_key Optional[str] default: "None" Your Fal.ai API key for authentication (required unless using a custom deployment) ​ url str default: "https://fal.run/fal-ai/smart-turn/raw" URL endpoint for the Smart Turn API (defaults to the official Fal deployment) ​ aiohttp_session aiohttp.ClientSession required An aiohttp client session for making HTTP requests ​ sample_rate Optional[int] default: "None" Audio sample rate (will be set by the transport if not provided) ​ params SmartTurnParams default: "SmartTurnParams()" Configuration parameters for turn detection. See SmartTurnParams for details. ​ Example Copy Ask AI import os import aiohttp from pipecat.audio.turn.smart_turn.fal_smart_turn import FalSmartTurnAnalyzer from pipecat.audio.vad.silero import SileroVADAnalyzer from pipecat.audio.vad.vad_analyzer import VADParams from pipecat.transports.base_transport import TransportParams async def setup_transport (): async with aiohttp.ClientSession() as session: transport = SmallWebRTCTransport( webrtc_connection = webrtc_connection, params = TransportParams( audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer( params = VADParams( stop_secs = 0.2 )), turn_analyzer = FalSmartTurnAnalyzer( api_key = os.getenv( "FAL_SMART_TURN_API_KEY" ), aiohttp_session = session ), ), ) # Continue with pipeline setup... ​ Custom Deployment You can also deploy the Smart Turn model yourself on Fal.ai and point to your custom deployment: Copy Ask AI turn_analyzer = FalSmartTurnAnalyzer( url = "https://fal.run/your-username/your-deployment/raw" , api_key = os.getenv( "FAL_API_KEY" ), aiohttp_session = session ) ​ Performance Considerations Latency : While Fal provides global infrastructure, there will be network latency compared to local inference Reliability : Depends on network connectivity and Fal.ai service availability Scalability : Handles scaling automatically based on your usage ​ Notes Fal handles the model hosting, scaling, and infrastructure management The session timeout is controlled by the stop_secs parameter For high-throughput applications, consider deploying your own inference service Smart Turn Overview Local CoreML Smart Turn On this page Overview Installation Requirements Configuration Constructor Parameters Example Custom Deployment Performance Considerations Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/smart-turn_local-coreml-smart-turn_0906791d.txt b/smart-turn_local-coreml-smart-turn_0906791d.txt
new file mode 100644
index 0000000000000000000000000000000000000000..909ddae5a05390ba9b80eccc415dfbcda3dc07af
--- /dev/null
+++ b/smart-turn_local-coreml-smart-turn_0906791d.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/smart-turn/local-coreml-smart-turn#param-params
+Title: Local CoreML Smart Turn - Pipecat
+==================================================
+
+Local CoreML Smart Turn - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Smart Turn Detection Local CoreML Smart Turn Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Smart Turn Overview Fal Smart Turn Local CoreML Smart Turn Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview LocalCoreMLSmartTurnAnalyzer runs Smart Turn inference directly on your Mac using Apple’s CoreML framework. This provides low-latency inference without external API dependencies, making it ideal for development and applications where network access is limited or latency is critical. ​ Installation Copy Ask AI pip install "pipecat-ai[local-smart-turn]" ​ Requirements Apple Silicon Mac (M1/M2/M3 series) macOS 11.0 or later ​ Local Model Setup To use the LocalCoreMLSmartTurnAnalyzer , you need to set up the CoreML model locally: Install Git LFS (Large File Storage): macOS Ubuntu/Debian Copy Ask AI brew install git-lfs Initialize Git LFS Copy Ask AI git lfs install Clone the Smart Turn model repository: Copy Ask AI git clone https://huggingface.co/pipecat-ai/smart-turn Set the environment variable to the cloned repository path: Copy Ask AI # Add to your .env file or environment export LOCAL_SMART_TURN_MODEL_PATH = / path / to / smart-turn ​ Configuration ​ Constructor Parameters ​ smart_turn_model_path str required Path to the directory containing the Smart Turn model files ​ sample_rate Optional[int] default: "None" Audio sample rate (will be set by the transport if not provided) ​ params SmartTurnParams default: "SmartTurnParams()" Configuration parameters for turn detection. See SmartTurnParams for details. ​ Example Copy Ask AI import os from pipecat.audio.turn.smart_turn.local_coreml_smart_turn import LocalCoreMLSmartTurnAnalyzer from pipecat.audio.turn.smart_turn.base_smart_turn import SmartTurnParams from pipecat.audio.vad.silero import SileroVADAnalyzer from pipecat.audio.vad.vad_analyzer import VADParams from pipecat.transports.base_transport import TransportParams # Get the path to the Smart Turn model smart_turn_model_path = os.getenv( "LOCAL_SMART_TURN_MODEL_PATH" ) # Create transport with local Smart Turn detection transport = SmallWebRTCTransport( webrtc_connection = webrtc_connection, params = TransportParams( audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer( params = VADParams( stop_secs = 0.2 )), turn_analyzer = LocalCoreMLSmartTurnAnalyzer( smart_turn_model_path = smart_turn_model_path, params = SmartTurnParams( stop_secs = 2.0 , # Shorter stop time when using Smart Turn pre_speech_ms = 0.0 , max_duration_secs = 8.0 ) ), ), ) ​ Performance Considerations Latency : Very low latency since inference happens locally Resource Usage : Uses local CPU/GPU resources Reliability : No dependency on external services or network connectivity ​ Notes Optimal for development environments and latency-sensitive applications The CoreML model is optimized for Apple Silicon but will work on Intel Macs with reduced performance First inference may be slower as the model is loaded and compiled Fal Smart Turn Watchdog Timers On this page Overview Installation Requirements Local Model Setup Configuration Constructor Parameters Example Performance Considerations Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/smart-turn_local-coreml-smart-turn_72407284.txt b/smart-turn_local-coreml-smart-turn_72407284.txt
new file mode 100644
index 0000000000000000000000000000000000000000..f63664a803da2481a5c833d392045960c311c193
--- /dev/null
+++ b/smart-turn_local-coreml-smart-turn_72407284.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/smart-turn/local-coreml-smart-turn
+Title: Local CoreML Smart Turn - Pipecat
+==================================================
+
+Local CoreML Smart Turn - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Smart Turn Detection Local CoreML Smart Turn Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Smart Turn Overview Fal Smart Turn Local CoreML Smart Turn Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview LocalCoreMLSmartTurnAnalyzer runs Smart Turn inference directly on your Mac using Apple’s CoreML framework. This provides low-latency inference without external API dependencies, making it ideal for development and applications where network access is limited or latency is critical. ​ Installation Copy Ask AI pip install "pipecat-ai[local-smart-turn]" ​ Requirements Apple Silicon Mac (M1/M2/M3 series) macOS 11.0 or later ​ Local Model Setup To use the LocalCoreMLSmartTurnAnalyzer , you need to set up the CoreML model locally: Install Git LFS (Large File Storage): macOS Ubuntu/Debian Copy Ask AI brew install git-lfs Initialize Git LFS Copy Ask AI git lfs install Clone the Smart Turn model repository: Copy Ask AI git clone https://huggingface.co/pipecat-ai/smart-turn Set the environment variable to the cloned repository path: Copy Ask AI # Add to your .env file or environment export LOCAL_SMART_TURN_MODEL_PATH = / path / to / smart-turn ​ Configuration ​ Constructor Parameters ​ smart_turn_model_path str required Path to the directory containing the Smart Turn model files ​ sample_rate Optional[int] default: "None" Audio sample rate (will be set by the transport if not provided) ​ params SmartTurnParams default: "SmartTurnParams()" Configuration parameters for turn detection. See SmartTurnParams for details. ​ Example Copy Ask AI import os from pipecat.audio.turn.smart_turn.local_coreml_smart_turn import LocalCoreMLSmartTurnAnalyzer from pipecat.audio.turn.smart_turn.base_smart_turn import SmartTurnParams from pipecat.audio.vad.silero import SileroVADAnalyzer from pipecat.audio.vad.vad_analyzer import VADParams from pipecat.transports.base_transport import TransportParams # Get the path to the Smart Turn model smart_turn_model_path = os.getenv( "LOCAL_SMART_TURN_MODEL_PATH" ) # Create transport with local Smart Turn detection transport = SmallWebRTCTransport( webrtc_connection = webrtc_connection, params = TransportParams( audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer( params = VADParams( stop_secs = 0.2 )), turn_analyzer = LocalCoreMLSmartTurnAnalyzer( smart_turn_model_path = smart_turn_model_path, params = SmartTurnParams( stop_secs = 2.0 , # Shorter stop time when using Smart Turn pre_speech_ms = 0.0 , max_duration_secs = 8.0 ) ), ), ) ​ Performance Considerations Latency : Very low latency since inference happens locally Resource Usage : Uses local CPU/GPU resources Reliability : No dependency on external services or network connectivity ​ Notes Optimal for development environments and latency-sensitive applications The CoreML model is optimized for Apple Silicon but will work on Intel Macs with reduced performance First inference may be slower as the model is loaded and compiled Fal Smart Turn Watchdog Timers On this page Overview Installation Requirements Local Model Setup Configuration Constructor Parameters Example Performance Considerations Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/smart-turn_local-coreml-smart-turn_7d7faa58.txt b/smart-turn_local-coreml-smart-turn_7d7faa58.txt
new file mode 100644
index 0000000000000000000000000000000000000000..c6ba2144b0b4fe50c0a611e601fa901b510d0be4
--- /dev/null
+++ b/smart-turn_local-coreml-smart-turn_7d7faa58.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/smart-turn/local-coreml-smart-turn#local-model-setup
+Title: Local CoreML Smart Turn - Pipecat
+==================================================
+
+Local CoreML Smart Turn - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Smart Turn Detection Local CoreML Smart Turn Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Smart Turn Overview Fal Smart Turn Local CoreML Smart Turn Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview LocalCoreMLSmartTurnAnalyzer runs Smart Turn inference directly on your Mac using Apple’s CoreML framework. This provides low-latency inference without external API dependencies, making it ideal for development and applications where network access is limited or latency is critical. ​ Installation Copy Ask AI pip install "pipecat-ai[local-smart-turn]" ​ Requirements Apple Silicon Mac (M1/M2/M3 series) macOS 11.0 or later ​ Local Model Setup To use the LocalCoreMLSmartTurnAnalyzer , you need to set up the CoreML model locally: Install Git LFS (Large File Storage): macOS Ubuntu/Debian Copy Ask AI brew install git-lfs Initialize Git LFS Copy Ask AI git lfs install Clone the Smart Turn model repository: Copy Ask AI git clone https://huggingface.co/pipecat-ai/smart-turn Set the environment variable to the cloned repository path: Copy Ask AI # Add to your .env file or environment export LOCAL_SMART_TURN_MODEL_PATH = / path / to / smart-turn ​ Configuration ​ Constructor Parameters ​ smart_turn_model_path str required Path to the directory containing the Smart Turn model files ​ sample_rate Optional[int] default: "None" Audio sample rate (will be set by the transport if not provided) ​ params SmartTurnParams default: "SmartTurnParams()" Configuration parameters for turn detection. See SmartTurnParams for details. ​ Example Copy Ask AI import os from pipecat.audio.turn.smart_turn.local_coreml_smart_turn import LocalCoreMLSmartTurnAnalyzer from pipecat.audio.turn.smart_turn.base_smart_turn import SmartTurnParams from pipecat.audio.vad.silero import SileroVADAnalyzer from pipecat.audio.vad.vad_analyzer import VADParams from pipecat.transports.base_transport import TransportParams # Get the path to the Smart Turn model smart_turn_model_path = os.getenv( "LOCAL_SMART_TURN_MODEL_PATH" ) # Create transport with local Smart Turn detection transport = SmallWebRTCTransport( webrtc_connection = webrtc_connection, params = TransportParams( audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer( params = VADParams( stop_secs = 0.2 )), turn_analyzer = LocalCoreMLSmartTurnAnalyzer( smart_turn_model_path = smart_turn_model_path, params = SmartTurnParams( stop_secs = 2.0 , # Shorter stop time when using Smart Turn pre_speech_ms = 0.0 , max_duration_secs = 8.0 ) ), ), ) ​ Performance Considerations Latency : Very low latency since inference happens locally Resource Usage : Uses local CPU/GPU resources Reliability : No dependency on external services or network connectivity ​ Notes Optimal for development environments and latency-sensitive applications The CoreML model is optimized for Apple Silicon but will work on Intel Macs with reduced performance First inference may be slower as the model is loaded and compiled Fal Smart Turn Watchdog Timers On this page Overview Installation Requirements Local Model Setup Configuration Constructor Parameters Example Performance Considerations Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/smart-turn_local-coreml-smart-turn_82be0862.txt b/smart-turn_local-coreml-smart-turn_82be0862.txt
new file mode 100644
index 0000000000000000000000000000000000000000..c0174a0eeae2bd279a5b6f34d573f5628d99e26f
--- /dev/null
+++ b/smart-turn_local-coreml-smart-turn_82be0862.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/smart-turn/local-coreml-smart-turn#requirements
+Title: Local CoreML Smart Turn - Pipecat
+==================================================
+
+Local CoreML Smart Turn - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Smart Turn Detection Local CoreML Smart Turn Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Smart Turn Overview Fal Smart Turn Local CoreML Smart Turn Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview LocalCoreMLSmartTurnAnalyzer runs Smart Turn inference directly on your Mac using Apple’s CoreML framework. This provides low-latency inference without external API dependencies, making it ideal for development and applications where network access is limited or latency is critical. ​ Installation Copy Ask AI pip install "pipecat-ai[local-smart-turn]" ​ Requirements Apple Silicon Mac (M1/M2/M3 series) macOS 11.0 or later ​ Local Model Setup To use the LocalCoreMLSmartTurnAnalyzer , you need to set up the CoreML model locally: Install Git LFS (Large File Storage): macOS Ubuntu/Debian Copy Ask AI brew install git-lfs Initialize Git LFS Copy Ask AI git lfs install Clone the Smart Turn model repository: Copy Ask AI git clone https://huggingface.co/pipecat-ai/smart-turn Set the environment variable to the cloned repository path: Copy Ask AI # Add to your .env file or environment export LOCAL_SMART_TURN_MODEL_PATH = / path / to / smart-turn ​ Configuration ​ Constructor Parameters ​ smart_turn_model_path str required Path to the directory containing the Smart Turn model files ​ sample_rate Optional[int] default: "None" Audio sample rate (will be set by the transport if not provided) ​ params SmartTurnParams default: "SmartTurnParams()" Configuration parameters for turn detection. See SmartTurnParams for details. ​ Example Copy Ask AI import os from pipecat.audio.turn.smart_turn.local_coreml_smart_turn import LocalCoreMLSmartTurnAnalyzer from pipecat.audio.turn.smart_turn.base_smart_turn import SmartTurnParams from pipecat.audio.vad.silero import SileroVADAnalyzer from pipecat.audio.vad.vad_analyzer import VADParams from pipecat.transports.base_transport import TransportParams # Get the path to the Smart Turn model smart_turn_model_path = os.getenv( "LOCAL_SMART_TURN_MODEL_PATH" ) # Create transport with local Smart Turn detection transport = SmallWebRTCTransport( webrtc_connection = webrtc_connection, params = TransportParams( audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer( params = VADParams( stop_secs = 0.2 )), turn_analyzer = LocalCoreMLSmartTurnAnalyzer( smart_turn_model_path = smart_turn_model_path, params = SmartTurnParams( stop_secs = 2.0 , # Shorter stop time when using Smart Turn pre_speech_ms = 0.0 , max_duration_secs = 8.0 ) ), ), ) ​ Performance Considerations Latency : Very low latency since inference happens locally Resource Usage : Uses local CPU/GPU resources Reliability : No dependency on external services or network connectivity ​ Notes Optimal for development environments and latency-sensitive applications The CoreML model is optimized for Apple Silicon but will work on Intel Macs with reduced performance First inference may be slower as the model is loaded and compiled Fal Smart Turn Watchdog Timers On this page Overview Installation Requirements Local Model Setup Configuration Constructor Parameters Example Performance Considerations Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/smart-turn_local-coreml-smart-turn_b454738f.txt b/smart-turn_local-coreml-smart-turn_b454738f.txt
new file mode 100644
index 0000000000000000000000000000000000000000..a834260529db768b8d50ffb6aff53b4cdb3097eb
--- /dev/null
+++ b/smart-turn_local-coreml-smart-turn_b454738f.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/smart-turn/local-coreml-smart-turn#example
+Title: Local CoreML Smart Turn - Pipecat
+==================================================
+
+Local CoreML Smart Turn - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Smart Turn Detection Local CoreML Smart Turn Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Smart Turn Overview Fal Smart Turn Local CoreML Smart Turn Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview LocalCoreMLSmartTurnAnalyzer runs Smart Turn inference directly on your Mac using Apple’s CoreML framework. This provides low-latency inference without external API dependencies, making it ideal for development and applications where network access is limited or latency is critical. ​ Installation Copy Ask AI pip install "pipecat-ai[local-smart-turn]" ​ Requirements Apple Silicon Mac (M1/M2/M3 series) macOS 11.0 or later ​ Local Model Setup To use the LocalCoreMLSmartTurnAnalyzer , you need to set up the CoreML model locally: Install Git LFS (Large File Storage): macOS Ubuntu/Debian Copy Ask AI brew install git-lfs Initialize Git LFS Copy Ask AI git lfs install Clone the Smart Turn model repository: Copy Ask AI git clone https://huggingface.co/pipecat-ai/smart-turn Set the environment variable to the cloned repository path: Copy Ask AI # Add to your .env file or environment export LOCAL_SMART_TURN_MODEL_PATH = / path / to / smart-turn ​ Configuration ​ Constructor Parameters ​ smart_turn_model_path str required Path to the directory containing the Smart Turn model files ​ sample_rate Optional[int] default: "None" Audio sample rate (will be set by the transport if not provided) ​ params SmartTurnParams default: "SmartTurnParams()" Configuration parameters for turn detection. See SmartTurnParams for details. ​ Example Copy Ask AI import os from pipecat.audio.turn.smart_turn.local_coreml_smart_turn import LocalCoreMLSmartTurnAnalyzer from pipecat.audio.turn.smart_turn.base_smart_turn import SmartTurnParams from pipecat.audio.vad.silero import SileroVADAnalyzer from pipecat.audio.vad.vad_analyzer import VADParams from pipecat.transports.base_transport import TransportParams # Get the path to the Smart Turn model smart_turn_model_path = os.getenv( "LOCAL_SMART_TURN_MODEL_PATH" ) # Create transport with local Smart Turn detection transport = SmallWebRTCTransport( webrtc_connection = webrtc_connection, params = TransportParams( audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer( params = VADParams( stop_secs = 0.2 )), turn_analyzer = LocalCoreMLSmartTurnAnalyzer( smart_turn_model_path = smart_turn_model_path, params = SmartTurnParams( stop_secs = 2.0 , # Shorter stop time when using Smart Turn pre_speech_ms = 0.0 , max_duration_secs = 8.0 ) ), ), ) ​ Performance Considerations Latency : Very low latency since inference happens locally Resource Usage : Uses local CPU/GPU resources Reliability : No dependency on external services or network connectivity ​ Notes Optimal for development environments and latency-sensitive applications The CoreML model is optimized for Apple Silicon but will work on Intel Macs with reduced performance First inference may be slower as the model is loaded and compiled Fal Smart Turn Watchdog Timers On this page Overview Installation Requirements Local Model Setup Configuration Constructor Parameters Example Performance Considerations Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/smart-turn_smart-turn-overview_1432caf4.txt b/smart-turn_smart-turn-overview_1432caf4.txt
new file mode 100644
index 0000000000000000000000000000000000000000..8feeaed1a6bae264a35c1fa9b4f29d4519959efb
--- /dev/null
+++ b/smart-turn_smart-turn-overview_1432caf4.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/smart-turn/smart-turn-overview#coreml-model-%26-pytorch-setup
+Title: Smart Turn Overview - Pipecat
+==================================================
+
+Smart Turn Overview - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Smart Turn Detection Smart Turn Overview Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Smart Turn Overview Fal Smart Turn Local CoreML Smart Turn Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview Smart Turn Detection is an advanced feature in Pipecat that determines when a user has finished speaking and the bot should respond. Unlike basic Voice Activity Detection (VAD) which only detects speech vs. non-speech, Smart Turn Detection uses a machine learning model to recognize natural conversational cues like intonation patterns and linguistic signals. Smart Turn Model Open source model for advanced conversational turn detection. Contribute to model training and development. Data Collector Contribute conversational data to improve the smart-turn model Data Classifier Help classify turn completion patterns in conversations Pipecat provides three implementations of Smart Turn Detection: FalSmartTurnAnalyzer - Uses a Fal’s hosted smart-turn model for inference LocalCoreMLSmartTurnAnalyzer - Runs inference locally on Apple Silicon using CoreML LocalSmartTurnAnalyzer - Runs inference locally using PyTorch and Hugging Face Transformers All implementations share the same underlying API and parameters, making it easy to switch between them based on your deployment requirements. ​ Installation The Smart Turn Detection feature requires additional dependencies depending on which implementation you choose. For Fal’s hosted service inference: Copy Ask AI pip install "pipecat-ai[remote-smart-turn]" For local inference (CoreML-based): Copy Ask AI pip install "pipecat-ai[local-smart-turn]" For local inference (PyTorch-based): Copy Ask AI pip install "pipecat-ai[local-smart-turn]" ​ Integration with Transport Smart Turn Detection is integrated into your application by setting one of the available turn analyzers as the turn_analyzer parameter in your transport configuration: Copy Ask AI from pipecat.transports.base_transport import TransportParams transport = SmallWebRTCTransport( webrtc_connection = webrtc_connection, params = TransportParams( # Other transport parameters... turn_analyzer = FalSmartTurnAnalyzer( url = remote_smart_turn_url), ), ) Smart Turn Detection requires VAD to be enabled and works best when the VAD analyzer is set to a short stop_secs value. We recommend 0.2 seconds. Copy Ask AI audio_in_enabled = True , vad_analyzer = SileroVADAnalyzer( params = VADParams( stop_secs = 0.2 )) ​ Configuration All implementations use the same SmartTurnParams class to configure behavior: ​ stop_secs float default: "3.0" Duration of silence in seconds required before triggering a silence-based end of turn ​ pre_speech_ms float default: "0.0" Amount of audio (in milliseconds) to include before speech is detected ​ max_duration_secs float default: "8.0" Maximum allowed segment duration in seconds. For segments longer than this value, a rolling window is used. ​ Remote Smart Turn The FalSmartTurnAnalyzer class uses a remote service for turn detection inference. ​ Constructor Parameters ​ url str required The URL of the remote Smart Turn service ​ sample_rate Optional[int] default: "None" Audio sample rate (will be set by the transport if not provided) ​ params SmartTurnParams default: "SmartTurnParams()" Configuration parameters for turn detection ​ Example Copy Ask AI import os from pipecat.audio.turn.smart_turn.fal_smart_turn import FalSmartTurnAnalyzer from pipecat.audio.turn.smart_turn.base_smart_turn import SmartTurnParams from pipecat.transports.base_transport import TransportParams # Get the URL for the remote Smart Turn service remote_smart_turn_url = os.getenv( "REMOTE_SMART_TURN_URL" ) # Create the transport with Smart Turn detection transport = SmallWebRTCTransport( webrtc_connection = webrtc_connection, params = TransportParams( audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer( params = VADParams( stop_secs = 0.2 )), turn_analyzer = FalSmartTurnAnalyzer( url = remote_smart_turn_url, params = SmartTurnParams( stop_secs = 3.0 , pre_speech_ms = 0.0 , max_duration_secs = 8.0 ) ), ), ) ​ Local Smart Turn (CoreML) The LocalCoreMLSmartTurnAnalyzer runs inference locally using CoreML, providing lower latency and no network dependencies. ​ Constructor Parameters ​ smart_turn_model_path str required Path to the directory containing the Smart Turn model ​ sample_rate Optional[int] default: "None" Audio sample rate (will be set by the transport if not provided) ​ params SmartTurnParams default: "SmartTurnParams()" Configuration parameters for turn detection ​ Example Copy Ask AI import os from pipecat.audio.turn.smart_turn.local_coreml_smart_turn import LocalCoreMLSmartTurnAnalyzer from pipecat.audio.turn.smart_turn.base_smart_turn import SmartTurnParams from pipecat.transports.base_transport import TransportParams # Path to the Smart Turn model directory smart_turn_model_path = os.getenv( "LOCAL_SMART_TURN_MODEL_PATH" ) # Create the transport with local Smart Turn detection transport = SmallWebRTCTransport( webrtc_connection = webrtc_connection, params = TransportParams( audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer( params = VADParams( stop_secs = 0.2 )), turn_analyzer = LocalCoreMLSmartTurnAnalyzer( smart_turn_model_path = smart_turn_model_path, params = SmartTurnParams( stop_secs = 2.0 , # Shorter stop time when using Smart Turn pre_speech_ms = 0.0 , max_duration_secs = 8.0 ) ), ), ) ​ Local Smart Turn (PyTorch) The LocalSmartTurnAnalyzer runs inference locally using PyTorch and Hugging Face Transformers, providing a cross-platform solution. ​ Constructor Parameters ​ smart_turn_model_path str default: "pipecat-ai/smart-turn" Path to the Smart Turn model or Hugging Face model identifier. Defaults to the official “pipecat-ai/smart-turn” model. ​ sample_rate Optional[int] default: "None" Audio sample rate (will be set by the transport if not provided) ​ params SmartTurnParams default: "SmartTurnParams()" Configuration parameters for turn detection ​ Example Copy Ask AI import os from pipecat.audio.turn.smart_turn.local_smart_turn import LocalSmartTurnAnalyzer from pipecat.audio.turn.smart_turn.base_smart_turn import SmartTurnParams from pipecat.transports.base_transport import TransportParams # Optional: Path to the local Smart Turn model # If not provided, it will download from Hugging Face smart_turn_model_path = os.getenv( "LOCAL_SMART_TURN_MODEL_PATH" ) # Create the transport with PyTorch-based Smart Turn detection transport = SmallWebRTCTransport( webrtc_connection = webrtc_connection, params = TransportParams( audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer( params = VADParams( stop_secs = 0.2 )), turn_analyzer = LocalSmartTurnAnalyzer( smart_turn_model_path = smart_turn_model_path, params = SmartTurnParams( stop_secs = 2.0 , pre_speech_ms = 0.0 , max_duration_secs = 8.0 ) ), ), ) ​ Local Model Setup ​ CoreML Model & PyTorch Setup To use the LocalCoreMLSmartTurnAnalyzer or LocalSmartTurnAnalyzer , you need to set up the model locally: Install Git LFS (Large File Storage): macOS Ubuntu/Debian Copy Ask AI brew install git-lfs Initialize Git LFS Copy Ask AI git lfs install Clone the Smart Turn model repository: Copy Ask AI git clone https://huggingface.co/pipecat-ai/smart-turn Set the environment variable to the cloned repository path: Copy Ask AI # Add to your .env file or environment export LOCAL_SMART_TURN_MODEL_PATH = / path / to / smart-turn Note that the CoreML model is optimized for Apple Silicon devices. If you’re using a different platform, consider using the PyTorch-based LocalSmartTurnAnalyzer or the remote Smart Turn service. Learn more about the CoreML setup in the official repository instructions ​ How It Works Smart Turn Detection continuously analyzes audio streams to identify natural turn completion points: Audio Buffering : The system continuously buffers audio with timestamps, maintaining a small buffer of pre-speech audio. VAD Processing : Voice Activity Detection segments the audio into speech and non-speech portions. Turn Analysis : When VAD detects a pause in speech: The ML model analyzes the speech segment for natural completion cues It identifies acoustic and linguistic patterns that indicate turn completion A decision is made whether the turn is complete or incomplete The system includes a fallback mechanism: if a turn is classified as incomplete but silence continues for longer than stop_secs , the turn is automatically marked as complete. ​ Notes The model is designed for English speech; performance may vary with other languages You can adjust the stop_secs parameter based on your application’s needs for responsiveness Smart Turn generally provides a more natural conversational experience but is computationally more intensive than simple VAD The PyTorch-based LocalSmartTurnAnalyzer runs on CPU by default but will use CUDA if available Daily REST Helper Fal Smart Turn On this page Overview Installation Integration with Transport Configuration Remote Smart Turn Constructor Parameters Example Local Smart Turn (CoreML) Constructor Parameters Example Local Smart Turn (PyTorch) Constructor Parameters Example Local Model Setup CoreML Model & PyTorch Setup How It Works Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/smart-turn_smart-turn-overview_2e761b58.txt b/smart-turn_smart-turn-overview_2e761b58.txt
new file mode 100644
index 0000000000000000000000000000000000000000..a132f7e75b0033b9d2f909deb908012b0d749b3c
--- /dev/null
+++ b/smart-turn_smart-turn-overview_2e761b58.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/smart-turn/smart-turn-overview#param-smart-turn-model-path-1
+Title: Smart Turn Overview - Pipecat
+==================================================
+
+Smart Turn Overview - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Smart Turn Detection Smart Turn Overview Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Smart Turn Overview Fal Smart Turn Local CoreML Smart Turn Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview Smart Turn Detection is an advanced feature in Pipecat that determines when a user has finished speaking and the bot should respond. Unlike basic Voice Activity Detection (VAD) which only detects speech vs. non-speech, Smart Turn Detection uses a machine learning model to recognize natural conversational cues like intonation patterns and linguistic signals. Smart Turn Model Open source model for advanced conversational turn detection. Contribute to model training and development. Data Collector Contribute conversational data to improve the smart-turn model Data Classifier Help classify turn completion patterns in conversations Pipecat provides three implementations of Smart Turn Detection: FalSmartTurnAnalyzer - Uses a Fal’s hosted smart-turn model for inference LocalCoreMLSmartTurnAnalyzer - Runs inference locally on Apple Silicon using CoreML LocalSmartTurnAnalyzer - Runs inference locally using PyTorch and Hugging Face Transformers All implementations share the same underlying API and parameters, making it easy to switch between them based on your deployment requirements. ​ Installation The Smart Turn Detection feature requires additional dependencies depending on which implementation you choose. For Fal’s hosted service inference: Copy Ask AI pip install "pipecat-ai[remote-smart-turn]" For local inference (CoreML-based): Copy Ask AI pip install "pipecat-ai[local-smart-turn]" For local inference (PyTorch-based): Copy Ask AI pip install "pipecat-ai[local-smart-turn]" ​ Integration with Transport Smart Turn Detection is integrated into your application by setting one of the available turn analyzers as the turn_analyzer parameter in your transport configuration: Copy Ask AI from pipecat.transports.base_transport import TransportParams transport = SmallWebRTCTransport( webrtc_connection = webrtc_connection, params = TransportParams( # Other transport parameters... turn_analyzer = FalSmartTurnAnalyzer( url = remote_smart_turn_url), ), ) Smart Turn Detection requires VAD to be enabled and works best when the VAD analyzer is set to a short stop_secs value. We recommend 0.2 seconds. Copy Ask AI audio_in_enabled = True , vad_analyzer = SileroVADAnalyzer( params = VADParams( stop_secs = 0.2 )) ​ Configuration All implementations use the same SmartTurnParams class to configure behavior: ​ stop_secs float default: "3.0" Duration of silence in seconds required before triggering a silence-based end of turn ​ pre_speech_ms float default: "0.0" Amount of audio (in milliseconds) to include before speech is detected ​ max_duration_secs float default: "8.0" Maximum allowed segment duration in seconds. For segments longer than this value, a rolling window is used. ​ Remote Smart Turn The FalSmartTurnAnalyzer class uses a remote service for turn detection inference. ​ Constructor Parameters ​ url str required The URL of the remote Smart Turn service ​ sample_rate Optional[int] default: "None" Audio sample rate (will be set by the transport if not provided) ​ params SmartTurnParams default: "SmartTurnParams()" Configuration parameters for turn detection ​ Example Copy Ask AI import os from pipecat.audio.turn.smart_turn.fal_smart_turn import FalSmartTurnAnalyzer from pipecat.audio.turn.smart_turn.base_smart_turn import SmartTurnParams from pipecat.transports.base_transport import TransportParams # Get the URL for the remote Smart Turn service remote_smart_turn_url = os.getenv( "REMOTE_SMART_TURN_URL" ) # Create the transport with Smart Turn detection transport = SmallWebRTCTransport( webrtc_connection = webrtc_connection, params = TransportParams( audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer( params = VADParams( stop_secs = 0.2 )), turn_analyzer = FalSmartTurnAnalyzer( url = remote_smart_turn_url, params = SmartTurnParams( stop_secs = 3.0 , pre_speech_ms = 0.0 , max_duration_secs = 8.0 ) ), ), ) ​ Local Smart Turn (CoreML) The LocalCoreMLSmartTurnAnalyzer runs inference locally using CoreML, providing lower latency and no network dependencies. ​ Constructor Parameters ​ smart_turn_model_path str required Path to the directory containing the Smart Turn model ​ sample_rate Optional[int] default: "None" Audio sample rate (will be set by the transport if not provided) ​ params SmartTurnParams default: "SmartTurnParams()" Configuration parameters for turn detection ​ Example Copy Ask AI import os from pipecat.audio.turn.smart_turn.local_coreml_smart_turn import LocalCoreMLSmartTurnAnalyzer from pipecat.audio.turn.smart_turn.base_smart_turn import SmartTurnParams from pipecat.transports.base_transport import TransportParams # Path to the Smart Turn model directory smart_turn_model_path = os.getenv( "LOCAL_SMART_TURN_MODEL_PATH" ) # Create the transport with local Smart Turn detection transport = SmallWebRTCTransport( webrtc_connection = webrtc_connection, params = TransportParams( audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer( params = VADParams( stop_secs = 0.2 )), turn_analyzer = LocalCoreMLSmartTurnAnalyzer( smart_turn_model_path = smart_turn_model_path, params = SmartTurnParams( stop_secs = 2.0 , # Shorter stop time when using Smart Turn pre_speech_ms = 0.0 , max_duration_secs = 8.0 ) ), ), ) ​ Local Smart Turn (PyTorch) The LocalSmartTurnAnalyzer runs inference locally using PyTorch and Hugging Face Transformers, providing a cross-platform solution. ​ Constructor Parameters ​ smart_turn_model_path str default: "pipecat-ai/smart-turn" Path to the Smart Turn model or Hugging Face model identifier. Defaults to the official “pipecat-ai/smart-turn” model. ​ sample_rate Optional[int] default: "None" Audio sample rate (will be set by the transport if not provided) ​ params SmartTurnParams default: "SmartTurnParams()" Configuration parameters for turn detection ​ Example Copy Ask AI import os from pipecat.audio.turn.smart_turn.local_smart_turn import LocalSmartTurnAnalyzer from pipecat.audio.turn.smart_turn.base_smart_turn import SmartTurnParams from pipecat.transports.base_transport import TransportParams # Optional: Path to the local Smart Turn model # If not provided, it will download from Hugging Face smart_turn_model_path = os.getenv( "LOCAL_SMART_TURN_MODEL_PATH" ) # Create the transport with PyTorch-based Smart Turn detection transport = SmallWebRTCTransport( webrtc_connection = webrtc_connection, params = TransportParams( audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer( params = VADParams( stop_secs = 0.2 )), turn_analyzer = LocalSmartTurnAnalyzer( smart_turn_model_path = smart_turn_model_path, params = SmartTurnParams( stop_secs = 2.0 , pre_speech_ms = 0.0 , max_duration_secs = 8.0 ) ), ), ) ​ Local Model Setup ​ CoreML Model & PyTorch Setup To use the LocalCoreMLSmartTurnAnalyzer or LocalSmartTurnAnalyzer , you need to set up the model locally: Install Git LFS (Large File Storage): macOS Ubuntu/Debian Copy Ask AI brew install git-lfs Initialize Git LFS Copy Ask AI git lfs install Clone the Smart Turn model repository: Copy Ask AI git clone https://huggingface.co/pipecat-ai/smart-turn Set the environment variable to the cloned repository path: Copy Ask AI # Add to your .env file or environment export LOCAL_SMART_TURN_MODEL_PATH = / path / to / smart-turn Note that the CoreML model is optimized for Apple Silicon devices. If you’re using a different platform, consider using the PyTorch-based LocalSmartTurnAnalyzer or the remote Smart Turn service. Learn more about the CoreML setup in the official repository instructions ​ How It Works Smart Turn Detection continuously analyzes audio streams to identify natural turn completion points: Audio Buffering : The system continuously buffers audio with timestamps, maintaining a small buffer of pre-speech audio. VAD Processing : Voice Activity Detection segments the audio into speech and non-speech portions. Turn Analysis : When VAD detects a pause in speech: The ML model analyzes the speech segment for natural completion cues It identifies acoustic and linguistic patterns that indicate turn completion A decision is made whether the turn is complete or incomplete The system includes a fallback mechanism: if a turn is classified as incomplete but silence continues for longer than stop_secs , the turn is automatically marked as complete. ​ Notes The model is designed for English speech; performance may vary with other languages You can adjust the stop_secs parameter based on your application’s needs for responsiveness Smart Turn generally provides a more natural conversational experience but is computationally more intensive than simple VAD The PyTorch-based LocalSmartTurnAnalyzer runs on CPU by default but will use CUDA if available Daily REST Helper Fal Smart Turn On this page Overview Installation Integration with Transport Configuration Remote Smart Turn Constructor Parameters Example Local Smart Turn (CoreML) Constructor Parameters Example Local Smart Turn (PyTorch) Constructor Parameters Example Local Model Setup CoreML Model & PyTorch Setup How It Works Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/smart-turn_smart-turn-overview_69b329a5.txt b/smart-turn_smart-turn-overview_69b329a5.txt
new file mode 100644
index 0000000000000000000000000000000000000000..96ea2a914575823cf01476ec19f424f0253ef2fa
--- /dev/null
+++ b/smart-turn_smart-turn-overview_69b329a5.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/smart-turn/smart-turn-overview#param-sample-rate-1
+Title: Smart Turn Overview - Pipecat
+==================================================
+
+Smart Turn Overview - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Smart Turn Detection Smart Turn Overview Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Smart Turn Overview Fal Smart Turn Local CoreML Smart Turn Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview Smart Turn Detection is an advanced feature in Pipecat that determines when a user has finished speaking and the bot should respond. Unlike basic Voice Activity Detection (VAD) which only detects speech vs. non-speech, Smart Turn Detection uses a machine learning model to recognize natural conversational cues like intonation patterns and linguistic signals. Smart Turn Model Open source model for advanced conversational turn detection. Contribute to model training and development. Data Collector Contribute conversational data to improve the smart-turn model Data Classifier Help classify turn completion patterns in conversations Pipecat provides three implementations of Smart Turn Detection: FalSmartTurnAnalyzer - Uses a Fal’s hosted smart-turn model for inference LocalCoreMLSmartTurnAnalyzer - Runs inference locally on Apple Silicon using CoreML LocalSmartTurnAnalyzer - Runs inference locally using PyTorch and Hugging Face Transformers All implementations share the same underlying API and parameters, making it easy to switch between them based on your deployment requirements. ​ Installation The Smart Turn Detection feature requires additional dependencies depending on which implementation you choose. For Fal’s hosted service inference: Copy Ask AI pip install "pipecat-ai[remote-smart-turn]" For local inference (CoreML-based): Copy Ask AI pip install "pipecat-ai[local-smart-turn]" For local inference (PyTorch-based): Copy Ask AI pip install "pipecat-ai[local-smart-turn]" ​ Integration with Transport Smart Turn Detection is integrated into your application by setting one of the available turn analyzers as the turn_analyzer parameter in your transport configuration: Copy Ask AI from pipecat.transports.base_transport import TransportParams transport = SmallWebRTCTransport( webrtc_connection = webrtc_connection, params = TransportParams( # Other transport parameters... turn_analyzer = FalSmartTurnAnalyzer( url = remote_smart_turn_url), ), ) Smart Turn Detection requires VAD to be enabled and works best when the VAD analyzer is set to a short stop_secs value. We recommend 0.2 seconds. Copy Ask AI audio_in_enabled = True , vad_analyzer = SileroVADAnalyzer( params = VADParams( stop_secs = 0.2 )) ​ Configuration All implementations use the same SmartTurnParams class to configure behavior: ​ stop_secs float default: "3.0" Duration of silence in seconds required before triggering a silence-based end of turn ​ pre_speech_ms float default: "0.0" Amount of audio (in milliseconds) to include before speech is detected ​ max_duration_secs float default: "8.0" Maximum allowed segment duration in seconds. For segments longer than this value, a rolling window is used. ​ Remote Smart Turn The FalSmartTurnAnalyzer class uses a remote service for turn detection inference. ​ Constructor Parameters ​ url str required The URL of the remote Smart Turn service ​ sample_rate Optional[int] default: "None" Audio sample rate (will be set by the transport if not provided) ​ params SmartTurnParams default: "SmartTurnParams()" Configuration parameters for turn detection ​ Example Copy Ask AI import os from pipecat.audio.turn.smart_turn.fal_smart_turn import FalSmartTurnAnalyzer from pipecat.audio.turn.smart_turn.base_smart_turn import SmartTurnParams from pipecat.transports.base_transport import TransportParams # Get the URL for the remote Smart Turn service remote_smart_turn_url = os.getenv( "REMOTE_SMART_TURN_URL" ) # Create the transport with Smart Turn detection transport = SmallWebRTCTransport( webrtc_connection = webrtc_connection, params = TransportParams( audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer( params = VADParams( stop_secs = 0.2 )), turn_analyzer = FalSmartTurnAnalyzer( url = remote_smart_turn_url, params = SmartTurnParams( stop_secs = 3.0 , pre_speech_ms = 0.0 , max_duration_secs = 8.0 ) ), ), ) ​ Local Smart Turn (CoreML) The LocalCoreMLSmartTurnAnalyzer runs inference locally using CoreML, providing lower latency and no network dependencies. ​ Constructor Parameters ​ smart_turn_model_path str required Path to the directory containing the Smart Turn model ​ sample_rate Optional[int] default: "None" Audio sample rate (will be set by the transport if not provided) ​ params SmartTurnParams default: "SmartTurnParams()" Configuration parameters for turn detection ​ Example Copy Ask AI import os from pipecat.audio.turn.smart_turn.local_coreml_smart_turn import LocalCoreMLSmartTurnAnalyzer from pipecat.audio.turn.smart_turn.base_smart_turn import SmartTurnParams from pipecat.transports.base_transport import TransportParams # Path to the Smart Turn model directory smart_turn_model_path = os.getenv( "LOCAL_SMART_TURN_MODEL_PATH" ) # Create the transport with local Smart Turn detection transport = SmallWebRTCTransport( webrtc_connection = webrtc_connection, params = TransportParams( audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer( params = VADParams( stop_secs = 0.2 )), turn_analyzer = LocalCoreMLSmartTurnAnalyzer( smart_turn_model_path = smart_turn_model_path, params = SmartTurnParams( stop_secs = 2.0 , # Shorter stop time when using Smart Turn pre_speech_ms = 0.0 , max_duration_secs = 8.0 ) ), ), ) ​ Local Smart Turn (PyTorch) The LocalSmartTurnAnalyzer runs inference locally using PyTorch and Hugging Face Transformers, providing a cross-platform solution. ​ Constructor Parameters ​ smart_turn_model_path str default: "pipecat-ai/smart-turn" Path to the Smart Turn model or Hugging Face model identifier. Defaults to the official “pipecat-ai/smart-turn” model. ​ sample_rate Optional[int] default: "None" Audio sample rate (will be set by the transport if not provided) ​ params SmartTurnParams default: "SmartTurnParams()" Configuration parameters for turn detection ​ Example Copy Ask AI import os from pipecat.audio.turn.smart_turn.local_smart_turn import LocalSmartTurnAnalyzer from pipecat.audio.turn.smart_turn.base_smart_turn import SmartTurnParams from pipecat.transports.base_transport import TransportParams # Optional: Path to the local Smart Turn model # If not provided, it will download from Hugging Face smart_turn_model_path = os.getenv( "LOCAL_SMART_TURN_MODEL_PATH" ) # Create the transport with PyTorch-based Smart Turn detection transport = SmallWebRTCTransport( webrtc_connection = webrtc_connection, params = TransportParams( audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer( params = VADParams( stop_secs = 0.2 )), turn_analyzer = LocalSmartTurnAnalyzer( smart_turn_model_path = smart_turn_model_path, params = SmartTurnParams( stop_secs = 2.0 , pre_speech_ms = 0.0 , max_duration_secs = 8.0 ) ), ), ) ​ Local Model Setup ​ CoreML Model & PyTorch Setup To use the LocalCoreMLSmartTurnAnalyzer or LocalSmartTurnAnalyzer , you need to set up the model locally: Install Git LFS (Large File Storage): macOS Ubuntu/Debian Copy Ask AI brew install git-lfs Initialize Git LFS Copy Ask AI git lfs install Clone the Smart Turn model repository: Copy Ask AI git clone https://huggingface.co/pipecat-ai/smart-turn Set the environment variable to the cloned repository path: Copy Ask AI # Add to your .env file or environment export LOCAL_SMART_TURN_MODEL_PATH = / path / to / smart-turn Note that the CoreML model is optimized for Apple Silicon devices. If you’re using a different platform, consider using the PyTorch-based LocalSmartTurnAnalyzer or the remote Smart Turn service. Learn more about the CoreML setup in the official repository instructions ​ How It Works Smart Turn Detection continuously analyzes audio streams to identify natural turn completion points: Audio Buffering : The system continuously buffers audio with timestamps, maintaining a small buffer of pre-speech audio. VAD Processing : Voice Activity Detection segments the audio into speech and non-speech portions. Turn Analysis : When VAD detects a pause in speech: The ML model analyzes the speech segment for natural completion cues It identifies acoustic and linguistic patterns that indicate turn completion A decision is made whether the turn is complete or incomplete The system includes a fallback mechanism: if a turn is classified as incomplete but silence continues for longer than stop_secs , the turn is automatically marked as complete. ​ Notes The model is designed for English speech; performance may vary with other languages You can adjust the stop_secs parameter based on your application’s needs for responsiveness Smart Turn generally provides a more natural conversational experience but is computationally more intensive than simple VAD The PyTorch-based LocalSmartTurnAnalyzer runs on CPU by default but will use CUDA if available Daily REST Helper Fal Smart Turn On this page Overview Installation Integration with Transport Configuration Remote Smart Turn Constructor Parameters Example Local Smart Turn (CoreML) Constructor Parameters Example Local Smart Turn (PyTorch) Constructor Parameters Example Local Model Setup CoreML Model & PyTorch Setup How It Works Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/smart-turn_smart-turn-overview_884c427f.txt b/smart-turn_smart-turn-overview_884c427f.txt
new file mode 100644
index 0000000000000000000000000000000000000000..4e35fe68873ca6f03e680c495694d1da9929f3ee
--- /dev/null
+++ b/smart-turn_smart-turn-overview_884c427f.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/smart-turn/smart-turn-overview#param-sample-rate-2
+Title: Smart Turn Overview - Pipecat
+==================================================
+
+Smart Turn Overview - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Smart Turn Detection Smart Turn Overview Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Smart Turn Overview Fal Smart Turn Local CoreML Smart Turn Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview Smart Turn Detection is an advanced feature in Pipecat that determines when a user has finished speaking and the bot should respond. Unlike basic Voice Activity Detection (VAD) which only detects speech vs. non-speech, Smart Turn Detection uses a machine learning model to recognize natural conversational cues like intonation patterns and linguistic signals. Smart Turn Model Open source model for advanced conversational turn detection. Contribute to model training and development. Data Collector Contribute conversational data to improve the smart-turn model Data Classifier Help classify turn completion patterns in conversations Pipecat provides three implementations of Smart Turn Detection: FalSmartTurnAnalyzer - Uses a Fal’s hosted smart-turn model for inference LocalCoreMLSmartTurnAnalyzer - Runs inference locally on Apple Silicon using CoreML LocalSmartTurnAnalyzer - Runs inference locally using PyTorch and Hugging Face Transformers All implementations share the same underlying API and parameters, making it easy to switch between them based on your deployment requirements. ​ Installation The Smart Turn Detection feature requires additional dependencies depending on which implementation you choose. For Fal’s hosted service inference: Copy Ask AI pip install "pipecat-ai[remote-smart-turn]" For local inference (CoreML-based): Copy Ask AI pip install "pipecat-ai[local-smart-turn]" For local inference (PyTorch-based): Copy Ask AI pip install "pipecat-ai[local-smart-turn]" ​ Integration with Transport Smart Turn Detection is integrated into your application by setting one of the available turn analyzers as the turn_analyzer parameter in your transport configuration: Copy Ask AI from pipecat.transports.base_transport import TransportParams transport = SmallWebRTCTransport( webrtc_connection = webrtc_connection, params = TransportParams( # Other transport parameters... turn_analyzer = FalSmartTurnAnalyzer( url = remote_smart_turn_url), ), ) Smart Turn Detection requires VAD to be enabled and works best when the VAD analyzer is set to a short stop_secs value. We recommend 0.2 seconds. Copy Ask AI audio_in_enabled = True , vad_analyzer = SileroVADAnalyzer( params = VADParams( stop_secs = 0.2 )) ​ Configuration All implementations use the same SmartTurnParams class to configure behavior: ​ stop_secs float default: "3.0" Duration of silence in seconds required before triggering a silence-based end of turn ​ pre_speech_ms float default: "0.0" Amount of audio (in milliseconds) to include before speech is detected ​ max_duration_secs float default: "8.0" Maximum allowed segment duration in seconds. For segments longer than this value, a rolling window is used. ​ Remote Smart Turn The FalSmartTurnAnalyzer class uses a remote service for turn detection inference. ​ Constructor Parameters ​ url str required The URL of the remote Smart Turn service ​ sample_rate Optional[int] default: "None" Audio sample rate (will be set by the transport if not provided) ​ params SmartTurnParams default: "SmartTurnParams()" Configuration parameters for turn detection ​ Example Copy Ask AI import os from pipecat.audio.turn.smart_turn.fal_smart_turn import FalSmartTurnAnalyzer from pipecat.audio.turn.smart_turn.base_smart_turn import SmartTurnParams from pipecat.transports.base_transport import TransportParams # Get the URL for the remote Smart Turn service remote_smart_turn_url = os.getenv( "REMOTE_SMART_TURN_URL" ) # Create the transport with Smart Turn detection transport = SmallWebRTCTransport( webrtc_connection = webrtc_connection, params = TransportParams( audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer( params = VADParams( stop_secs = 0.2 )), turn_analyzer = FalSmartTurnAnalyzer( url = remote_smart_turn_url, params = SmartTurnParams( stop_secs = 3.0 , pre_speech_ms = 0.0 , max_duration_secs = 8.0 ) ), ), ) ​ Local Smart Turn (CoreML) The LocalCoreMLSmartTurnAnalyzer runs inference locally using CoreML, providing lower latency and no network dependencies. ​ Constructor Parameters ​ smart_turn_model_path str required Path to the directory containing the Smart Turn model ​ sample_rate Optional[int] default: "None" Audio sample rate (will be set by the transport if not provided) ​ params SmartTurnParams default: "SmartTurnParams()" Configuration parameters for turn detection ​ Example Copy Ask AI import os from pipecat.audio.turn.smart_turn.local_coreml_smart_turn import LocalCoreMLSmartTurnAnalyzer from pipecat.audio.turn.smart_turn.base_smart_turn import SmartTurnParams from pipecat.transports.base_transport import TransportParams # Path to the Smart Turn model directory smart_turn_model_path = os.getenv( "LOCAL_SMART_TURN_MODEL_PATH" ) # Create the transport with local Smart Turn detection transport = SmallWebRTCTransport( webrtc_connection = webrtc_connection, params = TransportParams( audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer( params = VADParams( stop_secs = 0.2 )), turn_analyzer = LocalCoreMLSmartTurnAnalyzer( smart_turn_model_path = smart_turn_model_path, params = SmartTurnParams( stop_secs = 2.0 , # Shorter stop time when using Smart Turn pre_speech_ms = 0.0 , max_duration_secs = 8.0 ) ), ), ) ​ Local Smart Turn (PyTorch) The LocalSmartTurnAnalyzer runs inference locally using PyTorch and Hugging Face Transformers, providing a cross-platform solution. ​ Constructor Parameters ​ smart_turn_model_path str default: "pipecat-ai/smart-turn" Path to the Smart Turn model or Hugging Face model identifier. Defaults to the official “pipecat-ai/smart-turn” model. ​ sample_rate Optional[int] default: "None" Audio sample rate (will be set by the transport if not provided) ​ params SmartTurnParams default: "SmartTurnParams()" Configuration parameters for turn detection ​ Example Copy Ask AI import os from pipecat.audio.turn.smart_turn.local_smart_turn import LocalSmartTurnAnalyzer from pipecat.audio.turn.smart_turn.base_smart_turn import SmartTurnParams from pipecat.transports.base_transport import TransportParams # Optional: Path to the local Smart Turn model # If not provided, it will download from Hugging Face smart_turn_model_path = os.getenv( "LOCAL_SMART_TURN_MODEL_PATH" ) # Create the transport with PyTorch-based Smart Turn detection transport = SmallWebRTCTransport( webrtc_connection = webrtc_connection, params = TransportParams( audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer( params = VADParams( stop_secs = 0.2 )), turn_analyzer = LocalSmartTurnAnalyzer( smart_turn_model_path = smart_turn_model_path, params = SmartTurnParams( stop_secs = 2.0 , pre_speech_ms = 0.0 , max_duration_secs = 8.0 ) ), ), ) ​ Local Model Setup ​ CoreML Model & PyTorch Setup To use the LocalCoreMLSmartTurnAnalyzer or LocalSmartTurnAnalyzer , you need to set up the model locally: Install Git LFS (Large File Storage): macOS Ubuntu/Debian Copy Ask AI brew install git-lfs Initialize Git LFS Copy Ask AI git lfs install Clone the Smart Turn model repository: Copy Ask AI git clone https://huggingface.co/pipecat-ai/smart-turn Set the environment variable to the cloned repository path: Copy Ask AI # Add to your .env file or environment export LOCAL_SMART_TURN_MODEL_PATH = / path / to / smart-turn Note that the CoreML model is optimized for Apple Silicon devices. If you’re using a different platform, consider using the PyTorch-based LocalSmartTurnAnalyzer or the remote Smart Turn service. Learn more about the CoreML setup in the official repository instructions ​ How It Works Smart Turn Detection continuously analyzes audio streams to identify natural turn completion points: Audio Buffering : The system continuously buffers audio with timestamps, maintaining a small buffer of pre-speech audio. VAD Processing : Voice Activity Detection segments the audio into speech and non-speech portions. Turn Analysis : When VAD detects a pause in speech: The ML model analyzes the speech segment for natural completion cues It identifies acoustic and linguistic patterns that indicate turn completion A decision is made whether the turn is complete or incomplete The system includes a fallback mechanism: if a turn is classified as incomplete but silence continues for longer than stop_secs , the turn is automatically marked as complete. ​ Notes The model is designed for English speech; performance may vary with other languages You can adjust the stop_secs parameter based on your application’s needs for responsiveness Smart Turn generally provides a more natural conversational experience but is computationally more intensive than simple VAD The PyTorch-based LocalSmartTurnAnalyzer runs on CPU by default but will use CUDA if available Daily REST Helper Fal Smart Turn On this page Overview Installation Integration with Transport Configuration Remote Smart Turn Constructor Parameters Example Local Smart Turn (CoreML) Constructor Parameters Example Local Smart Turn (PyTorch) Constructor Parameters Example Local Model Setup CoreML Model & PyTorch Setup How It Works Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/smart-turn_smart-turn-overview_945bdac1.txt b/smart-turn_smart-turn-overview_945bdac1.txt
new file mode 100644
index 0000000000000000000000000000000000000000..32ada3035dfe88b2292fa042c20a665acf463252
--- /dev/null
+++ b/smart-turn_smart-turn-overview_945bdac1.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/smart-turn/smart-turn-overview#param-params-1
+Title: Smart Turn Overview - Pipecat
+==================================================
+
+Smart Turn Overview - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Smart Turn Detection Smart Turn Overview Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Smart Turn Overview Fal Smart Turn Local CoreML Smart Turn Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview Smart Turn Detection is an advanced feature in Pipecat that determines when a user has finished speaking and the bot should respond. Unlike basic Voice Activity Detection (VAD) which only detects speech vs. non-speech, Smart Turn Detection uses a machine learning model to recognize natural conversational cues like intonation patterns and linguistic signals. Smart Turn Model Open source model for advanced conversational turn detection. Contribute to model training and development. Data Collector Contribute conversational data to improve the smart-turn model Data Classifier Help classify turn completion patterns in conversations Pipecat provides three implementations of Smart Turn Detection: FalSmartTurnAnalyzer - Uses a Fal’s hosted smart-turn model for inference LocalCoreMLSmartTurnAnalyzer - Runs inference locally on Apple Silicon using CoreML LocalSmartTurnAnalyzer - Runs inference locally using PyTorch and Hugging Face Transformers All implementations share the same underlying API and parameters, making it easy to switch between them based on your deployment requirements. ​ Installation The Smart Turn Detection feature requires additional dependencies depending on which implementation you choose. For Fal’s hosted service inference: Copy Ask AI pip install "pipecat-ai[remote-smart-turn]" For local inference (CoreML-based): Copy Ask AI pip install "pipecat-ai[local-smart-turn]" For local inference (PyTorch-based): Copy Ask AI pip install "pipecat-ai[local-smart-turn]" ​ Integration with Transport Smart Turn Detection is integrated into your application by setting one of the available turn analyzers as the turn_analyzer parameter in your transport configuration: Copy Ask AI from pipecat.transports.base_transport import TransportParams transport = SmallWebRTCTransport( webrtc_connection = webrtc_connection, params = TransportParams( # Other transport parameters... turn_analyzer = FalSmartTurnAnalyzer( url = remote_smart_turn_url), ), ) Smart Turn Detection requires VAD to be enabled and works best when the VAD analyzer is set to a short stop_secs value. We recommend 0.2 seconds. Copy Ask AI audio_in_enabled = True , vad_analyzer = SileroVADAnalyzer( params = VADParams( stop_secs = 0.2 )) ​ Configuration All implementations use the same SmartTurnParams class to configure behavior: ​ stop_secs float default: "3.0" Duration of silence in seconds required before triggering a silence-based end of turn ​ pre_speech_ms float default: "0.0" Amount of audio (in milliseconds) to include before speech is detected ​ max_duration_secs float default: "8.0" Maximum allowed segment duration in seconds. For segments longer than this value, a rolling window is used. ​ Remote Smart Turn The FalSmartTurnAnalyzer class uses a remote service for turn detection inference. ​ Constructor Parameters ​ url str required The URL of the remote Smart Turn service ​ sample_rate Optional[int] default: "None" Audio sample rate (will be set by the transport if not provided) ​ params SmartTurnParams default: "SmartTurnParams()" Configuration parameters for turn detection ​ Example Copy Ask AI import os from pipecat.audio.turn.smart_turn.fal_smart_turn import FalSmartTurnAnalyzer from pipecat.audio.turn.smart_turn.base_smart_turn import SmartTurnParams from pipecat.transports.base_transport import TransportParams # Get the URL for the remote Smart Turn service remote_smart_turn_url = os.getenv( "REMOTE_SMART_TURN_URL" ) # Create the transport with Smart Turn detection transport = SmallWebRTCTransport( webrtc_connection = webrtc_connection, params = TransportParams( audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer( params = VADParams( stop_secs = 0.2 )), turn_analyzer = FalSmartTurnAnalyzer( url = remote_smart_turn_url, params = SmartTurnParams( stop_secs = 3.0 , pre_speech_ms = 0.0 , max_duration_secs = 8.0 ) ), ), ) ​ Local Smart Turn (CoreML) The LocalCoreMLSmartTurnAnalyzer runs inference locally using CoreML, providing lower latency and no network dependencies. ​ Constructor Parameters ​ smart_turn_model_path str required Path to the directory containing the Smart Turn model ​ sample_rate Optional[int] default: "None" Audio sample rate (will be set by the transport if not provided) ​ params SmartTurnParams default: "SmartTurnParams()" Configuration parameters for turn detection ​ Example Copy Ask AI import os from pipecat.audio.turn.smart_turn.local_coreml_smart_turn import LocalCoreMLSmartTurnAnalyzer from pipecat.audio.turn.smart_turn.base_smart_turn import SmartTurnParams from pipecat.transports.base_transport import TransportParams # Path to the Smart Turn model directory smart_turn_model_path = os.getenv( "LOCAL_SMART_TURN_MODEL_PATH" ) # Create the transport with local Smart Turn detection transport = SmallWebRTCTransport( webrtc_connection = webrtc_connection, params = TransportParams( audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer( params = VADParams( stop_secs = 0.2 )), turn_analyzer = LocalCoreMLSmartTurnAnalyzer( smart_turn_model_path = smart_turn_model_path, params = SmartTurnParams( stop_secs = 2.0 , # Shorter stop time when using Smart Turn pre_speech_ms = 0.0 , max_duration_secs = 8.0 ) ), ), ) ​ Local Smart Turn (PyTorch) The LocalSmartTurnAnalyzer runs inference locally using PyTorch and Hugging Face Transformers, providing a cross-platform solution. ​ Constructor Parameters ​ smart_turn_model_path str default: "pipecat-ai/smart-turn" Path to the Smart Turn model or Hugging Face model identifier. Defaults to the official “pipecat-ai/smart-turn” model. ​ sample_rate Optional[int] default: "None" Audio sample rate (will be set by the transport if not provided) ​ params SmartTurnParams default: "SmartTurnParams()" Configuration parameters for turn detection ​ Example Copy Ask AI import os from pipecat.audio.turn.smart_turn.local_smart_turn import LocalSmartTurnAnalyzer from pipecat.audio.turn.smart_turn.base_smart_turn import SmartTurnParams from pipecat.transports.base_transport import TransportParams # Optional: Path to the local Smart Turn model # If not provided, it will download from Hugging Face smart_turn_model_path = os.getenv( "LOCAL_SMART_TURN_MODEL_PATH" ) # Create the transport with PyTorch-based Smart Turn detection transport = SmallWebRTCTransport( webrtc_connection = webrtc_connection, params = TransportParams( audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer( params = VADParams( stop_secs = 0.2 )), turn_analyzer = LocalSmartTurnAnalyzer( smart_turn_model_path = smart_turn_model_path, params = SmartTurnParams( stop_secs = 2.0 , pre_speech_ms = 0.0 , max_duration_secs = 8.0 ) ), ), ) ​ Local Model Setup ​ CoreML Model & PyTorch Setup To use the LocalCoreMLSmartTurnAnalyzer or LocalSmartTurnAnalyzer , you need to set up the model locally: Install Git LFS (Large File Storage): macOS Ubuntu/Debian Copy Ask AI brew install git-lfs Initialize Git LFS Copy Ask AI git lfs install Clone the Smart Turn model repository: Copy Ask AI git clone https://huggingface.co/pipecat-ai/smart-turn Set the environment variable to the cloned repository path: Copy Ask AI # Add to your .env file or environment export LOCAL_SMART_TURN_MODEL_PATH = / path / to / smart-turn Note that the CoreML model is optimized for Apple Silicon devices. If you’re using a different platform, consider using the PyTorch-based LocalSmartTurnAnalyzer or the remote Smart Turn service. Learn more about the CoreML setup in the official repository instructions ​ How It Works Smart Turn Detection continuously analyzes audio streams to identify natural turn completion points: Audio Buffering : The system continuously buffers audio with timestamps, maintaining a small buffer of pre-speech audio. VAD Processing : Voice Activity Detection segments the audio into speech and non-speech portions. Turn Analysis : When VAD detects a pause in speech: The ML model analyzes the speech segment for natural completion cues It identifies acoustic and linguistic patterns that indicate turn completion A decision is made whether the turn is complete or incomplete The system includes a fallback mechanism: if a turn is classified as incomplete but silence continues for longer than stop_secs , the turn is automatically marked as complete. ​ Notes The model is designed for English speech; performance may vary with other languages You can adjust the stop_secs parameter based on your application’s needs for responsiveness Smart Turn generally provides a more natural conversational experience but is computationally more intensive than simple VAD The PyTorch-based LocalSmartTurnAnalyzer runs on CPU by default but will use CUDA if available Daily REST Helper Fal Smart Turn On this page Overview Installation Integration with Transport Configuration Remote Smart Turn Constructor Parameters Example Local Smart Turn (CoreML) Constructor Parameters Example Local Smart Turn (PyTorch) Constructor Parameters Example Local Model Setup CoreML Model & PyTorch Setup How It Works Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/smart-turn_smart-turn-overview_9d6bcd8f.txt b/smart-turn_smart-turn-overview_9d6bcd8f.txt
new file mode 100644
index 0000000000000000000000000000000000000000..b3921be6001898af0107633b02ac59ff528d6444
--- /dev/null
+++ b/smart-turn_smart-turn-overview_9d6bcd8f.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/smart-turn/smart-turn-overview#overview
+Title: Smart Turn Overview - Pipecat
+==================================================
+
+Smart Turn Overview - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Smart Turn Detection Smart Turn Overview Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Smart Turn Overview Fal Smart Turn Local CoreML Smart Turn Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview Smart Turn Detection is an advanced feature in Pipecat that determines when a user has finished speaking and the bot should respond. Unlike basic Voice Activity Detection (VAD) which only detects speech vs. non-speech, Smart Turn Detection uses a machine learning model to recognize natural conversational cues like intonation patterns and linguistic signals. Smart Turn Model Open source model for advanced conversational turn detection. Contribute to model training and development. Data Collector Contribute conversational data to improve the smart-turn model Data Classifier Help classify turn completion patterns in conversations Pipecat provides three implementations of Smart Turn Detection: FalSmartTurnAnalyzer - Uses a Fal’s hosted smart-turn model for inference LocalCoreMLSmartTurnAnalyzer - Runs inference locally on Apple Silicon using CoreML LocalSmartTurnAnalyzer - Runs inference locally using PyTorch and Hugging Face Transformers All implementations share the same underlying API and parameters, making it easy to switch between them based on your deployment requirements. ​ Installation The Smart Turn Detection feature requires additional dependencies depending on which implementation you choose. For Fal’s hosted service inference: Copy Ask AI pip install "pipecat-ai[remote-smart-turn]" For local inference (CoreML-based): Copy Ask AI pip install "pipecat-ai[local-smart-turn]" For local inference (PyTorch-based): Copy Ask AI pip install "pipecat-ai[local-smart-turn]" ​ Integration with Transport Smart Turn Detection is integrated into your application by setting one of the available turn analyzers as the turn_analyzer parameter in your transport configuration: Copy Ask AI from pipecat.transports.base_transport import TransportParams transport = SmallWebRTCTransport( webrtc_connection = webrtc_connection, params = TransportParams( # Other transport parameters... turn_analyzer = FalSmartTurnAnalyzer( url = remote_smart_turn_url), ), ) Smart Turn Detection requires VAD to be enabled and works best when the VAD analyzer is set to a short stop_secs value. We recommend 0.2 seconds. Copy Ask AI audio_in_enabled = True , vad_analyzer = SileroVADAnalyzer( params = VADParams( stop_secs = 0.2 )) ​ Configuration All implementations use the same SmartTurnParams class to configure behavior: ​ stop_secs float default: "3.0" Duration of silence in seconds required before triggering a silence-based end of turn ​ pre_speech_ms float default: "0.0" Amount of audio (in milliseconds) to include before speech is detected ​ max_duration_secs float default: "8.0" Maximum allowed segment duration in seconds. For segments longer than this value, a rolling window is used. ​ Remote Smart Turn The FalSmartTurnAnalyzer class uses a remote service for turn detection inference. ​ Constructor Parameters ​ url str required The URL of the remote Smart Turn service ​ sample_rate Optional[int] default: "None" Audio sample rate (will be set by the transport if not provided) ​ params SmartTurnParams default: "SmartTurnParams()" Configuration parameters for turn detection ​ Example Copy Ask AI import os from pipecat.audio.turn.smart_turn.fal_smart_turn import FalSmartTurnAnalyzer from pipecat.audio.turn.smart_turn.base_smart_turn import SmartTurnParams from pipecat.transports.base_transport import TransportParams # Get the URL for the remote Smart Turn service remote_smart_turn_url = os.getenv( "REMOTE_SMART_TURN_URL" ) # Create the transport with Smart Turn detection transport = SmallWebRTCTransport( webrtc_connection = webrtc_connection, params = TransportParams( audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer( params = VADParams( stop_secs = 0.2 )), turn_analyzer = FalSmartTurnAnalyzer( url = remote_smart_turn_url, params = SmartTurnParams( stop_secs = 3.0 , pre_speech_ms = 0.0 , max_duration_secs = 8.0 ) ), ), ) ​ Local Smart Turn (CoreML) The LocalCoreMLSmartTurnAnalyzer runs inference locally using CoreML, providing lower latency and no network dependencies. ​ Constructor Parameters ​ smart_turn_model_path str required Path to the directory containing the Smart Turn model ​ sample_rate Optional[int] default: "None" Audio sample rate (will be set by the transport if not provided) ​ params SmartTurnParams default: "SmartTurnParams()" Configuration parameters for turn detection ​ Example Copy Ask AI import os from pipecat.audio.turn.smart_turn.local_coreml_smart_turn import LocalCoreMLSmartTurnAnalyzer from pipecat.audio.turn.smart_turn.base_smart_turn import SmartTurnParams from pipecat.transports.base_transport import TransportParams # Path to the Smart Turn model directory smart_turn_model_path = os.getenv( "LOCAL_SMART_TURN_MODEL_PATH" ) # Create the transport with local Smart Turn detection transport = SmallWebRTCTransport( webrtc_connection = webrtc_connection, params = TransportParams( audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer( params = VADParams( stop_secs = 0.2 )), turn_analyzer = LocalCoreMLSmartTurnAnalyzer( smart_turn_model_path = smart_turn_model_path, params = SmartTurnParams( stop_secs = 2.0 , # Shorter stop time when using Smart Turn pre_speech_ms = 0.0 , max_duration_secs = 8.0 ) ), ), ) ​ Local Smart Turn (PyTorch) The LocalSmartTurnAnalyzer runs inference locally using PyTorch and Hugging Face Transformers, providing a cross-platform solution. ​ Constructor Parameters ​ smart_turn_model_path str default: "pipecat-ai/smart-turn" Path to the Smart Turn model or Hugging Face model identifier. Defaults to the official “pipecat-ai/smart-turn” model. ​ sample_rate Optional[int] default: "None" Audio sample rate (will be set by the transport if not provided) ​ params SmartTurnParams default: "SmartTurnParams()" Configuration parameters for turn detection ​ Example Copy Ask AI import os from pipecat.audio.turn.smart_turn.local_smart_turn import LocalSmartTurnAnalyzer from pipecat.audio.turn.smart_turn.base_smart_turn import SmartTurnParams from pipecat.transports.base_transport import TransportParams # Optional: Path to the local Smart Turn model # If not provided, it will download from Hugging Face smart_turn_model_path = os.getenv( "LOCAL_SMART_TURN_MODEL_PATH" ) # Create the transport with PyTorch-based Smart Turn detection transport = SmallWebRTCTransport( webrtc_connection = webrtc_connection, params = TransportParams( audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer( params = VADParams( stop_secs = 0.2 )), turn_analyzer = LocalSmartTurnAnalyzer( smart_turn_model_path = smart_turn_model_path, params = SmartTurnParams( stop_secs = 2.0 , pre_speech_ms = 0.0 , max_duration_secs = 8.0 ) ), ), ) ​ Local Model Setup ​ CoreML Model & PyTorch Setup To use the LocalCoreMLSmartTurnAnalyzer or LocalSmartTurnAnalyzer , you need to set up the model locally: Install Git LFS (Large File Storage): macOS Ubuntu/Debian Copy Ask AI brew install git-lfs Initialize Git LFS Copy Ask AI git lfs install Clone the Smart Turn model repository: Copy Ask AI git clone https://huggingface.co/pipecat-ai/smart-turn Set the environment variable to the cloned repository path: Copy Ask AI # Add to your .env file or environment export LOCAL_SMART_TURN_MODEL_PATH = / path / to / smart-turn Note that the CoreML model is optimized for Apple Silicon devices. If you’re using a different platform, consider using the PyTorch-based LocalSmartTurnAnalyzer or the remote Smart Turn service. Learn more about the CoreML setup in the official repository instructions ​ How It Works Smart Turn Detection continuously analyzes audio streams to identify natural turn completion points: Audio Buffering : The system continuously buffers audio with timestamps, maintaining a small buffer of pre-speech audio. VAD Processing : Voice Activity Detection segments the audio into speech and non-speech portions. Turn Analysis : When VAD detects a pause in speech: The ML model analyzes the speech segment for natural completion cues It identifies acoustic and linguistic patterns that indicate turn completion A decision is made whether the turn is complete or incomplete The system includes a fallback mechanism: if a turn is classified as incomplete but silence continues for longer than stop_secs , the turn is automatically marked as complete. ​ Notes The model is designed for English speech; performance may vary with other languages You can adjust the stop_secs parameter based on your application’s needs for responsiveness Smart Turn generally provides a more natural conversational experience but is computationally more intensive than simple VAD The PyTorch-based LocalSmartTurnAnalyzer runs on CPU by default but will use CUDA if available Daily REST Helper Fal Smart Turn On this page Overview Installation Integration with Transport Configuration Remote Smart Turn Constructor Parameters Example Local Smart Turn (CoreML) Constructor Parameters Example Local Smart Turn (PyTorch) Constructor Parameters Example Local Model Setup CoreML Model & PyTorch Setup How It Works Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/smart-turn_smart-turn-overview_b557e72d.txt b/smart-turn_smart-turn-overview_b557e72d.txt
new file mode 100644
index 0000000000000000000000000000000000000000..1dc86e49e85ca61bbef7e7513266e21e555a6d0a
--- /dev/null
+++ b/smart-turn_smart-turn-overview_b557e72d.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/smart-turn/smart-turn-overview#configuration
+Title: Smart Turn Overview - Pipecat
+==================================================
+
+Smart Turn Overview - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Smart Turn Detection Smart Turn Overview Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Smart Turn Overview Fal Smart Turn Local CoreML Smart Turn Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview Smart Turn Detection is an advanced feature in Pipecat that determines when a user has finished speaking and the bot should respond. Unlike basic Voice Activity Detection (VAD) which only detects speech vs. non-speech, Smart Turn Detection uses a machine learning model to recognize natural conversational cues like intonation patterns and linguistic signals. Smart Turn Model Open source model for advanced conversational turn detection. Contribute to model training and development. Data Collector Contribute conversational data to improve the smart-turn model Data Classifier Help classify turn completion patterns in conversations Pipecat provides three implementations of Smart Turn Detection: FalSmartTurnAnalyzer - Uses a Fal’s hosted smart-turn model for inference LocalCoreMLSmartTurnAnalyzer - Runs inference locally on Apple Silicon using CoreML LocalSmartTurnAnalyzer - Runs inference locally using PyTorch and Hugging Face Transformers All implementations share the same underlying API and parameters, making it easy to switch between them based on your deployment requirements. ​ Installation The Smart Turn Detection feature requires additional dependencies depending on which implementation you choose. For Fal’s hosted service inference: Copy Ask AI pip install "pipecat-ai[remote-smart-turn]" For local inference (CoreML-based): Copy Ask AI pip install "pipecat-ai[local-smart-turn]" For local inference (PyTorch-based): Copy Ask AI pip install "pipecat-ai[local-smart-turn]" ​ Integration with Transport Smart Turn Detection is integrated into your application by setting one of the available turn analyzers as the turn_analyzer parameter in your transport configuration: Copy Ask AI from pipecat.transports.base_transport import TransportParams transport = SmallWebRTCTransport( webrtc_connection = webrtc_connection, params = TransportParams( # Other transport parameters... turn_analyzer = FalSmartTurnAnalyzer( url = remote_smart_turn_url), ), ) Smart Turn Detection requires VAD to be enabled and works best when the VAD analyzer is set to a short stop_secs value. We recommend 0.2 seconds. Copy Ask AI audio_in_enabled = True , vad_analyzer = SileroVADAnalyzer( params = VADParams( stop_secs = 0.2 )) ​ Configuration All implementations use the same SmartTurnParams class to configure behavior: ​ stop_secs float default: "3.0" Duration of silence in seconds required before triggering a silence-based end of turn ​ pre_speech_ms float default: "0.0" Amount of audio (in milliseconds) to include before speech is detected ​ max_duration_secs float default: "8.0" Maximum allowed segment duration in seconds. For segments longer than this value, a rolling window is used. ​ Remote Smart Turn The FalSmartTurnAnalyzer class uses a remote service for turn detection inference. ​ Constructor Parameters ​ url str required The URL of the remote Smart Turn service ​ sample_rate Optional[int] default: "None" Audio sample rate (will be set by the transport if not provided) ​ params SmartTurnParams default: "SmartTurnParams()" Configuration parameters for turn detection ​ Example Copy Ask AI import os from pipecat.audio.turn.smart_turn.fal_smart_turn import FalSmartTurnAnalyzer from pipecat.audio.turn.smart_turn.base_smart_turn import SmartTurnParams from pipecat.transports.base_transport import TransportParams # Get the URL for the remote Smart Turn service remote_smart_turn_url = os.getenv( "REMOTE_SMART_TURN_URL" ) # Create the transport with Smart Turn detection transport = SmallWebRTCTransport( webrtc_connection = webrtc_connection, params = TransportParams( audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer( params = VADParams( stop_secs = 0.2 )), turn_analyzer = FalSmartTurnAnalyzer( url = remote_smart_turn_url, params = SmartTurnParams( stop_secs = 3.0 , pre_speech_ms = 0.0 , max_duration_secs = 8.0 ) ), ), ) ​ Local Smart Turn (CoreML) The LocalCoreMLSmartTurnAnalyzer runs inference locally using CoreML, providing lower latency and no network dependencies. ​ Constructor Parameters ​ smart_turn_model_path str required Path to the directory containing the Smart Turn model ​ sample_rate Optional[int] default: "None" Audio sample rate (will be set by the transport if not provided) ​ params SmartTurnParams default: "SmartTurnParams()" Configuration parameters for turn detection ​ Example Copy Ask AI import os from pipecat.audio.turn.smart_turn.local_coreml_smart_turn import LocalCoreMLSmartTurnAnalyzer from pipecat.audio.turn.smart_turn.base_smart_turn import SmartTurnParams from pipecat.transports.base_transport import TransportParams # Path to the Smart Turn model directory smart_turn_model_path = os.getenv( "LOCAL_SMART_TURN_MODEL_PATH" ) # Create the transport with local Smart Turn detection transport = SmallWebRTCTransport( webrtc_connection = webrtc_connection, params = TransportParams( audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer( params = VADParams( stop_secs = 0.2 )), turn_analyzer = LocalCoreMLSmartTurnAnalyzer( smart_turn_model_path = smart_turn_model_path, params = SmartTurnParams( stop_secs = 2.0 , # Shorter stop time when using Smart Turn pre_speech_ms = 0.0 , max_duration_secs = 8.0 ) ), ), ) ​ Local Smart Turn (PyTorch) The LocalSmartTurnAnalyzer runs inference locally using PyTorch and Hugging Face Transformers, providing a cross-platform solution. ​ Constructor Parameters ​ smart_turn_model_path str default: "pipecat-ai/smart-turn" Path to the Smart Turn model or Hugging Face model identifier. Defaults to the official “pipecat-ai/smart-turn” model. ​ sample_rate Optional[int] default: "None" Audio sample rate (will be set by the transport if not provided) ​ params SmartTurnParams default: "SmartTurnParams()" Configuration parameters for turn detection ​ Example Copy Ask AI import os from pipecat.audio.turn.smart_turn.local_smart_turn import LocalSmartTurnAnalyzer from pipecat.audio.turn.smart_turn.base_smart_turn import SmartTurnParams from pipecat.transports.base_transport import TransportParams # Optional: Path to the local Smart Turn model # If not provided, it will download from Hugging Face smart_turn_model_path = os.getenv( "LOCAL_SMART_TURN_MODEL_PATH" ) # Create the transport with PyTorch-based Smart Turn detection transport = SmallWebRTCTransport( webrtc_connection = webrtc_connection, params = TransportParams( audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer( params = VADParams( stop_secs = 0.2 )), turn_analyzer = LocalSmartTurnAnalyzer( smart_turn_model_path = smart_turn_model_path, params = SmartTurnParams( stop_secs = 2.0 , pre_speech_ms = 0.0 , max_duration_secs = 8.0 ) ), ), ) ​ Local Model Setup ​ CoreML Model & PyTorch Setup To use the LocalCoreMLSmartTurnAnalyzer or LocalSmartTurnAnalyzer , you need to set up the model locally: Install Git LFS (Large File Storage): macOS Ubuntu/Debian Copy Ask AI brew install git-lfs Initialize Git LFS Copy Ask AI git lfs install Clone the Smart Turn model repository: Copy Ask AI git clone https://huggingface.co/pipecat-ai/smart-turn Set the environment variable to the cloned repository path: Copy Ask AI # Add to your .env file or environment export LOCAL_SMART_TURN_MODEL_PATH = / path / to / smart-turn Note that the CoreML model is optimized for Apple Silicon devices. If you’re using a different platform, consider using the PyTorch-based LocalSmartTurnAnalyzer or the remote Smart Turn service. Learn more about the CoreML setup in the official repository instructions ​ How It Works Smart Turn Detection continuously analyzes audio streams to identify natural turn completion points: Audio Buffering : The system continuously buffers audio with timestamps, maintaining a small buffer of pre-speech audio. VAD Processing : Voice Activity Detection segments the audio into speech and non-speech portions. Turn Analysis : When VAD detects a pause in speech: The ML model analyzes the speech segment for natural completion cues It identifies acoustic and linguistic patterns that indicate turn completion A decision is made whether the turn is complete or incomplete The system includes a fallback mechanism: if a turn is classified as incomplete but silence continues for longer than stop_secs , the turn is automatically marked as complete. ​ Notes The model is designed for English speech; performance may vary with other languages You can adjust the stop_secs parameter based on your application’s needs for responsiveness Smart Turn generally provides a more natural conversational experience but is computationally more intensive than simple VAD The PyTorch-based LocalSmartTurnAnalyzer runs on CPU by default but will use CUDA if available Daily REST Helper Fal Smart Turn On this page Overview Installation Integration with Transport Configuration Remote Smart Turn Constructor Parameters Example Local Smart Turn (CoreML) Constructor Parameters Example Local Smart Turn (PyTorch) Constructor Parameters Example Local Model Setup CoreML Model & PyTorch Setup How It Works Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/smart-turn_smart-turn-overview_cd6e1d90.txt b/smart-turn_smart-turn-overview_cd6e1d90.txt
new file mode 100644
index 0000000000000000000000000000000000000000..52a5d26a8f8c46d93f5bb3a349bfd67fc886d188
--- /dev/null
+++ b/smart-turn_smart-turn-overview_cd6e1d90.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/smart-turn/smart-turn-overview#constructor-parameters-3
+Title: Smart Turn Overview - Pipecat
+==================================================
+
+Smart Turn Overview - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Smart Turn Detection Smart Turn Overview Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Smart Turn Overview Fal Smart Turn Local CoreML Smart Turn Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview Smart Turn Detection is an advanced feature in Pipecat that determines when a user has finished speaking and the bot should respond. Unlike basic Voice Activity Detection (VAD) which only detects speech vs. non-speech, Smart Turn Detection uses a machine learning model to recognize natural conversational cues like intonation patterns and linguistic signals. Smart Turn Model Open source model for advanced conversational turn detection. Contribute to model training and development. Data Collector Contribute conversational data to improve the smart-turn model Data Classifier Help classify turn completion patterns in conversations Pipecat provides three implementations of Smart Turn Detection: FalSmartTurnAnalyzer - Uses a Fal’s hosted smart-turn model for inference LocalCoreMLSmartTurnAnalyzer - Runs inference locally on Apple Silicon using CoreML LocalSmartTurnAnalyzer - Runs inference locally using PyTorch and Hugging Face Transformers All implementations share the same underlying API and parameters, making it easy to switch between them based on your deployment requirements. ​ Installation The Smart Turn Detection feature requires additional dependencies depending on which implementation you choose. For Fal’s hosted service inference: Copy Ask AI pip install "pipecat-ai[remote-smart-turn]" For local inference (CoreML-based): Copy Ask AI pip install "pipecat-ai[local-smart-turn]" For local inference (PyTorch-based): Copy Ask AI pip install "pipecat-ai[local-smart-turn]" ​ Integration with Transport Smart Turn Detection is integrated into your application by setting one of the available turn analyzers as the turn_analyzer parameter in your transport configuration: Copy Ask AI from pipecat.transports.base_transport import TransportParams transport = SmallWebRTCTransport( webrtc_connection = webrtc_connection, params = TransportParams( # Other transport parameters... turn_analyzer = FalSmartTurnAnalyzer( url = remote_smart_turn_url), ), ) Smart Turn Detection requires VAD to be enabled and works best when the VAD analyzer is set to a short stop_secs value. We recommend 0.2 seconds. Copy Ask AI audio_in_enabled = True , vad_analyzer = SileroVADAnalyzer( params = VADParams( stop_secs = 0.2 )) ​ Configuration All implementations use the same SmartTurnParams class to configure behavior: ​ stop_secs float default: "3.0" Duration of silence in seconds required before triggering a silence-based end of turn ​ pre_speech_ms float default: "0.0" Amount of audio (in milliseconds) to include before speech is detected ​ max_duration_secs float default: "8.0" Maximum allowed segment duration in seconds. For segments longer than this value, a rolling window is used. ​ Remote Smart Turn The FalSmartTurnAnalyzer class uses a remote service for turn detection inference. ​ Constructor Parameters ​ url str required The URL of the remote Smart Turn service ​ sample_rate Optional[int] default: "None" Audio sample rate (will be set by the transport if not provided) ​ params SmartTurnParams default: "SmartTurnParams()" Configuration parameters for turn detection ​ Example Copy Ask AI import os from pipecat.audio.turn.smart_turn.fal_smart_turn import FalSmartTurnAnalyzer from pipecat.audio.turn.smart_turn.base_smart_turn import SmartTurnParams from pipecat.transports.base_transport import TransportParams # Get the URL for the remote Smart Turn service remote_smart_turn_url = os.getenv( "REMOTE_SMART_TURN_URL" ) # Create the transport with Smart Turn detection transport = SmallWebRTCTransport( webrtc_connection = webrtc_connection, params = TransportParams( audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer( params = VADParams( stop_secs = 0.2 )), turn_analyzer = FalSmartTurnAnalyzer( url = remote_smart_turn_url, params = SmartTurnParams( stop_secs = 3.0 , pre_speech_ms = 0.0 , max_duration_secs = 8.0 ) ), ), ) ​ Local Smart Turn (CoreML) The LocalCoreMLSmartTurnAnalyzer runs inference locally using CoreML, providing lower latency and no network dependencies. ​ Constructor Parameters ​ smart_turn_model_path str required Path to the directory containing the Smart Turn model ​ sample_rate Optional[int] default: "None" Audio sample rate (will be set by the transport if not provided) ​ params SmartTurnParams default: "SmartTurnParams()" Configuration parameters for turn detection ​ Example Copy Ask AI import os from pipecat.audio.turn.smart_turn.local_coreml_smart_turn import LocalCoreMLSmartTurnAnalyzer from pipecat.audio.turn.smart_turn.base_smart_turn import SmartTurnParams from pipecat.transports.base_transport import TransportParams # Path to the Smart Turn model directory smart_turn_model_path = os.getenv( "LOCAL_SMART_TURN_MODEL_PATH" ) # Create the transport with local Smart Turn detection transport = SmallWebRTCTransport( webrtc_connection = webrtc_connection, params = TransportParams( audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer( params = VADParams( stop_secs = 0.2 )), turn_analyzer = LocalCoreMLSmartTurnAnalyzer( smart_turn_model_path = smart_turn_model_path, params = SmartTurnParams( stop_secs = 2.0 , # Shorter stop time when using Smart Turn pre_speech_ms = 0.0 , max_duration_secs = 8.0 ) ), ), ) ​ Local Smart Turn (PyTorch) The LocalSmartTurnAnalyzer runs inference locally using PyTorch and Hugging Face Transformers, providing a cross-platform solution. ​ Constructor Parameters ​ smart_turn_model_path str default: "pipecat-ai/smart-turn" Path to the Smart Turn model or Hugging Face model identifier. Defaults to the official “pipecat-ai/smart-turn” model. ​ sample_rate Optional[int] default: "None" Audio sample rate (will be set by the transport if not provided) ​ params SmartTurnParams default: "SmartTurnParams()" Configuration parameters for turn detection ​ Example Copy Ask AI import os from pipecat.audio.turn.smart_turn.local_smart_turn import LocalSmartTurnAnalyzer from pipecat.audio.turn.smart_turn.base_smart_turn import SmartTurnParams from pipecat.transports.base_transport import TransportParams # Optional: Path to the local Smart Turn model # If not provided, it will download from Hugging Face smart_turn_model_path = os.getenv( "LOCAL_SMART_TURN_MODEL_PATH" ) # Create the transport with PyTorch-based Smart Turn detection transport = SmallWebRTCTransport( webrtc_connection = webrtc_connection, params = TransportParams( audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer( params = VADParams( stop_secs = 0.2 )), turn_analyzer = LocalSmartTurnAnalyzer( smart_turn_model_path = smart_turn_model_path, params = SmartTurnParams( stop_secs = 2.0 , pre_speech_ms = 0.0 , max_duration_secs = 8.0 ) ), ), ) ​ Local Model Setup ​ CoreML Model & PyTorch Setup To use the LocalCoreMLSmartTurnAnalyzer or LocalSmartTurnAnalyzer , you need to set up the model locally: Install Git LFS (Large File Storage): macOS Ubuntu/Debian Copy Ask AI brew install git-lfs Initialize Git LFS Copy Ask AI git lfs install Clone the Smart Turn model repository: Copy Ask AI git clone https://huggingface.co/pipecat-ai/smart-turn Set the environment variable to the cloned repository path: Copy Ask AI # Add to your .env file or environment export LOCAL_SMART_TURN_MODEL_PATH = / path / to / smart-turn Note that the CoreML model is optimized for Apple Silicon devices. If you’re using a different platform, consider using the PyTorch-based LocalSmartTurnAnalyzer or the remote Smart Turn service. Learn more about the CoreML setup in the official repository instructions ​ How It Works Smart Turn Detection continuously analyzes audio streams to identify natural turn completion points: Audio Buffering : The system continuously buffers audio with timestamps, maintaining a small buffer of pre-speech audio. VAD Processing : Voice Activity Detection segments the audio into speech and non-speech portions. Turn Analysis : When VAD detects a pause in speech: The ML model analyzes the speech segment for natural completion cues It identifies acoustic and linguistic patterns that indicate turn completion A decision is made whether the turn is complete or incomplete The system includes a fallback mechanism: if a turn is classified as incomplete but silence continues for longer than stop_secs , the turn is automatically marked as complete. ​ Notes The model is designed for English speech; performance may vary with other languages You can adjust the stop_secs parameter based on your application’s needs for responsiveness Smart Turn generally provides a more natural conversational experience but is computationally more intensive than simple VAD The PyTorch-based LocalSmartTurnAnalyzer runs on CPU by default but will use CUDA if available Daily REST Helper Fal Smart Turn On this page Overview Installation Integration with Transport Configuration Remote Smart Turn Constructor Parameters Example Local Smart Turn (CoreML) Constructor Parameters Example Local Smart Turn (PyTorch) Constructor Parameters Example Local Model Setup CoreML Model & PyTorch Setup How It Works Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/smart-turn_smart-turn-overview_deeb8825.txt b/smart-turn_smart-turn-overview_deeb8825.txt
new file mode 100644
index 0000000000000000000000000000000000000000..321ec5c01c0f4e054cf95c760b130ec5d86d591c
--- /dev/null
+++ b/smart-turn_smart-turn-overview_deeb8825.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/smart-turn/smart-turn-overview#local-smart-turn-pytorch
+Title: Smart Turn Overview - Pipecat
+==================================================
+
+Smart Turn Overview - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Smart Turn Detection Smart Turn Overview Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Smart Turn Overview Fal Smart Turn Local CoreML Smart Turn Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview Smart Turn Detection is an advanced feature in Pipecat that determines when a user has finished speaking and the bot should respond. Unlike basic Voice Activity Detection (VAD) which only detects speech vs. non-speech, Smart Turn Detection uses a machine learning model to recognize natural conversational cues like intonation patterns and linguistic signals. Smart Turn Model Open source model for advanced conversational turn detection. Contribute to model training and development. Data Collector Contribute conversational data to improve the smart-turn model Data Classifier Help classify turn completion patterns in conversations Pipecat provides three implementations of Smart Turn Detection: FalSmartTurnAnalyzer - Uses a Fal’s hosted smart-turn model for inference LocalCoreMLSmartTurnAnalyzer - Runs inference locally on Apple Silicon using CoreML LocalSmartTurnAnalyzer - Runs inference locally using PyTorch and Hugging Face Transformers All implementations share the same underlying API and parameters, making it easy to switch between them based on your deployment requirements. ​ Installation The Smart Turn Detection feature requires additional dependencies depending on which implementation you choose. For Fal’s hosted service inference: Copy Ask AI pip install "pipecat-ai[remote-smart-turn]" For local inference (CoreML-based): Copy Ask AI pip install "pipecat-ai[local-smart-turn]" For local inference (PyTorch-based): Copy Ask AI pip install "pipecat-ai[local-smart-turn]" ​ Integration with Transport Smart Turn Detection is integrated into your application by setting one of the available turn analyzers as the turn_analyzer parameter in your transport configuration: Copy Ask AI from pipecat.transports.base_transport import TransportParams transport = SmallWebRTCTransport( webrtc_connection = webrtc_connection, params = TransportParams( # Other transport parameters... turn_analyzer = FalSmartTurnAnalyzer( url = remote_smart_turn_url), ), ) Smart Turn Detection requires VAD to be enabled and works best when the VAD analyzer is set to a short stop_secs value. We recommend 0.2 seconds. Copy Ask AI audio_in_enabled = True , vad_analyzer = SileroVADAnalyzer( params = VADParams( stop_secs = 0.2 )) ​ Configuration All implementations use the same SmartTurnParams class to configure behavior: ​ stop_secs float default: "3.0" Duration of silence in seconds required before triggering a silence-based end of turn ​ pre_speech_ms float default: "0.0" Amount of audio (in milliseconds) to include before speech is detected ​ max_duration_secs float default: "8.0" Maximum allowed segment duration in seconds. For segments longer than this value, a rolling window is used. ​ Remote Smart Turn The FalSmartTurnAnalyzer class uses a remote service for turn detection inference. ​ Constructor Parameters ​ url str required The URL of the remote Smart Turn service ​ sample_rate Optional[int] default: "None" Audio sample rate (will be set by the transport if not provided) ​ params SmartTurnParams default: "SmartTurnParams()" Configuration parameters for turn detection ​ Example Copy Ask AI import os from pipecat.audio.turn.smart_turn.fal_smart_turn import FalSmartTurnAnalyzer from pipecat.audio.turn.smart_turn.base_smart_turn import SmartTurnParams from pipecat.transports.base_transport import TransportParams # Get the URL for the remote Smart Turn service remote_smart_turn_url = os.getenv( "REMOTE_SMART_TURN_URL" ) # Create the transport with Smart Turn detection transport = SmallWebRTCTransport( webrtc_connection = webrtc_connection, params = TransportParams( audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer( params = VADParams( stop_secs = 0.2 )), turn_analyzer = FalSmartTurnAnalyzer( url = remote_smart_turn_url, params = SmartTurnParams( stop_secs = 3.0 , pre_speech_ms = 0.0 , max_duration_secs = 8.0 ) ), ), ) ​ Local Smart Turn (CoreML) The LocalCoreMLSmartTurnAnalyzer runs inference locally using CoreML, providing lower latency and no network dependencies. ​ Constructor Parameters ​ smart_turn_model_path str required Path to the directory containing the Smart Turn model ​ sample_rate Optional[int] default: "None" Audio sample rate (will be set by the transport if not provided) ​ params SmartTurnParams default: "SmartTurnParams()" Configuration parameters for turn detection ​ Example Copy Ask AI import os from pipecat.audio.turn.smart_turn.local_coreml_smart_turn import LocalCoreMLSmartTurnAnalyzer from pipecat.audio.turn.smart_turn.base_smart_turn import SmartTurnParams from pipecat.transports.base_transport import TransportParams # Path to the Smart Turn model directory smart_turn_model_path = os.getenv( "LOCAL_SMART_TURN_MODEL_PATH" ) # Create the transport with local Smart Turn detection transport = SmallWebRTCTransport( webrtc_connection = webrtc_connection, params = TransportParams( audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer( params = VADParams( stop_secs = 0.2 )), turn_analyzer = LocalCoreMLSmartTurnAnalyzer( smart_turn_model_path = smart_turn_model_path, params = SmartTurnParams( stop_secs = 2.0 , # Shorter stop time when using Smart Turn pre_speech_ms = 0.0 , max_duration_secs = 8.0 ) ), ), ) ​ Local Smart Turn (PyTorch) The LocalSmartTurnAnalyzer runs inference locally using PyTorch and Hugging Face Transformers, providing a cross-platform solution. ​ Constructor Parameters ​ smart_turn_model_path str default: "pipecat-ai/smart-turn" Path to the Smart Turn model or Hugging Face model identifier. Defaults to the official “pipecat-ai/smart-turn” model. ​ sample_rate Optional[int] default: "None" Audio sample rate (will be set by the transport if not provided) ​ params SmartTurnParams default: "SmartTurnParams()" Configuration parameters for turn detection ​ Example Copy Ask AI import os from pipecat.audio.turn.smart_turn.local_smart_turn import LocalSmartTurnAnalyzer from pipecat.audio.turn.smart_turn.base_smart_turn import SmartTurnParams from pipecat.transports.base_transport import TransportParams # Optional: Path to the local Smart Turn model # If not provided, it will download from Hugging Face smart_turn_model_path = os.getenv( "LOCAL_SMART_TURN_MODEL_PATH" ) # Create the transport with PyTorch-based Smart Turn detection transport = SmallWebRTCTransport( webrtc_connection = webrtc_connection, params = TransportParams( audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer( params = VADParams( stop_secs = 0.2 )), turn_analyzer = LocalSmartTurnAnalyzer( smart_turn_model_path = smart_turn_model_path, params = SmartTurnParams( stop_secs = 2.0 , pre_speech_ms = 0.0 , max_duration_secs = 8.0 ) ), ), ) ​ Local Model Setup ​ CoreML Model & PyTorch Setup To use the LocalCoreMLSmartTurnAnalyzer or LocalSmartTurnAnalyzer , you need to set up the model locally: Install Git LFS (Large File Storage): macOS Ubuntu/Debian Copy Ask AI brew install git-lfs Initialize Git LFS Copy Ask AI git lfs install Clone the Smart Turn model repository: Copy Ask AI git clone https://huggingface.co/pipecat-ai/smart-turn Set the environment variable to the cloned repository path: Copy Ask AI # Add to your .env file or environment export LOCAL_SMART_TURN_MODEL_PATH = / path / to / smart-turn Note that the CoreML model is optimized for Apple Silicon devices. If you’re using a different platform, consider using the PyTorch-based LocalSmartTurnAnalyzer or the remote Smart Turn service. Learn more about the CoreML setup in the official repository instructions ​ How It Works Smart Turn Detection continuously analyzes audio streams to identify natural turn completion points: Audio Buffering : The system continuously buffers audio with timestamps, maintaining a small buffer of pre-speech audio. VAD Processing : Voice Activity Detection segments the audio into speech and non-speech portions. Turn Analysis : When VAD detects a pause in speech: The ML model analyzes the speech segment for natural completion cues It identifies acoustic and linguistic patterns that indicate turn completion A decision is made whether the turn is complete or incomplete The system includes a fallback mechanism: if a turn is classified as incomplete but silence continues for longer than stop_secs , the turn is automatically marked as complete. ​ Notes The model is designed for English speech; performance may vary with other languages You can adjust the stop_secs parameter based on your application’s needs for responsiveness Smart Turn generally provides a more natural conversational experience but is computationally more intensive than simple VAD The PyTorch-based LocalSmartTurnAnalyzer runs on CPU by default but will use CUDA if available Daily REST Helper Fal Smart Turn On this page Overview Installation Integration with Transport Configuration Remote Smart Turn Constructor Parameters Example Local Smart Turn (CoreML) Constructor Parameters Example Local Smart Turn (PyTorch) Constructor Parameters Example Local Model Setup CoreML Model & PyTorch Setup How It Works Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/smart-turn_smart-turn-overview_fb346acf.txt b/smart-turn_smart-turn-overview_fb346acf.txt
new file mode 100644
index 0000000000000000000000000000000000000000..bb7bca72c552120409a730202d6d9369748bbd1c
--- /dev/null
+++ b/smart-turn_smart-turn-overview_fb346acf.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/smart-turn/smart-turn-overview
+Title: Smart Turn Overview - Pipecat
+==================================================
+
+Smart Turn Overview - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Smart Turn Detection Smart Turn Overview Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Smart Turn Overview Fal Smart Turn Local CoreML Smart Turn Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview Smart Turn Detection is an advanced feature in Pipecat that determines when a user has finished speaking and the bot should respond. Unlike basic Voice Activity Detection (VAD) which only detects speech vs. non-speech, Smart Turn Detection uses a machine learning model to recognize natural conversational cues like intonation patterns and linguistic signals. Smart Turn Model Open source model for advanced conversational turn detection. Contribute to model training and development. Data Collector Contribute conversational data to improve the smart-turn model Data Classifier Help classify turn completion patterns in conversations Pipecat provides three implementations of Smart Turn Detection: FalSmartTurnAnalyzer - Uses a Fal’s hosted smart-turn model for inference LocalCoreMLSmartTurnAnalyzer - Runs inference locally on Apple Silicon using CoreML LocalSmartTurnAnalyzer - Runs inference locally using PyTorch and Hugging Face Transformers All implementations share the same underlying API and parameters, making it easy to switch between them based on your deployment requirements. ​ Installation The Smart Turn Detection feature requires additional dependencies depending on which implementation you choose. For Fal’s hosted service inference: Copy Ask AI pip install "pipecat-ai[remote-smart-turn]" For local inference (CoreML-based): Copy Ask AI pip install "pipecat-ai[local-smart-turn]" For local inference (PyTorch-based): Copy Ask AI pip install "pipecat-ai[local-smart-turn]" ​ Integration with Transport Smart Turn Detection is integrated into your application by setting one of the available turn analyzers as the turn_analyzer parameter in your transport configuration: Copy Ask AI from pipecat.transports.base_transport import TransportParams transport = SmallWebRTCTransport( webrtc_connection = webrtc_connection, params = TransportParams( # Other transport parameters... turn_analyzer = FalSmartTurnAnalyzer( url = remote_smart_turn_url), ), ) Smart Turn Detection requires VAD to be enabled and works best when the VAD analyzer is set to a short stop_secs value. We recommend 0.2 seconds. Copy Ask AI audio_in_enabled = True , vad_analyzer = SileroVADAnalyzer( params = VADParams( stop_secs = 0.2 )) ​ Configuration All implementations use the same SmartTurnParams class to configure behavior: ​ stop_secs float default: "3.0" Duration of silence in seconds required before triggering a silence-based end of turn ​ pre_speech_ms float default: "0.0" Amount of audio (in milliseconds) to include before speech is detected ​ max_duration_secs float default: "8.0" Maximum allowed segment duration in seconds. For segments longer than this value, a rolling window is used. ​ Remote Smart Turn The FalSmartTurnAnalyzer class uses a remote service for turn detection inference. ​ Constructor Parameters ​ url str required The URL of the remote Smart Turn service ​ sample_rate Optional[int] default: "None" Audio sample rate (will be set by the transport if not provided) ​ params SmartTurnParams default: "SmartTurnParams()" Configuration parameters for turn detection ​ Example Copy Ask AI import os from pipecat.audio.turn.smart_turn.fal_smart_turn import FalSmartTurnAnalyzer from pipecat.audio.turn.smart_turn.base_smart_turn import SmartTurnParams from pipecat.transports.base_transport import TransportParams # Get the URL for the remote Smart Turn service remote_smart_turn_url = os.getenv( "REMOTE_SMART_TURN_URL" ) # Create the transport with Smart Turn detection transport = SmallWebRTCTransport( webrtc_connection = webrtc_connection, params = TransportParams( audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer( params = VADParams( stop_secs = 0.2 )), turn_analyzer = FalSmartTurnAnalyzer( url = remote_smart_turn_url, params = SmartTurnParams( stop_secs = 3.0 , pre_speech_ms = 0.0 , max_duration_secs = 8.0 ) ), ), ) ​ Local Smart Turn (CoreML) The LocalCoreMLSmartTurnAnalyzer runs inference locally using CoreML, providing lower latency and no network dependencies. ​ Constructor Parameters ​ smart_turn_model_path str required Path to the directory containing the Smart Turn model ​ sample_rate Optional[int] default: "None" Audio sample rate (will be set by the transport if not provided) ​ params SmartTurnParams default: "SmartTurnParams()" Configuration parameters for turn detection ​ Example Copy Ask AI import os from pipecat.audio.turn.smart_turn.local_coreml_smart_turn import LocalCoreMLSmartTurnAnalyzer from pipecat.audio.turn.smart_turn.base_smart_turn import SmartTurnParams from pipecat.transports.base_transport import TransportParams # Path to the Smart Turn model directory smart_turn_model_path = os.getenv( "LOCAL_SMART_TURN_MODEL_PATH" ) # Create the transport with local Smart Turn detection transport = SmallWebRTCTransport( webrtc_connection = webrtc_connection, params = TransportParams( audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer( params = VADParams( stop_secs = 0.2 )), turn_analyzer = LocalCoreMLSmartTurnAnalyzer( smart_turn_model_path = smart_turn_model_path, params = SmartTurnParams( stop_secs = 2.0 , # Shorter stop time when using Smart Turn pre_speech_ms = 0.0 , max_duration_secs = 8.0 ) ), ), ) ​ Local Smart Turn (PyTorch) The LocalSmartTurnAnalyzer runs inference locally using PyTorch and Hugging Face Transformers, providing a cross-platform solution. ​ Constructor Parameters ​ smart_turn_model_path str default: "pipecat-ai/smart-turn" Path to the Smart Turn model or Hugging Face model identifier. Defaults to the official “pipecat-ai/smart-turn” model. ​ sample_rate Optional[int] default: "None" Audio sample rate (will be set by the transport if not provided) ​ params SmartTurnParams default: "SmartTurnParams()" Configuration parameters for turn detection ​ Example Copy Ask AI import os from pipecat.audio.turn.smart_turn.local_smart_turn import LocalSmartTurnAnalyzer from pipecat.audio.turn.smart_turn.base_smart_turn import SmartTurnParams from pipecat.transports.base_transport import TransportParams # Optional: Path to the local Smart Turn model # If not provided, it will download from Hugging Face smart_turn_model_path = os.getenv( "LOCAL_SMART_TURN_MODEL_PATH" ) # Create the transport with PyTorch-based Smart Turn detection transport = SmallWebRTCTransport( webrtc_connection = webrtc_connection, params = TransportParams( audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer( params = VADParams( stop_secs = 0.2 )), turn_analyzer = LocalSmartTurnAnalyzer( smart_turn_model_path = smart_turn_model_path, params = SmartTurnParams( stop_secs = 2.0 , pre_speech_ms = 0.0 , max_duration_secs = 8.0 ) ), ), ) ​ Local Model Setup ​ CoreML Model & PyTorch Setup To use the LocalCoreMLSmartTurnAnalyzer or LocalSmartTurnAnalyzer , you need to set up the model locally: Install Git LFS (Large File Storage): macOS Ubuntu/Debian Copy Ask AI brew install git-lfs Initialize Git LFS Copy Ask AI git lfs install Clone the Smart Turn model repository: Copy Ask AI git clone https://huggingface.co/pipecat-ai/smart-turn Set the environment variable to the cloned repository path: Copy Ask AI # Add to your .env file or environment export LOCAL_SMART_TURN_MODEL_PATH = / path / to / smart-turn Note that the CoreML model is optimized for Apple Silicon devices. If you’re using a different platform, consider using the PyTorch-based LocalSmartTurnAnalyzer or the remote Smart Turn service. Learn more about the CoreML setup in the official repository instructions ​ How It Works Smart Turn Detection continuously analyzes audio streams to identify natural turn completion points: Audio Buffering : The system continuously buffers audio with timestamps, maintaining a small buffer of pre-speech audio. VAD Processing : Voice Activity Detection segments the audio into speech and non-speech portions. Turn Analysis : When VAD detects a pause in speech: The ML model analyzes the speech segment for natural completion cues It identifies acoustic and linguistic patterns that indicate turn completion A decision is made whether the turn is complete or incomplete The system includes a fallback mechanism: if a turn is classified as incomplete but silence continues for longer than stop_secs , the turn is automatically marked as complete. ​ Notes The model is designed for English speech; performance may vary with other languages You can adjust the stop_secs parameter based on your application’s needs for responsiveness Smart Turn generally provides a more natural conversational experience but is computationally more intensive than simple VAD The PyTorch-based LocalSmartTurnAnalyzer runs on CPU by default but will use CUDA if available Daily REST Helper Fal Smart Turn On this page Overview Installation Integration with Transport Configuration Remote Smart Turn Constructor Parameters Example Local Smart Turn (CoreML) Constructor Parameters Example Local Smart Turn (PyTorch) Constructor Parameters Example Local Model Setup CoreML Model & PyTorch Setup How It Works Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_assemblyai_10c27b26.txt b/stt_assemblyai_10c27b26.txt
new file mode 100644
index 0000000000000000000000000000000000000000..1059e8f09eb57820f7428f4a02c65f45b3f73d1f
--- /dev/null
+++ b/stt_assemblyai_10c27b26.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/assemblyai#param-language
+Title: AssemblyAI - Pipecat
+==================================================
+
+AssemblyAI - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text AssemblyAI Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview AssemblyAISTTService provides real-time speech-to-text capabilities using AssemblyAI’s WebSocket API. It supports streaming transcription with both interim and final results. ​ Installation To use AssemblyAISTTService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[assemblyai]" You’ll also need to set up your AssemblyAI API key as an environment variable: ASSEMBLYAI_API_KEY . You can obtain a AssemblyAI API key by signing up at AssemblyAI . ​ Configuration ​ Constructor Parameters ​ api_key str required Your AssemblyAI API key. ​ connection_params AssemblyAIConnectionParams Connection parameters for the AssemblyAI WebSocket connection. See below for details. ​ vad_force_turn_endpoint bool default: "True" When true, sends a ForceEndpoint event to AssemblyAI when a UserStoppedSpeakingFrame is received. Requires a VAD (Voice Activity Detection) processor in the pipeline to generate these frames. ​ language Language default: "Language.EN" Language for transcription. AssemblyAI currently only supports English Streaming transcription. ​ api_endpoint_base_url str Base URL for the WebSocket API endpoint. ​ Connection Parameters ​ sample_rate int default: "16000" The sample rate of the audio stream ​ encoding str default: "pcm_s16le" The encoding of the audio stream. Allowed values: pcm_s16le , pcm_mulaw ​ formatted_finals bool default: "True" Whether to return formatted final transcripts. If enabled, formatted final transcripts will be emitted shortly following an end-of-turn detection. ​ word_finalization_max_wait_time int The max amount of time in milliseconds to wait for a word to be finalized. ​ end_of_turn_confidence_threshold float The confidence threshold to use when determining if the end of a turn has been reached. ​ min_end_of_turn_silence_when_confident int The minimum amount of silence required to detect end of turn when confident. ​ max_turn_silence int The maximum amount of silence allowed in a turn before end of turn is triggered. ​ Input The service processes raw audio data with the following requirements: PCM audio format 16-bit depth 16kHz sample rate (default) Single channel (mono) ​ Output Frames The service produces two types of frames during transcription: ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Transcription language ​ InterimTranscriptionFrame Generated during ongoing speech, containing the same fields as TranscriptionFrame but with preliminary results. ​ Methods See the STT base class methods for additional functionality. ​ Language Support AssemblyAI Streaming STT currently only supports English. ​ Usage Example Copy Ask AI from pipecat.services.assemblyai.stt import AssemblyAISTTService # Configure service stt = AssemblyAISTTService( api_key = os.getenv( "ASSEMBLYAI_API_KEY" ), ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ Frame Flow ​ Metrics Support The service collects processing metrics: Time to First Byte (TTFB) Processing duration Connection status ​ Notes Currently supports English-only real-time transcription Handles WebSocket connection management Provides both interim and final transcriptions Thread-safe processing with proper event loop handling Automatic error handling and reporting Manages connection lifecycle TelnyxFrameSerializer AWS Transcribe On this page Overview Installation Configuration Constructor Parameters Connection Parameters Input Output Frames TranscriptionFrame InterimTranscriptionFrame Methods Language Support Usage Example Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_assemblyai_297aeb20.txt b/stt_assemblyai_297aeb20.txt
new file mode 100644
index 0000000000000000000000000000000000000000..513f7f1fbbe4f6ad9c8c999e8c2d60d8fb53dc33
--- /dev/null
+++ b/stt_assemblyai_297aeb20.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/assemblyai#param-vad-force-turn-endpoint
+Title: AssemblyAI - Pipecat
+==================================================
+
+AssemblyAI - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text AssemblyAI Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview AssemblyAISTTService provides real-time speech-to-text capabilities using AssemblyAI’s WebSocket API. It supports streaming transcription with both interim and final results. ​ Installation To use AssemblyAISTTService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[assemblyai]" You’ll also need to set up your AssemblyAI API key as an environment variable: ASSEMBLYAI_API_KEY . You can obtain a AssemblyAI API key by signing up at AssemblyAI . ​ Configuration ​ Constructor Parameters ​ api_key str required Your AssemblyAI API key. ​ connection_params AssemblyAIConnectionParams Connection parameters for the AssemblyAI WebSocket connection. See below for details. ​ vad_force_turn_endpoint bool default: "True" When true, sends a ForceEndpoint event to AssemblyAI when a UserStoppedSpeakingFrame is received. Requires a VAD (Voice Activity Detection) processor in the pipeline to generate these frames. ​ language Language default: "Language.EN" Language for transcription. AssemblyAI currently only supports English Streaming transcription. ​ api_endpoint_base_url str Base URL for the WebSocket API endpoint. ​ Connection Parameters ​ sample_rate int default: "16000" The sample rate of the audio stream ​ encoding str default: "pcm_s16le" The encoding of the audio stream. Allowed values: pcm_s16le , pcm_mulaw ​ formatted_finals bool default: "True" Whether to return formatted final transcripts. If enabled, formatted final transcripts will be emitted shortly following an end-of-turn detection. ​ word_finalization_max_wait_time int The max amount of time in milliseconds to wait for a word to be finalized. ​ end_of_turn_confidence_threshold float The confidence threshold to use when determining if the end of a turn has been reached. ​ min_end_of_turn_silence_when_confident int The minimum amount of silence required to detect end of turn when confident. ​ max_turn_silence int The maximum amount of silence allowed in a turn before end of turn is triggered. ​ Input The service processes raw audio data with the following requirements: PCM audio format 16-bit depth 16kHz sample rate (default) Single channel (mono) ​ Output Frames The service produces two types of frames during transcription: ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Transcription language ​ InterimTranscriptionFrame Generated during ongoing speech, containing the same fields as TranscriptionFrame but with preliminary results. ​ Methods See the STT base class methods for additional functionality. ​ Language Support AssemblyAI Streaming STT currently only supports English. ​ Usage Example Copy Ask AI from pipecat.services.assemblyai.stt import AssemblyAISTTService # Configure service stt = AssemblyAISTTService( api_key = os.getenv( "ASSEMBLYAI_API_KEY" ), ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ Frame Flow ​ Metrics Support The service collects processing metrics: Time to First Byte (TTFB) Processing duration Connection status ​ Notes Currently supports English-only real-time transcription Handles WebSocket connection management Provides both interim and final transcriptions Thread-safe processing with proper event loop handling Automatic error handling and reporting Manages connection lifecycle TelnyxFrameSerializer AWS Transcribe On this page Overview Installation Configuration Constructor Parameters Connection Parameters Input Output Frames TranscriptionFrame InterimTranscriptionFrame Methods Language Support Usage Example Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_assemblyai_536eda98.txt b/stt_assemblyai_536eda98.txt
new file mode 100644
index 0000000000000000000000000000000000000000..f7cd89bd310624780584f7d73309f1814dc2759b
--- /dev/null
+++ b/stt_assemblyai_536eda98.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/assemblyai#connection-parameters
+Title: AssemblyAI - Pipecat
+==================================================
+
+AssemblyAI - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text AssemblyAI Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview AssemblyAISTTService provides real-time speech-to-text capabilities using AssemblyAI’s WebSocket API. It supports streaming transcription with both interim and final results. ​ Installation To use AssemblyAISTTService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[assemblyai]" You’ll also need to set up your AssemblyAI API key as an environment variable: ASSEMBLYAI_API_KEY . You can obtain a AssemblyAI API key by signing up at AssemblyAI . ​ Configuration ​ Constructor Parameters ​ api_key str required Your AssemblyAI API key. ​ connection_params AssemblyAIConnectionParams Connection parameters for the AssemblyAI WebSocket connection. See below for details. ​ vad_force_turn_endpoint bool default: "True" When true, sends a ForceEndpoint event to AssemblyAI when a UserStoppedSpeakingFrame is received. Requires a VAD (Voice Activity Detection) processor in the pipeline to generate these frames. ​ language Language default: "Language.EN" Language for transcription. AssemblyAI currently only supports English Streaming transcription. ​ api_endpoint_base_url str Base URL for the WebSocket API endpoint. ​ Connection Parameters ​ sample_rate int default: "16000" The sample rate of the audio stream ​ encoding str default: "pcm_s16le" The encoding of the audio stream. Allowed values: pcm_s16le , pcm_mulaw ​ formatted_finals bool default: "True" Whether to return formatted final transcripts. If enabled, formatted final transcripts will be emitted shortly following an end-of-turn detection. ​ word_finalization_max_wait_time int The max amount of time in milliseconds to wait for a word to be finalized. ​ end_of_turn_confidence_threshold float The confidence threshold to use when determining if the end of a turn has been reached. ​ min_end_of_turn_silence_when_confident int The minimum amount of silence required to detect end of turn when confident. ​ max_turn_silence int The maximum amount of silence allowed in a turn before end of turn is triggered. ​ Input The service processes raw audio data with the following requirements: PCM audio format 16-bit depth 16kHz sample rate (default) Single channel (mono) ​ Output Frames The service produces two types of frames during transcription: ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Transcription language ​ InterimTranscriptionFrame Generated during ongoing speech, containing the same fields as TranscriptionFrame but with preliminary results. ​ Methods See the STT base class methods for additional functionality. ​ Language Support AssemblyAI Streaming STT currently only supports English. ​ Usage Example Copy Ask AI from pipecat.services.assemblyai.stt import AssemblyAISTTService # Configure service stt = AssemblyAISTTService( api_key = os.getenv( "ASSEMBLYAI_API_KEY" ), ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ Frame Flow ​ Metrics Support The service collects processing metrics: Time to First Byte (TTFB) Processing duration Connection status ​ Notes Currently supports English-only real-time transcription Handles WebSocket connection management Provides both interim and final transcriptions Thread-safe processing with proper event loop handling Automatic error handling and reporting Manages connection lifecycle TelnyxFrameSerializer AWS Transcribe On this page Overview Installation Configuration Constructor Parameters Connection Parameters Input Output Frames TranscriptionFrame InterimTranscriptionFrame Methods Language Support Usage Example Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_assemblyai_6e0c18cd.txt b/stt_assemblyai_6e0c18cd.txt
new file mode 100644
index 0000000000000000000000000000000000000000..6f07e01a5f6d83f0a8249ec42ca6e0caa33e5e1e
--- /dev/null
+++ b/stt_assemblyai_6e0c18cd.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/assemblyai#param-user-id
+Title: AssemblyAI - Pipecat
+==================================================
+
+AssemblyAI - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text AssemblyAI Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview AssemblyAISTTService provides real-time speech-to-text capabilities using AssemblyAI’s WebSocket API. It supports streaming transcription with both interim and final results. ​ Installation To use AssemblyAISTTService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[assemblyai]" You’ll also need to set up your AssemblyAI API key as an environment variable: ASSEMBLYAI_API_KEY . You can obtain a AssemblyAI API key by signing up at AssemblyAI . ​ Configuration ​ Constructor Parameters ​ api_key str required Your AssemblyAI API key. ​ connection_params AssemblyAIConnectionParams Connection parameters for the AssemblyAI WebSocket connection. See below for details. ​ vad_force_turn_endpoint bool default: "True" When true, sends a ForceEndpoint event to AssemblyAI when a UserStoppedSpeakingFrame is received. Requires a VAD (Voice Activity Detection) processor in the pipeline to generate these frames. ​ language Language default: "Language.EN" Language for transcription. AssemblyAI currently only supports English Streaming transcription. ​ api_endpoint_base_url str Base URL for the WebSocket API endpoint. ​ Connection Parameters ​ sample_rate int default: "16000" The sample rate of the audio stream ​ encoding str default: "pcm_s16le" The encoding of the audio stream. Allowed values: pcm_s16le , pcm_mulaw ​ formatted_finals bool default: "True" Whether to return formatted final transcripts. If enabled, formatted final transcripts will be emitted shortly following an end-of-turn detection. ​ word_finalization_max_wait_time int The max amount of time in milliseconds to wait for a word to be finalized. ​ end_of_turn_confidence_threshold float The confidence threshold to use when determining if the end of a turn has been reached. ​ min_end_of_turn_silence_when_confident int The minimum amount of silence required to detect end of turn when confident. ​ max_turn_silence int The maximum amount of silence allowed in a turn before end of turn is triggered. ​ Input The service processes raw audio data with the following requirements: PCM audio format 16-bit depth 16kHz sample rate (default) Single channel (mono) ​ Output Frames The service produces two types of frames during transcription: ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Transcription language ​ InterimTranscriptionFrame Generated during ongoing speech, containing the same fields as TranscriptionFrame but with preliminary results. ​ Methods See the STT base class methods for additional functionality. ​ Language Support AssemblyAI Streaming STT currently only supports English. ​ Usage Example Copy Ask AI from pipecat.services.assemblyai.stt import AssemblyAISTTService # Configure service stt = AssemblyAISTTService( api_key = os.getenv( "ASSEMBLYAI_API_KEY" ), ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ Frame Flow ​ Metrics Support The service collects processing metrics: Time to First Byte (TTFB) Processing duration Connection status ​ Notes Currently supports English-only real-time transcription Handles WebSocket connection management Provides both interim and final transcriptions Thread-safe processing with proper event loop handling Automatic error handling and reporting Manages connection lifecycle TelnyxFrameSerializer AWS Transcribe On this page Overview Installation Configuration Constructor Parameters Connection Parameters Input Output Frames TranscriptionFrame InterimTranscriptionFrame Methods Language Support Usage Example Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_assemblyai_9b489df8.txt b/stt_assemblyai_9b489df8.txt
new file mode 100644
index 0000000000000000000000000000000000000000..632106bc90cfad92e18a99315330912b4822f82d
--- /dev/null
+++ b/stt_assemblyai_9b489df8.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/assemblyai#frame-flow
+Title: AssemblyAI - Pipecat
+==================================================
+
+AssemblyAI - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text AssemblyAI Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview AssemblyAISTTService provides real-time speech-to-text capabilities using AssemblyAI’s WebSocket API. It supports streaming transcription with both interim and final results. ​ Installation To use AssemblyAISTTService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[assemblyai]" You’ll also need to set up your AssemblyAI API key as an environment variable: ASSEMBLYAI_API_KEY . You can obtain a AssemblyAI API key by signing up at AssemblyAI . ​ Configuration ​ Constructor Parameters ​ api_key str required Your AssemblyAI API key. ​ connection_params AssemblyAIConnectionParams Connection parameters for the AssemblyAI WebSocket connection. See below for details. ​ vad_force_turn_endpoint bool default: "True" When true, sends a ForceEndpoint event to AssemblyAI when a UserStoppedSpeakingFrame is received. Requires a VAD (Voice Activity Detection) processor in the pipeline to generate these frames. ​ language Language default: "Language.EN" Language for transcription. AssemblyAI currently only supports English Streaming transcription. ​ api_endpoint_base_url str Base URL for the WebSocket API endpoint. ​ Connection Parameters ​ sample_rate int default: "16000" The sample rate of the audio stream ​ encoding str default: "pcm_s16le" The encoding of the audio stream. Allowed values: pcm_s16le , pcm_mulaw ​ formatted_finals bool default: "True" Whether to return formatted final transcripts. If enabled, formatted final transcripts will be emitted shortly following an end-of-turn detection. ​ word_finalization_max_wait_time int The max amount of time in milliseconds to wait for a word to be finalized. ​ end_of_turn_confidence_threshold float The confidence threshold to use when determining if the end of a turn has been reached. ​ min_end_of_turn_silence_when_confident int The minimum amount of silence required to detect end of turn when confident. ​ max_turn_silence int The maximum amount of silence allowed in a turn before end of turn is triggered. ​ Input The service processes raw audio data with the following requirements: PCM audio format 16-bit depth 16kHz sample rate (default) Single channel (mono) ​ Output Frames The service produces two types of frames during transcription: ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Transcription language ​ InterimTranscriptionFrame Generated during ongoing speech, containing the same fields as TranscriptionFrame but with preliminary results. ​ Methods See the STT base class methods for additional functionality. ​ Language Support AssemblyAI Streaming STT currently only supports English. ​ Usage Example Copy Ask AI from pipecat.services.assemblyai.stt import AssemblyAISTTService # Configure service stt = AssemblyAISTTService( api_key = os.getenv( "ASSEMBLYAI_API_KEY" ), ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ Frame Flow ​ Metrics Support The service collects processing metrics: Time to First Byte (TTFB) Processing duration Connection status ​ Notes Currently supports English-only real-time transcription Handles WebSocket connection management Provides both interim and final transcriptions Thread-safe processing with proper event loop handling Automatic error handling and reporting Manages connection lifecycle TelnyxFrameSerializer AWS Transcribe On this page Overview Installation Configuration Constructor Parameters Connection Parameters Input Output Frames TranscriptionFrame InterimTranscriptionFrame Methods Language Support Usage Example Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_assemblyai_ab948aaf.txt b/stt_assemblyai_ab948aaf.txt
new file mode 100644
index 0000000000000000000000000000000000000000..84935951c59c056d8cc0f8b54cce778a1d107ff8
--- /dev/null
+++ b/stt_assemblyai_ab948aaf.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/assemblyai#param-text
+Title: AssemblyAI - Pipecat
+==================================================
+
+AssemblyAI - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text AssemblyAI Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview AssemblyAISTTService provides real-time speech-to-text capabilities using AssemblyAI’s WebSocket API. It supports streaming transcription with both interim and final results. ​ Installation To use AssemblyAISTTService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[assemblyai]" You’ll also need to set up your AssemblyAI API key as an environment variable: ASSEMBLYAI_API_KEY . You can obtain a AssemblyAI API key by signing up at AssemblyAI . ​ Configuration ​ Constructor Parameters ​ api_key str required Your AssemblyAI API key. ​ connection_params AssemblyAIConnectionParams Connection parameters for the AssemblyAI WebSocket connection. See below for details. ​ vad_force_turn_endpoint bool default: "True" When true, sends a ForceEndpoint event to AssemblyAI when a UserStoppedSpeakingFrame is received. Requires a VAD (Voice Activity Detection) processor in the pipeline to generate these frames. ​ language Language default: "Language.EN" Language for transcription. AssemblyAI currently only supports English Streaming transcription. ​ api_endpoint_base_url str Base URL for the WebSocket API endpoint. ​ Connection Parameters ​ sample_rate int default: "16000" The sample rate of the audio stream ​ encoding str default: "pcm_s16le" The encoding of the audio stream. Allowed values: pcm_s16le , pcm_mulaw ​ formatted_finals bool default: "True" Whether to return formatted final transcripts. If enabled, formatted final transcripts will be emitted shortly following an end-of-turn detection. ​ word_finalization_max_wait_time int The max amount of time in milliseconds to wait for a word to be finalized. ​ end_of_turn_confidence_threshold float The confidence threshold to use when determining if the end of a turn has been reached. ​ min_end_of_turn_silence_when_confident int The minimum amount of silence required to detect end of turn when confident. ​ max_turn_silence int The maximum amount of silence allowed in a turn before end of turn is triggered. ​ Input The service processes raw audio data with the following requirements: PCM audio format 16-bit depth 16kHz sample rate (default) Single channel (mono) ​ Output Frames The service produces two types of frames during transcription: ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Transcription language ​ InterimTranscriptionFrame Generated during ongoing speech, containing the same fields as TranscriptionFrame but with preliminary results. ​ Methods See the STT base class methods for additional functionality. ​ Language Support AssemblyAI Streaming STT currently only supports English. ​ Usage Example Copy Ask AI from pipecat.services.assemblyai.stt import AssemblyAISTTService # Configure service stt = AssemblyAISTTService( api_key = os.getenv( "ASSEMBLYAI_API_KEY" ), ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ Frame Flow ​ Metrics Support The service collects processing metrics: Time to First Byte (TTFB) Processing duration Connection status ​ Notes Currently supports English-only real-time transcription Handles WebSocket connection management Provides both interim and final transcriptions Thread-safe processing with proper event loop handling Automatic error handling and reporting Manages connection lifecycle TelnyxFrameSerializer AWS Transcribe On this page Overview Installation Configuration Constructor Parameters Connection Parameters Input Output Frames TranscriptionFrame InterimTranscriptionFrame Methods Language Support Usage Example Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_assemblyai_c1e5f26b.txt b/stt_assemblyai_c1e5f26b.txt
new file mode 100644
index 0000000000000000000000000000000000000000..89287188b02ef825e1337eb8cab3cd857ddd3560
--- /dev/null
+++ b/stt_assemblyai_c1e5f26b.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/assemblyai
+Title: AssemblyAI - Pipecat
+==================================================
+
+AssemblyAI - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text AssemblyAI Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview AssemblyAISTTService provides real-time speech-to-text capabilities using AssemblyAI’s WebSocket API. It supports streaming transcription with both interim and final results. ​ Installation To use AssemblyAISTTService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[assemblyai]" You’ll also need to set up your AssemblyAI API key as an environment variable: ASSEMBLYAI_API_KEY . You can obtain a AssemblyAI API key by signing up at AssemblyAI . ​ Configuration ​ Constructor Parameters ​ api_key str required Your AssemblyAI API key. ​ connection_params AssemblyAIConnectionParams Connection parameters for the AssemblyAI WebSocket connection. See below for details. ​ vad_force_turn_endpoint bool default: "True" When true, sends a ForceEndpoint event to AssemblyAI when a UserStoppedSpeakingFrame is received. Requires a VAD (Voice Activity Detection) processor in the pipeline to generate these frames. ​ language Language default: "Language.EN" Language for transcription. AssemblyAI currently only supports English Streaming transcription. ​ api_endpoint_base_url str Base URL for the WebSocket API endpoint. ​ Connection Parameters ​ sample_rate int default: "16000" The sample rate of the audio stream ​ encoding str default: "pcm_s16le" The encoding of the audio stream. Allowed values: pcm_s16le , pcm_mulaw ​ formatted_finals bool default: "True" Whether to return formatted final transcripts. If enabled, formatted final transcripts will be emitted shortly following an end-of-turn detection. ​ word_finalization_max_wait_time int The max amount of time in milliseconds to wait for a word to be finalized. ​ end_of_turn_confidence_threshold float The confidence threshold to use when determining if the end of a turn has been reached. ​ min_end_of_turn_silence_when_confident int The minimum amount of silence required to detect end of turn when confident. ​ max_turn_silence int The maximum amount of silence allowed in a turn before end of turn is triggered. ​ Input The service processes raw audio data with the following requirements: PCM audio format 16-bit depth 16kHz sample rate (default) Single channel (mono) ​ Output Frames The service produces two types of frames during transcription: ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Transcription language ​ InterimTranscriptionFrame Generated during ongoing speech, containing the same fields as TranscriptionFrame but with preliminary results. ​ Methods See the STT base class methods for additional functionality. ​ Language Support AssemblyAI Streaming STT currently only supports English. ​ Usage Example Copy Ask AI from pipecat.services.assemblyai.stt import AssemblyAISTTService # Configure service stt = AssemblyAISTTService( api_key = os.getenv( "ASSEMBLYAI_API_KEY" ), ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ Frame Flow ​ Metrics Support The service collects processing metrics: Time to First Byte (TTFB) Processing duration Connection status ​ Notes Currently supports English-only real-time transcription Handles WebSocket connection management Provides both interim and final transcriptions Thread-safe processing with proper event loop handling Automatic error handling and reporting Manages connection lifecycle TelnyxFrameSerializer AWS Transcribe On this page Overview Installation Configuration Constructor Parameters Connection Parameters Input Output Frames TranscriptionFrame InterimTranscriptionFrame Methods Language Support Usage Example Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_assemblyai_cc35799a.txt b/stt_assemblyai_cc35799a.txt
new file mode 100644
index 0000000000000000000000000000000000000000..007c02ce08f2ddd0cdb6532b73ea77a26b034ce8
--- /dev/null
+++ b/stt_assemblyai_cc35799a.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/assemblyai#overview
+Title: AssemblyAI - Pipecat
+==================================================
+
+AssemblyAI - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text AssemblyAI Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview AssemblyAISTTService provides real-time speech-to-text capabilities using AssemblyAI’s WebSocket API. It supports streaming transcription with both interim and final results. ​ Installation To use AssemblyAISTTService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[assemblyai]" You’ll also need to set up your AssemblyAI API key as an environment variable: ASSEMBLYAI_API_KEY . You can obtain a AssemblyAI API key by signing up at AssemblyAI . ​ Configuration ​ Constructor Parameters ​ api_key str required Your AssemblyAI API key. ​ connection_params AssemblyAIConnectionParams Connection parameters for the AssemblyAI WebSocket connection. See below for details. ​ vad_force_turn_endpoint bool default: "True" When true, sends a ForceEndpoint event to AssemblyAI when a UserStoppedSpeakingFrame is received. Requires a VAD (Voice Activity Detection) processor in the pipeline to generate these frames. ​ language Language default: "Language.EN" Language for transcription. AssemblyAI currently only supports English Streaming transcription. ​ api_endpoint_base_url str Base URL for the WebSocket API endpoint. ​ Connection Parameters ​ sample_rate int default: "16000" The sample rate of the audio stream ​ encoding str default: "pcm_s16le" The encoding of the audio stream. Allowed values: pcm_s16le , pcm_mulaw ​ formatted_finals bool default: "True" Whether to return formatted final transcripts. If enabled, formatted final transcripts will be emitted shortly following an end-of-turn detection. ​ word_finalization_max_wait_time int The max amount of time in milliseconds to wait for a word to be finalized. ​ end_of_turn_confidence_threshold float The confidence threshold to use when determining if the end of a turn has been reached. ​ min_end_of_turn_silence_when_confident int The minimum amount of silence required to detect end of turn when confident. ​ max_turn_silence int The maximum amount of silence allowed in a turn before end of turn is triggered. ​ Input The service processes raw audio data with the following requirements: PCM audio format 16-bit depth 16kHz sample rate (default) Single channel (mono) ​ Output Frames The service produces two types of frames during transcription: ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Transcription language ​ InterimTranscriptionFrame Generated during ongoing speech, containing the same fields as TranscriptionFrame but with preliminary results. ​ Methods See the STT base class methods for additional functionality. ​ Language Support AssemblyAI Streaming STT currently only supports English. ​ Usage Example Copy Ask AI from pipecat.services.assemblyai.stt import AssemblyAISTTService # Configure service stt = AssemblyAISTTService( api_key = os.getenv( "ASSEMBLYAI_API_KEY" ), ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ Frame Flow ​ Metrics Support The service collects processing metrics: Time to First Byte (TTFB) Processing duration Connection status ​ Notes Currently supports English-only real-time transcription Handles WebSocket connection management Provides both interim and final transcriptions Thread-safe processing with proper event loop handling Automatic error handling and reporting Manages connection lifecycle TelnyxFrameSerializer AWS Transcribe On this page Overview Installation Configuration Constructor Parameters Connection Parameters Input Output Frames TranscriptionFrame InterimTranscriptionFrame Methods Language Support Usage Example Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_aws_2adfbf07.txt b/stt_aws_2adfbf07.txt
new file mode 100644
index 0000000000000000000000000000000000000000..8080c54ce54ef635d6c6686dd24120f8c5fac342
--- /dev/null
+++ b/stt_aws_2adfbf07.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/aws#output-frames
+Title: AWS Transcribe - Pipecat
+==================================================
+
+AWS Transcribe - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text AWS Transcribe Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview AWSTranscribeSTTService provides real-time speech-to-text capabilities using Amazon Transcribe’s WebSocket API. It supports interim results, adjustable quality levels, and can handle continuous audio streams. ​ Installation To use AWSTranscribeSTTService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[aws]" You’ll also need to set up your AWS credentials as environment variables: AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_SESSION_TOKEN (if using temporary credentials) AWS_REGION (defaults to “us-east-1”) You can obtain AWS credentials by setting up an IAM user with access to Amazon Transcribe in your AWS account. ​ Configuration ​ Constructor Parameters ​ api_key str Your AWS secret access key (can also use environment variable) ​ aws_access_key_id str Your AWS access key ID (can also use environment variable) ​ aws_session_token str Your AWS session token for temporary credentials (can also use environment variable) ​ region str default: "us-east-1" AWS region to use for Transcribe service ​ sample_rate int default: "16000" Audio sample rate in Hz (only 8000 Hz or 16000 Hz are supported) ​ language Language default: "Language.EN" Language for transcription ​ Default Settings Copy Ask AI { "sample_rate" : 16000 , "language" : Language. EN , "media_encoding" : "linear16" , # AWS expects raw PCM "number_of_channels" : 1 , "show_speaker_label" : False , "enable_channel_identification" : False } ​ Input The service processes InputAudioRawFrame instances containing: Raw PCM audio data 16-bit depth 8kHz or 16kHz sample rate (will convert to 16kHz if another rate is provided) Single channel (mono) ​ Output Frames The service produces two types of frames during transcription: ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Language used for transcription ​ InterimTranscriptionFrame Generated during ongoing speech, containing same fields as TranscriptionFrame but with preliminary results. ​ Methods See the STT base class methods for additional functionality. ​ Language Setting Copy Ask AI await service.set_language(Language. FR ) ​ Usage Example Copy Ask AI from pipecat.services.aws.stt import AWSTranscribeSTTService # Configure service using environment variables for credentials stt = AWSTranscribeSTTService( region = "us-west-2" , sample_rate = 16000 , language = Language. EN ) # Or provide credentials directly stt = AWSTranscribeSTTService( aws_access_key_id = "YOUR_ACCESS_KEY_ID" , api_key = "YOUR_SECRET_ACCESS_KEY" , region = "us-west-2" , sample_rate = 16000 , language = Language. EN ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ Language Support AWS Transcribe STT supports the following languages: Language Code Description Service Codes Language.EN English (US) en-US Language.ES Spanish es-US Language.FR French fr-FR Language.DE German de-DE Language.IT Italian it-IT Language.PT Portuguese (Brazil) pt-BR Language.JA Japanese ja-JP Language.KO Korean ko-KR Language.ZH Chinese (Mandarin) zh-CN AWS Transcribe supports additional languages and regional variants. See the AWS Transcribe documentation for a complete list. ​ Frame Flow ​ Metrics Support The service supports the following metrics: Time to First Byte (TTFB) Processing duration ​ Notes Requires valid AWS credentials with access to Amazon Transcribe Supports real-time transcription with interim results Handles WebSocket connection management and reconnection Only supports mono audio (single channel) Automatically handles audio format conversion to PCM Manages connection lifecycle (start, stop, cancel) AssemblyAI Azure On this page Overview Installation Configuration Constructor Parameters Default Settings Input Output Frames TranscriptionFrame InterimTranscriptionFrame Methods Language Setting Usage Example Language Support Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_aws_5cba929f.txt b/stt_aws_5cba929f.txt
new file mode 100644
index 0000000000000000000000000000000000000000..e3e391e93f1436a826fcf210490a1b9f889279a0
--- /dev/null
+++ b/stt_aws_5cba929f.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/aws
+Title: AWS Transcribe - Pipecat
+==================================================
+
+AWS Transcribe - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text AWS Transcribe Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview AWSTranscribeSTTService provides real-time speech-to-text capabilities using Amazon Transcribe’s WebSocket API. It supports interim results, adjustable quality levels, and can handle continuous audio streams. ​ Installation To use AWSTranscribeSTTService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[aws]" You’ll also need to set up your AWS credentials as environment variables: AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_SESSION_TOKEN (if using temporary credentials) AWS_REGION (defaults to “us-east-1”) You can obtain AWS credentials by setting up an IAM user with access to Amazon Transcribe in your AWS account. ​ Configuration ​ Constructor Parameters ​ api_key str Your AWS secret access key (can also use environment variable) ​ aws_access_key_id str Your AWS access key ID (can also use environment variable) ​ aws_session_token str Your AWS session token for temporary credentials (can also use environment variable) ​ region str default: "us-east-1" AWS region to use for Transcribe service ​ sample_rate int default: "16000" Audio sample rate in Hz (only 8000 Hz or 16000 Hz are supported) ​ language Language default: "Language.EN" Language for transcription ​ Default Settings Copy Ask AI { "sample_rate" : 16000 , "language" : Language. EN , "media_encoding" : "linear16" , # AWS expects raw PCM "number_of_channels" : 1 , "show_speaker_label" : False , "enable_channel_identification" : False } ​ Input The service processes InputAudioRawFrame instances containing: Raw PCM audio data 16-bit depth 8kHz or 16kHz sample rate (will convert to 16kHz if another rate is provided) Single channel (mono) ​ Output Frames The service produces two types of frames during transcription: ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Language used for transcription ​ InterimTranscriptionFrame Generated during ongoing speech, containing same fields as TranscriptionFrame but with preliminary results. ​ Methods See the STT base class methods for additional functionality. ​ Language Setting Copy Ask AI await service.set_language(Language. FR ) ​ Usage Example Copy Ask AI from pipecat.services.aws.stt import AWSTranscribeSTTService # Configure service using environment variables for credentials stt = AWSTranscribeSTTService( region = "us-west-2" , sample_rate = 16000 , language = Language. EN ) # Or provide credentials directly stt = AWSTranscribeSTTService( aws_access_key_id = "YOUR_ACCESS_KEY_ID" , api_key = "YOUR_SECRET_ACCESS_KEY" , region = "us-west-2" , sample_rate = 16000 , language = Language. EN ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ Language Support AWS Transcribe STT supports the following languages: Language Code Description Service Codes Language.EN English (US) en-US Language.ES Spanish es-US Language.FR French fr-FR Language.DE German de-DE Language.IT Italian it-IT Language.PT Portuguese (Brazil) pt-BR Language.JA Japanese ja-JP Language.KO Korean ko-KR Language.ZH Chinese (Mandarin) zh-CN AWS Transcribe supports additional languages and regional variants. See the AWS Transcribe documentation for a complete list. ​ Frame Flow ​ Metrics Support The service supports the following metrics: Time to First Byte (TTFB) Processing duration ​ Notes Requires valid AWS credentials with access to Amazon Transcribe Supports real-time transcription with interim results Handles WebSocket connection management and reconnection Only supports mono audio (single channel) Automatically handles audio format conversion to PCM Manages connection lifecycle (start, stop, cancel) AssemblyAI Azure On this page Overview Installation Configuration Constructor Parameters Default Settings Input Output Frames TranscriptionFrame InterimTranscriptionFrame Methods Language Setting Usage Example Language Support Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_aws_941d3471.txt b/stt_aws_941d3471.txt
new file mode 100644
index 0000000000000000000000000000000000000000..d1530832a77ef6b83e83f377c294dd07ac1c7320
--- /dev/null
+++ b/stt_aws_941d3471.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/aws#language-setting
+Title: AWS Transcribe - Pipecat
+==================================================
+
+AWS Transcribe - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text AWS Transcribe Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview AWSTranscribeSTTService provides real-time speech-to-text capabilities using Amazon Transcribe’s WebSocket API. It supports interim results, adjustable quality levels, and can handle continuous audio streams. ​ Installation To use AWSTranscribeSTTService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[aws]" You’ll also need to set up your AWS credentials as environment variables: AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_SESSION_TOKEN (if using temporary credentials) AWS_REGION (defaults to “us-east-1”) You can obtain AWS credentials by setting up an IAM user with access to Amazon Transcribe in your AWS account. ​ Configuration ​ Constructor Parameters ​ api_key str Your AWS secret access key (can also use environment variable) ​ aws_access_key_id str Your AWS access key ID (can also use environment variable) ​ aws_session_token str Your AWS session token for temporary credentials (can also use environment variable) ​ region str default: "us-east-1" AWS region to use for Transcribe service ​ sample_rate int default: "16000" Audio sample rate in Hz (only 8000 Hz or 16000 Hz are supported) ​ language Language default: "Language.EN" Language for transcription ​ Default Settings Copy Ask AI { "sample_rate" : 16000 , "language" : Language. EN , "media_encoding" : "linear16" , # AWS expects raw PCM "number_of_channels" : 1 , "show_speaker_label" : False , "enable_channel_identification" : False } ​ Input The service processes InputAudioRawFrame instances containing: Raw PCM audio data 16-bit depth 8kHz or 16kHz sample rate (will convert to 16kHz if another rate is provided) Single channel (mono) ​ Output Frames The service produces two types of frames during transcription: ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Language used for transcription ​ InterimTranscriptionFrame Generated during ongoing speech, containing same fields as TranscriptionFrame but with preliminary results. ​ Methods See the STT base class methods for additional functionality. ​ Language Setting Copy Ask AI await service.set_language(Language. FR ) ​ Usage Example Copy Ask AI from pipecat.services.aws.stt import AWSTranscribeSTTService # Configure service using environment variables for credentials stt = AWSTranscribeSTTService( region = "us-west-2" , sample_rate = 16000 , language = Language. EN ) # Or provide credentials directly stt = AWSTranscribeSTTService( aws_access_key_id = "YOUR_ACCESS_KEY_ID" , api_key = "YOUR_SECRET_ACCESS_KEY" , region = "us-west-2" , sample_rate = 16000 , language = Language. EN ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ Language Support AWS Transcribe STT supports the following languages: Language Code Description Service Codes Language.EN English (US) en-US Language.ES Spanish es-US Language.FR French fr-FR Language.DE German de-DE Language.IT Italian it-IT Language.PT Portuguese (Brazil) pt-BR Language.JA Japanese ja-JP Language.KO Korean ko-KR Language.ZH Chinese (Mandarin) zh-CN AWS Transcribe supports additional languages and regional variants. See the AWS Transcribe documentation for a complete list. ​ Frame Flow ​ Metrics Support The service supports the following metrics: Time to First Byte (TTFB) Processing duration ​ Notes Requires valid AWS credentials with access to Amazon Transcribe Supports real-time transcription with interim results Handles WebSocket connection management and reconnection Only supports mono audio (single channel) Automatically handles audio format conversion to PCM Manages connection lifecycle (start, stop, cancel) AssemblyAI Azure On this page Overview Installation Configuration Constructor Parameters Default Settings Input Output Frames TranscriptionFrame InterimTranscriptionFrame Methods Language Setting Usage Example Language Support Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_aws_95a5a488.txt b/stt_aws_95a5a488.txt
new file mode 100644
index 0000000000000000000000000000000000000000..c6057bf53f1f7e26dfbf57c2e5bf656ac27760cb
--- /dev/null
+++ b/stt_aws_95a5a488.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/aws#language-support
+Title: AWS Transcribe - Pipecat
+==================================================
+
+AWS Transcribe - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text AWS Transcribe Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview AWSTranscribeSTTService provides real-time speech-to-text capabilities using Amazon Transcribe’s WebSocket API. It supports interim results, adjustable quality levels, and can handle continuous audio streams. ​ Installation To use AWSTranscribeSTTService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[aws]" You’ll also need to set up your AWS credentials as environment variables: AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_SESSION_TOKEN (if using temporary credentials) AWS_REGION (defaults to “us-east-1”) You can obtain AWS credentials by setting up an IAM user with access to Amazon Transcribe in your AWS account. ​ Configuration ​ Constructor Parameters ​ api_key str Your AWS secret access key (can also use environment variable) ​ aws_access_key_id str Your AWS access key ID (can also use environment variable) ​ aws_session_token str Your AWS session token for temporary credentials (can also use environment variable) ​ region str default: "us-east-1" AWS region to use for Transcribe service ​ sample_rate int default: "16000" Audio sample rate in Hz (only 8000 Hz or 16000 Hz are supported) ​ language Language default: "Language.EN" Language for transcription ​ Default Settings Copy Ask AI { "sample_rate" : 16000 , "language" : Language. EN , "media_encoding" : "linear16" , # AWS expects raw PCM "number_of_channels" : 1 , "show_speaker_label" : False , "enable_channel_identification" : False } ​ Input The service processes InputAudioRawFrame instances containing: Raw PCM audio data 16-bit depth 8kHz or 16kHz sample rate (will convert to 16kHz if another rate is provided) Single channel (mono) ​ Output Frames The service produces two types of frames during transcription: ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Language used for transcription ​ InterimTranscriptionFrame Generated during ongoing speech, containing same fields as TranscriptionFrame but with preliminary results. ​ Methods See the STT base class methods for additional functionality. ​ Language Setting Copy Ask AI await service.set_language(Language. FR ) ​ Usage Example Copy Ask AI from pipecat.services.aws.stt import AWSTranscribeSTTService # Configure service using environment variables for credentials stt = AWSTranscribeSTTService( region = "us-west-2" , sample_rate = 16000 , language = Language. EN ) # Or provide credentials directly stt = AWSTranscribeSTTService( aws_access_key_id = "YOUR_ACCESS_KEY_ID" , api_key = "YOUR_SECRET_ACCESS_KEY" , region = "us-west-2" , sample_rate = 16000 , language = Language. EN ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ Language Support AWS Transcribe STT supports the following languages: Language Code Description Service Codes Language.EN English (US) en-US Language.ES Spanish es-US Language.FR French fr-FR Language.DE German de-DE Language.IT Italian it-IT Language.PT Portuguese (Brazil) pt-BR Language.JA Japanese ja-JP Language.KO Korean ko-KR Language.ZH Chinese (Mandarin) zh-CN AWS Transcribe supports additional languages and regional variants. See the AWS Transcribe documentation for a complete list. ​ Frame Flow ​ Metrics Support The service supports the following metrics: Time to First Byte (TTFB) Processing duration ​ Notes Requires valid AWS credentials with access to Amazon Transcribe Supports real-time transcription with interim results Handles WebSocket connection management and reconnection Only supports mono audio (single channel) Automatically handles audio format conversion to PCM Manages connection lifecycle (start, stop, cancel) AssemblyAI Azure On this page Overview Installation Configuration Constructor Parameters Default Settings Input Output Frames TranscriptionFrame InterimTranscriptionFrame Methods Language Setting Usage Example Language Support Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_aws_aed56446.txt b/stt_aws_aed56446.txt
new file mode 100644
index 0000000000000000000000000000000000000000..d41457355123ba6e3c2df6899481d852f6b57812
--- /dev/null
+++ b/stt_aws_aed56446.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/aws#param-aws-access-key-id
+Title: AWS Transcribe - Pipecat
+==================================================
+
+AWS Transcribe - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text AWS Transcribe Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview AWSTranscribeSTTService provides real-time speech-to-text capabilities using Amazon Transcribe’s WebSocket API. It supports interim results, adjustable quality levels, and can handle continuous audio streams. ​ Installation To use AWSTranscribeSTTService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[aws]" You’ll also need to set up your AWS credentials as environment variables: AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_SESSION_TOKEN (if using temporary credentials) AWS_REGION (defaults to “us-east-1”) You can obtain AWS credentials by setting up an IAM user with access to Amazon Transcribe in your AWS account. ​ Configuration ​ Constructor Parameters ​ api_key str Your AWS secret access key (can also use environment variable) ​ aws_access_key_id str Your AWS access key ID (can also use environment variable) ​ aws_session_token str Your AWS session token for temporary credentials (can also use environment variable) ​ region str default: "us-east-1" AWS region to use for Transcribe service ​ sample_rate int default: "16000" Audio sample rate in Hz (only 8000 Hz or 16000 Hz are supported) ​ language Language default: "Language.EN" Language for transcription ​ Default Settings Copy Ask AI { "sample_rate" : 16000 , "language" : Language. EN , "media_encoding" : "linear16" , # AWS expects raw PCM "number_of_channels" : 1 , "show_speaker_label" : False , "enable_channel_identification" : False } ​ Input The service processes InputAudioRawFrame instances containing: Raw PCM audio data 16-bit depth 8kHz or 16kHz sample rate (will convert to 16kHz if another rate is provided) Single channel (mono) ​ Output Frames The service produces two types of frames during transcription: ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Language used for transcription ​ InterimTranscriptionFrame Generated during ongoing speech, containing same fields as TranscriptionFrame but with preliminary results. ​ Methods See the STT base class methods for additional functionality. ​ Language Setting Copy Ask AI await service.set_language(Language. FR ) ​ Usage Example Copy Ask AI from pipecat.services.aws.stt import AWSTranscribeSTTService # Configure service using environment variables for credentials stt = AWSTranscribeSTTService( region = "us-west-2" , sample_rate = 16000 , language = Language. EN ) # Or provide credentials directly stt = AWSTranscribeSTTService( aws_access_key_id = "YOUR_ACCESS_KEY_ID" , api_key = "YOUR_SECRET_ACCESS_KEY" , region = "us-west-2" , sample_rate = 16000 , language = Language. EN ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ Language Support AWS Transcribe STT supports the following languages: Language Code Description Service Codes Language.EN English (US) en-US Language.ES Spanish es-US Language.FR French fr-FR Language.DE German de-DE Language.IT Italian it-IT Language.PT Portuguese (Brazil) pt-BR Language.JA Japanese ja-JP Language.KO Korean ko-KR Language.ZH Chinese (Mandarin) zh-CN AWS Transcribe supports additional languages and regional variants. See the AWS Transcribe documentation for a complete list. ​ Frame Flow ​ Metrics Support The service supports the following metrics: Time to First Byte (TTFB) Processing duration ​ Notes Requires valid AWS credentials with access to Amazon Transcribe Supports real-time transcription with interim results Handles WebSocket connection management and reconnection Only supports mono audio (single channel) Automatically handles audio format conversion to PCM Manages connection lifecycle (start, stop, cancel) AssemblyAI Azure On this page Overview Installation Configuration Constructor Parameters Default Settings Input Output Frames TranscriptionFrame InterimTranscriptionFrame Methods Language Setting Usage Example Language Support Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_aws_d008e289.txt b/stt_aws_d008e289.txt
new file mode 100644
index 0000000000000000000000000000000000000000..8b62eff44e29bf35aa05c1eae4df0b3430364eb1
--- /dev/null
+++ b/stt_aws_d008e289.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/aws#param-user-id
+Title: AWS Transcribe - Pipecat
+==================================================
+
+AWS Transcribe - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text AWS Transcribe Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview AWSTranscribeSTTService provides real-time speech-to-text capabilities using Amazon Transcribe’s WebSocket API. It supports interim results, adjustable quality levels, and can handle continuous audio streams. ​ Installation To use AWSTranscribeSTTService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[aws]" You’ll also need to set up your AWS credentials as environment variables: AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_SESSION_TOKEN (if using temporary credentials) AWS_REGION (defaults to “us-east-1”) You can obtain AWS credentials by setting up an IAM user with access to Amazon Transcribe in your AWS account. ​ Configuration ​ Constructor Parameters ​ api_key str Your AWS secret access key (can also use environment variable) ​ aws_access_key_id str Your AWS access key ID (can also use environment variable) ​ aws_session_token str Your AWS session token for temporary credentials (can also use environment variable) ​ region str default: "us-east-1" AWS region to use for Transcribe service ​ sample_rate int default: "16000" Audio sample rate in Hz (only 8000 Hz or 16000 Hz are supported) ​ language Language default: "Language.EN" Language for transcription ​ Default Settings Copy Ask AI { "sample_rate" : 16000 , "language" : Language. EN , "media_encoding" : "linear16" , # AWS expects raw PCM "number_of_channels" : 1 , "show_speaker_label" : False , "enable_channel_identification" : False } ​ Input The service processes InputAudioRawFrame instances containing: Raw PCM audio data 16-bit depth 8kHz or 16kHz sample rate (will convert to 16kHz if another rate is provided) Single channel (mono) ​ Output Frames The service produces two types of frames during transcription: ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Language used for transcription ​ InterimTranscriptionFrame Generated during ongoing speech, containing same fields as TranscriptionFrame but with preliminary results. ​ Methods See the STT base class methods for additional functionality. ​ Language Setting Copy Ask AI await service.set_language(Language. FR ) ​ Usage Example Copy Ask AI from pipecat.services.aws.stt import AWSTranscribeSTTService # Configure service using environment variables for credentials stt = AWSTranscribeSTTService( region = "us-west-2" , sample_rate = 16000 , language = Language. EN ) # Or provide credentials directly stt = AWSTranscribeSTTService( aws_access_key_id = "YOUR_ACCESS_KEY_ID" , api_key = "YOUR_SECRET_ACCESS_KEY" , region = "us-west-2" , sample_rate = 16000 , language = Language. EN ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ Language Support AWS Transcribe STT supports the following languages: Language Code Description Service Codes Language.EN English (US) en-US Language.ES Spanish es-US Language.FR French fr-FR Language.DE German de-DE Language.IT Italian it-IT Language.PT Portuguese (Brazil) pt-BR Language.JA Japanese ja-JP Language.KO Korean ko-KR Language.ZH Chinese (Mandarin) zh-CN AWS Transcribe supports additional languages and regional variants. See the AWS Transcribe documentation for a complete list. ​ Frame Flow ​ Metrics Support The service supports the following metrics: Time to First Byte (TTFB) Processing duration ​ Notes Requires valid AWS credentials with access to Amazon Transcribe Supports real-time transcription with interim results Handles WebSocket connection management and reconnection Only supports mono audio (single channel) Automatically handles audio format conversion to PCM Manages connection lifecycle (start, stop, cancel) AssemblyAI Azure On this page Overview Installation Configuration Constructor Parameters Default Settings Input Output Frames TranscriptionFrame InterimTranscriptionFrame Methods Language Setting Usage Example Language Support Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_aws_d0527973.txt b/stt_aws_d0527973.txt
new file mode 100644
index 0000000000000000000000000000000000000000..c4890651c90fad8e902fdf037a5c8c788e44eb85
--- /dev/null
+++ b/stt_aws_d0527973.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/aws#notes
+Title: AWS Transcribe - Pipecat
+==================================================
+
+AWS Transcribe - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text AWS Transcribe Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview AWSTranscribeSTTService provides real-time speech-to-text capabilities using Amazon Transcribe’s WebSocket API. It supports interim results, adjustable quality levels, and can handle continuous audio streams. ​ Installation To use AWSTranscribeSTTService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[aws]" You’ll also need to set up your AWS credentials as environment variables: AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_SESSION_TOKEN (if using temporary credentials) AWS_REGION (defaults to “us-east-1”) You can obtain AWS credentials by setting up an IAM user with access to Amazon Transcribe in your AWS account. ​ Configuration ​ Constructor Parameters ​ api_key str Your AWS secret access key (can also use environment variable) ​ aws_access_key_id str Your AWS access key ID (can also use environment variable) ​ aws_session_token str Your AWS session token for temporary credentials (can also use environment variable) ​ region str default: "us-east-1" AWS region to use for Transcribe service ​ sample_rate int default: "16000" Audio sample rate in Hz (only 8000 Hz or 16000 Hz are supported) ​ language Language default: "Language.EN" Language for transcription ​ Default Settings Copy Ask AI { "sample_rate" : 16000 , "language" : Language. EN , "media_encoding" : "linear16" , # AWS expects raw PCM "number_of_channels" : 1 , "show_speaker_label" : False , "enable_channel_identification" : False } ​ Input The service processes InputAudioRawFrame instances containing: Raw PCM audio data 16-bit depth 8kHz or 16kHz sample rate (will convert to 16kHz if another rate is provided) Single channel (mono) ​ Output Frames The service produces two types of frames during transcription: ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Language used for transcription ​ InterimTranscriptionFrame Generated during ongoing speech, containing same fields as TranscriptionFrame but with preliminary results. ​ Methods See the STT base class methods for additional functionality. ​ Language Setting Copy Ask AI await service.set_language(Language. FR ) ​ Usage Example Copy Ask AI from pipecat.services.aws.stt import AWSTranscribeSTTService # Configure service using environment variables for credentials stt = AWSTranscribeSTTService( region = "us-west-2" , sample_rate = 16000 , language = Language. EN ) # Or provide credentials directly stt = AWSTranscribeSTTService( aws_access_key_id = "YOUR_ACCESS_KEY_ID" , api_key = "YOUR_SECRET_ACCESS_KEY" , region = "us-west-2" , sample_rate = 16000 , language = Language. EN ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ Language Support AWS Transcribe STT supports the following languages: Language Code Description Service Codes Language.EN English (US) en-US Language.ES Spanish es-US Language.FR French fr-FR Language.DE German de-DE Language.IT Italian it-IT Language.PT Portuguese (Brazil) pt-BR Language.JA Japanese ja-JP Language.KO Korean ko-KR Language.ZH Chinese (Mandarin) zh-CN AWS Transcribe supports additional languages and regional variants. See the AWS Transcribe documentation for a complete list. ​ Frame Flow ​ Metrics Support The service supports the following metrics: Time to First Byte (TTFB) Processing duration ​ Notes Requires valid AWS credentials with access to Amazon Transcribe Supports real-time transcription with interim results Handles WebSocket connection management and reconnection Only supports mono audio (single channel) Automatically handles audio format conversion to PCM Manages connection lifecycle (start, stop, cancel) AssemblyAI Azure On this page Overview Installation Configuration Constructor Parameters Default Settings Input Output Frames TranscriptionFrame InterimTranscriptionFrame Methods Language Setting Usage Example Language Support Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_azure_11bdfa1b.txt b/stt_azure_11bdfa1b.txt
new file mode 100644
index 0000000000000000000000000000000000000000..a6946bc62e8414c2bae82e4eb80bea56f23c9701
--- /dev/null
+++ b/stt_azure_11bdfa1b.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/azure#output-frames
+Title: Azure - Pipecat
+==================================================
+
+Azure - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text Azure Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview AzureSTTService provides real-time speech recognition using Azure’s Cognitive Services Speech SDK. It supports continuous recognition and multiple languages. ​ Installation To use AzureSTTService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[azure]" You’ll also need to set up the following environment variables: AZURE_API_KEY AZURE_REGION ​ Configuration ​ Constructor Parameters ​ api_key str required Azure Speech Service API key ​ region str required Azure region identifier ​ language Language default: "Language.EN_US" Recognition language ​ sample_rate int default: "None" Input audio sample rate in Hz ​ channels int default: "1" Number of audio channels ​ Input The service processes audio data through a PushAudioInputStream : PCM format Configurable sample rate Mono or stereo input ​ Output Frames ​ TranscriptionFrame Frame Contains: - Recognized text - Empty user ID - ISO 8601 formatted timestamp ​ Methods See the STT base class methods for additional functionality. ​ Language Setting Copy Ask AI await service.set_language(Language. FR ) ​ Language Support Azure STT supports the following languages and regional variants: Language Code Description Service Codes Language.ZH Chinese zh-CN Language.EN_US English (US) en-US Language.EN_IN English (India) en-IN Language.FR French fr-FR Language.DE German de-DE Language.HI Hindi hi-IN Language.IT Italian it-IT Language.JA Japanese ja-JP Language.KO Korean ko-KR Language.PT_BR Portuguese (Brazil) pt-BR Language.ES Spanish es-ES , es-MX ​ Usage Example Copy Ask AI # Configure service stt = AzureSTTService( api_key = "your-api-key" , region = "eastus" , language = Language. EN_US , sample_rate = 16000 , channels = 1 ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ Frame Flow ​ Notes Supports continuous recognition Handles automatic reconnection Provides real-time transcription Thread-safe processing Automatic resource cleanup AWS Transcribe Cartesia On this page Overview Installation Configuration Constructor Parameters Input Output Frames Methods Language Setting Language Support Usage Example Frame Flow Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_azure_75bf264d.txt b/stt_azure_75bf264d.txt
new file mode 100644
index 0000000000000000000000000000000000000000..7cc3cd39347e98e243ae3dccd965c14e8bb4d71e
--- /dev/null
+++ b/stt_azure_75bf264d.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/azure#configuration
+Title: Azure - Pipecat
+==================================================
+
+Azure - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text Azure Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview AzureSTTService provides real-time speech recognition using Azure’s Cognitive Services Speech SDK. It supports continuous recognition and multiple languages. ​ Installation To use AzureSTTService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[azure]" You’ll also need to set up the following environment variables: AZURE_API_KEY AZURE_REGION ​ Configuration ​ Constructor Parameters ​ api_key str required Azure Speech Service API key ​ region str required Azure region identifier ​ language Language default: "Language.EN_US" Recognition language ​ sample_rate int default: "None" Input audio sample rate in Hz ​ channels int default: "1" Number of audio channels ​ Input The service processes audio data through a PushAudioInputStream : PCM format Configurable sample rate Mono or stereo input ​ Output Frames ​ TranscriptionFrame Frame Contains: - Recognized text - Empty user ID - ISO 8601 formatted timestamp ​ Methods See the STT base class methods for additional functionality. ​ Language Setting Copy Ask AI await service.set_language(Language. FR ) ​ Language Support Azure STT supports the following languages and regional variants: Language Code Description Service Codes Language.ZH Chinese zh-CN Language.EN_US English (US) en-US Language.EN_IN English (India) en-IN Language.FR French fr-FR Language.DE German de-DE Language.HI Hindi hi-IN Language.IT Italian it-IT Language.JA Japanese ja-JP Language.KO Korean ko-KR Language.PT_BR Portuguese (Brazil) pt-BR Language.ES Spanish es-ES , es-MX ​ Usage Example Copy Ask AI # Configure service stt = AzureSTTService( api_key = "your-api-key" , region = "eastus" , language = Language. EN_US , sample_rate = 16000 , channels = 1 ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ Frame Flow ​ Notes Supports continuous recognition Handles automatic reconnection Provides real-time transcription Thread-safe processing Automatic resource cleanup AWS Transcribe Cartesia On this page Overview Installation Configuration Constructor Parameters Input Output Frames Methods Language Setting Language Support Usage Example Frame Flow Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_azure_7675c9d4.txt b/stt_azure_7675c9d4.txt
new file mode 100644
index 0000000000000000000000000000000000000000..9ab7d40b4b75bc07138c03a14d25b8356c4e882d
--- /dev/null
+++ b/stt_azure_7675c9d4.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/azure#constructor-parameters
+Title: Azure - Pipecat
+==================================================
+
+Azure - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text Azure Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview AzureSTTService provides real-time speech recognition using Azure’s Cognitive Services Speech SDK. It supports continuous recognition and multiple languages. ​ Installation To use AzureSTTService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[azure]" You’ll also need to set up the following environment variables: AZURE_API_KEY AZURE_REGION ​ Configuration ​ Constructor Parameters ​ api_key str required Azure Speech Service API key ​ region str required Azure region identifier ​ language Language default: "Language.EN_US" Recognition language ​ sample_rate int default: "None" Input audio sample rate in Hz ​ channels int default: "1" Number of audio channels ​ Input The service processes audio data through a PushAudioInputStream : PCM format Configurable sample rate Mono or stereo input ​ Output Frames ​ TranscriptionFrame Frame Contains: - Recognized text - Empty user ID - ISO 8601 formatted timestamp ​ Methods See the STT base class methods for additional functionality. ​ Language Setting Copy Ask AI await service.set_language(Language. FR ) ​ Language Support Azure STT supports the following languages and regional variants: Language Code Description Service Codes Language.ZH Chinese zh-CN Language.EN_US English (US) en-US Language.EN_IN English (India) en-IN Language.FR French fr-FR Language.DE German de-DE Language.HI Hindi hi-IN Language.IT Italian it-IT Language.JA Japanese ja-JP Language.KO Korean ko-KR Language.PT_BR Portuguese (Brazil) pt-BR Language.ES Spanish es-ES , es-MX ​ Usage Example Copy Ask AI # Configure service stt = AzureSTTService( api_key = "your-api-key" , region = "eastus" , language = Language. EN_US , sample_rate = 16000 , channels = 1 ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ Frame Flow ​ Notes Supports continuous recognition Handles automatic reconnection Provides real-time transcription Thread-safe processing Automatic resource cleanup AWS Transcribe Cartesia On this page Overview Installation Configuration Constructor Parameters Input Output Frames Methods Language Setting Language Support Usage Example Frame Flow Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_azure_a5afc866.txt b/stt_azure_a5afc866.txt
new file mode 100644
index 0000000000000000000000000000000000000000..612fc69b5a8765780bf4349aa702fe46be12e650
--- /dev/null
+++ b/stt_azure_a5afc866.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/azure#param-transcription-frame
+Title: Azure - Pipecat
+==================================================
+
+Azure - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text Azure Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview AzureSTTService provides real-time speech recognition using Azure’s Cognitive Services Speech SDK. It supports continuous recognition and multiple languages. ​ Installation To use AzureSTTService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[azure]" You’ll also need to set up the following environment variables: AZURE_API_KEY AZURE_REGION ​ Configuration ​ Constructor Parameters ​ api_key str required Azure Speech Service API key ​ region str required Azure region identifier ​ language Language default: "Language.EN_US" Recognition language ​ sample_rate int default: "None" Input audio sample rate in Hz ​ channels int default: "1" Number of audio channels ​ Input The service processes audio data through a PushAudioInputStream : PCM format Configurable sample rate Mono or stereo input ​ Output Frames ​ TranscriptionFrame Frame Contains: - Recognized text - Empty user ID - ISO 8601 formatted timestamp ​ Methods See the STT base class methods for additional functionality. ​ Language Setting Copy Ask AI await service.set_language(Language. FR ) ​ Language Support Azure STT supports the following languages and regional variants: Language Code Description Service Codes Language.ZH Chinese zh-CN Language.EN_US English (US) en-US Language.EN_IN English (India) en-IN Language.FR French fr-FR Language.DE German de-DE Language.HI Hindi hi-IN Language.IT Italian it-IT Language.JA Japanese ja-JP Language.KO Korean ko-KR Language.PT_BR Portuguese (Brazil) pt-BR Language.ES Spanish es-ES , es-MX ​ Usage Example Copy Ask AI # Configure service stt = AzureSTTService( api_key = "your-api-key" , region = "eastus" , language = Language. EN_US , sample_rate = 16000 , channels = 1 ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ Frame Flow ​ Notes Supports continuous recognition Handles automatic reconnection Provides real-time transcription Thread-safe processing Automatic resource cleanup AWS Transcribe Cartesia On this page Overview Installation Configuration Constructor Parameters Input Output Frames Methods Language Setting Language Support Usage Example Frame Flow Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_azure_aac67627.txt b/stt_azure_aac67627.txt
new file mode 100644
index 0000000000000000000000000000000000000000..a41c96254f674169d6cd7ed827324877337ed90b
--- /dev/null
+++ b/stt_azure_aac67627.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/azure#param-channels
+Title: Azure - Pipecat
+==================================================
+
+Azure - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text Azure Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview AzureSTTService provides real-time speech recognition using Azure’s Cognitive Services Speech SDK. It supports continuous recognition and multiple languages. ​ Installation To use AzureSTTService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[azure]" You’ll also need to set up the following environment variables: AZURE_API_KEY AZURE_REGION ​ Configuration ​ Constructor Parameters ​ api_key str required Azure Speech Service API key ​ region str required Azure region identifier ​ language Language default: "Language.EN_US" Recognition language ​ sample_rate int default: "None" Input audio sample rate in Hz ​ channels int default: "1" Number of audio channels ​ Input The service processes audio data through a PushAudioInputStream : PCM format Configurable sample rate Mono or stereo input ​ Output Frames ​ TranscriptionFrame Frame Contains: - Recognized text - Empty user ID - ISO 8601 formatted timestamp ​ Methods See the STT base class methods for additional functionality. ​ Language Setting Copy Ask AI await service.set_language(Language. FR ) ​ Language Support Azure STT supports the following languages and regional variants: Language Code Description Service Codes Language.ZH Chinese zh-CN Language.EN_US English (US) en-US Language.EN_IN English (India) en-IN Language.FR French fr-FR Language.DE German de-DE Language.HI Hindi hi-IN Language.IT Italian it-IT Language.JA Japanese ja-JP Language.KO Korean ko-KR Language.PT_BR Portuguese (Brazil) pt-BR Language.ES Spanish es-ES , es-MX ​ Usage Example Copy Ask AI # Configure service stt = AzureSTTService( api_key = "your-api-key" , region = "eastus" , language = Language. EN_US , sample_rate = 16000 , channels = 1 ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ Frame Flow ​ Notes Supports continuous recognition Handles automatic reconnection Provides real-time transcription Thread-safe processing Automatic resource cleanup AWS Transcribe Cartesia On this page Overview Installation Configuration Constructor Parameters Input Output Frames Methods Language Setting Language Support Usage Example Frame Flow Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_cartesia_13e95a36.txt b/stt_cartesia_13e95a36.txt
new file mode 100644
index 0000000000000000000000000000000000000000..87bd6ebdfa5d3b9014b6944ba2ce6c0c1aee2657
--- /dev/null
+++ b/stt_cartesia_13e95a36.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/cartesia#param-sample-rate
+Title: Cartesia - Pipecat
+==================================================
+
+Cartesia - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text Cartesia Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview CartesiaSTTService provides real-time speech-to-text capabilities using Cartesia’s WebSocket API. It supports streaming transcription with both interim and final results using the ink-whisper model. ​ Installation To use CartesiaSTTService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[cartesia]" You’ll also need to set up your Cartesia API key as an environment variable: CARTESIA_API_KEY . You can obtain a Cartesia API key by signing up at Cartesia . ​ Configuration ​ Constructor Parameters ​ api_key str required Your Cartesia API key ​ base_url str default: "api.cartesia.ai" Custom Cartesia API endpoint URL ​ sample_rate int default: "16000" Audio sample rate in Hz ​ live_options CartesiaLiveOptions Custom transcription options ​ CartesiaLiveOptions ​ model str default: "ink-whisper" The Cartesia transcription model to use ​ language str default: "en" Language code for transcription ​ encoding str default: "pcm_s16le" Audio encoding format ​ sample_rate int default: "16000" Audio sample rate in Hz ​ Default Options Copy Ask AI CartesiaLiveOptions( model = "ink-whisper" , language = "en" , encoding = "pcm_s16le" , sample_rate = 16000 ) ​ Input The service processes raw audio data with the following requirements: PCM audio format ( pcm_s16le ) 16-bit depth 16kHz sample rate (default) Single channel (mono) ​ Output Frames The service produces two types of frames during transcription: ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Final transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Detected or configured language ​ InterimTranscriptionFrame Generated during ongoing speech, containing the same fields as TranscriptionFrame but with preliminary results. ​ Methods See the STT base class methods for additional functionality. ​ Language Setting The service supports language configuration through the CartesiaLiveOptions : Copy Ask AI live_options = CartesiaLiveOptions( language = "es" ) ​ Model Selection Copy Ask AI live_options = CartesiaLiveOptions( model = "ink-whisper" ) ​ Usage Example Copy Ask AI from pipecat.services.cartesia.stt import CartesiaSTTService, CartesiaLiveOptions from pipecat.transcriptions.language import Language # Basic configuration stt = CartesiaSTTService( api_key = os.getenv( "CARTESIA_API_KEY" ) ) # Advanced configuration live_options = CartesiaLiveOptions( model = "ink-whisper" , language = Language. ES .value, sample_rate = 16000 , encoding = "pcm_s16le" ) stt = CartesiaSTTService( api_key = os.getenv( "CARTESIA_API_KEY" ), live_options = live_options ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ Frame Flow ​ Connection Management The service automatically manages WebSocket connections: Auto-reconnect : Reconnects automatically when the connection is closed due to timeout Finalization : Sends a “finalize” command when user stops speaking to flush the transcription session Error handling : Gracefully handles connection errors and WebSocket exceptions ​ Metrics Support The service supports comprehensive metrics collection: Time to First Byte (TTFB) Processing duration Speech detection events Connection status ​ Notes Requires valid Cartesia API key Supports real-time streaming transcription Handles automatic WebSocket connection management Includes comprehensive error handling Manages connection lifecycle automatically Azure Deepgram On this page Overview Installation Configuration Constructor Parameters CartesiaLiveOptions Default Options Input Output Frames TranscriptionFrame InterimTranscriptionFrame Methods Language Setting Model Selection Usage Example Frame Flow Connection Management Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_cartesia_16bbd65c.txt b/stt_cartesia_16bbd65c.txt
new file mode 100644
index 0000000000000000000000000000000000000000..fbcd1f91eff1045414345d500d21baec07adce6b
--- /dev/null
+++ b/stt_cartesia_16bbd65c.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/cartesia#cartesialiveoptions
+Title: Cartesia - Pipecat
+==================================================
+
+Cartesia - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text Cartesia Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview CartesiaSTTService provides real-time speech-to-text capabilities using Cartesia’s WebSocket API. It supports streaming transcription with both interim and final results using the ink-whisper model. ​ Installation To use CartesiaSTTService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[cartesia]" You’ll also need to set up your Cartesia API key as an environment variable: CARTESIA_API_KEY . You can obtain a Cartesia API key by signing up at Cartesia . ​ Configuration ​ Constructor Parameters ​ api_key str required Your Cartesia API key ​ base_url str default: "api.cartesia.ai" Custom Cartesia API endpoint URL ​ sample_rate int default: "16000" Audio sample rate in Hz ​ live_options CartesiaLiveOptions Custom transcription options ​ CartesiaLiveOptions ​ model str default: "ink-whisper" The Cartesia transcription model to use ​ language str default: "en" Language code for transcription ​ encoding str default: "pcm_s16le" Audio encoding format ​ sample_rate int default: "16000" Audio sample rate in Hz ​ Default Options Copy Ask AI CartesiaLiveOptions( model = "ink-whisper" , language = "en" , encoding = "pcm_s16le" , sample_rate = 16000 ) ​ Input The service processes raw audio data with the following requirements: PCM audio format ( pcm_s16le ) 16-bit depth 16kHz sample rate (default) Single channel (mono) ​ Output Frames The service produces two types of frames during transcription: ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Final transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Detected or configured language ​ InterimTranscriptionFrame Generated during ongoing speech, containing the same fields as TranscriptionFrame but with preliminary results. ​ Methods See the STT base class methods for additional functionality. ​ Language Setting The service supports language configuration through the CartesiaLiveOptions : Copy Ask AI live_options = CartesiaLiveOptions( language = "es" ) ​ Model Selection Copy Ask AI live_options = CartesiaLiveOptions( model = "ink-whisper" ) ​ Usage Example Copy Ask AI from pipecat.services.cartesia.stt import CartesiaSTTService, CartesiaLiveOptions from pipecat.transcriptions.language import Language # Basic configuration stt = CartesiaSTTService( api_key = os.getenv( "CARTESIA_API_KEY" ) ) # Advanced configuration live_options = CartesiaLiveOptions( model = "ink-whisper" , language = Language. ES .value, sample_rate = 16000 , encoding = "pcm_s16le" ) stt = CartesiaSTTService( api_key = os.getenv( "CARTESIA_API_KEY" ), live_options = live_options ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ Frame Flow ​ Connection Management The service automatically manages WebSocket connections: Auto-reconnect : Reconnects automatically when the connection is closed due to timeout Finalization : Sends a “finalize” command when user stops speaking to flush the transcription session Error handling : Gracefully handles connection errors and WebSocket exceptions ​ Metrics Support The service supports comprehensive metrics collection: Time to First Byte (TTFB) Processing duration Speech detection events Connection status ​ Notes Requires valid Cartesia API key Supports real-time streaming transcription Handles automatic WebSocket connection management Includes comprehensive error handling Manages connection lifecycle automatically Azure Deepgram On this page Overview Installation Configuration Constructor Parameters CartesiaLiveOptions Default Options Input Output Frames TranscriptionFrame InterimTranscriptionFrame Methods Language Setting Model Selection Usage Example Frame Flow Connection Management Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_cartesia_2e6f89c9.txt b/stt_cartesia_2e6f89c9.txt
new file mode 100644
index 0000000000000000000000000000000000000000..fbd7398e4947e10379a537de32f02de6d7195b64
--- /dev/null
+++ b/stt_cartesia_2e6f89c9.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/cartesia#constructor-parameters
+Title: Cartesia - Pipecat
+==================================================
+
+Cartesia - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text Cartesia Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview CartesiaSTTService provides real-time speech-to-text capabilities using Cartesia’s WebSocket API. It supports streaming transcription with both interim and final results using the ink-whisper model. ​ Installation To use CartesiaSTTService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[cartesia]" You’ll also need to set up your Cartesia API key as an environment variable: CARTESIA_API_KEY . You can obtain a Cartesia API key by signing up at Cartesia . ​ Configuration ​ Constructor Parameters ​ api_key str required Your Cartesia API key ​ base_url str default: "api.cartesia.ai" Custom Cartesia API endpoint URL ​ sample_rate int default: "16000" Audio sample rate in Hz ​ live_options CartesiaLiveOptions Custom transcription options ​ CartesiaLiveOptions ​ model str default: "ink-whisper" The Cartesia transcription model to use ​ language str default: "en" Language code for transcription ​ encoding str default: "pcm_s16le" Audio encoding format ​ sample_rate int default: "16000" Audio sample rate in Hz ​ Default Options Copy Ask AI CartesiaLiveOptions( model = "ink-whisper" , language = "en" , encoding = "pcm_s16le" , sample_rate = 16000 ) ​ Input The service processes raw audio data with the following requirements: PCM audio format ( pcm_s16le ) 16-bit depth 16kHz sample rate (default) Single channel (mono) ​ Output Frames The service produces two types of frames during transcription: ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Final transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Detected or configured language ​ InterimTranscriptionFrame Generated during ongoing speech, containing the same fields as TranscriptionFrame but with preliminary results. ​ Methods See the STT base class methods for additional functionality. ​ Language Setting The service supports language configuration through the CartesiaLiveOptions : Copy Ask AI live_options = CartesiaLiveOptions( language = "es" ) ​ Model Selection Copy Ask AI live_options = CartesiaLiveOptions( model = "ink-whisper" ) ​ Usage Example Copy Ask AI from pipecat.services.cartesia.stt import CartesiaSTTService, CartesiaLiveOptions from pipecat.transcriptions.language import Language # Basic configuration stt = CartesiaSTTService( api_key = os.getenv( "CARTESIA_API_KEY" ) ) # Advanced configuration live_options = CartesiaLiveOptions( model = "ink-whisper" , language = Language. ES .value, sample_rate = 16000 , encoding = "pcm_s16le" ) stt = CartesiaSTTService( api_key = os.getenv( "CARTESIA_API_KEY" ), live_options = live_options ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ Frame Flow ​ Connection Management The service automatically manages WebSocket connections: Auto-reconnect : Reconnects automatically when the connection is closed due to timeout Finalization : Sends a “finalize” command when user stops speaking to flush the transcription session Error handling : Gracefully handles connection errors and WebSocket exceptions ​ Metrics Support The service supports comprehensive metrics collection: Time to First Byte (TTFB) Processing duration Speech detection events Connection status ​ Notes Requires valid Cartesia API key Supports real-time streaming transcription Handles automatic WebSocket connection management Includes comprehensive error handling Manages connection lifecycle automatically Azure Deepgram On this page Overview Installation Configuration Constructor Parameters CartesiaLiveOptions Default Options Input Output Frames TranscriptionFrame InterimTranscriptionFrame Methods Language Setting Model Selection Usage Example Frame Flow Connection Management Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_cartesia_2e769bd5.txt b/stt_cartesia_2e769bd5.txt
new file mode 100644
index 0000000000000000000000000000000000000000..331ade3e6efb8f6ce7daf4590c138da319e87ca4
--- /dev/null
+++ b/stt_cartesia_2e769bd5.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/cartesia#param-live-options
+Title: Cartesia - Pipecat
+==================================================
+
+Cartesia - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text Cartesia Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview CartesiaSTTService provides real-time speech-to-text capabilities using Cartesia’s WebSocket API. It supports streaming transcription with both interim and final results using the ink-whisper model. ​ Installation To use CartesiaSTTService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[cartesia]" You’ll also need to set up your Cartesia API key as an environment variable: CARTESIA_API_KEY . You can obtain a Cartesia API key by signing up at Cartesia . ​ Configuration ​ Constructor Parameters ​ api_key str required Your Cartesia API key ​ base_url str default: "api.cartesia.ai" Custom Cartesia API endpoint URL ​ sample_rate int default: "16000" Audio sample rate in Hz ​ live_options CartesiaLiveOptions Custom transcription options ​ CartesiaLiveOptions ​ model str default: "ink-whisper" The Cartesia transcription model to use ​ language str default: "en" Language code for transcription ​ encoding str default: "pcm_s16le" Audio encoding format ​ sample_rate int default: "16000" Audio sample rate in Hz ​ Default Options Copy Ask AI CartesiaLiveOptions( model = "ink-whisper" , language = "en" , encoding = "pcm_s16le" , sample_rate = 16000 ) ​ Input The service processes raw audio data with the following requirements: PCM audio format ( pcm_s16le ) 16-bit depth 16kHz sample rate (default) Single channel (mono) ​ Output Frames The service produces two types of frames during transcription: ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Final transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Detected or configured language ​ InterimTranscriptionFrame Generated during ongoing speech, containing the same fields as TranscriptionFrame but with preliminary results. ​ Methods See the STT base class methods for additional functionality. ​ Language Setting The service supports language configuration through the CartesiaLiveOptions : Copy Ask AI live_options = CartesiaLiveOptions( language = "es" ) ​ Model Selection Copy Ask AI live_options = CartesiaLiveOptions( model = "ink-whisper" ) ​ Usage Example Copy Ask AI from pipecat.services.cartesia.stt import CartesiaSTTService, CartesiaLiveOptions from pipecat.transcriptions.language import Language # Basic configuration stt = CartesiaSTTService( api_key = os.getenv( "CARTESIA_API_KEY" ) ) # Advanced configuration live_options = CartesiaLiveOptions( model = "ink-whisper" , language = Language. ES .value, sample_rate = 16000 , encoding = "pcm_s16le" ) stt = CartesiaSTTService( api_key = os.getenv( "CARTESIA_API_KEY" ), live_options = live_options ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ Frame Flow ​ Connection Management The service automatically manages WebSocket connections: Auto-reconnect : Reconnects automatically when the connection is closed due to timeout Finalization : Sends a “finalize” command when user stops speaking to flush the transcription session Error handling : Gracefully handles connection errors and WebSocket exceptions ​ Metrics Support The service supports comprehensive metrics collection: Time to First Byte (TTFB) Processing duration Speech detection events Connection status ​ Notes Requires valid Cartesia API key Supports real-time streaming transcription Handles automatic WebSocket connection management Includes comprehensive error handling Manages connection lifecycle automatically Azure Deepgram On this page Overview Installation Configuration Constructor Parameters CartesiaLiveOptions Default Options Input Output Frames TranscriptionFrame InterimTranscriptionFrame Methods Language Setting Model Selection Usage Example Frame Flow Connection Management Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_cartesia_3b721e51.txt b/stt_cartesia_3b721e51.txt
new file mode 100644
index 0000000000000000000000000000000000000000..81d16755149c27be7f5b3045c27038163096eeb8
--- /dev/null
+++ b/stt_cartesia_3b721e51.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/cartesia#configuration
+Title: Cartesia - Pipecat
+==================================================
+
+Cartesia - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text Cartesia Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview CartesiaSTTService provides real-time speech-to-text capabilities using Cartesia’s WebSocket API. It supports streaming transcription with both interim and final results using the ink-whisper model. ​ Installation To use CartesiaSTTService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[cartesia]" You’ll also need to set up your Cartesia API key as an environment variable: CARTESIA_API_KEY . You can obtain a Cartesia API key by signing up at Cartesia . ​ Configuration ​ Constructor Parameters ​ api_key str required Your Cartesia API key ​ base_url str default: "api.cartesia.ai" Custom Cartesia API endpoint URL ​ sample_rate int default: "16000" Audio sample rate in Hz ​ live_options CartesiaLiveOptions Custom transcription options ​ CartesiaLiveOptions ​ model str default: "ink-whisper" The Cartesia transcription model to use ​ language str default: "en" Language code for transcription ​ encoding str default: "pcm_s16le" Audio encoding format ​ sample_rate int default: "16000" Audio sample rate in Hz ​ Default Options Copy Ask AI CartesiaLiveOptions( model = "ink-whisper" , language = "en" , encoding = "pcm_s16le" , sample_rate = 16000 ) ​ Input The service processes raw audio data with the following requirements: PCM audio format ( pcm_s16le ) 16-bit depth 16kHz sample rate (default) Single channel (mono) ​ Output Frames The service produces two types of frames during transcription: ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Final transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Detected or configured language ​ InterimTranscriptionFrame Generated during ongoing speech, containing the same fields as TranscriptionFrame but with preliminary results. ​ Methods See the STT base class methods for additional functionality. ​ Language Setting The service supports language configuration through the CartesiaLiveOptions : Copy Ask AI live_options = CartesiaLiveOptions( language = "es" ) ​ Model Selection Copy Ask AI live_options = CartesiaLiveOptions( model = "ink-whisper" ) ​ Usage Example Copy Ask AI from pipecat.services.cartesia.stt import CartesiaSTTService, CartesiaLiveOptions from pipecat.transcriptions.language import Language # Basic configuration stt = CartesiaSTTService( api_key = os.getenv( "CARTESIA_API_KEY" ) ) # Advanced configuration live_options = CartesiaLiveOptions( model = "ink-whisper" , language = Language. ES .value, sample_rate = 16000 , encoding = "pcm_s16le" ) stt = CartesiaSTTService( api_key = os.getenv( "CARTESIA_API_KEY" ), live_options = live_options ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ Frame Flow ​ Connection Management The service automatically manages WebSocket connections: Auto-reconnect : Reconnects automatically when the connection is closed due to timeout Finalization : Sends a “finalize” command when user stops speaking to flush the transcription session Error handling : Gracefully handles connection errors and WebSocket exceptions ​ Metrics Support The service supports comprehensive metrics collection: Time to First Byte (TTFB) Processing duration Speech detection events Connection status ​ Notes Requires valid Cartesia API key Supports real-time streaming transcription Handles automatic WebSocket connection management Includes comprehensive error handling Manages connection lifecycle automatically Azure Deepgram On this page Overview Installation Configuration Constructor Parameters CartesiaLiveOptions Default Options Input Output Frames TranscriptionFrame InterimTranscriptionFrame Methods Language Setting Model Selection Usage Example Frame Flow Connection Management Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_cartesia_8583493a.txt b/stt_cartesia_8583493a.txt
new file mode 100644
index 0000000000000000000000000000000000000000..26bf3b97c8a1318115fee3ee2fe4dbe79242d98b
--- /dev/null
+++ b/stt_cartesia_8583493a.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/cartesia#language-setting
+Title: Cartesia - Pipecat
+==================================================
+
+Cartesia - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text Cartesia Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview CartesiaSTTService provides real-time speech-to-text capabilities using Cartesia’s WebSocket API. It supports streaming transcription with both interim and final results using the ink-whisper model. ​ Installation To use CartesiaSTTService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[cartesia]" You’ll also need to set up your Cartesia API key as an environment variable: CARTESIA_API_KEY . You can obtain a Cartesia API key by signing up at Cartesia . ​ Configuration ​ Constructor Parameters ​ api_key str required Your Cartesia API key ​ base_url str default: "api.cartesia.ai" Custom Cartesia API endpoint URL ​ sample_rate int default: "16000" Audio sample rate in Hz ​ live_options CartesiaLiveOptions Custom transcription options ​ CartesiaLiveOptions ​ model str default: "ink-whisper" The Cartesia transcription model to use ​ language str default: "en" Language code for transcription ​ encoding str default: "pcm_s16le" Audio encoding format ​ sample_rate int default: "16000" Audio sample rate in Hz ​ Default Options Copy Ask AI CartesiaLiveOptions( model = "ink-whisper" , language = "en" , encoding = "pcm_s16le" , sample_rate = 16000 ) ​ Input The service processes raw audio data with the following requirements: PCM audio format ( pcm_s16le ) 16-bit depth 16kHz sample rate (default) Single channel (mono) ​ Output Frames The service produces two types of frames during transcription: ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Final transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Detected or configured language ​ InterimTranscriptionFrame Generated during ongoing speech, containing the same fields as TranscriptionFrame but with preliminary results. ​ Methods See the STT base class methods for additional functionality. ​ Language Setting The service supports language configuration through the CartesiaLiveOptions : Copy Ask AI live_options = CartesiaLiveOptions( language = "es" ) ​ Model Selection Copy Ask AI live_options = CartesiaLiveOptions( model = "ink-whisper" ) ​ Usage Example Copy Ask AI from pipecat.services.cartesia.stt import CartesiaSTTService, CartesiaLiveOptions from pipecat.transcriptions.language import Language # Basic configuration stt = CartesiaSTTService( api_key = os.getenv( "CARTESIA_API_KEY" ) ) # Advanced configuration live_options = CartesiaLiveOptions( model = "ink-whisper" , language = Language. ES .value, sample_rate = 16000 , encoding = "pcm_s16le" ) stt = CartesiaSTTService( api_key = os.getenv( "CARTESIA_API_KEY" ), live_options = live_options ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ Frame Flow ​ Connection Management The service automatically manages WebSocket connections: Auto-reconnect : Reconnects automatically when the connection is closed due to timeout Finalization : Sends a “finalize” command when user stops speaking to flush the transcription session Error handling : Gracefully handles connection errors and WebSocket exceptions ​ Metrics Support The service supports comprehensive metrics collection: Time to First Byte (TTFB) Processing duration Speech detection events Connection status ​ Notes Requires valid Cartesia API key Supports real-time streaming transcription Handles automatic WebSocket connection management Includes comprehensive error handling Manages connection lifecycle automatically Azure Deepgram On this page Overview Installation Configuration Constructor Parameters CartesiaLiveOptions Default Options Input Output Frames TranscriptionFrame InterimTranscriptionFrame Methods Language Setting Model Selection Usage Example Frame Flow Connection Management Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_cartesia_8716cadd.txt b/stt_cartesia_8716cadd.txt
new file mode 100644
index 0000000000000000000000000000000000000000..44b2564bf802f2944006024bdf761c7f36682790
--- /dev/null
+++ b/stt_cartesia_8716cadd.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/cartesia#model-selection
+Title: Cartesia - Pipecat
+==================================================
+
+Cartesia - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text Cartesia Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview CartesiaSTTService provides real-time speech-to-text capabilities using Cartesia’s WebSocket API. It supports streaming transcription with both interim and final results using the ink-whisper model. ​ Installation To use CartesiaSTTService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[cartesia]" You’ll also need to set up your Cartesia API key as an environment variable: CARTESIA_API_KEY . You can obtain a Cartesia API key by signing up at Cartesia . ​ Configuration ​ Constructor Parameters ​ api_key str required Your Cartesia API key ​ base_url str default: "api.cartesia.ai" Custom Cartesia API endpoint URL ​ sample_rate int default: "16000" Audio sample rate in Hz ​ live_options CartesiaLiveOptions Custom transcription options ​ CartesiaLiveOptions ​ model str default: "ink-whisper" The Cartesia transcription model to use ​ language str default: "en" Language code for transcription ​ encoding str default: "pcm_s16le" Audio encoding format ​ sample_rate int default: "16000" Audio sample rate in Hz ​ Default Options Copy Ask AI CartesiaLiveOptions( model = "ink-whisper" , language = "en" , encoding = "pcm_s16le" , sample_rate = 16000 ) ​ Input The service processes raw audio data with the following requirements: PCM audio format ( pcm_s16le ) 16-bit depth 16kHz sample rate (default) Single channel (mono) ​ Output Frames The service produces two types of frames during transcription: ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Final transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Detected or configured language ​ InterimTranscriptionFrame Generated during ongoing speech, containing the same fields as TranscriptionFrame but with preliminary results. ​ Methods See the STT base class methods for additional functionality. ​ Language Setting The service supports language configuration through the CartesiaLiveOptions : Copy Ask AI live_options = CartesiaLiveOptions( language = "es" ) ​ Model Selection Copy Ask AI live_options = CartesiaLiveOptions( model = "ink-whisper" ) ​ Usage Example Copy Ask AI from pipecat.services.cartesia.stt import CartesiaSTTService, CartesiaLiveOptions from pipecat.transcriptions.language import Language # Basic configuration stt = CartesiaSTTService( api_key = os.getenv( "CARTESIA_API_KEY" ) ) # Advanced configuration live_options = CartesiaLiveOptions( model = "ink-whisper" , language = Language. ES .value, sample_rate = 16000 , encoding = "pcm_s16le" ) stt = CartesiaSTTService( api_key = os.getenv( "CARTESIA_API_KEY" ), live_options = live_options ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ Frame Flow ​ Connection Management The service automatically manages WebSocket connections: Auto-reconnect : Reconnects automatically when the connection is closed due to timeout Finalization : Sends a “finalize” command when user stops speaking to flush the transcription session Error handling : Gracefully handles connection errors and WebSocket exceptions ​ Metrics Support The service supports comprehensive metrics collection: Time to First Byte (TTFB) Processing duration Speech detection events Connection status ​ Notes Requires valid Cartesia API key Supports real-time streaming transcription Handles automatic WebSocket connection management Includes comprehensive error handling Manages connection lifecycle automatically Azure Deepgram On this page Overview Installation Configuration Constructor Parameters CartesiaLiveOptions Default Options Input Output Frames TranscriptionFrame InterimTranscriptionFrame Methods Language Setting Model Selection Usage Example Frame Flow Connection Management Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_cartesia_9f00523b.txt b/stt_cartesia_9f00523b.txt
new file mode 100644
index 0000000000000000000000000000000000000000..f395090a24159a1aa135c843481523794e831ef1
--- /dev/null
+++ b/stt_cartesia_9f00523b.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/cartesia#default-options
+Title: Cartesia - Pipecat
+==================================================
+
+Cartesia - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text Cartesia Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview CartesiaSTTService provides real-time speech-to-text capabilities using Cartesia’s WebSocket API. It supports streaming transcription with both interim and final results using the ink-whisper model. ​ Installation To use CartesiaSTTService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[cartesia]" You’ll also need to set up your Cartesia API key as an environment variable: CARTESIA_API_KEY . You can obtain a Cartesia API key by signing up at Cartesia . ​ Configuration ​ Constructor Parameters ​ api_key str required Your Cartesia API key ​ base_url str default: "api.cartesia.ai" Custom Cartesia API endpoint URL ​ sample_rate int default: "16000" Audio sample rate in Hz ​ live_options CartesiaLiveOptions Custom transcription options ​ CartesiaLiveOptions ​ model str default: "ink-whisper" The Cartesia transcription model to use ​ language str default: "en" Language code for transcription ​ encoding str default: "pcm_s16le" Audio encoding format ​ sample_rate int default: "16000" Audio sample rate in Hz ​ Default Options Copy Ask AI CartesiaLiveOptions( model = "ink-whisper" , language = "en" , encoding = "pcm_s16le" , sample_rate = 16000 ) ​ Input The service processes raw audio data with the following requirements: PCM audio format ( pcm_s16le ) 16-bit depth 16kHz sample rate (default) Single channel (mono) ​ Output Frames The service produces two types of frames during transcription: ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Final transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Detected or configured language ​ InterimTranscriptionFrame Generated during ongoing speech, containing the same fields as TranscriptionFrame but with preliminary results. ​ Methods See the STT base class methods for additional functionality. ​ Language Setting The service supports language configuration through the CartesiaLiveOptions : Copy Ask AI live_options = CartesiaLiveOptions( language = "es" ) ​ Model Selection Copy Ask AI live_options = CartesiaLiveOptions( model = "ink-whisper" ) ​ Usage Example Copy Ask AI from pipecat.services.cartesia.stt import CartesiaSTTService, CartesiaLiveOptions from pipecat.transcriptions.language import Language # Basic configuration stt = CartesiaSTTService( api_key = os.getenv( "CARTESIA_API_KEY" ) ) # Advanced configuration live_options = CartesiaLiveOptions( model = "ink-whisper" , language = Language. ES .value, sample_rate = 16000 , encoding = "pcm_s16le" ) stt = CartesiaSTTService( api_key = os.getenv( "CARTESIA_API_KEY" ), live_options = live_options ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ Frame Flow ​ Connection Management The service automatically manages WebSocket connections: Auto-reconnect : Reconnects automatically when the connection is closed due to timeout Finalization : Sends a “finalize” command when user stops speaking to flush the transcription session Error handling : Gracefully handles connection errors and WebSocket exceptions ​ Metrics Support The service supports comprehensive metrics collection: Time to First Byte (TTFB) Processing duration Speech detection events Connection status ​ Notes Requires valid Cartesia API key Supports real-time streaming transcription Handles automatic WebSocket connection management Includes comprehensive error handling Manages connection lifecycle automatically Azure Deepgram On this page Overview Installation Configuration Constructor Parameters CartesiaLiveOptions Default Options Input Output Frames TranscriptionFrame InterimTranscriptionFrame Methods Language Setting Model Selection Usage Example Frame Flow Connection Management Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_cartesia_ba7893b0.txt b/stt_cartesia_ba7893b0.txt
new file mode 100644
index 0000000000000000000000000000000000000000..096728ad1c1700dea663a573bc7729e4aaf5e424
--- /dev/null
+++ b/stt_cartesia_ba7893b0.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/cartesia#param-user-id
+Title: Cartesia - Pipecat
+==================================================
+
+Cartesia - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text Cartesia Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview CartesiaSTTService provides real-time speech-to-text capabilities using Cartesia’s WebSocket API. It supports streaming transcription with both interim and final results using the ink-whisper model. ​ Installation To use CartesiaSTTService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[cartesia]" You’ll also need to set up your Cartesia API key as an environment variable: CARTESIA_API_KEY . You can obtain a Cartesia API key by signing up at Cartesia . ​ Configuration ​ Constructor Parameters ​ api_key str required Your Cartesia API key ​ base_url str default: "api.cartesia.ai" Custom Cartesia API endpoint URL ​ sample_rate int default: "16000" Audio sample rate in Hz ​ live_options CartesiaLiveOptions Custom transcription options ​ CartesiaLiveOptions ​ model str default: "ink-whisper" The Cartesia transcription model to use ​ language str default: "en" Language code for transcription ​ encoding str default: "pcm_s16le" Audio encoding format ​ sample_rate int default: "16000" Audio sample rate in Hz ​ Default Options Copy Ask AI CartesiaLiveOptions( model = "ink-whisper" , language = "en" , encoding = "pcm_s16le" , sample_rate = 16000 ) ​ Input The service processes raw audio data with the following requirements: PCM audio format ( pcm_s16le ) 16-bit depth 16kHz sample rate (default) Single channel (mono) ​ Output Frames The service produces two types of frames during transcription: ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Final transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Detected or configured language ​ InterimTranscriptionFrame Generated during ongoing speech, containing the same fields as TranscriptionFrame but with preliminary results. ​ Methods See the STT base class methods for additional functionality. ​ Language Setting The service supports language configuration through the CartesiaLiveOptions : Copy Ask AI live_options = CartesiaLiveOptions( language = "es" ) ​ Model Selection Copy Ask AI live_options = CartesiaLiveOptions( model = "ink-whisper" ) ​ Usage Example Copy Ask AI from pipecat.services.cartesia.stt import CartesiaSTTService, CartesiaLiveOptions from pipecat.transcriptions.language import Language # Basic configuration stt = CartesiaSTTService( api_key = os.getenv( "CARTESIA_API_KEY" ) ) # Advanced configuration live_options = CartesiaLiveOptions( model = "ink-whisper" , language = Language. ES .value, sample_rate = 16000 , encoding = "pcm_s16le" ) stt = CartesiaSTTService( api_key = os.getenv( "CARTESIA_API_KEY" ), live_options = live_options ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ Frame Flow ​ Connection Management The service automatically manages WebSocket connections: Auto-reconnect : Reconnects automatically when the connection is closed due to timeout Finalization : Sends a “finalize” command when user stops speaking to flush the transcription session Error handling : Gracefully handles connection errors and WebSocket exceptions ​ Metrics Support The service supports comprehensive metrics collection: Time to First Byte (TTFB) Processing duration Speech detection events Connection status ​ Notes Requires valid Cartesia API key Supports real-time streaming transcription Handles automatic WebSocket connection management Includes comprehensive error handling Manages connection lifecycle automatically Azure Deepgram On this page Overview Installation Configuration Constructor Parameters CartesiaLiveOptions Default Options Input Output Frames TranscriptionFrame InterimTranscriptionFrame Methods Language Setting Model Selection Usage Example Frame Flow Connection Management Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_cartesia_ba9d21a5.txt b/stt_cartesia_ba9d21a5.txt
new file mode 100644
index 0000000000000000000000000000000000000000..6eaa31392e5fb3a2cbafb6b0ff81ba0cc60f72e1
--- /dev/null
+++ b/stt_cartesia_ba9d21a5.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/cartesia#frame-flow
+Title: Cartesia - Pipecat
+==================================================
+
+Cartesia - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text Cartesia Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview CartesiaSTTService provides real-time speech-to-text capabilities using Cartesia’s WebSocket API. It supports streaming transcription with both interim and final results using the ink-whisper model. ​ Installation To use CartesiaSTTService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[cartesia]" You’ll also need to set up your Cartesia API key as an environment variable: CARTESIA_API_KEY . You can obtain a Cartesia API key by signing up at Cartesia . ​ Configuration ​ Constructor Parameters ​ api_key str required Your Cartesia API key ​ base_url str default: "api.cartesia.ai" Custom Cartesia API endpoint URL ​ sample_rate int default: "16000" Audio sample rate in Hz ​ live_options CartesiaLiveOptions Custom transcription options ​ CartesiaLiveOptions ​ model str default: "ink-whisper" The Cartesia transcription model to use ​ language str default: "en" Language code for transcription ​ encoding str default: "pcm_s16le" Audio encoding format ​ sample_rate int default: "16000" Audio sample rate in Hz ​ Default Options Copy Ask AI CartesiaLiveOptions( model = "ink-whisper" , language = "en" , encoding = "pcm_s16le" , sample_rate = 16000 ) ​ Input The service processes raw audio data with the following requirements: PCM audio format ( pcm_s16le ) 16-bit depth 16kHz sample rate (default) Single channel (mono) ​ Output Frames The service produces two types of frames during transcription: ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Final transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Detected or configured language ​ InterimTranscriptionFrame Generated during ongoing speech, containing the same fields as TranscriptionFrame but with preliminary results. ​ Methods See the STT base class methods for additional functionality. ​ Language Setting The service supports language configuration through the CartesiaLiveOptions : Copy Ask AI live_options = CartesiaLiveOptions( language = "es" ) ​ Model Selection Copy Ask AI live_options = CartesiaLiveOptions( model = "ink-whisper" ) ​ Usage Example Copy Ask AI from pipecat.services.cartesia.stt import CartesiaSTTService, CartesiaLiveOptions from pipecat.transcriptions.language import Language # Basic configuration stt = CartesiaSTTService( api_key = os.getenv( "CARTESIA_API_KEY" ) ) # Advanced configuration live_options = CartesiaLiveOptions( model = "ink-whisper" , language = Language. ES .value, sample_rate = 16000 , encoding = "pcm_s16le" ) stt = CartesiaSTTService( api_key = os.getenv( "CARTESIA_API_KEY" ), live_options = live_options ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ Frame Flow ​ Connection Management The service automatically manages WebSocket connections: Auto-reconnect : Reconnects automatically when the connection is closed due to timeout Finalization : Sends a “finalize” command when user stops speaking to flush the transcription session Error handling : Gracefully handles connection errors and WebSocket exceptions ​ Metrics Support The service supports comprehensive metrics collection: Time to First Byte (TTFB) Processing duration Speech detection events Connection status ​ Notes Requires valid Cartesia API key Supports real-time streaming transcription Handles automatic WebSocket connection management Includes comprehensive error handling Manages connection lifecycle automatically Azure Deepgram On this page Overview Installation Configuration Constructor Parameters CartesiaLiveOptions Default Options Input Output Frames TranscriptionFrame InterimTranscriptionFrame Methods Language Setting Model Selection Usage Example Frame Flow Connection Management Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_cartesia_cca65ff5.txt b/stt_cartesia_cca65ff5.txt
new file mode 100644
index 0000000000000000000000000000000000000000..bc064976201368a76c5c015df3b7d314de231c15
--- /dev/null
+++ b/stt_cartesia_cca65ff5.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/cartesia#output-frames
+Title: Cartesia - Pipecat
+==================================================
+
+Cartesia - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text Cartesia Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview CartesiaSTTService provides real-time speech-to-text capabilities using Cartesia’s WebSocket API. It supports streaming transcription with both interim and final results using the ink-whisper model. ​ Installation To use CartesiaSTTService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[cartesia]" You’ll also need to set up your Cartesia API key as an environment variable: CARTESIA_API_KEY . You can obtain a Cartesia API key by signing up at Cartesia . ​ Configuration ​ Constructor Parameters ​ api_key str required Your Cartesia API key ​ base_url str default: "api.cartesia.ai" Custom Cartesia API endpoint URL ​ sample_rate int default: "16000" Audio sample rate in Hz ​ live_options CartesiaLiveOptions Custom transcription options ​ CartesiaLiveOptions ​ model str default: "ink-whisper" The Cartesia transcription model to use ​ language str default: "en" Language code for transcription ​ encoding str default: "pcm_s16le" Audio encoding format ​ sample_rate int default: "16000" Audio sample rate in Hz ​ Default Options Copy Ask AI CartesiaLiveOptions( model = "ink-whisper" , language = "en" , encoding = "pcm_s16le" , sample_rate = 16000 ) ​ Input The service processes raw audio data with the following requirements: PCM audio format ( pcm_s16le ) 16-bit depth 16kHz sample rate (default) Single channel (mono) ​ Output Frames The service produces two types of frames during transcription: ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Final transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Detected or configured language ​ InterimTranscriptionFrame Generated during ongoing speech, containing the same fields as TranscriptionFrame but with preliminary results. ​ Methods See the STT base class methods for additional functionality. ​ Language Setting The service supports language configuration through the CartesiaLiveOptions : Copy Ask AI live_options = CartesiaLiveOptions( language = "es" ) ​ Model Selection Copy Ask AI live_options = CartesiaLiveOptions( model = "ink-whisper" ) ​ Usage Example Copy Ask AI from pipecat.services.cartesia.stt import CartesiaSTTService, CartesiaLiveOptions from pipecat.transcriptions.language import Language # Basic configuration stt = CartesiaSTTService( api_key = os.getenv( "CARTESIA_API_KEY" ) ) # Advanced configuration live_options = CartesiaLiveOptions( model = "ink-whisper" , language = Language. ES .value, sample_rate = 16000 , encoding = "pcm_s16le" ) stt = CartesiaSTTService( api_key = os.getenv( "CARTESIA_API_KEY" ), live_options = live_options ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ Frame Flow ​ Connection Management The service automatically manages WebSocket connections: Auto-reconnect : Reconnects automatically when the connection is closed due to timeout Finalization : Sends a “finalize” command when user stops speaking to flush the transcription session Error handling : Gracefully handles connection errors and WebSocket exceptions ​ Metrics Support The service supports comprehensive metrics collection: Time to First Byte (TTFB) Processing duration Speech detection events Connection status ​ Notes Requires valid Cartesia API key Supports real-time streaming transcription Handles automatic WebSocket connection management Includes comprehensive error handling Manages connection lifecycle automatically Azure Deepgram On this page Overview Installation Configuration Constructor Parameters CartesiaLiveOptions Default Options Input Output Frames TranscriptionFrame InterimTranscriptionFrame Methods Language Setting Model Selection Usage Example Frame Flow Connection Management Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_cartesia_e7cf5276.txt b/stt_cartesia_e7cf5276.txt
new file mode 100644
index 0000000000000000000000000000000000000000..1ca7655d3ee02996d33388b1990e807250bb09f3
--- /dev/null
+++ b/stt_cartesia_e7cf5276.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/cartesia#param-api-key
+Title: Cartesia - Pipecat
+==================================================
+
+Cartesia - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text Cartesia Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview CartesiaSTTService provides real-time speech-to-text capabilities using Cartesia’s WebSocket API. It supports streaming transcription with both interim and final results using the ink-whisper model. ​ Installation To use CartesiaSTTService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[cartesia]" You’ll also need to set up your Cartesia API key as an environment variable: CARTESIA_API_KEY . You can obtain a Cartesia API key by signing up at Cartesia . ​ Configuration ​ Constructor Parameters ​ api_key str required Your Cartesia API key ​ base_url str default: "api.cartesia.ai" Custom Cartesia API endpoint URL ​ sample_rate int default: "16000" Audio sample rate in Hz ​ live_options CartesiaLiveOptions Custom transcription options ​ CartesiaLiveOptions ​ model str default: "ink-whisper" The Cartesia transcription model to use ​ language str default: "en" Language code for transcription ​ encoding str default: "pcm_s16le" Audio encoding format ​ sample_rate int default: "16000" Audio sample rate in Hz ​ Default Options Copy Ask AI CartesiaLiveOptions( model = "ink-whisper" , language = "en" , encoding = "pcm_s16le" , sample_rate = 16000 ) ​ Input The service processes raw audio data with the following requirements: PCM audio format ( pcm_s16le ) 16-bit depth 16kHz sample rate (default) Single channel (mono) ​ Output Frames The service produces two types of frames during transcription: ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Final transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Detected or configured language ​ InterimTranscriptionFrame Generated during ongoing speech, containing the same fields as TranscriptionFrame but with preliminary results. ​ Methods See the STT base class methods for additional functionality. ​ Language Setting The service supports language configuration through the CartesiaLiveOptions : Copy Ask AI live_options = CartesiaLiveOptions( language = "es" ) ​ Model Selection Copy Ask AI live_options = CartesiaLiveOptions( model = "ink-whisper" ) ​ Usage Example Copy Ask AI from pipecat.services.cartesia.stt import CartesiaSTTService, CartesiaLiveOptions from pipecat.transcriptions.language import Language # Basic configuration stt = CartesiaSTTService( api_key = os.getenv( "CARTESIA_API_KEY" ) ) # Advanced configuration live_options = CartesiaLiveOptions( model = "ink-whisper" , language = Language. ES .value, sample_rate = 16000 , encoding = "pcm_s16le" ) stt = CartesiaSTTService( api_key = os.getenv( "CARTESIA_API_KEY" ), live_options = live_options ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ Frame Flow ​ Connection Management The service automatically manages WebSocket connections: Auto-reconnect : Reconnects automatically when the connection is closed due to timeout Finalization : Sends a “finalize” command when user stops speaking to flush the transcription session Error handling : Gracefully handles connection errors and WebSocket exceptions ​ Metrics Support The service supports comprehensive metrics collection: Time to First Byte (TTFB) Processing duration Speech detection events Connection status ​ Notes Requires valid Cartesia API key Supports real-time streaming transcription Handles automatic WebSocket connection management Includes comprehensive error handling Manages connection lifecycle automatically Azure Deepgram On this page Overview Installation Configuration Constructor Parameters CartesiaLiveOptions Default Options Input Output Frames TranscriptionFrame InterimTranscriptionFrame Methods Language Setting Model Selection Usage Example Frame Flow Connection Management Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_deepgram_377c13b8.txt b/stt_deepgram_377c13b8.txt
new file mode 100644
index 0000000000000000000000000000000000000000..0cd70afa25941a558bcb9cf00f00023e8599f386
--- /dev/null
+++ b/stt_deepgram_377c13b8.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/deepgram#output
+Title: Deepgram - Pipecat
+==================================================
+
+Deepgram - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text Deepgram Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview DeepgramSTTService provides real-time speech recognition using Deepgram’s WebSocket API with support for interim results, language detection, and voice activity detection (VAD). API Reference Complete API documentation and method details Deepgram Docs Official Deepgram documentation and features Example Code Working example with interruption handling ​ Installation To use DeepgramSTTService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[deepgram]" You’ll also need to set up your Deepgram API key as an environment variable: DEEPGRAM_API_KEY . Get your API key from the Deepgram Console . ​ Frames ​ Input InputAudioRawFrame - Raw PCM audio data (16-bit, 16kHz, mono) STTUpdateSettingsFrame - Runtime transcription configuration updates STTMuteFrame - Mute audio input for transcription ​ Output InterimTranscriptionFrame - Real-time transcription updates TranscriptionFrame - Final transcription results ErrorFrame - Connection or processing errors ​ Language Support Deepgram STT supports the following languages and regional variants: Language Code Description Service Codes Language.BG Bulgarian bg Language.CA Catalan ca Language.ZH Chinese (Mandarin, Simplified) zh , zh-CN , zh-Hans Language.ZH_TW Chinese (Mandarin, Traditional) zh-TW , zh-Hant Language.ZH_HK Chinese (Cantonese, Traditional) zh-HK Language.CS Czech cs Language.DA Danish da , da-DK Language.NL Dutch nl Language.NL_BE Dutch (Flemish) nl-BE Language.EN English en Language.EN_US English (US) en-US Language.EN_AU English (Australia) en-AU Language.EN_GB English (UK) en-GB Language.EN_NZ English (New Zealand) en-NZ Language.EN_IN English (India) en-IN Language.ET Estonian et Language.FI Finnish fi Language.FR French fr Language.FR_CA French (Canada) fr-CA Language.DE German de Language.DE_CH German (Switzerland) de-CH Language.EL Greek el Language.HI Hindi hi Language.HU Hungarian hu Language.ID Indonesian id Language.IT Italian it Language.JA Japanese ja Language.KO Korean ko , ko-KR Language.LV Latvian lv Language.LT Lithuanian lt Language.MS Malay ms Language.NO Norwegian no Language.PL Polish pl Language.PT Portuguese pt Language.PT_BR Portuguese (Brazil) pt-BR Language.PT_PT Portuguese (Portugal) pt-PT Language.RO Romanian ro Language.RU Russian ru Language.SK Slovak sk Language.ES Spanish es , es-419 Language.SV Swedish sv , sv-SE Language.TH Thai th , th-TH Language.TR Turkish tr Language.UK Ukrainian uk Language.VI Vietnamese vi ​ Usage Example Copy Ask AI from pipecat.services.deepgram.stt import DeepgramSTTService from deepgram import LiveOptions from pipecat.transcriptions.language import Language # Configure service stt = DeepgramSTTService( api_key = "your-api-key" , live_options = LiveOptions( model = "nova-3-general" , language = "en-US" , smart_format = True ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, tts, transport.output() ]) # Change language dynamically await stt.set_language(Language. FR ) ​ Metrics The service provides: Time to First Byte (TTFB) - Latency from audio input to first transcription Processing Duration - Total time spent processing audio ​ Additional Notes Connection Management : Automatically handles WebSocket connections and reconnections VAD Integration : Supports Deepgram’s built-in VAD, though we recommend using local VAD services like Silero for better performance Sample Rate : Can be configured per service, but we recommend setting it globally in PipelineParams for consistency across services Cartesia Fal (Wizper) On this page Overview Installation Frames Input Output Language Support Usage Example Metrics Additional Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_deepgram_4d0cb34c.txt b/stt_deepgram_4d0cb34c.txt
new file mode 100644
index 0000000000000000000000000000000000000000..aef956326efd62c30dd9d159420708ea5c4ad236
--- /dev/null
+++ b/stt_deepgram_4d0cb34c.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/deepgram#input
+Title: Deepgram - Pipecat
+==================================================
+
+Deepgram - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text Deepgram Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview DeepgramSTTService provides real-time speech recognition using Deepgram’s WebSocket API with support for interim results, language detection, and voice activity detection (VAD). API Reference Complete API documentation and method details Deepgram Docs Official Deepgram documentation and features Example Code Working example with interruption handling ​ Installation To use DeepgramSTTService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[deepgram]" You’ll also need to set up your Deepgram API key as an environment variable: DEEPGRAM_API_KEY . Get your API key from the Deepgram Console . ​ Frames ​ Input InputAudioRawFrame - Raw PCM audio data (16-bit, 16kHz, mono) STTUpdateSettingsFrame - Runtime transcription configuration updates STTMuteFrame - Mute audio input for transcription ​ Output InterimTranscriptionFrame - Real-time transcription updates TranscriptionFrame - Final transcription results ErrorFrame - Connection or processing errors ​ Language Support Deepgram STT supports the following languages and regional variants: Language Code Description Service Codes Language.BG Bulgarian bg Language.CA Catalan ca Language.ZH Chinese (Mandarin, Simplified) zh , zh-CN , zh-Hans Language.ZH_TW Chinese (Mandarin, Traditional) zh-TW , zh-Hant Language.ZH_HK Chinese (Cantonese, Traditional) zh-HK Language.CS Czech cs Language.DA Danish da , da-DK Language.NL Dutch nl Language.NL_BE Dutch (Flemish) nl-BE Language.EN English en Language.EN_US English (US) en-US Language.EN_AU English (Australia) en-AU Language.EN_GB English (UK) en-GB Language.EN_NZ English (New Zealand) en-NZ Language.EN_IN English (India) en-IN Language.ET Estonian et Language.FI Finnish fi Language.FR French fr Language.FR_CA French (Canada) fr-CA Language.DE German de Language.DE_CH German (Switzerland) de-CH Language.EL Greek el Language.HI Hindi hi Language.HU Hungarian hu Language.ID Indonesian id Language.IT Italian it Language.JA Japanese ja Language.KO Korean ko , ko-KR Language.LV Latvian lv Language.LT Lithuanian lt Language.MS Malay ms Language.NO Norwegian no Language.PL Polish pl Language.PT Portuguese pt Language.PT_BR Portuguese (Brazil) pt-BR Language.PT_PT Portuguese (Portugal) pt-PT Language.RO Romanian ro Language.RU Russian ru Language.SK Slovak sk Language.ES Spanish es , es-419 Language.SV Swedish sv , sv-SE Language.TH Thai th , th-TH Language.TR Turkish tr Language.UK Ukrainian uk Language.VI Vietnamese vi ​ Usage Example Copy Ask AI from pipecat.services.deepgram.stt import DeepgramSTTService from deepgram import LiveOptions from pipecat.transcriptions.language import Language # Configure service stt = DeepgramSTTService( api_key = "your-api-key" , live_options = LiveOptions( model = "nova-3-general" , language = "en-US" , smart_format = True ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, tts, transport.output() ]) # Change language dynamically await stt.set_language(Language. FR ) ​ Metrics The service provides: Time to First Byte (TTFB) - Latency from audio input to first transcription Processing Duration - Total time spent processing audio ​ Additional Notes Connection Management : Automatically handles WebSocket connections and reconnections VAD Integration : Supports Deepgram’s built-in VAD, though we recommend using local VAD services like Silero for better performance Sample Rate : Can be configured per service, but we recommend setting it globally in PipelineParams for consistency across services Cartesia Fal (Wizper) On this page Overview Installation Frames Input Output Language Support Usage Example Metrics Additional Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_deepgram_b832241c.txt b/stt_deepgram_b832241c.txt
new file mode 100644
index 0000000000000000000000000000000000000000..43f2103716e526a09a716d4718390b76ead25536
--- /dev/null
+++ b/stt_deepgram_b832241c.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/deepgram#language-support
+Title: Deepgram - Pipecat
+==================================================
+
+Deepgram - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text Deepgram Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview DeepgramSTTService provides real-time speech recognition using Deepgram’s WebSocket API with support for interim results, language detection, and voice activity detection (VAD). API Reference Complete API documentation and method details Deepgram Docs Official Deepgram documentation and features Example Code Working example with interruption handling ​ Installation To use DeepgramSTTService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[deepgram]" You’ll also need to set up your Deepgram API key as an environment variable: DEEPGRAM_API_KEY . Get your API key from the Deepgram Console . ​ Frames ​ Input InputAudioRawFrame - Raw PCM audio data (16-bit, 16kHz, mono) STTUpdateSettingsFrame - Runtime transcription configuration updates STTMuteFrame - Mute audio input for transcription ​ Output InterimTranscriptionFrame - Real-time transcription updates TranscriptionFrame - Final transcription results ErrorFrame - Connection or processing errors ​ Language Support Deepgram STT supports the following languages and regional variants: Language Code Description Service Codes Language.BG Bulgarian bg Language.CA Catalan ca Language.ZH Chinese (Mandarin, Simplified) zh , zh-CN , zh-Hans Language.ZH_TW Chinese (Mandarin, Traditional) zh-TW , zh-Hant Language.ZH_HK Chinese (Cantonese, Traditional) zh-HK Language.CS Czech cs Language.DA Danish da , da-DK Language.NL Dutch nl Language.NL_BE Dutch (Flemish) nl-BE Language.EN English en Language.EN_US English (US) en-US Language.EN_AU English (Australia) en-AU Language.EN_GB English (UK) en-GB Language.EN_NZ English (New Zealand) en-NZ Language.EN_IN English (India) en-IN Language.ET Estonian et Language.FI Finnish fi Language.FR French fr Language.FR_CA French (Canada) fr-CA Language.DE German de Language.DE_CH German (Switzerland) de-CH Language.EL Greek el Language.HI Hindi hi Language.HU Hungarian hu Language.ID Indonesian id Language.IT Italian it Language.JA Japanese ja Language.KO Korean ko , ko-KR Language.LV Latvian lv Language.LT Lithuanian lt Language.MS Malay ms Language.NO Norwegian no Language.PL Polish pl Language.PT Portuguese pt Language.PT_BR Portuguese (Brazil) pt-BR Language.PT_PT Portuguese (Portugal) pt-PT Language.RO Romanian ro Language.RU Russian ru Language.SK Slovak sk Language.ES Spanish es , es-419 Language.SV Swedish sv , sv-SE Language.TH Thai th , th-TH Language.TR Turkish tr Language.UK Ukrainian uk Language.VI Vietnamese vi ​ Usage Example Copy Ask AI from pipecat.services.deepgram.stt import DeepgramSTTService from deepgram import LiveOptions from pipecat.transcriptions.language import Language # Configure service stt = DeepgramSTTService( api_key = "your-api-key" , live_options = LiveOptions( model = "nova-3-general" , language = "en-US" , smart_format = True ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, tts, transport.output() ]) # Change language dynamically await stt.set_language(Language. FR ) ​ Metrics The service provides: Time to First Byte (TTFB) - Latency from audio input to first transcription Processing Duration - Total time spent processing audio ​ Additional Notes Connection Management : Automatically handles WebSocket connections and reconnections VAD Integration : Supports Deepgram’s built-in VAD, though we recommend using local VAD services like Silero for better performance Sample Rate : Can be configured per service, but we recommend setting it globally in PipelineParams for consistency across services Cartesia Fal (Wizper) On this page Overview Installation Frames Input Output Language Support Usage Example Metrics Additional Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_fal_03f0e317.txt b/stt_fal_03f0e317.txt
new file mode 100644
index 0000000000000000000000000000000000000000..70f5a234201eee70b13ed884a18016d6026e190c
--- /dev/null
+++ b/stt_fal_03f0e317.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/fal#overview
+Title: Fal (Wizper) - Pipecat
+==================================================
+
+Fal (Wizper) - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text Fal (Wizper) Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview FalSTTService provides speech-to-text capabilities using Fal’s Wizper API. It offers high-quality transcription with minimal setup. The service uses Voice Activity Detection (VAD) to process only speech segments, optimizing API usage and improving response time. ​ Installation To use FalSTTService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[fal]" You’ll need to set up your Fal API key as an environment variable: FAL_KEY . You can obtain a Fal API key from the Fal platform . ​ Configuration ​ Constructor Parameters ​ api_key str Your Fal API key. If not provided, will use the FAL_KEY environment variable. ​ sample_rate int Audio sample rate in Hz. If not provided, uses the pipeline’s sample rate. ​ params InputParams Configuration parameters for the Wizper API. See InputParams below. ​ InputParams ​ language Language default: "Language.EN" Language of the audio input. Defaults to English. ​ task str default: "transcribe" Task to perform. Options are ‘transcribe’ or ‘translate’. ​ chunk_level str default: "segment" Level of chunking for the audio. Default is ‘segment’. ​ version str default: "3" Version of Wizper model to use. ​ Input The service processes audio data with the following requirements: PCM audio format 16-bit depth Single channel (mono) ​ Output Frames The service produces two types of frames during transcription: ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Detected language (if available) ​ ErrorFrame Generated when transcription errors occur, containing error details. ​ Methods ​ Set Model Copy Ask AI await service.set_model( "wizper-v3" ) See the STT base class methods for additional functionality. ​ Language Support Fal Wizper supports a wide range of languages. The service automatically maps Language enum values to the appropriate Wizper language codes. Language Code Description Wizper Code Language.AF Afrikaans af Language.AM Amharic am Language.AR Arabic ar Language.AS Assamese as Language.AZ Azerbaijani az Language.BA Bashkir ba Language.BE Belarusian be Language.BG Bulgarian bg Language.BN Bengali bn Language.BO Tibetan bo Language.BR Breton br Language.BS Bosnian bs Language.CA Catalan ca Language.CS Czech cs Language.CY Welsh cy Language.DA Danish da Language.DE German de Language.EL Greek el Language.EN English en Language.ES Spanish es Language.ET Estonian et Language.EU Basque eu Language.FA Persian fa Language.FI Finnish fi Language.FO Faroese fo Language.FR French fr Language.GL Galician gl Language.GU Gujarati gu Language.HA Hausa ha Language.HE Hebrew he Language.HI Hindi hi Language.HR Croatian hr Language.HT Haitian Creole ht Language.HU Hungarian hu Language.HY Armenian hy Language.ID Indonesian id Language.IS Icelandic is Language.IT Italian it Language.JA Japanese ja Language.JW Javanese jw Language.KA Georgian ka Language.KK Kazakh kk Language.KM Khmer km Language.KN Kannada kn Language.KO Korean ko Language.LA Latin la Language.LB Luxembourgish lb Language.LN Lingala ln Language.LO Lao lo Language.LT Lithuanian lt Language.LV Latvian lv Language.MG Malagasy mg Language.MI Maori mi Language.MK Macedonian mk Language.ML Malayalam ml Language.MN Mongolian mn Language.MR Marathi mr Language.MS Malay ms Language.MT Maltese mt Language.MY Burmese my Language.NE Nepali ne Language.NL Dutch nl Language.NN Norwegian Nynorsk nn Language.NO Norwegian no Language.OC Occitan oc Language.PA Punjabi pa Language.PL Polish pl Language.PS Pashto ps Language.PT Portuguese pt Language.RO Romanian ro Language.RU Russian ru Language.SA Sanskrit sa Language.SD Sindhi sd Language.SI Sinhala si Language.SK Slovak sk Language.SL Slovenian sl Language.SN Shona sn Language.SO Somali so Language.SQ Albanian sq Language.SR Serbian sr Language.SU Sundanese su Language.SV Swedish sv Language.SW Swahili sw Language.TA Tamil ta Language.TE Telugu te Language.TG Tajik tg Language.TH Thai th Language.TK Turkmen tk Language.TL Tagalog tl Language.TR Turkish tr Language.TT Tatar tt Language.UK Ukrainian uk Language.UR Urdu ur Language.UZ Uzbek uz Language.VI Vietnamese vi Language.YI Yiddish yi Language.YO Yoruba yo Language.ZH Chinese zh Fal Wizper supports a range of languages and dialects. For the most accurate transcription, specify the correct language for your audio input. ​ Usage Example Copy Ask AI from pipecat.services.fal.stt import FalSTTService from pipecat.transcriptions.language import Language # Configure service stt = FalSTTService( api_key = "your-fal-api-key" , params = FalSTTService.InputParams( language = Language. EN , ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ Voice Activity Detection Integration This service inherits from SegmentedSTTService , which uses Voice Activity Detection (VAD) to identify speech segments for processing. This approach: Processes only actual speech, not silence or background noise Maintains a small audio buffer (default 1 second) to capture speech that occurs slightly before VAD detection Receives UserStartedSpeakingFrame and UserStoppedSpeakingFrame from a VAD component in the pipeline Only sends complete utterances to the API when speech has ended Ensure your transport includes a VAD component (like SileroVADAnalyzer ) to properly detect speech segments. ​ Metrics Support The service collects the following metrics: Processing duration API response time Success/failure rates ​ Notes Requires valid Fal API key Uses Fal’s Wizper model Requires VAD component in transport Processes complete utterances, not continuous audio Thread-safe processing ​ Error Handling The service handles common API errors including: Authentication errors API availability issues Invalid audio format Network connectivity issues API timeouts Errors are propagated through ErrorFrames with descriptive messages. Deepgram Gladia On this page Overview Installation Configuration Constructor Parameters InputParams Input Output Frames TranscriptionFrame ErrorFrame Methods Set Model Language Support Usage Example Voice Activity Detection Integration Metrics Support Notes Error Handling Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_fal_66bad817.txt b/stt_fal_66bad817.txt
new file mode 100644
index 0000000000000000000000000000000000000000..d2bdf4be383045d03ac3ff0341b3424effbc9420
--- /dev/null
+++ b/stt_fal_66bad817.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/fal#param-chunk-level
+Title: Fal (Wizper) - Pipecat
+==================================================
+
+Fal (Wizper) - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text Fal (Wizper) Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview FalSTTService provides speech-to-text capabilities using Fal’s Wizper API. It offers high-quality transcription with minimal setup. The service uses Voice Activity Detection (VAD) to process only speech segments, optimizing API usage and improving response time. ​ Installation To use FalSTTService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[fal]" You’ll need to set up your Fal API key as an environment variable: FAL_KEY . You can obtain a Fal API key from the Fal platform . ​ Configuration ​ Constructor Parameters ​ api_key str Your Fal API key. If not provided, will use the FAL_KEY environment variable. ​ sample_rate int Audio sample rate in Hz. If not provided, uses the pipeline’s sample rate. ​ params InputParams Configuration parameters for the Wizper API. See InputParams below. ​ InputParams ​ language Language default: "Language.EN" Language of the audio input. Defaults to English. ​ task str default: "transcribe" Task to perform. Options are ‘transcribe’ or ‘translate’. ​ chunk_level str default: "segment" Level of chunking for the audio. Default is ‘segment’. ​ version str default: "3" Version of Wizper model to use. ​ Input The service processes audio data with the following requirements: PCM audio format 16-bit depth Single channel (mono) ​ Output Frames The service produces two types of frames during transcription: ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Detected language (if available) ​ ErrorFrame Generated when transcription errors occur, containing error details. ​ Methods ​ Set Model Copy Ask AI await service.set_model( "wizper-v3" ) See the STT base class methods for additional functionality. ​ Language Support Fal Wizper supports a wide range of languages. The service automatically maps Language enum values to the appropriate Wizper language codes. Language Code Description Wizper Code Language.AF Afrikaans af Language.AM Amharic am Language.AR Arabic ar Language.AS Assamese as Language.AZ Azerbaijani az Language.BA Bashkir ba Language.BE Belarusian be Language.BG Bulgarian bg Language.BN Bengali bn Language.BO Tibetan bo Language.BR Breton br Language.BS Bosnian bs Language.CA Catalan ca Language.CS Czech cs Language.CY Welsh cy Language.DA Danish da Language.DE German de Language.EL Greek el Language.EN English en Language.ES Spanish es Language.ET Estonian et Language.EU Basque eu Language.FA Persian fa Language.FI Finnish fi Language.FO Faroese fo Language.FR French fr Language.GL Galician gl Language.GU Gujarati gu Language.HA Hausa ha Language.HE Hebrew he Language.HI Hindi hi Language.HR Croatian hr Language.HT Haitian Creole ht Language.HU Hungarian hu Language.HY Armenian hy Language.ID Indonesian id Language.IS Icelandic is Language.IT Italian it Language.JA Japanese ja Language.JW Javanese jw Language.KA Georgian ka Language.KK Kazakh kk Language.KM Khmer km Language.KN Kannada kn Language.KO Korean ko Language.LA Latin la Language.LB Luxembourgish lb Language.LN Lingala ln Language.LO Lao lo Language.LT Lithuanian lt Language.LV Latvian lv Language.MG Malagasy mg Language.MI Maori mi Language.MK Macedonian mk Language.ML Malayalam ml Language.MN Mongolian mn Language.MR Marathi mr Language.MS Malay ms Language.MT Maltese mt Language.MY Burmese my Language.NE Nepali ne Language.NL Dutch nl Language.NN Norwegian Nynorsk nn Language.NO Norwegian no Language.OC Occitan oc Language.PA Punjabi pa Language.PL Polish pl Language.PS Pashto ps Language.PT Portuguese pt Language.RO Romanian ro Language.RU Russian ru Language.SA Sanskrit sa Language.SD Sindhi sd Language.SI Sinhala si Language.SK Slovak sk Language.SL Slovenian sl Language.SN Shona sn Language.SO Somali so Language.SQ Albanian sq Language.SR Serbian sr Language.SU Sundanese su Language.SV Swedish sv Language.SW Swahili sw Language.TA Tamil ta Language.TE Telugu te Language.TG Tajik tg Language.TH Thai th Language.TK Turkmen tk Language.TL Tagalog tl Language.TR Turkish tr Language.TT Tatar tt Language.UK Ukrainian uk Language.UR Urdu ur Language.UZ Uzbek uz Language.VI Vietnamese vi Language.YI Yiddish yi Language.YO Yoruba yo Language.ZH Chinese zh Fal Wizper supports a range of languages and dialects. For the most accurate transcription, specify the correct language for your audio input. ​ Usage Example Copy Ask AI from pipecat.services.fal.stt import FalSTTService from pipecat.transcriptions.language import Language # Configure service stt = FalSTTService( api_key = "your-fal-api-key" , params = FalSTTService.InputParams( language = Language. EN , ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ Voice Activity Detection Integration This service inherits from SegmentedSTTService , which uses Voice Activity Detection (VAD) to identify speech segments for processing. This approach: Processes only actual speech, not silence or background noise Maintains a small audio buffer (default 1 second) to capture speech that occurs slightly before VAD detection Receives UserStartedSpeakingFrame and UserStoppedSpeakingFrame from a VAD component in the pipeline Only sends complete utterances to the API when speech has ended Ensure your transport includes a VAD component (like SileroVADAnalyzer ) to properly detect speech segments. ​ Metrics Support The service collects the following metrics: Processing duration API response time Success/failure rates ​ Notes Requires valid Fal API key Uses Fal’s Wizper model Requires VAD component in transport Processes complete utterances, not continuous audio Thread-safe processing ​ Error Handling The service handles common API errors including: Authentication errors API availability issues Invalid audio format Network connectivity issues API timeouts Errors are propagated through ErrorFrames with descriptive messages. Deepgram Gladia On this page Overview Installation Configuration Constructor Parameters InputParams Input Output Frames TranscriptionFrame ErrorFrame Methods Set Model Language Support Usage Example Voice Activity Detection Integration Metrics Support Notes Error Handling Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_fal_856a6511.txt b/stt_fal_856a6511.txt
new file mode 100644
index 0000000000000000000000000000000000000000..32e14afbe7c279c77354a896fb728cb98b49b8cb
--- /dev/null
+++ b/stt_fal_856a6511.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/fal#param-language-1
+Title: Fal (Wizper) - Pipecat
+==================================================
+
+Fal (Wizper) - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text Fal (Wizper) Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview FalSTTService provides speech-to-text capabilities using Fal’s Wizper API. It offers high-quality transcription with minimal setup. The service uses Voice Activity Detection (VAD) to process only speech segments, optimizing API usage and improving response time. ​ Installation To use FalSTTService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[fal]" You’ll need to set up your Fal API key as an environment variable: FAL_KEY . You can obtain a Fal API key from the Fal platform . ​ Configuration ​ Constructor Parameters ​ api_key str Your Fal API key. If not provided, will use the FAL_KEY environment variable. ​ sample_rate int Audio sample rate in Hz. If not provided, uses the pipeline’s sample rate. ​ params InputParams Configuration parameters for the Wizper API. See InputParams below. ​ InputParams ​ language Language default: "Language.EN" Language of the audio input. Defaults to English. ​ task str default: "transcribe" Task to perform. Options are ‘transcribe’ or ‘translate’. ​ chunk_level str default: "segment" Level of chunking for the audio. Default is ‘segment’. ​ version str default: "3" Version of Wizper model to use. ​ Input The service processes audio data with the following requirements: PCM audio format 16-bit depth Single channel (mono) ​ Output Frames The service produces two types of frames during transcription: ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Detected language (if available) ​ ErrorFrame Generated when transcription errors occur, containing error details. ​ Methods ​ Set Model Copy Ask AI await service.set_model( "wizper-v3" ) See the STT base class methods for additional functionality. ​ Language Support Fal Wizper supports a wide range of languages. The service automatically maps Language enum values to the appropriate Wizper language codes. Language Code Description Wizper Code Language.AF Afrikaans af Language.AM Amharic am Language.AR Arabic ar Language.AS Assamese as Language.AZ Azerbaijani az Language.BA Bashkir ba Language.BE Belarusian be Language.BG Bulgarian bg Language.BN Bengali bn Language.BO Tibetan bo Language.BR Breton br Language.BS Bosnian bs Language.CA Catalan ca Language.CS Czech cs Language.CY Welsh cy Language.DA Danish da Language.DE German de Language.EL Greek el Language.EN English en Language.ES Spanish es Language.ET Estonian et Language.EU Basque eu Language.FA Persian fa Language.FI Finnish fi Language.FO Faroese fo Language.FR French fr Language.GL Galician gl Language.GU Gujarati gu Language.HA Hausa ha Language.HE Hebrew he Language.HI Hindi hi Language.HR Croatian hr Language.HT Haitian Creole ht Language.HU Hungarian hu Language.HY Armenian hy Language.ID Indonesian id Language.IS Icelandic is Language.IT Italian it Language.JA Japanese ja Language.JW Javanese jw Language.KA Georgian ka Language.KK Kazakh kk Language.KM Khmer km Language.KN Kannada kn Language.KO Korean ko Language.LA Latin la Language.LB Luxembourgish lb Language.LN Lingala ln Language.LO Lao lo Language.LT Lithuanian lt Language.LV Latvian lv Language.MG Malagasy mg Language.MI Maori mi Language.MK Macedonian mk Language.ML Malayalam ml Language.MN Mongolian mn Language.MR Marathi mr Language.MS Malay ms Language.MT Maltese mt Language.MY Burmese my Language.NE Nepali ne Language.NL Dutch nl Language.NN Norwegian Nynorsk nn Language.NO Norwegian no Language.OC Occitan oc Language.PA Punjabi pa Language.PL Polish pl Language.PS Pashto ps Language.PT Portuguese pt Language.RO Romanian ro Language.RU Russian ru Language.SA Sanskrit sa Language.SD Sindhi sd Language.SI Sinhala si Language.SK Slovak sk Language.SL Slovenian sl Language.SN Shona sn Language.SO Somali so Language.SQ Albanian sq Language.SR Serbian sr Language.SU Sundanese su Language.SV Swedish sv Language.SW Swahili sw Language.TA Tamil ta Language.TE Telugu te Language.TG Tajik tg Language.TH Thai th Language.TK Turkmen tk Language.TL Tagalog tl Language.TR Turkish tr Language.TT Tatar tt Language.UK Ukrainian uk Language.UR Urdu ur Language.UZ Uzbek uz Language.VI Vietnamese vi Language.YI Yiddish yi Language.YO Yoruba yo Language.ZH Chinese zh Fal Wizper supports a range of languages and dialects. For the most accurate transcription, specify the correct language for your audio input. ​ Usage Example Copy Ask AI from pipecat.services.fal.stt import FalSTTService from pipecat.transcriptions.language import Language # Configure service stt = FalSTTService( api_key = "your-fal-api-key" , params = FalSTTService.InputParams( language = Language. EN , ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ Voice Activity Detection Integration This service inherits from SegmentedSTTService , which uses Voice Activity Detection (VAD) to identify speech segments for processing. This approach: Processes only actual speech, not silence or background noise Maintains a small audio buffer (default 1 second) to capture speech that occurs slightly before VAD detection Receives UserStartedSpeakingFrame and UserStoppedSpeakingFrame from a VAD component in the pipeline Only sends complete utterances to the API when speech has ended Ensure your transport includes a VAD component (like SileroVADAnalyzer ) to properly detect speech segments. ​ Metrics Support The service collects the following metrics: Processing duration API response time Success/failure rates ​ Notes Requires valid Fal API key Uses Fal’s Wizper model Requires VAD component in transport Processes complete utterances, not continuous audio Thread-safe processing ​ Error Handling The service handles common API errors including: Authentication errors API availability issues Invalid audio format Network connectivity issues API timeouts Errors are propagated through ErrorFrames with descriptive messages. Deepgram Gladia On this page Overview Installation Configuration Constructor Parameters InputParams Input Output Frames TranscriptionFrame ErrorFrame Methods Set Model Language Support Usage Example Voice Activity Detection Integration Metrics Support Notes Error Handling Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_fal_85baea9d.txt b/stt_fal_85baea9d.txt
new file mode 100644
index 0000000000000000000000000000000000000000..7090769872008bc0e8157762c9c1e090534a04f0
--- /dev/null
+++ b/stt_fal_85baea9d.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/fal#errorframe
+Title: Fal (Wizper) - Pipecat
+==================================================
+
+Fal (Wizper) - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text Fal (Wizper) Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview FalSTTService provides speech-to-text capabilities using Fal’s Wizper API. It offers high-quality transcription with minimal setup. The service uses Voice Activity Detection (VAD) to process only speech segments, optimizing API usage and improving response time. ​ Installation To use FalSTTService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[fal]" You’ll need to set up your Fal API key as an environment variable: FAL_KEY . You can obtain a Fal API key from the Fal platform . ​ Configuration ​ Constructor Parameters ​ api_key str Your Fal API key. If not provided, will use the FAL_KEY environment variable. ​ sample_rate int Audio sample rate in Hz. If not provided, uses the pipeline’s sample rate. ​ params InputParams Configuration parameters for the Wizper API. See InputParams below. ​ InputParams ​ language Language default: "Language.EN" Language of the audio input. Defaults to English. ​ task str default: "transcribe" Task to perform. Options are ‘transcribe’ or ‘translate’. ​ chunk_level str default: "segment" Level of chunking for the audio. Default is ‘segment’. ​ version str default: "3" Version of Wizper model to use. ​ Input The service processes audio data with the following requirements: PCM audio format 16-bit depth Single channel (mono) ​ Output Frames The service produces two types of frames during transcription: ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Detected language (if available) ​ ErrorFrame Generated when transcription errors occur, containing error details. ​ Methods ​ Set Model Copy Ask AI await service.set_model( "wizper-v3" ) See the STT base class methods for additional functionality. ​ Language Support Fal Wizper supports a wide range of languages. The service automatically maps Language enum values to the appropriate Wizper language codes. Language Code Description Wizper Code Language.AF Afrikaans af Language.AM Amharic am Language.AR Arabic ar Language.AS Assamese as Language.AZ Azerbaijani az Language.BA Bashkir ba Language.BE Belarusian be Language.BG Bulgarian bg Language.BN Bengali bn Language.BO Tibetan bo Language.BR Breton br Language.BS Bosnian bs Language.CA Catalan ca Language.CS Czech cs Language.CY Welsh cy Language.DA Danish da Language.DE German de Language.EL Greek el Language.EN English en Language.ES Spanish es Language.ET Estonian et Language.EU Basque eu Language.FA Persian fa Language.FI Finnish fi Language.FO Faroese fo Language.FR French fr Language.GL Galician gl Language.GU Gujarati gu Language.HA Hausa ha Language.HE Hebrew he Language.HI Hindi hi Language.HR Croatian hr Language.HT Haitian Creole ht Language.HU Hungarian hu Language.HY Armenian hy Language.ID Indonesian id Language.IS Icelandic is Language.IT Italian it Language.JA Japanese ja Language.JW Javanese jw Language.KA Georgian ka Language.KK Kazakh kk Language.KM Khmer km Language.KN Kannada kn Language.KO Korean ko Language.LA Latin la Language.LB Luxembourgish lb Language.LN Lingala ln Language.LO Lao lo Language.LT Lithuanian lt Language.LV Latvian lv Language.MG Malagasy mg Language.MI Maori mi Language.MK Macedonian mk Language.ML Malayalam ml Language.MN Mongolian mn Language.MR Marathi mr Language.MS Malay ms Language.MT Maltese mt Language.MY Burmese my Language.NE Nepali ne Language.NL Dutch nl Language.NN Norwegian Nynorsk nn Language.NO Norwegian no Language.OC Occitan oc Language.PA Punjabi pa Language.PL Polish pl Language.PS Pashto ps Language.PT Portuguese pt Language.RO Romanian ro Language.RU Russian ru Language.SA Sanskrit sa Language.SD Sindhi sd Language.SI Sinhala si Language.SK Slovak sk Language.SL Slovenian sl Language.SN Shona sn Language.SO Somali so Language.SQ Albanian sq Language.SR Serbian sr Language.SU Sundanese su Language.SV Swedish sv Language.SW Swahili sw Language.TA Tamil ta Language.TE Telugu te Language.TG Tajik tg Language.TH Thai th Language.TK Turkmen tk Language.TL Tagalog tl Language.TR Turkish tr Language.TT Tatar tt Language.UK Ukrainian uk Language.UR Urdu ur Language.UZ Uzbek uz Language.VI Vietnamese vi Language.YI Yiddish yi Language.YO Yoruba yo Language.ZH Chinese zh Fal Wizper supports a range of languages and dialects. For the most accurate transcription, specify the correct language for your audio input. ​ Usage Example Copy Ask AI from pipecat.services.fal.stt import FalSTTService from pipecat.transcriptions.language import Language # Configure service stt = FalSTTService( api_key = "your-fal-api-key" , params = FalSTTService.InputParams( language = Language. EN , ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ Voice Activity Detection Integration This service inherits from SegmentedSTTService , which uses Voice Activity Detection (VAD) to identify speech segments for processing. This approach: Processes only actual speech, not silence or background noise Maintains a small audio buffer (default 1 second) to capture speech that occurs slightly before VAD detection Receives UserStartedSpeakingFrame and UserStoppedSpeakingFrame from a VAD component in the pipeline Only sends complete utterances to the API when speech has ended Ensure your transport includes a VAD component (like SileroVADAnalyzer ) to properly detect speech segments. ​ Metrics Support The service collects the following metrics: Processing duration API response time Success/failure rates ​ Notes Requires valid Fal API key Uses Fal’s Wizper model Requires VAD component in transport Processes complete utterances, not continuous audio Thread-safe processing ​ Error Handling The service handles common API errors including: Authentication errors API availability issues Invalid audio format Network connectivity issues API timeouts Errors are propagated through ErrorFrames with descriptive messages. Deepgram Gladia On this page Overview Installation Configuration Constructor Parameters InputParams Input Output Frames TranscriptionFrame ErrorFrame Methods Set Model Language Support Usage Example Voice Activity Detection Integration Metrics Support Notes Error Handling Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_fal_b6629a4a.txt b/stt_fal_b6629a4a.txt
new file mode 100644
index 0000000000000000000000000000000000000000..d2e3844b81c8aae0cda5254246f8f55ace891afe
--- /dev/null
+++ b/stt_fal_b6629a4a.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/fal#param-user-id
+Title: Fal (Wizper) - Pipecat
+==================================================
+
+Fal (Wizper) - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text Fal (Wizper) Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview FalSTTService provides speech-to-text capabilities using Fal’s Wizper API. It offers high-quality transcription with minimal setup. The service uses Voice Activity Detection (VAD) to process only speech segments, optimizing API usage and improving response time. ​ Installation To use FalSTTService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[fal]" You’ll need to set up your Fal API key as an environment variable: FAL_KEY . You can obtain a Fal API key from the Fal platform . ​ Configuration ​ Constructor Parameters ​ api_key str Your Fal API key. If not provided, will use the FAL_KEY environment variable. ​ sample_rate int Audio sample rate in Hz. If not provided, uses the pipeline’s sample rate. ​ params InputParams Configuration parameters for the Wizper API. See InputParams below. ​ InputParams ​ language Language default: "Language.EN" Language of the audio input. Defaults to English. ​ task str default: "transcribe" Task to perform. Options are ‘transcribe’ or ‘translate’. ​ chunk_level str default: "segment" Level of chunking for the audio. Default is ‘segment’. ​ version str default: "3" Version of Wizper model to use. ​ Input The service processes audio data with the following requirements: PCM audio format 16-bit depth Single channel (mono) ​ Output Frames The service produces two types of frames during transcription: ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Detected language (if available) ​ ErrorFrame Generated when transcription errors occur, containing error details. ​ Methods ​ Set Model Copy Ask AI await service.set_model( "wizper-v3" ) See the STT base class methods for additional functionality. ​ Language Support Fal Wizper supports a wide range of languages. The service automatically maps Language enum values to the appropriate Wizper language codes. Language Code Description Wizper Code Language.AF Afrikaans af Language.AM Amharic am Language.AR Arabic ar Language.AS Assamese as Language.AZ Azerbaijani az Language.BA Bashkir ba Language.BE Belarusian be Language.BG Bulgarian bg Language.BN Bengali bn Language.BO Tibetan bo Language.BR Breton br Language.BS Bosnian bs Language.CA Catalan ca Language.CS Czech cs Language.CY Welsh cy Language.DA Danish da Language.DE German de Language.EL Greek el Language.EN English en Language.ES Spanish es Language.ET Estonian et Language.EU Basque eu Language.FA Persian fa Language.FI Finnish fi Language.FO Faroese fo Language.FR French fr Language.GL Galician gl Language.GU Gujarati gu Language.HA Hausa ha Language.HE Hebrew he Language.HI Hindi hi Language.HR Croatian hr Language.HT Haitian Creole ht Language.HU Hungarian hu Language.HY Armenian hy Language.ID Indonesian id Language.IS Icelandic is Language.IT Italian it Language.JA Japanese ja Language.JW Javanese jw Language.KA Georgian ka Language.KK Kazakh kk Language.KM Khmer km Language.KN Kannada kn Language.KO Korean ko Language.LA Latin la Language.LB Luxembourgish lb Language.LN Lingala ln Language.LO Lao lo Language.LT Lithuanian lt Language.LV Latvian lv Language.MG Malagasy mg Language.MI Maori mi Language.MK Macedonian mk Language.ML Malayalam ml Language.MN Mongolian mn Language.MR Marathi mr Language.MS Malay ms Language.MT Maltese mt Language.MY Burmese my Language.NE Nepali ne Language.NL Dutch nl Language.NN Norwegian Nynorsk nn Language.NO Norwegian no Language.OC Occitan oc Language.PA Punjabi pa Language.PL Polish pl Language.PS Pashto ps Language.PT Portuguese pt Language.RO Romanian ro Language.RU Russian ru Language.SA Sanskrit sa Language.SD Sindhi sd Language.SI Sinhala si Language.SK Slovak sk Language.SL Slovenian sl Language.SN Shona sn Language.SO Somali so Language.SQ Albanian sq Language.SR Serbian sr Language.SU Sundanese su Language.SV Swedish sv Language.SW Swahili sw Language.TA Tamil ta Language.TE Telugu te Language.TG Tajik tg Language.TH Thai th Language.TK Turkmen tk Language.TL Tagalog tl Language.TR Turkish tr Language.TT Tatar tt Language.UK Ukrainian uk Language.UR Urdu ur Language.UZ Uzbek uz Language.VI Vietnamese vi Language.YI Yiddish yi Language.YO Yoruba yo Language.ZH Chinese zh Fal Wizper supports a range of languages and dialects. For the most accurate transcription, specify the correct language for your audio input. ​ Usage Example Copy Ask AI from pipecat.services.fal.stt import FalSTTService from pipecat.transcriptions.language import Language # Configure service stt = FalSTTService( api_key = "your-fal-api-key" , params = FalSTTService.InputParams( language = Language. EN , ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ Voice Activity Detection Integration This service inherits from SegmentedSTTService , which uses Voice Activity Detection (VAD) to identify speech segments for processing. This approach: Processes only actual speech, not silence or background noise Maintains a small audio buffer (default 1 second) to capture speech that occurs slightly before VAD detection Receives UserStartedSpeakingFrame and UserStoppedSpeakingFrame from a VAD component in the pipeline Only sends complete utterances to the API when speech has ended Ensure your transport includes a VAD component (like SileroVADAnalyzer ) to properly detect speech segments. ​ Metrics Support The service collects the following metrics: Processing duration API response time Success/failure rates ​ Notes Requires valid Fal API key Uses Fal’s Wizper model Requires VAD component in transport Processes complete utterances, not continuous audio Thread-safe processing ​ Error Handling The service handles common API errors including: Authentication errors API availability issues Invalid audio format Network connectivity issues API timeouts Errors are propagated through ErrorFrames with descriptive messages. Deepgram Gladia On this page Overview Installation Configuration Constructor Parameters InputParams Input Output Frames TranscriptionFrame ErrorFrame Methods Set Model Language Support Usage Example Voice Activity Detection Integration Metrics Support Notes Error Handling Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_fal_bf9feeb3.txt b/stt_fal_bf9feeb3.txt
new file mode 100644
index 0000000000000000000000000000000000000000..ba583d2b27eb35769707721b5d583e3aa096c390
--- /dev/null
+++ b/stt_fal_bf9feeb3.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/fal#inputparams
+Title: Fal (Wizper) - Pipecat
+==================================================
+
+Fal (Wizper) - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text Fal (Wizper) Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview FalSTTService provides speech-to-text capabilities using Fal’s Wizper API. It offers high-quality transcription with minimal setup. The service uses Voice Activity Detection (VAD) to process only speech segments, optimizing API usage and improving response time. ​ Installation To use FalSTTService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[fal]" You’ll need to set up your Fal API key as an environment variable: FAL_KEY . You can obtain a Fal API key from the Fal platform . ​ Configuration ​ Constructor Parameters ​ api_key str Your Fal API key. If not provided, will use the FAL_KEY environment variable. ​ sample_rate int Audio sample rate in Hz. If not provided, uses the pipeline’s sample rate. ​ params InputParams Configuration parameters for the Wizper API. See InputParams below. ​ InputParams ​ language Language default: "Language.EN" Language of the audio input. Defaults to English. ​ task str default: "transcribe" Task to perform. Options are ‘transcribe’ or ‘translate’. ​ chunk_level str default: "segment" Level of chunking for the audio. Default is ‘segment’. ​ version str default: "3" Version of Wizper model to use. ​ Input The service processes audio data with the following requirements: PCM audio format 16-bit depth Single channel (mono) ​ Output Frames The service produces two types of frames during transcription: ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Detected language (if available) ​ ErrorFrame Generated when transcription errors occur, containing error details. ​ Methods ​ Set Model Copy Ask AI await service.set_model( "wizper-v3" ) See the STT base class methods for additional functionality. ​ Language Support Fal Wizper supports a wide range of languages. The service automatically maps Language enum values to the appropriate Wizper language codes. Language Code Description Wizper Code Language.AF Afrikaans af Language.AM Amharic am Language.AR Arabic ar Language.AS Assamese as Language.AZ Azerbaijani az Language.BA Bashkir ba Language.BE Belarusian be Language.BG Bulgarian bg Language.BN Bengali bn Language.BO Tibetan bo Language.BR Breton br Language.BS Bosnian bs Language.CA Catalan ca Language.CS Czech cs Language.CY Welsh cy Language.DA Danish da Language.DE German de Language.EL Greek el Language.EN English en Language.ES Spanish es Language.ET Estonian et Language.EU Basque eu Language.FA Persian fa Language.FI Finnish fi Language.FO Faroese fo Language.FR French fr Language.GL Galician gl Language.GU Gujarati gu Language.HA Hausa ha Language.HE Hebrew he Language.HI Hindi hi Language.HR Croatian hr Language.HT Haitian Creole ht Language.HU Hungarian hu Language.HY Armenian hy Language.ID Indonesian id Language.IS Icelandic is Language.IT Italian it Language.JA Japanese ja Language.JW Javanese jw Language.KA Georgian ka Language.KK Kazakh kk Language.KM Khmer km Language.KN Kannada kn Language.KO Korean ko Language.LA Latin la Language.LB Luxembourgish lb Language.LN Lingala ln Language.LO Lao lo Language.LT Lithuanian lt Language.LV Latvian lv Language.MG Malagasy mg Language.MI Maori mi Language.MK Macedonian mk Language.ML Malayalam ml Language.MN Mongolian mn Language.MR Marathi mr Language.MS Malay ms Language.MT Maltese mt Language.MY Burmese my Language.NE Nepali ne Language.NL Dutch nl Language.NN Norwegian Nynorsk nn Language.NO Norwegian no Language.OC Occitan oc Language.PA Punjabi pa Language.PL Polish pl Language.PS Pashto ps Language.PT Portuguese pt Language.RO Romanian ro Language.RU Russian ru Language.SA Sanskrit sa Language.SD Sindhi sd Language.SI Sinhala si Language.SK Slovak sk Language.SL Slovenian sl Language.SN Shona sn Language.SO Somali so Language.SQ Albanian sq Language.SR Serbian sr Language.SU Sundanese su Language.SV Swedish sv Language.SW Swahili sw Language.TA Tamil ta Language.TE Telugu te Language.TG Tajik tg Language.TH Thai th Language.TK Turkmen tk Language.TL Tagalog tl Language.TR Turkish tr Language.TT Tatar tt Language.UK Ukrainian uk Language.UR Urdu ur Language.UZ Uzbek uz Language.VI Vietnamese vi Language.YI Yiddish yi Language.YO Yoruba yo Language.ZH Chinese zh Fal Wizper supports a range of languages and dialects. For the most accurate transcription, specify the correct language for your audio input. ​ Usage Example Copy Ask AI from pipecat.services.fal.stt import FalSTTService from pipecat.transcriptions.language import Language # Configure service stt = FalSTTService( api_key = "your-fal-api-key" , params = FalSTTService.InputParams( language = Language. EN , ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ Voice Activity Detection Integration This service inherits from SegmentedSTTService , which uses Voice Activity Detection (VAD) to identify speech segments for processing. This approach: Processes only actual speech, not silence or background noise Maintains a small audio buffer (default 1 second) to capture speech that occurs slightly before VAD detection Receives UserStartedSpeakingFrame and UserStoppedSpeakingFrame from a VAD component in the pipeline Only sends complete utterances to the API when speech has ended Ensure your transport includes a VAD component (like SileroVADAnalyzer ) to properly detect speech segments. ​ Metrics Support The service collects the following metrics: Processing duration API response time Success/failure rates ​ Notes Requires valid Fal API key Uses Fal’s Wizper model Requires VAD component in transport Processes complete utterances, not continuous audio Thread-safe processing ​ Error Handling The service handles common API errors including: Authentication errors API availability issues Invalid audio format Network connectivity issues API timeouts Errors are propagated through ErrorFrames with descriptive messages. Deepgram Gladia On this page Overview Installation Configuration Constructor Parameters InputParams Input Output Frames TranscriptionFrame ErrorFrame Methods Set Model Language Support Usage Example Voice Activity Detection Integration Metrics Support Notes Error Handling Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_fal_f29debe2.txt b/stt_fal_f29debe2.txt
new file mode 100644
index 0000000000000000000000000000000000000000..271ec52ec00a973f1982411d3407fcd36865abfd
--- /dev/null
+++ b/stt_fal_f29debe2.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/fal#language-support
+Title: Fal (Wizper) - Pipecat
+==================================================
+
+Fal (Wizper) - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text Fal (Wizper) Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview FalSTTService provides speech-to-text capabilities using Fal’s Wizper API. It offers high-quality transcription with minimal setup. The service uses Voice Activity Detection (VAD) to process only speech segments, optimizing API usage and improving response time. ​ Installation To use FalSTTService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[fal]" You’ll need to set up your Fal API key as an environment variable: FAL_KEY . You can obtain a Fal API key from the Fal platform . ​ Configuration ​ Constructor Parameters ​ api_key str Your Fal API key. If not provided, will use the FAL_KEY environment variable. ​ sample_rate int Audio sample rate in Hz. If not provided, uses the pipeline’s sample rate. ​ params InputParams Configuration parameters for the Wizper API. See InputParams below. ​ InputParams ​ language Language default: "Language.EN" Language of the audio input. Defaults to English. ​ task str default: "transcribe" Task to perform. Options are ‘transcribe’ or ‘translate’. ​ chunk_level str default: "segment" Level of chunking for the audio. Default is ‘segment’. ​ version str default: "3" Version of Wizper model to use. ​ Input The service processes audio data with the following requirements: PCM audio format 16-bit depth Single channel (mono) ​ Output Frames The service produces two types of frames during transcription: ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Detected language (if available) ​ ErrorFrame Generated when transcription errors occur, containing error details. ​ Methods ​ Set Model Copy Ask AI await service.set_model( "wizper-v3" ) See the STT base class methods for additional functionality. ​ Language Support Fal Wizper supports a wide range of languages. The service automatically maps Language enum values to the appropriate Wizper language codes. Language Code Description Wizper Code Language.AF Afrikaans af Language.AM Amharic am Language.AR Arabic ar Language.AS Assamese as Language.AZ Azerbaijani az Language.BA Bashkir ba Language.BE Belarusian be Language.BG Bulgarian bg Language.BN Bengali bn Language.BO Tibetan bo Language.BR Breton br Language.BS Bosnian bs Language.CA Catalan ca Language.CS Czech cs Language.CY Welsh cy Language.DA Danish da Language.DE German de Language.EL Greek el Language.EN English en Language.ES Spanish es Language.ET Estonian et Language.EU Basque eu Language.FA Persian fa Language.FI Finnish fi Language.FO Faroese fo Language.FR French fr Language.GL Galician gl Language.GU Gujarati gu Language.HA Hausa ha Language.HE Hebrew he Language.HI Hindi hi Language.HR Croatian hr Language.HT Haitian Creole ht Language.HU Hungarian hu Language.HY Armenian hy Language.ID Indonesian id Language.IS Icelandic is Language.IT Italian it Language.JA Japanese ja Language.JW Javanese jw Language.KA Georgian ka Language.KK Kazakh kk Language.KM Khmer km Language.KN Kannada kn Language.KO Korean ko Language.LA Latin la Language.LB Luxembourgish lb Language.LN Lingala ln Language.LO Lao lo Language.LT Lithuanian lt Language.LV Latvian lv Language.MG Malagasy mg Language.MI Maori mi Language.MK Macedonian mk Language.ML Malayalam ml Language.MN Mongolian mn Language.MR Marathi mr Language.MS Malay ms Language.MT Maltese mt Language.MY Burmese my Language.NE Nepali ne Language.NL Dutch nl Language.NN Norwegian Nynorsk nn Language.NO Norwegian no Language.OC Occitan oc Language.PA Punjabi pa Language.PL Polish pl Language.PS Pashto ps Language.PT Portuguese pt Language.RO Romanian ro Language.RU Russian ru Language.SA Sanskrit sa Language.SD Sindhi sd Language.SI Sinhala si Language.SK Slovak sk Language.SL Slovenian sl Language.SN Shona sn Language.SO Somali so Language.SQ Albanian sq Language.SR Serbian sr Language.SU Sundanese su Language.SV Swedish sv Language.SW Swahili sw Language.TA Tamil ta Language.TE Telugu te Language.TG Tajik tg Language.TH Thai th Language.TK Turkmen tk Language.TL Tagalog tl Language.TR Turkish tr Language.TT Tatar tt Language.UK Ukrainian uk Language.UR Urdu ur Language.UZ Uzbek uz Language.VI Vietnamese vi Language.YI Yiddish yi Language.YO Yoruba yo Language.ZH Chinese zh Fal Wizper supports a range of languages and dialects. For the most accurate transcription, specify the correct language for your audio input. ​ Usage Example Copy Ask AI from pipecat.services.fal.stt import FalSTTService from pipecat.transcriptions.language import Language # Configure service stt = FalSTTService( api_key = "your-fal-api-key" , params = FalSTTService.InputParams( language = Language. EN , ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ Voice Activity Detection Integration This service inherits from SegmentedSTTService , which uses Voice Activity Detection (VAD) to identify speech segments for processing. This approach: Processes only actual speech, not silence or background noise Maintains a small audio buffer (default 1 second) to capture speech that occurs slightly before VAD detection Receives UserStartedSpeakingFrame and UserStoppedSpeakingFrame from a VAD component in the pipeline Only sends complete utterances to the API when speech has ended Ensure your transport includes a VAD component (like SileroVADAnalyzer ) to properly detect speech segments. ​ Metrics Support The service collects the following metrics: Processing duration API response time Success/failure rates ​ Notes Requires valid Fal API key Uses Fal’s Wizper model Requires VAD component in transport Processes complete utterances, not continuous audio Thread-safe processing ​ Error Handling The service handles common API errors including: Authentication errors API availability issues Invalid audio format Network connectivity issues API timeouts Errors are propagated through ErrorFrames with descriptive messages. Deepgram Gladia On this page Overview Installation Configuration Constructor Parameters InputParams Input Output Frames TranscriptionFrame ErrorFrame Methods Set Model Language Support Usage Example Voice Activity Detection Integration Metrics Support Notes Error Handling Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_gladia_010efffd.txt b/stt_gladia_010efffd.txt
new file mode 100644
index 0000000000000000000000000000000000000000..a40484cf1d60f98dcb27ffc6263b5be618f6c7b7
--- /dev/null
+++ b/stt_gladia_010efffd.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/gladia#multi-language-support
+Title: Gladia - Pipecat
+==================================================
+
+Gladia - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text Gladia Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview GladiaSTTService is a speech-to-text (STT) service that integrates with Gladia’s API to provide real-time transcription capabilities. It processes audio input and produces transcription frames in real-time with support for multiple languages, custom vocabulary, and various processing options. ​ Installation To use GladiaSTTService , you need to install the Gladia dependencies: Copy Ask AI pip install "pipecat-ai[gladia]" You’ll also need to set up your Gladia API key as an environment variable: GLADIA_API_KEY . ​ Configuration ​ Service Parameters ​ api_key string required Your Gladia API key for authentication ​ url string default: "https://api.gladia.io/v2/live" Gladia API endpoint URL ​ confidence float default: "0.5" Minimum confidence threshold to create interim and final transcriptions. Values range from 0 to 1. ​ sample_rate integer default: "None" Audio sample rate in Hz ​ model string default: "solaria-1" Model to use for transcription. Options include solaria-1 solaria-mini-1 fast accurate See Gladia’s docs for the latest supported models. ​ params GladiaInputParams default: "GladiaInputParams()" Additional configuration parameters for the service ​ GladiaInputParams ​ encoding string default: "wav/pcm" Audio encoding format ​ bit_depth integer default: "16" Audio bit depth ​ channels integer default: "1" Number of audio channels ​ custom_metadata Dict[str, Any] Additional metadata to include with requests ​ endpointing float Silence duration in seconds to mark end of speech ​ maximum_duration_without_endpointing integer default: "10" Maximum utterance duration without silence ​ language Language deprecated Primary language for transcription. Deprecated: use language_config instead. ​ language_config LanguageConfig Detailed language configuration ​ pre_processing PreProcessingConfig Audio pre-processing options ​ realtime_processing RealtimeProcessingConfig Real-time processing features ​ messages_config MessagesConfig WebSocket message filtering options ​ LanguageConfig ​ languages List[str] Specify language(s) for transcription. If one language is set, it will be used for all transcription. If multiple languages are provided or none, language will be auto-detected by the model. ​ code_switching boolean default: "false" If true, language will be auto-detected on each utterance. Otherwise, language will be auto-detected on first utterance and then used for the rest of the transcription. If one language is set, this option will be ignored. ​ PreProcessingConfig ​ speech_threshold float default: "0.8" Sensitivity configuration for Speech Threshold. A value close to 1 will apply stricter thresholds, making it less likely to detect background sounds as speech. Must be between 0 and 1. ​ CustomVocabularyConfig ​ vocabulary List[Union[str, CustomVocabularyItem]] required Specific vocabulary list to feed the transcription model with. Can be a list of strings or CustomVocabularyItem objects. ​ default_intensity float Default intensity for the custom vocabulary. Must be between 0 and 1. ​ CustomSpellingConfig ​ spelling_dictionary Dict[str, List[str]] required The list of spelling rules applied on the audio transcription. Keys are the correct spellings and values are lists of phonetic variations. ​ TranslationConfig ​ target_languages List[str] required The target language(s) in ISO639-1 format (e.g., “en”, “fr”, “es”) ​ model string default: "base" Translation model to use. Options: “base” or “enhanced” ​ match_original_utterances boolean default: "true" Align translated utterances with the original ones ​ RealtimeProcessingConfig ​ words_accurate_timestamps boolean Whether to provide per-word timestamps ​ custom_vocabulary boolean Whether to enable custom vocabulary ​ custom_vocabulary_config CustomVocabularyConfig Custom vocabulary configuration ​ custom_spelling boolean Whether to enable custom spelling ​ custom_spelling_config CustomSpellingConfig Custom spelling configuration ​ translation boolean Whether to enable translation ​ translation_config TranslationConfig Translation configuration ​ named_entity_recognition boolean Whether to enable named entity recognition ​ sentiment_analysis boolean Whether to enable sentiment analysis ​ MessagesConfig ​ receive_partial_transcripts boolean default: "true" If true, partial utterances will be sent via WebSocket ​ receive_final_transcripts boolean default: "true" If true, final utterances will be sent via WebSocket ​ receive_speech_events boolean default: "true" If true, begin and end speech events will be sent via WebSocket ​ receive_pre_processing_events boolean default: "true" If true, pre-processing events will be sent via WebSocket ​ receive_realtime_processing_events boolean default: "true" If true, realtime processing events will be sent via WebSocket ​ receive_post_processing_events boolean default: "true" If true, post-processing events will be sent via WebSocket ​ receive_acknowledgments boolean default: "true" If true, acknowledgments will be sent via WebSocket ​ receive_errors boolean default: "true" If true, errors will be sent via WebSocket ​ receive_lifecycle_events boolean default: "false" If true, lifecycle events will be sent via WebSocket ​ Input The service processes raw audio data with the following requirements: PCM audio format 16-bit depth 16kHz sample rate (default) Single channel (mono) ​ Output The service produces two types of frames during transcription: ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Transcription language ​ InterimTranscriptionFrame Generated during ongoing speech, containing the same fields as TranscriptionFrame but with preliminary results. ​ ErrorFrame Generated when transcription errors occur, containing error details. ​ Methods See the STT base class methods for additional functionality. ​ Language Setting Copy Ask AI await service.set_language(Language. FR ) ​ Language Support Gladia STT supports a wide range of languages. Here’s a partial list: Language Code Description Service Code Language.AF Afrikaans af Language.AM Amharic am Language.AR Arabic ar Language.AS Assamese as Language.AZ Azerbaijani az Language.BA Bashkir ba Language.BE Belarusian be Language.BG Bulgarian bg Language.BN Bengali bn Language.BO Tibetan bo Language.BR Breton br Language.BS Bosnian bs Language.CA Catalan ca Language.CS Czech cs Language.CY Welsh cy Language.DA Danish da Language.DE German de Language.EL Greek el Language.EN English en Language.ES Spanish es Language.ET Estonian et Language.EU Basque eu Language.FA Persian fa Language.FI Finnish fi Language.FO Faroese fo Language.FR French fr Language.GL Galician gl Language.GU Gujarati gu Language.HA Hausa ha Language.HAW Hawaiian haw Language.HE Hebrew he Language.HI Hindi hi Language.HR Croatian hr Language.HT Haitian Creole ht Language.HU Hungarian hu Language.HY Armenian hy Language.ID Indonesian id Language.IS Icelandic is Language.IT Italian it Language.JA Japanese ja Language.JV Javanese jv Language.KA Georgian ka Language.KK Kazakh kk Language.KM Khmer km Language.KN Kannada kn Language.KO Korean ko Language.LA Latin la Language.LB Luxembourgish lb Language.LN Lingala ln Language.LO Lao lo Language.LT Lithuanian lt Language.LV Latvian lv Language.MG Malagasy mg Language.MI Maori mi Language.MK Macedonian mk Language.ML Malayalam ml Language.MN Mongolian mn Language.MR Marathi mr Language.MS Malay ms Language.MT Maltese mt Language.MY_MR Burmese mymr Language.NE Nepali ne Language.NL Dutch nl Language.NN Norwegian (Nynorsk) nn Language.NO Norwegian no Language.OC Occitan oc Language.PA Punjabi pa Language.PL Polish pl Language.PS Pashto ps Language.PT Portuguese pt Language.RO Romanian ro Language.RU Russian ru Language.SA Sanskrit sa Language.SD Sindhi sd Language.SI Sinhala si Language.SK Slovak sk Language.SL Slovenian sl Language.SN Shona sn Language.SO Somali so Language.SQ Albanian sq Language.SR Serbian sr Language.SU Sundanese su Language.SV Swedish sv Language.SW Swahili sw Language.TA Tamil ta Language.TE Telugu te Language.TG Tajik tg Language.TH Thai th Language.TK Turkmen tk Language.TL Tagalog tl Language.TR Turkish tr Language.TT Tatar tt Language.UK Ukrainian uk Language.UR Urdu ur Language.UZ Uzbek uz Language.VI Vietnamese vi Language.YI Yiddish yi Language.YO Yoruba yo Language.ZH Chinese zh For a complete list of supported languages, refer to Gladia’s documentation . ​ Advanced Features ​ Custom Vocabulary You can provide custom vocabulary items with bias intensity: Copy Ask AI from pipecat.services.gladia.config import CustomVocabularyItem, CustomVocabularyConfig, RealtimeProcessingConfig custom_vocab = CustomVocabularyConfig( vocabulary = [ CustomVocabularyItem( value = "Pipecat" , intensity = 0.8 ), CustomVocabularyItem( value = "Daily" , intensity = 0.7 ), ], default_intensity = 0.5 ) realtime_config = RealtimeProcessingConfig( custom_vocabulary = True , custom_vocabulary_config = custom_vocab ) ​ Translation Enable real-time translation: Copy Ask AI from pipecat.services.gladia.config import TranslationConfig, RealtimeProcessingConfig translation_config = TranslationConfig( target_languages = [ "fr" , "es" , "de" ], model = "enhanced" , match_original_utterances = True ) realtime_config = RealtimeProcessingConfig( translation = True , translation_config = translation_config ) ​ Multi-language Support Configure multiple languages with automatic language switching: Copy Ask AI from pipecat.services.gladia.config import LanguageConfig, GladiaInputParams language_config = LanguageConfig( languages = [ "en" , "fr" , "es" ], code_switching = True ) params = GladiaInputParams( language_config = language_config ) ​ Usage Example Copy Ask AI from pipecat.pipeline.pipeline import Pipeline from pipecat.services.gladia.stt import GladiaSTTService from pipecat.services.gladia.config import ( GladiaInputParams, LanguageConfig, RealtimeProcessingConfig ) from pipecat.transcriptions.language import Language # Configure the service stt = GladiaSTTService( api_key = "your-api-key" , model = "solaria-1" , params = GladiaInputParams( language_config = LanguageConfig( languages = [Language. EN , Language. FR ], code_switching = True ), ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ Frame Flow ​ Metrics Support The service collects processing metrics: Time to First Byte (TTFB) Processing duration Connection status ​ Notes Audio input must be in PCM format Transcription frames are only generated when confidence threshold is met Service automatically handles websocket connections and cleanup Real-time processing occurs in parallel for natural conversation flow Fal (Wizper) Google On this page Overview Installation Configuration Service Parameters GladiaInputParams LanguageConfig PreProcessingConfig CustomVocabularyConfig CustomSpellingConfig TranslationConfig RealtimeProcessingConfig MessagesConfig Input Output TranscriptionFrame InterimTranscriptionFrame ErrorFrame Methods Language Setting Language Support Advanced Features Custom Vocabulary Translation Multi-language Support Usage Example Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_gladia_1187a821.txt b/stt_gladia_1187a821.txt
new file mode 100644
index 0000000000000000000000000000000000000000..63e0315d02213e1269a4028b886e6d1c7ab4d2be
--- /dev/null
+++ b/stt_gladia_1187a821.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/gladia#param-custom-spelling-config
+Title: Gladia - Pipecat
+==================================================
+
+Gladia - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text Gladia Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview GladiaSTTService is a speech-to-text (STT) service that integrates with Gladia’s API to provide real-time transcription capabilities. It processes audio input and produces transcription frames in real-time with support for multiple languages, custom vocabulary, and various processing options. ​ Installation To use GladiaSTTService , you need to install the Gladia dependencies: Copy Ask AI pip install "pipecat-ai[gladia]" You’ll also need to set up your Gladia API key as an environment variable: GLADIA_API_KEY . ​ Configuration ​ Service Parameters ​ api_key string required Your Gladia API key for authentication ​ url string default: "https://api.gladia.io/v2/live" Gladia API endpoint URL ​ confidence float default: "0.5" Minimum confidence threshold to create interim and final transcriptions. Values range from 0 to 1. ​ sample_rate integer default: "None" Audio sample rate in Hz ​ model string default: "solaria-1" Model to use for transcription. Options include solaria-1 solaria-mini-1 fast accurate See Gladia’s docs for the latest supported models. ​ params GladiaInputParams default: "GladiaInputParams()" Additional configuration parameters for the service ​ GladiaInputParams ​ encoding string default: "wav/pcm" Audio encoding format ​ bit_depth integer default: "16" Audio bit depth ​ channels integer default: "1" Number of audio channels ​ custom_metadata Dict[str, Any] Additional metadata to include with requests ​ endpointing float Silence duration in seconds to mark end of speech ​ maximum_duration_without_endpointing integer default: "10" Maximum utterance duration without silence ​ language Language deprecated Primary language for transcription. Deprecated: use language_config instead. ​ language_config LanguageConfig Detailed language configuration ​ pre_processing PreProcessingConfig Audio pre-processing options ​ realtime_processing RealtimeProcessingConfig Real-time processing features ​ messages_config MessagesConfig WebSocket message filtering options ​ LanguageConfig ​ languages List[str] Specify language(s) for transcription. If one language is set, it will be used for all transcription. If multiple languages are provided or none, language will be auto-detected by the model. ​ code_switching boolean default: "false" If true, language will be auto-detected on each utterance. Otherwise, language will be auto-detected on first utterance and then used for the rest of the transcription. If one language is set, this option will be ignored. ​ PreProcessingConfig ​ speech_threshold float default: "0.8" Sensitivity configuration for Speech Threshold. A value close to 1 will apply stricter thresholds, making it less likely to detect background sounds as speech. Must be between 0 and 1. ​ CustomVocabularyConfig ​ vocabulary List[Union[str, CustomVocabularyItem]] required Specific vocabulary list to feed the transcription model with. Can be a list of strings or CustomVocabularyItem objects. ​ default_intensity float Default intensity for the custom vocabulary. Must be between 0 and 1. ​ CustomSpellingConfig ​ spelling_dictionary Dict[str, List[str]] required The list of spelling rules applied on the audio transcription. Keys are the correct spellings and values are lists of phonetic variations. ​ TranslationConfig ​ target_languages List[str] required The target language(s) in ISO639-1 format (e.g., “en”, “fr”, “es”) ​ model string default: "base" Translation model to use. Options: “base” or “enhanced” ​ match_original_utterances boolean default: "true" Align translated utterances with the original ones ​ RealtimeProcessingConfig ​ words_accurate_timestamps boolean Whether to provide per-word timestamps ​ custom_vocabulary boolean Whether to enable custom vocabulary ​ custom_vocabulary_config CustomVocabularyConfig Custom vocabulary configuration ​ custom_spelling boolean Whether to enable custom spelling ​ custom_spelling_config CustomSpellingConfig Custom spelling configuration ​ translation boolean Whether to enable translation ​ translation_config TranslationConfig Translation configuration ​ named_entity_recognition boolean Whether to enable named entity recognition ​ sentiment_analysis boolean Whether to enable sentiment analysis ​ MessagesConfig ​ receive_partial_transcripts boolean default: "true" If true, partial utterances will be sent via WebSocket ​ receive_final_transcripts boolean default: "true" If true, final utterances will be sent via WebSocket ​ receive_speech_events boolean default: "true" If true, begin and end speech events will be sent via WebSocket ​ receive_pre_processing_events boolean default: "true" If true, pre-processing events will be sent via WebSocket ​ receive_realtime_processing_events boolean default: "true" If true, realtime processing events will be sent via WebSocket ​ receive_post_processing_events boolean default: "true" If true, post-processing events will be sent via WebSocket ​ receive_acknowledgments boolean default: "true" If true, acknowledgments will be sent via WebSocket ​ receive_errors boolean default: "true" If true, errors will be sent via WebSocket ​ receive_lifecycle_events boolean default: "false" If true, lifecycle events will be sent via WebSocket ​ Input The service processes raw audio data with the following requirements: PCM audio format 16-bit depth 16kHz sample rate (default) Single channel (mono) ​ Output The service produces two types of frames during transcription: ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Transcription language ​ InterimTranscriptionFrame Generated during ongoing speech, containing the same fields as TranscriptionFrame but with preliminary results. ​ ErrorFrame Generated when transcription errors occur, containing error details. ​ Methods See the STT base class methods for additional functionality. ​ Language Setting Copy Ask AI await service.set_language(Language. FR ) ​ Language Support Gladia STT supports a wide range of languages. Here’s a partial list: Language Code Description Service Code Language.AF Afrikaans af Language.AM Amharic am Language.AR Arabic ar Language.AS Assamese as Language.AZ Azerbaijani az Language.BA Bashkir ba Language.BE Belarusian be Language.BG Bulgarian bg Language.BN Bengali bn Language.BO Tibetan bo Language.BR Breton br Language.BS Bosnian bs Language.CA Catalan ca Language.CS Czech cs Language.CY Welsh cy Language.DA Danish da Language.DE German de Language.EL Greek el Language.EN English en Language.ES Spanish es Language.ET Estonian et Language.EU Basque eu Language.FA Persian fa Language.FI Finnish fi Language.FO Faroese fo Language.FR French fr Language.GL Galician gl Language.GU Gujarati gu Language.HA Hausa ha Language.HAW Hawaiian haw Language.HE Hebrew he Language.HI Hindi hi Language.HR Croatian hr Language.HT Haitian Creole ht Language.HU Hungarian hu Language.HY Armenian hy Language.ID Indonesian id Language.IS Icelandic is Language.IT Italian it Language.JA Japanese ja Language.JV Javanese jv Language.KA Georgian ka Language.KK Kazakh kk Language.KM Khmer km Language.KN Kannada kn Language.KO Korean ko Language.LA Latin la Language.LB Luxembourgish lb Language.LN Lingala ln Language.LO Lao lo Language.LT Lithuanian lt Language.LV Latvian lv Language.MG Malagasy mg Language.MI Maori mi Language.MK Macedonian mk Language.ML Malayalam ml Language.MN Mongolian mn Language.MR Marathi mr Language.MS Malay ms Language.MT Maltese mt Language.MY_MR Burmese mymr Language.NE Nepali ne Language.NL Dutch nl Language.NN Norwegian (Nynorsk) nn Language.NO Norwegian no Language.OC Occitan oc Language.PA Punjabi pa Language.PL Polish pl Language.PS Pashto ps Language.PT Portuguese pt Language.RO Romanian ro Language.RU Russian ru Language.SA Sanskrit sa Language.SD Sindhi sd Language.SI Sinhala si Language.SK Slovak sk Language.SL Slovenian sl Language.SN Shona sn Language.SO Somali so Language.SQ Albanian sq Language.SR Serbian sr Language.SU Sundanese su Language.SV Swedish sv Language.SW Swahili sw Language.TA Tamil ta Language.TE Telugu te Language.TG Tajik tg Language.TH Thai th Language.TK Turkmen tk Language.TL Tagalog tl Language.TR Turkish tr Language.TT Tatar tt Language.UK Ukrainian uk Language.UR Urdu ur Language.UZ Uzbek uz Language.VI Vietnamese vi Language.YI Yiddish yi Language.YO Yoruba yo Language.ZH Chinese zh For a complete list of supported languages, refer to Gladia’s documentation . ​ Advanced Features ​ Custom Vocabulary You can provide custom vocabulary items with bias intensity: Copy Ask AI from pipecat.services.gladia.config import CustomVocabularyItem, CustomVocabularyConfig, RealtimeProcessingConfig custom_vocab = CustomVocabularyConfig( vocabulary = [ CustomVocabularyItem( value = "Pipecat" , intensity = 0.8 ), CustomVocabularyItem( value = "Daily" , intensity = 0.7 ), ], default_intensity = 0.5 ) realtime_config = RealtimeProcessingConfig( custom_vocabulary = True , custom_vocabulary_config = custom_vocab ) ​ Translation Enable real-time translation: Copy Ask AI from pipecat.services.gladia.config import TranslationConfig, RealtimeProcessingConfig translation_config = TranslationConfig( target_languages = [ "fr" , "es" , "de" ], model = "enhanced" , match_original_utterances = True ) realtime_config = RealtimeProcessingConfig( translation = True , translation_config = translation_config ) ​ Multi-language Support Configure multiple languages with automatic language switching: Copy Ask AI from pipecat.services.gladia.config import LanguageConfig, GladiaInputParams language_config = LanguageConfig( languages = [ "en" , "fr" , "es" ], code_switching = True ) params = GladiaInputParams( language_config = language_config ) ​ Usage Example Copy Ask AI from pipecat.pipeline.pipeline import Pipeline from pipecat.services.gladia.stt import GladiaSTTService from pipecat.services.gladia.config import ( GladiaInputParams, LanguageConfig, RealtimeProcessingConfig ) from pipecat.transcriptions.language import Language # Configure the service stt = GladiaSTTService( api_key = "your-api-key" , model = "solaria-1" , params = GladiaInputParams( language_config = LanguageConfig( languages = [Language. EN , Language. FR ], code_switching = True ), ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ Frame Flow ​ Metrics Support The service collects processing metrics: Time to First Byte (TTFB) Processing duration Connection status ​ Notes Audio input must be in PCM format Transcription frames are only generated when confidence threshold is met Service automatically handles websocket connections and cleanup Real-time processing occurs in parallel for natural conversation flow Fal (Wizper) Google On this page Overview Installation Configuration Service Parameters GladiaInputParams LanguageConfig PreProcessingConfig CustomVocabularyConfig CustomSpellingConfig TranslationConfig RealtimeProcessingConfig MessagesConfig Input Output TranscriptionFrame InterimTranscriptionFrame ErrorFrame Methods Language Setting Language Support Advanced Features Custom Vocabulary Translation Multi-language Support Usage Example Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_gladia_12457eae.txt b/stt_gladia_12457eae.txt
new file mode 100644
index 0000000000000000000000000000000000000000..12f2301f7c8a17907cf26173f65ca3b7c3ae1592
--- /dev/null
+++ b/stt_gladia_12457eae.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/gladia#param-model
+Title: Gladia - Pipecat
+==================================================
+
+Gladia - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text Gladia Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview GladiaSTTService is a speech-to-text (STT) service that integrates with Gladia’s API to provide real-time transcription capabilities. It processes audio input and produces transcription frames in real-time with support for multiple languages, custom vocabulary, and various processing options. ​ Installation To use GladiaSTTService , you need to install the Gladia dependencies: Copy Ask AI pip install "pipecat-ai[gladia]" You’ll also need to set up your Gladia API key as an environment variable: GLADIA_API_KEY . ​ Configuration ​ Service Parameters ​ api_key string required Your Gladia API key for authentication ​ url string default: "https://api.gladia.io/v2/live" Gladia API endpoint URL ​ confidence float default: "0.5" Minimum confidence threshold to create interim and final transcriptions. Values range from 0 to 1. ​ sample_rate integer default: "None" Audio sample rate in Hz ​ model string default: "solaria-1" Model to use for transcription. Options include solaria-1 solaria-mini-1 fast accurate See Gladia’s docs for the latest supported models. ​ params GladiaInputParams default: "GladiaInputParams()" Additional configuration parameters for the service ​ GladiaInputParams ​ encoding string default: "wav/pcm" Audio encoding format ​ bit_depth integer default: "16" Audio bit depth ​ channels integer default: "1" Number of audio channels ​ custom_metadata Dict[str, Any] Additional metadata to include with requests ​ endpointing float Silence duration in seconds to mark end of speech ​ maximum_duration_without_endpointing integer default: "10" Maximum utterance duration without silence ​ language Language deprecated Primary language for transcription. Deprecated: use language_config instead. ​ language_config LanguageConfig Detailed language configuration ​ pre_processing PreProcessingConfig Audio pre-processing options ​ realtime_processing RealtimeProcessingConfig Real-time processing features ​ messages_config MessagesConfig WebSocket message filtering options ​ LanguageConfig ​ languages List[str] Specify language(s) for transcription. If one language is set, it will be used for all transcription. If multiple languages are provided or none, language will be auto-detected by the model. ​ code_switching boolean default: "false" If true, language will be auto-detected on each utterance. Otherwise, language will be auto-detected on first utterance and then used for the rest of the transcription. If one language is set, this option will be ignored. ​ PreProcessingConfig ​ speech_threshold float default: "0.8" Sensitivity configuration for Speech Threshold. A value close to 1 will apply stricter thresholds, making it less likely to detect background sounds as speech. Must be between 0 and 1. ​ CustomVocabularyConfig ​ vocabulary List[Union[str, CustomVocabularyItem]] required Specific vocabulary list to feed the transcription model with. Can be a list of strings or CustomVocabularyItem objects. ​ default_intensity float Default intensity for the custom vocabulary. Must be between 0 and 1. ​ CustomSpellingConfig ​ spelling_dictionary Dict[str, List[str]] required The list of spelling rules applied on the audio transcription. Keys are the correct spellings and values are lists of phonetic variations. ​ TranslationConfig ​ target_languages List[str] required The target language(s) in ISO639-1 format (e.g., “en”, “fr”, “es”) ​ model string default: "base" Translation model to use. Options: “base” or “enhanced” ​ match_original_utterances boolean default: "true" Align translated utterances with the original ones ​ RealtimeProcessingConfig ​ words_accurate_timestamps boolean Whether to provide per-word timestamps ​ custom_vocabulary boolean Whether to enable custom vocabulary ​ custom_vocabulary_config CustomVocabularyConfig Custom vocabulary configuration ​ custom_spelling boolean Whether to enable custom spelling ​ custom_spelling_config CustomSpellingConfig Custom spelling configuration ​ translation boolean Whether to enable translation ​ translation_config TranslationConfig Translation configuration ​ named_entity_recognition boolean Whether to enable named entity recognition ​ sentiment_analysis boolean Whether to enable sentiment analysis ​ MessagesConfig ​ receive_partial_transcripts boolean default: "true" If true, partial utterances will be sent via WebSocket ​ receive_final_transcripts boolean default: "true" If true, final utterances will be sent via WebSocket ​ receive_speech_events boolean default: "true" If true, begin and end speech events will be sent via WebSocket ​ receive_pre_processing_events boolean default: "true" If true, pre-processing events will be sent via WebSocket ​ receive_realtime_processing_events boolean default: "true" If true, realtime processing events will be sent via WebSocket ​ receive_post_processing_events boolean default: "true" If true, post-processing events will be sent via WebSocket ​ receive_acknowledgments boolean default: "true" If true, acknowledgments will be sent via WebSocket ​ receive_errors boolean default: "true" If true, errors will be sent via WebSocket ​ receive_lifecycle_events boolean default: "false" If true, lifecycle events will be sent via WebSocket ​ Input The service processes raw audio data with the following requirements: PCM audio format 16-bit depth 16kHz sample rate (default) Single channel (mono) ​ Output The service produces two types of frames during transcription: ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Transcription language ​ InterimTranscriptionFrame Generated during ongoing speech, containing the same fields as TranscriptionFrame but with preliminary results. ​ ErrorFrame Generated when transcription errors occur, containing error details. ​ Methods See the STT base class methods for additional functionality. ​ Language Setting Copy Ask AI await service.set_language(Language. FR ) ​ Language Support Gladia STT supports a wide range of languages. Here’s a partial list: Language Code Description Service Code Language.AF Afrikaans af Language.AM Amharic am Language.AR Arabic ar Language.AS Assamese as Language.AZ Azerbaijani az Language.BA Bashkir ba Language.BE Belarusian be Language.BG Bulgarian bg Language.BN Bengali bn Language.BO Tibetan bo Language.BR Breton br Language.BS Bosnian bs Language.CA Catalan ca Language.CS Czech cs Language.CY Welsh cy Language.DA Danish da Language.DE German de Language.EL Greek el Language.EN English en Language.ES Spanish es Language.ET Estonian et Language.EU Basque eu Language.FA Persian fa Language.FI Finnish fi Language.FO Faroese fo Language.FR French fr Language.GL Galician gl Language.GU Gujarati gu Language.HA Hausa ha Language.HAW Hawaiian haw Language.HE Hebrew he Language.HI Hindi hi Language.HR Croatian hr Language.HT Haitian Creole ht Language.HU Hungarian hu Language.HY Armenian hy Language.ID Indonesian id Language.IS Icelandic is Language.IT Italian it Language.JA Japanese ja Language.JV Javanese jv Language.KA Georgian ka Language.KK Kazakh kk Language.KM Khmer km Language.KN Kannada kn Language.KO Korean ko Language.LA Latin la Language.LB Luxembourgish lb Language.LN Lingala ln Language.LO Lao lo Language.LT Lithuanian lt Language.LV Latvian lv Language.MG Malagasy mg Language.MI Maori mi Language.MK Macedonian mk Language.ML Malayalam ml Language.MN Mongolian mn Language.MR Marathi mr Language.MS Malay ms Language.MT Maltese mt Language.MY_MR Burmese mymr Language.NE Nepali ne Language.NL Dutch nl Language.NN Norwegian (Nynorsk) nn Language.NO Norwegian no Language.OC Occitan oc Language.PA Punjabi pa Language.PL Polish pl Language.PS Pashto ps Language.PT Portuguese pt Language.RO Romanian ro Language.RU Russian ru Language.SA Sanskrit sa Language.SD Sindhi sd Language.SI Sinhala si Language.SK Slovak sk Language.SL Slovenian sl Language.SN Shona sn Language.SO Somali so Language.SQ Albanian sq Language.SR Serbian sr Language.SU Sundanese su Language.SV Swedish sv Language.SW Swahili sw Language.TA Tamil ta Language.TE Telugu te Language.TG Tajik tg Language.TH Thai th Language.TK Turkmen tk Language.TL Tagalog tl Language.TR Turkish tr Language.TT Tatar tt Language.UK Ukrainian uk Language.UR Urdu ur Language.UZ Uzbek uz Language.VI Vietnamese vi Language.YI Yiddish yi Language.YO Yoruba yo Language.ZH Chinese zh For a complete list of supported languages, refer to Gladia’s documentation . ​ Advanced Features ​ Custom Vocabulary You can provide custom vocabulary items with bias intensity: Copy Ask AI from pipecat.services.gladia.config import CustomVocabularyItem, CustomVocabularyConfig, RealtimeProcessingConfig custom_vocab = CustomVocabularyConfig( vocabulary = [ CustomVocabularyItem( value = "Pipecat" , intensity = 0.8 ), CustomVocabularyItem( value = "Daily" , intensity = 0.7 ), ], default_intensity = 0.5 ) realtime_config = RealtimeProcessingConfig( custom_vocabulary = True , custom_vocabulary_config = custom_vocab ) ​ Translation Enable real-time translation: Copy Ask AI from pipecat.services.gladia.config import TranslationConfig, RealtimeProcessingConfig translation_config = TranslationConfig( target_languages = [ "fr" , "es" , "de" ], model = "enhanced" , match_original_utterances = True ) realtime_config = RealtimeProcessingConfig( translation = True , translation_config = translation_config ) ​ Multi-language Support Configure multiple languages with automatic language switching: Copy Ask AI from pipecat.services.gladia.config import LanguageConfig, GladiaInputParams language_config = LanguageConfig( languages = [ "en" , "fr" , "es" ], code_switching = True ) params = GladiaInputParams( language_config = language_config ) ​ Usage Example Copy Ask AI from pipecat.pipeline.pipeline import Pipeline from pipecat.services.gladia.stt import GladiaSTTService from pipecat.services.gladia.config import ( GladiaInputParams, LanguageConfig, RealtimeProcessingConfig ) from pipecat.transcriptions.language import Language # Configure the service stt = GladiaSTTService( api_key = "your-api-key" , model = "solaria-1" , params = GladiaInputParams( language_config = LanguageConfig( languages = [Language. EN , Language. FR ], code_switching = True ), ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ Frame Flow ​ Metrics Support The service collects processing metrics: Time to First Byte (TTFB) Processing duration Connection status ​ Notes Audio input must be in PCM format Transcription frames are only generated when confidence threshold is met Service automatically handles websocket connections and cleanup Real-time processing occurs in parallel for natural conversation flow Fal (Wizper) Google On this page Overview Installation Configuration Service Parameters GladiaInputParams LanguageConfig PreProcessingConfig CustomVocabularyConfig CustomSpellingConfig TranslationConfig RealtimeProcessingConfig MessagesConfig Input Output TranscriptionFrame InterimTranscriptionFrame ErrorFrame Methods Language Setting Language Support Advanced Features Custom Vocabulary Translation Multi-language Support Usage Example Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_gladia_19d6c019.txt b/stt_gladia_19d6c019.txt
new file mode 100644
index 0000000000000000000000000000000000000000..d8ef499c1ee46c041fac094f343092be1c85bef7
--- /dev/null
+++ b/stt_gladia_19d6c019.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/gladia#param-receive-final-transcripts
+Title: Gladia - Pipecat
+==================================================
+
+Gladia - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text Gladia Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview GladiaSTTService is a speech-to-text (STT) service that integrates with Gladia’s API to provide real-time transcription capabilities. It processes audio input and produces transcription frames in real-time with support for multiple languages, custom vocabulary, and various processing options. ​ Installation To use GladiaSTTService , you need to install the Gladia dependencies: Copy Ask AI pip install "pipecat-ai[gladia]" You’ll also need to set up your Gladia API key as an environment variable: GLADIA_API_KEY . ​ Configuration ​ Service Parameters ​ api_key string required Your Gladia API key for authentication ​ url string default: "https://api.gladia.io/v2/live" Gladia API endpoint URL ​ confidence float default: "0.5" Minimum confidence threshold to create interim and final transcriptions. Values range from 0 to 1. ​ sample_rate integer default: "None" Audio sample rate in Hz ​ model string default: "solaria-1" Model to use for transcription. Options include solaria-1 solaria-mini-1 fast accurate See Gladia’s docs for the latest supported models. ​ params GladiaInputParams default: "GladiaInputParams()" Additional configuration parameters for the service ​ GladiaInputParams ​ encoding string default: "wav/pcm" Audio encoding format ​ bit_depth integer default: "16" Audio bit depth ​ channels integer default: "1" Number of audio channels ​ custom_metadata Dict[str, Any] Additional metadata to include with requests ​ endpointing float Silence duration in seconds to mark end of speech ​ maximum_duration_without_endpointing integer default: "10" Maximum utterance duration without silence ​ language Language deprecated Primary language for transcription. Deprecated: use language_config instead. ​ language_config LanguageConfig Detailed language configuration ​ pre_processing PreProcessingConfig Audio pre-processing options ​ realtime_processing RealtimeProcessingConfig Real-time processing features ​ messages_config MessagesConfig WebSocket message filtering options ​ LanguageConfig ​ languages List[str] Specify language(s) for transcription. If one language is set, it will be used for all transcription. If multiple languages are provided or none, language will be auto-detected by the model. ​ code_switching boolean default: "false" If true, language will be auto-detected on each utterance. Otherwise, language will be auto-detected on first utterance and then used for the rest of the transcription. If one language is set, this option will be ignored. ​ PreProcessingConfig ​ speech_threshold float default: "0.8" Sensitivity configuration for Speech Threshold. A value close to 1 will apply stricter thresholds, making it less likely to detect background sounds as speech. Must be between 0 and 1. ​ CustomVocabularyConfig ​ vocabulary List[Union[str, CustomVocabularyItem]] required Specific vocabulary list to feed the transcription model with. Can be a list of strings or CustomVocabularyItem objects. ​ default_intensity float Default intensity for the custom vocabulary. Must be between 0 and 1. ​ CustomSpellingConfig ​ spelling_dictionary Dict[str, List[str]] required The list of spelling rules applied on the audio transcription. Keys are the correct spellings and values are lists of phonetic variations. ​ TranslationConfig ​ target_languages List[str] required The target language(s) in ISO639-1 format (e.g., “en”, “fr”, “es”) ​ model string default: "base" Translation model to use. Options: “base” or “enhanced” ​ match_original_utterances boolean default: "true" Align translated utterances with the original ones ​ RealtimeProcessingConfig ​ words_accurate_timestamps boolean Whether to provide per-word timestamps ​ custom_vocabulary boolean Whether to enable custom vocabulary ​ custom_vocabulary_config CustomVocabularyConfig Custom vocabulary configuration ​ custom_spelling boolean Whether to enable custom spelling ​ custom_spelling_config CustomSpellingConfig Custom spelling configuration ​ translation boolean Whether to enable translation ​ translation_config TranslationConfig Translation configuration ​ named_entity_recognition boolean Whether to enable named entity recognition ​ sentiment_analysis boolean Whether to enable sentiment analysis ​ MessagesConfig ​ receive_partial_transcripts boolean default: "true" If true, partial utterances will be sent via WebSocket ​ receive_final_transcripts boolean default: "true" If true, final utterances will be sent via WebSocket ​ receive_speech_events boolean default: "true" If true, begin and end speech events will be sent via WebSocket ​ receive_pre_processing_events boolean default: "true" If true, pre-processing events will be sent via WebSocket ​ receive_realtime_processing_events boolean default: "true" If true, realtime processing events will be sent via WebSocket ​ receive_post_processing_events boolean default: "true" If true, post-processing events will be sent via WebSocket ​ receive_acknowledgments boolean default: "true" If true, acknowledgments will be sent via WebSocket ​ receive_errors boolean default: "true" If true, errors will be sent via WebSocket ​ receive_lifecycle_events boolean default: "false" If true, lifecycle events will be sent via WebSocket ​ Input The service processes raw audio data with the following requirements: PCM audio format 16-bit depth 16kHz sample rate (default) Single channel (mono) ​ Output The service produces two types of frames during transcription: ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Transcription language ​ InterimTranscriptionFrame Generated during ongoing speech, containing the same fields as TranscriptionFrame but with preliminary results. ​ ErrorFrame Generated when transcription errors occur, containing error details. ​ Methods See the STT base class methods for additional functionality. ​ Language Setting Copy Ask AI await service.set_language(Language. FR ) ​ Language Support Gladia STT supports a wide range of languages. Here’s a partial list: Language Code Description Service Code Language.AF Afrikaans af Language.AM Amharic am Language.AR Arabic ar Language.AS Assamese as Language.AZ Azerbaijani az Language.BA Bashkir ba Language.BE Belarusian be Language.BG Bulgarian bg Language.BN Bengali bn Language.BO Tibetan bo Language.BR Breton br Language.BS Bosnian bs Language.CA Catalan ca Language.CS Czech cs Language.CY Welsh cy Language.DA Danish da Language.DE German de Language.EL Greek el Language.EN English en Language.ES Spanish es Language.ET Estonian et Language.EU Basque eu Language.FA Persian fa Language.FI Finnish fi Language.FO Faroese fo Language.FR French fr Language.GL Galician gl Language.GU Gujarati gu Language.HA Hausa ha Language.HAW Hawaiian haw Language.HE Hebrew he Language.HI Hindi hi Language.HR Croatian hr Language.HT Haitian Creole ht Language.HU Hungarian hu Language.HY Armenian hy Language.ID Indonesian id Language.IS Icelandic is Language.IT Italian it Language.JA Japanese ja Language.JV Javanese jv Language.KA Georgian ka Language.KK Kazakh kk Language.KM Khmer km Language.KN Kannada kn Language.KO Korean ko Language.LA Latin la Language.LB Luxembourgish lb Language.LN Lingala ln Language.LO Lao lo Language.LT Lithuanian lt Language.LV Latvian lv Language.MG Malagasy mg Language.MI Maori mi Language.MK Macedonian mk Language.ML Malayalam ml Language.MN Mongolian mn Language.MR Marathi mr Language.MS Malay ms Language.MT Maltese mt Language.MY_MR Burmese mymr Language.NE Nepali ne Language.NL Dutch nl Language.NN Norwegian (Nynorsk) nn Language.NO Norwegian no Language.OC Occitan oc Language.PA Punjabi pa Language.PL Polish pl Language.PS Pashto ps Language.PT Portuguese pt Language.RO Romanian ro Language.RU Russian ru Language.SA Sanskrit sa Language.SD Sindhi sd Language.SI Sinhala si Language.SK Slovak sk Language.SL Slovenian sl Language.SN Shona sn Language.SO Somali so Language.SQ Albanian sq Language.SR Serbian sr Language.SU Sundanese su Language.SV Swedish sv Language.SW Swahili sw Language.TA Tamil ta Language.TE Telugu te Language.TG Tajik tg Language.TH Thai th Language.TK Turkmen tk Language.TL Tagalog tl Language.TR Turkish tr Language.TT Tatar tt Language.UK Ukrainian uk Language.UR Urdu ur Language.UZ Uzbek uz Language.VI Vietnamese vi Language.YI Yiddish yi Language.YO Yoruba yo Language.ZH Chinese zh For a complete list of supported languages, refer to Gladia’s documentation . ​ Advanced Features ​ Custom Vocabulary You can provide custom vocabulary items with bias intensity: Copy Ask AI from pipecat.services.gladia.config import CustomVocabularyItem, CustomVocabularyConfig, RealtimeProcessingConfig custom_vocab = CustomVocabularyConfig( vocabulary = [ CustomVocabularyItem( value = "Pipecat" , intensity = 0.8 ), CustomVocabularyItem( value = "Daily" , intensity = 0.7 ), ], default_intensity = 0.5 ) realtime_config = RealtimeProcessingConfig( custom_vocabulary = True , custom_vocabulary_config = custom_vocab ) ​ Translation Enable real-time translation: Copy Ask AI from pipecat.services.gladia.config import TranslationConfig, RealtimeProcessingConfig translation_config = TranslationConfig( target_languages = [ "fr" , "es" , "de" ], model = "enhanced" , match_original_utterances = True ) realtime_config = RealtimeProcessingConfig( translation = True , translation_config = translation_config ) ​ Multi-language Support Configure multiple languages with automatic language switching: Copy Ask AI from pipecat.services.gladia.config import LanguageConfig, GladiaInputParams language_config = LanguageConfig( languages = [ "en" , "fr" , "es" ], code_switching = True ) params = GladiaInputParams( language_config = language_config ) ​ Usage Example Copy Ask AI from pipecat.pipeline.pipeline import Pipeline from pipecat.services.gladia.stt import GladiaSTTService from pipecat.services.gladia.config import ( GladiaInputParams, LanguageConfig, RealtimeProcessingConfig ) from pipecat.transcriptions.language import Language # Configure the service stt = GladiaSTTService( api_key = "your-api-key" , model = "solaria-1" , params = GladiaInputParams( language_config = LanguageConfig( languages = [Language. EN , Language. FR ], code_switching = True ), ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ Frame Flow ​ Metrics Support The service collects processing metrics: Time to First Byte (TTFB) Processing duration Connection status ​ Notes Audio input must be in PCM format Transcription frames are only generated when confidence threshold is met Service automatically handles websocket connections and cleanup Real-time processing occurs in parallel for natural conversation flow Fal (Wizper) Google On this page Overview Installation Configuration Service Parameters GladiaInputParams LanguageConfig PreProcessingConfig CustomVocabularyConfig CustomSpellingConfig TranslationConfig RealtimeProcessingConfig MessagesConfig Input Output TranscriptionFrame InterimTranscriptionFrame ErrorFrame Methods Language Setting Language Support Advanced Features Custom Vocabulary Translation Multi-language Support Usage Example Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_gladia_205e942e.txt b/stt_gladia_205e942e.txt
new file mode 100644
index 0000000000000000000000000000000000000000..29fe4de51f244c0379dab7aa86760d60a546c700
--- /dev/null
+++ b/stt_gladia_205e942e.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/gladia#param-params
+Title: Gladia - Pipecat
+==================================================
+
+Gladia - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text Gladia Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview GladiaSTTService is a speech-to-text (STT) service that integrates with Gladia’s API to provide real-time transcription capabilities. It processes audio input and produces transcription frames in real-time with support for multiple languages, custom vocabulary, and various processing options. ​ Installation To use GladiaSTTService , you need to install the Gladia dependencies: Copy Ask AI pip install "pipecat-ai[gladia]" You’ll also need to set up your Gladia API key as an environment variable: GLADIA_API_KEY . ​ Configuration ​ Service Parameters ​ api_key string required Your Gladia API key for authentication ​ url string default: "https://api.gladia.io/v2/live" Gladia API endpoint URL ​ confidence float default: "0.5" Minimum confidence threshold to create interim and final transcriptions. Values range from 0 to 1. ​ sample_rate integer default: "None" Audio sample rate in Hz ​ model string default: "solaria-1" Model to use for transcription. Options include solaria-1 solaria-mini-1 fast accurate See Gladia’s docs for the latest supported models. ​ params GladiaInputParams default: "GladiaInputParams()" Additional configuration parameters for the service ​ GladiaInputParams ​ encoding string default: "wav/pcm" Audio encoding format ​ bit_depth integer default: "16" Audio bit depth ​ channels integer default: "1" Number of audio channels ​ custom_metadata Dict[str, Any] Additional metadata to include with requests ​ endpointing float Silence duration in seconds to mark end of speech ​ maximum_duration_without_endpointing integer default: "10" Maximum utterance duration without silence ​ language Language deprecated Primary language for transcription. Deprecated: use language_config instead. ​ language_config LanguageConfig Detailed language configuration ​ pre_processing PreProcessingConfig Audio pre-processing options ​ realtime_processing RealtimeProcessingConfig Real-time processing features ​ messages_config MessagesConfig WebSocket message filtering options ​ LanguageConfig ​ languages List[str] Specify language(s) for transcription. If one language is set, it will be used for all transcription. If multiple languages are provided or none, language will be auto-detected by the model. ​ code_switching boolean default: "false" If true, language will be auto-detected on each utterance. Otherwise, language will be auto-detected on first utterance and then used for the rest of the transcription. If one language is set, this option will be ignored. ​ PreProcessingConfig ​ speech_threshold float default: "0.8" Sensitivity configuration for Speech Threshold. A value close to 1 will apply stricter thresholds, making it less likely to detect background sounds as speech. Must be between 0 and 1. ​ CustomVocabularyConfig ​ vocabulary List[Union[str, CustomVocabularyItem]] required Specific vocabulary list to feed the transcription model with. Can be a list of strings or CustomVocabularyItem objects. ​ default_intensity float Default intensity for the custom vocabulary. Must be between 0 and 1. ​ CustomSpellingConfig ​ spelling_dictionary Dict[str, List[str]] required The list of spelling rules applied on the audio transcription. Keys are the correct spellings and values are lists of phonetic variations. ​ TranslationConfig ​ target_languages List[str] required The target language(s) in ISO639-1 format (e.g., “en”, “fr”, “es”) ​ model string default: "base" Translation model to use. Options: “base” or “enhanced” ​ match_original_utterances boolean default: "true" Align translated utterances with the original ones ​ RealtimeProcessingConfig ​ words_accurate_timestamps boolean Whether to provide per-word timestamps ​ custom_vocabulary boolean Whether to enable custom vocabulary ​ custom_vocabulary_config CustomVocabularyConfig Custom vocabulary configuration ​ custom_spelling boolean Whether to enable custom spelling ​ custom_spelling_config CustomSpellingConfig Custom spelling configuration ​ translation boolean Whether to enable translation ​ translation_config TranslationConfig Translation configuration ​ named_entity_recognition boolean Whether to enable named entity recognition ​ sentiment_analysis boolean Whether to enable sentiment analysis ​ MessagesConfig ​ receive_partial_transcripts boolean default: "true" If true, partial utterances will be sent via WebSocket ​ receive_final_transcripts boolean default: "true" If true, final utterances will be sent via WebSocket ​ receive_speech_events boolean default: "true" If true, begin and end speech events will be sent via WebSocket ​ receive_pre_processing_events boolean default: "true" If true, pre-processing events will be sent via WebSocket ​ receive_realtime_processing_events boolean default: "true" If true, realtime processing events will be sent via WebSocket ​ receive_post_processing_events boolean default: "true" If true, post-processing events will be sent via WebSocket ​ receive_acknowledgments boolean default: "true" If true, acknowledgments will be sent via WebSocket ​ receive_errors boolean default: "true" If true, errors will be sent via WebSocket ​ receive_lifecycle_events boolean default: "false" If true, lifecycle events will be sent via WebSocket ​ Input The service processes raw audio data with the following requirements: PCM audio format 16-bit depth 16kHz sample rate (default) Single channel (mono) ​ Output The service produces two types of frames during transcription: ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Transcription language ​ InterimTranscriptionFrame Generated during ongoing speech, containing the same fields as TranscriptionFrame but with preliminary results. ​ ErrorFrame Generated when transcription errors occur, containing error details. ​ Methods See the STT base class methods for additional functionality. ​ Language Setting Copy Ask AI await service.set_language(Language. FR ) ​ Language Support Gladia STT supports a wide range of languages. Here’s a partial list: Language Code Description Service Code Language.AF Afrikaans af Language.AM Amharic am Language.AR Arabic ar Language.AS Assamese as Language.AZ Azerbaijani az Language.BA Bashkir ba Language.BE Belarusian be Language.BG Bulgarian bg Language.BN Bengali bn Language.BO Tibetan bo Language.BR Breton br Language.BS Bosnian bs Language.CA Catalan ca Language.CS Czech cs Language.CY Welsh cy Language.DA Danish da Language.DE German de Language.EL Greek el Language.EN English en Language.ES Spanish es Language.ET Estonian et Language.EU Basque eu Language.FA Persian fa Language.FI Finnish fi Language.FO Faroese fo Language.FR French fr Language.GL Galician gl Language.GU Gujarati gu Language.HA Hausa ha Language.HAW Hawaiian haw Language.HE Hebrew he Language.HI Hindi hi Language.HR Croatian hr Language.HT Haitian Creole ht Language.HU Hungarian hu Language.HY Armenian hy Language.ID Indonesian id Language.IS Icelandic is Language.IT Italian it Language.JA Japanese ja Language.JV Javanese jv Language.KA Georgian ka Language.KK Kazakh kk Language.KM Khmer km Language.KN Kannada kn Language.KO Korean ko Language.LA Latin la Language.LB Luxembourgish lb Language.LN Lingala ln Language.LO Lao lo Language.LT Lithuanian lt Language.LV Latvian lv Language.MG Malagasy mg Language.MI Maori mi Language.MK Macedonian mk Language.ML Malayalam ml Language.MN Mongolian mn Language.MR Marathi mr Language.MS Malay ms Language.MT Maltese mt Language.MY_MR Burmese mymr Language.NE Nepali ne Language.NL Dutch nl Language.NN Norwegian (Nynorsk) nn Language.NO Norwegian no Language.OC Occitan oc Language.PA Punjabi pa Language.PL Polish pl Language.PS Pashto ps Language.PT Portuguese pt Language.RO Romanian ro Language.RU Russian ru Language.SA Sanskrit sa Language.SD Sindhi sd Language.SI Sinhala si Language.SK Slovak sk Language.SL Slovenian sl Language.SN Shona sn Language.SO Somali so Language.SQ Albanian sq Language.SR Serbian sr Language.SU Sundanese su Language.SV Swedish sv Language.SW Swahili sw Language.TA Tamil ta Language.TE Telugu te Language.TG Tajik tg Language.TH Thai th Language.TK Turkmen tk Language.TL Tagalog tl Language.TR Turkish tr Language.TT Tatar tt Language.UK Ukrainian uk Language.UR Urdu ur Language.UZ Uzbek uz Language.VI Vietnamese vi Language.YI Yiddish yi Language.YO Yoruba yo Language.ZH Chinese zh For a complete list of supported languages, refer to Gladia’s documentation . ​ Advanced Features ​ Custom Vocabulary You can provide custom vocabulary items with bias intensity: Copy Ask AI from pipecat.services.gladia.config import CustomVocabularyItem, CustomVocabularyConfig, RealtimeProcessingConfig custom_vocab = CustomVocabularyConfig( vocabulary = [ CustomVocabularyItem( value = "Pipecat" , intensity = 0.8 ), CustomVocabularyItem( value = "Daily" , intensity = 0.7 ), ], default_intensity = 0.5 ) realtime_config = RealtimeProcessingConfig( custom_vocabulary = True , custom_vocabulary_config = custom_vocab ) ​ Translation Enable real-time translation: Copy Ask AI from pipecat.services.gladia.config import TranslationConfig, RealtimeProcessingConfig translation_config = TranslationConfig( target_languages = [ "fr" , "es" , "de" ], model = "enhanced" , match_original_utterances = True ) realtime_config = RealtimeProcessingConfig( translation = True , translation_config = translation_config ) ​ Multi-language Support Configure multiple languages with automatic language switching: Copy Ask AI from pipecat.services.gladia.config import LanguageConfig, GladiaInputParams language_config = LanguageConfig( languages = [ "en" , "fr" , "es" ], code_switching = True ) params = GladiaInputParams( language_config = language_config ) ​ Usage Example Copy Ask AI from pipecat.pipeline.pipeline import Pipeline from pipecat.services.gladia.stt import GladiaSTTService from pipecat.services.gladia.config import ( GladiaInputParams, LanguageConfig, RealtimeProcessingConfig ) from pipecat.transcriptions.language import Language # Configure the service stt = GladiaSTTService( api_key = "your-api-key" , model = "solaria-1" , params = GladiaInputParams( language_config = LanguageConfig( languages = [Language. EN , Language. FR ], code_switching = True ), ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ Frame Flow ​ Metrics Support The service collects processing metrics: Time to First Byte (TTFB) Processing duration Connection status ​ Notes Audio input must be in PCM format Transcription frames are only generated when confidence threshold is met Service automatically handles websocket connections and cleanup Real-time processing occurs in parallel for natural conversation flow Fal (Wizper) Google On this page Overview Installation Configuration Service Parameters GladiaInputParams LanguageConfig PreProcessingConfig CustomVocabularyConfig CustomSpellingConfig TranslationConfig RealtimeProcessingConfig MessagesConfig Input Output TranscriptionFrame InterimTranscriptionFrame ErrorFrame Methods Language Setting Language Support Advanced Features Custom Vocabulary Translation Multi-language Support Usage Example Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_gladia_225bcb74.txt b/stt_gladia_225bcb74.txt
new file mode 100644
index 0000000000000000000000000000000000000000..3aa643514587a7b2d40e628a1addb9eb181b7aa2
--- /dev/null
+++ b/stt_gladia_225bcb74.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/gladia#customvocabularyconfig
+Title: Gladia - Pipecat
+==================================================
+
+Gladia - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text Gladia Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview GladiaSTTService is a speech-to-text (STT) service that integrates with Gladia’s API to provide real-time transcription capabilities. It processes audio input and produces transcription frames in real-time with support for multiple languages, custom vocabulary, and various processing options. ​ Installation To use GladiaSTTService , you need to install the Gladia dependencies: Copy Ask AI pip install "pipecat-ai[gladia]" You’ll also need to set up your Gladia API key as an environment variable: GLADIA_API_KEY . ​ Configuration ​ Service Parameters ​ api_key string required Your Gladia API key for authentication ​ url string default: "https://api.gladia.io/v2/live" Gladia API endpoint URL ​ confidence float default: "0.5" Minimum confidence threshold to create interim and final transcriptions. Values range from 0 to 1. ​ sample_rate integer default: "None" Audio sample rate in Hz ​ model string default: "solaria-1" Model to use for transcription. Options include solaria-1 solaria-mini-1 fast accurate See Gladia’s docs for the latest supported models. ​ params GladiaInputParams default: "GladiaInputParams()" Additional configuration parameters for the service ​ GladiaInputParams ​ encoding string default: "wav/pcm" Audio encoding format ​ bit_depth integer default: "16" Audio bit depth ​ channels integer default: "1" Number of audio channels ​ custom_metadata Dict[str, Any] Additional metadata to include with requests ​ endpointing float Silence duration in seconds to mark end of speech ​ maximum_duration_without_endpointing integer default: "10" Maximum utterance duration without silence ​ language Language deprecated Primary language for transcription. Deprecated: use language_config instead. ​ language_config LanguageConfig Detailed language configuration ​ pre_processing PreProcessingConfig Audio pre-processing options ​ realtime_processing RealtimeProcessingConfig Real-time processing features ​ messages_config MessagesConfig WebSocket message filtering options ​ LanguageConfig ​ languages List[str] Specify language(s) for transcription. If one language is set, it will be used for all transcription. If multiple languages are provided or none, language will be auto-detected by the model. ​ code_switching boolean default: "false" If true, language will be auto-detected on each utterance. Otherwise, language will be auto-detected on first utterance and then used for the rest of the transcription. If one language is set, this option will be ignored. ​ PreProcessingConfig ​ speech_threshold float default: "0.8" Sensitivity configuration for Speech Threshold. A value close to 1 will apply stricter thresholds, making it less likely to detect background sounds as speech. Must be between 0 and 1. ​ CustomVocabularyConfig ​ vocabulary List[Union[str, CustomVocabularyItem]] required Specific vocabulary list to feed the transcription model with. Can be a list of strings or CustomVocabularyItem objects. ​ default_intensity float Default intensity for the custom vocabulary. Must be between 0 and 1. ​ CustomSpellingConfig ​ spelling_dictionary Dict[str, List[str]] required The list of spelling rules applied on the audio transcription. Keys are the correct spellings and values are lists of phonetic variations. ​ TranslationConfig ​ target_languages List[str] required The target language(s) in ISO639-1 format (e.g., “en”, “fr”, “es”) ​ model string default: "base" Translation model to use. Options: “base” or “enhanced” ​ match_original_utterances boolean default: "true" Align translated utterances with the original ones ​ RealtimeProcessingConfig ​ words_accurate_timestamps boolean Whether to provide per-word timestamps ​ custom_vocabulary boolean Whether to enable custom vocabulary ​ custom_vocabulary_config CustomVocabularyConfig Custom vocabulary configuration ​ custom_spelling boolean Whether to enable custom spelling ​ custom_spelling_config CustomSpellingConfig Custom spelling configuration ​ translation boolean Whether to enable translation ​ translation_config TranslationConfig Translation configuration ​ named_entity_recognition boolean Whether to enable named entity recognition ​ sentiment_analysis boolean Whether to enable sentiment analysis ​ MessagesConfig ​ receive_partial_transcripts boolean default: "true" If true, partial utterances will be sent via WebSocket ​ receive_final_transcripts boolean default: "true" If true, final utterances will be sent via WebSocket ​ receive_speech_events boolean default: "true" If true, begin and end speech events will be sent via WebSocket ​ receive_pre_processing_events boolean default: "true" If true, pre-processing events will be sent via WebSocket ​ receive_realtime_processing_events boolean default: "true" If true, realtime processing events will be sent via WebSocket ​ receive_post_processing_events boolean default: "true" If true, post-processing events will be sent via WebSocket ​ receive_acknowledgments boolean default: "true" If true, acknowledgments will be sent via WebSocket ​ receive_errors boolean default: "true" If true, errors will be sent via WebSocket ​ receive_lifecycle_events boolean default: "false" If true, lifecycle events will be sent via WebSocket ​ Input The service processes raw audio data with the following requirements: PCM audio format 16-bit depth 16kHz sample rate (default) Single channel (mono) ​ Output The service produces two types of frames during transcription: ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Transcription language ​ InterimTranscriptionFrame Generated during ongoing speech, containing the same fields as TranscriptionFrame but with preliminary results. ​ ErrorFrame Generated when transcription errors occur, containing error details. ​ Methods See the STT base class methods for additional functionality. ​ Language Setting Copy Ask AI await service.set_language(Language. FR ) ​ Language Support Gladia STT supports a wide range of languages. Here’s a partial list: Language Code Description Service Code Language.AF Afrikaans af Language.AM Amharic am Language.AR Arabic ar Language.AS Assamese as Language.AZ Azerbaijani az Language.BA Bashkir ba Language.BE Belarusian be Language.BG Bulgarian bg Language.BN Bengali bn Language.BO Tibetan bo Language.BR Breton br Language.BS Bosnian bs Language.CA Catalan ca Language.CS Czech cs Language.CY Welsh cy Language.DA Danish da Language.DE German de Language.EL Greek el Language.EN English en Language.ES Spanish es Language.ET Estonian et Language.EU Basque eu Language.FA Persian fa Language.FI Finnish fi Language.FO Faroese fo Language.FR French fr Language.GL Galician gl Language.GU Gujarati gu Language.HA Hausa ha Language.HAW Hawaiian haw Language.HE Hebrew he Language.HI Hindi hi Language.HR Croatian hr Language.HT Haitian Creole ht Language.HU Hungarian hu Language.HY Armenian hy Language.ID Indonesian id Language.IS Icelandic is Language.IT Italian it Language.JA Japanese ja Language.JV Javanese jv Language.KA Georgian ka Language.KK Kazakh kk Language.KM Khmer km Language.KN Kannada kn Language.KO Korean ko Language.LA Latin la Language.LB Luxembourgish lb Language.LN Lingala ln Language.LO Lao lo Language.LT Lithuanian lt Language.LV Latvian lv Language.MG Malagasy mg Language.MI Maori mi Language.MK Macedonian mk Language.ML Malayalam ml Language.MN Mongolian mn Language.MR Marathi mr Language.MS Malay ms Language.MT Maltese mt Language.MY_MR Burmese mymr Language.NE Nepali ne Language.NL Dutch nl Language.NN Norwegian (Nynorsk) nn Language.NO Norwegian no Language.OC Occitan oc Language.PA Punjabi pa Language.PL Polish pl Language.PS Pashto ps Language.PT Portuguese pt Language.RO Romanian ro Language.RU Russian ru Language.SA Sanskrit sa Language.SD Sindhi sd Language.SI Sinhala si Language.SK Slovak sk Language.SL Slovenian sl Language.SN Shona sn Language.SO Somali so Language.SQ Albanian sq Language.SR Serbian sr Language.SU Sundanese su Language.SV Swedish sv Language.SW Swahili sw Language.TA Tamil ta Language.TE Telugu te Language.TG Tajik tg Language.TH Thai th Language.TK Turkmen tk Language.TL Tagalog tl Language.TR Turkish tr Language.TT Tatar tt Language.UK Ukrainian uk Language.UR Urdu ur Language.UZ Uzbek uz Language.VI Vietnamese vi Language.YI Yiddish yi Language.YO Yoruba yo Language.ZH Chinese zh For a complete list of supported languages, refer to Gladia’s documentation . ​ Advanced Features ​ Custom Vocabulary You can provide custom vocabulary items with bias intensity: Copy Ask AI from pipecat.services.gladia.config import CustomVocabularyItem, CustomVocabularyConfig, RealtimeProcessingConfig custom_vocab = CustomVocabularyConfig( vocabulary = [ CustomVocabularyItem( value = "Pipecat" , intensity = 0.8 ), CustomVocabularyItem( value = "Daily" , intensity = 0.7 ), ], default_intensity = 0.5 ) realtime_config = RealtimeProcessingConfig( custom_vocabulary = True , custom_vocabulary_config = custom_vocab ) ​ Translation Enable real-time translation: Copy Ask AI from pipecat.services.gladia.config import TranslationConfig, RealtimeProcessingConfig translation_config = TranslationConfig( target_languages = [ "fr" , "es" , "de" ], model = "enhanced" , match_original_utterances = True ) realtime_config = RealtimeProcessingConfig( translation = True , translation_config = translation_config ) ​ Multi-language Support Configure multiple languages with automatic language switching: Copy Ask AI from pipecat.services.gladia.config import LanguageConfig, GladiaInputParams language_config = LanguageConfig( languages = [ "en" , "fr" , "es" ], code_switching = True ) params = GladiaInputParams( language_config = language_config ) ​ Usage Example Copy Ask AI from pipecat.pipeline.pipeline import Pipeline from pipecat.services.gladia.stt import GladiaSTTService from pipecat.services.gladia.config import ( GladiaInputParams, LanguageConfig, RealtimeProcessingConfig ) from pipecat.transcriptions.language import Language # Configure the service stt = GladiaSTTService( api_key = "your-api-key" , model = "solaria-1" , params = GladiaInputParams( language_config = LanguageConfig( languages = [Language. EN , Language. FR ], code_switching = True ), ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ Frame Flow ​ Metrics Support The service collects processing metrics: Time to First Byte (TTFB) Processing duration Connection status ​ Notes Audio input must be in PCM format Transcription frames are only generated when confidence threshold is met Service automatically handles websocket connections and cleanup Real-time processing occurs in parallel for natural conversation flow Fal (Wizper) Google On this page Overview Installation Configuration Service Parameters GladiaInputParams LanguageConfig PreProcessingConfig CustomVocabularyConfig CustomSpellingConfig TranslationConfig RealtimeProcessingConfig MessagesConfig Input Output TranscriptionFrame InterimTranscriptionFrame ErrorFrame Methods Language Setting Language Support Advanced Features Custom Vocabulary Translation Multi-language Support Usage Example Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_gladia_26a0bee9.txt b/stt_gladia_26a0bee9.txt
new file mode 100644
index 0000000000000000000000000000000000000000..b7c4ea835b238eddb83d36c4ad5dcb3fc5c0b86c
--- /dev/null
+++ b/stt_gladia_26a0bee9.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/gladia#param-receive-acknowledgments
+Title: Gladia - Pipecat
+==================================================
+
+Gladia - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text Gladia Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview GladiaSTTService is a speech-to-text (STT) service that integrates with Gladia’s API to provide real-time transcription capabilities. It processes audio input and produces transcription frames in real-time with support for multiple languages, custom vocabulary, and various processing options. ​ Installation To use GladiaSTTService , you need to install the Gladia dependencies: Copy Ask AI pip install "pipecat-ai[gladia]" You’ll also need to set up your Gladia API key as an environment variable: GLADIA_API_KEY . ​ Configuration ​ Service Parameters ​ api_key string required Your Gladia API key for authentication ​ url string default: "https://api.gladia.io/v2/live" Gladia API endpoint URL ​ confidence float default: "0.5" Minimum confidence threshold to create interim and final transcriptions. Values range from 0 to 1. ​ sample_rate integer default: "None" Audio sample rate in Hz ​ model string default: "solaria-1" Model to use for transcription. Options include solaria-1 solaria-mini-1 fast accurate See Gladia’s docs for the latest supported models. ​ params GladiaInputParams default: "GladiaInputParams()" Additional configuration parameters for the service ​ GladiaInputParams ​ encoding string default: "wav/pcm" Audio encoding format ​ bit_depth integer default: "16" Audio bit depth ​ channels integer default: "1" Number of audio channels ​ custom_metadata Dict[str, Any] Additional metadata to include with requests ​ endpointing float Silence duration in seconds to mark end of speech ​ maximum_duration_without_endpointing integer default: "10" Maximum utterance duration without silence ​ language Language deprecated Primary language for transcription. Deprecated: use language_config instead. ​ language_config LanguageConfig Detailed language configuration ​ pre_processing PreProcessingConfig Audio pre-processing options ​ realtime_processing RealtimeProcessingConfig Real-time processing features ​ messages_config MessagesConfig WebSocket message filtering options ​ LanguageConfig ​ languages List[str] Specify language(s) for transcription. If one language is set, it will be used for all transcription. If multiple languages are provided or none, language will be auto-detected by the model. ​ code_switching boolean default: "false" If true, language will be auto-detected on each utterance. Otherwise, language will be auto-detected on first utterance and then used for the rest of the transcription. If one language is set, this option will be ignored. ​ PreProcessingConfig ​ speech_threshold float default: "0.8" Sensitivity configuration for Speech Threshold. A value close to 1 will apply stricter thresholds, making it less likely to detect background sounds as speech. Must be between 0 and 1. ​ CustomVocabularyConfig ​ vocabulary List[Union[str, CustomVocabularyItem]] required Specific vocabulary list to feed the transcription model with. Can be a list of strings or CustomVocabularyItem objects. ​ default_intensity float Default intensity for the custom vocabulary. Must be between 0 and 1. ​ CustomSpellingConfig ​ spelling_dictionary Dict[str, List[str]] required The list of spelling rules applied on the audio transcription. Keys are the correct spellings and values are lists of phonetic variations. ​ TranslationConfig ​ target_languages List[str] required The target language(s) in ISO639-1 format (e.g., “en”, “fr”, “es”) ​ model string default: "base" Translation model to use. Options: “base” or “enhanced” ​ match_original_utterances boolean default: "true" Align translated utterances with the original ones ​ RealtimeProcessingConfig ​ words_accurate_timestamps boolean Whether to provide per-word timestamps ​ custom_vocabulary boolean Whether to enable custom vocabulary ​ custom_vocabulary_config CustomVocabularyConfig Custom vocabulary configuration ​ custom_spelling boolean Whether to enable custom spelling ​ custom_spelling_config CustomSpellingConfig Custom spelling configuration ​ translation boolean Whether to enable translation ​ translation_config TranslationConfig Translation configuration ​ named_entity_recognition boolean Whether to enable named entity recognition ​ sentiment_analysis boolean Whether to enable sentiment analysis ​ MessagesConfig ​ receive_partial_transcripts boolean default: "true" If true, partial utterances will be sent via WebSocket ​ receive_final_transcripts boolean default: "true" If true, final utterances will be sent via WebSocket ​ receive_speech_events boolean default: "true" If true, begin and end speech events will be sent via WebSocket ​ receive_pre_processing_events boolean default: "true" If true, pre-processing events will be sent via WebSocket ​ receive_realtime_processing_events boolean default: "true" If true, realtime processing events will be sent via WebSocket ​ receive_post_processing_events boolean default: "true" If true, post-processing events will be sent via WebSocket ​ receive_acknowledgments boolean default: "true" If true, acknowledgments will be sent via WebSocket ​ receive_errors boolean default: "true" If true, errors will be sent via WebSocket ​ receive_lifecycle_events boolean default: "false" If true, lifecycle events will be sent via WebSocket ​ Input The service processes raw audio data with the following requirements: PCM audio format 16-bit depth 16kHz sample rate (default) Single channel (mono) ​ Output The service produces two types of frames during transcription: ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Transcription language ​ InterimTranscriptionFrame Generated during ongoing speech, containing the same fields as TranscriptionFrame but with preliminary results. ​ ErrorFrame Generated when transcription errors occur, containing error details. ​ Methods See the STT base class methods for additional functionality. ​ Language Setting Copy Ask AI await service.set_language(Language. FR ) ​ Language Support Gladia STT supports a wide range of languages. Here’s a partial list: Language Code Description Service Code Language.AF Afrikaans af Language.AM Amharic am Language.AR Arabic ar Language.AS Assamese as Language.AZ Azerbaijani az Language.BA Bashkir ba Language.BE Belarusian be Language.BG Bulgarian bg Language.BN Bengali bn Language.BO Tibetan bo Language.BR Breton br Language.BS Bosnian bs Language.CA Catalan ca Language.CS Czech cs Language.CY Welsh cy Language.DA Danish da Language.DE German de Language.EL Greek el Language.EN English en Language.ES Spanish es Language.ET Estonian et Language.EU Basque eu Language.FA Persian fa Language.FI Finnish fi Language.FO Faroese fo Language.FR French fr Language.GL Galician gl Language.GU Gujarati gu Language.HA Hausa ha Language.HAW Hawaiian haw Language.HE Hebrew he Language.HI Hindi hi Language.HR Croatian hr Language.HT Haitian Creole ht Language.HU Hungarian hu Language.HY Armenian hy Language.ID Indonesian id Language.IS Icelandic is Language.IT Italian it Language.JA Japanese ja Language.JV Javanese jv Language.KA Georgian ka Language.KK Kazakh kk Language.KM Khmer km Language.KN Kannada kn Language.KO Korean ko Language.LA Latin la Language.LB Luxembourgish lb Language.LN Lingala ln Language.LO Lao lo Language.LT Lithuanian lt Language.LV Latvian lv Language.MG Malagasy mg Language.MI Maori mi Language.MK Macedonian mk Language.ML Malayalam ml Language.MN Mongolian mn Language.MR Marathi mr Language.MS Malay ms Language.MT Maltese mt Language.MY_MR Burmese mymr Language.NE Nepali ne Language.NL Dutch nl Language.NN Norwegian (Nynorsk) nn Language.NO Norwegian no Language.OC Occitan oc Language.PA Punjabi pa Language.PL Polish pl Language.PS Pashto ps Language.PT Portuguese pt Language.RO Romanian ro Language.RU Russian ru Language.SA Sanskrit sa Language.SD Sindhi sd Language.SI Sinhala si Language.SK Slovak sk Language.SL Slovenian sl Language.SN Shona sn Language.SO Somali so Language.SQ Albanian sq Language.SR Serbian sr Language.SU Sundanese su Language.SV Swedish sv Language.SW Swahili sw Language.TA Tamil ta Language.TE Telugu te Language.TG Tajik tg Language.TH Thai th Language.TK Turkmen tk Language.TL Tagalog tl Language.TR Turkish tr Language.TT Tatar tt Language.UK Ukrainian uk Language.UR Urdu ur Language.UZ Uzbek uz Language.VI Vietnamese vi Language.YI Yiddish yi Language.YO Yoruba yo Language.ZH Chinese zh For a complete list of supported languages, refer to Gladia’s documentation . ​ Advanced Features ​ Custom Vocabulary You can provide custom vocabulary items with bias intensity: Copy Ask AI from pipecat.services.gladia.config import CustomVocabularyItem, CustomVocabularyConfig, RealtimeProcessingConfig custom_vocab = CustomVocabularyConfig( vocabulary = [ CustomVocabularyItem( value = "Pipecat" , intensity = 0.8 ), CustomVocabularyItem( value = "Daily" , intensity = 0.7 ), ], default_intensity = 0.5 ) realtime_config = RealtimeProcessingConfig( custom_vocabulary = True , custom_vocabulary_config = custom_vocab ) ​ Translation Enable real-time translation: Copy Ask AI from pipecat.services.gladia.config import TranslationConfig, RealtimeProcessingConfig translation_config = TranslationConfig( target_languages = [ "fr" , "es" , "de" ], model = "enhanced" , match_original_utterances = True ) realtime_config = RealtimeProcessingConfig( translation = True , translation_config = translation_config ) ​ Multi-language Support Configure multiple languages with automatic language switching: Copy Ask AI from pipecat.services.gladia.config import LanguageConfig, GladiaInputParams language_config = LanguageConfig( languages = [ "en" , "fr" , "es" ], code_switching = True ) params = GladiaInputParams( language_config = language_config ) ​ Usage Example Copy Ask AI from pipecat.pipeline.pipeline import Pipeline from pipecat.services.gladia.stt import GladiaSTTService from pipecat.services.gladia.config import ( GladiaInputParams, LanguageConfig, RealtimeProcessingConfig ) from pipecat.transcriptions.language import Language # Configure the service stt = GladiaSTTService( api_key = "your-api-key" , model = "solaria-1" , params = GladiaInputParams( language_config = LanguageConfig( languages = [Language. EN , Language. FR ], code_switching = True ), ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ Frame Flow ​ Metrics Support The service collects processing metrics: Time to First Byte (TTFB) Processing duration Connection status ​ Notes Audio input must be in PCM format Transcription frames are only generated when confidence threshold is met Service automatically handles websocket connections and cleanup Real-time processing occurs in parallel for natural conversation flow Fal (Wizper) Google On this page Overview Installation Configuration Service Parameters GladiaInputParams LanguageConfig PreProcessingConfig CustomVocabularyConfig CustomSpellingConfig TranslationConfig RealtimeProcessingConfig MessagesConfig Input Output TranscriptionFrame InterimTranscriptionFrame ErrorFrame Methods Language Setting Language Support Advanced Features Custom Vocabulary Translation Multi-language Support Usage Example Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_gladia_2786cdce.txt b/stt_gladia_2786cdce.txt
new file mode 100644
index 0000000000000000000000000000000000000000..720838db5cbd39b9e7854ee8f726922d0bb72c66
--- /dev/null
+++ b/stt_gladia_2786cdce.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/gladia#param-maximum-duration-without-endpointing
+Title: Gladia - Pipecat
+==================================================
+
+Gladia - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text Gladia Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview GladiaSTTService is a speech-to-text (STT) service that integrates with Gladia’s API to provide real-time transcription capabilities. It processes audio input and produces transcription frames in real-time with support for multiple languages, custom vocabulary, and various processing options. ​ Installation To use GladiaSTTService , you need to install the Gladia dependencies: Copy Ask AI pip install "pipecat-ai[gladia]" You’ll also need to set up your Gladia API key as an environment variable: GLADIA_API_KEY . ​ Configuration ​ Service Parameters ​ api_key string required Your Gladia API key for authentication ​ url string default: "https://api.gladia.io/v2/live" Gladia API endpoint URL ​ confidence float default: "0.5" Minimum confidence threshold to create interim and final transcriptions. Values range from 0 to 1. ​ sample_rate integer default: "None" Audio sample rate in Hz ​ model string default: "solaria-1" Model to use for transcription. Options include solaria-1 solaria-mini-1 fast accurate See Gladia’s docs for the latest supported models. ​ params GladiaInputParams default: "GladiaInputParams()" Additional configuration parameters for the service ​ GladiaInputParams ​ encoding string default: "wav/pcm" Audio encoding format ​ bit_depth integer default: "16" Audio bit depth ​ channels integer default: "1" Number of audio channels ​ custom_metadata Dict[str, Any] Additional metadata to include with requests ​ endpointing float Silence duration in seconds to mark end of speech ​ maximum_duration_without_endpointing integer default: "10" Maximum utterance duration without silence ​ language Language deprecated Primary language for transcription. Deprecated: use language_config instead. ​ language_config LanguageConfig Detailed language configuration ​ pre_processing PreProcessingConfig Audio pre-processing options ​ realtime_processing RealtimeProcessingConfig Real-time processing features ​ messages_config MessagesConfig WebSocket message filtering options ​ LanguageConfig ​ languages List[str] Specify language(s) for transcription. If one language is set, it will be used for all transcription. If multiple languages are provided or none, language will be auto-detected by the model. ​ code_switching boolean default: "false" If true, language will be auto-detected on each utterance. Otherwise, language will be auto-detected on first utterance and then used for the rest of the transcription. If one language is set, this option will be ignored. ​ PreProcessingConfig ​ speech_threshold float default: "0.8" Sensitivity configuration for Speech Threshold. A value close to 1 will apply stricter thresholds, making it less likely to detect background sounds as speech. Must be between 0 and 1. ​ CustomVocabularyConfig ​ vocabulary List[Union[str, CustomVocabularyItem]] required Specific vocabulary list to feed the transcription model with. Can be a list of strings or CustomVocabularyItem objects. ​ default_intensity float Default intensity for the custom vocabulary. Must be between 0 and 1. ​ CustomSpellingConfig ​ spelling_dictionary Dict[str, List[str]] required The list of spelling rules applied on the audio transcription. Keys are the correct spellings and values are lists of phonetic variations. ​ TranslationConfig ​ target_languages List[str] required The target language(s) in ISO639-1 format (e.g., “en”, “fr”, “es”) ​ model string default: "base" Translation model to use. Options: “base” or “enhanced” ​ match_original_utterances boolean default: "true" Align translated utterances with the original ones ​ RealtimeProcessingConfig ​ words_accurate_timestamps boolean Whether to provide per-word timestamps ​ custom_vocabulary boolean Whether to enable custom vocabulary ​ custom_vocabulary_config CustomVocabularyConfig Custom vocabulary configuration ​ custom_spelling boolean Whether to enable custom spelling ​ custom_spelling_config CustomSpellingConfig Custom spelling configuration ​ translation boolean Whether to enable translation ​ translation_config TranslationConfig Translation configuration ​ named_entity_recognition boolean Whether to enable named entity recognition ​ sentiment_analysis boolean Whether to enable sentiment analysis ​ MessagesConfig ​ receive_partial_transcripts boolean default: "true" If true, partial utterances will be sent via WebSocket ​ receive_final_transcripts boolean default: "true" If true, final utterances will be sent via WebSocket ​ receive_speech_events boolean default: "true" If true, begin and end speech events will be sent via WebSocket ​ receive_pre_processing_events boolean default: "true" If true, pre-processing events will be sent via WebSocket ​ receive_realtime_processing_events boolean default: "true" If true, realtime processing events will be sent via WebSocket ​ receive_post_processing_events boolean default: "true" If true, post-processing events will be sent via WebSocket ​ receive_acknowledgments boolean default: "true" If true, acknowledgments will be sent via WebSocket ​ receive_errors boolean default: "true" If true, errors will be sent via WebSocket ​ receive_lifecycle_events boolean default: "false" If true, lifecycle events will be sent via WebSocket ​ Input The service processes raw audio data with the following requirements: PCM audio format 16-bit depth 16kHz sample rate (default) Single channel (mono) ​ Output The service produces two types of frames during transcription: ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Transcription language ​ InterimTranscriptionFrame Generated during ongoing speech, containing the same fields as TranscriptionFrame but with preliminary results. ​ ErrorFrame Generated when transcription errors occur, containing error details. ​ Methods See the STT base class methods for additional functionality. ​ Language Setting Copy Ask AI await service.set_language(Language. FR ) ​ Language Support Gladia STT supports a wide range of languages. Here’s a partial list: Language Code Description Service Code Language.AF Afrikaans af Language.AM Amharic am Language.AR Arabic ar Language.AS Assamese as Language.AZ Azerbaijani az Language.BA Bashkir ba Language.BE Belarusian be Language.BG Bulgarian bg Language.BN Bengali bn Language.BO Tibetan bo Language.BR Breton br Language.BS Bosnian bs Language.CA Catalan ca Language.CS Czech cs Language.CY Welsh cy Language.DA Danish da Language.DE German de Language.EL Greek el Language.EN English en Language.ES Spanish es Language.ET Estonian et Language.EU Basque eu Language.FA Persian fa Language.FI Finnish fi Language.FO Faroese fo Language.FR French fr Language.GL Galician gl Language.GU Gujarati gu Language.HA Hausa ha Language.HAW Hawaiian haw Language.HE Hebrew he Language.HI Hindi hi Language.HR Croatian hr Language.HT Haitian Creole ht Language.HU Hungarian hu Language.HY Armenian hy Language.ID Indonesian id Language.IS Icelandic is Language.IT Italian it Language.JA Japanese ja Language.JV Javanese jv Language.KA Georgian ka Language.KK Kazakh kk Language.KM Khmer km Language.KN Kannada kn Language.KO Korean ko Language.LA Latin la Language.LB Luxembourgish lb Language.LN Lingala ln Language.LO Lao lo Language.LT Lithuanian lt Language.LV Latvian lv Language.MG Malagasy mg Language.MI Maori mi Language.MK Macedonian mk Language.ML Malayalam ml Language.MN Mongolian mn Language.MR Marathi mr Language.MS Malay ms Language.MT Maltese mt Language.MY_MR Burmese mymr Language.NE Nepali ne Language.NL Dutch nl Language.NN Norwegian (Nynorsk) nn Language.NO Norwegian no Language.OC Occitan oc Language.PA Punjabi pa Language.PL Polish pl Language.PS Pashto ps Language.PT Portuguese pt Language.RO Romanian ro Language.RU Russian ru Language.SA Sanskrit sa Language.SD Sindhi sd Language.SI Sinhala si Language.SK Slovak sk Language.SL Slovenian sl Language.SN Shona sn Language.SO Somali so Language.SQ Albanian sq Language.SR Serbian sr Language.SU Sundanese su Language.SV Swedish sv Language.SW Swahili sw Language.TA Tamil ta Language.TE Telugu te Language.TG Tajik tg Language.TH Thai th Language.TK Turkmen tk Language.TL Tagalog tl Language.TR Turkish tr Language.TT Tatar tt Language.UK Ukrainian uk Language.UR Urdu ur Language.UZ Uzbek uz Language.VI Vietnamese vi Language.YI Yiddish yi Language.YO Yoruba yo Language.ZH Chinese zh For a complete list of supported languages, refer to Gladia’s documentation . ​ Advanced Features ​ Custom Vocabulary You can provide custom vocabulary items with bias intensity: Copy Ask AI from pipecat.services.gladia.config import CustomVocabularyItem, CustomVocabularyConfig, RealtimeProcessingConfig custom_vocab = CustomVocabularyConfig( vocabulary = [ CustomVocabularyItem( value = "Pipecat" , intensity = 0.8 ), CustomVocabularyItem( value = "Daily" , intensity = 0.7 ), ], default_intensity = 0.5 ) realtime_config = RealtimeProcessingConfig( custom_vocabulary = True , custom_vocabulary_config = custom_vocab ) ​ Translation Enable real-time translation: Copy Ask AI from pipecat.services.gladia.config import TranslationConfig, RealtimeProcessingConfig translation_config = TranslationConfig( target_languages = [ "fr" , "es" , "de" ], model = "enhanced" , match_original_utterances = True ) realtime_config = RealtimeProcessingConfig( translation = True , translation_config = translation_config ) ​ Multi-language Support Configure multiple languages with automatic language switching: Copy Ask AI from pipecat.services.gladia.config import LanguageConfig, GladiaInputParams language_config = LanguageConfig( languages = [ "en" , "fr" , "es" ], code_switching = True ) params = GladiaInputParams( language_config = language_config ) ​ Usage Example Copy Ask AI from pipecat.pipeline.pipeline import Pipeline from pipecat.services.gladia.stt import GladiaSTTService from pipecat.services.gladia.config import ( GladiaInputParams, LanguageConfig, RealtimeProcessingConfig ) from pipecat.transcriptions.language import Language # Configure the service stt = GladiaSTTService( api_key = "your-api-key" , model = "solaria-1" , params = GladiaInputParams( language_config = LanguageConfig( languages = [Language. EN , Language. FR ], code_switching = True ), ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ Frame Flow ​ Metrics Support The service collects processing metrics: Time to First Byte (TTFB) Processing duration Connection status ​ Notes Audio input must be in PCM format Transcription frames are only generated when confidence threshold is met Service automatically handles websocket connections and cleanup Real-time processing occurs in parallel for natural conversation flow Fal (Wizper) Google On this page Overview Installation Configuration Service Parameters GladiaInputParams LanguageConfig PreProcessingConfig CustomVocabularyConfig CustomSpellingConfig TranslationConfig RealtimeProcessingConfig MessagesConfig Input Output TranscriptionFrame InterimTranscriptionFrame ErrorFrame Methods Language Setting Language Support Advanced Features Custom Vocabulary Translation Multi-language Support Usage Example Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_gladia_2a452a63.txt b/stt_gladia_2a452a63.txt
new file mode 100644
index 0000000000000000000000000000000000000000..566b39bd9a79c864ce96acb88725f6590d23b89a
--- /dev/null
+++ b/stt_gladia_2a452a63.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/gladia#input
+Title: Gladia - Pipecat
+==================================================
+
+Gladia - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text Gladia Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview GladiaSTTService is a speech-to-text (STT) service that integrates with Gladia’s API to provide real-time transcription capabilities. It processes audio input and produces transcription frames in real-time with support for multiple languages, custom vocabulary, and various processing options. ​ Installation To use GladiaSTTService , you need to install the Gladia dependencies: Copy Ask AI pip install "pipecat-ai[gladia]" You’ll also need to set up your Gladia API key as an environment variable: GLADIA_API_KEY . ​ Configuration ​ Service Parameters ​ api_key string required Your Gladia API key for authentication ​ url string default: "https://api.gladia.io/v2/live" Gladia API endpoint URL ​ confidence float default: "0.5" Minimum confidence threshold to create interim and final transcriptions. Values range from 0 to 1. ​ sample_rate integer default: "None" Audio sample rate in Hz ​ model string default: "solaria-1" Model to use for transcription. Options include solaria-1 solaria-mini-1 fast accurate See Gladia’s docs for the latest supported models. ​ params GladiaInputParams default: "GladiaInputParams()" Additional configuration parameters for the service ​ GladiaInputParams ​ encoding string default: "wav/pcm" Audio encoding format ​ bit_depth integer default: "16" Audio bit depth ​ channels integer default: "1" Number of audio channels ​ custom_metadata Dict[str, Any] Additional metadata to include with requests ​ endpointing float Silence duration in seconds to mark end of speech ​ maximum_duration_without_endpointing integer default: "10" Maximum utterance duration without silence ​ language Language deprecated Primary language for transcription. Deprecated: use language_config instead. ​ language_config LanguageConfig Detailed language configuration ​ pre_processing PreProcessingConfig Audio pre-processing options ​ realtime_processing RealtimeProcessingConfig Real-time processing features ​ messages_config MessagesConfig WebSocket message filtering options ​ LanguageConfig ​ languages List[str] Specify language(s) for transcription. If one language is set, it will be used for all transcription. If multiple languages are provided or none, language will be auto-detected by the model. ​ code_switching boolean default: "false" If true, language will be auto-detected on each utterance. Otherwise, language will be auto-detected on first utterance and then used for the rest of the transcription. If one language is set, this option will be ignored. ​ PreProcessingConfig ​ speech_threshold float default: "0.8" Sensitivity configuration for Speech Threshold. A value close to 1 will apply stricter thresholds, making it less likely to detect background sounds as speech. Must be between 0 and 1. ​ CustomVocabularyConfig ​ vocabulary List[Union[str, CustomVocabularyItem]] required Specific vocabulary list to feed the transcription model with. Can be a list of strings or CustomVocabularyItem objects. ​ default_intensity float Default intensity for the custom vocabulary. Must be between 0 and 1. ​ CustomSpellingConfig ​ spelling_dictionary Dict[str, List[str]] required The list of spelling rules applied on the audio transcription. Keys are the correct spellings and values are lists of phonetic variations. ​ TranslationConfig ​ target_languages List[str] required The target language(s) in ISO639-1 format (e.g., “en”, “fr”, “es”) ​ model string default: "base" Translation model to use. Options: “base” or “enhanced” ​ match_original_utterances boolean default: "true" Align translated utterances with the original ones ​ RealtimeProcessingConfig ​ words_accurate_timestamps boolean Whether to provide per-word timestamps ​ custom_vocabulary boolean Whether to enable custom vocabulary ​ custom_vocabulary_config CustomVocabularyConfig Custom vocabulary configuration ​ custom_spelling boolean Whether to enable custom spelling ​ custom_spelling_config CustomSpellingConfig Custom spelling configuration ​ translation boolean Whether to enable translation ​ translation_config TranslationConfig Translation configuration ​ named_entity_recognition boolean Whether to enable named entity recognition ​ sentiment_analysis boolean Whether to enable sentiment analysis ​ MessagesConfig ​ receive_partial_transcripts boolean default: "true" If true, partial utterances will be sent via WebSocket ​ receive_final_transcripts boolean default: "true" If true, final utterances will be sent via WebSocket ​ receive_speech_events boolean default: "true" If true, begin and end speech events will be sent via WebSocket ​ receive_pre_processing_events boolean default: "true" If true, pre-processing events will be sent via WebSocket ​ receive_realtime_processing_events boolean default: "true" If true, realtime processing events will be sent via WebSocket ​ receive_post_processing_events boolean default: "true" If true, post-processing events will be sent via WebSocket ​ receive_acknowledgments boolean default: "true" If true, acknowledgments will be sent via WebSocket ​ receive_errors boolean default: "true" If true, errors will be sent via WebSocket ​ receive_lifecycle_events boolean default: "false" If true, lifecycle events will be sent via WebSocket ​ Input The service processes raw audio data with the following requirements: PCM audio format 16-bit depth 16kHz sample rate (default) Single channel (mono) ​ Output The service produces two types of frames during transcription: ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Transcription language ​ InterimTranscriptionFrame Generated during ongoing speech, containing the same fields as TranscriptionFrame but with preliminary results. ​ ErrorFrame Generated when transcription errors occur, containing error details. ​ Methods See the STT base class methods for additional functionality. ​ Language Setting Copy Ask AI await service.set_language(Language. FR ) ​ Language Support Gladia STT supports a wide range of languages. Here’s a partial list: Language Code Description Service Code Language.AF Afrikaans af Language.AM Amharic am Language.AR Arabic ar Language.AS Assamese as Language.AZ Azerbaijani az Language.BA Bashkir ba Language.BE Belarusian be Language.BG Bulgarian bg Language.BN Bengali bn Language.BO Tibetan bo Language.BR Breton br Language.BS Bosnian bs Language.CA Catalan ca Language.CS Czech cs Language.CY Welsh cy Language.DA Danish da Language.DE German de Language.EL Greek el Language.EN English en Language.ES Spanish es Language.ET Estonian et Language.EU Basque eu Language.FA Persian fa Language.FI Finnish fi Language.FO Faroese fo Language.FR French fr Language.GL Galician gl Language.GU Gujarati gu Language.HA Hausa ha Language.HAW Hawaiian haw Language.HE Hebrew he Language.HI Hindi hi Language.HR Croatian hr Language.HT Haitian Creole ht Language.HU Hungarian hu Language.HY Armenian hy Language.ID Indonesian id Language.IS Icelandic is Language.IT Italian it Language.JA Japanese ja Language.JV Javanese jv Language.KA Georgian ka Language.KK Kazakh kk Language.KM Khmer km Language.KN Kannada kn Language.KO Korean ko Language.LA Latin la Language.LB Luxembourgish lb Language.LN Lingala ln Language.LO Lao lo Language.LT Lithuanian lt Language.LV Latvian lv Language.MG Malagasy mg Language.MI Maori mi Language.MK Macedonian mk Language.ML Malayalam ml Language.MN Mongolian mn Language.MR Marathi mr Language.MS Malay ms Language.MT Maltese mt Language.MY_MR Burmese mymr Language.NE Nepali ne Language.NL Dutch nl Language.NN Norwegian (Nynorsk) nn Language.NO Norwegian no Language.OC Occitan oc Language.PA Punjabi pa Language.PL Polish pl Language.PS Pashto ps Language.PT Portuguese pt Language.RO Romanian ro Language.RU Russian ru Language.SA Sanskrit sa Language.SD Sindhi sd Language.SI Sinhala si Language.SK Slovak sk Language.SL Slovenian sl Language.SN Shona sn Language.SO Somali so Language.SQ Albanian sq Language.SR Serbian sr Language.SU Sundanese su Language.SV Swedish sv Language.SW Swahili sw Language.TA Tamil ta Language.TE Telugu te Language.TG Tajik tg Language.TH Thai th Language.TK Turkmen tk Language.TL Tagalog tl Language.TR Turkish tr Language.TT Tatar tt Language.UK Ukrainian uk Language.UR Urdu ur Language.UZ Uzbek uz Language.VI Vietnamese vi Language.YI Yiddish yi Language.YO Yoruba yo Language.ZH Chinese zh For a complete list of supported languages, refer to Gladia’s documentation . ​ Advanced Features ​ Custom Vocabulary You can provide custom vocabulary items with bias intensity: Copy Ask AI from pipecat.services.gladia.config import CustomVocabularyItem, CustomVocabularyConfig, RealtimeProcessingConfig custom_vocab = CustomVocabularyConfig( vocabulary = [ CustomVocabularyItem( value = "Pipecat" , intensity = 0.8 ), CustomVocabularyItem( value = "Daily" , intensity = 0.7 ), ], default_intensity = 0.5 ) realtime_config = RealtimeProcessingConfig( custom_vocabulary = True , custom_vocabulary_config = custom_vocab ) ​ Translation Enable real-time translation: Copy Ask AI from pipecat.services.gladia.config import TranslationConfig, RealtimeProcessingConfig translation_config = TranslationConfig( target_languages = [ "fr" , "es" , "de" ], model = "enhanced" , match_original_utterances = True ) realtime_config = RealtimeProcessingConfig( translation = True , translation_config = translation_config ) ​ Multi-language Support Configure multiple languages with automatic language switching: Copy Ask AI from pipecat.services.gladia.config import LanguageConfig, GladiaInputParams language_config = LanguageConfig( languages = [ "en" , "fr" , "es" ], code_switching = True ) params = GladiaInputParams( language_config = language_config ) ​ Usage Example Copy Ask AI from pipecat.pipeline.pipeline import Pipeline from pipecat.services.gladia.stt import GladiaSTTService from pipecat.services.gladia.config import ( GladiaInputParams, LanguageConfig, RealtimeProcessingConfig ) from pipecat.transcriptions.language import Language # Configure the service stt = GladiaSTTService( api_key = "your-api-key" , model = "solaria-1" , params = GladiaInputParams( language_config = LanguageConfig( languages = [Language. EN , Language. FR ], code_switching = True ), ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ Frame Flow ​ Metrics Support The service collects processing metrics: Time to First Byte (TTFB) Processing duration Connection status ​ Notes Audio input must be in PCM format Transcription frames are only generated when confidence threshold is met Service automatically handles websocket connections and cleanup Real-time processing occurs in parallel for natural conversation flow Fal (Wizper) Google On this page Overview Installation Configuration Service Parameters GladiaInputParams LanguageConfig PreProcessingConfig CustomVocabularyConfig CustomSpellingConfig TranslationConfig RealtimeProcessingConfig MessagesConfig Input Output TranscriptionFrame InterimTranscriptionFrame ErrorFrame Methods Language Setting Language Support Advanced Features Custom Vocabulary Translation Multi-language Support Usage Example Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_gladia_31176740.txt b/stt_gladia_31176740.txt
new file mode 100644
index 0000000000000000000000000000000000000000..83c2f031b24ffdca37f848f5b831f22f44b13b78
--- /dev/null
+++ b/stt_gladia_31176740.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/gladia#advanced-features
+Title: Gladia - Pipecat
+==================================================
+
+Gladia - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text Gladia Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview GladiaSTTService is a speech-to-text (STT) service that integrates with Gladia’s API to provide real-time transcription capabilities. It processes audio input and produces transcription frames in real-time with support for multiple languages, custom vocabulary, and various processing options. ​ Installation To use GladiaSTTService , you need to install the Gladia dependencies: Copy Ask AI pip install "pipecat-ai[gladia]" You’ll also need to set up your Gladia API key as an environment variable: GLADIA_API_KEY . ​ Configuration ​ Service Parameters ​ api_key string required Your Gladia API key for authentication ​ url string default: "https://api.gladia.io/v2/live" Gladia API endpoint URL ​ confidence float default: "0.5" Minimum confidence threshold to create interim and final transcriptions. Values range from 0 to 1. ​ sample_rate integer default: "None" Audio sample rate in Hz ​ model string default: "solaria-1" Model to use for transcription. Options include solaria-1 solaria-mini-1 fast accurate See Gladia’s docs for the latest supported models. ​ params GladiaInputParams default: "GladiaInputParams()" Additional configuration parameters for the service ​ GladiaInputParams ​ encoding string default: "wav/pcm" Audio encoding format ​ bit_depth integer default: "16" Audio bit depth ​ channels integer default: "1" Number of audio channels ​ custom_metadata Dict[str, Any] Additional metadata to include with requests ​ endpointing float Silence duration in seconds to mark end of speech ​ maximum_duration_without_endpointing integer default: "10" Maximum utterance duration without silence ​ language Language deprecated Primary language for transcription. Deprecated: use language_config instead. ​ language_config LanguageConfig Detailed language configuration ​ pre_processing PreProcessingConfig Audio pre-processing options ​ realtime_processing RealtimeProcessingConfig Real-time processing features ​ messages_config MessagesConfig WebSocket message filtering options ​ LanguageConfig ​ languages List[str] Specify language(s) for transcription. If one language is set, it will be used for all transcription. If multiple languages are provided or none, language will be auto-detected by the model. ​ code_switching boolean default: "false" If true, language will be auto-detected on each utterance. Otherwise, language will be auto-detected on first utterance and then used for the rest of the transcription. If one language is set, this option will be ignored. ​ PreProcessingConfig ​ speech_threshold float default: "0.8" Sensitivity configuration for Speech Threshold. A value close to 1 will apply stricter thresholds, making it less likely to detect background sounds as speech. Must be between 0 and 1. ​ CustomVocabularyConfig ​ vocabulary List[Union[str, CustomVocabularyItem]] required Specific vocabulary list to feed the transcription model with. Can be a list of strings or CustomVocabularyItem objects. ​ default_intensity float Default intensity for the custom vocabulary. Must be between 0 and 1. ​ CustomSpellingConfig ​ spelling_dictionary Dict[str, List[str]] required The list of spelling rules applied on the audio transcription. Keys are the correct spellings and values are lists of phonetic variations. ​ TranslationConfig ​ target_languages List[str] required The target language(s) in ISO639-1 format (e.g., “en”, “fr”, “es”) ​ model string default: "base" Translation model to use. Options: “base” or “enhanced” ​ match_original_utterances boolean default: "true" Align translated utterances with the original ones ​ RealtimeProcessingConfig ​ words_accurate_timestamps boolean Whether to provide per-word timestamps ​ custom_vocabulary boolean Whether to enable custom vocabulary ​ custom_vocabulary_config CustomVocabularyConfig Custom vocabulary configuration ​ custom_spelling boolean Whether to enable custom spelling ​ custom_spelling_config CustomSpellingConfig Custom spelling configuration ​ translation boolean Whether to enable translation ​ translation_config TranslationConfig Translation configuration ​ named_entity_recognition boolean Whether to enable named entity recognition ​ sentiment_analysis boolean Whether to enable sentiment analysis ​ MessagesConfig ​ receive_partial_transcripts boolean default: "true" If true, partial utterances will be sent via WebSocket ​ receive_final_transcripts boolean default: "true" If true, final utterances will be sent via WebSocket ​ receive_speech_events boolean default: "true" If true, begin and end speech events will be sent via WebSocket ​ receive_pre_processing_events boolean default: "true" If true, pre-processing events will be sent via WebSocket ​ receive_realtime_processing_events boolean default: "true" If true, realtime processing events will be sent via WebSocket ​ receive_post_processing_events boolean default: "true" If true, post-processing events will be sent via WebSocket ​ receive_acknowledgments boolean default: "true" If true, acknowledgments will be sent via WebSocket ​ receive_errors boolean default: "true" If true, errors will be sent via WebSocket ​ receive_lifecycle_events boolean default: "false" If true, lifecycle events will be sent via WebSocket ​ Input The service processes raw audio data with the following requirements: PCM audio format 16-bit depth 16kHz sample rate (default) Single channel (mono) ​ Output The service produces two types of frames during transcription: ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Transcription language ​ InterimTranscriptionFrame Generated during ongoing speech, containing the same fields as TranscriptionFrame but with preliminary results. ​ ErrorFrame Generated when transcription errors occur, containing error details. ​ Methods See the STT base class methods for additional functionality. ​ Language Setting Copy Ask AI await service.set_language(Language. FR ) ​ Language Support Gladia STT supports a wide range of languages. Here’s a partial list: Language Code Description Service Code Language.AF Afrikaans af Language.AM Amharic am Language.AR Arabic ar Language.AS Assamese as Language.AZ Azerbaijani az Language.BA Bashkir ba Language.BE Belarusian be Language.BG Bulgarian bg Language.BN Bengali bn Language.BO Tibetan bo Language.BR Breton br Language.BS Bosnian bs Language.CA Catalan ca Language.CS Czech cs Language.CY Welsh cy Language.DA Danish da Language.DE German de Language.EL Greek el Language.EN English en Language.ES Spanish es Language.ET Estonian et Language.EU Basque eu Language.FA Persian fa Language.FI Finnish fi Language.FO Faroese fo Language.FR French fr Language.GL Galician gl Language.GU Gujarati gu Language.HA Hausa ha Language.HAW Hawaiian haw Language.HE Hebrew he Language.HI Hindi hi Language.HR Croatian hr Language.HT Haitian Creole ht Language.HU Hungarian hu Language.HY Armenian hy Language.ID Indonesian id Language.IS Icelandic is Language.IT Italian it Language.JA Japanese ja Language.JV Javanese jv Language.KA Georgian ka Language.KK Kazakh kk Language.KM Khmer km Language.KN Kannada kn Language.KO Korean ko Language.LA Latin la Language.LB Luxembourgish lb Language.LN Lingala ln Language.LO Lao lo Language.LT Lithuanian lt Language.LV Latvian lv Language.MG Malagasy mg Language.MI Maori mi Language.MK Macedonian mk Language.ML Malayalam ml Language.MN Mongolian mn Language.MR Marathi mr Language.MS Malay ms Language.MT Maltese mt Language.MY_MR Burmese mymr Language.NE Nepali ne Language.NL Dutch nl Language.NN Norwegian (Nynorsk) nn Language.NO Norwegian no Language.OC Occitan oc Language.PA Punjabi pa Language.PL Polish pl Language.PS Pashto ps Language.PT Portuguese pt Language.RO Romanian ro Language.RU Russian ru Language.SA Sanskrit sa Language.SD Sindhi sd Language.SI Sinhala si Language.SK Slovak sk Language.SL Slovenian sl Language.SN Shona sn Language.SO Somali so Language.SQ Albanian sq Language.SR Serbian sr Language.SU Sundanese su Language.SV Swedish sv Language.SW Swahili sw Language.TA Tamil ta Language.TE Telugu te Language.TG Tajik tg Language.TH Thai th Language.TK Turkmen tk Language.TL Tagalog tl Language.TR Turkish tr Language.TT Tatar tt Language.UK Ukrainian uk Language.UR Urdu ur Language.UZ Uzbek uz Language.VI Vietnamese vi Language.YI Yiddish yi Language.YO Yoruba yo Language.ZH Chinese zh For a complete list of supported languages, refer to Gladia’s documentation . ​ Advanced Features ​ Custom Vocabulary You can provide custom vocabulary items with bias intensity: Copy Ask AI from pipecat.services.gladia.config import CustomVocabularyItem, CustomVocabularyConfig, RealtimeProcessingConfig custom_vocab = CustomVocabularyConfig( vocabulary = [ CustomVocabularyItem( value = "Pipecat" , intensity = 0.8 ), CustomVocabularyItem( value = "Daily" , intensity = 0.7 ), ], default_intensity = 0.5 ) realtime_config = RealtimeProcessingConfig( custom_vocabulary = True , custom_vocabulary_config = custom_vocab ) ​ Translation Enable real-time translation: Copy Ask AI from pipecat.services.gladia.config import TranslationConfig, RealtimeProcessingConfig translation_config = TranslationConfig( target_languages = [ "fr" , "es" , "de" ], model = "enhanced" , match_original_utterances = True ) realtime_config = RealtimeProcessingConfig( translation = True , translation_config = translation_config ) ​ Multi-language Support Configure multiple languages with automatic language switching: Copy Ask AI from pipecat.services.gladia.config import LanguageConfig, GladiaInputParams language_config = LanguageConfig( languages = [ "en" , "fr" , "es" ], code_switching = True ) params = GladiaInputParams( language_config = language_config ) ​ Usage Example Copy Ask AI from pipecat.pipeline.pipeline import Pipeline from pipecat.services.gladia.stt import GladiaSTTService from pipecat.services.gladia.config import ( GladiaInputParams, LanguageConfig, RealtimeProcessingConfig ) from pipecat.transcriptions.language import Language # Configure the service stt = GladiaSTTService( api_key = "your-api-key" , model = "solaria-1" , params = GladiaInputParams( language_config = LanguageConfig( languages = [Language. EN , Language. FR ], code_switching = True ), ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ Frame Flow ​ Metrics Support The service collects processing metrics: Time to First Byte (TTFB) Processing duration Connection status ​ Notes Audio input must be in PCM format Transcription frames are only generated when confidence threshold is met Service automatically handles websocket connections and cleanup Real-time processing occurs in parallel for natural conversation flow Fal (Wizper) Google On this page Overview Installation Configuration Service Parameters GladiaInputParams LanguageConfig PreProcessingConfig CustomVocabularyConfig CustomSpellingConfig TranslationConfig RealtimeProcessingConfig MessagesConfig Input Output TranscriptionFrame InterimTranscriptionFrame ErrorFrame Methods Language Setting Language Support Advanced Features Custom Vocabulary Translation Multi-language Support Usage Example Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_gladia_33484e21.txt b/stt_gladia_33484e21.txt
new file mode 100644
index 0000000000000000000000000000000000000000..b30c1f4e7d199011612f7405e6ad3af3c749af1f
--- /dev/null
+++ b/stt_gladia_33484e21.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/gladia#param-custom-vocabulary-config
+Title: Gladia - Pipecat
+==================================================
+
+Gladia - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text Gladia Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview GladiaSTTService is a speech-to-text (STT) service that integrates with Gladia’s API to provide real-time transcription capabilities. It processes audio input and produces transcription frames in real-time with support for multiple languages, custom vocabulary, and various processing options. ​ Installation To use GladiaSTTService , you need to install the Gladia dependencies: Copy Ask AI pip install "pipecat-ai[gladia]" You’ll also need to set up your Gladia API key as an environment variable: GLADIA_API_KEY . ​ Configuration ​ Service Parameters ​ api_key string required Your Gladia API key for authentication ​ url string default: "https://api.gladia.io/v2/live" Gladia API endpoint URL ​ confidence float default: "0.5" Minimum confidence threshold to create interim and final transcriptions. Values range from 0 to 1. ​ sample_rate integer default: "None" Audio sample rate in Hz ​ model string default: "solaria-1" Model to use for transcription. Options include solaria-1 solaria-mini-1 fast accurate See Gladia’s docs for the latest supported models. ​ params GladiaInputParams default: "GladiaInputParams()" Additional configuration parameters for the service ​ GladiaInputParams ​ encoding string default: "wav/pcm" Audio encoding format ​ bit_depth integer default: "16" Audio bit depth ​ channels integer default: "1" Number of audio channels ​ custom_metadata Dict[str, Any] Additional metadata to include with requests ​ endpointing float Silence duration in seconds to mark end of speech ​ maximum_duration_without_endpointing integer default: "10" Maximum utterance duration without silence ​ language Language deprecated Primary language for transcription. Deprecated: use language_config instead. ​ language_config LanguageConfig Detailed language configuration ​ pre_processing PreProcessingConfig Audio pre-processing options ​ realtime_processing RealtimeProcessingConfig Real-time processing features ​ messages_config MessagesConfig WebSocket message filtering options ​ LanguageConfig ​ languages List[str] Specify language(s) for transcription. If one language is set, it will be used for all transcription. If multiple languages are provided or none, language will be auto-detected by the model. ​ code_switching boolean default: "false" If true, language will be auto-detected on each utterance. Otherwise, language will be auto-detected on first utterance and then used for the rest of the transcription. If one language is set, this option will be ignored. ​ PreProcessingConfig ​ speech_threshold float default: "0.8" Sensitivity configuration for Speech Threshold. A value close to 1 will apply stricter thresholds, making it less likely to detect background sounds as speech. Must be between 0 and 1. ​ CustomVocabularyConfig ​ vocabulary List[Union[str, CustomVocabularyItem]] required Specific vocabulary list to feed the transcription model with. Can be a list of strings or CustomVocabularyItem objects. ​ default_intensity float Default intensity for the custom vocabulary. Must be between 0 and 1. ​ CustomSpellingConfig ​ spelling_dictionary Dict[str, List[str]] required The list of spelling rules applied on the audio transcription. Keys are the correct spellings and values are lists of phonetic variations. ​ TranslationConfig ​ target_languages List[str] required The target language(s) in ISO639-1 format (e.g., “en”, “fr”, “es”) ​ model string default: "base" Translation model to use. Options: “base” or “enhanced” ​ match_original_utterances boolean default: "true" Align translated utterances with the original ones ​ RealtimeProcessingConfig ​ words_accurate_timestamps boolean Whether to provide per-word timestamps ​ custom_vocabulary boolean Whether to enable custom vocabulary ​ custom_vocabulary_config CustomVocabularyConfig Custom vocabulary configuration ​ custom_spelling boolean Whether to enable custom spelling ​ custom_spelling_config CustomSpellingConfig Custom spelling configuration ​ translation boolean Whether to enable translation ​ translation_config TranslationConfig Translation configuration ​ named_entity_recognition boolean Whether to enable named entity recognition ​ sentiment_analysis boolean Whether to enable sentiment analysis ​ MessagesConfig ​ receive_partial_transcripts boolean default: "true" If true, partial utterances will be sent via WebSocket ​ receive_final_transcripts boolean default: "true" If true, final utterances will be sent via WebSocket ​ receive_speech_events boolean default: "true" If true, begin and end speech events will be sent via WebSocket ​ receive_pre_processing_events boolean default: "true" If true, pre-processing events will be sent via WebSocket ​ receive_realtime_processing_events boolean default: "true" If true, realtime processing events will be sent via WebSocket ​ receive_post_processing_events boolean default: "true" If true, post-processing events will be sent via WebSocket ​ receive_acknowledgments boolean default: "true" If true, acknowledgments will be sent via WebSocket ​ receive_errors boolean default: "true" If true, errors will be sent via WebSocket ​ receive_lifecycle_events boolean default: "false" If true, lifecycle events will be sent via WebSocket ​ Input The service processes raw audio data with the following requirements: PCM audio format 16-bit depth 16kHz sample rate (default) Single channel (mono) ​ Output The service produces two types of frames during transcription: ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Transcription language ​ InterimTranscriptionFrame Generated during ongoing speech, containing the same fields as TranscriptionFrame but with preliminary results. ​ ErrorFrame Generated when transcription errors occur, containing error details. ​ Methods See the STT base class methods for additional functionality. ​ Language Setting Copy Ask AI await service.set_language(Language. FR ) ​ Language Support Gladia STT supports a wide range of languages. Here’s a partial list: Language Code Description Service Code Language.AF Afrikaans af Language.AM Amharic am Language.AR Arabic ar Language.AS Assamese as Language.AZ Azerbaijani az Language.BA Bashkir ba Language.BE Belarusian be Language.BG Bulgarian bg Language.BN Bengali bn Language.BO Tibetan bo Language.BR Breton br Language.BS Bosnian bs Language.CA Catalan ca Language.CS Czech cs Language.CY Welsh cy Language.DA Danish da Language.DE German de Language.EL Greek el Language.EN English en Language.ES Spanish es Language.ET Estonian et Language.EU Basque eu Language.FA Persian fa Language.FI Finnish fi Language.FO Faroese fo Language.FR French fr Language.GL Galician gl Language.GU Gujarati gu Language.HA Hausa ha Language.HAW Hawaiian haw Language.HE Hebrew he Language.HI Hindi hi Language.HR Croatian hr Language.HT Haitian Creole ht Language.HU Hungarian hu Language.HY Armenian hy Language.ID Indonesian id Language.IS Icelandic is Language.IT Italian it Language.JA Japanese ja Language.JV Javanese jv Language.KA Georgian ka Language.KK Kazakh kk Language.KM Khmer km Language.KN Kannada kn Language.KO Korean ko Language.LA Latin la Language.LB Luxembourgish lb Language.LN Lingala ln Language.LO Lao lo Language.LT Lithuanian lt Language.LV Latvian lv Language.MG Malagasy mg Language.MI Maori mi Language.MK Macedonian mk Language.ML Malayalam ml Language.MN Mongolian mn Language.MR Marathi mr Language.MS Malay ms Language.MT Maltese mt Language.MY_MR Burmese mymr Language.NE Nepali ne Language.NL Dutch nl Language.NN Norwegian (Nynorsk) nn Language.NO Norwegian no Language.OC Occitan oc Language.PA Punjabi pa Language.PL Polish pl Language.PS Pashto ps Language.PT Portuguese pt Language.RO Romanian ro Language.RU Russian ru Language.SA Sanskrit sa Language.SD Sindhi sd Language.SI Sinhala si Language.SK Slovak sk Language.SL Slovenian sl Language.SN Shona sn Language.SO Somali so Language.SQ Albanian sq Language.SR Serbian sr Language.SU Sundanese su Language.SV Swedish sv Language.SW Swahili sw Language.TA Tamil ta Language.TE Telugu te Language.TG Tajik tg Language.TH Thai th Language.TK Turkmen tk Language.TL Tagalog tl Language.TR Turkish tr Language.TT Tatar tt Language.UK Ukrainian uk Language.UR Urdu ur Language.UZ Uzbek uz Language.VI Vietnamese vi Language.YI Yiddish yi Language.YO Yoruba yo Language.ZH Chinese zh For a complete list of supported languages, refer to Gladia’s documentation . ​ Advanced Features ​ Custom Vocabulary You can provide custom vocabulary items with bias intensity: Copy Ask AI from pipecat.services.gladia.config import CustomVocabularyItem, CustomVocabularyConfig, RealtimeProcessingConfig custom_vocab = CustomVocabularyConfig( vocabulary = [ CustomVocabularyItem( value = "Pipecat" , intensity = 0.8 ), CustomVocabularyItem( value = "Daily" , intensity = 0.7 ), ], default_intensity = 0.5 ) realtime_config = RealtimeProcessingConfig( custom_vocabulary = True , custom_vocabulary_config = custom_vocab ) ​ Translation Enable real-time translation: Copy Ask AI from pipecat.services.gladia.config import TranslationConfig, RealtimeProcessingConfig translation_config = TranslationConfig( target_languages = [ "fr" , "es" , "de" ], model = "enhanced" , match_original_utterances = True ) realtime_config = RealtimeProcessingConfig( translation = True , translation_config = translation_config ) ​ Multi-language Support Configure multiple languages with automatic language switching: Copy Ask AI from pipecat.services.gladia.config import LanguageConfig, GladiaInputParams language_config = LanguageConfig( languages = [ "en" , "fr" , "es" ], code_switching = True ) params = GladiaInputParams( language_config = language_config ) ​ Usage Example Copy Ask AI from pipecat.pipeline.pipeline import Pipeline from pipecat.services.gladia.stt import GladiaSTTService from pipecat.services.gladia.config import ( GladiaInputParams, LanguageConfig, RealtimeProcessingConfig ) from pipecat.transcriptions.language import Language # Configure the service stt = GladiaSTTService( api_key = "your-api-key" , model = "solaria-1" , params = GladiaInputParams( language_config = LanguageConfig( languages = [Language. EN , Language. FR ], code_switching = True ), ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ Frame Flow ​ Metrics Support The service collects processing metrics: Time to First Byte (TTFB) Processing duration Connection status ​ Notes Audio input must be in PCM format Transcription frames are only generated when confidence threshold is met Service automatically handles websocket connections and cleanup Real-time processing occurs in parallel for natural conversation flow Fal (Wizper) Google On this page Overview Installation Configuration Service Parameters GladiaInputParams LanguageConfig PreProcessingConfig CustomVocabularyConfig CustomSpellingConfig TranslationConfig RealtimeProcessingConfig MessagesConfig Input Output TranscriptionFrame InterimTranscriptionFrame ErrorFrame Methods Language Setting Language Support Advanced Features Custom Vocabulary Translation Multi-language Support Usage Example Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_gladia_35491c1d.txt b/stt_gladia_35491c1d.txt
new file mode 100644
index 0000000000000000000000000000000000000000..664322fd58e461601eb94243c44e12950b337d54
--- /dev/null
+++ b/stt_gladia_35491c1d.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/gladia#param-custom-vocabulary
+Title: Gladia - Pipecat
+==================================================
+
+Gladia - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text Gladia Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview GladiaSTTService is a speech-to-text (STT) service that integrates with Gladia’s API to provide real-time transcription capabilities. It processes audio input and produces transcription frames in real-time with support for multiple languages, custom vocabulary, and various processing options. ​ Installation To use GladiaSTTService , you need to install the Gladia dependencies: Copy Ask AI pip install "pipecat-ai[gladia]" You’ll also need to set up your Gladia API key as an environment variable: GLADIA_API_KEY . ​ Configuration ​ Service Parameters ​ api_key string required Your Gladia API key for authentication ​ url string default: "https://api.gladia.io/v2/live" Gladia API endpoint URL ​ confidence float default: "0.5" Minimum confidence threshold to create interim and final transcriptions. Values range from 0 to 1. ​ sample_rate integer default: "None" Audio sample rate in Hz ​ model string default: "solaria-1" Model to use for transcription. Options include solaria-1 solaria-mini-1 fast accurate See Gladia’s docs for the latest supported models. ​ params GladiaInputParams default: "GladiaInputParams()" Additional configuration parameters for the service ​ GladiaInputParams ​ encoding string default: "wav/pcm" Audio encoding format ​ bit_depth integer default: "16" Audio bit depth ​ channels integer default: "1" Number of audio channels ​ custom_metadata Dict[str, Any] Additional metadata to include with requests ​ endpointing float Silence duration in seconds to mark end of speech ​ maximum_duration_without_endpointing integer default: "10" Maximum utterance duration without silence ​ language Language deprecated Primary language for transcription. Deprecated: use language_config instead. ​ language_config LanguageConfig Detailed language configuration ​ pre_processing PreProcessingConfig Audio pre-processing options ​ realtime_processing RealtimeProcessingConfig Real-time processing features ​ messages_config MessagesConfig WebSocket message filtering options ​ LanguageConfig ​ languages List[str] Specify language(s) for transcription. If one language is set, it will be used for all transcription. If multiple languages are provided or none, language will be auto-detected by the model. ​ code_switching boolean default: "false" If true, language will be auto-detected on each utterance. Otherwise, language will be auto-detected on first utterance and then used for the rest of the transcription. If one language is set, this option will be ignored. ​ PreProcessingConfig ​ speech_threshold float default: "0.8" Sensitivity configuration for Speech Threshold. A value close to 1 will apply stricter thresholds, making it less likely to detect background sounds as speech. Must be between 0 and 1. ​ CustomVocabularyConfig ​ vocabulary List[Union[str, CustomVocabularyItem]] required Specific vocabulary list to feed the transcription model with. Can be a list of strings or CustomVocabularyItem objects. ​ default_intensity float Default intensity for the custom vocabulary. Must be between 0 and 1. ​ CustomSpellingConfig ​ spelling_dictionary Dict[str, List[str]] required The list of spelling rules applied on the audio transcription. Keys are the correct spellings and values are lists of phonetic variations. ​ TranslationConfig ​ target_languages List[str] required The target language(s) in ISO639-1 format (e.g., “en”, “fr”, “es”) ​ model string default: "base" Translation model to use. Options: “base” or “enhanced” ​ match_original_utterances boolean default: "true" Align translated utterances with the original ones ​ RealtimeProcessingConfig ​ words_accurate_timestamps boolean Whether to provide per-word timestamps ​ custom_vocabulary boolean Whether to enable custom vocabulary ​ custom_vocabulary_config CustomVocabularyConfig Custom vocabulary configuration ​ custom_spelling boolean Whether to enable custom spelling ​ custom_spelling_config CustomSpellingConfig Custom spelling configuration ​ translation boolean Whether to enable translation ​ translation_config TranslationConfig Translation configuration ​ named_entity_recognition boolean Whether to enable named entity recognition ​ sentiment_analysis boolean Whether to enable sentiment analysis ​ MessagesConfig ​ receive_partial_transcripts boolean default: "true" If true, partial utterances will be sent via WebSocket ​ receive_final_transcripts boolean default: "true" If true, final utterances will be sent via WebSocket ​ receive_speech_events boolean default: "true" If true, begin and end speech events will be sent via WebSocket ​ receive_pre_processing_events boolean default: "true" If true, pre-processing events will be sent via WebSocket ​ receive_realtime_processing_events boolean default: "true" If true, realtime processing events will be sent via WebSocket ​ receive_post_processing_events boolean default: "true" If true, post-processing events will be sent via WebSocket ​ receive_acknowledgments boolean default: "true" If true, acknowledgments will be sent via WebSocket ​ receive_errors boolean default: "true" If true, errors will be sent via WebSocket ​ receive_lifecycle_events boolean default: "false" If true, lifecycle events will be sent via WebSocket ​ Input The service processes raw audio data with the following requirements: PCM audio format 16-bit depth 16kHz sample rate (default) Single channel (mono) ​ Output The service produces two types of frames during transcription: ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Transcription language ​ InterimTranscriptionFrame Generated during ongoing speech, containing the same fields as TranscriptionFrame but with preliminary results. ​ ErrorFrame Generated when transcription errors occur, containing error details. ​ Methods See the STT base class methods for additional functionality. ​ Language Setting Copy Ask AI await service.set_language(Language. FR ) ​ Language Support Gladia STT supports a wide range of languages. Here’s a partial list: Language Code Description Service Code Language.AF Afrikaans af Language.AM Amharic am Language.AR Arabic ar Language.AS Assamese as Language.AZ Azerbaijani az Language.BA Bashkir ba Language.BE Belarusian be Language.BG Bulgarian bg Language.BN Bengali bn Language.BO Tibetan bo Language.BR Breton br Language.BS Bosnian bs Language.CA Catalan ca Language.CS Czech cs Language.CY Welsh cy Language.DA Danish da Language.DE German de Language.EL Greek el Language.EN English en Language.ES Spanish es Language.ET Estonian et Language.EU Basque eu Language.FA Persian fa Language.FI Finnish fi Language.FO Faroese fo Language.FR French fr Language.GL Galician gl Language.GU Gujarati gu Language.HA Hausa ha Language.HAW Hawaiian haw Language.HE Hebrew he Language.HI Hindi hi Language.HR Croatian hr Language.HT Haitian Creole ht Language.HU Hungarian hu Language.HY Armenian hy Language.ID Indonesian id Language.IS Icelandic is Language.IT Italian it Language.JA Japanese ja Language.JV Javanese jv Language.KA Georgian ka Language.KK Kazakh kk Language.KM Khmer km Language.KN Kannada kn Language.KO Korean ko Language.LA Latin la Language.LB Luxembourgish lb Language.LN Lingala ln Language.LO Lao lo Language.LT Lithuanian lt Language.LV Latvian lv Language.MG Malagasy mg Language.MI Maori mi Language.MK Macedonian mk Language.ML Malayalam ml Language.MN Mongolian mn Language.MR Marathi mr Language.MS Malay ms Language.MT Maltese mt Language.MY_MR Burmese mymr Language.NE Nepali ne Language.NL Dutch nl Language.NN Norwegian (Nynorsk) nn Language.NO Norwegian no Language.OC Occitan oc Language.PA Punjabi pa Language.PL Polish pl Language.PS Pashto ps Language.PT Portuguese pt Language.RO Romanian ro Language.RU Russian ru Language.SA Sanskrit sa Language.SD Sindhi sd Language.SI Sinhala si Language.SK Slovak sk Language.SL Slovenian sl Language.SN Shona sn Language.SO Somali so Language.SQ Albanian sq Language.SR Serbian sr Language.SU Sundanese su Language.SV Swedish sv Language.SW Swahili sw Language.TA Tamil ta Language.TE Telugu te Language.TG Tajik tg Language.TH Thai th Language.TK Turkmen tk Language.TL Tagalog tl Language.TR Turkish tr Language.TT Tatar tt Language.UK Ukrainian uk Language.UR Urdu ur Language.UZ Uzbek uz Language.VI Vietnamese vi Language.YI Yiddish yi Language.YO Yoruba yo Language.ZH Chinese zh For a complete list of supported languages, refer to Gladia’s documentation . ​ Advanced Features ​ Custom Vocabulary You can provide custom vocabulary items with bias intensity: Copy Ask AI from pipecat.services.gladia.config import CustomVocabularyItem, CustomVocabularyConfig, RealtimeProcessingConfig custom_vocab = CustomVocabularyConfig( vocabulary = [ CustomVocabularyItem( value = "Pipecat" , intensity = 0.8 ), CustomVocabularyItem( value = "Daily" , intensity = 0.7 ), ], default_intensity = 0.5 ) realtime_config = RealtimeProcessingConfig( custom_vocabulary = True , custom_vocabulary_config = custom_vocab ) ​ Translation Enable real-time translation: Copy Ask AI from pipecat.services.gladia.config import TranslationConfig, RealtimeProcessingConfig translation_config = TranslationConfig( target_languages = [ "fr" , "es" , "de" ], model = "enhanced" , match_original_utterances = True ) realtime_config = RealtimeProcessingConfig( translation = True , translation_config = translation_config ) ​ Multi-language Support Configure multiple languages with automatic language switching: Copy Ask AI from pipecat.services.gladia.config import LanguageConfig, GladiaInputParams language_config = LanguageConfig( languages = [ "en" , "fr" , "es" ], code_switching = True ) params = GladiaInputParams( language_config = language_config ) ​ Usage Example Copy Ask AI from pipecat.pipeline.pipeline import Pipeline from pipecat.services.gladia.stt import GladiaSTTService from pipecat.services.gladia.config import ( GladiaInputParams, LanguageConfig, RealtimeProcessingConfig ) from pipecat.transcriptions.language import Language # Configure the service stt = GladiaSTTService( api_key = "your-api-key" , model = "solaria-1" , params = GladiaInputParams( language_config = LanguageConfig( languages = [Language. EN , Language. FR ], code_switching = True ), ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ Frame Flow ​ Metrics Support The service collects processing metrics: Time to First Byte (TTFB) Processing duration Connection status ​ Notes Audio input must be in PCM format Transcription frames are only generated when confidence threshold is met Service automatically handles websocket connections and cleanup Real-time processing occurs in parallel for natural conversation flow Fal (Wizper) Google On this page Overview Installation Configuration Service Parameters GladiaInputParams LanguageConfig PreProcessingConfig CustomVocabularyConfig CustomSpellingConfig TranslationConfig RealtimeProcessingConfig MessagesConfig Input Output TranscriptionFrame InterimTranscriptionFrame ErrorFrame Methods Language Setting Language Support Advanced Features Custom Vocabulary Translation Multi-language Support Usage Example Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_gladia_38f904f5.txt b/stt_gladia_38f904f5.txt
new file mode 100644
index 0000000000000000000000000000000000000000..1e8d42b0968a1d03a825a6154fe30c3a58c1e1d6
--- /dev/null
+++ b/stt_gladia_38f904f5.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/gladia#param-languages
+Title: Gladia - Pipecat
+==================================================
+
+Gladia - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text Gladia Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview GladiaSTTService is a speech-to-text (STT) service that integrates with Gladia’s API to provide real-time transcription capabilities. It processes audio input and produces transcription frames in real-time with support for multiple languages, custom vocabulary, and various processing options. ​ Installation To use GladiaSTTService , you need to install the Gladia dependencies: Copy Ask AI pip install "pipecat-ai[gladia]" You’ll also need to set up your Gladia API key as an environment variable: GLADIA_API_KEY . ​ Configuration ​ Service Parameters ​ api_key string required Your Gladia API key for authentication ​ url string default: "https://api.gladia.io/v2/live" Gladia API endpoint URL ​ confidence float default: "0.5" Minimum confidence threshold to create interim and final transcriptions. Values range from 0 to 1. ​ sample_rate integer default: "None" Audio sample rate in Hz ​ model string default: "solaria-1" Model to use for transcription. Options include solaria-1 solaria-mini-1 fast accurate See Gladia’s docs for the latest supported models. ​ params GladiaInputParams default: "GladiaInputParams()" Additional configuration parameters for the service ​ GladiaInputParams ​ encoding string default: "wav/pcm" Audio encoding format ​ bit_depth integer default: "16" Audio bit depth ​ channels integer default: "1" Number of audio channels ​ custom_metadata Dict[str, Any] Additional metadata to include with requests ​ endpointing float Silence duration in seconds to mark end of speech ​ maximum_duration_without_endpointing integer default: "10" Maximum utterance duration without silence ​ language Language deprecated Primary language for transcription. Deprecated: use language_config instead. ​ language_config LanguageConfig Detailed language configuration ​ pre_processing PreProcessingConfig Audio pre-processing options ​ realtime_processing RealtimeProcessingConfig Real-time processing features ​ messages_config MessagesConfig WebSocket message filtering options ​ LanguageConfig ​ languages List[str] Specify language(s) for transcription. If one language is set, it will be used for all transcription. If multiple languages are provided or none, language will be auto-detected by the model. ​ code_switching boolean default: "false" If true, language will be auto-detected on each utterance. Otherwise, language will be auto-detected on first utterance and then used for the rest of the transcription. If one language is set, this option will be ignored. ​ PreProcessingConfig ​ speech_threshold float default: "0.8" Sensitivity configuration for Speech Threshold. A value close to 1 will apply stricter thresholds, making it less likely to detect background sounds as speech. Must be between 0 and 1. ​ CustomVocabularyConfig ​ vocabulary List[Union[str, CustomVocabularyItem]] required Specific vocabulary list to feed the transcription model with. Can be a list of strings or CustomVocabularyItem objects. ​ default_intensity float Default intensity for the custom vocabulary. Must be between 0 and 1. ​ CustomSpellingConfig ​ spelling_dictionary Dict[str, List[str]] required The list of spelling rules applied on the audio transcription. Keys are the correct spellings and values are lists of phonetic variations. ​ TranslationConfig ​ target_languages List[str] required The target language(s) in ISO639-1 format (e.g., “en”, “fr”, “es”) ​ model string default: "base" Translation model to use. Options: “base” or “enhanced” ​ match_original_utterances boolean default: "true" Align translated utterances with the original ones ​ RealtimeProcessingConfig ​ words_accurate_timestamps boolean Whether to provide per-word timestamps ​ custom_vocabulary boolean Whether to enable custom vocabulary ​ custom_vocabulary_config CustomVocabularyConfig Custom vocabulary configuration ​ custom_spelling boolean Whether to enable custom spelling ​ custom_spelling_config CustomSpellingConfig Custom spelling configuration ​ translation boolean Whether to enable translation ​ translation_config TranslationConfig Translation configuration ​ named_entity_recognition boolean Whether to enable named entity recognition ​ sentiment_analysis boolean Whether to enable sentiment analysis ​ MessagesConfig ​ receive_partial_transcripts boolean default: "true" If true, partial utterances will be sent via WebSocket ​ receive_final_transcripts boolean default: "true" If true, final utterances will be sent via WebSocket ​ receive_speech_events boolean default: "true" If true, begin and end speech events will be sent via WebSocket ​ receive_pre_processing_events boolean default: "true" If true, pre-processing events will be sent via WebSocket ​ receive_realtime_processing_events boolean default: "true" If true, realtime processing events will be sent via WebSocket ​ receive_post_processing_events boolean default: "true" If true, post-processing events will be sent via WebSocket ​ receive_acknowledgments boolean default: "true" If true, acknowledgments will be sent via WebSocket ​ receive_errors boolean default: "true" If true, errors will be sent via WebSocket ​ receive_lifecycle_events boolean default: "false" If true, lifecycle events will be sent via WebSocket ​ Input The service processes raw audio data with the following requirements: PCM audio format 16-bit depth 16kHz sample rate (default) Single channel (mono) ​ Output The service produces two types of frames during transcription: ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Transcription language ​ InterimTranscriptionFrame Generated during ongoing speech, containing the same fields as TranscriptionFrame but with preliminary results. ​ ErrorFrame Generated when transcription errors occur, containing error details. ​ Methods See the STT base class methods for additional functionality. ​ Language Setting Copy Ask AI await service.set_language(Language. FR ) ​ Language Support Gladia STT supports a wide range of languages. Here’s a partial list: Language Code Description Service Code Language.AF Afrikaans af Language.AM Amharic am Language.AR Arabic ar Language.AS Assamese as Language.AZ Azerbaijani az Language.BA Bashkir ba Language.BE Belarusian be Language.BG Bulgarian bg Language.BN Bengali bn Language.BO Tibetan bo Language.BR Breton br Language.BS Bosnian bs Language.CA Catalan ca Language.CS Czech cs Language.CY Welsh cy Language.DA Danish da Language.DE German de Language.EL Greek el Language.EN English en Language.ES Spanish es Language.ET Estonian et Language.EU Basque eu Language.FA Persian fa Language.FI Finnish fi Language.FO Faroese fo Language.FR French fr Language.GL Galician gl Language.GU Gujarati gu Language.HA Hausa ha Language.HAW Hawaiian haw Language.HE Hebrew he Language.HI Hindi hi Language.HR Croatian hr Language.HT Haitian Creole ht Language.HU Hungarian hu Language.HY Armenian hy Language.ID Indonesian id Language.IS Icelandic is Language.IT Italian it Language.JA Japanese ja Language.JV Javanese jv Language.KA Georgian ka Language.KK Kazakh kk Language.KM Khmer km Language.KN Kannada kn Language.KO Korean ko Language.LA Latin la Language.LB Luxembourgish lb Language.LN Lingala ln Language.LO Lao lo Language.LT Lithuanian lt Language.LV Latvian lv Language.MG Malagasy mg Language.MI Maori mi Language.MK Macedonian mk Language.ML Malayalam ml Language.MN Mongolian mn Language.MR Marathi mr Language.MS Malay ms Language.MT Maltese mt Language.MY_MR Burmese mymr Language.NE Nepali ne Language.NL Dutch nl Language.NN Norwegian (Nynorsk) nn Language.NO Norwegian no Language.OC Occitan oc Language.PA Punjabi pa Language.PL Polish pl Language.PS Pashto ps Language.PT Portuguese pt Language.RO Romanian ro Language.RU Russian ru Language.SA Sanskrit sa Language.SD Sindhi sd Language.SI Sinhala si Language.SK Slovak sk Language.SL Slovenian sl Language.SN Shona sn Language.SO Somali so Language.SQ Albanian sq Language.SR Serbian sr Language.SU Sundanese su Language.SV Swedish sv Language.SW Swahili sw Language.TA Tamil ta Language.TE Telugu te Language.TG Tajik tg Language.TH Thai th Language.TK Turkmen tk Language.TL Tagalog tl Language.TR Turkish tr Language.TT Tatar tt Language.UK Ukrainian uk Language.UR Urdu ur Language.UZ Uzbek uz Language.VI Vietnamese vi Language.YI Yiddish yi Language.YO Yoruba yo Language.ZH Chinese zh For a complete list of supported languages, refer to Gladia’s documentation . ​ Advanced Features ​ Custom Vocabulary You can provide custom vocabulary items with bias intensity: Copy Ask AI from pipecat.services.gladia.config import CustomVocabularyItem, CustomVocabularyConfig, RealtimeProcessingConfig custom_vocab = CustomVocabularyConfig( vocabulary = [ CustomVocabularyItem( value = "Pipecat" , intensity = 0.8 ), CustomVocabularyItem( value = "Daily" , intensity = 0.7 ), ], default_intensity = 0.5 ) realtime_config = RealtimeProcessingConfig( custom_vocabulary = True , custom_vocabulary_config = custom_vocab ) ​ Translation Enable real-time translation: Copy Ask AI from pipecat.services.gladia.config import TranslationConfig, RealtimeProcessingConfig translation_config = TranslationConfig( target_languages = [ "fr" , "es" , "de" ], model = "enhanced" , match_original_utterances = True ) realtime_config = RealtimeProcessingConfig( translation = True , translation_config = translation_config ) ​ Multi-language Support Configure multiple languages with automatic language switching: Copy Ask AI from pipecat.services.gladia.config import LanguageConfig, GladiaInputParams language_config = LanguageConfig( languages = [ "en" , "fr" , "es" ], code_switching = True ) params = GladiaInputParams( language_config = language_config ) ​ Usage Example Copy Ask AI from pipecat.pipeline.pipeline import Pipeline from pipecat.services.gladia.stt import GladiaSTTService from pipecat.services.gladia.config import ( GladiaInputParams, LanguageConfig, RealtimeProcessingConfig ) from pipecat.transcriptions.language import Language # Configure the service stt = GladiaSTTService( api_key = "your-api-key" , model = "solaria-1" , params = GladiaInputParams( language_config = LanguageConfig( languages = [Language. EN , Language. FR ], code_switching = True ), ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ Frame Flow ​ Metrics Support The service collects processing metrics: Time to First Byte (TTFB) Processing duration Connection status ​ Notes Audio input must be in PCM format Transcription frames are only generated when confidence threshold is met Service automatically handles websocket connections and cleanup Real-time processing occurs in parallel for natural conversation flow Fal (Wizper) Google On this page Overview Installation Configuration Service Parameters GladiaInputParams LanguageConfig PreProcessingConfig CustomVocabularyConfig CustomSpellingConfig TranslationConfig RealtimeProcessingConfig MessagesConfig Input Output TranscriptionFrame InterimTranscriptionFrame ErrorFrame Methods Language Setting Language Support Advanced Features Custom Vocabulary Translation Multi-language Support Usage Example Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_gladia_4f0260cc.txt b/stt_gladia_4f0260cc.txt
new file mode 100644
index 0000000000000000000000000000000000000000..a25c2853f7a7316d087886d3413e6f399855b6f7
--- /dev/null
+++ b/stt_gladia_4f0260cc.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/gladia#param-custom-metadata
+Title: Gladia - Pipecat
+==================================================
+
+Gladia - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text Gladia Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview GladiaSTTService is a speech-to-text (STT) service that integrates with Gladia’s API to provide real-time transcription capabilities. It processes audio input and produces transcription frames in real-time with support for multiple languages, custom vocabulary, and various processing options. ​ Installation To use GladiaSTTService , you need to install the Gladia dependencies: Copy Ask AI pip install "pipecat-ai[gladia]" You’ll also need to set up your Gladia API key as an environment variable: GLADIA_API_KEY . ​ Configuration ​ Service Parameters ​ api_key string required Your Gladia API key for authentication ​ url string default: "https://api.gladia.io/v2/live" Gladia API endpoint URL ​ confidence float default: "0.5" Minimum confidence threshold to create interim and final transcriptions. Values range from 0 to 1. ​ sample_rate integer default: "None" Audio sample rate in Hz ​ model string default: "solaria-1" Model to use for transcription. Options include solaria-1 solaria-mini-1 fast accurate See Gladia’s docs for the latest supported models. ​ params GladiaInputParams default: "GladiaInputParams()" Additional configuration parameters for the service ​ GladiaInputParams ​ encoding string default: "wav/pcm" Audio encoding format ​ bit_depth integer default: "16" Audio bit depth ​ channels integer default: "1" Number of audio channels ​ custom_metadata Dict[str, Any] Additional metadata to include with requests ​ endpointing float Silence duration in seconds to mark end of speech ​ maximum_duration_without_endpointing integer default: "10" Maximum utterance duration without silence ​ language Language deprecated Primary language for transcription. Deprecated: use language_config instead. ​ language_config LanguageConfig Detailed language configuration ​ pre_processing PreProcessingConfig Audio pre-processing options ​ realtime_processing RealtimeProcessingConfig Real-time processing features ​ messages_config MessagesConfig WebSocket message filtering options ​ LanguageConfig ​ languages List[str] Specify language(s) for transcription. If one language is set, it will be used for all transcription. If multiple languages are provided or none, language will be auto-detected by the model. ​ code_switching boolean default: "false" If true, language will be auto-detected on each utterance. Otherwise, language will be auto-detected on first utterance and then used for the rest of the transcription. If one language is set, this option will be ignored. ​ PreProcessingConfig ​ speech_threshold float default: "0.8" Sensitivity configuration for Speech Threshold. A value close to 1 will apply stricter thresholds, making it less likely to detect background sounds as speech. Must be between 0 and 1. ​ CustomVocabularyConfig ​ vocabulary List[Union[str, CustomVocabularyItem]] required Specific vocabulary list to feed the transcription model with. Can be a list of strings or CustomVocabularyItem objects. ​ default_intensity float Default intensity for the custom vocabulary. Must be between 0 and 1. ​ CustomSpellingConfig ​ spelling_dictionary Dict[str, List[str]] required The list of spelling rules applied on the audio transcription. Keys are the correct spellings and values are lists of phonetic variations. ​ TranslationConfig ​ target_languages List[str] required The target language(s) in ISO639-1 format (e.g., “en”, “fr”, “es”) ​ model string default: "base" Translation model to use. Options: “base” or “enhanced” ​ match_original_utterances boolean default: "true" Align translated utterances with the original ones ​ RealtimeProcessingConfig ​ words_accurate_timestamps boolean Whether to provide per-word timestamps ​ custom_vocabulary boolean Whether to enable custom vocabulary ​ custom_vocabulary_config CustomVocabularyConfig Custom vocabulary configuration ​ custom_spelling boolean Whether to enable custom spelling ​ custom_spelling_config CustomSpellingConfig Custom spelling configuration ​ translation boolean Whether to enable translation ​ translation_config TranslationConfig Translation configuration ​ named_entity_recognition boolean Whether to enable named entity recognition ​ sentiment_analysis boolean Whether to enable sentiment analysis ​ MessagesConfig ​ receive_partial_transcripts boolean default: "true" If true, partial utterances will be sent via WebSocket ​ receive_final_transcripts boolean default: "true" If true, final utterances will be sent via WebSocket ​ receive_speech_events boolean default: "true" If true, begin and end speech events will be sent via WebSocket ​ receive_pre_processing_events boolean default: "true" If true, pre-processing events will be sent via WebSocket ​ receive_realtime_processing_events boolean default: "true" If true, realtime processing events will be sent via WebSocket ​ receive_post_processing_events boolean default: "true" If true, post-processing events will be sent via WebSocket ​ receive_acknowledgments boolean default: "true" If true, acknowledgments will be sent via WebSocket ​ receive_errors boolean default: "true" If true, errors will be sent via WebSocket ​ receive_lifecycle_events boolean default: "false" If true, lifecycle events will be sent via WebSocket ​ Input The service processes raw audio data with the following requirements: PCM audio format 16-bit depth 16kHz sample rate (default) Single channel (mono) ​ Output The service produces two types of frames during transcription: ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Transcription language ​ InterimTranscriptionFrame Generated during ongoing speech, containing the same fields as TranscriptionFrame but with preliminary results. ​ ErrorFrame Generated when transcription errors occur, containing error details. ​ Methods See the STT base class methods for additional functionality. ​ Language Setting Copy Ask AI await service.set_language(Language. FR ) ​ Language Support Gladia STT supports a wide range of languages. Here’s a partial list: Language Code Description Service Code Language.AF Afrikaans af Language.AM Amharic am Language.AR Arabic ar Language.AS Assamese as Language.AZ Azerbaijani az Language.BA Bashkir ba Language.BE Belarusian be Language.BG Bulgarian bg Language.BN Bengali bn Language.BO Tibetan bo Language.BR Breton br Language.BS Bosnian bs Language.CA Catalan ca Language.CS Czech cs Language.CY Welsh cy Language.DA Danish da Language.DE German de Language.EL Greek el Language.EN English en Language.ES Spanish es Language.ET Estonian et Language.EU Basque eu Language.FA Persian fa Language.FI Finnish fi Language.FO Faroese fo Language.FR French fr Language.GL Galician gl Language.GU Gujarati gu Language.HA Hausa ha Language.HAW Hawaiian haw Language.HE Hebrew he Language.HI Hindi hi Language.HR Croatian hr Language.HT Haitian Creole ht Language.HU Hungarian hu Language.HY Armenian hy Language.ID Indonesian id Language.IS Icelandic is Language.IT Italian it Language.JA Japanese ja Language.JV Javanese jv Language.KA Georgian ka Language.KK Kazakh kk Language.KM Khmer km Language.KN Kannada kn Language.KO Korean ko Language.LA Latin la Language.LB Luxembourgish lb Language.LN Lingala ln Language.LO Lao lo Language.LT Lithuanian lt Language.LV Latvian lv Language.MG Malagasy mg Language.MI Maori mi Language.MK Macedonian mk Language.ML Malayalam ml Language.MN Mongolian mn Language.MR Marathi mr Language.MS Malay ms Language.MT Maltese mt Language.MY_MR Burmese mymr Language.NE Nepali ne Language.NL Dutch nl Language.NN Norwegian (Nynorsk) nn Language.NO Norwegian no Language.OC Occitan oc Language.PA Punjabi pa Language.PL Polish pl Language.PS Pashto ps Language.PT Portuguese pt Language.RO Romanian ro Language.RU Russian ru Language.SA Sanskrit sa Language.SD Sindhi sd Language.SI Sinhala si Language.SK Slovak sk Language.SL Slovenian sl Language.SN Shona sn Language.SO Somali so Language.SQ Albanian sq Language.SR Serbian sr Language.SU Sundanese su Language.SV Swedish sv Language.SW Swahili sw Language.TA Tamil ta Language.TE Telugu te Language.TG Tajik tg Language.TH Thai th Language.TK Turkmen tk Language.TL Tagalog tl Language.TR Turkish tr Language.TT Tatar tt Language.UK Ukrainian uk Language.UR Urdu ur Language.UZ Uzbek uz Language.VI Vietnamese vi Language.YI Yiddish yi Language.YO Yoruba yo Language.ZH Chinese zh For a complete list of supported languages, refer to Gladia’s documentation . ​ Advanced Features ​ Custom Vocabulary You can provide custom vocabulary items with bias intensity: Copy Ask AI from pipecat.services.gladia.config import CustomVocabularyItem, CustomVocabularyConfig, RealtimeProcessingConfig custom_vocab = CustomVocabularyConfig( vocabulary = [ CustomVocabularyItem( value = "Pipecat" , intensity = 0.8 ), CustomVocabularyItem( value = "Daily" , intensity = 0.7 ), ], default_intensity = 0.5 ) realtime_config = RealtimeProcessingConfig( custom_vocabulary = True , custom_vocabulary_config = custom_vocab ) ​ Translation Enable real-time translation: Copy Ask AI from pipecat.services.gladia.config import TranslationConfig, RealtimeProcessingConfig translation_config = TranslationConfig( target_languages = [ "fr" , "es" , "de" ], model = "enhanced" , match_original_utterances = True ) realtime_config = RealtimeProcessingConfig( translation = True , translation_config = translation_config ) ​ Multi-language Support Configure multiple languages with automatic language switching: Copy Ask AI from pipecat.services.gladia.config import LanguageConfig, GladiaInputParams language_config = LanguageConfig( languages = [ "en" , "fr" , "es" ], code_switching = True ) params = GladiaInputParams( language_config = language_config ) ​ Usage Example Copy Ask AI from pipecat.pipeline.pipeline import Pipeline from pipecat.services.gladia.stt import GladiaSTTService from pipecat.services.gladia.config import ( GladiaInputParams, LanguageConfig, RealtimeProcessingConfig ) from pipecat.transcriptions.language import Language # Configure the service stt = GladiaSTTService( api_key = "your-api-key" , model = "solaria-1" , params = GladiaInputParams( language_config = LanguageConfig( languages = [Language. EN , Language. FR ], code_switching = True ), ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ Frame Flow ​ Metrics Support The service collects processing metrics: Time to First Byte (TTFB) Processing duration Connection status ​ Notes Audio input must be in PCM format Transcription frames are only generated when confidence threshold is met Service automatically handles websocket connections and cleanup Real-time processing occurs in parallel for natural conversation flow Fal (Wizper) Google On this page Overview Installation Configuration Service Parameters GladiaInputParams LanguageConfig PreProcessingConfig CustomVocabularyConfig CustomSpellingConfig TranslationConfig RealtimeProcessingConfig MessagesConfig Input Output TranscriptionFrame InterimTranscriptionFrame ErrorFrame Methods Language Setting Language Support Advanced Features Custom Vocabulary Translation Multi-language Support Usage Example Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_gladia_51effc0e.txt b/stt_gladia_51effc0e.txt
new file mode 100644
index 0000000000000000000000000000000000000000..0d1a7b48e5485c0fac54d339304c47cd28386b49
--- /dev/null
+++ b/stt_gladia_51effc0e.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/gladia#param-target-languages
+Title: Gladia - Pipecat
+==================================================
+
+Gladia - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text Gladia Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview GladiaSTTService is a speech-to-text (STT) service that integrates with Gladia’s API to provide real-time transcription capabilities. It processes audio input and produces transcription frames in real-time with support for multiple languages, custom vocabulary, and various processing options. ​ Installation To use GladiaSTTService , you need to install the Gladia dependencies: Copy Ask AI pip install "pipecat-ai[gladia]" You’ll also need to set up your Gladia API key as an environment variable: GLADIA_API_KEY . ​ Configuration ​ Service Parameters ​ api_key string required Your Gladia API key for authentication ​ url string default: "https://api.gladia.io/v2/live" Gladia API endpoint URL ​ confidence float default: "0.5" Minimum confidence threshold to create interim and final transcriptions. Values range from 0 to 1. ​ sample_rate integer default: "None" Audio sample rate in Hz ​ model string default: "solaria-1" Model to use for transcription. Options include solaria-1 solaria-mini-1 fast accurate See Gladia’s docs for the latest supported models. ​ params GladiaInputParams default: "GladiaInputParams()" Additional configuration parameters for the service ​ GladiaInputParams ​ encoding string default: "wav/pcm" Audio encoding format ​ bit_depth integer default: "16" Audio bit depth ​ channels integer default: "1" Number of audio channels ​ custom_metadata Dict[str, Any] Additional metadata to include with requests ​ endpointing float Silence duration in seconds to mark end of speech ​ maximum_duration_without_endpointing integer default: "10" Maximum utterance duration without silence ​ language Language deprecated Primary language for transcription. Deprecated: use language_config instead. ​ language_config LanguageConfig Detailed language configuration ​ pre_processing PreProcessingConfig Audio pre-processing options ​ realtime_processing RealtimeProcessingConfig Real-time processing features ​ messages_config MessagesConfig WebSocket message filtering options ​ LanguageConfig ​ languages List[str] Specify language(s) for transcription. If one language is set, it will be used for all transcription. If multiple languages are provided or none, language will be auto-detected by the model. ​ code_switching boolean default: "false" If true, language will be auto-detected on each utterance. Otherwise, language will be auto-detected on first utterance and then used for the rest of the transcription. If one language is set, this option will be ignored. ​ PreProcessingConfig ​ speech_threshold float default: "0.8" Sensitivity configuration for Speech Threshold. A value close to 1 will apply stricter thresholds, making it less likely to detect background sounds as speech. Must be between 0 and 1. ​ CustomVocabularyConfig ​ vocabulary List[Union[str, CustomVocabularyItem]] required Specific vocabulary list to feed the transcription model with. Can be a list of strings or CustomVocabularyItem objects. ​ default_intensity float Default intensity for the custom vocabulary. Must be between 0 and 1. ​ CustomSpellingConfig ​ spelling_dictionary Dict[str, List[str]] required The list of spelling rules applied on the audio transcription. Keys are the correct spellings and values are lists of phonetic variations. ​ TranslationConfig ​ target_languages List[str] required The target language(s) in ISO639-1 format (e.g., “en”, “fr”, “es”) ​ model string default: "base" Translation model to use. Options: “base” or “enhanced” ​ match_original_utterances boolean default: "true" Align translated utterances with the original ones ​ RealtimeProcessingConfig ​ words_accurate_timestamps boolean Whether to provide per-word timestamps ​ custom_vocabulary boolean Whether to enable custom vocabulary ​ custom_vocabulary_config CustomVocabularyConfig Custom vocabulary configuration ​ custom_spelling boolean Whether to enable custom spelling ​ custom_spelling_config CustomSpellingConfig Custom spelling configuration ​ translation boolean Whether to enable translation ​ translation_config TranslationConfig Translation configuration ​ named_entity_recognition boolean Whether to enable named entity recognition ​ sentiment_analysis boolean Whether to enable sentiment analysis ​ MessagesConfig ​ receive_partial_transcripts boolean default: "true" If true, partial utterances will be sent via WebSocket ​ receive_final_transcripts boolean default: "true" If true, final utterances will be sent via WebSocket ​ receive_speech_events boolean default: "true" If true, begin and end speech events will be sent via WebSocket ​ receive_pre_processing_events boolean default: "true" If true, pre-processing events will be sent via WebSocket ​ receive_realtime_processing_events boolean default: "true" If true, realtime processing events will be sent via WebSocket ​ receive_post_processing_events boolean default: "true" If true, post-processing events will be sent via WebSocket ​ receive_acknowledgments boolean default: "true" If true, acknowledgments will be sent via WebSocket ​ receive_errors boolean default: "true" If true, errors will be sent via WebSocket ​ receive_lifecycle_events boolean default: "false" If true, lifecycle events will be sent via WebSocket ​ Input The service processes raw audio data with the following requirements: PCM audio format 16-bit depth 16kHz sample rate (default) Single channel (mono) ​ Output The service produces two types of frames during transcription: ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Transcription language ​ InterimTranscriptionFrame Generated during ongoing speech, containing the same fields as TranscriptionFrame but with preliminary results. ​ ErrorFrame Generated when transcription errors occur, containing error details. ​ Methods See the STT base class methods for additional functionality. ​ Language Setting Copy Ask AI await service.set_language(Language. FR ) ​ Language Support Gladia STT supports a wide range of languages. Here’s a partial list: Language Code Description Service Code Language.AF Afrikaans af Language.AM Amharic am Language.AR Arabic ar Language.AS Assamese as Language.AZ Azerbaijani az Language.BA Bashkir ba Language.BE Belarusian be Language.BG Bulgarian bg Language.BN Bengali bn Language.BO Tibetan bo Language.BR Breton br Language.BS Bosnian bs Language.CA Catalan ca Language.CS Czech cs Language.CY Welsh cy Language.DA Danish da Language.DE German de Language.EL Greek el Language.EN English en Language.ES Spanish es Language.ET Estonian et Language.EU Basque eu Language.FA Persian fa Language.FI Finnish fi Language.FO Faroese fo Language.FR French fr Language.GL Galician gl Language.GU Gujarati gu Language.HA Hausa ha Language.HAW Hawaiian haw Language.HE Hebrew he Language.HI Hindi hi Language.HR Croatian hr Language.HT Haitian Creole ht Language.HU Hungarian hu Language.HY Armenian hy Language.ID Indonesian id Language.IS Icelandic is Language.IT Italian it Language.JA Japanese ja Language.JV Javanese jv Language.KA Georgian ka Language.KK Kazakh kk Language.KM Khmer km Language.KN Kannada kn Language.KO Korean ko Language.LA Latin la Language.LB Luxembourgish lb Language.LN Lingala ln Language.LO Lao lo Language.LT Lithuanian lt Language.LV Latvian lv Language.MG Malagasy mg Language.MI Maori mi Language.MK Macedonian mk Language.ML Malayalam ml Language.MN Mongolian mn Language.MR Marathi mr Language.MS Malay ms Language.MT Maltese mt Language.MY_MR Burmese mymr Language.NE Nepali ne Language.NL Dutch nl Language.NN Norwegian (Nynorsk) nn Language.NO Norwegian no Language.OC Occitan oc Language.PA Punjabi pa Language.PL Polish pl Language.PS Pashto ps Language.PT Portuguese pt Language.RO Romanian ro Language.RU Russian ru Language.SA Sanskrit sa Language.SD Sindhi sd Language.SI Sinhala si Language.SK Slovak sk Language.SL Slovenian sl Language.SN Shona sn Language.SO Somali so Language.SQ Albanian sq Language.SR Serbian sr Language.SU Sundanese su Language.SV Swedish sv Language.SW Swahili sw Language.TA Tamil ta Language.TE Telugu te Language.TG Tajik tg Language.TH Thai th Language.TK Turkmen tk Language.TL Tagalog tl Language.TR Turkish tr Language.TT Tatar tt Language.UK Ukrainian uk Language.UR Urdu ur Language.UZ Uzbek uz Language.VI Vietnamese vi Language.YI Yiddish yi Language.YO Yoruba yo Language.ZH Chinese zh For a complete list of supported languages, refer to Gladia’s documentation . ​ Advanced Features ​ Custom Vocabulary You can provide custom vocabulary items with bias intensity: Copy Ask AI from pipecat.services.gladia.config import CustomVocabularyItem, CustomVocabularyConfig, RealtimeProcessingConfig custom_vocab = CustomVocabularyConfig( vocabulary = [ CustomVocabularyItem( value = "Pipecat" , intensity = 0.8 ), CustomVocabularyItem( value = "Daily" , intensity = 0.7 ), ], default_intensity = 0.5 ) realtime_config = RealtimeProcessingConfig( custom_vocabulary = True , custom_vocabulary_config = custom_vocab ) ​ Translation Enable real-time translation: Copy Ask AI from pipecat.services.gladia.config import TranslationConfig, RealtimeProcessingConfig translation_config = TranslationConfig( target_languages = [ "fr" , "es" , "de" ], model = "enhanced" , match_original_utterances = True ) realtime_config = RealtimeProcessingConfig( translation = True , translation_config = translation_config ) ​ Multi-language Support Configure multiple languages with automatic language switching: Copy Ask AI from pipecat.services.gladia.config import LanguageConfig, GladiaInputParams language_config = LanguageConfig( languages = [ "en" , "fr" , "es" ], code_switching = True ) params = GladiaInputParams( language_config = language_config ) ​ Usage Example Copy Ask AI from pipecat.pipeline.pipeline import Pipeline from pipecat.services.gladia.stt import GladiaSTTService from pipecat.services.gladia.config import ( GladiaInputParams, LanguageConfig, RealtimeProcessingConfig ) from pipecat.transcriptions.language import Language # Configure the service stt = GladiaSTTService( api_key = "your-api-key" , model = "solaria-1" , params = GladiaInputParams( language_config = LanguageConfig( languages = [Language. EN , Language. FR ], code_switching = True ), ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ Frame Flow ​ Metrics Support The service collects processing metrics: Time to First Byte (TTFB) Processing duration Connection status ​ Notes Audio input must be in PCM format Transcription frames are only generated when confidence threshold is met Service automatically handles websocket connections and cleanup Real-time processing occurs in parallel for natural conversation flow Fal (Wizper) Google On this page Overview Installation Configuration Service Parameters GladiaInputParams LanguageConfig PreProcessingConfig CustomVocabularyConfig CustomSpellingConfig TranslationConfig RealtimeProcessingConfig MessagesConfig Input Output TranscriptionFrame InterimTranscriptionFrame ErrorFrame Methods Language Setting Language Support Advanced Features Custom Vocabulary Translation Multi-language Support Usage Example Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_gladia_56d1948c.txt b/stt_gladia_56d1948c.txt
new file mode 100644
index 0000000000000000000000000000000000000000..7e1cfa25811bf729a31eecaa7c14dafdf6d3fbef
--- /dev/null
+++ b/stt_gladia_56d1948c.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/gladia#param-receive-pre-processing-events
+Title: Gladia - Pipecat
+==================================================
+
+Gladia - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text Gladia Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview GladiaSTTService is a speech-to-text (STT) service that integrates with Gladia’s API to provide real-time transcription capabilities. It processes audio input and produces transcription frames in real-time with support for multiple languages, custom vocabulary, and various processing options. ​ Installation To use GladiaSTTService , you need to install the Gladia dependencies: Copy Ask AI pip install "pipecat-ai[gladia]" You’ll also need to set up your Gladia API key as an environment variable: GLADIA_API_KEY . ​ Configuration ​ Service Parameters ​ api_key string required Your Gladia API key for authentication ​ url string default: "https://api.gladia.io/v2/live" Gladia API endpoint URL ​ confidence float default: "0.5" Minimum confidence threshold to create interim and final transcriptions. Values range from 0 to 1. ​ sample_rate integer default: "None" Audio sample rate in Hz ​ model string default: "solaria-1" Model to use for transcription. Options include solaria-1 solaria-mini-1 fast accurate See Gladia’s docs for the latest supported models. ​ params GladiaInputParams default: "GladiaInputParams()" Additional configuration parameters for the service ​ GladiaInputParams ​ encoding string default: "wav/pcm" Audio encoding format ​ bit_depth integer default: "16" Audio bit depth ​ channels integer default: "1" Number of audio channels ​ custom_metadata Dict[str, Any] Additional metadata to include with requests ​ endpointing float Silence duration in seconds to mark end of speech ​ maximum_duration_without_endpointing integer default: "10" Maximum utterance duration without silence ​ language Language deprecated Primary language for transcription. Deprecated: use language_config instead. ​ language_config LanguageConfig Detailed language configuration ​ pre_processing PreProcessingConfig Audio pre-processing options ​ realtime_processing RealtimeProcessingConfig Real-time processing features ​ messages_config MessagesConfig WebSocket message filtering options ​ LanguageConfig ​ languages List[str] Specify language(s) for transcription. If one language is set, it will be used for all transcription. If multiple languages are provided or none, language will be auto-detected by the model. ​ code_switching boolean default: "false" If true, language will be auto-detected on each utterance. Otherwise, language will be auto-detected on first utterance and then used for the rest of the transcription. If one language is set, this option will be ignored. ​ PreProcessingConfig ​ speech_threshold float default: "0.8" Sensitivity configuration for Speech Threshold. A value close to 1 will apply stricter thresholds, making it less likely to detect background sounds as speech. Must be between 0 and 1. ​ CustomVocabularyConfig ​ vocabulary List[Union[str, CustomVocabularyItem]] required Specific vocabulary list to feed the transcription model with. Can be a list of strings or CustomVocabularyItem objects. ​ default_intensity float Default intensity for the custom vocabulary. Must be between 0 and 1. ​ CustomSpellingConfig ​ spelling_dictionary Dict[str, List[str]] required The list of spelling rules applied on the audio transcription. Keys are the correct spellings and values are lists of phonetic variations. ​ TranslationConfig ​ target_languages List[str] required The target language(s) in ISO639-1 format (e.g., “en”, “fr”, “es”) ​ model string default: "base" Translation model to use. Options: “base” or “enhanced” ​ match_original_utterances boolean default: "true" Align translated utterances with the original ones ​ RealtimeProcessingConfig ​ words_accurate_timestamps boolean Whether to provide per-word timestamps ​ custom_vocabulary boolean Whether to enable custom vocabulary ​ custom_vocabulary_config CustomVocabularyConfig Custom vocabulary configuration ​ custom_spelling boolean Whether to enable custom spelling ​ custom_spelling_config CustomSpellingConfig Custom spelling configuration ​ translation boolean Whether to enable translation ​ translation_config TranslationConfig Translation configuration ​ named_entity_recognition boolean Whether to enable named entity recognition ​ sentiment_analysis boolean Whether to enable sentiment analysis ​ MessagesConfig ​ receive_partial_transcripts boolean default: "true" If true, partial utterances will be sent via WebSocket ​ receive_final_transcripts boolean default: "true" If true, final utterances will be sent via WebSocket ​ receive_speech_events boolean default: "true" If true, begin and end speech events will be sent via WebSocket ​ receive_pre_processing_events boolean default: "true" If true, pre-processing events will be sent via WebSocket ​ receive_realtime_processing_events boolean default: "true" If true, realtime processing events will be sent via WebSocket ​ receive_post_processing_events boolean default: "true" If true, post-processing events will be sent via WebSocket ​ receive_acknowledgments boolean default: "true" If true, acknowledgments will be sent via WebSocket ​ receive_errors boolean default: "true" If true, errors will be sent via WebSocket ​ receive_lifecycle_events boolean default: "false" If true, lifecycle events will be sent via WebSocket ​ Input The service processes raw audio data with the following requirements: PCM audio format 16-bit depth 16kHz sample rate (default) Single channel (mono) ​ Output The service produces two types of frames during transcription: ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Transcription language ​ InterimTranscriptionFrame Generated during ongoing speech, containing the same fields as TranscriptionFrame but with preliminary results. ​ ErrorFrame Generated when transcription errors occur, containing error details. ​ Methods See the STT base class methods for additional functionality. ​ Language Setting Copy Ask AI await service.set_language(Language. FR ) ​ Language Support Gladia STT supports a wide range of languages. Here’s a partial list: Language Code Description Service Code Language.AF Afrikaans af Language.AM Amharic am Language.AR Arabic ar Language.AS Assamese as Language.AZ Azerbaijani az Language.BA Bashkir ba Language.BE Belarusian be Language.BG Bulgarian bg Language.BN Bengali bn Language.BO Tibetan bo Language.BR Breton br Language.BS Bosnian bs Language.CA Catalan ca Language.CS Czech cs Language.CY Welsh cy Language.DA Danish da Language.DE German de Language.EL Greek el Language.EN English en Language.ES Spanish es Language.ET Estonian et Language.EU Basque eu Language.FA Persian fa Language.FI Finnish fi Language.FO Faroese fo Language.FR French fr Language.GL Galician gl Language.GU Gujarati gu Language.HA Hausa ha Language.HAW Hawaiian haw Language.HE Hebrew he Language.HI Hindi hi Language.HR Croatian hr Language.HT Haitian Creole ht Language.HU Hungarian hu Language.HY Armenian hy Language.ID Indonesian id Language.IS Icelandic is Language.IT Italian it Language.JA Japanese ja Language.JV Javanese jv Language.KA Georgian ka Language.KK Kazakh kk Language.KM Khmer km Language.KN Kannada kn Language.KO Korean ko Language.LA Latin la Language.LB Luxembourgish lb Language.LN Lingala ln Language.LO Lao lo Language.LT Lithuanian lt Language.LV Latvian lv Language.MG Malagasy mg Language.MI Maori mi Language.MK Macedonian mk Language.ML Malayalam ml Language.MN Mongolian mn Language.MR Marathi mr Language.MS Malay ms Language.MT Maltese mt Language.MY_MR Burmese mymr Language.NE Nepali ne Language.NL Dutch nl Language.NN Norwegian (Nynorsk) nn Language.NO Norwegian no Language.OC Occitan oc Language.PA Punjabi pa Language.PL Polish pl Language.PS Pashto ps Language.PT Portuguese pt Language.RO Romanian ro Language.RU Russian ru Language.SA Sanskrit sa Language.SD Sindhi sd Language.SI Sinhala si Language.SK Slovak sk Language.SL Slovenian sl Language.SN Shona sn Language.SO Somali so Language.SQ Albanian sq Language.SR Serbian sr Language.SU Sundanese su Language.SV Swedish sv Language.SW Swahili sw Language.TA Tamil ta Language.TE Telugu te Language.TG Tajik tg Language.TH Thai th Language.TK Turkmen tk Language.TL Tagalog tl Language.TR Turkish tr Language.TT Tatar tt Language.UK Ukrainian uk Language.UR Urdu ur Language.UZ Uzbek uz Language.VI Vietnamese vi Language.YI Yiddish yi Language.YO Yoruba yo Language.ZH Chinese zh For a complete list of supported languages, refer to Gladia’s documentation . ​ Advanced Features ​ Custom Vocabulary You can provide custom vocabulary items with bias intensity: Copy Ask AI from pipecat.services.gladia.config import CustomVocabularyItem, CustomVocabularyConfig, RealtimeProcessingConfig custom_vocab = CustomVocabularyConfig( vocabulary = [ CustomVocabularyItem( value = "Pipecat" , intensity = 0.8 ), CustomVocabularyItem( value = "Daily" , intensity = 0.7 ), ], default_intensity = 0.5 ) realtime_config = RealtimeProcessingConfig( custom_vocabulary = True , custom_vocabulary_config = custom_vocab ) ​ Translation Enable real-time translation: Copy Ask AI from pipecat.services.gladia.config import TranslationConfig, RealtimeProcessingConfig translation_config = TranslationConfig( target_languages = [ "fr" , "es" , "de" ], model = "enhanced" , match_original_utterances = True ) realtime_config = RealtimeProcessingConfig( translation = True , translation_config = translation_config ) ​ Multi-language Support Configure multiple languages with automatic language switching: Copy Ask AI from pipecat.services.gladia.config import LanguageConfig, GladiaInputParams language_config = LanguageConfig( languages = [ "en" , "fr" , "es" ], code_switching = True ) params = GladiaInputParams( language_config = language_config ) ​ Usage Example Copy Ask AI from pipecat.pipeline.pipeline import Pipeline from pipecat.services.gladia.stt import GladiaSTTService from pipecat.services.gladia.config import ( GladiaInputParams, LanguageConfig, RealtimeProcessingConfig ) from pipecat.transcriptions.language import Language # Configure the service stt = GladiaSTTService( api_key = "your-api-key" , model = "solaria-1" , params = GladiaInputParams( language_config = LanguageConfig( languages = [Language. EN , Language. FR ], code_switching = True ), ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ Frame Flow ​ Metrics Support The service collects processing metrics: Time to First Byte (TTFB) Processing duration Connection status ​ Notes Audio input must be in PCM format Transcription frames are only generated when confidence threshold is met Service automatically handles websocket connections and cleanup Real-time processing occurs in parallel for natural conversation flow Fal (Wizper) Google On this page Overview Installation Configuration Service Parameters GladiaInputParams LanguageConfig PreProcessingConfig CustomVocabularyConfig CustomSpellingConfig TranslationConfig RealtimeProcessingConfig MessagesConfig Input Output TranscriptionFrame InterimTranscriptionFrame ErrorFrame Methods Language Setting Language Support Advanced Features Custom Vocabulary Translation Multi-language Support Usage Example Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_gladia_5c9b248a.txt b/stt_gladia_5c9b248a.txt
new file mode 100644
index 0000000000000000000000000000000000000000..10a8f1d90887609df0e96527e388c636de66b855
--- /dev/null
+++ b/stt_gladia_5c9b248a.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/gladia#messagesconfig
+Title: Gladia - Pipecat
+==================================================
+
+Gladia - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text Gladia Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview GladiaSTTService is a speech-to-text (STT) service that integrates with Gladia’s API to provide real-time transcription capabilities. It processes audio input and produces transcription frames in real-time with support for multiple languages, custom vocabulary, and various processing options. ​ Installation To use GladiaSTTService , you need to install the Gladia dependencies: Copy Ask AI pip install "pipecat-ai[gladia]" You’ll also need to set up your Gladia API key as an environment variable: GLADIA_API_KEY . ​ Configuration ​ Service Parameters ​ api_key string required Your Gladia API key for authentication ​ url string default: "https://api.gladia.io/v2/live" Gladia API endpoint URL ​ confidence float default: "0.5" Minimum confidence threshold to create interim and final transcriptions. Values range from 0 to 1. ​ sample_rate integer default: "None" Audio sample rate in Hz ​ model string default: "solaria-1" Model to use for transcription. Options include solaria-1 solaria-mini-1 fast accurate See Gladia’s docs for the latest supported models. ​ params GladiaInputParams default: "GladiaInputParams()" Additional configuration parameters for the service ​ GladiaInputParams ​ encoding string default: "wav/pcm" Audio encoding format ​ bit_depth integer default: "16" Audio bit depth ​ channels integer default: "1" Number of audio channels ​ custom_metadata Dict[str, Any] Additional metadata to include with requests ​ endpointing float Silence duration in seconds to mark end of speech ​ maximum_duration_without_endpointing integer default: "10" Maximum utterance duration without silence ​ language Language deprecated Primary language for transcription. Deprecated: use language_config instead. ​ language_config LanguageConfig Detailed language configuration ​ pre_processing PreProcessingConfig Audio pre-processing options ​ realtime_processing RealtimeProcessingConfig Real-time processing features ​ messages_config MessagesConfig WebSocket message filtering options ​ LanguageConfig ​ languages List[str] Specify language(s) for transcription. If one language is set, it will be used for all transcription. If multiple languages are provided or none, language will be auto-detected by the model. ​ code_switching boolean default: "false" If true, language will be auto-detected on each utterance. Otherwise, language will be auto-detected on first utterance and then used for the rest of the transcription. If one language is set, this option will be ignored. ​ PreProcessingConfig ​ speech_threshold float default: "0.8" Sensitivity configuration for Speech Threshold. A value close to 1 will apply stricter thresholds, making it less likely to detect background sounds as speech. Must be between 0 and 1. ​ CustomVocabularyConfig ​ vocabulary List[Union[str, CustomVocabularyItem]] required Specific vocabulary list to feed the transcription model with. Can be a list of strings or CustomVocabularyItem objects. ​ default_intensity float Default intensity for the custom vocabulary. Must be between 0 and 1. ​ CustomSpellingConfig ​ spelling_dictionary Dict[str, List[str]] required The list of spelling rules applied on the audio transcription. Keys are the correct spellings and values are lists of phonetic variations. ​ TranslationConfig ​ target_languages List[str] required The target language(s) in ISO639-1 format (e.g., “en”, “fr”, “es”) ​ model string default: "base" Translation model to use. Options: “base” or “enhanced” ​ match_original_utterances boolean default: "true" Align translated utterances with the original ones ​ RealtimeProcessingConfig ​ words_accurate_timestamps boolean Whether to provide per-word timestamps ​ custom_vocabulary boolean Whether to enable custom vocabulary ​ custom_vocabulary_config CustomVocabularyConfig Custom vocabulary configuration ​ custom_spelling boolean Whether to enable custom spelling ​ custom_spelling_config CustomSpellingConfig Custom spelling configuration ​ translation boolean Whether to enable translation ​ translation_config TranslationConfig Translation configuration ​ named_entity_recognition boolean Whether to enable named entity recognition ​ sentiment_analysis boolean Whether to enable sentiment analysis ​ MessagesConfig ​ receive_partial_transcripts boolean default: "true" If true, partial utterances will be sent via WebSocket ​ receive_final_transcripts boolean default: "true" If true, final utterances will be sent via WebSocket ​ receive_speech_events boolean default: "true" If true, begin and end speech events will be sent via WebSocket ​ receive_pre_processing_events boolean default: "true" If true, pre-processing events will be sent via WebSocket ​ receive_realtime_processing_events boolean default: "true" If true, realtime processing events will be sent via WebSocket ​ receive_post_processing_events boolean default: "true" If true, post-processing events will be sent via WebSocket ​ receive_acknowledgments boolean default: "true" If true, acknowledgments will be sent via WebSocket ​ receive_errors boolean default: "true" If true, errors will be sent via WebSocket ​ receive_lifecycle_events boolean default: "false" If true, lifecycle events will be sent via WebSocket ​ Input The service processes raw audio data with the following requirements: PCM audio format 16-bit depth 16kHz sample rate (default) Single channel (mono) ​ Output The service produces two types of frames during transcription: ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Transcription language ​ InterimTranscriptionFrame Generated during ongoing speech, containing the same fields as TranscriptionFrame but with preliminary results. ​ ErrorFrame Generated when transcription errors occur, containing error details. ​ Methods See the STT base class methods for additional functionality. ​ Language Setting Copy Ask AI await service.set_language(Language. FR ) ​ Language Support Gladia STT supports a wide range of languages. Here’s a partial list: Language Code Description Service Code Language.AF Afrikaans af Language.AM Amharic am Language.AR Arabic ar Language.AS Assamese as Language.AZ Azerbaijani az Language.BA Bashkir ba Language.BE Belarusian be Language.BG Bulgarian bg Language.BN Bengali bn Language.BO Tibetan bo Language.BR Breton br Language.BS Bosnian bs Language.CA Catalan ca Language.CS Czech cs Language.CY Welsh cy Language.DA Danish da Language.DE German de Language.EL Greek el Language.EN English en Language.ES Spanish es Language.ET Estonian et Language.EU Basque eu Language.FA Persian fa Language.FI Finnish fi Language.FO Faroese fo Language.FR French fr Language.GL Galician gl Language.GU Gujarati gu Language.HA Hausa ha Language.HAW Hawaiian haw Language.HE Hebrew he Language.HI Hindi hi Language.HR Croatian hr Language.HT Haitian Creole ht Language.HU Hungarian hu Language.HY Armenian hy Language.ID Indonesian id Language.IS Icelandic is Language.IT Italian it Language.JA Japanese ja Language.JV Javanese jv Language.KA Georgian ka Language.KK Kazakh kk Language.KM Khmer km Language.KN Kannada kn Language.KO Korean ko Language.LA Latin la Language.LB Luxembourgish lb Language.LN Lingala ln Language.LO Lao lo Language.LT Lithuanian lt Language.LV Latvian lv Language.MG Malagasy mg Language.MI Maori mi Language.MK Macedonian mk Language.ML Malayalam ml Language.MN Mongolian mn Language.MR Marathi mr Language.MS Malay ms Language.MT Maltese mt Language.MY_MR Burmese mymr Language.NE Nepali ne Language.NL Dutch nl Language.NN Norwegian (Nynorsk) nn Language.NO Norwegian no Language.OC Occitan oc Language.PA Punjabi pa Language.PL Polish pl Language.PS Pashto ps Language.PT Portuguese pt Language.RO Romanian ro Language.RU Russian ru Language.SA Sanskrit sa Language.SD Sindhi sd Language.SI Sinhala si Language.SK Slovak sk Language.SL Slovenian sl Language.SN Shona sn Language.SO Somali so Language.SQ Albanian sq Language.SR Serbian sr Language.SU Sundanese su Language.SV Swedish sv Language.SW Swahili sw Language.TA Tamil ta Language.TE Telugu te Language.TG Tajik tg Language.TH Thai th Language.TK Turkmen tk Language.TL Tagalog tl Language.TR Turkish tr Language.TT Tatar tt Language.UK Ukrainian uk Language.UR Urdu ur Language.UZ Uzbek uz Language.VI Vietnamese vi Language.YI Yiddish yi Language.YO Yoruba yo Language.ZH Chinese zh For a complete list of supported languages, refer to Gladia’s documentation . ​ Advanced Features ​ Custom Vocabulary You can provide custom vocabulary items with bias intensity: Copy Ask AI from pipecat.services.gladia.config import CustomVocabularyItem, CustomVocabularyConfig, RealtimeProcessingConfig custom_vocab = CustomVocabularyConfig( vocabulary = [ CustomVocabularyItem( value = "Pipecat" , intensity = 0.8 ), CustomVocabularyItem( value = "Daily" , intensity = 0.7 ), ], default_intensity = 0.5 ) realtime_config = RealtimeProcessingConfig( custom_vocabulary = True , custom_vocabulary_config = custom_vocab ) ​ Translation Enable real-time translation: Copy Ask AI from pipecat.services.gladia.config import TranslationConfig, RealtimeProcessingConfig translation_config = TranslationConfig( target_languages = [ "fr" , "es" , "de" ], model = "enhanced" , match_original_utterances = True ) realtime_config = RealtimeProcessingConfig( translation = True , translation_config = translation_config ) ​ Multi-language Support Configure multiple languages with automatic language switching: Copy Ask AI from pipecat.services.gladia.config import LanguageConfig, GladiaInputParams language_config = LanguageConfig( languages = [ "en" , "fr" , "es" ], code_switching = True ) params = GladiaInputParams( language_config = language_config ) ​ Usage Example Copy Ask AI from pipecat.pipeline.pipeline import Pipeline from pipecat.services.gladia.stt import GladiaSTTService from pipecat.services.gladia.config import ( GladiaInputParams, LanguageConfig, RealtimeProcessingConfig ) from pipecat.transcriptions.language import Language # Configure the service stt = GladiaSTTService( api_key = "your-api-key" , model = "solaria-1" , params = GladiaInputParams( language_config = LanguageConfig( languages = [Language. EN , Language. FR ], code_switching = True ), ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ Frame Flow ​ Metrics Support The service collects processing metrics: Time to First Byte (TTFB) Processing duration Connection status ​ Notes Audio input must be in PCM format Transcription frames are only generated when confidence threshold is met Service automatically handles websocket connections and cleanup Real-time processing occurs in parallel for natural conversation flow Fal (Wizper) Google On this page Overview Installation Configuration Service Parameters GladiaInputParams LanguageConfig PreProcessingConfig CustomVocabularyConfig CustomSpellingConfig TranslationConfig RealtimeProcessingConfig MessagesConfig Input Output TranscriptionFrame InterimTranscriptionFrame ErrorFrame Methods Language Setting Language Support Advanced Features Custom Vocabulary Translation Multi-language Support Usage Example Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_gladia_6930b7bb.txt b/stt_gladia_6930b7bb.txt
new file mode 100644
index 0000000000000000000000000000000000000000..4fc5e1c6e23b23dd995b0814cfce0a6b731ba360
--- /dev/null
+++ b/stt_gladia_6930b7bb.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/gladia#param-language-1
+Title: Gladia - Pipecat
+==================================================
+
+Gladia - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text Gladia Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview GladiaSTTService is a speech-to-text (STT) service that integrates with Gladia’s API to provide real-time transcription capabilities. It processes audio input and produces transcription frames in real-time with support for multiple languages, custom vocabulary, and various processing options. ​ Installation To use GladiaSTTService , you need to install the Gladia dependencies: Copy Ask AI pip install "pipecat-ai[gladia]" You’ll also need to set up your Gladia API key as an environment variable: GLADIA_API_KEY . ​ Configuration ​ Service Parameters ​ api_key string required Your Gladia API key for authentication ​ url string default: "https://api.gladia.io/v2/live" Gladia API endpoint URL ​ confidence float default: "0.5" Minimum confidence threshold to create interim and final transcriptions. Values range from 0 to 1. ​ sample_rate integer default: "None" Audio sample rate in Hz ​ model string default: "solaria-1" Model to use for transcription. Options include solaria-1 solaria-mini-1 fast accurate See Gladia’s docs for the latest supported models. ​ params GladiaInputParams default: "GladiaInputParams()" Additional configuration parameters for the service ​ GladiaInputParams ​ encoding string default: "wav/pcm" Audio encoding format ​ bit_depth integer default: "16" Audio bit depth ​ channels integer default: "1" Number of audio channels ​ custom_metadata Dict[str, Any] Additional metadata to include with requests ​ endpointing float Silence duration in seconds to mark end of speech ​ maximum_duration_without_endpointing integer default: "10" Maximum utterance duration without silence ​ language Language deprecated Primary language for transcription. Deprecated: use language_config instead. ​ language_config LanguageConfig Detailed language configuration ​ pre_processing PreProcessingConfig Audio pre-processing options ​ realtime_processing RealtimeProcessingConfig Real-time processing features ​ messages_config MessagesConfig WebSocket message filtering options ​ LanguageConfig ​ languages List[str] Specify language(s) for transcription. If one language is set, it will be used for all transcription. If multiple languages are provided or none, language will be auto-detected by the model. ​ code_switching boolean default: "false" If true, language will be auto-detected on each utterance. Otherwise, language will be auto-detected on first utterance and then used for the rest of the transcription. If one language is set, this option will be ignored. ​ PreProcessingConfig ​ speech_threshold float default: "0.8" Sensitivity configuration for Speech Threshold. A value close to 1 will apply stricter thresholds, making it less likely to detect background sounds as speech. Must be between 0 and 1. ​ CustomVocabularyConfig ​ vocabulary List[Union[str, CustomVocabularyItem]] required Specific vocabulary list to feed the transcription model with. Can be a list of strings or CustomVocabularyItem objects. ​ default_intensity float Default intensity for the custom vocabulary. Must be between 0 and 1. ​ CustomSpellingConfig ​ spelling_dictionary Dict[str, List[str]] required The list of spelling rules applied on the audio transcription. Keys are the correct spellings and values are lists of phonetic variations. ​ TranslationConfig ​ target_languages List[str] required The target language(s) in ISO639-1 format (e.g., “en”, “fr”, “es”) ​ model string default: "base" Translation model to use. Options: “base” or “enhanced” ​ match_original_utterances boolean default: "true" Align translated utterances with the original ones ​ RealtimeProcessingConfig ​ words_accurate_timestamps boolean Whether to provide per-word timestamps ​ custom_vocabulary boolean Whether to enable custom vocabulary ​ custom_vocabulary_config CustomVocabularyConfig Custom vocabulary configuration ​ custom_spelling boolean Whether to enable custom spelling ​ custom_spelling_config CustomSpellingConfig Custom spelling configuration ​ translation boolean Whether to enable translation ​ translation_config TranslationConfig Translation configuration ​ named_entity_recognition boolean Whether to enable named entity recognition ​ sentiment_analysis boolean Whether to enable sentiment analysis ​ MessagesConfig ​ receive_partial_transcripts boolean default: "true" If true, partial utterances will be sent via WebSocket ​ receive_final_transcripts boolean default: "true" If true, final utterances will be sent via WebSocket ​ receive_speech_events boolean default: "true" If true, begin and end speech events will be sent via WebSocket ​ receive_pre_processing_events boolean default: "true" If true, pre-processing events will be sent via WebSocket ​ receive_realtime_processing_events boolean default: "true" If true, realtime processing events will be sent via WebSocket ​ receive_post_processing_events boolean default: "true" If true, post-processing events will be sent via WebSocket ​ receive_acknowledgments boolean default: "true" If true, acknowledgments will be sent via WebSocket ​ receive_errors boolean default: "true" If true, errors will be sent via WebSocket ​ receive_lifecycle_events boolean default: "false" If true, lifecycle events will be sent via WebSocket ​ Input The service processes raw audio data with the following requirements: PCM audio format 16-bit depth 16kHz sample rate (default) Single channel (mono) ​ Output The service produces two types of frames during transcription: ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Transcription language ​ InterimTranscriptionFrame Generated during ongoing speech, containing the same fields as TranscriptionFrame but with preliminary results. ​ ErrorFrame Generated when transcription errors occur, containing error details. ​ Methods See the STT base class methods for additional functionality. ​ Language Setting Copy Ask AI await service.set_language(Language. FR ) ​ Language Support Gladia STT supports a wide range of languages. Here’s a partial list: Language Code Description Service Code Language.AF Afrikaans af Language.AM Amharic am Language.AR Arabic ar Language.AS Assamese as Language.AZ Azerbaijani az Language.BA Bashkir ba Language.BE Belarusian be Language.BG Bulgarian bg Language.BN Bengali bn Language.BO Tibetan bo Language.BR Breton br Language.BS Bosnian bs Language.CA Catalan ca Language.CS Czech cs Language.CY Welsh cy Language.DA Danish da Language.DE German de Language.EL Greek el Language.EN English en Language.ES Spanish es Language.ET Estonian et Language.EU Basque eu Language.FA Persian fa Language.FI Finnish fi Language.FO Faroese fo Language.FR French fr Language.GL Galician gl Language.GU Gujarati gu Language.HA Hausa ha Language.HAW Hawaiian haw Language.HE Hebrew he Language.HI Hindi hi Language.HR Croatian hr Language.HT Haitian Creole ht Language.HU Hungarian hu Language.HY Armenian hy Language.ID Indonesian id Language.IS Icelandic is Language.IT Italian it Language.JA Japanese ja Language.JV Javanese jv Language.KA Georgian ka Language.KK Kazakh kk Language.KM Khmer km Language.KN Kannada kn Language.KO Korean ko Language.LA Latin la Language.LB Luxembourgish lb Language.LN Lingala ln Language.LO Lao lo Language.LT Lithuanian lt Language.LV Latvian lv Language.MG Malagasy mg Language.MI Maori mi Language.MK Macedonian mk Language.ML Malayalam ml Language.MN Mongolian mn Language.MR Marathi mr Language.MS Malay ms Language.MT Maltese mt Language.MY_MR Burmese mymr Language.NE Nepali ne Language.NL Dutch nl Language.NN Norwegian (Nynorsk) nn Language.NO Norwegian no Language.OC Occitan oc Language.PA Punjabi pa Language.PL Polish pl Language.PS Pashto ps Language.PT Portuguese pt Language.RO Romanian ro Language.RU Russian ru Language.SA Sanskrit sa Language.SD Sindhi sd Language.SI Sinhala si Language.SK Slovak sk Language.SL Slovenian sl Language.SN Shona sn Language.SO Somali so Language.SQ Albanian sq Language.SR Serbian sr Language.SU Sundanese su Language.SV Swedish sv Language.SW Swahili sw Language.TA Tamil ta Language.TE Telugu te Language.TG Tajik tg Language.TH Thai th Language.TK Turkmen tk Language.TL Tagalog tl Language.TR Turkish tr Language.TT Tatar tt Language.UK Ukrainian uk Language.UR Urdu ur Language.UZ Uzbek uz Language.VI Vietnamese vi Language.YI Yiddish yi Language.YO Yoruba yo Language.ZH Chinese zh For a complete list of supported languages, refer to Gladia’s documentation . ​ Advanced Features ​ Custom Vocabulary You can provide custom vocabulary items with bias intensity: Copy Ask AI from pipecat.services.gladia.config import CustomVocabularyItem, CustomVocabularyConfig, RealtimeProcessingConfig custom_vocab = CustomVocabularyConfig( vocabulary = [ CustomVocabularyItem( value = "Pipecat" , intensity = 0.8 ), CustomVocabularyItem( value = "Daily" , intensity = 0.7 ), ], default_intensity = 0.5 ) realtime_config = RealtimeProcessingConfig( custom_vocabulary = True , custom_vocabulary_config = custom_vocab ) ​ Translation Enable real-time translation: Copy Ask AI from pipecat.services.gladia.config import TranslationConfig, RealtimeProcessingConfig translation_config = TranslationConfig( target_languages = [ "fr" , "es" , "de" ], model = "enhanced" , match_original_utterances = True ) realtime_config = RealtimeProcessingConfig( translation = True , translation_config = translation_config ) ​ Multi-language Support Configure multiple languages with automatic language switching: Copy Ask AI from pipecat.services.gladia.config import LanguageConfig, GladiaInputParams language_config = LanguageConfig( languages = [ "en" , "fr" , "es" ], code_switching = True ) params = GladiaInputParams( language_config = language_config ) ​ Usage Example Copy Ask AI from pipecat.pipeline.pipeline import Pipeline from pipecat.services.gladia.stt import GladiaSTTService from pipecat.services.gladia.config import ( GladiaInputParams, LanguageConfig, RealtimeProcessingConfig ) from pipecat.transcriptions.language import Language # Configure the service stt = GladiaSTTService( api_key = "your-api-key" , model = "solaria-1" , params = GladiaInputParams( language_config = LanguageConfig( languages = [Language. EN , Language. FR ], code_switching = True ), ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ Frame Flow ​ Metrics Support The service collects processing metrics: Time to First Byte (TTFB) Processing duration Connection status ​ Notes Audio input must be in PCM format Transcription frames are only generated when confidence threshold is met Service automatically handles websocket connections and cleanup Real-time processing occurs in parallel for natural conversation flow Fal (Wizper) Google On this page Overview Installation Configuration Service Parameters GladiaInputParams LanguageConfig PreProcessingConfig CustomVocabularyConfig CustomSpellingConfig TranslationConfig RealtimeProcessingConfig MessagesConfig Input Output TranscriptionFrame InterimTranscriptionFrame ErrorFrame Methods Language Setting Language Support Advanced Features Custom Vocabulary Translation Multi-language Support Usage Example Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_gladia_6cf819ee.txt b/stt_gladia_6cf819ee.txt
new file mode 100644
index 0000000000000000000000000000000000000000..ff0fcb6b1c055daee4bf5540e75cfd94b5683c26
--- /dev/null
+++ b/stt_gladia_6cf819ee.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/gladia#notes
+Title: Gladia - Pipecat
+==================================================
+
+Gladia - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text Gladia Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview GladiaSTTService is a speech-to-text (STT) service that integrates with Gladia’s API to provide real-time transcription capabilities. It processes audio input and produces transcription frames in real-time with support for multiple languages, custom vocabulary, and various processing options. ​ Installation To use GladiaSTTService , you need to install the Gladia dependencies: Copy Ask AI pip install "pipecat-ai[gladia]" You’ll also need to set up your Gladia API key as an environment variable: GLADIA_API_KEY . ​ Configuration ​ Service Parameters ​ api_key string required Your Gladia API key for authentication ​ url string default: "https://api.gladia.io/v2/live" Gladia API endpoint URL ​ confidence float default: "0.5" Minimum confidence threshold to create interim and final transcriptions. Values range from 0 to 1. ​ sample_rate integer default: "None" Audio sample rate in Hz ​ model string default: "solaria-1" Model to use for transcription. Options include solaria-1 solaria-mini-1 fast accurate See Gladia’s docs for the latest supported models. ​ params GladiaInputParams default: "GladiaInputParams()" Additional configuration parameters for the service ​ GladiaInputParams ​ encoding string default: "wav/pcm" Audio encoding format ​ bit_depth integer default: "16" Audio bit depth ​ channels integer default: "1" Number of audio channels ​ custom_metadata Dict[str, Any] Additional metadata to include with requests ​ endpointing float Silence duration in seconds to mark end of speech ​ maximum_duration_without_endpointing integer default: "10" Maximum utterance duration without silence ​ language Language deprecated Primary language for transcription. Deprecated: use language_config instead. ​ language_config LanguageConfig Detailed language configuration ​ pre_processing PreProcessingConfig Audio pre-processing options ​ realtime_processing RealtimeProcessingConfig Real-time processing features ​ messages_config MessagesConfig WebSocket message filtering options ​ LanguageConfig ​ languages List[str] Specify language(s) for transcription. If one language is set, it will be used for all transcription. If multiple languages are provided or none, language will be auto-detected by the model. ​ code_switching boolean default: "false" If true, language will be auto-detected on each utterance. Otherwise, language will be auto-detected on first utterance and then used for the rest of the transcription. If one language is set, this option will be ignored. ​ PreProcessingConfig ​ speech_threshold float default: "0.8" Sensitivity configuration for Speech Threshold. A value close to 1 will apply stricter thresholds, making it less likely to detect background sounds as speech. Must be between 0 and 1. ​ CustomVocabularyConfig ​ vocabulary List[Union[str, CustomVocabularyItem]] required Specific vocabulary list to feed the transcription model with. Can be a list of strings or CustomVocabularyItem objects. ​ default_intensity float Default intensity for the custom vocabulary. Must be between 0 and 1. ​ CustomSpellingConfig ​ spelling_dictionary Dict[str, List[str]] required The list of spelling rules applied on the audio transcription. Keys are the correct spellings and values are lists of phonetic variations. ​ TranslationConfig ​ target_languages List[str] required The target language(s) in ISO639-1 format (e.g., “en”, “fr”, “es”) ​ model string default: "base" Translation model to use. Options: “base” or “enhanced” ​ match_original_utterances boolean default: "true" Align translated utterances with the original ones ​ RealtimeProcessingConfig ​ words_accurate_timestamps boolean Whether to provide per-word timestamps ​ custom_vocabulary boolean Whether to enable custom vocabulary ​ custom_vocabulary_config CustomVocabularyConfig Custom vocabulary configuration ​ custom_spelling boolean Whether to enable custom spelling ​ custom_spelling_config CustomSpellingConfig Custom spelling configuration ​ translation boolean Whether to enable translation ​ translation_config TranslationConfig Translation configuration ​ named_entity_recognition boolean Whether to enable named entity recognition ​ sentiment_analysis boolean Whether to enable sentiment analysis ​ MessagesConfig ​ receive_partial_transcripts boolean default: "true" If true, partial utterances will be sent via WebSocket ​ receive_final_transcripts boolean default: "true" If true, final utterances will be sent via WebSocket ​ receive_speech_events boolean default: "true" If true, begin and end speech events will be sent via WebSocket ​ receive_pre_processing_events boolean default: "true" If true, pre-processing events will be sent via WebSocket ​ receive_realtime_processing_events boolean default: "true" If true, realtime processing events will be sent via WebSocket ​ receive_post_processing_events boolean default: "true" If true, post-processing events will be sent via WebSocket ​ receive_acknowledgments boolean default: "true" If true, acknowledgments will be sent via WebSocket ​ receive_errors boolean default: "true" If true, errors will be sent via WebSocket ​ receive_lifecycle_events boolean default: "false" If true, lifecycle events will be sent via WebSocket ​ Input The service processes raw audio data with the following requirements: PCM audio format 16-bit depth 16kHz sample rate (default) Single channel (mono) ​ Output The service produces two types of frames during transcription: ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Transcription language ​ InterimTranscriptionFrame Generated during ongoing speech, containing the same fields as TranscriptionFrame but with preliminary results. ​ ErrorFrame Generated when transcription errors occur, containing error details. ​ Methods See the STT base class methods for additional functionality. ​ Language Setting Copy Ask AI await service.set_language(Language. FR ) ​ Language Support Gladia STT supports a wide range of languages. Here’s a partial list: Language Code Description Service Code Language.AF Afrikaans af Language.AM Amharic am Language.AR Arabic ar Language.AS Assamese as Language.AZ Azerbaijani az Language.BA Bashkir ba Language.BE Belarusian be Language.BG Bulgarian bg Language.BN Bengali bn Language.BO Tibetan bo Language.BR Breton br Language.BS Bosnian bs Language.CA Catalan ca Language.CS Czech cs Language.CY Welsh cy Language.DA Danish da Language.DE German de Language.EL Greek el Language.EN English en Language.ES Spanish es Language.ET Estonian et Language.EU Basque eu Language.FA Persian fa Language.FI Finnish fi Language.FO Faroese fo Language.FR French fr Language.GL Galician gl Language.GU Gujarati gu Language.HA Hausa ha Language.HAW Hawaiian haw Language.HE Hebrew he Language.HI Hindi hi Language.HR Croatian hr Language.HT Haitian Creole ht Language.HU Hungarian hu Language.HY Armenian hy Language.ID Indonesian id Language.IS Icelandic is Language.IT Italian it Language.JA Japanese ja Language.JV Javanese jv Language.KA Georgian ka Language.KK Kazakh kk Language.KM Khmer km Language.KN Kannada kn Language.KO Korean ko Language.LA Latin la Language.LB Luxembourgish lb Language.LN Lingala ln Language.LO Lao lo Language.LT Lithuanian lt Language.LV Latvian lv Language.MG Malagasy mg Language.MI Maori mi Language.MK Macedonian mk Language.ML Malayalam ml Language.MN Mongolian mn Language.MR Marathi mr Language.MS Malay ms Language.MT Maltese mt Language.MY_MR Burmese mymr Language.NE Nepali ne Language.NL Dutch nl Language.NN Norwegian (Nynorsk) nn Language.NO Norwegian no Language.OC Occitan oc Language.PA Punjabi pa Language.PL Polish pl Language.PS Pashto ps Language.PT Portuguese pt Language.RO Romanian ro Language.RU Russian ru Language.SA Sanskrit sa Language.SD Sindhi sd Language.SI Sinhala si Language.SK Slovak sk Language.SL Slovenian sl Language.SN Shona sn Language.SO Somali so Language.SQ Albanian sq Language.SR Serbian sr Language.SU Sundanese su Language.SV Swedish sv Language.SW Swahili sw Language.TA Tamil ta Language.TE Telugu te Language.TG Tajik tg Language.TH Thai th Language.TK Turkmen tk Language.TL Tagalog tl Language.TR Turkish tr Language.TT Tatar tt Language.UK Ukrainian uk Language.UR Urdu ur Language.UZ Uzbek uz Language.VI Vietnamese vi Language.YI Yiddish yi Language.YO Yoruba yo Language.ZH Chinese zh For a complete list of supported languages, refer to Gladia’s documentation . ​ Advanced Features ​ Custom Vocabulary You can provide custom vocabulary items with bias intensity: Copy Ask AI from pipecat.services.gladia.config import CustomVocabularyItem, CustomVocabularyConfig, RealtimeProcessingConfig custom_vocab = CustomVocabularyConfig( vocabulary = [ CustomVocabularyItem( value = "Pipecat" , intensity = 0.8 ), CustomVocabularyItem( value = "Daily" , intensity = 0.7 ), ], default_intensity = 0.5 ) realtime_config = RealtimeProcessingConfig( custom_vocabulary = True , custom_vocabulary_config = custom_vocab ) ​ Translation Enable real-time translation: Copy Ask AI from pipecat.services.gladia.config import TranslationConfig, RealtimeProcessingConfig translation_config = TranslationConfig( target_languages = [ "fr" , "es" , "de" ], model = "enhanced" , match_original_utterances = True ) realtime_config = RealtimeProcessingConfig( translation = True , translation_config = translation_config ) ​ Multi-language Support Configure multiple languages with automatic language switching: Copy Ask AI from pipecat.services.gladia.config import LanguageConfig, GladiaInputParams language_config = LanguageConfig( languages = [ "en" , "fr" , "es" ], code_switching = True ) params = GladiaInputParams( language_config = language_config ) ​ Usage Example Copy Ask AI from pipecat.pipeline.pipeline import Pipeline from pipecat.services.gladia.stt import GladiaSTTService from pipecat.services.gladia.config import ( GladiaInputParams, LanguageConfig, RealtimeProcessingConfig ) from pipecat.transcriptions.language import Language # Configure the service stt = GladiaSTTService( api_key = "your-api-key" , model = "solaria-1" , params = GladiaInputParams( language_config = LanguageConfig( languages = [Language. EN , Language. FR ], code_switching = True ), ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ Frame Flow ​ Metrics Support The service collects processing metrics: Time to First Byte (TTFB) Processing duration Connection status ​ Notes Audio input must be in PCM format Transcription frames are only generated when confidence threshold is met Service automatically handles websocket connections and cleanup Real-time processing occurs in parallel for natural conversation flow Fal (Wizper) Google On this page Overview Installation Configuration Service Parameters GladiaInputParams LanguageConfig PreProcessingConfig CustomVocabularyConfig CustomSpellingConfig TranslationConfig RealtimeProcessingConfig MessagesConfig Input Output TranscriptionFrame InterimTranscriptionFrame ErrorFrame Methods Language Setting Language Support Advanced Features Custom Vocabulary Translation Multi-language Support Usage Example Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_gladia_88aa8677.txt b/stt_gladia_88aa8677.txt
new file mode 100644
index 0000000000000000000000000000000000000000..ffd426156761e3ca59524b35b9d8ca9157b2c441
--- /dev/null
+++ b/stt_gladia_88aa8677.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/gladia#gladiainputparams
+Title: Gladia - Pipecat
+==================================================
+
+Gladia - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text Gladia Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview GladiaSTTService is a speech-to-text (STT) service that integrates with Gladia’s API to provide real-time transcription capabilities. It processes audio input and produces transcription frames in real-time with support for multiple languages, custom vocabulary, and various processing options. ​ Installation To use GladiaSTTService , you need to install the Gladia dependencies: Copy Ask AI pip install "pipecat-ai[gladia]" You’ll also need to set up your Gladia API key as an environment variable: GLADIA_API_KEY . ​ Configuration ​ Service Parameters ​ api_key string required Your Gladia API key for authentication ​ url string default: "https://api.gladia.io/v2/live" Gladia API endpoint URL ​ confidence float default: "0.5" Minimum confidence threshold to create interim and final transcriptions. Values range from 0 to 1. ​ sample_rate integer default: "None" Audio sample rate in Hz ​ model string default: "solaria-1" Model to use for transcription. Options include solaria-1 solaria-mini-1 fast accurate See Gladia’s docs for the latest supported models. ​ params GladiaInputParams default: "GladiaInputParams()" Additional configuration parameters for the service ​ GladiaInputParams ​ encoding string default: "wav/pcm" Audio encoding format ​ bit_depth integer default: "16" Audio bit depth ​ channels integer default: "1" Number of audio channels ​ custom_metadata Dict[str, Any] Additional metadata to include with requests ​ endpointing float Silence duration in seconds to mark end of speech ​ maximum_duration_without_endpointing integer default: "10" Maximum utterance duration without silence ​ language Language deprecated Primary language for transcription. Deprecated: use language_config instead. ​ language_config LanguageConfig Detailed language configuration ​ pre_processing PreProcessingConfig Audio pre-processing options ​ realtime_processing RealtimeProcessingConfig Real-time processing features ​ messages_config MessagesConfig WebSocket message filtering options ​ LanguageConfig ​ languages List[str] Specify language(s) for transcription. If one language is set, it will be used for all transcription. If multiple languages are provided or none, language will be auto-detected by the model. ​ code_switching boolean default: "false" If true, language will be auto-detected on each utterance. Otherwise, language will be auto-detected on first utterance and then used for the rest of the transcription. If one language is set, this option will be ignored. ​ PreProcessingConfig ​ speech_threshold float default: "0.8" Sensitivity configuration for Speech Threshold. A value close to 1 will apply stricter thresholds, making it less likely to detect background sounds as speech. Must be between 0 and 1. ​ CustomVocabularyConfig ​ vocabulary List[Union[str, CustomVocabularyItem]] required Specific vocabulary list to feed the transcription model with. Can be a list of strings or CustomVocabularyItem objects. ​ default_intensity float Default intensity for the custom vocabulary. Must be between 0 and 1. ​ CustomSpellingConfig ​ spelling_dictionary Dict[str, List[str]] required The list of spelling rules applied on the audio transcription. Keys are the correct spellings and values are lists of phonetic variations. ​ TranslationConfig ​ target_languages List[str] required The target language(s) in ISO639-1 format (e.g., “en”, “fr”, “es”) ​ model string default: "base" Translation model to use. Options: “base” or “enhanced” ​ match_original_utterances boolean default: "true" Align translated utterances with the original ones ​ RealtimeProcessingConfig ​ words_accurate_timestamps boolean Whether to provide per-word timestamps ​ custom_vocabulary boolean Whether to enable custom vocabulary ​ custom_vocabulary_config CustomVocabularyConfig Custom vocabulary configuration ​ custom_spelling boolean Whether to enable custom spelling ​ custom_spelling_config CustomSpellingConfig Custom spelling configuration ​ translation boolean Whether to enable translation ​ translation_config TranslationConfig Translation configuration ​ named_entity_recognition boolean Whether to enable named entity recognition ​ sentiment_analysis boolean Whether to enable sentiment analysis ​ MessagesConfig ​ receive_partial_transcripts boolean default: "true" If true, partial utterances will be sent via WebSocket ​ receive_final_transcripts boolean default: "true" If true, final utterances will be sent via WebSocket ​ receive_speech_events boolean default: "true" If true, begin and end speech events will be sent via WebSocket ​ receive_pre_processing_events boolean default: "true" If true, pre-processing events will be sent via WebSocket ​ receive_realtime_processing_events boolean default: "true" If true, realtime processing events will be sent via WebSocket ​ receive_post_processing_events boolean default: "true" If true, post-processing events will be sent via WebSocket ​ receive_acknowledgments boolean default: "true" If true, acknowledgments will be sent via WebSocket ​ receive_errors boolean default: "true" If true, errors will be sent via WebSocket ​ receive_lifecycle_events boolean default: "false" If true, lifecycle events will be sent via WebSocket ​ Input The service processes raw audio data with the following requirements: PCM audio format 16-bit depth 16kHz sample rate (default) Single channel (mono) ​ Output The service produces two types of frames during transcription: ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Transcription language ​ InterimTranscriptionFrame Generated during ongoing speech, containing the same fields as TranscriptionFrame but with preliminary results. ​ ErrorFrame Generated when transcription errors occur, containing error details. ​ Methods See the STT base class methods for additional functionality. ​ Language Setting Copy Ask AI await service.set_language(Language. FR ) ​ Language Support Gladia STT supports a wide range of languages. Here’s a partial list: Language Code Description Service Code Language.AF Afrikaans af Language.AM Amharic am Language.AR Arabic ar Language.AS Assamese as Language.AZ Azerbaijani az Language.BA Bashkir ba Language.BE Belarusian be Language.BG Bulgarian bg Language.BN Bengali bn Language.BO Tibetan bo Language.BR Breton br Language.BS Bosnian bs Language.CA Catalan ca Language.CS Czech cs Language.CY Welsh cy Language.DA Danish da Language.DE German de Language.EL Greek el Language.EN English en Language.ES Spanish es Language.ET Estonian et Language.EU Basque eu Language.FA Persian fa Language.FI Finnish fi Language.FO Faroese fo Language.FR French fr Language.GL Galician gl Language.GU Gujarati gu Language.HA Hausa ha Language.HAW Hawaiian haw Language.HE Hebrew he Language.HI Hindi hi Language.HR Croatian hr Language.HT Haitian Creole ht Language.HU Hungarian hu Language.HY Armenian hy Language.ID Indonesian id Language.IS Icelandic is Language.IT Italian it Language.JA Japanese ja Language.JV Javanese jv Language.KA Georgian ka Language.KK Kazakh kk Language.KM Khmer km Language.KN Kannada kn Language.KO Korean ko Language.LA Latin la Language.LB Luxembourgish lb Language.LN Lingala ln Language.LO Lao lo Language.LT Lithuanian lt Language.LV Latvian lv Language.MG Malagasy mg Language.MI Maori mi Language.MK Macedonian mk Language.ML Malayalam ml Language.MN Mongolian mn Language.MR Marathi mr Language.MS Malay ms Language.MT Maltese mt Language.MY_MR Burmese mymr Language.NE Nepali ne Language.NL Dutch nl Language.NN Norwegian (Nynorsk) nn Language.NO Norwegian no Language.OC Occitan oc Language.PA Punjabi pa Language.PL Polish pl Language.PS Pashto ps Language.PT Portuguese pt Language.RO Romanian ro Language.RU Russian ru Language.SA Sanskrit sa Language.SD Sindhi sd Language.SI Sinhala si Language.SK Slovak sk Language.SL Slovenian sl Language.SN Shona sn Language.SO Somali so Language.SQ Albanian sq Language.SR Serbian sr Language.SU Sundanese su Language.SV Swedish sv Language.SW Swahili sw Language.TA Tamil ta Language.TE Telugu te Language.TG Tajik tg Language.TH Thai th Language.TK Turkmen tk Language.TL Tagalog tl Language.TR Turkish tr Language.TT Tatar tt Language.UK Ukrainian uk Language.UR Urdu ur Language.UZ Uzbek uz Language.VI Vietnamese vi Language.YI Yiddish yi Language.YO Yoruba yo Language.ZH Chinese zh For a complete list of supported languages, refer to Gladia’s documentation . ​ Advanced Features ​ Custom Vocabulary You can provide custom vocabulary items with bias intensity: Copy Ask AI from pipecat.services.gladia.config import CustomVocabularyItem, CustomVocabularyConfig, RealtimeProcessingConfig custom_vocab = CustomVocabularyConfig( vocabulary = [ CustomVocabularyItem( value = "Pipecat" , intensity = 0.8 ), CustomVocabularyItem( value = "Daily" , intensity = 0.7 ), ], default_intensity = 0.5 ) realtime_config = RealtimeProcessingConfig( custom_vocabulary = True , custom_vocabulary_config = custom_vocab ) ​ Translation Enable real-time translation: Copy Ask AI from pipecat.services.gladia.config import TranslationConfig, RealtimeProcessingConfig translation_config = TranslationConfig( target_languages = [ "fr" , "es" , "de" ], model = "enhanced" , match_original_utterances = True ) realtime_config = RealtimeProcessingConfig( translation = True , translation_config = translation_config ) ​ Multi-language Support Configure multiple languages with automatic language switching: Copy Ask AI from pipecat.services.gladia.config import LanguageConfig, GladiaInputParams language_config = LanguageConfig( languages = [ "en" , "fr" , "es" ], code_switching = True ) params = GladiaInputParams( language_config = language_config ) ​ Usage Example Copy Ask AI from pipecat.pipeline.pipeline import Pipeline from pipecat.services.gladia.stt import GladiaSTTService from pipecat.services.gladia.config import ( GladiaInputParams, LanguageConfig, RealtimeProcessingConfig ) from pipecat.transcriptions.language import Language # Configure the service stt = GladiaSTTService( api_key = "your-api-key" , model = "solaria-1" , params = GladiaInputParams( language_config = LanguageConfig( languages = [Language. EN , Language. FR ], code_switching = True ), ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ Frame Flow ​ Metrics Support The service collects processing metrics: Time to First Byte (TTFB) Processing duration Connection status ​ Notes Audio input must be in PCM format Transcription frames are only generated when confidence threshold is met Service automatically handles websocket connections and cleanup Real-time processing occurs in parallel for natural conversation flow Fal (Wizper) Google On this page Overview Installation Configuration Service Parameters GladiaInputParams LanguageConfig PreProcessingConfig CustomVocabularyConfig CustomSpellingConfig TranslationConfig RealtimeProcessingConfig MessagesConfig Input Output TranscriptionFrame InterimTranscriptionFrame ErrorFrame Methods Language Setting Language Support Advanced Features Custom Vocabulary Translation Multi-language Support Usage Example Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_gladia_a2a8034f.txt b/stt_gladia_a2a8034f.txt
new file mode 100644
index 0000000000000000000000000000000000000000..ae94c3907ecaba70b0bc51d5f5af2de9842c13dc
--- /dev/null
+++ b/stt_gladia_a2a8034f.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/gladia
+Title: Gladia - Pipecat
+==================================================
+
+Gladia - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text Gladia Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview GladiaSTTService is a speech-to-text (STT) service that integrates with Gladia’s API to provide real-time transcription capabilities. It processes audio input and produces transcription frames in real-time with support for multiple languages, custom vocabulary, and various processing options. ​ Installation To use GladiaSTTService , you need to install the Gladia dependencies: Copy Ask AI pip install "pipecat-ai[gladia]" You’ll also need to set up your Gladia API key as an environment variable: GLADIA_API_KEY . ​ Configuration ​ Service Parameters ​ api_key string required Your Gladia API key for authentication ​ url string default: "https://api.gladia.io/v2/live" Gladia API endpoint URL ​ confidence float default: "0.5" Minimum confidence threshold to create interim and final transcriptions. Values range from 0 to 1. ​ sample_rate integer default: "None" Audio sample rate in Hz ​ model string default: "solaria-1" Model to use for transcription. Options include solaria-1 solaria-mini-1 fast accurate See Gladia’s docs for the latest supported models. ​ params GladiaInputParams default: "GladiaInputParams()" Additional configuration parameters for the service ​ GladiaInputParams ​ encoding string default: "wav/pcm" Audio encoding format ​ bit_depth integer default: "16" Audio bit depth ​ channels integer default: "1" Number of audio channels ​ custom_metadata Dict[str, Any] Additional metadata to include with requests ​ endpointing float Silence duration in seconds to mark end of speech ​ maximum_duration_without_endpointing integer default: "10" Maximum utterance duration without silence ​ language Language deprecated Primary language for transcription. Deprecated: use language_config instead. ​ language_config LanguageConfig Detailed language configuration ​ pre_processing PreProcessingConfig Audio pre-processing options ​ realtime_processing RealtimeProcessingConfig Real-time processing features ​ messages_config MessagesConfig WebSocket message filtering options ​ LanguageConfig ​ languages List[str] Specify language(s) for transcription. If one language is set, it will be used for all transcription. If multiple languages are provided or none, language will be auto-detected by the model. ​ code_switching boolean default: "false" If true, language will be auto-detected on each utterance. Otherwise, language will be auto-detected on first utterance and then used for the rest of the transcription. If one language is set, this option will be ignored. ​ PreProcessingConfig ​ speech_threshold float default: "0.8" Sensitivity configuration for Speech Threshold. A value close to 1 will apply stricter thresholds, making it less likely to detect background sounds as speech. Must be between 0 and 1. ​ CustomVocabularyConfig ​ vocabulary List[Union[str, CustomVocabularyItem]] required Specific vocabulary list to feed the transcription model with. Can be a list of strings or CustomVocabularyItem objects. ​ default_intensity float Default intensity for the custom vocabulary. Must be between 0 and 1. ​ CustomSpellingConfig ​ spelling_dictionary Dict[str, List[str]] required The list of spelling rules applied on the audio transcription. Keys are the correct spellings and values are lists of phonetic variations. ​ TranslationConfig ​ target_languages List[str] required The target language(s) in ISO639-1 format (e.g., “en”, “fr”, “es”) ​ model string default: "base" Translation model to use. Options: “base” or “enhanced” ​ match_original_utterances boolean default: "true" Align translated utterances with the original ones ​ RealtimeProcessingConfig ​ words_accurate_timestamps boolean Whether to provide per-word timestamps ​ custom_vocabulary boolean Whether to enable custom vocabulary ​ custom_vocabulary_config CustomVocabularyConfig Custom vocabulary configuration ​ custom_spelling boolean Whether to enable custom spelling ​ custom_spelling_config CustomSpellingConfig Custom spelling configuration ​ translation boolean Whether to enable translation ​ translation_config TranslationConfig Translation configuration ​ named_entity_recognition boolean Whether to enable named entity recognition ​ sentiment_analysis boolean Whether to enable sentiment analysis ​ MessagesConfig ​ receive_partial_transcripts boolean default: "true" If true, partial utterances will be sent via WebSocket ​ receive_final_transcripts boolean default: "true" If true, final utterances will be sent via WebSocket ​ receive_speech_events boolean default: "true" If true, begin and end speech events will be sent via WebSocket ​ receive_pre_processing_events boolean default: "true" If true, pre-processing events will be sent via WebSocket ​ receive_realtime_processing_events boolean default: "true" If true, realtime processing events will be sent via WebSocket ​ receive_post_processing_events boolean default: "true" If true, post-processing events will be sent via WebSocket ​ receive_acknowledgments boolean default: "true" If true, acknowledgments will be sent via WebSocket ​ receive_errors boolean default: "true" If true, errors will be sent via WebSocket ​ receive_lifecycle_events boolean default: "false" If true, lifecycle events will be sent via WebSocket ​ Input The service processes raw audio data with the following requirements: PCM audio format 16-bit depth 16kHz sample rate (default) Single channel (mono) ​ Output The service produces two types of frames during transcription: ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Transcription language ​ InterimTranscriptionFrame Generated during ongoing speech, containing the same fields as TranscriptionFrame but with preliminary results. ​ ErrorFrame Generated when transcription errors occur, containing error details. ​ Methods See the STT base class methods for additional functionality. ​ Language Setting Copy Ask AI await service.set_language(Language. FR ) ​ Language Support Gladia STT supports a wide range of languages. Here’s a partial list: Language Code Description Service Code Language.AF Afrikaans af Language.AM Amharic am Language.AR Arabic ar Language.AS Assamese as Language.AZ Azerbaijani az Language.BA Bashkir ba Language.BE Belarusian be Language.BG Bulgarian bg Language.BN Bengali bn Language.BO Tibetan bo Language.BR Breton br Language.BS Bosnian bs Language.CA Catalan ca Language.CS Czech cs Language.CY Welsh cy Language.DA Danish da Language.DE German de Language.EL Greek el Language.EN English en Language.ES Spanish es Language.ET Estonian et Language.EU Basque eu Language.FA Persian fa Language.FI Finnish fi Language.FO Faroese fo Language.FR French fr Language.GL Galician gl Language.GU Gujarati gu Language.HA Hausa ha Language.HAW Hawaiian haw Language.HE Hebrew he Language.HI Hindi hi Language.HR Croatian hr Language.HT Haitian Creole ht Language.HU Hungarian hu Language.HY Armenian hy Language.ID Indonesian id Language.IS Icelandic is Language.IT Italian it Language.JA Japanese ja Language.JV Javanese jv Language.KA Georgian ka Language.KK Kazakh kk Language.KM Khmer km Language.KN Kannada kn Language.KO Korean ko Language.LA Latin la Language.LB Luxembourgish lb Language.LN Lingala ln Language.LO Lao lo Language.LT Lithuanian lt Language.LV Latvian lv Language.MG Malagasy mg Language.MI Maori mi Language.MK Macedonian mk Language.ML Malayalam ml Language.MN Mongolian mn Language.MR Marathi mr Language.MS Malay ms Language.MT Maltese mt Language.MY_MR Burmese mymr Language.NE Nepali ne Language.NL Dutch nl Language.NN Norwegian (Nynorsk) nn Language.NO Norwegian no Language.OC Occitan oc Language.PA Punjabi pa Language.PL Polish pl Language.PS Pashto ps Language.PT Portuguese pt Language.RO Romanian ro Language.RU Russian ru Language.SA Sanskrit sa Language.SD Sindhi sd Language.SI Sinhala si Language.SK Slovak sk Language.SL Slovenian sl Language.SN Shona sn Language.SO Somali so Language.SQ Albanian sq Language.SR Serbian sr Language.SU Sundanese su Language.SV Swedish sv Language.SW Swahili sw Language.TA Tamil ta Language.TE Telugu te Language.TG Tajik tg Language.TH Thai th Language.TK Turkmen tk Language.TL Tagalog tl Language.TR Turkish tr Language.TT Tatar tt Language.UK Ukrainian uk Language.UR Urdu ur Language.UZ Uzbek uz Language.VI Vietnamese vi Language.YI Yiddish yi Language.YO Yoruba yo Language.ZH Chinese zh For a complete list of supported languages, refer to Gladia’s documentation . ​ Advanced Features ​ Custom Vocabulary You can provide custom vocabulary items with bias intensity: Copy Ask AI from pipecat.services.gladia.config import CustomVocabularyItem, CustomVocabularyConfig, RealtimeProcessingConfig custom_vocab = CustomVocabularyConfig( vocabulary = [ CustomVocabularyItem( value = "Pipecat" , intensity = 0.8 ), CustomVocabularyItem( value = "Daily" , intensity = 0.7 ), ], default_intensity = 0.5 ) realtime_config = RealtimeProcessingConfig( custom_vocabulary = True , custom_vocabulary_config = custom_vocab ) ​ Translation Enable real-time translation: Copy Ask AI from pipecat.services.gladia.config import TranslationConfig, RealtimeProcessingConfig translation_config = TranslationConfig( target_languages = [ "fr" , "es" , "de" ], model = "enhanced" , match_original_utterances = True ) realtime_config = RealtimeProcessingConfig( translation = True , translation_config = translation_config ) ​ Multi-language Support Configure multiple languages with automatic language switching: Copy Ask AI from pipecat.services.gladia.config import LanguageConfig, GladiaInputParams language_config = LanguageConfig( languages = [ "en" , "fr" , "es" ], code_switching = True ) params = GladiaInputParams( language_config = language_config ) ​ Usage Example Copy Ask AI from pipecat.pipeline.pipeline import Pipeline from pipecat.services.gladia.stt import GladiaSTTService from pipecat.services.gladia.config import ( GladiaInputParams, LanguageConfig, RealtimeProcessingConfig ) from pipecat.transcriptions.language import Language # Configure the service stt = GladiaSTTService( api_key = "your-api-key" , model = "solaria-1" , params = GladiaInputParams( language_config = LanguageConfig( languages = [Language. EN , Language. FR ], code_switching = True ), ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ Frame Flow ​ Metrics Support The service collects processing metrics: Time to First Byte (TTFB) Processing duration Connection status ​ Notes Audio input must be in PCM format Transcription frames are only generated when confidence threshold is met Service automatically handles websocket connections and cleanup Real-time processing occurs in parallel for natural conversation flow Fal (Wizper) Google On this page Overview Installation Configuration Service Parameters GladiaInputParams LanguageConfig PreProcessingConfig CustomVocabularyConfig CustomSpellingConfig TranslationConfig RealtimeProcessingConfig MessagesConfig Input Output TranscriptionFrame InterimTranscriptionFrame ErrorFrame Methods Language Setting Language Support Advanced Features Custom Vocabulary Translation Multi-language Support Usage Example Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_gladia_a7341516.txt b/stt_gladia_a7341516.txt
new file mode 100644
index 0000000000000000000000000000000000000000..ab193a6680dd5b074c2895ecadad35bd5d831e59
--- /dev/null
+++ b/stt_gladia_a7341516.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/gladia#installation
+Title: Gladia - Pipecat
+==================================================
+
+Gladia - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text Gladia Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview GladiaSTTService is a speech-to-text (STT) service that integrates with Gladia’s API to provide real-time transcription capabilities. It processes audio input and produces transcription frames in real-time with support for multiple languages, custom vocabulary, and various processing options. ​ Installation To use GladiaSTTService , you need to install the Gladia dependencies: Copy Ask AI pip install "pipecat-ai[gladia]" You’ll also need to set up your Gladia API key as an environment variable: GLADIA_API_KEY . ​ Configuration ​ Service Parameters ​ api_key string required Your Gladia API key for authentication ​ url string default: "https://api.gladia.io/v2/live" Gladia API endpoint URL ​ confidence float default: "0.5" Minimum confidence threshold to create interim and final transcriptions. Values range from 0 to 1. ​ sample_rate integer default: "None" Audio sample rate in Hz ​ model string default: "solaria-1" Model to use for transcription. Options include solaria-1 solaria-mini-1 fast accurate See Gladia’s docs for the latest supported models. ​ params GladiaInputParams default: "GladiaInputParams()" Additional configuration parameters for the service ​ GladiaInputParams ​ encoding string default: "wav/pcm" Audio encoding format ​ bit_depth integer default: "16" Audio bit depth ​ channels integer default: "1" Number of audio channels ​ custom_metadata Dict[str, Any] Additional metadata to include with requests ​ endpointing float Silence duration in seconds to mark end of speech ​ maximum_duration_without_endpointing integer default: "10" Maximum utterance duration without silence ​ language Language deprecated Primary language for transcription. Deprecated: use language_config instead. ​ language_config LanguageConfig Detailed language configuration ​ pre_processing PreProcessingConfig Audio pre-processing options ​ realtime_processing RealtimeProcessingConfig Real-time processing features ​ messages_config MessagesConfig WebSocket message filtering options ​ LanguageConfig ​ languages List[str] Specify language(s) for transcription. If one language is set, it will be used for all transcription. If multiple languages are provided or none, language will be auto-detected by the model. ​ code_switching boolean default: "false" If true, language will be auto-detected on each utterance. Otherwise, language will be auto-detected on first utterance and then used for the rest of the transcription. If one language is set, this option will be ignored. ​ PreProcessingConfig ​ speech_threshold float default: "0.8" Sensitivity configuration for Speech Threshold. A value close to 1 will apply stricter thresholds, making it less likely to detect background sounds as speech. Must be between 0 and 1. ​ CustomVocabularyConfig ​ vocabulary List[Union[str, CustomVocabularyItem]] required Specific vocabulary list to feed the transcription model with. Can be a list of strings or CustomVocabularyItem objects. ​ default_intensity float Default intensity for the custom vocabulary. Must be between 0 and 1. ​ CustomSpellingConfig ​ spelling_dictionary Dict[str, List[str]] required The list of spelling rules applied on the audio transcription. Keys are the correct spellings and values are lists of phonetic variations. ​ TranslationConfig ​ target_languages List[str] required The target language(s) in ISO639-1 format (e.g., “en”, “fr”, “es”) ​ model string default: "base" Translation model to use. Options: “base” or “enhanced” ​ match_original_utterances boolean default: "true" Align translated utterances with the original ones ​ RealtimeProcessingConfig ​ words_accurate_timestamps boolean Whether to provide per-word timestamps ​ custom_vocabulary boolean Whether to enable custom vocabulary ​ custom_vocabulary_config CustomVocabularyConfig Custom vocabulary configuration ​ custom_spelling boolean Whether to enable custom spelling ​ custom_spelling_config CustomSpellingConfig Custom spelling configuration ​ translation boolean Whether to enable translation ​ translation_config TranslationConfig Translation configuration ​ named_entity_recognition boolean Whether to enable named entity recognition ​ sentiment_analysis boolean Whether to enable sentiment analysis ​ MessagesConfig ​ receive_partial_transcripts boolean default: "true" If true, partial utterances will be sent via WebSocket ​ receive_final_transcripts boolean default: "true" If true, final utterances will be sent via WebSocket ​ receive_speech_events boolean default: "true" If true, begin and end speech events will be sent via WebSocket ​ receive_pre_processing_events boolean default: "true" If true, pre-processing events will be sent via WebSocket ​ receive_realtime_processing_events boolean default: "true" If true, realtime processing events will be sent via WebSocket ​ receive_post_processing_events boolean default: "true" If true, post-processing events will be sent via WebSocket ​ receive_acknowledgments boolean default: "true" If true, acknowledgments will be sent via WebSocket ​ receive_errors boolean default: "true" If true, errors will be sent via WebSocket ​ receive_lifecycle_events boolean default: "false" If true, lifecycle events will be sent via WebSocket ​ Input The service processes raw audio data with the following requirements: PCM audio format 16-bit depth 16kHz sample rate (default) Single channel (mono) ​ Output The service produces two types of frames during transcription: ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Transcription language ​ InterimTranscriptionFrame Generated during ongoing speech, containing the same fields as TranscriptionFrame but with preliminary results. ​ ErrorFrame Generated when transcription errors occur, containing error details. ​ Methods See the STT base class methods for additional functionality. ​ Language Setting Copy Ask AI await service.set_language(Language. FR ) ​ Language Support Gladia STT supports a wide range of languages. Here’s a partial list: Language Code Description Service Code Language.AF Afrikaans af Language.AM Amharic am Language.AR Arabic ar Language.AS Assamese as Language.AZ Azerbaijani az Language.BA Bashkir ba Language.BE Belarusian be Language.BG Bulgarian bg Language.BN Bengali bn Language.BO Tibetan bo Language.BR Breton br Language.BS Bosnian bs Language.CA Catalan ca Language.CS Czech cs Language.CY Welsh cy Language.DA Danish da Language.DE German de Language.EL Greek el Language.EN English en Language.ES Spanish es Language.ET Estonian et Language.EU Basque eu Language.FA Persian fa Language.FI Finnish fi Language.FO Faroese fo Language.FR French fr Language.GL Galician gl Language.GU Gujarati gu Language.HA Hausa ha Language.HAW Hawaiian haw Language.HE Hebrew he Language.HI Hindi hi Language.HR Croatian hr Language.HT Haitian Creole ht Language.HU Hungarian hu Language.HY Armenian hy Language.ID Indonesian id Language.IS Icelandic is Language.IT Italian it Language.JA Japanese ja Language.JV Javanese jv Language.KA Georgian ka Language.KK Kazakh kk Language.KM Khmer km Language.KN Kannada kn Language.KO Korean ko Language.LA Latin la Language.LB Luxembourgish lb Language.LN Lingala ln Language.LO Lao lo Language.LT Lithuanian lt Language.LV Latvian lv Language.MG Malagasy mg Language.MI Maori mi Language.MK Macedonian mk Language.ML Malayalam ml Language.MN Mongolian mn Language.MR Marathi mr Language.MS Malay ms Language.MT Maltese mt Language.MY_MR Burmese mymr Language.NE Nepali ne Language.NL Dutch nl Language.NN Norwegian (Nynorsk) nn Language.NO Norwegian no Language.OC Occitan oc Language.PA Punjabi pa Language.PL Polish pl Language.PS Pashto ps Language.PT Portuguese pt Language.RO Romanian ro Language.RU Russian ru Language.SA Sanskrit sa Language.SD Sindhi sd Language.SI Sinhala si Language.SK Slovak sk Language.SL Slovenian sl Language.SN Shona sn Language.SO Somali so Language.SQ Albanian sq Language.SR Serbian sr Language.SU Sundanese su Language.SV Swedish sv Language.SW Swahili sw Language.TA Tamil ta Language.TE Telugu te Language.TG Tajik tg Language.TH Thai th Language.TK Turkmen tk Language.TL Tagalog tl Language.TR Turkish tr Language.TT Tatar tt Language.UK Ukrainian uk Language.UR Urdu ur Language.UZ Uzbek uz Language.VI Vietnamese vi Language.YI Yiddish yi Language.YO Yoruba yo Language.ZH Chinese zh For a complete list of supported languages, refer to Gladia’s documentation . ​ Advanced Features ​ Custom Vocabulary You can provide custom vocabulary items with bias intensity: Copy Ask AI from pipecat.services.gladia.config import CustomVocabularyItem, CustomVocabularyConfig, RealtimeProcessingConfig custom_vocab = CustomVocabularyConfig( vocabulary = [ CustomVocabularyItem( value = "Pipecat" , intensity = 0.8 ), CustomVocabularyItem( value = "Daily" , intensity = 0.7 ), ], default_intensity = 0.5 ) realtime_config = RealtimeProcessingConfig( custom_vocabulary = True , custom_vocabulary_config = custom_vocab ) ​ Translation Enable real-time translation: Copy Ask AI from pipecat.services.gladia.config import TranslationConfig, RealtimeProcessingConfig translation_config = TranslationConfig( target_languages = [ "fr" , "es" , "de" ], model = "enhanced" , match_original_utterances = True ) realtime_config = RealtimeProcessingConfig( translation = True , translation_config = translation_config ) ​ Multi-language Support Configure multiple languages with automatic language switching: Copy Ask AI from pipecat.services.gladia.config import LanguageConfig, GladiaInputParams language_config = LanguageConfig( languages = [ "en" , "fr" , "es" ], code_switching = True ) params = GladiaInputParams( language_config = language_config ) ​ Usage Example Copy Ask AI from pipecat.pipeline.pipeline import Pipeline from pipecat.services.gladia.stt import GladiaSTTService from pipecat.services.gladia.config import ( GladiaInputParams, LanguageConfig, RealtimeProcessingConfig ) from pipecat.transcriptions.language import Language # Configure the service stt = GladiaSTTService( api_key = "your-api-key" , model = "solaria-1" , params = GladiaInputParams( language_config = LanguageConfig( languages = [Language. EN , Language. FR ], code_switching = True ), ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ Frame Flow ​ Metrics Support The service collects processing metrics: Time to First Byte (TTFB) Processing duration Connection status ​ Notes Audio input must be in PCM format Transcription frames are only generated when confidence threshold is met Service automatically handles websocket connections and cleanup Real-time processing occurs in parallel for natural conversation flow Fal (Wizper) Google On this page Overview Installation Configuration Service Parameters GladiaInputParams LanguageConfig PreProcessingConfig CustomVocabularyConfig CustomSpellingConfig TranslationConfig RealtimeProcessingConfig MessagesConfig Input Output TranscriptionFrame InterimTranscriptionFrame ErrorFrame Methods Language Setting Language Support Advanced Features Custom Vocabulary Translation Multi-language Support Usage Example Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_gladia_b740fdde.txt b/stt_gladia_b740fdde.txt
new file mode 100644
index 0000000000000000000000000000000000000000..d3a120b0b41d8b7bc2f9fce690e9b71292cd8dc8
--- /dev/null
+++ b/stt_gladia_b740fdde.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/gladia#language-setting
+Title: Gladia - Pipecat
+==================================================
+
+Gladia - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text Gladia Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview GladiaSTTService is a speech-to-text (STT) service that integrates with Gladia’s API to provide real-time transcription capabilities. It processes audio input and produces transcription frames in real-time with support for multiple languages, custom vocabulary, and various processing options. ​ Installation To use GladiaSTTService , you need to install the Gladia dependencies: Copy Ask AI pip install "pipecat-ai[gladia]" You’ll also need to set up your Gladia API key as an environment variable: GLADIA_API_KEY . ​ Configuration ​ Service Parameters ​ api_key string required Your Gladia API key for authentication ​ url string default: "https://api.gladia.io/v2/live" Gladia API endpoint URL ​ confidence float default: "0.5" Minimum confidence threshold to create interim and final transcriptions. Values range from 0 to 1. ​ sample_rate integer default: "None" Audio sample rate in Hz ​ model string default: "solaria-1" Model to use for transcription. Options include solaria-1 solaria-mini-1 fast accurate See Gladia’s docs for the latest supported models. ​ params GladiaInputParams default: "GladiaInputParams()" Additional configuration parameters for the service ​ GladiaInputParams ​ encoding string default: "wav/pcm" Audio encoding format ​ bit_depth integer default: "16" Audio bit depth ​ channels integer default: "1" Number of audio channels ​ custom_metadata Dict[str, Any] Additional metadata to include with requests ​ endpointing float Silence duration in seconds to mark end of speech ​ maximum_duration_without_endpointing integer default: "10" Maximum utterance duration without silence ​ language Language deprecated Primary language for transcription. Deprecated: use language_config instead. ​ language_config LanguageConfig Detailed language configuration ​ pre_processing PreProcessingConfig Audio pre-processing options ​ realtime_processing RealtimeProcessingConfig Real-time processing features ​ messages_config MessagesConfig WebSocket message filtering options ​ LanguageConfig ​ languages List[str] Specify language(s) for transcription. If one language is set, it will be used for all transcription. If multiple languages are provided or none, language will be auto-detected by the model. ​ code_switching boolean default: "false" If true, language will be auto-detected on each utterance. Otherwise, language will be auto-detected on first utterance and then used for the rest of the transcription. If one language is set, this option will be ignored. ​ PreProcessingConfig ​ speech_threshold float default: "0.8" Sensitivity configuration for Speech Threshold. A value close to 1 will apply stricter thresholds, making it less likely to detect background sounds as speech. Must be between 0 and 1. ​ CustomVocabularyConfig ​ vocabulary List[Union[str, CustomVocabularyItem]] required Specific vocabulary list to feed the transcription model with. Can be a list of strings or CustomVocabularyItem objects. ​ default_intensity float Default intensity for the custom vocabulary. Must be between 0 and 1. ​ CustomSpellingConfig ​ spelling_dictionary Dict[str, List[str]] required The list of spelling rules applied on the audio transcription. Keys are the correct spellings and values are lists of phonetic variations. ​ TranslationConfig ​ target_languages List[str] required The target language(s) in ISO639-1 format (e.g., “en”, “fr”, “es”) ​ model string default: "base" Translation model to use. Options: “base” or “enhanced” ​ match_original_utterances boolean default: "true" Align translated utterances with the original ones ​ RealtimeProcessingConfig ​ words_accurate_timestamps boolean Whether to provide per-word timestamps ​ custom_vocabulary boolean Whether to enable custom vocabulary ​ custom_vocabulary_config CustomVocabularyConfig Custom vocabulary configuration ​ custom_spelling boolean Whether to enable custom spelling ​ custom_spelling_config CustomSpellingConfig Custom spelling configuration ​ translation boolean Whether to enable translation ​ translation_config TranslationConfig Translation configuration ​ named_entity_recognition boolean Whether to enable named entity recognition ​ sentiment_analysis boolean Whether to enable sentiment analysis ​ MessagesConfig ​ receive_partial_transcripts boolean default: "true" If true, partial utterances will be sent via WebSocket ​ receive_final_transcripts boolean default: "true" If true, final utterances will be sent via WebSocket ​ receive_speech_events boolean default: "true" If true, begin and end speech events will be sent via WebSocket ​ receive_pre_processing_events boolean default: "true" If true, pre-processing events will be sent via WebSocket ​ receive_realtime_processing_events boolean default: "true" If true, realtime processing events will be sent via WebSocket ​ receive_post_processing_events boolean default: "true" If true, post-processing events will be sent via WebSocket ​ receive_acknowledgments boolean default: "true" If true, acknowledgments will be sent via WebSocket ​ receive_errors boolean default: "true" If true, errors will be sent via WebSocket ​ receive_lifecycle_events boolean default: "false" If true, lifecycle events will be sent via WebSocket ​ Input The service processes raw audio data with the following requirements: PCM audio format 16-bit depth 16kHz sample rate (default) Single channel (mono) ​ Output The service produces two types of frames during transcription: ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Transcription language ​ InterimTranscriptionFrame Generated during ongoing speech, containing the same fields as TranscriptionFrame but with preliminary results. ​ ErrorFrame Generated when transcription errors occur, containing error details. ​ Methods See the STT base class methods for additional functionality. ​ Language Setting Copy Ask AI await service.set_language(Language. FR ) ​ Language Support Gladia STT supports a wide range of languages. Here’s a partial list: Language Code Description Service Code Language.AF Afrikaans af Language.AM Amharic am Language.AR Arabic ar Language.AS Assamese as Language.AZ Azerbaijani az Language.BA Bashkir ba Language.BE Belarusian be Language.BG Bulgarian bg Language.BN Bengali bn Language.BO Tibetan bo Language.BR Breton br Language.BS Bosnian bs Language.CA Catalan ca Language.CS Czech cs Language.CY Welsh cy Language.DA Danish da Language.DE German de Language.EL Greek el Language.EN English en Language.ES Spanish es Language.ET Estonian et Language.EU Basque eu Language.FA Persian fa Language.FI Finnish fi Language.FO Faroese fo Language.FR French fr Language.GL Galician gl Language.GU Gujarati gu Language.HA Hausa ha Language.HAW Hawaiian haw Language.HE Hebrew he Language.HI Hindi hi Language.HR Croatian hr Language.HT Haitian Creole ht Language.HU Hungarian hu Language.HY Armenian hy Language.ID Indonesian id Language.IS Icelandic is Language.IT Italian it Language.JA Japanese ja Language.JV Javanese jv Language.KA Georgian ka Language.KK Kazakh kk Language.KM Khmer km Language.KN Kannada kn Language.KO Korean ko Language.LA Latin la Language.LB Luxembourgish lb Language.LN Lingala ln Language.LO Lao lo Language.LT Lithuanian lt Language.LV Latvian lv Language.MG Malagasy mg Language.MI Maori mi Language.MK Macedonian mk Language.ML Malayalam ml Language.MN Mongolian mn Language.MR Marathi mr Language.MS Malay ms Language.MT Maltese mt Language.MY_MR Burmese mymr Language.NE Nepali ne Language.NL Dutch nl Language.NN Norwegian (Nynorsk) nn Language.NO Norwegian no Language.OC Occitan oc Language.PA Punjabi pa Language.PL Polish pl Language.PS Pashto ps Language.PT Portuguese pt Language.RO Romanian ro Language.RU Russian ru Language.SA Sanskrit sa Language.SD Sindhi sd Language.SI Sinhala si Language.SK Slovak sk Language.SL Slovenian sl Language.SN Shona sn Language.SO Somali so Language.SQ Albanian sq Language.SR Serbian sr Language.SU Sundanese su Language.SV Swedish sv Language.SW Swahili sw Language.TA Tamil ta Language.TE Telugu te Language.TG Tajik tg Language.TH Thai th Language.TK Turkmen tk Language.TL Tagalog tl Language.TR Turkish tr Language.TT Tatar tt Language.UK Ukrainian uk Language.UR Urdu ur Language.UZ Uzbek uz Language.VI Vietnamese vi Language.YI Yiddish yi Language.YO Yoruba yo Language.ZH Chinese zh For a complete list of supported languages, refer to Gladia’s documentation . ​ Advanced Features ​ Custom Vocabulary You can provide custom vocabulary items with bias intensity: Copy Ask AI from pipecat.services.gladia.config import CustomVocabularyItem, CustomVocabularyConfig, RealtimeProcessingConfig custom_vocab = CustomVocabularyConfig( vocabulary = [ CustomVocabularyItem( value = "Pipecat" , intensity = 0.8 ), CustomVocabularyItem( value = "Daily" , intensity = 0.7 ), ], default_intensity = 0.5 ) realtime_config = RealtimeProcessingConfig( custom_vocabulary = True , custom_vocabulary_config = custom_vocab ) ​ Translation Enable real-time translation: Copy Ask AI from pipecat.services.gladia.config import TranslationConfig, RealtimeProcessingConfig translation_config = TranslationConfig( target_languages = [ "fr" , "es" , "de" ], model = "enhanced" , match_original_utterances = True ) realtime_config = RealtimeProcessingConfig( translation = True , translation_config = translation_config ) ​ Multi-language Support Configure multiple languages with automatic language switching: Copy Ask AI from pipecat.services.gladia.config import LanguageConfig, GladiaInputParams language_config = LanguageConfig( languages = [ "en" , "fr" , "es" ], code_switching = True ) params = GladiaInputParams( language_config = language_config ) ​ Usage Example Copy Ask AI from pipecat.pipeline.pipeline import Pipeline from pipecat.services.gladia.stt import GladiaSTTService from pipecat.services.gladia.config import ( GladiaInputParams, LanguageConfig, RealtimeProcessingConfig ) from pipecat.transcriptions.language import Language # Configure the service stt = GladiaSTTService( api_key = "your-api-key" , model = "solaria-1" , params = GladiaInputParams( language_config = LanguageConfig( languages = [Language. EN , Language. FR ], code_switching = True ), ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ Frame Flow ​ Metrics Support The service collects processing metrics: Time to First Byte (TTFB) Processing duration Connection status ​ Notes Audio input must be in PCM format Transcription frames are only generated when confidence threshold is met Service automatically handles websocket connections and cleanup Real-time processing occurs in parallel for natural conversation flow Fal (Wizper) Google On this page Overview Installation Configuration Service Parameters GladiaInputParams LanguageConfig PreProcessingConfig CustomVocabularyConfig CustomSpellingConfig TranslationConfig RealtimeProcessingConfig MessagesConfig Input Output TranscriptionFrame InterimTranscriptionFrame ErrorFrame Methods Language Setting Language Support Advanced Features Custom Vocabulary Translation Multi-language Support Usage Example Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_gladia_c38d9875.txt b/stt_gladia_c38d9875.txt
new file mode 100644
index 0000000000000000000000000000000000000000..69cfdc1fd05ee3dde8cdd671eaa6b9dc4281acfe
--- /dev/null
+++ b/stt_gladia_c38d9875.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/gladia#param-receive-errors
+Title: Gladia - Pipecat
+==================================================
+
+Gladia - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text Gladia Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview GladiaSTTService is a speech-to-text (STT) service that integrates with Gladia’s API to provide real-time transcription capabilities. It processes audio input and produces transcription frames in real-time with support for multiple languages, custom vocabulary, and various processing options. ​ Installation To use GladiaSTTService , you need to install the Gladia dependencies: Copy Ask AI pip install "pipecat-ai[gladia]" You’ll also need to set up your Gladia API key as an environment variable: GLADIA_API_KEY . ​ Configuration ​ Service Parameters ​ api_key string required Your Gladia API key for authentication ​ url string default: "https://api.gladia.io/v2/live" Gladia API endpoint URL ​ confidence float default: "0.5" Minimum confidence threshold to create interim and final transcriptions. Values range from 0 to 1. ​ sample_rate integer default: "None" Audio sample rate in Hz ​ model string default: "solaria-1" Model to use for transcription. Options include solaria-1 solaria-mini-1 fast accurate See Gladia’s docs for the latest supported models. ​ params GladiaInputParams default: "GladiaInputParams()" Additional configuration parameters for the service ​ GladiaInputParams ​ encoding string default: "wav/pcm" Audio encoding format ​ bit_depth integer default: "16" Audio bit depth ​ channels integer default: "1" Number of audio channels ​ custom_metadata Dict[str, Any] Additional metadata to include with requests ​ endpointing float Silence duration in seconds to mark end of speech ​ maximum_duration_without_endpointing integer default: "10" Maximum utterance duration without silence ​ language Language deprecated Primary language for transcription. Deprecated: use language_config instead. ​ language_config LanguageConfig Detailed language configuration ​ pre_processing PreProcessingConfig Audio pre-processing options ​ realtime_processing RealtimeProcessingConfig Real-time processing features ​ messages_config MessagesConfig WebSocket message filtering options ​ LanguageConfig ​ languages List[str] Specify language(s) for transcription. If one language is set, it will be used for all transcription. If multiple languages are provided or none, language will be auto-detected by the model. ​ code_switching boolean default: "false" If true, language will be auto-detected on each utterance. Otherwise, language will be auto-detected on first utterance and then used for the rest of the transcription. If one language is set, this option will be ignored. ​ PreProcessingConfig ​ speech_threshold float default: "0.8" Sensitivity configuration for Speech Threshold. A value close to 1 will apply stricter thresholds, making it less likely to detect background sounds as speech. Must be between 0 and 1. ​ CustomVocabularyConfig ​ vocabulary List[Union[str, CustomVocabularyItem]] required Specific vocabulary list to feed the transcription model with. Can be a list of strings or CustomVocabularyItem objects. ​ default_intensity float Default intensity for the custom vocabulary. Must be between 0 and 1. ​ CustomSpellingConfig ​ spelling_dictionary Dict[str, List[str]] required The list of spelling rules applied on the audio transcription. Keys are the correct spellings and values are lists of phonetic variations. ​ TranslationConfig ​ target_languages List[str] required The target language(s) in ISO639-1 format (e.g., “en”, “fr”, “es”) ​ model string default: "base" Translation model to use. Options: “base” or “enhanced” ​ match_original_utterances boolean default: "true" Align translated utterances with the original ones ​ RealtimeProcessingConfig ​ words_accurate_timestamps boolean Whether to provide per-word timestamps ​ custom_vocabulary boolean Whether to enable custom vocabulary ​ custom_vocabulary_config CustomVocabularyConfig Custom vocabulary configuration ​ custom_spelling boolean Whether to enable custom spelling ​ custom_spelling_config CustomSpellingConfig Custom spelling configuration ​ translation boolean Whether to enable translation ​ translation_config TranslationConfig Translation configuration ​ named_entity_recognition boolean Whether to enable named entity recognition ​ sentiment_analysis boolean Whether to enable sentiment analysis ​ MessagesConfig ​ receive_partial_transcripts boolean default: "true" If true, partial utterances will be sent via WebSocket ​ receive_final_transcripts boolean default: "true" If true, final utterances will be sent via WebSocket ​ receive_speech_events boolean default: "true" If true, begin and end speech events will be sent via WebSocket ​ receive_pre_processing_events boolean default: "true" If true, pre-processing events will be sent via WebSocket ​ receive_realtime_processing_events boolean default: "true" If true, realtime processing events will be sent via WebSocket ​ receive_post_processing_events boolean default: "true" If true, post-processing events will be sent via WebSocket ​ receive_acknowledgments boolean default: "true" If true, acknowledgments will be sent via WebSocket ​ receive_errors boolean default: "true" If true, errors will be sent via WebSocket ​ receive_lifecycle_events boolean default: "false" If true, lifecycle events will be sent via WebSocket ​ Input The service processes raw audio data with the following requirements: PCM audio format 16-bit depth 16kHz sample rate (default) Single channel (mono) ​ Output The service produces two types of frames during transcription: ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Transcription language ​ InterimTranscriptionFrame Generated during ongoing speech, containing the same fields as TranscriptionFrame but with preliminary results. ​ ErrorFrame Generated when transcription errors occur, containing error details. ​ Methods See the STT base class methods for additional functionality. ​ Language Setting Copy Ask AI await service.set_language(Language. FR ) ​ Language Support Gladia STT supports a wide range of languages. Here’s a partial list: Language Code Description Service Code Language.AF Afrikaans af Language.AM Amharic am Language.AR Arabic ar Language.AS Assamese as Language.AZ Azerbaijani az Language.BA Bashkir ba Language.BE Belarusian be Language.BG Bulgarian bg Language.BN Bengali bn Language.BO Tibetan bo Language.BR Breton br Language.BS Bosnian bs Language.CA Catalan ca Language.CS Czech cs Language.CY Welsh cy Language.DA Danish da Language.DE German de Language.EL Greek el Language.EN English en Language.ES Spanish es Language.ET Estonian et Language.EU Basque eu Language.FA Persian fa Language.FI Finnish fi Language.FO Faroese fo Language.FR French fr Language.GL Galician gl Language.GU Gujarati gu Language.HA Hausa ha Language.HAW Hawaiian haw Language.HE Hebrew he Language.HI Hindi hi Language.HR Croatian hr Language.HT Haitian Creole ht Language.HU Hungarian hu Language.HY Armenian hy Language.ID Indonesian id Language.IS Icelandic is Language.IT Italian it Language.JA Japanese ja Language.JV Javanese jv Language.KA Georgian ka Language.KK Kazakh kk Language.KM Khmer km Language.KN Kannada kn Language.KO Korean ko Language.LA Latin la Language.LB Luxembourgish lb Language.LN Lingala ln Language.LO Lao lo Language.LT Lithuanian lt Language.LV Latvian lv Language.MG Malagasy mg Language.MI Maori mi Language.MK Macedonian mk Language.ML Malayalam ml Language.MN Mongolian mn Language.MR Marathi mr Language.MS Malay ms Language.MT Maltese mt Language.MY_MR Burmese mymr Language.NE Nepali ne Language.NL Dutch nl Language.NN Norwegian (Nynorsk) nn Language.NO Norwegian no Language.OC Occitan oc Language.PA Punjabi pa Language.PL Polish pl Language.PS Pashto ps Language.PT Portuguese pt Language.RO Romanian ro Language.RU Russian ru Language.SA Sanskrit sa Language.SD Sindhi sd Language.SI Sinhala si Language.SK Slovak sk Language.SL Slovenian sl Language.SN Shona sn Language.SO Somali so Language.SQ Albanian sq Language.SR Serbian sr Language.SU Sundanese su Language.SV Swedish sv Language.SW Swahili sw Language.TA Tamil ta Language.TE Telugu te Language.TG Tajik tg Language.TH Thai th Language.TK Turkmen tk Language.TL Tagalog tl Language.TR Turkish tr Language.TT Tatar tt Language.UK Ukrainian uk Language.UR Urdu ur Language.UZ Uzbek uz Language.VI Vietnamese vi Language.YI Yiddish yi Language.YO Yoruba yo Language.ZH Chinese zh For a complete list of supported languages, refer to Gladia’s documentation . ​ Advanced Features ​ Custom Vocabulary You can provide custom vocabulary items with bias intensity: Copy Ask AI from pipecat.services.gladia.config import CustomVocabularyItem, CustomVocabularyConfig, RealtimeProcessingConfig custom_vocab = CustomVocabularyConfig( vocabulary = [ CustomVocabularyItem( value = "Pipecat" , intensity = 0.8 ), CustomVocabularyItem( value = "Daily" , intensity = 0.7 ), ], default_intensity = 0.5 ) realtime_config = RealtimeProcessingConfig( custom_vocabulary = True , custom_vocabulary_config = custom_vocab ) ​ Translation Enable real-time translation: Copy Ask AI from pipecat.services.gladia.config import TranslationConfig, RealtimeProcessingConfig translation_config = TranslationConfig( target_languages = [ "fr" , "es" , "de" ], model = "enhanced" , match_original_utterances = True ) realtime_config = RealtimeProcessingConfig( translation = True , translation_config = translation_config ) ​ Multi-language Support Configure multiple languages with automatic language switching: Copy Ask AI from pipecat.services.gladia.config import LanguageConfig, GladiaInputParams language_config = LanguageConfig( languages = [ "en" , "fr" , "es" ], code_switching = True ) params = GladiaInputParams( language_config = language_config ) ​ Usage Example Copy Ask AI from pipecat.pipeline.pipeline import Pipeline from pipecat.services.gladia.stt import GladiaSTTService from pipecat.services.gladia.config import ( GladiaInputParams, LanguageConfig, RealtimeProcessingConfig ) from pipecat.transcriptions.language import Language # Configure the service stt = GladiaSTTService( api_key = "your-api-key" , model = "solaria-1" , params = GladiaInputParams( language_config = LanguageConfig( languages = [Language. EN , Language. FR ], code_switching = True ), ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ Frame Flow ​ Metrics Support The service collects processing metrics: Time to First Byte (TTFB) Processing duration Connection status ​ Notes Audio input must be in PCM format Transcription frames are only generated when confidence threshold is met Service automatically handles websocket connections and cleanup Real-time processing occurs in parallel for natural conversation flow Fal (Wizper) Google On this page Overview Installation Configuration Service Parameters GladiaInputParams LanguageConfig PreProcessingConfig CustomVocabularyConfig CustomSpellingConfig TranslationConfig RealtimeProcessingConfig MessagesConfig Input Output TranscriptionFrame InterimTranscriptionFrame ErrorFrame Methods Language Setting Language Support Advanced Features Custom Vocabulary Translation Multi-language Support Usage Example Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_gladia_c9adb440.txt b/stt_gladia_c9adb440.txt
new file mode 100644
index 0000000000000000000000000000000000000000..d9fe8447f0ca404202f223dbca83bb277de179f6
--- /dev/null
+++ b/stt_gladia_c9adb440.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/gladia#errorframe
+Title: Gladia - Pipecat
+==================================================
+
+Gladia - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text Gladia Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview GladiaSTTService is a speech-to-text (STT) service that integrates with Gladia’s API to provide real-time transcription capabilities. It processes audio input and produces transcription frames in real-time with support for multiple languages, custom vocabulary, and various processing options. ​ Installation To use GladiaSTTService , you need to install the Gladia dependencies: Copy Ask AI pip install "pipecat-ai[gladia]" You’ll also need to set up your Gladia API key as an environment variable: GLADIA_API_KEY . ​ Configuration ​ Service Parameters ​ api_key string required Your Gladia API key for authentication ​ url string default: "https://api.gladia.io/v2/live" Gladia API endpoint URL ​ confidence float default: "0.5" Minimum confidence threshold to create interim and final transcriptions. Values range from 0 to 1. ​ sample_rate integer default: "None" Audio sample rate in Hz ​ model string default: "solaria-1" Model to use for transcription. Options include solaria-1 solaria-mini-1 fast accurate See Gladia’s docs for the latest supported models. ​ params GladiaInputParams default: "GladiaInputParams()" Additional configuration parameters for the service ​ GladiaInputParams ​ encoding string default: "wav/pcm" Audio encoding format ​ bit_depth integer default: "16" Audio bit depth ​ channels integer default: "1" Number of audio channels ​ custom_metadata Dict[str, Any] Additional metadata to include with requests ​ endpointing float Silence duration in seconds to mark end of speech ​ maximum_duration_without_endpointing integer default: "10" Maximum utterance duration without silence ​ language Language deprecated Primary language for transcription. Deprecated: use language_config instead. ​ language_config LanguageConfig Detailed language configuration ​ pre_processing PreProcessingConfig Audio pre-processing options ​ realtime_processing RealtimeProcessingConfig Real-time processing features ​ messages_config MessagesConfig WebSocket message filtering options ​ LanguageConfig ​ languages List[str] Specify language(s) for transcription. If one language is set, it will be used for all transcription. If multiple languages are provided or none, language will be auto-detected by the model. ​ code_switching boolean default: "false" If true, language will be auto-detected on each utterance. Otherwise, language will be auto-detected on first utterance and then used for the rest of the transcription. If one language is set, this option will be ignored. ​ PreProcessingConfig ​ speech_threshold float default: "0.8" Sensitivity configuration for Speech Threshold. A value close to 1 will apply stricter thresholds, making it less likely to detect background sounds as speech. Must be between 0 and 1. ​ CustomVocabularyConfig ​ vocabulary List[Union[str, CustomVocabularyItem]] required Specific vocabulary list to feed the transcription model with. Can be a list of strings or CustomVocabularyItem objects. ​ default_intensity float Default intensity for the custom vocabulary. Must be between 0 and 1. ​ CustomSpellingConfig ​ spelling_dictionary Dict[str, List[str]] required The list of spelling rules applied on the audio transcription. Keys are the correct spellings and values are lists of phonetic variations. ​ TranslationConfig ​ target_languages List[str] required The target language(s) in ISO639-1 format (e.g., “en”, “fr”, “es”) ​ model string default: "base" Translation model to use. Options: “base” or “enhanced” ​ match_original_utterances boolean default: "true" Align translated utterances with the original ones ​ RealtimeProcessingConfig ​ words_accurate_timestamps boolean Whether to provide per-word timestamps ​ custom_vocabulary boolean Whether to enable custom vocabulary ​ custom_vocabulary_config CustomVocabularyConfig Custom vocabulary configuration ​ custom_spelling boolean Whether to enable custom spelling ​ custom_spelling_config CustomSpellingConfig Custom spelling configuration ​ translation boolean Whether to enable translation ​ translation_config TranslationConfig Translation configuration ​ named_entity_recognition boolean Whether to enable named entity recognition ​ sentiment_analysis boolean Whether to enable sentiment analysis ​ MessagesConfig ​ receive_partial_transcripts boolean default: "true" If true, partial utterances will be sent via WebSocket ​ receive_final_transcripts boolean default: "true" If true, final utterances will be sent via WebSocket ​ receive_speech_events boolean default: "true" If true, begin and end speech events will be sent via WebSocket ​ receive_pre_processing_events boolean default: "true" If true, pre-processing events will be sent via WebSocket ​ receive_realtime_processing_events boolean default: "true" If true, realtime processing events will be sent via WebSocket ​ receive_post_processing_events boolean default: "true" If true, post-processing events will be sent via WebSocket ​ receive_acknowledgments boolean default: "true" If true, acknowledgments will be sent via WebSocket ​ receive_errors boolean default: "true" If true, errors will be sent via WebSocket ​ receive_lifecycle_events boolean default: "false" If true, lifecycle events will be sent via WebSocket ​ Input The service processes raw audio data with the following requirements: PCM audio format 16-bit depth 16kHz sample rate (default) Single channel (mono) ​ Output The service produces two types of frames during transcription: ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Transcription language ​ InterimTranscriptionFrame Generated during ongoing speech, containing the same fields as TranscriptionFrame but with preliminary results. ​ ErrorFrame Generated when transcription errors occur, containing error details. ​ Methods See the STT base class methods for additional functionality. ​ Language Setting Copy Ask AI await service.set_language(Language. FR ) ​ Language Support Gladia STT supports a wide range of languages. Here’s a partial list: Language Code Description Service Code Language.AF Afrikaans af Language.AM Amharic am Language.AR Arabic ar Language.AS Assamese as Language.AZ Azerbaijani az Language.BA Bashkir ba Language.BE Belarusian be Language.BG Bulgarian bg Language.BN Bengali bn Language.BO Tibetan bo Language.BR Breton br Language.BS Bosnian bs Language.CA Catalan ca Language.CS Czech cs Language.CY Welsh cy Language.DA Danish da Language.DE German de Language.EL Greek el Language.EN English en Language.ES Spanish es Language.ET Estonian et Language.EU Basque eu Language.FA Persian fa Language.FI Finnish fi Language.FO Faroese fo Language.FR French fr Language.GL Galician gl Language.GU Gujarati gu Language.HA Hausa ha Language.HAW Hawaiian haw Language.HE Hebrew he Language.HI Hindi hi Language.HR Croatian hr Language.HT Haitian Creole ht Language.HU Hungarian hu Language.HY Armenian hy Language.ID Indonesian id Language.IS Icelandic is Language.IT Italian it Language.JA Japanese ja Language.JV Javanese jv Language.KA Georgian ka Language.KK Kazakh kk Language.KM Khmer km Language.KN Kannada kn Language.KO Korean ko Language.LA Latin la Language.LB Luxembourgish lb Language.LN Lingala ln Language.LO Lao lo Language.LT Lithuanian lt Language.LV Latvian lv Language.MG Malagasy mg Language.MI Maori mi Language.MK Macedonian mk Language.ML Malayalam ml Language.MN Mongolian mn Language.MR Marathi mr Language.MS Malay ms Language.MT Maltese mt Language.MY_MR Burmese mymr Language.NE Nepali ne Language.NL Dutch nl Language.NN Norwegian (Nynorsk) nn Language.NO Norwegian no Language.OC Occitan oc Language.PA Punjabi pa Language.PL Polish pl Language.PS Pashto ps Language.PT Portuguese pt Language.RO Romanian ro Language.RU Russian ru Language.SA Sanskrit sa Language.SD Sindhi sd Language.SI Sinhala si Language.SK Slovak sk Language.SL Slovenian sl Language.SN Shona sn Language.SO Somali so Language.SQ Albanian sq Language.SR Serbian sr Language.SU Sundanese su Language.SV Swedish sv Language.SW Swahili sw Language.TA Tamil ta Language.TE Telugu te Language.TG Tajik tg Language.TH Thai th Language.TK Turkmen tk Language.TL Tagalog tl Language.TR Turkish tr Language.TT Tatar tt Language.UK Ukrainian uk Language.UR Urdu ur Language.UZ Uzbek uz Language.VI Vietnamese vi Language.YI Yiddish yi Language.YO Yoruba yo Language.ZH Chinese zh For a complete list of supported languages, refer to Gladia’s documentation . ​ Advanced Features ​ Custom Vocabulary You can provide custom vocabulary items with bias intensity: Copy Ask AI from pipecat.services.gladia.config import CustomVocabularyItem, CustomVocabularyConfig, RealtimeProcessingConfig custom_vocab = CustomVocabularyConfig( vocabulary = [ CustomVocabularyItem( value = "Pipecat" , intensity = 0.8 ), CustomVocabularyItem( value = "Daily" , intensity = 0.7 ), ], default_intensity = 0.5 ) realtime_config = RealtimeProcessingConfig( custom_vocabulary = True , custom_vocabulary_config = custom_vocab ) ​ Translation Enable real-time translation: Copy Ask AI from pipecat.services.gladia.config import TranslationConfig, RealtimeProcessingConfig translation_config = TranslationConfig( target_languages = [ "fr" , "es" , "de" ], model = "enhanced" , match_original_utterances = True ) realtime_config = RealtimeProcessingConfig( translation = True , translation_config = translation_config ) ​ Multi-language Support Configure multiple languages with automatic language switching: Copy Ask AI from pipecat.services.gladia.config import LanguageConfig, GladiaInputParams language_config = LanguageConfig( languages = [ "en" , "fr" , "es" ], code_switching = True ) params = GladiaInputParams( language_config = language_config ) ​ Usage Example Copy Ask AI from pipecat.pipeline.pipeline import Pipeline from pipecat.services.gladia.stt import GladiaSTTService from pipecat.services.gladia.config import ( GladiaInputParams, LanguageConfig, RealtimeProcessingConfig ) from pipecat.transcriptions.language import Language # Configure the service stt = GladiaSTTService( api_key = "your-api-key" , model = "solaria-1" , params = GladiaInputParams( language_config = LanguageConfig( languages = [Language. EN , Language. FR ], code_switching = True ), ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ Frame Flow ​ Metrics Support The service collects processing metrics: Time to First Byte (TTFB) Processing duration Connection status ​ Notes Audio input must be in PCM format Transcription frames are only generated when confidence threshold is met Service automatically handles websocket connections and cleanup Real-time processing occurs in parallel for natural conversation flow Fal (Wizper) Google On this page Overview Installation Configuration Service Parameters GladiaInputParams LanguageConfig PreProcessingConfig CustomVocabularyConfig CustomSpellingConfig TranslationConfig RealtimeProcessingConfig MessagesConfig Input Output TranscriptionFrame InterimTranscriptionFrame ErrorFrame Methods Language Setting Language Support Advanced Features Custom Vocabulary Translation Multi-language Support Usage Example Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_gladia_d781d152.txt b/stt_gladia_d781d152.txt
new file mode 100644
index 0000000000000000000000000000000000000000..30c4f5e33356656c8edd16ab8fd004903325fb41
--- /dev/null
+++ b/stt_gladia_d781d152.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/gladia#overview
+Title: Gladia - Pipecat
+==================================================
+
+Gladia - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text Gladia Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview GladiaSTTService is a speech-to-text (STT) service that integrates with Gladia’s API to provide real-time transcription capabilities. It processes audio input and produces transcription frames in real-time with support for multiple languages, custom vocabulary, and various processing options. ​ Installation To use GladiaSTTService , you need to install the Gladia dependencies: Copy Ask AI pip install "pipecat-ai[gladia]" You’ll also need to set up your Gladia API key as an environment variable: GLADIA_API_KEY . ​ Configuration ​ Service Parameters ​ api_key string required Your Gladia API key for authentication ​ url string default: "https://api.gladia.io/v2/live" Gladia API endpoint URL ​ confidence float default: "0.5" Minimum confidence threshold to create interim and final transcriptions. Values range from 0 to 1. ​ sample_rate integer default: "None" Audio sample rate in Hz ​ model string default: "solaria-1" Model to use for transcription. Options include solaria-1 solaria-mini-1 fast accurate See Gladia’s docs for the latest supported models. ​ params GladiaInputParams default: "GladiaInputParams()" Additional configuration parameters for the service ​ GladiaInputParams ​ encoding string default: "wav/pcm" Audio encoding format ​ bit_depth integer default: "16" Audio bit depth ​ channels integer default: "1" Number of audio channels ​ custom_metadata Dict[str, Any] Additional metadata to include with requests ​ endpointing float Silence duration in seconds to mark end of speech ​ maximum_duration_without_endpointing integer default: "10" Maximum utterance duration without silence ​ language Language deprecated Primary language for transcription. Deprecated: use language_config instead. ​ language_config LanguageConfig Detailed language configuration ​ pre_processing PreProcessingConfig Audio pre-processing options ​ realtime_processing RealtimeProcessingConfig Real-time processing features ​ messages_config MessagesConfig WebSocket message filtering options ​ LanguageConfig ​ languages List[str] Specify language(s) for transcription. If one language is set, it will be used for all transcription. If multiple languages are provided or none, language will be auto-detected by the model. ​ code_switching boolean default: "false" If true, language will be auto-detected on each utterance. Otherwise, language will be auto-detected on first utterance and then used for the rest of the transcription. If one language is set, this option will be ignored. ​ PreProcessingConfig ​ speech_threshold float default: "0.8" Sensitivity configuration for Speech Threshold. A value close to 1 will apply stricter thresholds, making it less likely to detect background sounds as speech. Must be between 0 and 1. ​ CustomVocabularyConfig ​ vocabulary List[Union[str, CustomVocabularyItem]] required Specific vocabulary list to feed the transcription model with. Can be a list of strings or CustomVocabularyItem objects. ​ default_intensity float Default intensity for the custom vocabulary. Must be between 0 and 1. ​ CustomSpellingConfig ​ spelling_dictionary Dict[str, List[str]] required The list of spelling rules applied on the audio transcription. Keys are the correct spellings and values are lists of phonetic variations. ​ TranslationConfig ​ target_languages List[str] required The target language(s) in ISO639-1 format (e.g., “en”, “fr”, “es”) ​ model string default: "base" Translation model to use. Options: “base” or “enhanced” ​ match_original_utterances boolean default: "true" Align translated utterances with the original ones ​ RealtimeProcessingConfig ​ words_accurate_timestamps boolean Whether to provide per-word timestamps ​ custom_vocabulary boolean Whether to enable custom vocabulary ​ custom_vocabulary_config CustomVocabularyConfig Custom vocabulary configuration ​ custom_spelling boolean Whether to enable custom spelling ​ custom_spelling_config CustomSpellingConfig Custom spelling configuration ​ translation boolean Whether to enable translation ​ translation_config TranslationConfig Translation configuration ​ named_entity_recognition boolean Whether to enable named entity recognition ​ sentiment_analysis boolean Whether to enable sentiment analysis ​ MessagesConfig ​ receive_partial_transcripts boolean default: "true" If true, partial utterances will be sent via WebSocket ​ receive_final_transcripts boolean default: "true" If true, final utterances will be sent via WebSocket ​ receive_speech_events boolean default: "true" If true, begin and end speech events will be sent via WebSocket ​ receive_pre_processing_events boolean default: "true" If true, pre-processing events will be sent via WebSocket ​ receive_realtime_processing_events boolean default: "true" If true, realtime processing events will be sent via WebSocket ​ receive_post_processing_events boolean default: "true" If true, post-processing events will be sent via WebSocket ​ receive_acknowledgments boolean default: "true" If true, acknowledgments will be sent via WebSocket ​ receive_errors boolean default: "true" If true, errors will be sent via WebSocket ​ receive_lifecycle_events boolean default: "false" If true, lifecycle events will be sent via WebSocket ​ Input The service processes raw audio data with the following requirements: PCM audio format 16-bit depth 16kHz sample rate (default) Single channel (mono) ​ Output The service produces two types of frames during transcription: ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Transcription language ​ InterimTranscriptionFrame Generated during ongoing speech, containing the same fields as TranscriptionFrame but with preliminary results. ​ ErrorFrame Generated when transcription errors occur, containing error details. ​ Methods See the STT base class methods for additional functionality. ​ Language Setting Copy Ask AI await service.set_language(Language. FR ) ​ Language Support Gladia STT supports a wide range of languages. Here’s a partial list: Language Code Description Service Code Language.AF Afrikaans af Language.AM Amharic am Language.AR Arabic ar Language.AS Assamese as Language.AZ Azerbaijani az Language.BA Bashkir ba Language.BE Belarusian be Language.BG Bulgarian bg Language.BN Bengali bn Language.BO Tibetan bo Language.BR Breton br Language.BS Bosnian bs Language.CA Catalan ca Language.CS Czech cs Language.CY Welsh cy Language.DA Danish da Language.DE German de Language.EL Greek el Language.EN English en Language.ES Spanish es Language.ET Estonian et Language.EU Basque eu Language.FA Persian fa Language.FI Finnish fi Language.FO Faroese fo Language.FR French fr Language.GL Galician gl Language.GU Gujarati gu Language.HA Hausa ha Language.HAW Hawaiian haw Language.HE Hebrew he Language.HI Hindi hi Language.HR Croatian hr Language.HT Haitian Creole ht Language.HU Hungarian hu Language.HY Armenian hy Language.ID Indonesian id Language.IS Icelandic is Language.IT Italian it Language.JA Japanese ja Language.JV Javanese jv Language.KA Georgian ka Language.KK Kazakh kk Language.KM Khmer km Language.KN Kannada kn Language.KO Korean ko Language.LA Latin la Language.LB Luxembourgish lb Language.LN Lingala ln Language.LO Lao lo Language.LT Lithuanian lt Language.LV Latvian lv Language.MG Malagasy mg Language.MI Maori mi Language.MK Macedonian mk Language.ML Malayalam ml Language.MN Mongolian mn Language.MR Marathi mr Language.MS Malay ms Language.MT Maltese mt Language.MY_MR Burmese mymr Language.NE Nepali ne Language.NL Dutch nl Language.NN Norwegian (Nynorsk) nn Language.NO Norwegian no Language.OC Occitan oc Language.PA Punjabi pa Language.PL Polish pl Language.PS Pashto ps Language.PT Portuguese pt Language.RO Romanian ro Language.RU Russian ru Language.SA Sanskrit sa Language.SD Sindhi sd Language.SI Sinhala si Language.SK Slovak sk Language.SL Slovenian sl Language.SN Shona sn Language.SO Somali so Language.SQ Albanian sq Language.SR Serbian sr Language.SU Sundanese su Language.SV Swedish sv Language.SW Swahili sw Language.TA Tamil ta Language.TE Telugu te Language.TG Tajik tg Language.TH Thai th Language.TK Turkmen tk Language.TL Tagalog tl Language.TR Turkish tr Language.TT Tatar tt Language.UK Ukrainian uk Language.UR Urdu ur Language.UZ Uzbek uz Language.VI Vietnamese vi Language.YI Yiddish yi Language.YO Yoruba yo Language.ZH Chinese zh For a complete list of supported languages, refer to Gladia’s documentation . ​ Advanced Features ​ Custom Vocabulary You can provide custom vocabulary items with bias intensity: Copy Ask AI from pipecat.services.gladia.config import CustomVocabularyItem, CustomVocabularyConfig, RealtimeProcessingConfig custom_vocab = CustomVocabularyConfig( vocabulary = [ CustomVocabularyItem( value = "Pipecat" , intensity = 0.8 ), CustomVocabularyItem( value = "Daily" , intensity = 0.7 ), ], default_intensity = 0.5 ) realtime_config = RealtimeProcessingConfig( custom_vocabulary = True , custom_vocabulary_config = custom_vocab ) ​ Translation Enable real-time translation: Copy Ask AI from pipecat.services.gladia.config import TranslationConfig, RealtimeProcessingConfig translation_config = TranslationConfig( target_languages = [ "fr" , "es" , "de" ], model = "enhanced" , match_original_utterances = True ) realtime_config = RealtimeProcessingConfig( translation = True , translation_config = translation_config ) ​ Multi-language Support Configure multiple languages with automatic language switching: Copy Ask AI from pipecat.services.gladia.config import LanguageConfig, GladiaInputParams language_config = LanguageConfig( languages = [ "en" , "fr" , "es" ], code_switching = True ) params = GladiaInputParams( language_config = language_config ) ​ Usage Example Copy Ask AI from pipecat.pipeline.pipeline import Pipeline from pipecat.services.gladia.stt import GladiaSTTService from pipecat.services.gladia.config import ( GladiaInputParams, LanguageConfig, RealtimeProcessingConfig ) from pipecat.transcriptions.language import Language # Configure the service stt = GladiaSTTService( api_key = "your-api-key" , model = "solaria-1" , params = GladiaInputParams( language_config = LanguageConfig( languages = [Language. EN , Language. FR ], code_switching = True ), ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ Frame Flow ​ Metrics Support The service collects processing metrics: Time to First Byte (TTFB) Processing duration Connection status ​ Notes Audio input must be in PCM format Transcription frames are only generated when confidence threshold is met Service automatically handles websocket connections and cleanup Real-time processing occurs in parallel for natural conversation flow Fal (Wizper) Google On this page Overview Installation Configuration Service Parameters GladiaInputParams LanguageConfig PreProcessingConfig CustomVocabularyConfig CustomSpellingConfig TranslationConfig RealtimeProcessingConfig MessagesConfig Input Output TranscriptionFrame InterimTranscriptionFrame ErrorFrame Methods Language Setting Language Support Advanced Features Custom Vocabulary Translation Multi-language Support Usage Example Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_gladia_dc130296.txt b/stt_gladia_dc130296.txt
new file mode 100644
index 0000000000000000000000000000000000000000..9c8e377364b6ebc3a1987a6442d3dee4b085be01
--- /dev/null
+++ b/stt_gladia_dc130296.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/gladia#param-user-id
+Title: Gladia - Pipecat
+==================================================
+
+Gladia - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text Gladia Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview GladiaSTTService is a speech-to-text (STT) service that integrates with Gladia’s API to provide real-time transcription capabilities. It processes audio input and produces transcription frames in real-time with support for multiple languages, custom vocabulary, and various processing options. ​ Installation To use GladiaSTTService , you need to install the Gladia dependencies: Copy Ask AI pip install "pipecat-ai[gladia]" You’ll also need to set up your Gladia API key as an environment variable: GLADIA_API_KEY . ​ Configuration ​ Service Parameters ​ api_key string required Your Gladia API key for authentication ​ url string default: "https://api.gladia.io/v2/live" Gladia API endpoint URL ​ confidence float default: "0.5" Minimum confidence threshold to create interim and final transcriptions. Values range from 0 to 1. ​ sample_rate integer default: "None" Audio sample rate in Hz ​ model string default: "solaria-1" Model to use for transcription. Options include solaria-1 solaria-mini-1 fast accurate See Gladia’s docs for the latest supported models. ​ params GladiaInputParams default: "GladiaInputParams()" Additional configuration parameters for the service ​ GladiaInputParams ​ encoding string default: "wav/pcm" Audio encoding format ​ bit_depth integer default: "16" Audio bit depth ​ channels integer default: "1" Number of audio channels ​ custom_metadata Dict[str, Any] Additional metadata to include with requests ​ endpointing float Silence duration in seconds to mark end of speech ​ maximum_duration_without_endpointing integer default: "10" Maximum utterance duration without silence ​ language Language deprecated Primary language for transcription. Deprecated: use language_config instead. ​ language_config LanguageConfig Detailed language configuration ​ pre_processing PreProcessingConfig Audio pre-processing options ​ realtime_processing RealtimeProcessingConfig Real-time processing features ​ messages_config MessagesConfig WebSocket message filtering options ​ LanguageConfig ​ languages List[str] Specify language(s) for transcription. If one language is set, it will be used for all transcription. If multiple languages are provided or none, language will be auto-detected by the model. ​ code_switching boolean default: "false" If true, language will be auto-detected on each utterance. Otherwise, language will be auto-detected on first utterance and then used for the rest of the transcription. If one language is set, this option will be ignored. ​ PreProcessingConfig ​ speech_threshold float default: "0.8" Sensitivity configuration for Speech Threshold. A value close to 1 will apply stricter thresholds, making it less likely to detect background sounds as speech. Must be between 0 and 1. ​ CustomVocabularyConfig ​ vocabulary List[Union[str, CustomVocabularyItem]] required Specific vocabulary list to feed the transcription model with. Can be a list of strings or CustomVocabularyItem objects. ​ default_intensity float Default intensity for the custom vocabulary. Must be between 0 and 1. ​ CustomSpellingConfig ​ spelling_dictionary Dict[str, List[str]] required The list of spelling rules applied on the audio transcription. Keys are the correct spellings and values are lists of phonetic variations. ​ TranslationConfig ​ target_languages List[str] required The target language(s) in ISO639-1 format (e.g., “en”, “fr”, “es”) ​ model string default: "base" Translation model to use. Options: “base” or “enhanced” ​ match_original_utterances boolean default: "true" Align translated utterances with the original ones ​ RealtimeProcessingConfig ​ words_accurate_timestamps boolean Whether to provide per-word timestamps ​ custom_vocabulary boolean Whether to enable custom vocabulary ​ custom_vocabulary_config CustomVocabularyConfig Custom vocabulary configuration ​ custom_spelling boolean Whether to enable custom spelling ​ custom_spelling_config CustomSpellingConfig Custom spelling configuration ​ translation boolean Whether to enable translation ​ translation_config TranslationConfig Translation configuration ​ named_entity_recognition boolean Whether to enable named entity recognition ​ sentiment_analysis boolean Whether to enable sentiment analysis ​ MessagesConfig ​ receive_partial_transcripts boolean default: "true" If true, partial utterances will be sent via WebSocket ​ receive_final_transcripts boolean default: "true" If true, final utterances will be sent via WebSocket ​ receive_speech_events boolean default: "true" If true, begin and end speech events will be sent via WebSocket ​ receive_pre_processing_events boolean default: "true" If true, pre-processing events will be sent via WebSocket ​ receive_realtime_processing_events boolean default: "true" If true, realtime processing events will be sent via WebSocket ​ receive_post_processing_events boolean default: "true" If true, post-processing events will be sent via WebSocket ​ receive_acknowledgments boolean default: "true" If true, acknowledgments will be sent via WebSocket ​ receive_errors boolean default: "true" If true, errors will be sent via WebSocket ​ receive_lifecycle_events boolean default: "false" If true, lifecycle events will be sent via WebSocket ​ Input The service processes raw audio data with the following requirements: PCM audio format 16-bit depth 16kHz sample rate (default) Single channel (mono) ​ Output The service produces two types of frames during transcription: ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Transcription language ​ InterimTranscriptionFrame Generated during ongoing speech, containing the same fields as TranscriptionFrame but with preliminary results. ​ ErrorFrame Generated when transcription errors occur, containing error details. ​ Methods See the STT base class methods for additional functionality. ​ Language Setting Copy Ask AI await service.set_language(Language. FR ) ​ Language Support Gladia STT supports a wide range of languages. Here’s a partial list: Language Code Description Service Code Language.AF Afrikaans af Language.AM Amharic am Language.AR Arabic ar Language.AS Assamese as Language.AZ Azerbaijani az Language.BA Bashkir ba Language.BE Belarusian be Language.BG Bulgarian bg Language.BN Bengali bn Language.BO Tibetan bo Language.BR Breton br Language.BS Bosnian bs Language.CA Catalan ca Language.CS Czech cs Language.CY Welsh cy Language.DA Danish da Language.DE German de Language.EL Greek el Language.EN English en Language.ES Spanish es Language.ET Estonian et Language.EU Basque eu Language.FA Persian fa Language.FI Finnish fi Language.FO Faroese fo Language.FR French fr Language.GL Galician gl Language.GU Gujarati gu Language.HA Hausa ha Language.HAW Hawaiian haw Language.HE Hebrew he Language.HI Hindi hi Language.HR Croatian hr Language.HT Haitian Creole ht Language.HU Hungarian hu Language.HY Armenian hy Language.ID Indonesian id Language.IS Icelandic is Language.IT Italian it Language.JA Japanese ja Language.JV Javanese jv Language.KA Georgian ka Language.KK Kazakh kk Language.KM Khmer km Language.KN Kannada kn Language.KO Korean ko Language.LA Latin la Language.LB Luxembourgish lb Language.LN Lingala ln Language.LO Lao lo Language.LT Lithuanian lt Language.LV Latvian lv Language.MG Malagasy mg Language.MI Maori mi Language.MK Macedonian mk Language.ML Malayalam ml Language.MN Mongolian mn Language.MR Marathi mr Language.MS Malay ms Language.MT Maltese mt Language.MY_MR Burmese mymr Language.NE Nepali ne Language.NL Dutch nl Language.NN Norwegian (Nynorsk) nn Language.NO Norwegian no Language.OC Occitan oc Language.PA Punjabi pa Language.PL Polish pl Language.PS Pashto ps Language.PT Portuguese pt Language.RO Romanian ro Language.RU Russian ru Language.SA Sanskrit sa Language.SD Sindhi sd Language.SI Sinhala si Language.SK Slovak sk Language.SL Slovenian sl Language.SN Shona sn Language.SO Somali so Language.SQ Albanian sq Language.SR Serbian sr Language.SU Sundanese su Language.SV Swedish sv Language.SW Swahili sw Language.TA Tamil ta Language.TE Telugu te Language.TG Tajik tg Language.TH Thai th Language.TK Turkmen tk Language.TL Tagalog tl Language.TR Turkish tr Language.TT Tatar tt Language.UK Ukrainian uk Language.UR Urdu ur Language.UZ Uzbek uz Language.VI Vietnamese vi Language.YI Yiddish yi Language.YO Yoruba yo Language.ZH Chinese zh For a complete list of supported languages, refer to Gladia’s documentation . ​ Advanced Features ​ Custom Vocabulary You can provide custom vocabulary items with bias intensity: Copy Ask AI from pipecat.services.gladia.config import CustomVocabularyItem, CustomVocabularyConfig, RealtimeProcessingConfig custom_vocab = CustomVocabularyConfig( vocabulary = [ CustomVocabularyItem( value = "Pipecat" , intensity = 0.8 ), CustomVocabularyItem( value = "Daily" , intensity = 0.7 ), ], default_intensity = 0.5 ) realtime_config = RealtimeProcessingConfig( custom_vocabulary = True , custom_vocabulary_config = custom_vocab ) ​ Translation Enable real-time translation: Copy Ask AI from pipecat.services.gladia.config import TranslationConfig, RealtimeProcessingConfig translation_config = TranslationConfig( target_languages = [ "fr" , "es" , "de" ], model = "enhanced" , match_original_utterances = True ) realtime_config = RealtimeProcessingConfig( translation = True , translation_config = translation_config ) ​ Multi-language Support Configure multiple languages with automatic language switching: Copy Ask AI from pipecat.services.gladia.config import LanguageConfig, GladiaInputParams language_config = LanguageConfig( languages = [ "en" , "fr" , "es" ], code_switching = True ) params = GladiaInputParams( language_config = language_config ) ​ Usage Example Copy Ask AI from pipecat.pipeline.pipeline import Pipeline from pipecat.services.gladia.stt import GladiaSTTService from pipecat.services.gladia.config import ( GladiaInputParams, LanguageConfig, RealtimeProcessingConfig ) from pipecat.transcriptions.language import Language # Configure the service stt = GladiaSTTService( api_key = "your-api-key" , model = "solaria-1" , params = GladiaInputParams( language_config = LanguageConfig( languages = [Language. EN , Language. FR ], code_switching = True ), ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ Frame Flow ​ Metrics Support The service collects processing metrics: Time to First Byte (TTFB) Processing duration Connection status ​ Notes Audio input must be in PCM format Transcription frames are only generated when confidence threshold is met Service automatically handles websocket connections and cleanup Real-time processing occurs in parallel for natural conversation flow Fal (Wizper) Google On this page Overview Installation Configuration Service Parameters GladiaInputParams LanguageConfig PreProcessingConfig CustomVocabularyConfig CustomSpellingConfig TranslationConfig RealtimeProcessingConfig MessagesConfig Input Output TranscriptionFrame InterimTranscriptionFrame ErrorFrame Methods Language Setting Language Support Advanced Features Custom Vocabulary Translation Multi-language Support Usage Example Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_gladia_de6cb95f.txt b/stt_gladia_de6cb95f.txt
new file mode 100644
index 0000000000000000000000000000000000000000..1ef3c4f5edf363b5de92af7a00f945f745472771
--- /dev/null
+++ b/stt_gladia_de6cb95f.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/gladia#param-pre-processing
+Title: Gladia - Pipecat
+==================================================
+
+Gladia - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text Gladia Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview GladiaSTTService is a speech-to-text (STT) service that integrates with Gladia’s API to provide real-time transcription capabilities. It processes audio input and produces transcription frames in real-time with support for multiple languages, custom vocabulary, and various processing options. ​ Installation To use GladiaSTTService , you need to install the Gladia dependencies: Copy Ask AI pip install "pipecat-ai[gladia]" You’ll also need to set up your Gladia API key as an environment variable: GLADIA_API_KEY . ​ Configuration ​ Service Parameters ​ api_key string required Your Gladia API key for authentication ​ url string default: "https://api.gladia.io/v2/live" Gladia API endpoint URL ​ confidence float default: "0.5" Minimum confidence threshold to create interim and final transcriptions. Values range from 0 to 1. ​ sample_rate integer default: "None" Audio sample rate in Hz ​ model string default: "solaria-1" Model to use for transcription. Options include solaria-1 solaria-mini-1 fast accurate See Gladia’s docs for the latest supported models. ​ params GladiaInputParams default: "GladiaInputParams()" Additional configuration parameters for the service ​ GladiaInputParams ​ encoding string default: "wav/pcm" Audio encoding format ​ bit_depth integer default: "16" Audio bit depth ​ channels integer default: "1" Number of audio channels ​ custom_metadata Dict[str, Any] Additional metadata to include with requests ​ endpointing float Silence duration in seconds to mark end of speech ​ maximum_duration_without_endpointing integer default: "10" Maximum utterance duration without silence ​ language Language deprecated Primary language for transcription. Deprecated: use language_config instead. ​ language_config LanguageConfig Detailed language configuration ​ pre_processing PreProcessingConfig Audio pre-processing options ​ realtime_processing RealtimeProcessingConfig Real-time processing features ​ messages_config MessagesConfig WebSocket message filtering options ​ LanguageConfig ​ languages List[str] Specify language(s) for transcription. If one language is set, it will be used for all transcription. If multiple languages are provided or none, language will be auto-detected by the model. ​ code_switching boolean default: "false" If true, language will be auto-detected on each utterance. Otherwise, language will be auto-detected on first utterance and then used for the rest of the transcription. If one language is set, this option will be ignored. ​ PreProcessingConfig ​ speech_threshold float default: "0.8" Sensitivity configuration for Speech Threshold. A value close to 1 will apply stricter thresholds, making it less likely to detect background sounds as speech. Must be between 0 and 1. ​ CustomVocabularyConfig ​ vocabulary List[Union[str, CustomVocabularyItem]] required Specific vocabulary list to feed the transcription model with. Can be a list of strings or CustomVocabularyItem objects. ​ default_intensity float Default intensity for the custom vocabulary. Must be between 0 and 1. ​ CustomSpellingConfig ​ spelling_dictionary Dict[str, List[str]] required The list of spelling rules applied on the audio transcription. Keys are the correct spellings and values are lists of phonetic variations. ​ TranslationConfig ​ target_languages List[str] required The target language(s) in ISO639-1 format (e.g., “en”, “fr”, “es”) ​ model string default: "base" Translation model to use. Options: “base” or “enhanced” ​ match_original_utterances boolean default: "true" Align translated utterances with the original ones ​ RealtimeProcessingConfig ​ words_accurate_timestamps boolean Whether to provide per-word timestamps ​ custom_vocabulary boolean Whether to enable custom vocabulary ​ custom_vocabulary_config CustomVocabularyConfig Custom vocabulary configuration ​ custom_spelling boolean Whether to enable custom spelling ​ custom_spelling_config CustomSpellingConfig Custom spelling configuration ​ translation boolean Whether to enable translation ​ translation_config TranslationConfig Translation configuration ​ named_entity_recognition boolean Whether to enable named entity recognition ​ sentiment_analysis boolean Whether to enable sentiment analysis ​ MessagesConfig ​ receive_partial_transcripts boolean default: "true" If true, partial utterances will be sent via WebSocket ​ receive_final_transcripts boolean default: "true" If true, final utterances will be sent via WebSocket ​ receive_speech_events boolean default: "true" If true, begin and end speech events will be sent via WebSocket ​ receive_pre_processing_events boolean default: "true" If true, pre-processing events will be sent via WebSocket ​ receive_realtime_processing_events boolean default: "true" If true, realtime processing events will be sent via WebSocket ​ receive_post_processing_events boolean default: "true" If true, post-processing events will be sent via WebSocket ​ receive_acknowledgments boolean default: "true" If true, acknowledgments will be sent via WebSocket ​ receive_errors boolean default: "true" If true, errors will be sent via WebSocket ​ receive_lifecycle_events boolean default: "false" If true, lifecycle events will be sent via WebSocket ​ Input The service processes raw audio data with the following requirements: PCM audio format 16-bit depth 16kHz sample rate (default) Single channel (mono) ​ Output The service produces two types of frames during transcription: ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Transcription language ​ InterimTranscriptionFrame Generated during ongoing speech, containing the same fields as TranscriptionFrame but with preliminary results. ​ ErrorFrame Generated when transcription errors occur, containing error details. ​ Methods See the STT base class methods for additional functionality. ​ Language Setting Copy Ask AI await service.set_language(Language. FR ) ​ Language Support Gladia STT supports a wide range of languages. Here’s a partial list: Language Code Description Service Code Language.AF Afrikaans af Language.AM Amharic am Language.AR Arabic ar Language.AS Assamese as Language.AZ Azerbaijani az Language.BA Bashkir ba Language.BE Belarusian be Language.BG Bulgarian bg Language.BN Bengali bn Language.BO Tibetan bo Language.BR Breton br Language.BS Bosnian bs Language.CA Catalan ca Language.CS Czech cs Language.CY Welsh cy Language.DA Danish da Language.DE German de Language.EL Greek el Language.EN English en Language.ES Spanish es Language.ET Estonian et Language.EU Basque eu Language.FA Persian fa Language.FI Finnish fi Language.FO Faroese fo Language.FR French fr Language.GL Galician gl Language.GU Gujarati gu Language.HA Hausa ha Language.HAW Hawaiian haw Language.HE Hebrew he Language.HI Hindi hi Language.HR Croatian hr Language.HT Haitian Creole ht Language.HU Hungarian hu Language.HY Armenian hy Language.ID Indonesian id Language.IS Icelandic is Language.IT Italian it Language.JA Japanese ja Language.JV Javanese jv Language.KA Georgian ka Language.KK Kazakh kk Language.KM Khmer km Language.KN Kannada kn Language.KO Korean ko Language.LA Latin la Language.LB Luxembourgish lb Language.LN Lingala ln Language.LO Lao lo Language.LT Lithuanian lt Language.LV Latvian lv Language.MG Malagasy mg Language.MI Maori mi Language.MK Macedonian mk Language.ML Malayalam ml Language.MN Mongolian mn Language.MR Marathi mr Language.MS Malay ms Language.MT Maltese mt Language.MY_MR Burmese mymr Language.NE Nepali ne Language.NL Dutch nl Language.NN Norwegian (Nynorsk) nn Language.NO Norwegian no Language.OC Occitan oc Language.PA Punjabi pa Language.PL Polish pl Language.PS Pashto ps Language.PT Portuguese pt Language.RO Romanian ro Language.RU Russian ru Language.SA Sanskrit sa Language.SD Sindhi sd Language.SI Sinhala si Language.SK Slovak sk Language.SL Slovenian sl Language.SN Shona sn Language.SO Somali so Language.SQ Albanian sq Language.SR Serbian sr Language.SU Sundanese su Language.SV Swedish sv Language.SW Swahili sw Language.TA Tamil ta Language.TE Telugu te Language.TG Tajik tg Language.TH Thai th Language.TK Turkmen tk Language.TL Tagalog tl Language.TR Turkish tr Language.TT Tatar tt Language.UK Ukrainian uk Language.UR Urdu ur Language.UZ Uzbek uz Language.VI Vietnamese vi Language.YI Yiddish yi Language.YO Yoruba yo Language.ZH Chinese zh For a complete list of supported languages, refer to Gladia’s documentation . ​ Advanced Features ​ Custom Vocabulary You can provide custom vocabulary items with bias intensity: Copy Ask AI from pipecat.services.gladia.config import CustomVocabularyItem, CustomVocabularyConfig, RealtimeProcessingConfig custom_vocab = CustomVocabularyConfig( vocabulary = [ CustomVocabularyItem( value = "Pipecat" , intensity = 0.8 ), CustomVocabularyItem( value = "Daily" , intensity = 0.7 ), ], default_intensity = 0.5 ) realtime_config = RealtimeProcessingConfig( custom_vocabulary = True , custom_vocabulary_config = custom_vocab ) ​ Translation Enable real-time translation: Copy Ask AI from pipecat.services.gladia.config import TranslationConfig, RealtimeProcessingConfig translation_config = TranslationConfig( target_languages = [ "fr" , "es" , "de" ], model = "enhanced" , match_original_utterances = True ) realtime_config = RealtimeProcessingConfig( translation = True , translation_config = translation_config ) ​ Multi-language Support Configure multiple languages with automatic language switching: Copy Ask AI from pipecat.services.gladia.config import LanguageConfig, GladiaInputParams language_config = LanguageConfig( languages = [ "en" , "fr" , "es" ], code_switching = True ) params = GladiaInputParams( language_config = language_config ) ​ Usage Example Copy Ask AI from pipecat.pipeline.pipeline import Pipeline from pipecat.services.gladia.stt import GladiaSTTService from pipecat.services.gladia.config import ( GladiaInputParams, LanguageConfig, RealtimeProcessingConfig ) from pipecat.transcriptions.language import Language # Configure the service stt = GladiaSTTService( api_key = "your-api-key" , model = "solaria-1" , params = GladiaInputParams( language_config = LanguageConfig( languages = [Language. EN , Language. FR ], code_switching = True ), ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ Frame Flow ​ Metrics Support The service collects processing metrics: Time to First Byte (TTFB) Processing duration Connection status ​ Notes Audio input must be in PCM format Transcription frames are only generated when confidence threshold is met Service automatically handles websocket connections and cleanup Real-time processing occurs in parallel for natural conversation flow Fal (Wizper) Google On this page Overview Installation Configuration Service Parameters GladiaInputParams LanguageConfig PreProcessingConfig CustomVocabularyConfig CustomSpellingConfig TranslationConfig RealtimeProcessingConfig MessagesConfig Input Output TranscriptionFrame InterimTranscriptionFrame ErrorFrame Methods Language Setting Language Support Advanced Features Custom Vocabulary Translation Multi-language Support Usage Example Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_gladia_fa248931.txt b/stt_gladia_fa248931.txt
new file mode 100644
index 0000000000000000000000000000000000000000..3926d6db2af44023b034c8f814a1bc07e81e3603
--- /dev/null
+++ b/stt_gladia_fa248931.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/gladia#param-model-1
+Title: Gladia - Pipecat
+==================================================
+
+Gladia - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text Gladia Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview GladiaSTTService is a speech-to-text (STT) service that integrates with Gladia’s API to provide real-time transcription capabilities. It processes audio input and produces transcription frames in real-time with support for multiple languages, custom vocabulary, and various processing options. ​ Installation To use GladiaSTTService , you need to install the Gladia dependencies: Copy Ask AI pip install "pipecat-ai[gladia]" You’ll also need to set up your Gladia API key as an environment variable: GLADIA_API_KEY . ​ Configuration ​ Service Parameters ​ api_key string required Your Gladia API key for authentication ​ url string default: "https://api.gladia.io/v2/live" Gladia API endpoint URL ​ confidence float default: "0.5" Minimum confidence threshold to create interim and final transcriptions. Values range from 0 to 1. ​ sample_rate integer default: "None" Audio sample rate in Hz ​ model string default: "solaria-1" Model to use for transcription. Options include solaria-1 solaria-mini-1 fast accurate See Gladia’s docs for the latest supported models. ​ params GladiaInputParams default: "GladiaInputParams()" Additional configuration parameters for the service ​ GladiaInputParams ​ encoding string default: "wav/pcm" Audio encoding format ​ bit_depth integer default: "16" Audio bit depth ​ channels integer default: "1" Number of audio channels ​ custom_metadata Dict[str, Any] Additional metadata to include with requests ​ endpointing float Silence duration in seconds to mark end of speech ​ maximum_duration_without_endpointing integer default: "10" Maximum utterance duration without silence ​ language Language deprecated Primary language for transcription. Deprecated: use language_config instead. ​ language_config LanguageConfig Detailed language configuration ​ pre_processing PreProcessingConfig Audio pre-processing options ​ realtime_processing RealtimeProcessingConfig Real-time processing features ​ messages_config MessagesConfig WebSocket message filtering options ​ LanguageConfig ​ languages List[str] Specify language(s) for transcription. If one language is set, it will be used for all transcription. If multiple languages are provided or none, language will be auto-detected by the model. ​ code_switching boolean default: "false" If true, language will be auto-detected on each utterance. Otherwise, language will be auto-detected on first utterance and then used for the rest of the transcription. If one language is set, this option will be ignored. ​ PreProcessingConfig ​ speech_threshold float default: "0.8" Sensitivity configuration for Speech Threshold. A value close to 1 will apply stricter thresholds, making it less likely to detect background sounds as speech. Must be between 0 and 1. ​ CustomVocabularyConfig ​ vocabulary List[Union[str, CustomVocabularyItem]] required Specific vocabulary list to feed the transcription model with. Can be a list of strings or CustomVocabularyItem objects. ​ default_intensity float Default intensity for the custom vocabulary. Must be between 0 and 1. ​ CustomSpellingConfig ​ spelling_dictionary Dict[str, List[str]] required The list of spelling rules applied on the audio transcription. Keys are the correct spellings and values are lists of phonetic variations. ​ TranslationConfig ​ target_languages List[str] required The target language(s) in ISO639-1 format (e.g., “en”, “fr”, “es”) ​ model string default: "base" Translation model to use. Options: “base” or “enhanced” ​ match_original_utterances boolean default: "true" Align translated utterances with the original ones ​ RealtimeProcessingConfig ​ words_accurate_timestamps boolean Whether to provide per-word timestamps ​ custom_vocabulary boolean Whether to enable custom vocabulary ​ custom_vocabulary_config CustomVocabularyConfig Custom vocabulary configuration ​ custom_spelling boolean Whether to enable custom spelling ​ custom_spelling_config CustomSpellingConfig Custom spelling configuration ​ translation boolean Whether to enable translation ​ translation_config TranslationConfig Translation configuration ​ named_entity_recognition boolean Whether to enable named entity recognition ​ sentiment_analysis boolean Whether to enable sentiment analysis ​ MessagesConfig ​ receive_partial_transcripts boolean default: "true" If true, partial utterances will be sent via WebSocket ​ receive_final_transcripts boolean default: "true" If true, final utterances will be sent via WebSocket ​ receive_speech_events boolean default: "true" If true, begin and end speech events will be sent via WebSocket ​ receive_pre_processing_events boolean default: "true" If true, pre-processing events will be sent via WebSocket ​ receive_realtime_processing_events boolean default: "true" If true, realtime processing events will be sent via WebSocket ​ receive_post_processing_events boolean default: "true" If true, post-processing events will be sent via WebSocket ​ receive_acknowledgments boolean default: "true" If true, acknowledgments will be sent via WebSocket ​ receive_errors boolean default: "true" If true, errors will be sent via WebSocket ​ receive_lifecycle_events boolean default: "false" If true, lifecycle events will be sent via WebSocket ​ Input The service processes raw audio data with the following requirements: PCM audio format 16-bit depth 16kHz sample rate (default) Single channel (mono) ​ Output The service produces two types of frames during transcription: ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Transcription language ​ InterimTranscriptionFrame Generated during ongoing speech, containing the same fields as TranscriptionFrame but with preliminary results. ​ ErrorFrame Generated when transcription errors occur, containing error details. ​ Methods See the STT base class methods for additional functionality. ​ Language Setting Copy Ask AI await service.set_language(Language. FR ) ​ Language Support Gladia STT supports a wide range of languages. Here’s a partial list: Language Code Description Service Code Language.AF Afrikaans af Language.AM Amharic am Language.AR Arabic ar Language.AS Assamese as Language.AZ Azerbaijani az Language.BA Bashkir ba Language.BE Belarusian be Language.BG Bulgarian bg Language.BN Bengali bn Language.BO Tibetan bo Language.BR Breton br Language.BS Bosnian bs Language.CA Catalan ca Language.CS Czech cs Language.CY Welsh cy Language.DA Danish da Language.DE German de Language.EL Greek el Language.EN English en Language.ES Spanish es Language.ET Estonian et Language.EU Basque eu Language.FA Persian fa Language.FI Finnish fi Language.FO Faroese fo Language.FR French fr Language.GL Galician gl Language.GU Gujarati gu Language.HA Hausa ha Language.HAW Hawaiian haw Language.HE Hebrew he Language.HI Hindi hi Language.HR Croatian hr Language.HT Haitian Creole ht Language.HU Hungarian hu Language.HY Armenian hy Language.ID Indonesian id Language.IS Icelandic is Language.IT Italian it Language.JA Japanese ja Language.JV Javanese jv Language.KA Georgian ka Language.KK Kazakh kk Language.KM Khmer km Language.KN Kannada kn Language.KO Korean ko Language.LA Latin la Language.LB Luxembourgish lb Language.LN Lingala ln Language.LO Lao lo Language.LT Lithuanian lt Language.LV Latvian lv Language.MG Malagasy mg Language.MI Maori mi Language.MK Macedonian mk Language.ML Malayalam ml Language.MN Mongolian mn Language.MR Marathi mr Language.MS Malay ms Language.MT Maltese mt Language.MY_MR Burmese mymr Language.NE Nepali ne Language.NL Dutch nl Language.NN Norwegian (Nynorsk) nn Language.NO Norwegian no Language.OC Occitan oc Language.PA Punjabi pa Language.PL Polish pl Language.PS Pashto ps Language.PT Portuguese pt Language.RO Romanian ro Language.RU Russian ru Language.SA Sanskrit sa Language.SD Sindhi sd Language.SI Sinhala si Language.SK Slovak sk Language.SL Slovenian sl Language.SN Shona sn Language.SO Somali so Language.SQ Albanian sq Language.SR Serbian sr Language.SU Sundanese su Language.SV Swedish sv Language.SW Swahili sw Language.TA Tamil ta Language.TE Telugu te Language.TG Tajik tg Language.TH Thai th Language.TK Turkmen tk Language.TL Tagalog tl Language.TR Turkish tr Language.TT Tatar tt Language.UK Ukrainian uk Language.UR Urdu ur Language.UZ Uzbek uz Language.VI Vietnamese vi Language.YI Yiddish yi Language.YO Yoruba yo Language.ZH Chinese zh For a complete list of supported languages, refer to Gladia’s documentation . ​ Advanced Features ​ Custom Vocabulary You can provide custom vocabulary items with bias intensity: Copy Ask AI from pipecat.services.gladia.config import CustomVocabularyItem, CustomVocabularyConfig, RealtimeProcessingConfig custom_vocab = CustomVocabularyConfig( vocabulary = [ CustomVocabularyItem( value = "Pipecat" , intensity = 0.8 ), CustomVocabularyItem( value = "Daily" , intensity = 0.7 ), ], default_intensity = 0.5 ) realtime_config = RealtimeProcessingConfig( custom_vocabulary = True , custom_vocabulary_config = custom_vocab ) ​ Translation Enable real-time translation: Copy Ask AI from pipecat.services.gladia.config import TranslationConfig, RealtimeProcessingConfig translation_config = TranslationConfig( target_languages = [ "fr" , "es" , "de" ], model = "enhanced" , match_original_utterances = True ) realtime_config = RealtimeProcessingConfig( translation = True , translation_config = translation_config ) ​ Multi-language Support Configure multiple languages with automatic language switching: Copy Ask AI from pipecat.services.gladia.config import LanguageConfig, GladiaInputParams language_config = LanguageConfig( languages = [ "en" , "fr" , "es" ], code_switching = True ) params = GladiaInputParams( language_config = language_config ) ​ Usage Example Copy Ask AI from pipecat.pipeline.pipeline import Pipeline from pipecat.services.gladia.stt import GladiaSTTService from pipecat.services.gladia.config import ( GladiaInputParams, LanguageConfig, RealtimeProcessingConfig ) from pipecat.transcriptions.language import Language # Configure the service stt = GladiaSTTService( api_key = "your-api-key" , model = "solaria-1" , params = GladiaInputParams( language_config = LanguageConfig( languages = [Language. EN , Language. FR ], code_switching = True ), ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ Frame Flow ​ Metrics Support The service collects processing metrics: Time to First Byte (TTFB) Processing duration Connection status ​ Notes Audio input must be in PCM format Transcription frames are only generated when confidence threshold is met Service automatically handles websocket connections and cleanup Real-time processing occurs in parallel for natural conversation flow Fal (Wizper) Google On this page Overview Installation Configuration Service Parameters GladiaInputParams LanguageConfig PreProcessingConfig CustomVocabularyConfig CustomSpellingConfig TranslationConfig RealtimeProcessingConfig MessagesConfig Input Output TranscriptionFrame InterimTranscriptionFrame ErrorFrame Methods Language Setting Language Support Advanced Features Custom Vocabulary Translation Multi-language Support Usage Example Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_gladia_ffd1a6c4.txt b/stt_gladia_ffd1a6c4.txt
new file mode 100644
index 0000000000000000000000000000000000000000..46c3e50b421cfcce184350b0e3896fd3bf49498d
--- /dev/null
+++ b/stt_gladia_ffd1a6c4.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/gladia#param-endpointing
+Title: Gladia - Pipecat
+==================================================
+
+Gladia - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text Gladia Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview GladiaSTTService is a speech-to-text (STT) service that integrates with Gladia’s API to provide real-time transcription capabilities. It processes audio input and produces transcription frames in real-time with support for multiple languages, custom vocabulary, and various processing options. ​ Installation To use GladiaSTTService , you need to install the Gladia dependencies: Copy Ask AI pip install "pipecat-ai[gladia]" You’ll also need to set up your Gladia API key as an environment variable: GLADIA_API_KEY . ​ Configuration ​ Service Parameters ​ api_key string required Your Gladia API key for authentication ​ url string default: "https://api.gladia.io/v2/live" Gladia API endpoint URL ​ confidence float default: "0.5" Minimum confidence threshold to create interim and final transcriptions. Values range from 0 to 1. ​ sample_rate integer default: "None" Audio sample rate in Hz ​ model string default: "solaria-1" Model to use for transcription. Options include solaria-1 solaria-mini-1 fast accurate See Gladia’s docs for the latest supported models. ​ params GladiaInputParams default: "GladiaInputParams()" Additional configuration parameters for the service ​ GladiaInputParams ​ encoding string default: "wav/pcm" Audio encoding format ​ bit_depth integer default: "16" Audio bit depth ​ channels integer default: "1" Number of audio channels ​ custom_metadata Dict[str, Any] Additional metadata to include with requests ​ endpointing float Silence duration in seconds to mark end of speech ​ maximum_duration_without_endpointing integer default: "10" Maximum utterance duration without silence ​ language Language deprecated Primary language for transcription. Deprecated: use language_config instead. ​ language_config LanguageConfig Detailed language configuration ​ pre_processing PreProcessingConfig Audio pre-processing options ​ realtime_processing RealtimeProcessingConfig Real-time processing features ​ messages_config MessagesConfig WebSocket message filtering options ​ LanguageConfig ​ languages List[str] Specify language(s) for transcription. If one language is set, it will be used for all transcription. If multiple languages are provided or none, language will be auto-detected by the model. ​ code_switching boolean default: "false" If true, language will be auto-detected on each utterance. Otherwise, language will be auto-detected on first utterance and then used for the rest of the transcription. If one language is set, this option will be ignored. ​ PreProcessingConfig ​ speech_threshold float default: "0.8" Sensitivity configuration for Speech Threshold. A value close to 1 will apply stricter thresholds, making it less likely to detect background sounds as speech. Must be between 0 and 1. ​ CustomVocabularyConfig ​ vocabulary List[Union[str, CustomVocabularyItem]] required Specific vocabulary list to feed the transcription model with. Can be a list of strings or CustomVocabularyItem objects. ​ default_intensity float Default intensity for the custom vocabulary. Must be between 0 and 1. ​ CustomSpellingConfig ​ spelling_dictionary Dict[str, List[str]] required The list of spelling rules applied on the audio transcription. Keys are the correct spellings and values are lists of phonetic variations. ​ TranslationConfig ​ target_languages List[str] required The target language(s) in ISO639-1 format (e.g., “en”, “fr”, “es”) ​ model string default: "base" Translation model to use. Options: “base” or “enhanced” ​ match_original_utterances boolean default: "true" Align translated utterances with the original ones ​ RealtimeProcessingConfig ​ words_accurate_timestamps boolean Whether to provide per-word timestamps ​ custom_vocabulary boolean Whether to enable custom vocabulary ​ custom_vocabulary_config CustomVocabularyConfig Custom vocabulary configuration ​ custom_spelling boolean Whether to enable custom spelling ​ custom_spelling_config CustomSpellingConfig Custom spelling configuration ​ translation boolean Whether to enable translation ​ translation_config TranslationConfig Translation configuration ​ named_entity_recognition boolean Whether to enable named entity recognition ​ sentiment_analysis boolean Whether to enable sentiment analysis ​ MessagesConfig ​ receive_partial_transcripts boolean default: "true" If true, partial utterances will be sent via WebSocket ​ receive_final_transcripts boolean default: "true" If true, final utterances will be sent via WebSocket ​ receive_speech_events boolean default: "true" If true, begin and end speech events will be sent via WebSocket ​ receive_pre_processing_events boolean default: "true" If true, pre-processing events will be sent via WebSocket ​ receive_realtime_processing_events boolean default: "true" If true, realtime processing events will be sent via WebSocket ​ receive_post_processing_events boolean default: "true" If true, post-processing events will be sent via WebSocket ​ receive_acknowledgments boolean default: "true" If true, acknowledgments will be sent via WebSocket ​ receive_errors boolean default: "true" If true, errors will be sent via WebSocket ​ receive_lifecycle_events boolean default: "false" If true, lifecycle events will be sent via WebSocket ​ Input The service processes raw audio data with the following requirements: PCM audio format 16-bit depth 16kHz sample rate (default) Single channel (mono) ​ Output The service produces two types of frames during transcription: ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Transcription language ​ InterimTranscriptionFrame Generated during ongoing speech, containing the same fields as TranscriptionFrame but with preliminary results. ​ ErrorFrame Generated when transcription errors occur, containing error details. ​ Methods See the STT base class methods for additional functionality. ​ Language Setting Copy Ask AI await service.set_language(Language. FR ) ​ Language Support Gladia STT supports a wide range of languages. Here’s a partial list: Language Code Description Service Code Language.AF Afrikaans af Language.AM Amharic am Language.AR Arabic ar Language.AS Assamese as Language.AZ Azerbaijani az Language.BA Bashkir ba Language.BE Belarusian be Language.BG Bulgarian bg Language.BN Bengali bn Language.BO Tibetan bo Language.BR Breton br Language.BS Bosnian bs Language.CA Catalan ca Language.CS Czech cs Language.CY Welsh cy Language.DA Danish da Language.DE German de Language.EL Greek el Language.EN English en Language.ES Spanish es Language.ET Estonian et Language.EU Basque eu Language.FA Persian fa Language.FI Finnish fi Language.FO Faroese fo Language.FR French fr Language.GL Galician gl Language.GU Gujarati gu Language.HA Hausa ha Language.HAW Hawaiian haw Language.HE Hebrew he Language.HI Hindi hi Language.HR Croatian hr Language.HT Haitian Creole ht Language.HU Hungarian hu Language.HY Armenian hy Language.ID Indonesian id Language.IS Icelandic is Language.IT Italian it Language.JA Japanese ja Language.JV Javanese jv Language.KA Georgian ka Language.KK Kazakh kk Language.KM Khmer km Language.KN Kannada kn Language.KO Korean ko Language.LA Latin la Language.LB Luxembourgish lb Language.LN Lingala ln Language.LO Lao lo Language.LT Lithuanian lt Language.LV Latvian lv Language.MG Malagasy mg Language.MI Maori mi Language.MK Macedonian mk Language.ML Malayalam ml Language.MN Mongolian mn Language.MR Marathi mr Language.MS Malay ms Language.MT Maltese mt Language.MY_MR Burmese mymr Language.NE Nepali ne Language.NL Dutch nl Language.NN Norwegian (Nynorsk) nn Language.NO Norwegian no Language.OC Occitan oc Language.PA Punjabi pa Language.PL Polish pl Language.PS Pashto ps Language.PT Portuguese pt Language.RO Romanian ro Language.RU Russian ru Language.SA Sanskrit sa Language.SD Sindhi sd Language.SI Sinhala si Language.SK Slovak sk Language.SL Slovenian sl Language.SN Shona sn Language.SO Somali so Language.SQ Albanian sq Language.SR Serbian sr Language.SU Sundanese su Language.SV Swedish sv Language.SW Swahili sw Language.TA Tamil ta Language.TE Telugu te Language.TG Tajik tg Language.TH Thai th Language.TK Turkmen tk Language.TL Tagalog tl Language.TR Turkish tr Language.TT Tatar tt Language.UK Ukrainian uk Language.UR Urdu ur Language.UZ Uzbek uz Language.VI Vietnamese vi Language.YI Yiddish yi Language.YO Yoruba yo Language.ZH Chinese zh For a complete list of supported languages, refer to Gladia’s documentation . ​ Advanced Features ​ Custom Vocabulary You can provide custom vocabulary items with bias intensity: Copy Ask AI from pipecat.services.gladia.config import CustomVocabularyItem, CustomVocabularyConfig, RealtimeProcessingConfig custom_vocab = CustomVocabularyConfig( vocabulary = [ CustomVocabularyItem( value = "Pipecat" , intensity = 0.8 ), CustomVocabularyItem( value = "Daily" , intensity = 0.7 ), ], default_intensity = 0.5 ) realtime_config = RealtimeProcessingConfig( custom_vocabulary = True , custom_vocabulary_config = custom_vocab ) ​ Translation Enable real-time translation: Copy Ask AI from pipecat.services.gladia.config import TranslationConfig, RealtimeProcessingConfig translation_config = TranslationConfig( target_languages = [ "fr" , "es" , "de" ], model = "enhanced" , match_original_utterances = True ) realtime_config = RealtimeProcessingConfig( translation = True , translation_config = translation_config ) ​ Multi-language Support Configure multiple languages with automatic language switching: Copy Ask AI from pipecat.services.gladia.config import LanguageConfig, GladiaInputParams language_config = LanguageConfig( languages = [ "en" , "fr" , "es" ], code_switching = True ) params = GladiaInputParams( language_config = language_config ) ​ Usage Example Copy Ask AI from pipecat.pipeline.pipeline import Pipeline from pipecat.services.gladia.stt import GladiaSTTService from pipecat.services.gladia.config import ( GladiaInputParams, LanguageConfig, RealtimeProcessingConfig ) from pipecat.transcriptions.language import Language # Configure the service stt = GladiaSTTService( api_key = "your-api-key" , model = "solaria-1" , params = GladiaInputParams( language_config = LanguageConfig( languages = [Language. EN , Language. FR ], code_switching = True ), ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ Frame Flow ​ Metrics Support The service collects processing metrics: Time to First Byte (TTFB) Processing duration Connection status ​ Notes Audio input must be in PCM format Transcription frames are only generated when confidence threshold is met Service automatically handles websocket connections and cleanup Real-time processing occurs in parallel for natural conversation flow Fal (Wizper) Google On this page Overview Installation Configuration Service Parameters GladiaInputParams LanguageConfig PreProcessingConfig CustomVocabularyConfig CustomSpellingConfig TranslationConfig RealtimeProcessingConfig MessagesConfig Input Output TranscriptionFrame InterimTranscriptionFrame ErrorFrame Methods Language Setting Language Support Advanced Features Custom Vocabulary Translation Multi-language Support Usage Example Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_google_03edde75.txt b/stt_google_03edde75.txt
new file mode 100644
index 0000000000000000000000000000000000000000..e5d29bcf813a37dbd326ff7cf217f7affce5cdea
--- /dev/null
+++ b/stt_google_03edde75.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/google#param-credentials-path
+Title: Google - Pipecat
+==================================================
+
+Google - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text Google Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview GoogleSTTService provides real-time speech-to-text capabilities using Google Cloud’s Speech-to-Text V2 API. It supports interim results, multiple languages, and voice activity detection (VAD). ​ Installation To use GoogleSTTService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[google]" You’ll need Google Cloud credentials either as a JSON string or file. You can obtain Google Cloud credentials by creating a service account in the Google Cloud Console . ​ Configuration ​ Constructor Parameters ​ credentials str Google Cloud service account credentials as JSON string ​ credentials_path str Path to service account credentials JSON file ​ location str default: "global" Google Cloud location for the service ​ sample_rate int Audio sample rate in Hertz ​ params InputParams Configuration parameters for the service ​ InputParams The InputParams class provides configuration options for the Google STT service. ​ languages Language | List[Language] default: "Language.EN_US" Single language or list of recognition languages. First language is primary. Examples: Language.EN_US [Language.EN_US, Language.ES_US] The first language in the list is considered primary. Recognition accuracy may vary with multiple languages. When using multiple languages, list them in order of expected usage frequency for optimal recognition results. ​ model str default: "latest_long" Speech recognition model to use. ​ use_separate_recognition_per_channel bool default: "False" Process each audio channel separately for multi-channel audio. ​ enable_automatic_punctuation bool default: "True" Automatically add punctuation marks to transcriptions. ​ enable_spoken_punctuation bool default: "False" Include spoken punctuation (e.g., “period”, “comma”) in transcript. ​ enable_spoken_emojis bool default: "False" Include spoken emojis (e.g., “smiley face”) in transcript. ​ profanity_filter bool default: "False" Filter profanity from transcriptions. ​ enable_word_time_offsets bool default: "False" Include timing information for each word. ​ enable_word_confidence bool default: "False" Include confidence scores for each word. ​ enable_interim_results bool default: "True" Stream partial recognition results as they become available. ​ enable_voice_activity_events bool default: "False" Enable voice activity detection events. Not all features are available for all models or languages Some combinations of options may affect latency or accuracy Model selection should match your use case for best results ​ Input The service processes raw audio data with: Linear16 PCM encoding 16-bit depth Configurable sample rate Single channel (mono) ​ Output Frames The service produces two types of frames: ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Recognition language ​ InterimTranscriptionFrame Generated during ongoing speech, containing same fields as TranscriptionFrame but with preliminary results. ​ Methods ​ set_languages method Updates the service’s recognition language. Copy Ask AI async def set_languages ( language : List[Language]) -> None Example: Copy Ask AI await service.set_languages([Language. FR_FR ]) ​ set_model method Updates the service’s recognition model. Copy Ask AI async def set_model ( model : str ) -> None Example: Copy Ask AI await service.set_model( "medical_dictation" ) ​ update_options method Updates multiple service options dynamically. Copy Ask AI async def update_options ( * , languages : Optional[List[Language]] = None , model : Optional[ str ] = None , enable_automatic_punctuation : Optional[ bool ] = None , enable_spoken_punctuation : Optional[ bool ] = None , enable_spoken_emojis : Optional[ bool ] = None , profanity_filter : Optional[ bool ] = None , enable_word_time_offsets : Optional[ bool ] = None , enable_word_confidence : Optional[ bool ] = None , enable_interim_results : Optional[ bool ] = None , enable_voice_activity_events : Optional[ bool ] = None , location : Optional[ str ] = None , ) -> None Example: Copy Ask AI await service.update_options( languages = [Language. ES_ES , Language. EN_US ], enable_interim_results = True , profanity_filter = True ) See the STT base class methods for additional functionality. ​ Usage Example Copy Ask AI from pipecat.services.google.stt import GoogleSTTService from pipecat.transcriptions.language import Language # Configure service stt = GoogleSTTService( credentials_path = "path/to/credentials.json" , params = GoogleSTTService.InputParams( languages = Language. EN_US , model = "latest_long" , enable_automatic_punctuation = True , enable_interim_results = True ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, context_aggregator.user(), llm, ... ]) ​ Regional Support Google Cloud Speech-to-Text V2 supports different regional endpoints for improved latency and data residency requirements. ​ Available Regions See supported languages, models, and features for each region in Google’s Speech-to-Text documentation . ​ Configuration Specify the region during service initialization: Copy Ask AI stt = GoogleSTTService( credentials = credentials, location = "us-central1" , # Use us-central1 endpoint params = GoogleSTTService.InputParams( model = "chirp_2" ) ) ​ Dynamic Region Updates The region can be updated during runtime: Copy Ask AI await stt.update_options( location = "asia" ) ​ Notes The global endpoint is used by default Regional endpoints may provide lower latency for users in those regions Some features or models might only be available in specific regions Regional selection may affect pricing Data residency requirements may dictate region selection ​ Models Model Name Description Best For chirp_2 Google’s latest ASR model General use cases latest_long Latest model optimized for long-form speech Conversations, meetings latest_short Latest model optimized for short-form speech Short messages, notes telephony Optimized for phone calls Call centers medical_dictation Optimized for medical terminology Healthcare dictation medical_conversation Optimized for doctor-patient interactions Medical consultations See Google Cloud’s Speech-to-Text documentation for more details. ​ Language Support Language Code Description Service Codes Language.AF Afrikaans af-ZA Language.SQ Albanian sq-AL Language.AM Amharic am-ET Language.AR Arabic (Default: Egypt) ar-EG Language.AR_AE Arabic (UAE) ar-AE Language.AR_BH Arabic (Bahrain) ar-BH Language.AR_DZ Arabic (Algeria) ar-DZ Language.AR_EG Arabic (Egypt) ar-EG Language.AR_IQ Arabic (Iraq) ar-IQ Language.AR_JO Arabic (Jordan) ar-JO Language.AR_KW Arabic (Kuwait) ar-KW Language.AR_LB Arabic (Lebanon) ar-LB Language.AR_MA Arabic (Morocco) ar-MA Language.AR_OM Arabic (Oman) ar-OM Language.AR_QA Arabic (Qatar) ar-QA Language.AR_SA Arabic (Saudi Arabia) ar-SA Language.AR_SY Arabic (Syria) ar-SY Language.AR_TN Arabic (Tunisia) ar-TN Language.AR_YE Arabic (Yemen) ar-YE Language.HY Armenian hy-AM Language.AZ Azerbaijani az-AZ Language.EU Basque eu-ES Language.BN Bengali (Default: India) bn-IN Language.BN_BD Bengali (Bangladesh) bn-BD Language.BN_IN Bengali (India) bn-IN Language.BS Bosnian bs-BA Language.BG Bulgarian bg-BG Language.MY Burmese my-MM Language.CA Catalan ca-ES Language.ZH Chinese (Default: Simplified) cmn-Hans-CN Language.ZH_CN Chinese (Simplified) cmn-Hans-CN Language.ZH_HK Chinese (Hong Kong) cmn-Hans-HK Language.ZH_TW Chinese (Traditional) cmn-Hant-TW Language.YUE Chinese (Cantonese) yue-Hant-HK Language.HR Croatian hr-HR Language.CS Czech cs-CZ Language.DA Danish da-DK Language.NL Dutch (Default: Netherlands) nl-NL Language.NL_BE Dutch (Belgium) nl-BE Language.NL_NL Dutch (Netherlands) nl-NL Language.EN English (Default: US) en-US Language.EN_AU English (Australia) en-AU Language.EN_CA English (Canada) en-CA Language.EN_GB English (UK) en-GB Language.EN_GH English (Ghana) en-GH Language.EN_HK English (Hong Kong) en-HK Language.EN_IN English (India) en-IN Language.EN_IE English (Ireland) en-IE Language.EN_KE English (Kenya) en-KE Language.EN_NG English (Nigeria) en-NG Language.EN_NZ English (New Zealand) en-NZ Language.EN_PH English (Philippines) en-PH Language.EN_SG English (Singapore) en-SG Language.EN_TZ English (Tanzania) en-TZ Language.EN_US English (US) en-US Language.EN_ZA English (South Africa) en-ZA Language.ET Estonian et-EE Language.FIL Filipino fil-PH Language.FI Finnish fi-FI Language.FR French (Default: France) fr-FR Language.FR_BE French (Belgium) fr-BE Language.FR_CA French (Canada) fr-CA Language.FR_CH French (Switzerland) fr-CH Language.GL Galician gl-ES Language.KA Georgian ka-GE Language.DE German (Default: Germany) de-DE Language.DE_AT German (Austria) de-AT Language.DE_CH German (Switzerland) de-CH Language.EL Greek el-GR Language.GU Gujarati gu-IN Language.HE Hebrew iw-IL Language.HI Hindi hi-IN Language.HU Hungarian hu-HU Language.IS Icelandic is-IS Language.ID Indonesian id-ID Language.IT Italian it-IT Language.IT_CH Italian (Switzerland) it-CH Language.JA Japanese ja-JP Language.JV Javanese jv-ID Language.KN Kannada kn-IN Language.KK Kazakh kk-KZ Language.KM Khmer km-KH Language.KO Korean ko-KR Language.LO Lao lo-LA Language.LV Latvian lv-LV Language.LT Lithuanian lt-LT Language.MK Macedonian mk-MK Language.MS Malay ms-MY Language.ML Malayalam ml-IN Language.MR Marathi mr-IN Language.MN Mongolian mn-MN Language.NE Nepali ne-NP Language.NO Norwegian no-NO Language.FA Persian fa-IR Language.PL Polish pl-PL Language.PT Portuguese (Default: Portugal) pt-PT Language.PT_BR Portuguese (Brazil) pt-BR Language.PT_PT Portuguese (Portugal) pt-PT Language.PA Punjabi pa-Guru-IN Language.RO Romanian ro-RO Language.RU Russian ru-RU Language.SR Serbian sr-RS Language.SI Sinhala si-LK Language.SK Slovak sk-SK Language.SL Slovenian sl-SI Language.ES Spanish (Default: Spain) es-ES Language.ES_AR Spanish (Argentina) es-AR Language.ES_BO Spanish (Bolivia) es-BO Language.ES_CL Spanish (Chile) es-CL Language.ES_CO Spanish (Colombia) es-CO Language.ES_CR Spanish (Costa Rica) es-CR Language.ES_DO Spanish (Dominican Republic) es-DO Language.ES_EC Spanish (Ecuador) es-EC Language.ES_GT Spanish (Guatemala) es-GT Language.ES_HN Spanish (Honduras) es-HN Language.ES_MX Spanish (Mexico) es-MX Language.ES_NI Spanish (Nicaragua) es-NI Language.ES_PA Spanish (Panama) es-PA Language.ES_PE Spanish (Peru) es-PE Language.ES_PR Spanish (Puerto Rico) es-PR Language.ES_PY Spanish (Paraguay) es-PY Language.ES_SV Spanish (El Salvador) es-SV Language.ES_US Spanish (US) es-US Language.ES_UY Spanish (Uruguay) es-UY Language.ES_VE Spanish (Venezuela) es-VE Language.SU Sundanese su-ID Language.SW Swahili (Default: Tanzania) sw-TZ Language.SW_KE Swahili (Kenya) sw-KE Language.SW_TZ Swahili (Tanzania) sw-TZ Language.SV Swedish sv-SE Language.TA Tamil (Default: India) ta-IN Language.TA_IN Tamil (India) ta-IN Language.TA_MY Tamil (Malaysia) ta-MY Language.TA_SG Tamil (Singapore) ta-SG Language.TA_LK Tamil (Sri Lanka) ta-LK Language.TE Telugu te-IN Language.TH Thai th-TH Language.TR Turkish tr-TR Language.UK Ukrainian uk-UA Language.UR Urdu (Default: India) ur-IN Language.UR_IN Urdu (India) ur-IN Language.UR_PK Urdu (Pakistan) ur-PK Language.UZ Uzbek uz-UZ Language.VI Vietnamese vi-VN Language.XH Xhosa xh-ZA Language.ZU Zulu zu-ZA ​ Special Features Supports multiple languages simultaneously Provides regional variants for many languages Handles different Chinese scripts (simplified/traditional) Supports medical-specific models ​ Frame Flow ​ Notes Requires Google Cloud credentials Supports real-time transcription Handles streaming connection management Provides dynamic configuration updates Supports model switching Includes VAD capabilities Manages connection lifecycle Gladia Groq (Whisper) On this page Overview Installation Configuration Constructor Parameters InputParams Input Output Frames TranscriptionFrame InterimTranscriptionFrame Methods Usage Example Regional Support Available Regions Configuration Dynamic Region Updates Notes Models Language Support Special Features Frame Flow Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_google_17647674.txt b/stt_google_17647674.txt
new file mode 100644
index 0000000000000000000000000000000000000000..9924559d929a9e2f30772494bf223bd1ade066a5
--- /dev/null
+++ b/stt_google_17647674.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/google#output-frames
+Title: Google - Pipecat
+==================================================
+
+Google - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text Google Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview GoogleSTTService provides real-time speech-to-text capabilities using Google Cloud’s Speech-to-Text V2 API. It supports interim results, multiple languages, and voice activity detection (VAD). ​ Installation To use GoogleSTTService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[google]" You’ll need Google Cloud credentials either as a JSON string or file. You can obtain Google Cloud credentials by creating a service account in the Google Cloud Console . ​ Configuration ​ Constructor Parameters ​ credentials str Google Cloud service account credentials as JSON string ​ credentials_path str Path to service account credentials JSON file ​ location str default: "global" Google Cloud location for the service ​ sample_rate int Audio sample rate in Hertz ​ params InputParams Configuration parameters for the service ​ InputParams The InputParams class provides configuration options for the Google STT service. ​ languages Language | List[Language] default: "Language.EN_US" Single language or list of recognition languages. First language is primary. Examples: Language.EN_US [Language.EN_US, Language.ES_US] The first language in the list is considered primary. Recognition accuracy may vary with multiple languages. When using multiple languages, list them in order of expected usage frequency for optimal recognition results. ​ model str default: "latest_long" Speech recognition model to use. ​ use_separate_recognition_per_channel bool default: "False" Process each audio channel separately for multi-channel audio. ​ enable_automatic_punctuation bool default: "True" Automatically add punctuation marks to transcriptions. ​ enable_spoken_punctuation bool default: "False" Include spoken punctuation (e.g., “period”, “comma”) in transcript. ​ enable_spoken_emojis bool default: "False" Include spoken emojis (e.g., “smiley face”) in transcript. ​ profanity_filter bool default: "False" Filter profanity from transcriptions. ​ enable_word_time_offsets bool default: "False" Include timing information for each word. ​ enable_word_confidence bool default: "False" Include confidence scores for each word. ​ enable_interim_results bool default: "True" Stream partial recognition results as they become available. ​ enable_voice_activity_events bool default: "False" Enable voice activity detection events. Not all features are available for all models or languages Some combinations of options may affect latency or accuracy Model selection should match your use case for best results ​ Input The service processes raw audio data with: Linear16 PCM encoding 16-bit depth Configurable sample rate Single channel (mono) ​ Output Frames The service produces two types of frames: ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Recognition language ​ InterimTranscriptionFrame Generated during ongoing speech, containing same fields as TranscriptionFrame but with preliminary results. ​ Methods ​ set_languages method Updates the service’s recognition language. Copy Ask AI async def set_languages ( language : List[Language]) -> None Example: Copy Ask AI await service.set_languages([Language. FR_FR ]) ​ set_model method Updates the service’s recognition model. Copy Ask AI async def set_model ( model : str ) -> None Example: Copy Ask AI await service.set_model( "medical_dictation" ) ​ update_options method Updates multiple service options dynamically. Copy Ask AI async def update_options ( * , languages : Optional[List[Language]] = None , model : Optional[ str ] = None , enable_automatic_punctuation : Optional[ bool ] = None , enable_spoken_punctuation : Optional[ bool ] = None , enable_spoken_emojis : Optional[ bool ] = None , profanity_filter : Optional[ bool ] = None , enable_word_time_offsets : Optional[ bool ] = None , enable_word_confidence : Optional[ bool ] = None , enable_interim_results : Optional[ bool ] = None , enable_voice_activity_events : Optional[ bool ] = None , location : Optional[ str ] = None , ) -> None Example: Copy Ask AI await service.update_options( languages = [Language. ES_ES , Language. EN_US ], enable_interim_results = True , profanity_filter = True ) See the STT base class methods for additional functionality. ​ Usage Example Copy Ask AI from pipecat.services.google.stt import GoogleSTTService from pipecat.transcriptions.language import Language # Configure service stt = GoogleSTTService( credentials_path = "path/to/credentials.json" , params = GoogleSTTService.InputParams( languages = Language. EN_US , model = "latest_long" , enable_automatic_punctuation = True , enable_interim_results = True ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, context_aggregator.user(), llm, ... ]) ​ Regional Support Google Cloud Speech-to-Text V2 supports different regional endpoints for improved latency and data residency requirements. ​ Available Regions See supported languages, models, and features for each region in Google’s Speech-to-Text documentation . ​ Configuration Specify the region during service initialization: Copy Ask AI stt = GoogleSTTService( credentials = credentials, location = "us-central1" , # Use us-central1 endpoint params = GoogleSTTService.InputParams( model = "chirp_2" ) ) ​ Dynamic Region Updates The region can be updated during runtime: Copy Ask AI await stt.update_options( location = "asia" ) ​ Notes The global endpoint is used by default Regional endpoints may provide lower latency for users in those regions Some features or models might only be available in specific regions Regional selection may affect pricing Data residency requirements may dictate region selection ​ Models Model Name Description Best For chirp_2 Google’s latest ASR model General use cases latest_long Latest model optimized for long-form speech Conversations, meetings latest_short Latest model optimized for short-form speech Short messages, notes telephony Optimized for phone calls Call centers medical_dictation Optimized for medical terminology Healthcare dictation medical_conversation Optimized for doctor-patient interactions Medical consultations See Google Cloud’s Speech-to-Text documentation for more details. ​ Language Support Language Code Description Service Codes Language.AF Afrikaans af-ZA Language.SQ Albanian sq-AL Language.AM Amharic am-ET Language.AR Arabic (Default: Egypt) ar-EG Language.AR_AE Arabic (UAE) ar-AE Language.AR_BH Arabic (Bahrain) ar-BH Language.AR_DZ Arabic (Algeria) ar-DZ Language.AR_EG Arabic (Egypt) ar-EG Language.AR_IQ Arabic (Iraq) ar-IQ Language.AR_JO Arabic (Jordan) ar-JO Language.AR_KW Arabic (Kuwait) ar-KW Language.AR_LB Arabic (Lebanon) ar-LB Language.AR_MA Arabic (Morocco) ar-MA Language.AR_OM Arabic (Oman) ar-OM Language.AR_QA Arabic (Qatar) ar-QA Language.AR_SA Arabic (Saudi Arabia) ar-SA Language.AR_SY Arabic (Syria) ar-SY Language.AR_TN Arabic (Tunisia) ar-TN Language.AR_YE Arabic (Yemen) ar-YE Language.HY Armenian hy-AM Language.AZ Azerbaijani az-AZ Language.EU Basque eu-ES Language.BN Bengali (Default: India) bn-IN Language.BN_BD Bengali (Bangladesh) bn-BD Language.BN_IN Bengali (India) bn-IN Language.BS Bosnian bs-BA Language.BG Bulgarian bg-BG Language.MY Burmese my-MM Language.CA Catalan ca-ES Language.ZH Chinese (Default: Simplified) cmn-Hans-CN Language.ZH_CN Chinese (Simplified) cmn-Hans-CN Language.ZH_HK Chinese (Hong Kong) cmn-Hans-HK Language.ZH_TW Chinese (Traditional) cmn-Hant-TW Language.YUE Chinese (Cantonese) yue-Hant-HK Language.HR Croatian hr-HR Language.CS Czech cs-CZ Language.DA Danish da-DK Language.NL Dutch (Default: Netherlands) nl-NL Language.NL_BE Dutch (Belgium) nl-BE Language.NL_NL Dutch (Netherlands) nl-NL Language.EN English (Default: US) en-US Language.EN_AU English (Australia) en-AU Language.EN_CA English (Canada) en-CA Language.EN_GB English (UK) en-GB Language.EN_GH English (Ghana) en-GH Language.EN_HK English (Hong Kong) en-HK Language.EN_IN English (India) en-IN Language.EN_IE English (Ireland) en-IE Language.EN_KE English (Kenya) en-KE Language.EN_NG English (Nigeria) en-NG Language.EN_NZ English (New Zealand) en-NZ Language.EN_PH English (Philippines) en-PH Language.EN_SG English (Singapore) en-SG Language.EN_TZ English (Tanzania) en-TZ Language.EN_US English (US) en-US Language.EN_ZA English (South Africa) en-ZA Language.ET Estonian et-EE Language.FIL Filipino fil-PH Language.FI Finnish fi-FI Language.FR French (Default: France) fr-FR Language.FR_BE French (Belgium) fr-BE Language.FR_CA French (Canada) fr-CA Language.FR_CH French (Switzerland) fr-CH Language.GL Galician gl-ES Language.KA Georgian ka-GE Language.DE German (Default: Germany) de-DE Language.DE_AT German (Austria) de-AT Language.DE_CH German (Switzerland) de-CH Language.EL Greek el-GR Language.GU Gujarati gu-IN Language.HE Hebrew iw-IL Language.HI Hindi hi-IN Language.HU Hungarian hu-HU Language.IS Icelandic is-IS Language.ID Indonesian id-ID Language.IT Italian it-IT Language.IT_CH Italian (Switzerland) it-CH Language.JA Japanese ja-JP Language.JV Javanese jv-ID Language.KN Kannada kn-IN Language.KK Kazakh kk-KZ Language.KM Khmer km-KH Language.KO Korean ko-KR Language.LO Lao lo-LA Language.LV Latvian lv-LV Language.LT Lithuanian lt-LT Language.MK Macedonian mk-MK Language.MS Malay ms-MY Language.ML Malayalam ml-IN Language.MR Marathi mr-IN Language.MN Mongolian mn-MN Language.NE Nepali ne-NP Language.NO Norwegian no-NO Language.FA Persian fa-IR Language.PL Polish pl-PL Language.PT Portuguese (Default: Portugal) pt-PT Language.PT_BR Portuguese (Brazil) pt-BR Language.PT_PT Portuguese (Portugal) pt-PT Language.PA Punjabi pa-Guru-IN Language.RO Romanian ro-RO Language.RU Russian ru-RU Language.SR Serbian sr-RS Language.SI Sinhala si-LK Language.SK Slovak sk-SK Language.SL Slovenian sl-SI Language.ES Spanish (Default: Spain) es-ES Language.ES_AR Spanish (Argentina) es-AR Language.ES_BO Spanish (Bolivia) es-BO Language.ES_CL Spanish (Chile) es-CL Language.ES_CO Spanish (Colombia) es-CO Language.ES_CR Spanish (Costa Rica) es-CR Language.ES_DO Spanish (Dominican Republic) es-DO Language.ES_EC Spanish (Ecuador) es-EC Language.ES_GT Spanish (Guatemala) es-GT Language.ES_HN Spanish (Honduras) es-HN Language.ES_MX Spanish (Mexico) es-MX Language.ES_NI Spanish (Nicaragua) es-NI Language.ES_PA Spanish (Panama) es-PA Language.ES_PE Spanish (Peru) es-PE Language.ES_PR Spanish (Puerto Rico) es-PR Language.ES_PY Spanish (Paraguay) es-PY Language.ES_SV Spanish (El Salvador) es-SV Language.ES_US Spanish (US) es-US Language.ES_UY Spanish (Uruguay) es-UY Language.ES_VE Spanish (Venezuela) es-VE Language.SU Sundanese su-ID Language.SW Swahili (Default: Tanzania) sw-TZ Language.SW_KE Swahili (Kenya) sw-KE Language.SW_TZ Swahili (Tanzania) sw-TZ Language.SV Swedish sv-SE Language.TA Tamil (Default: India) ta-IN Language.TA_IN Tamil (India) ta-IN Language.TA_MY Tamil (Malaysia) ta-MY Language.TA_SG Tamil (Singapore) ta-SG Language.TA_LK Tamil (Sri Lanka) ta-LK Language.TE Telugu te-IN Language.TH Thai th-TH Language.TR Turkish tr-TR Language.UK Ukrainian uk-UA Language.UR Urdu (Default: India) ur-IN Language.UR_IN Urdu (India) ur-IN Language.UR_PK Urdu (Pakistan) ur-PK Language.UZ Uzbek uz-UZ Language.VI Vietnamese vi-VN Language.XH Xhosa xh-ZA Language.ZU Zulu zu-ZA ​ Special Features Supports multiple languages simultaneously Provides regional variants for many languages Handles different Chinese scripts (simplified/traditional) Supports medical-specific models ​ Frame Flow ​ Notes Requires Google Cloud credentials Supports real-time transcription Handles streaming connection management Provides dynamic configuration updates Supports model switching Includes VAD capabilities Manages connection lifecycle Gladia Groq (Whisper) On this page Overview Installation Configuration Constructor Parameters InputParams Input Output Frames TranscriptionFrame InterimTranscriptionFrame Methods Usage Example Regional Support Available Regions Configuration Dynamic Region Updates Notes Models Language Support Special Features Frame Flow Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_google_295bb484.txt b/stt_google_295bb484.txt
new file mode 100644
index 0000000000000000000000000000000000000000..2c441b771de6206c7ed9a564e9dfa8efd82f38e0
--- /dev/null
+++ b/stt_google_295bb484.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/google#param-enable-automatic-punctuation
+Title: Google - Pipecat
+==================================================
+
+Google - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text Google Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview GoogleSTTService provides real-time speech-to-text capabilities using Google Cloud’s Speech-to-Text V2 API. It supports interim results, multiple languages, and voice activity detection (VAD). ​ Installation To use GoogleSTTService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[google]" You’ll need Google Cloud credentials either as a JSON string or file. You can obtain Google Cloud credentials by creating a service account in the Google Cloud Console . ​ Configuration ​ Constructor Parameters ​ credentials str Google Cloud service account credentials as JSON string ​ credentials_path str Path to service account credentials JSON file ​ location str default: "global" Google Cloud location for the service ​ sample_rate int Audio sample rate in Hertz ​ params InputParams Configuration parameters for the service ​ InputParams The InputParams class provides configuration options for the Google STT service. ​ languages Language | List[Language] default: "Language.EN_US" Single language or list of recognition languages. First language is primary. Examples: Language.EN_US [Language.EN_US, Language.ES_US] The first language in the list is considered primary. Recognition accuracy may vary with multiple languages. When using multiple languages, list them in order of expected usage frequency for optimal recognition results. ​ model str default: "latest_long" Speech recognition model to use. ​ use_separate_recognition_per_channel bool default: "False" Process each audio channel separately for multi-channel audio. ​ enable_automatic_punctuation bool default: "True" Automatically add punctuation marks to transcriptions. ​ enable_spoken_punctuation bool default: "False" Include spoken punctuation (e.g., “period”, “comma”) in transcript. ​ enable_spoken_emojis bool default: "False" Include spoken emojis (e.g., “smiley face”) in transcript. ​ profanity_filter bool default: "False" Filter profanity from transcriptions. ​ enable_word_time_offsets bool default: "False" Include timing information for each word. ​ enable_word_confidence bool default: "False" Include confidence scores for each word. ​ enable_interim_results bool default: "True" Stream partial recognition results as they become available. ​ enable_voice_activity_events bool default: "False" Enable voice activity detection events. Not all features are available for all models or languages Some combinations of options may affect latency or accuracy Model selection should match your use case for best results ​ Input The service processes raw audio data with: Linear16 PCM encoding 16-bit depth Configurable sample rate Single channel (mono) ​ Output Frames The service produces two types of frames: ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Recognition language ​ InterimTranscriptionFrame Generated during ongoing speech, containing same fields as TranscriptionFrame but with preliminary results. ​ Methods ​ set_languages method Updates the service’s recognition language. Copy Ask AI async def set_languages ( language : List[Language]) -> None Example: Copy Ask AI await service.set_languages([Language. FR_FR ]) ​ set_model method Updates the service’s recognition model. Copy Ask AI async def set_model ( model : str ) -> None Example: Copy Ask AI await service.set_model( "medical_dictation" ) ​ update_options method Updates multiple service options dynamically. Copy Ask AI async def update_options ( * , languages : Optional[List[Language]] = None , model : Optional[ str ] = None , enable_automatic_punctuation : Optional[ bool ] = None , enable_spoken_punctuation : Optional[ bool ] = None , enable_spoken_emojis : Optional[ bool ] = None , profanity_filter : Optional[ bool ] = None , enable_word_time_offsets : Optional[ bool ] = None , enable_word_confidence : Optional[ bool ] = None , enable_interim_results : Optional[ bool ] = None , enable_voice_activity_events : Optional[ bool ] = None , location : Optional[ str ] = None , ) -> None Example: Copy Ask AI await service.update_options( languages = [Language. ES_ES , Language. EN_US ], enable_interim_results = True , profanity_filter = True ) See the STT base class methods for additional functionality. ​ Usage Example Copy Ask AI from pipecat.services.google.stt import GoogleSTTService from pipecat.transcriptions.language import Language # Configure service stt = GoogleSTTService( credentials_path = "path/to/credentials.json" , params = GoogleSTTService.InputParams( languages = Language. EN_US , model = "latest_long" , enable_automatic_punctuation = True , enable_interim_results = True ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, context_aggregator.user(), llm, ... ]) ​ Regional Support Google Cloud Speech-to-Text V2 supports different regional endpoints for improved latency and data residency requirements. ​ Available Regions See supported languages, models, and features for each region in Google’s Speech-to-Text documentation . ​ Configuration Specify the region during service initialization: Copy Ask AI stt = GoogleSTTService( credentials = credentials, location = "us-central1" , # Use us-central1 endpoint params = GoogleSTTService.InputParams( model = "chirp_2" ) ) ​ Dynamic Region Updates The region can be updated during runtime: Copy Ask AI await stt.update_options( location = "asia" ) ​ Notes The global endpoint is used by default Regional endpoints may provide lower latency for users in those regions Some features or models might only be available in specific regions Regional selection may affect pricing Data residency requirements may dictate region selection ​ Models Model Name Description Best For chirp_2 Google’s latest ASR model General use cases latest_long Latest model optimized for long-form speech Conversations, meetings latest_short Latest model optimized for short-form speech Short messages, notes telephony Optimized for phone calls Call centers medical_dictation Optimized for medical terminology Healthcare dictation medical_conversation Optimized for doctor-patient interactions Medical consultations See Google Cloud’s Speech-to-Text documentation for more details. ​ Language Support Language Code Description Service Codes Language.AF Afrikaans af-ZA Language.SQ Albanian sq-AL Language.AM Amharic am-ET Language.AR Arabic (Default: Egypt) ar-EG Language.AR_AE Arabic (UAE) ar-AE Language.AR_BH Arabic (Bahrain) ar-BH Language.AR_DZ Arabic (Algeria) ar-DZ Language.AR_EG Arabic (Egypt) ar-EG Language.AR_IQ Arabic (Iraq) ar-IQ Language.AR_JO Arabic (Jordan) ar-JO Language.AR_KW Arabic (Kuwait) ar-KW Language.AR_LB Arabic (Lebanon) ar-LB Language.AR_MA Arabic (Morocco) ar-MA Language.AR_OM Arabic (Oman) ar-OM Language.AR_QA Arabic (Qatar) ar-QA Language.AR_SA Arabic (Saudi Arabia) ar-SA Language.AR_SY Arabic (Syria) ar-SY Language.AR_TN Arabic (Tunisia) ar-TN Language.AR_YE Arabic (Yemen) ar-YE Language.HY Armenian hy-AM Language.AZ Azerbaijani az-AZ Language.EU Basque eu-ES Language.BN Bengali (Default: India) bn-IN Language.BN_BD Bengali (Bangladesh) bn-BD Language.BN_IN Bengali (India) bn-IN Language.BS Bosnian bs-BA Language.BG Bulgarian bg-BG Language.MY Burmese my-MM Language.CA Catalan ca-ES Language.ZH Chinese (Default: Simplified) cmn-Hans-CN Language.ZH_CN Chinese (Simplified) cmn-Hans-CN Language.ZH_HK Chinese (Hong Kong) cmn-Hans-HK Language.ZH_TW Chinese (Traditional) cmn-Hant-TW Language.YUE Chinese (Cantonese) yue-Hant-HK Language.HR Croatian hr-HR Language.CS Czech cs-CZ Language.DA Danish da-DK Language.NL Dutch (Default: Netherlands) nl-NL Language.NL_BE Dutch (Belgium) nl-BE Language.NL_NL Dutch (Netherlands) nl-NL Language.EN English (Default: US) en-US Language.EN_AU English (Australia) en-AU Language.EN_CA English (Canada) en-CA Language.EN_GB English (UK) en-GB Language.EN_GH English (Ghana) en-GH Language.EN_HK English (Hong Kong) en-HK Language.EN_IN English (India) en-IN Language.EN_IE English (Ireland) en-IE Language.EN_KE English (Kenya) en-KE Language.EN_NG English (Nigeria) en-NG Language.EN_NZ English (New Zealand) en-NZ Language.EN_PH English (Philippines) en-PH Language.EN_SG English (Singapore) en-SG Language.EN_TZ English (Tanzania) en-TZ Language.EN_US English (US) en-US Language.EN_ZA English (South Africa) en-ZA Language.ET Estonian et-EE Language.FIL Filipino fil-PH Language.FI Finnish fi-FI Language.FR French (Default: France) fr-FR Language.FR_BE French (Belgium) fr-BE Language.FR_CA French (Canada) fr-CA Language.FR_CH French (Switzerland) fr-CH Language.GL Galician gl-ES Language.KA Georgian ka-GE Language.DE German (Default: Germany) de-DE Language.DE_AT German (Austria) de-AT Language.DE_CH German (Switzerland) de-CH Language.EL Greek el-GR Language.GU Gujarati gu-IN Language.HE Hebrew iw-IL Language.HI Hindi hi-IN Language.HU Hungarian hu-HU Language.IS Icelandic is-IS Language.ID Indonesian id-ID Language.IT Italian it-IT Language.IT_CH Italian (Switzerland) it-CH Language.JA Japanese ja-JP Language.JV Javanese jv-ID Language.KN Kannada kn-IN Language.KK Kazakh kk-KZ Language.KM Khmer km-KH Language.KO Korean ko-KR Language.LO Lao lo-LA Language.LV Latvian lv-LV Language.LT Lithuanian lt-LT Language.MK Macedonian mk-MK Language.MS Malay ms-MY Language.ML Malayalam ml-IN Language.MR Marathi mr-IN Language.MN Mongolian mn-MN Language.NE Nepali ne-NP Language.NO Norwegian no-NO Language.FA Persian fa-IR Language.PL Polish pl-PL Language.PT Portuguese (Default: Portugal) pt-PT Language.PT_BR Portuguese (Brazil) pt-BR Language.PT_PT Portuguese (Portugal) pt-PT Language.PA Punjabi pa-Guru-IN Language.RO Romanian ro-RO Language.RU Russian ru-RU Language.SR Serbian sr-RS Language.SI Sinhala si-LK Language.SK Slovak sk-SK Language.SL Slovenian sl-SI Language.ES Spanish (Default: Spain) es-ES Language.ES_AR Spanish (Argentina) es-AR Language.ES_BO Spanish (Bolivia) es-BO Language.ES_CL Spanish (Chile) es-CL Language.ES_CO Spanish (Colombia) es-CO Language.ES_CR Spanish (Costa Rica) es-CR Language.ES_DO Spanish (Dominican Republic) es-DO Language.ES_EC Spanish (Ecuador) es-EC Language.ES_GT Spanish (Guatemala) es-GT Language.ES_HN Spanish (Honduras) es-HN Language.ES_MX Spanish (Mexico) es-MX Language.ES_NI Spanish (Nicaragua) es-NI Language.ES_PA Spanish (Panama) es-PA Language.ES_PE Spanish (Peru) es-PE Language.ES_PR Spanish (Puerto Rico) es-PR Language.ES_PY Spanish (Paraguay) es-PY Language.ES_SV Spanish (El Salvador) es-SV Language.ES_US Spanish (US) es-US Language.ES_UY Spanish (Uruguay) es-UY Language.ES_VE Spanish (Venezuela) es-VE Language.SU Sundanese su-ID Language.SW Swahili (Default: Tanzania) sw-TZ Language.SW_KE Swahili (Kenya) sw-KE Language.SW_TZ Swahili (Tanzania) sw-TZ Language.SV Swedish sv-SE Language.TA Tamil (Default: India) ta-IN Language.TA_IN Tamil (India) ta-IN Language.TA_MY Tamil (Malaysia) ta-MY Language.TA_SG Tamil (Singapore) ta-SG Language.TA_LK Tamil (Sri Lanka) ta-LK Language.TE Telugu te-IN Language.TH Thai th-TH Language.TR Turkish tr-TR Language.UK Ukrainian uk-UA Language.UR Urdu (Default: India) ur-IN Language.UR_IN Urdu (India) ur-IN Language.UR_PK Urdu (Pakistan) ur-PK Language.UZ Uzbek uz-UZ Language.VI Vietnamese vi-VN Language.XH Xhosa xh-ZA Language.ZU Zulu zu-ZA ​ Special Features Supports multiple languages simultaneously Provides regional variants for many languages Handles different Chinese scripts (simplified/traditional) Supports medical-specific models ​ Frame Flow ​ Notes Requires Google Cloud credentials Supports real-time transcription Handles streaming connection management Provides dynamic configuration updates Supports model switching Includes VAD capabilities Manages connection lifecycle Gladia Groq (Whisper) On this page Overview Installation Configuration Constructor Parameters InputParams Input Output Frames TranscriptionFrame InterimTranscriptionFrame Methods Usage Example Regional Support Available Regions Configuration Dynamic Region Updates Notes Models Language Support Special Features Frame Flow Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_google_41021e1e.txt b/stt_google_41021e1e.txt
new file mode 100644
index 0000000000000000000000000000000000000000..ec296cee1e1b2d659ce850536c74b3a9f4b7ae4b
--- /dev/null
+++ b/stt_google_41021e1e.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/google#param-sample-rate
+Title: Google - Pipecat
+==================================================
+
+Google - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text Google Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview GoogleSTTService provides real-time speech-to-text capabilities using Google Cloud’s Speech-to-Text V2 API. It supports interim results, multiple languages, and voice activity detection (VAD). ​ Installation To use GoogleSTTService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[google]" You’ll need Google Cloud credentials either as a JSON string or file. You can obtain Google Cloud credentials by creating a service account in the Google Cloud Console . ​ Configuration ​ Constructor Parameters ​ credentials str Google Cloud service account credentials as JSON string ​ credentials_path str Path to service account credentials JSON file ​ location str default: "global" Google Cloud location for the service ​ sample_rate int Audio sample rate in Hertz ​ params InputParams Configuration parameters for the service ​ InputParams The InputParams class provides configuration options for the Google STT service. ​ languages Language | List[Language] default: "Language.EN_US" Single language or list of recognition languages. First language is primary. Examples: Language.EN_US [Language.EN_US, Language.ES_US] The first language in the list is considered primary. Recognition accuracy may vary with multiple languages. When using multiple languages, list them in order of expected usage frequency for optimal recognition results. ​ model str default: "latest_long" Speech recognition model to use. ​ use_separate_recognition_per_channel bool default: "False" Process each audio channel separately for multi-channel audio. ​ enable_automatic_punctuation bool default: "True" Automatically add punctuation marks to transcriptions. ​ enable_spoken_punctuation bool default: "False" Include spoken punctuation (e.g., “period”, “comma”) in transcript. ​ enable_spoken_emojis bool default: "False" Include spoken emojis (e.g., “smiley face”) in transcript. ​ profanity_filter bool default: "False" Filter profanity from transcriptions. ​ enable_word_time_offsets bool default: "False" Include timing information for each word. ​ enable_word_confidence bool default: "False" Include confidence scores for each word. ​ enable_interim_results bool default: "True" Stream partial recognition results as they become available. ​ enable_voice_activity_events bool default: "False" Enable voice activity detection events. Not all features are available for all models or languages Some combinations of options may affect latency or accuracy Model selection should match your use case for best results ​ Input The service processes raw audio data with: Linear16 PCM encoding 16-bit depth Configurable sample rate Single channel (mono) ​ Output Frames The service produces two types of frames: ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Recognition language ​ InterimTranscriptionFrame Generated during ongoing speech, containing same fields as TranscriptionFrame but with preliminary results. ​ Methods ​ set_languages method Updates the service’s recognition language. Copy Ask AI async def set_languages ( language : List[Language]) -> None Example: Copy Ask AI await service.set_languages([Language. FR_FR ]) ​ set_model method Updates the service’s recognition model. Copy Ask AI async def set_model ( model : str ) -> None Example: Copy Ask AI await service.set_model( "medical_dictation" ) ​ update_options method Updates multiple service options dynamically. Copy Ask AI async def update_options ( * , languages : Optional[List[Language]] = None , model : Optional[ str ] = None , enable_automatic_punctuation : Optional[ bool ] = None , enable_spoken_punctuation : Optional[ bool ] = None , enable_spoken_emojis : Optional[ bool ] = None , profanity_filter : Optional[ bool ] = None , enable_word_time_offsets : Optional[ bool ] = None , enable_word_confidence : Optional[ bool ] = None , enable_interim_results : Optional[ bool ] = None , enable_voice_activity_events : Optional[ bool ] = None , location : Optional[ str ] = None , ) -> None Example: Copy Ask AI await service.update_options( languages = [Language. ES_ES , Language. EN_US ], enable_interim_results = True , profanity_filter = True ) See the STT base class methods for additional functionality. ​ Usage Example Copy Ask AI from pipecat.services.google.stt import GoogleSTTService from pipecat.transcriptions.language import Language # Configure service stt = GoogleSTTService( credentials_path = "path/to/credentials.json" , params = GoogleSTTService.InputParams( languages = Language. EN_US , model = "latest_long" , enable_automatic_punctuation = True , enable_interim_results = True ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, context_aggregator.user(), llm, ... ]) ​ Regional Support Google Cloud Speech-to-Text V2 supports different regional endpoints for improved latency and data residency requirements. ​ Available Regions See supported languages, models, and features for each region in Google’s Speech-to-Text documentation . ​ Configuration Specify the region during service initialization: Copy Ask AI stt = GoogleSTTService( credentials = credentials, location = "us-central1" , # Use us-central1 endpoint params = GoogleSTTService.InputParams( model = "chirp_2" ) ) ​ Dynamic Region Updates The region can be updated during runtime: Copy Ask AI await stt.update_options( location = "asia" ) ​ Notes The global endpoint is used by default Regional endpoints may provide lower latency for users in those regions Some features or models might only be available in specific regions Regional selection may affect pricing Data residency requirements may dictate region selection ​ Models Model Name Description Best For chirp_2 Google’s latest ASR model General use cases latest_long Latest model optimized for long-form speech Conversations, meetings latest_short Latest model optimized for short-form speech Short messages, notes telephony Optimized for phone calls Call centers medical_dictation Optimized for medical terminology Healthcare dictation medical_conversation Optimized for doctor-patient interactions Medical consultations See Google Cloud’s Speech-to-Text documentation for more details. ​ Language Support Language Code Description Service Codes Language.AF Afrikaans af-ZA Language.SQ Albanian sq-AL Language.AM Amharic am-ET Language.AR Arabic (Default: Egypt) ar-EG Language.AR_AE Arabic (UAE) ar-AE Language.AR_BH Arabic (Bahrain) ar-BH Language.AR_DZ Arabic (Algeria) ar-DZ Language.AR_EG Arabic (Egypt) ar-EG Language.AR_IQ Arabic (Iraq) ar-IQ Language.AR_JO Arabic (Jordan) ar-JO Language.AR_KW Arabic (Kuwait) ar-KW Language.AR_LB Arabic (Lebanon) ar-LB Language.AR_MA Arabic (Morocco) ar-MA Language.AR_OM Arabic (Oman) ar-OM Language.AR_QA Arabic (Qatar) ar-QA Language.AR_SA Arabic (Saudi Arabia) ar-SA Language.AR_SY Arabic (Syria) ar-SY Language.AR_TN Arabic (Tunisia) ar-TN Language.AR_YE Arabic (Yemen) ar-YE Language.HY Armenian hy-AM Language.AZ Azerbaijani az-AZ Language.EU Basque eu-ES Language.BN Bengali (Default: India) bn-IN Language.BN_BD Bengali (Bangladesh) bn-BD Language.BN_IN Bengali (India) bn-IN Language.BS Bosnian bs-BA Language.BG Bulgarian bg-BG Language.MY Burmese my-MM Language.CA Catalan ca-ES Language.ZH Chinese (Default: Simplified) cmn-Hans-CN Language.ZH_CN Chinese (Simplified) cmn-Hans-CN Language.ZH_HK Chinese (Hong Kong) cmn-Hans-HK Language.ZH_TW Chinese (Traditional) cmn-Hant-TW Language.YUE Chinese (Cantonese) yue-Hant-HK Language.HR Croatian hr-HR Language.CS Czech cs-CZ Language.DA Danish da-DK Language.NL Dutch (Default: Netherlands) nl-NL Language.NL_BE Dutch (Belgium) nl-BE Language.NL_NL Dutch (Netherlands) nl-NL Language.EN English (Default: US) en-US Language.EN_AU English (Australia) en-AU Language.EN_CA English (Canada) en-CA Language.EN_GB English (UK) en-GB Language.EN_GH English (Ghana) en-GH Language.EN_HK English (Hong Kong) en-HK Language.EN_IN English (India) en-IN Language.EN_IE English (Ireland) en-IE Language.EN_KE English (Kenya) en-KE Language.EN_NG English (Nigeria) en-NG Language.EN_NZ English (New Zealand) en-NZ Language.EN_PH English (Philippines) en-PH Language.EN_SG English (Singapore) en-SG Language.EN_TZ English (Tanzania) en-TZ Language.EN_US English (US) en-US Language.EN_ZA English (South Africa) en-ZA Language.ET Estonian et-EE Language.FIL Filipino fil-PH Language.FI Finnish fi-FI Language.FR French (Default: France) fr-FR Language.FR_BE French (Belgium) fr-BE Language.FR_CA French (Canada) fr-CA Language.FR_CH French (Switzerland) fr-CH Language.GL Galician gl-ES Language.KA Georgian ka-GE Language.DE German (Default: Germany) de-DE Language.DE_AT German (Austria) de-AT Language.DE_CH German (Switzerland) de-CH Language.EL Greek el-GR Language.GU Gujarati gu-IN Language.HE Hebrew iw-IL Language.HI Hindi hi-IN Language.HU Hungarian hu-HU Language.IS Icelandic is-IS Language.ID Indonesian id-ID Language.IT Italian it-IT Language.IT_CH Italian (Switzerland) it-CH Language.JA Japanese ja-JP Language.JV Javanese jv-ID Language.KN Kannada kn-IN Language.KK Kazakh kk-KZ Language.KM Khmer km-KH Language.KO Korean ko-KR Language.LO Lao lo-LA Language.LV Latvian lv-LV Language.LT Lithuanian lt-LT Language.MK Macedonian mk-MK Language.MS Malay ms-MY Language.ML Malayalam ml-IN Language.MR Marathi mr-IN Language.MN Mongolian mn-MN Language.NE Nepali ne-NP Language.NO Norwegian no-NO Language.FA Persian fa-IR Language.PL Polish pl-PL Language.PT Portuguese (Default: Portugal) pt-PT Language.PT_BR Portuguese (Brazil) pt-BR Language.PT_PT Portuguese (Portugal) pt-PT Language.PA Punjabi pa-Guru-IN Language.RO Romanian ro-RO Language.RU Russian ru-RU Language.SR Serbian sr-RS Language.SI Sinhala si-LK Language.SK Slovak sk-SK Language.SL Slovenian sl-SI Language.ES Spanish (Default: Spain) es-ES Language.ES_AR Spanish (Argentina) es-AR Language.ES_BO Spanish (Bolivia) es-BO Language.ES_CL Spanish (Chile) es-CL Language.ES_CO Spanish (Colombia) es-CO Language.ES_CR Spanish (Costa Rica) es-CR Language.ES_DO Spanish (Dominican Republic) es-DO Language.ES_EC Spanish (Ecuador) es-EC Language.ES_GT Spanish (Guatemala) es-GT Language.ES_HN Spanish (Honduras) es-HN Language.ES_MX Spanish (Mexico) es-MX Language.ES_NI Spanish (Nicaragua) es-NI Language.ES_PA Spanish (Panama) es-PA Language.ES_PE Spanish (Peru) es-PE Language.ES_PR Spanish (Puerto Rico) es-PR Language.ES_PY Spanish (Paraguay) es-PY Language.ES_SV Spanish (El Salvador) es-SV Language.ES_US Spanish (US) es-US Language.ES_UY Spanish (Uruguay) es-UY Language.ES_VE Spanish (Venezuela) es-VE Language.SU Sundanese su-ID Language.SW Swahili (Default: Tanzania) sw-TZ Language.SW_KE Swahili (Kenya) sw-KE Language.SW_TZ Swahili (Tanzania) sw-TZ Language.SV Swedish sv-SE Language.TA Tamil (Default: India) ta-IN Language.TA_IN Tamil (India) ta-IN Language.TA_MY Tamil (Malaysia) ta-MY Language.TA_SG Tamil (Singapore) ta-SG Language.TA_LK Tamil (Sri Lanka) ta-LK Language.TE Telugu te-IN Language.TH Thai th-TH Language.TR Turkish tr-TR Language.UK Ukrainian uk-UA Language.UR Urdu (Default: India) ur-IN Language.UR_IN Urdu (India) ur-IN Language.UR_PK Urdu (Pakistan) ur-PK Language.UZ Uzbek uz-UZ Language.VI Vietnamese vi-VN Language.XH Xhosa xh-ZA Language.ZU Zulu zu-ZA ​ Special Features Supports multiple languages simultaneously Provides regional variants for many languages Handles different Chinese scripts (simplified/traditional) Supports medical-specific models ​ Frame Flow ​ Notes Requires Google Cloud credentials Supports real-time transcription Handles streaming connection management Provides dynamic configuration updates Supports model switching Includes VAD capabilities Manages connection lifecycle Gladia Groq (Whisper) On this page Overview Installation Configuration Constructor Parameters InputParams Input Output Frames TranscriptionFrame InterimTranscriptionFrame Methods Usage Example Regional Support Available Regions Configuration Dynamic Region Updates Notes Models Language Support Special Features Frame Flow Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_google_42de0583.txt b/stt_google_42de0583.txt
new file mode 100644
index 0000000000000000000000000000000000000000..28fc7a293740144df9a23b992df29c745655b418
--- /dev/null
+++ b/stt_google_42de0583.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/google#special-features
+Title: Google - Pipecat
+==================================================
+
+Google - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text Google Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview GoogleSTTService provides real-time speech-to-text capabilities using Google Cloud’s Speech-to-Text V2 API. It supports interim results, multiple languages, and voice activity detection (VAD). ​ Installation To use GoogleSTTService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[google]" You’ll need Google Cloud credentials either as a JSON string or file. You can obtain Google Cloud credentials by creating a service account in the Google Cloud Console . ​ Configuration ​ Constructor Parameters ​ credentials str Google Cloud service account credentials as JSON string ​ credentials_path str Path to service account credentials JSON file ​ location str default: "global" Google Cloud location for the service ​ sample_rate int Audio sample rate in Hertz ​ params InputParams Configuration parameters for the service ​ InputParams The InputParams class provides configuration options for the Google STT service. ​ languages Language | List[Language] default: "Language.EN_US" Single language or list of recognition languages. First language is primary. Examples: Language.EN_US [Language.EN_US, Language.ES_US] The first language in the list is considered primary. Recognition accuracy may vary with multiple languages. When using multiple languages, list them in order of expected usage frequency for optimal recognition results. ​ model str default: "latest_long" Speech recognition model to use. ​ use_separate_recognition_per_channel bool default: "False" Process each audio channel separately for multi-channel audio. ​ enable_automatic_punctuation bool default: "True" Automatically add punctuation marks to transcriptions. ​ enable_spoken_punctuation bool default: "False" Include spoken punctuation (e.g., “period”, “comma”) in transcript. ​ enable_spoken_emojis bool default: "False" Include spoken emojis (e.g., “smiley face”) in transcript. ​ profanity_filter bool default: "False" Filter profanity from transcriptions. ​ enable_word_time_offsets bool default: "False" Include timing information for each word. ​ enable_word_confidence bool default: "False" Include confidence scores for each word. ​ enable_interim_results bool default: "True" Stream partial recognition results as they become available. ​ enable_voice_activity_events bool default: "False" Enable voice activity detection events. Not all features are available for all models or languages Some combinations of options may affect latency or accuracy Model selection should match your use case for best results ​ Input The service processes raw audio data with: Linear16 PCM encoding 16-bit depth Configurable sample rate Single channel (mono) ​ Output Frames The service produces two types of frames: ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Recognition language ​ InterimTranscriptionFrame Generated during ongoing speech, containing same fields as TranscriptionFrame but with preliminary results. ​ Methods ​ set_languages method Updates the service’s recognition language. Copy Ask AI async def set_languages ( language : List[Language]) -> None Example: Copy Ask AI await service.set_languages([Language. FR_FR ]) ​ set_model method Updates the service’s recognition model. Copy Ask AI async def set_model ( model : str ) -> None Example: Copy Ask AI await service.set_model( "medical_dictation" ) ​ update_options method Updates multiple service options dynamically. Copy Ask AI async def update_options ( * , languages : Optional[List[Language]] = None , model : Optional[ str ] = None , enable_automatic_punctuation : Optional[ bool ] = None , enable_spoken_punctuation : Optional[ bool ] = None , enable_spoken_emojis : Optional[ bool ] = None , profanity_filter : Optional[ bool ] = None , enable_word_time_offsets : Optional[ bool ] = None , enable_word_confidence : Optional[ bool ] = None , enable_interim_results : Optional[ bool ] = None , enable_voice_activity_events : Optional[ bool ] = None , location : Optional[ str ] = None , ) -> None Example: Copy Ask AI await service.update_options( languages = [Language. ES_ES , Language. EN_US ], enable_interim_results = True , profanity_filter = True ) See the STT base class methods for additional functionality. ​ Usage Example Copy Ask AI from pipecat.services.google.stt import GoogleSTTService from pipecat.transcriptions.language import Language # Configure service stt = GoogleSTTService( credentials_path = "path/to/credentials.json" , params = GoogleSTTService.InputParams( languages = Language. EN_US , model = "latest_long" , enable_automatic_punctuation = True , enable_interim_results = True ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, context_aggregator.user(), llm, ... ]) ​ Regional Support Google Cloud Speech-to-Text V2 supports different regional endpoints for improved latency and data residency requirements. ​ Available Regions See supported languages, models, and features for each region in Google’s Speech-to-Text documentation . ​ Configuration Specify the region during service initialization: Copy Ask AI stt = GoogleSTTService( credentials = credentials, location = "us-central1" , # Use us-central1 endpoint params = GoogleSTTService.InputParams( model = "chirp_2" ) ) ​ Dynamic Region Updates The region can be updated during runtime: Copy Ask AI await stt.update_options( location = "asia" ) ​ Notes The global endpoint is used by default Regional endpoints may provide lower latency for users in those regions Some features or models might only be available in specific regions Regional selection may affect pricing Data residency requirements may dictate region selection ​ Models Model Name Description Best For chirp_2 Google’s latest ASR model General use cases latest_long Latest model optimized for long-form speech Conversations, meetings latest_short Latest model optimized for short-form speech Short messages, notes telephony Optimized for phone calls Call centers medical_dictation Optimized for medical terminology Healthcare dictation medical_conversation Optimized for doctor-patient interactions Medical consultations See Google Cloud’s Speech-to-Text documentation for more details. ​ Language Support Language Code Description Service Codes Language.AF Afrikaans af-ZA Language.SQ Albanian sq-AL Language.AM Amharic am-ET Language.AR Arabic (Default: Egypt) ar-EG Language.AR_AE Arabic (UAE) ar-AE Language.AR_BH Arabic (Bahrain) ar-BH Language.AR_DZ Arabic (Algeria) ar-DZ Language.AR_EG Arabic (Egypt) ar-EG Language.AR_IQ Arabic (Iraq) ar-IQ Language.AR_JO Arabic (Jordan) ar-JO Language.AR_KW Arabic (Kuwait) ar-KW Language.AR_LB Arabic (Lebanon) ar-LB Language.AR_MA Arabic (Morocco) ar-MA Language.AR_OM Arabic (Oman) ar-OM Language.AR_QA Arabic (Qatar) ar-QA Language.AR_SA Arabic (Saudi Arabia) ar-SA Language.AR_SY Arabic (Syria) ar-SY Language.AR_TN Arabic (Tunisia) ar-TN Language.AR_YE Arabic (Yemen) ar-YE Language.HY Armenian hy-AM Language.AZ Azerbaijani az-AZ Language.EU Basque eu-ES Language.BN Bengali (Default: India) bn-IN Language.BN_BD Bengali (Bangladesh) bn-BD Language.BN_IN Bengali (India) bn-IN Language.BS Bosnian bs-BA Language.BG Bulgarian bg-BG Language.MY Burmese my-MM Language.CA Catalan ca-ES Language.ZH Chinese (Default: Simplified) cmn-Hans-CN Language.ZH_CN Chinese (Simplified) cmn-Hans-CN Language.ZH_HK Chinese (Hong Kong) cmn-Hans-HK Language.ZH_TW Chinese (Traditional) cmn-Hant-TW Language.YUE Chinese (Cantonese) yue-Hant-HK Language.HR Croatian hr-HR Language.CS Czech cs-CZ Language.DA Danish da-DK Language.NL Dutch (Default: Netherlands) nl-NL Language.NL_BE Dutch (Belgium) nl-BE Language.NL_NL Dutch (Netherlands) nl-NL Language.EN English (Default: US) en-US Language.EN_AU English (Australia) en-AU Language.EN_CA English (Canada) en-CA Language.EN_GB English (UK) en-GB Language.EN_GH English (Ghana) en-GH Language.EN_HK English (Hong Kong) en-HK Language.EN_IN English (India) en-IN Language.EN_IE English (Ireland) en-IE Language.EN_KE English (Kenya) en-KE Language.EN_NG English (Nigeria) en-NG Language.EN_NZ English (New Zealand) en-NZ Language.EN_PH English (Philippines) en-PH Language.EN_SG English (Singapore) en-SG Language.EN_TZ English (Tanzania) en-TZ Language.EN_US English (US) en-US Language.EN_ZA English (South Africa) en-ZA Language.ET Estonian et-EE Language.FIL Filipino fil-PH Language.FI Finnish fi-FI Language.FR French (Default: France) fr-FR Language.FR_BE French (Belgium) fr-BE Language.FR_CA French (Canada) fr-CA Language.FR_CH French (Switzerland) fr-CH Language.GL Galician gl-ES Language.KA Georgian ka-GE Language.DE German (Default: Germany) de-DE Language.DE_AT German (Austria) de-AT Language.DE_CH German (Switzerland) de-CH Language.EL Greek el-GR Language.GU Gujarati gu-IN Language.HE Hebrew iw-IL Language.HI Hindi hi-IN Language.HU Hungarian hu-HU Language.IS Icelandic is-IS Language.ID Indonesian id-ID Language.IT Italian it-IT Language.IT_CH Italian (Switzerland) it-CH Language.JA Japanese ja-JP Language.JV Javanese jv-ID Language.KN Kannada kn-IN Language.KK Kazakh kk-KZ Language.KM Khmer km-KH Language.KO Korean ko-KR Language.LO Lao lo-LA Language.LV Latvian lv-LV Language.LT Lithuanian lt-LT Language.MK Macedonian mk-MK Language.MS Malay ms-MY Language.ML Malayalam ml-IN Language.MR Marathi mr-IN Language.MN Mongolian mn-MN Language.NE Nepali ne-NP Language.NO Norwegian no-NO Language.FA Persian fa-IR Language.PL Polish pl-PL Language.PT Portuguese (Default: Portugal) pt-PT Language.PT_BR Portuguese (Brazil) pt-BR Language.PT_PT Portuguese (Portugal) pt-PT Language.PA Punjabi pa-Guru-IN Language.RO Romanian ro-RO Language.RU Russian ru-RU Language.SR Serbian sr-RS Language.SI Sinhala si-LK Language.SK Slovak sk-SK Language.SL Slovenian sl-SI Language.ES Spanish (Default: Spain) es-ES Language.ES_AR Spanish (Argentina) es-AR Language.ES_BO Spanish (Bolivia) es-BO Language.ES_CL Spanish (Chile) es-CL Language.ES_CO Spanish (Colombia) es-CO Language.ES_CR Spanish (Costa Rica) es-CR Language.ES_DO Spanish (Dominican Republic) es-DO Language.ES_EC Spanish (Ecuador) es-EC Language.ES_GT Spanish (Guatemala) es-GT Language.ES_HN Spanish (Honduras) es-HN Language.ES_MX Spanish (Mexico) es-MX Language.ES_NI Spanish (Nicaragua) es-NI Language.ES_PA Spanish (Panama) es-PA Language.ES_PE Spanish (Peru) es-PE Language.ES_PR Spanish (Puerto Rico) es-PR Language.ES_PY Spanish (Paraguay) es-PY Language.ES_SV Spanish (El Salvador) es-SV Language.ES_US Spanish (US) es-US Language.ES_UY Spanish (Uruguay) es-UY Language.ES_VE Spanish (Venezuela) es-VE Language.SU Sundanese su-ID Language.SW Swahili (Default: Tanzania) sw-TZ Language.SW_KE Swahili (Kenya) sw-KE Language.SW_TZ Swahili (Tanzania) sw-TZ Language.SV Swedish sv-SE Language.TA Tamil (Default: India) ta-IN Language.TA_IN Tamil (India) ta-IN Language.TA_MY Tamil (Malaysia) ta-MY Language.TA_SG Tamil (Singapore) ta-SG Language.TA_LK Tamil (Sri Lanka) ta-LK Language.TE Telugu te-IN Language.TH Thai th-TH Language.TR Turkish tr-TR Language.UK Ukrainian uk-UA Language.UR Urdu (Default: India) ur-IN Language.UR_IN Urdu (India) ur-IN Language.UR_PK Urdu (Pakistan) ur-PK Language.UZ Uzbek uz-UZ Language.VI Vietnamese vi-VN Language.XH Xhosa xh-ZA Language.ZU Zulu zu-ZA ​ Special Features Supports multiple languages simultaneously Provides regional variants for many languages Handles different Chinese scripts (simplified/traditional) Supports medical-specific models ​ Frame Flow ​ Notes Requires Google Cloud credentials Supports real-time transcription Handles streaming connection management Provides dynamic configuration updates Supports model switching Includes VAD capabilities Manages connection lifecycle Gladia Groq (Whisper) On this page Overview Installation Configuration Constructor Parameters InputParams Input Output Frames TranscriptionFrame InterimTranscriptionFrame Methods Usage Example Regional Support Available Regions Configuration Dynamic Region Updates Notes Models Language Support Special Features Frame Flow Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_google_63593eea.txt b/stt_google_63593eea.txt
new file mode 100644
index 0000000000000000000000000000000000000000..35e025d29e9d16f4abaf22981d4db80ba47da541
--- /dev/null
+++ b/stt_google_63593eea.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/google#param-text
+Title: Google - Pipecat
+==================================================
+
+Google - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text Google Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview GoogleSTTService provides real-time speech-to-text capabilities using Google Cloud’s Speech-to-Text V2 API. It supports interim results, multiple languages, and voice activity detection (VAD). ​ Installation To use GoogleSTTService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[google]" You’ll need Google Cloud credentials either as a JSON string or file. You can obtain Google Cloud credentials by creating a service account in the Google Cloud Console . ​ Configuration ​ Constructor Parameters ​ credentials str Google Cloud service account credentials as JSON string ​ credentials_path str Path to service account credentials JSON file ​ location str default: "global" Google Cloud location for the service ​ sample_rate int Audio sample rate in Hertz ​ params InputParams Configuration parameters for the service ​ InputParams The InputParams class provides configuration options for the Google STT service. ​ languages Language | List[Language] default: "Language.EN_US" Single language or list of recognition languages. First language is primary. Examples: Language.EN_US [Language.EN_US, Language.ES_US] The first language in the list is considered primary. Recognition accuracy may vary with multiple languages. When using multiple languages, list them in order of expected usage frequency for optimal recognition results. ​ model str default: "latest_long" Speech recognition model to use. ​ use_separate_recognition_per_channel bool default: "False" Process each audio channel separately for multi-channel audio. ​ enable_automatic_punctuation bool default: "True" Automatically add punctuation marks to transcriptions. ​ enable_spoken_punctuation bool default: "False" Include spoken punctuation (e.g., “period”, “comma”) in transcript. ​ enable_spoken_emojis bool default: "False" Include spoken emojis (e.g., “smiley face”) in transcript. ​ profanity_filter bool default: "False" Filter profanity from transcriptions. ​ enable_word_time_offsets bool default: "False" Include timing information for each word. ​ enable_word_confidence bool default: "False" Include confidence scores for each word. ​ enable_interim_results bool default: "True" Stream partial recognition results as they become available. ​ enable_voice_activity_events bool default: "False" Enable voice activity detection events. Not all features are available for all models or languages Some combinations of options may affect latency or accuracy Model selection should match your use case for best results ​ Input The service processes raw audio data with: Linear16 PCM encoding 16-bit depth Configurable sample rate Single channel (mono) ​ Output Frames The service produces two types of frames: ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Recognition language ​ InterimTranscriptionFrame Generated during ongoing speech, containing same fields as TranscriptionFrame but with preliminary results. ​ Methods ​ set_languages method Updates the service’s recognition language. Copy Ask AI async def set_languages ( language : List[Language]) -> None Example: Copy Ask AI await service.set_languages([Language. FR_FR ]) ​ set_model method Updates the service’s recognition model. Copy Ask AI async def set_model ( model : str ) -> None Example: Copy Ask AI await service.set_model( "medical_dictation" ) ​ update_options method Updates multiple service options dynamically. Copy Ask AI async def update_options ( * , languages : Optional[List[Language]] = None , model : Optional[ str ] = None , enable_automatic_punctuation : Optional[ bool ] = None , enable_spoken_punctuation : Optional[ bool ] = None , enable_spoken_emojis : Optional[ bool ] = None , profanity_filter : Optional[ bool ] = None , enable_word_time_offsets : Optional[ bool ] = None , enable_word_confidence : Optional[ bool ] = None , enable_interim_results : Optional[ bool ] = None , enable_voice_activity_events : Optional[ bool ] = None , location : Optional[ str ] = None , ) -> None Example: Copy Ask AI await service.update_options( languages = [Language. ES_ES , Language. EN_US ], enable_interim_results = True , profanity_filter = True ) See the STT base class methods for additional functionality. ​ Usage Example Copy Ask AI from pipecat.services.google.stt import GoogleSTTService from pipecat.transcriptions.language import Language # Configure service stt = GoogleSTTService( credentials_path = "path/to/credentials.json" , params = GoogleSTTService.InputParams( languages = Language. EN_US , model = "latest_long" , enable_automatic_punctuation = True , enable_interim_results = True ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, context_aggregator.user(), llm, ... ]) ​ Regional Support Google Cloud Speech-to-Text V2 supports different regional endpoints for improved latency and data residency requirements. ​ Available Regions See supported languages, models, and features for each region in Google’s Speech-to-Text documentation . ​ Configuration Specify the region during service initialization: Copy Ask AI stt = GoogleSTTService( credentials = credentials, location = "us-central1" , # Use us-central1 endpoint params = GoogleSTTService.InputParams( model = "chirp_2" ) ) ​ Dynamic Region Updates The region can be updated during runtime: Copy Ask AI await stt.update_options( location = "asia" ) ​ Notes The global endpoint is used by default Regional endpoints may provide lower latency for users in those regions Some features or models might only be available in specific regions Regional selection may affect pricing Data residency requirements may dictate region selection ​ Models Model Name Description Best For chirp_2 Google’s latest ASR model General use cases latest_long Latest model optimized for long-form speech Conversations, meetings latest_short Latest model optimized for short-form speech Short messages, notes telephony Optimized for phone calls Call centers medical_dictation Optimized for medical terminology Healthcare dictation medical_conversation Optimized for doctor-patient interactions Medical consultations See Google Cloud’s Speech-to-Text documentation for more details. ​ Language Support Language Code Description Service Codes Language.AF Afrikaans af-ZA Language.SQ Albanian sq-AL Language.AM Amharic am-ET Language.AR Arabic (Default: Egypt) ar-EG Language.AR_AE Arabic (UAE) ar-AE Language.AR_BH Arabic (Bahrain) ar-BH Language.AR_DZ Arabic (Algeria) ar-DZ Language.AR_EG Arabic (Egypt) ar-EG Language.AR_IQ Arabic (Iraq) ar-IQ Language.AR_JO Arabic (Jordan) ar-JO Language.AR_KW Arabic (Kuwait) ar-KW Language.AR_LB Arabic (Lebanon) ar-LB Language.AR_MA Arabic (Morocco) ar-MA Language.AR_OM Arabic (Oman) ar-OM Language.AR_QA Arabic (Qatar) ar-QA Language.AR_SA Arabic (Saudi Arabia) ar-SA Language.AR_SY Arabic (Syria) ar-SY Language.AR_TN Arabic (Tunisia) ar-TN Language.AR_YE Arabic (Yemen) ar-YE Language.HY Armenian hy-AM Language.AZ Azerbaijani az-AZ Language.EU Basque eu-ES Language.BN Bengali (Default: India) bn-IN Language.BN_BD Bengali (Bangladesh) bn-BD Language.BN_IN Bengali (India) bn-IN Language.BS Bosnian bs-BA Language.BG Bulgarian bg-BG Language.MY Burmese my-MM Language.CA Catalan ca-ES Language.ZH Chinese (Default: Simplified) cmn-Hans-CN Language.ZH_CN Chinese (Simplified) cmn-Hans-CN Language.ZH_HK Chinese (Hong Kong) cmn-Hans-HK Language.ZH_TW Chinese (Traditional) cmn-Hant-TW Language.YUE Chinese (Cantonese) yue-Hant-HK Language.HR Croatian hr-HR Language.CS Czech cs-CZ Language.DA Danish da-DK Language.NL Dutch (Default: Netherlands) nl-NL Language.NL_BE Dutch (Belgium) nl-BE Language.NL_NL Dutch (Netherlands) nl-NL Language.EN English (Default: US) en-US Language.EN_AU English (Australia) en-AU Language.EN_CA English (Canada) en-CA Language.EN_GB English (UK) en-GB Language.EN_GH English (Ghana) en-GH Language.EN_HK English (Hong Kong) en-HK Language.EN_IN English (India) en-IN Language.EN_IE English (Ireland) en-IE Language.EN_KE English (Kenya) en-KE Language.EN_NG English (Nigeria) en-NG Language.EN_NZ English (New Zealand) en-NZ Language.EN_PH English (Philippines) en-PH Language.EN_SG English (Singapore) en-SG Language.EN_TZ English (Tanzania) en-TZ Language.EN_US English (US) en-US Language.EN_ZA English (South Africa) en-ZA Language.ET Estonian et-EE Language.FIL Filipino fil-PH Language.FI Finnish fi-FI Language.FR French (Default: France) fr-FR Language.FR_BE French (Belgium) fr-BE Language.FR_CA French (Canada) fr-CA Language.FR_CH French (Switzerland) fr-CH Language.GL Galician gl-ES Language.KA Georgian ka-GE Language.DE German (Default: Germany) de-DE Language.DE_AT German (Austria) de-AT Language.DE_CH German (Switzerland) de-CH Language.EL Greek el-GR Language.GU Gujarati gu-IN Language.HE Hebrew iw-IL Language.HI Hindi hi-IN Language.HU Hungarian hu-HU Language.IS Icelandic is-IS Language.ID Indonesian id-ID Language.IT Italian it-IT Language.IT_CH Italian (Switzerland) it-CH Language.JA Japanese ja-JP Language.JV Javanese jv-ID Language.KN Kannada kn-IN Language.KK Kazakh kk-KZ Language.KM Khmer km-KH Language.KO Korean ko-KR Language.LO Lao lo-LA Language.LV Latvian lv-LV Language.LT Lithuanian lt-LT Language.MK Macedonian mk-MK Language.MS Malay ms-MY Language.ML Malayalam ml-IN Language.MR Marathi mr-IN Language.MN Mongolian mn-MN Language.NE Nepali ne-NP Language.NO Norwegian no-NO Language.FA Persian fa-IR Language.PL Polish pl-PL Language.PT Portuguese (Default: Portugal) pt-PT Language.PT_BR Portuguese (Brazil) pt-BR Language.PT_PT Portuguese (Portugal) pt-PT Language.PA Punjabi pa-Guru-IN Language.RO Romanian ro-RO Language.RU Russian ru-RU Language.SR Serbian sr-RS Language.SI Sinhala si-LK Language.SK Slovak sk-SK Language.SL Slovenian sl-SI Language.ES Spanish (Default: Spain) es-ES Language.ES_AR Spanish (Argentina) es-AR Language.ES_BO Spanish (Bolivia) es-BO Language.ES_CL Spanish (Chile) es-CL Language.ES_CO Spanish (Colombia) es-CO Language.ES_CR Spanish (Costa Rica) es-CR Language.ES_DO Spanish (Dominican Republic) es-DO Language.ES_EC Spanish (Ecuador) es-EC Language.ES_GT Spanish (Guatemala) es-GT Language.ES_HN Spanish (Honduras) es-HN Language.ES_MX Spanish (Mexico) es-MX Language.ES_NI Spanish (Nicaragua) es-NI Language.ES_PA Spanish (Panama) es-PA Language.ES_PE Spanish (Peru) es-PE Language.ES_PR Spanish (Puerto Rico) es-PR Language.ES_PY Spanish (Paraguay) es-PY Language.ES_SV Spanish (El Salvador) es-SV Language.ES_US Spanish (US) es-US Language.ES_UY Spanish (Uruguay) es-UY Language.ES_VE Spanish (Venezuela) es-VE Language.SU Sundanese su-ID Language.SW Swahili (Default: Tanzania) sw-TZ Language.SW_KE Swahili (Kenya) sw-KE Language.SW_TZ Swahili (Tanzania) sw-TZ Language.SV Swedish sv-SE Language.TA Tamil (Default: India) ta-IN Language.TA_IN Tamil (India) ta-IN Language.TA_MY Tamil (Malaysia) ta-MY Language.TA_SG Tamil (Singapore) ta-SG Language.TA_LK Tamil (Sri Lanka) ta-LK Language.TE Telugu te-IN Language.TH Thai th-TH Language.TR Turkish tr-TR Language.UK Ukrainian uk-UA Language.UR Urdu (Default: India) ur-IN Language.UR_IN Urdu (India) ur-IN Language.UR_PK Urdu (Pakistan) ur-PK Language.UZ Uzbek uz-UZ Language.VI Vietnamese vi-VN Language.XH Xhosa xh-ZA Language.ZU Zulu zu-ZA ​ Special Features Supports multiple languages simultaneously Provides regional variants for many languages Handles different Chinese scripts (simplified/traditional) Supports medical-specific models ​ Frame Flow ​ Notes Requires Google Cloud credentials Supports real-time transcription Handles streaming connection management Provides dynamic configuration updates Supports model switching Includes VAD capabilities Manages connection lifecycle Gladia Groq (Whisper) On this page Overview Installation Configuration Constructor Parameters InputParams Input Output Frames TranscriptionFrame InterimTranscriptionFrame Methods Usage Example Regional Support Available Regions Configuration Dynamic Region Updates Notes Models Language Support Special Features Frame Flow Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_google_6bd96ffa.txt b/stt_google_6bd96ffa.txt
new file mode 100644
index 0000000000000000000000000000000000000000..fe6415e3a3dbc468aa80e77e14485bc03eb8ad2f
--- /dev/null
+++ b/stt_google_6bd96ffa.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/google#input
+Title: Google - Pipecat
+==================================================
+
+Google - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text Google Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview GoogleSTTService provides real-time speech-to-text capabilities using Google Cloud’s Speech-to-Text V2 API. It supports interim results, multiple languages, and voice activity detection (VAD). ​ Installation To use GoogleSTTService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[google]" You’ll need Google Cloud credentials either as a JSON string or file. You can obtain Google Cloud credentials by creating a service account in the Google Cloud Console . ​ Configuration ​ Constructor Parameters ​ credentials str Google Cloud service account credentials as JSON string ​ credentials_path str Path to service account credentials JSON file ​ location str default: "global" Google Cloud location for the service ​ sample_rate int Audio sample rate in Hertz ​ params InputParams Configuration parameters for the service ​ InputParams The InputParams class provides configuration options for the Google STT service. ​ languages Language | List[Language] default: "Language.EN_US" Single language or list of recognition languages. First language is primary. Examples: Language.EN_US [Language.EN_US, Language.ES_US] The first language in the list is considered primary. Recognition accuracy may vary with multiple languages. When using multiple languages, list them in order of expected usage frequency for optimal recognition results. ​ model str default: "latest_long" Speech recognition model to use. ​ use_separate_recognition_per_channel bool default: "False" Process each audio channel separately for multi-channel audio. ​ enable_automatic_punctuation bool default: "True" Automatically add punctuation marks to transcriptions. ​ enable_spoken_punctuation bool default: "False" Include spoken punctuation (e.g., “period”, “comma”) in transcript. ​ enable_spoken_emojis bool default: "False" Include spoken emojis (e.g., “smiley face”) in transcript. ​ profanity_filter bool default: "False" Filter profanity from transcriptions. ​ enable_word_time_offsets bool default: "False" Include timing information for each word. ​ enable_word_confidence bool default: "False" Include confidence scores for each word. ​ enable_interim_results bool default: "True" Stream partial recognition results as they become available. ​ enable_voice_activity_events bool default: "False" Enable voice activity detection events. Not all features are available for all models or languages Some combinations of options may affect latency or accuracy Model selection should match your use case for best results ​ Input The service processes raw audio data with: Linear16 PCM encoding 16-bit depth Configurable sample rate Single channel (mono) ​ Output Frames The service produces two types of frames: ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Recognition language ​ InterimTranscriptionFrame Generated during ongoing speech, containing same fields as TranscriptionFrame but with preliminary results. ​ Methods ​ set_languages method Updates the service’s recognition language. Copy Ask AI async def set_languages ( language : List[Language]) -> None Example: Copy Ask AI await service.set_languages([Language. FR_FR ]) ​ set_model method Updates the service’s recognition model. Copy Ask AI async def set_model ( model : str ) -> None Example: Copy Ask AI await service.set_model( "medical_dictation" ) ​ update_options method Updates multiple service options dynamically. Copy Ask AI async def update_options ( * , languages : Optional[List[Language]] = None , model : Optional[ str ] = None , enable_automatic_punctuation : Optional[ bool ] = None , enable_spoken_punctuation : Optional[ bool ] = None , enable_spoken_emojis : Optional[ bool ] = None , profanity_filter : Optional[ bool ] = None , enable_word_time_offsets : Optional[ bool ] = None , enable_word_confidence : Optional[ bool ] = None , enable_interim_results : Optional[ bool ] = None , enable_voice_activity_events : Optional[ bool ] = None , location : Optional[ str ] = None , ) -> None Example: Copy Ask AI await service.update_options( languages = [Language. ES_ES , Language. EN_US ], enable_interim_results = True , profanity_filter = True ) See the STT base class methods for additional functionality. ​ Usage Example Copy Ask AI from pipecat.services.google.stt import GoogleSTTService from pipecat.transcriptions.language import Language # Configure service stt = GoogleSTTService( credentials_path = "path/to/credentials.json" , params = GoogleSTTService.InputParams( languages = Language. EN_US , model = "latest_long" , enable_automatic_punctuation = True , enable_interim_results = True ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, context_aggregator.user(), llm, ... ]) ​ Regional Support Google Cloud Speech-to-Text V2 supports different regional endpoints for improved latency and data residency requirements. ​ Available Regions See supported languages, models, and features for each region in Google’s Speech-to-Text documentation . ​ Configuration Specify the region during service initialization: Copy Ask AI stt = GoogleSTTService( credentials = credentials, location = "us-central1" , # Use us-central1 endpoint params = GoogleSTTService.InputParams( model = "chirp_2" ) ) ​ Dynamic Region Updates The region can be updated during runtime: Copy Ask AI await stt.update_options( location = "asia" ) ​ Notes The global endpoint is used by default Regional endpoints may provide lower latency for users in those regions Some features or models might only be available in specific regions Regional selection may affect pricing Data residency requirements may dictate region selection ​ Models Model Name Description Best For chirp_2 Google’s latest ASR model General use cases latest_long Latest model optimized for long-form speech Conversations, meetings latest_short Latest model optimized for short-form speech Short messages, notes telephony Optimized for phone calls Call centers medical_dictation Optimized for medical terminology Healthcare dictation medical_conversation Optimized for doctor-patient interactions Medical consultations See Google Cloud’s Speech-to-Text documentation for more details. ​ Language Support Language Code Description Service Codes Language.AF Afrikaans af-ZA Language.SQ Albanian sq-AL Language.AM Amharic am-ET Language.AR Arabic (Default: Egypt) ar-EG Language.AR_AE Arabic (UAE) ar-AE Language.AR_BH Arabic (Bahrain) ar-BH Language.AR_DZ Arabic (Algeria) ar-DZ Language.AR_EG Arabic (Egypt) ar-EG Language.AR_IQ Arabic (Iraq) ar-IQ Language.AR_JO Arabic (Jordan) ar-JO Language.AR_KW Arabic (Kuwait) ar-KW Language.AR_LB Arabic (Lebanon) ar-LB Language.AR_MA Arabic (Morocco) ar-MA Language.AR_OM Arabic (Oman) ar-OM Language.AR_QA Arabic (Qatar) ar-QA Language.AR_SA Arabic (Saudi Arabia) ar-SA Language.AR_SY Arabic (Syria) ar-SY Language.AR_TN Arabic (Tunisia) ar-TN Language.AR_YE Arabic (Yemen) ar-YE Language.HY Armenian hy-AM Language.AZ Azerbaijani az-AZ Language.EU Basque eu-ES Language.BN Bengali (Default: India) bn-IN Language.BN_BD Bengali (Bangladesh) bn-BD Language.BN_IN Bengali (India) bn-IN Language.BS Bosnian bs-BA Language.BG Bulgarian bg-BG Language.MY Burmese my-MM Language.CA Catalan ca-ES Language.ZH Chinese (Default: Simplified) cmn-Hans-CN Language.ZH_CN Chinese (Simplified) cmn-Hans-CN Language.ZH_HK Chinese (Hong Kong) cmn-Hans-HK Language.ZH_TW Chinese (Traditional) cmn-Hant-TW Language.YUE Chinese (Cantonese) yue-Hant-HK Language.HR Croatian hr-HR Language.CS Czech cs-CZ Language.DA Danish da-DK Language.NL Dutch (Default: Netherlands) nl-NL Language.NL_BE Dutch (Belgium) nl-BE Language.NL_NL Dutch (Netherlands) nl-NL Language.EN English (Default: US) en-US Language.EN_AU English (Australia) en-AU Language.EN_CA English (Canada) en-CA Language.EN_GB English (UK) en-GB Language.EN_GH English (Ghana) en-GH Language.EN_HK English (Hong Kong) en-HK Language.EN_IN English (India) en-IN Language.EN_IE English (Ireland) en-IE Language.EN_KE English (Kenya) en-KE Language.EN_NG English (Nigeria) en-NG Language.EN_NZ English (New Zealand) en-NZ Language.EN_PH English (Philippines) en-PH Language.EN_SG English (Singapore) en-SG Language.EN_TZ English (Tanzania) en-TZ Language.EN_US English (US) en-US Language.EN_ZA English (South Africa) en-ZA Language.ET Estonian et-EE Language.FIL Filipino fil-PH Language.FI Finnish fi-FI Language.FR French (Default: France) fr-FR Language.FR_BE French (Belgium) fr-BE Language.FR_CA French (Canada) fr-CA Language.FR_CH French (Switzerland) fr-CH Language.GL Galician gl-ES Language.KA Georgian ka-GE Language.DE German (Default: Germany) de-DE Language.DE_AT German (Austria) de-AT Language.DE_CH German (Switzerland) de-CH Language.EL Greek el-GR Language.GU Gujarati gu-IN Language.HE Hebrew iw-IL Language.HI Hindi hi-IN Language.HU Hungarian hu-HU Language.IS Icelandic is-IS Language.ID Indonesian id-ID Language.IT Italian it-IT Language.IT_CH Italian (Switzerland) it-CH Language.JA Japanese ja-JP Language.JV Javanese jv-ID Language.KN Kannada kn-IN Language.KK Kazakh kk-KZ Language.KM Khmer km-KH Language.KO Korean ko-KR Language.LO Lao lo-LA Language.LV Latvian lv-LV Language.LT Lithuanian lt-LT Language.MK Macedonian mk-MK Language.MS Malay ms-MY Language.ML Malayalam ml-IN Language.MR Marathi mr-IN Language.MN Mongolian mn-MN Language.NE Nepali ne-NP Language.NO Norwegian no-NO Language.FA Persian fa-IR Language.PL Polish pl-PL Language.PT Portuguese (Default: Portugal) pt-PT Language.PT_BR Portuguese (Brazil) pt-BR Language.PT_PT Portuguese (Portugal) pt-PT Language.PA Punjabi pa-Guru-IN Language.RO Romanian ro-RO Language.RU Russian ru-RU Language.SR Serbian sr-RS Language.SI Sinhala si-LK Language.SK Slovak sk-SK Language.SL Slovenian sl-SI Language.ES Spanish (Default: Spain) es-ES Language.ES_AR Spanish (Argentina) es-AR Language.ES_BO Spanish (Bolivia) es-BO Language.ES_CL Spanish (Chile) es-CL Language.ES_CO Spanish (Colombia) es-CO Language.ES_CR Spanish (Costa Rica) es-CR Language.ES_DO Spanish (Dominican Republic) es-DO Language.ES_EC Spanish (Ecuador) es-EC Language.ES_GT Spanish (Guatemala) es-GT Language.ES_HN Spanish (Honduras) es-HN Language.ES_MX Spanish (Mexico) es-MX Language.ES_NI Spanish (Nicaragua) es-NI Language.ES_PA Spanish (Panama) es-PA Language.ES_PE Spanish (Peru) es-PE Language.ES_PR Spanish (Puerto Rico) es-PR Language.ES_PY Spanish (Paraguay) es-PY Language.ES_SV Spanish (El Salvador) es-SV Language.ES_US Spanish (US) es-US Language.ES_UY Spanish (Uruguay) es-UY Language.ES_VE Spanish (Venezuela) es-VE Language.SU Sundanese su-ID Language.SW Swahili (Default: Tanzania) sw-TZ Language.SW_KE Swahili (Kenya) sw-KE Language.SW_TZ Swahili (Tanzania) sw-TZ Language.SV Swedish sv-SE Language.TA Tamil (Default: India) ta-IN Language.TA_IN Tamil (India) ta-IN Language.TA_MY Tamil (Malaysia) ta-MY Language.TA_SG Tamil (Singapore) ta-SG Language.TA_LK Tamil (Sri Lanka) ta-LK Language.TE Telugu te-IN Language.TH Thai th-TH Language.TR Turkish tr-TR Language.UK Ukrainian uk-UA Language.UR Urdu (Default: India) ur-IN Language.UR_IN Urdu (India) ur-IN Language.UR_PK Urdu (Pakistan) ur-PK Language.UZ Uzbek uz-UZ Language.VI Vietnamese vi-VN Language.XH Xhosa xh-ZA Language.ZU Zulu zu-ZA ​ Special Features Supports multiple languages simultaneously Provides regional variants for many languages Handles different Chinese scripts (simplified/traditional) Supports medical-specific models ​ Frame Flow ​ Notes Requires Google Cloud credentials Supports real-time transcription Handles streaming connection management Provides dynamic configuration updates Supports model switching Includes VAD capabilities Manages connection lifecycle Gladia Groq (Whisper) On this page Overview Installation Configuration Constructor Parameters InputParams Input Output Frames TranscriptionFrame InterimTranscriptionFrame Methods Usage Example Regional Support Available Regions Configuration Dynamic Region Updates Notes Models Language Support Special Features Frame Flow Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_google_7af26a4f.txt b/stt_google_7af26a4f.txt
new file mode 100644
index 0000000000000000000000000000000000000000..48a876f77942b115060324d79933757c3f307227
--- /dev/null
+++ b/stt_google_7af26a4f.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/google#methods
+Title: Google - Pipecat
+==================================================
+
+Google - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text Google Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview GoogleSTTService provides real-time speech-to-text capabilities using Google Cloud’s Speech-to-Text V2 API. It supports interim results, multiple languages, and voice activity detection (VAD). ​ Installation To use GoogleSTTService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[google]" You’ll need Google Cloud credentials either as a JSON string or file. You can obtain Google Cloud credentials by creating a service account in the Google Cloud Console . ​ Configuration ​ Constructor Parameters ​ credentials str Google Cloud service account credentials as JSON string ​ credentials_path str Path to service account credentials JSON file ​ location str default: "global" Google Cloud location for the service ​ sample_rate int Audio sample rate in Hertz ​ params InputParams Configuration parameters for the service ​ InputParams The InputParams class provides configuration options for the Google STT service. ​ languages Language | List[Language] default: "Language.EN_US" Single language or list of recognition languages. First language is primary. Examples: Language.EN_US [Language.EN_US, Language.ES_US] The first language in the list is considered primary. Recognition accuracy may vary with multiple languages. When using multiple languages, list them in order of expected usage frequency for optimal recognition results. ​ model str default: "latest_long" Speech recognition model to use. ​ use_separate_recognition_per_channel bool default: "False" Process each audio channel separately for multi-channel audio. ​ enable_automatic_punctuation bool default: "True" Automatically add punctuation marks to transcriptions. ​ enable_spoken_punctuation bool default: "False" Include spoken punctuation (e.g., “period”, “comma”) in transcript. ​ enable_spoken_emojis bool default: "False" Include spoken emojis (e.g., “smiley face”) in transcript. ​ profanity_filter bool default: "False" Filter profanity from transcriptions. ​ enable_word_time_offsets bool default: "False" Include timing information for each word. ​ enable_word_confidence bool default: "False" Include confidence scores for each word. ​ enable_interim_results bool default: "True" Stream partial recognition results as they become available. ​ enable_voice_activity_events bool default: "False" Enable voice activity detection events. Not all features are available for all models or languages Some combinations of options may affect latency or accuracy Model selection should match your use case for best results ​ Input The service processes raw audio data with: Linear16 PCM encoding 16-bit depth Configurable sample rate Single channel (mono) ​ Output Frames The service produces two types of frames: ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Recognition language ​ InterimTranscriptionFrame Generated during ongoing speech, containing same fields as TranscriptionFrame but with preliminary results. ​ Methods ​ set_languages method Updates the service’s recognition language. Copy Ask AI async def set_languages ( language : List[Language]) -> None Example: Copy Ask AI await service.set_languages([Language. FR_FR ]) ​ set_model method Updates the service’s recognition model. Copy Ask AI async def set_model ( model : str ) -> None Example: Copy Ask AI await service.set_model( "medical_dictation" ) ​ update_options method Updates multiple service options dynamically. Copy Ask AI async def update_options ( * , languages : Optional[List[Language]] = None , model : Optional[ str ] = None , enable_automatic_punctuation : Optional[ bool ] = None , enable_spoken_punctuation : Optional[ bool ] = None , enable_spoken_emojis : Optional[ bool ] = None , profanity_filter : Optional[ bool ] = None , enable_word_time_offsets : Optional[ bool ] = None , enable_word_confidence : Optional[ bool ] = None , enable_interim_results : Optional[ bool ] = None , enable_voice_activity_events : Optional[ bool ] = None , location : Optional[ str ] = None , ) -> None Example: Copy Ask AI await service.update_options( languages = [Language. ES_ES , Language. EN_US ], enable_interim_results = True , profanity_filter = True ) See the STT base class methods for additional functionality. ​ Usage Example Copy Ask AI from pipecat.services.google.stt import GoogleSTTService from pipecat.transcriptions.language import Language # Configure service stt = GoogleSTTService( credentials_path = "path/to/credentials.json" , params = GoogleSTTService.InputParams( languages = Language. EN_US , model = "latest_long" , enable_automatic_punctuation = True , enable_interim_results = True ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, context_aggregator.user(), llm, ... ]) ​ Regional Support Google Cloud Speech-to-Text V2 supports different regional endpoints for improved latency and data residency requirements. ​ Available Regions See supported languages, models, and features for each region in Google’s Speech-to-Text documentation . ​ Configuration Specify the region during service initialization: Copy Ask AI stt = GoogleSTTService( credentials = credentials, location = "us-central1" , # Use us-central1 endpoint params = GoogleSTTService.InputParams( model = "chirp_2" ) ) ​ Dynamic Region Updates The region can be updated during runtime: Copy Ask AI await stt.update_options( location = "asia" ) ​ Notes The global endpoint is used by default Regional endpoints may provide lower latency for users in those regions Some features or models might only be available in specific regions Regional selection may affect pricing Data residency requirements may dictate region selection ​ Models Model Name Description Best For chirp_2 Google’s latest ASR model General use cases latest_long Latest model optimized for long-form speech Conversations, meetings latest_short Latest model optimized for short-form speech Short messages, notes telephony Optimized for phone calls Call centers medical_dictation Optimized for medical terminology Healthcare dictation medical_conversation Optimized for doctor-patient interactions Medical consultations See Google Cloud’s Speech-to-Text documentation for more details. ​ Language Support Language Code Description Service Codes Language.AF Afrikaans af-ZA Language.SQ Albanian sq-AL Language.AM Amharic am-ET Language.AR Arabic (Default: Egypt) ar-EG Language.AR_AE Arabic (UAE) ar-AE Language.AR_BH Arabic (Bahrain) ar-BH Language.AR_DZ Arabic (Algeria) ar-DZ Language.AR_EG Arabic (Egypt) ar-EG Language.AR_IQ Arabic (Iraq) ar-IQ Language.AR_JO Arabic (Jordan) ar-JO Language.AR_KW Arabic (Kuwait) ar-KW Language.AR_LB Arabic (Lebanon) ar-LB Language.AR_MA Arabic (Morocco) ar-MA Language.AR_OM Arabic (Oman) ar-OM Language.AR_QA Arabic (Qatar) ar-QA Language.AR_SA Arabic (Saudi Arabia) ar-SA Language.AR_SY Arabic (Syria) ar-SY Language.AR_TN Arabic (Tunisia) ar-TN Language.AR_YE Arabic (Yemen) ar-YE Language.HY Armenian hy-AM Language.AZ Azerbaijani az-AZ Language.EU Basque eu-ES Language.BN Bengali (Default: India) bn-IN Language.BN_BD Bengali (Bangladesh) bn-BD Language.BN_IN Bengali (India) bn-IN Language.BS Bosnian bs-BA Language.BG Bulgarian bg-BG Language.MY Burmese my-MM Language.CA Catalan ca-ES Language.ZH Chinese (Default: Simplified) cmn-Hans-CN Language.ZH_CN Chinese (Simplified) cmn-Hans-CN Language.ZH_HK Chinese (Hong Kong) cmn-Hans-HK Language.ZH_TW Chinese (Traditional) cmn-Hant-TW Language.YUE Chinese (Cantonese) yue-Hant-HK Language.HR Croatian hr-HR Language.CS Czech cs-CZ Language.DA Danish da-DK Language.NL Dutch (Default: Netherlands) nl-NL Language.NL_BE Dutch (Belgium) nl-BE Language.NL_NL Dutch (Netherlands) nl-NL Language.EN English (Default: US) en-US Language.EN_AU English (Australia) en-AU Language.EN_CA English (Canada) en-CA Language.EN_GB English (UK) en-GB Language.EN_GH English (Ghana) en-GH Language.EN_HK English (Hong Kong) en-HK Language.EN_IN English (India) en-IN Language.EN_IE English (Ireland) en-IE Language.EN_KE English (Kenya) en-KE Language.EN_NG English (Nigeria) en-NG Language.EN_NZ English (New Zealand) en-NZ Language.EN_PH English (Philippines) en-PH Language.EN_SG English (Singapore) en-SG Language.EN_TZ English (Tanzania) en-TZ Language.EN_US English (US) en-US Language.EN_ZA English (South Africa) en-ZA Language.ET Estonian et-EE Language.FIL Filipino fil-PH Language.FI Finnish fi-FI Language.FR French (Default: France) fr-FR Language.FR_BE French (Belgium) fr-BE Language.FR_CA French (Canada) fr-CA Language.FR_CH French (Switzerland) fr-CH Language.GL Galician gl-ES Language.KA Georgian ka-GE Language.DE German (Default: Germany) de-DE Language.DE_AT German (Austria) de-AT Language.DE_CH German (Switzerland) de-CH Language.EL Greek el-GR Language.GU Gujarati gu-IN Language.HE Hebrew iw-IL Language.HI Hindi hi-IN Language.HU Hungarian hu-HU Language.IS Icelandic is-IS Language.ID Indonesian id-ID Language.IT Italian it-IT Language.IT_CH Italian (Switzerland) it-CH Language.JA Japanese ja-JP Language.JV Javanese jv-ID Language.KN Kannada kn-IN Language.KK Kazakh kk-KZ Language.KM Khmer km-KH Language.KO Korean ko-KR Language.LO Lao lo-LA Language.LV Latvian lv-LV Language.LT Lithuanian lt-LT Language.MK Macedonian mk-MK Language.MS Malay ms-MY Language.ML Malayalam ml-IN Language.MR Marathi mr-IN Language.MN Mongolian mn-MN Language.NE Nepali ne-NP Language.NO Norwegian no-NO Language.FA Persian fa-IR Language.PL Polish pl-PL Language.PT Portuguese (Default: Portugal) pt-PT Language.PT_BR Portuguese (Brazil) pt-BR Language.PT_PT Portuguese (Portugal) pt-PT Language.PA Punjabi pa-Guru-IN Language.RO Romanian ro-RO Language.RU Russian ru-RU Language.SR Serbian sr-RS Language.SI Sinhala si-LK Language.SK Slovak sk-SK Language.SL Slovenian sl-SI Language.ES Spanish (Default: Spain) es-ES Language.ES_AR Spanish (Argentina) es-AR Language.ES_BO Spanish (Bolivia) es-BO Language.ES_CL Spanish (Chile) es-CL Language.ES_CO Spanish (Colombia) es-CO Language.ES_CR Spanish (Costa Rica) es-CR Language.ES_DO Spanish (Dominican Republic) es-DO Language.ES_EC Spanish (Ecuador) es-EC Language.ES_GT Spanish (Guatemala) es-GT Language.ES_HN Spanish (Honduras) es-HN Language.ES_MX Spanish (Mexico) es-MX Language.ES_NI Spanish (Nicaragua) es-NI Language.ES_PA Spanish (Panama) es-PA Language.ES_PE Spanish (Peru) es-PE Language.ES_PR Spanish (Puerto Rico) es-PR Language.ES_PY Spanish (Paraguay) es-PY Language.ES_SV Spanish (El Salvador) es-SV Language.ES_US Spanish (US) es-US Language.ES_UY Spanish (Uruguay) es-UY Language.ES_VE Spanish (Venezuela) es-VE Language.SU Sundanese su-ID Language.SW Swahili (Default: Tanzania) sw-TZ Language.SW_KE Swahili (Kenya) sw-KE Language.SW_TZ Swahili (Tanzania) sw-TZ Language.SV Swedish sv-SE Language.TA Tamil (Default: India) ta-IN Language.TA_IN Tamil (India) ta-IN Language.TA_MY Tamil (Malaysia) ta-MY Language.TA_SG Tamil (Singapore) ta-SG Language.TA_LK Tamil (Sri Lanka) ta-LK Language.TE Telugu te-IN Language.TH Thai th-TH Language.TR Turkish tr-TR Language.UK Ukrainian uk-UA Language.UR Urdu (Default: India) ur-IN Language.UR_IN Urdu (India) ur-IN Language.UR_PK Urdu (Pakistan) ur-PK Language.UZ Uzbek uz-UZ Language.VI Vietnamese vi-VN Language.XH Xhosa xh-ZA Language.ZU Zulu zu-ZA ​ Special Features Supports multiple languages simultaneously Provides regional variants for many languages Handles different Chinese scripts (simplified/traditional) Supports medical-specific models ​ Frame Flow ​ Notes Requires Google Cloud credentials Supports real-time transcription Handles streaming connection management Provides dynamic configuration updates Supports model switching Includes VAD capabilities Manages connection lifecycle Gladia Groq (Whisper) On this page Overview Installation Configuration Constructor Parameters InputParams Input Output Frames TranscriptionFrame InterimTranscriptionFrame Methods Usage Example Regional Support Available Regions Configuration Dynamic Region Updates Notes Models Language Support Special Features Frame Flow Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_google_a857326b.txt b/stt_google_a857326b.txt
new file mode 100644
index 0000000000000000000000000000000000000000..cf0e7cdcf55abab07039772fe1807f4ee0bfe833
--- /dev/null
+++ b/stt_google_a857326b.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/google#usage-example
+Title: Google - Pipecat
+==================================================
+
+Google - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text Google Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview GoogleSTTService provides real-time speech-to-text capabilities using Google Cloud’s Speech-to-Text V2 API. It supports interim results, multiple languages, and voice activity detection (VAD). ​ Installation To use GoogleSTTService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[google]" You’ll need Google Cloud credentials either as a JSON string or file. You can obtain Google Cloud credentials by creating a service account in the Google Cloud Console . ​ Configuration ​ Constructor Parameters ​ credentials str Google Cloud service account credentials as JSON string ​ credentials_path str Path to service account credentials JSON file ​ location str default: "global" Google Cloud location for the service ​ sample_rate int Audio sample rate in Hertz ​ params InputParams Configuration parameters for the service ​ InputParams The InputParams class provides configuration options for the Google STT service. ​ languages Language | List[Language] default: "Language.EN_US" Single language or list of recognition languages. First language is primary. Examples: Language.EN_US [Language.EN_US, Language.ES_US] The first language in the list is considered primary. Recognition accuracy may vary with multiple languages. When using multiple languages, list them in order of expected usage frequency for optimal recognition results. ​ model str default: "latest_long" Speech recognition model to use. ​ use_separate_recognition_per_channel bool default: "False" Process each audio channel separately for multi-channel audio. ​ enable_automatic_punctuation bool default: "True" Automatically add punctuation marks to transcriptions. ​ enable_spoken_punctuation bool default: "False" Include spoken punctuation (e.g., “period”, “comma”) in transcript. ​ enable_spoken_emojis bool default: "False" Include spoken emojis (e.g., “smiley face”) in transcript. ​ profanity_filter bool default: "False" Filter profanity from transcriptions. ​ enable_word_time_offsets bool default: "False" Include timing information for each word. ​ enable_word_confidence bool default: "False" Include confidence scores for each word. ​ enable_interim_results bool default: "True" Stream partial recognition results as they become available. ​ enable_voice_activity_events bool default: "False" Enable voice activity detection events. Not all features are available for all models or languages Some combinations of options may affect latency or accuracy Model selection should match your use case for best results ​ Input The service processes raw audio data with: Linear16 PCM encoding 16-bit depth Configurable sample rate Single channel (mono) ​ Output Frames The service produces two types of frames: ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Recognition language ​ InterimTranscriptionFrame Generated during ongoing speech, containing same fields as TranscriptionFrame but with preliminary results. ​ Methods ​ set_languages method Updates the service’s recognition language. Copy Ask AI async def set_languages ( language : List[Language]) -> None Example: Copy Ask AI await service.set_languages([Language. FR_FR ]) ​ set_model method Updates the service’s recognition model. Copy Ask AI async def set_model ( model : str ) -> None Example: Copy Ask AI await service.set_model( "medical_dictation" ) ​ update_options method Updates multiple service options dynamically. Copy Ask AI async def update_options ( * , languages : Optional[List[Language]] = None , model : Optional[ str ] = None , enable_automatic_punctuation : Optional[ bool ] = None , enable_spoken_punctuation : Optional[ bool ] = None , enable_spoken_emojis : Optional[ bool ] = None , profanity_filter : Optional[ bool ] = None , enable_word_time_offsets : Optional[ bool ] = None , enable_word_confidence : Optional[ bool ] = None , enable_interim_results : Optional[ bool ] = None , enable_voice_activity_events : Optional[ bool ] = None , location : Optional[ str ] = None , ) -> None Example: Copy Ask AI await service.update_options( languages = [Language. ES_ES , Language. EN_US ], enable_interim_results = True , profanity_filter = True ) See the STT base class methods for additional functionality. ​ Usage Example Copy Ask AI from pipecat.services.google.stt import GoogleSTTService from pipecat.transcriptions.language import Language # Configure service stt = GoogleSTTService( credentials_path = "path/to/credentials.json" , params = GoogleSTTService.InputParams( languages = Language. EN_US , model = "latest_long" , enable_automatic_punctuation = True , enable_interim_results = True ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, context_aggregator.user(), llm, ... ]) ​ Regional Support Google Cloud Speech-to-Text V2 supports different regional endpoints for improved latency and data residency requirements. ​ Available Regions See supported languages, models, and features for each region in Google’s Speech-to-Text documentation . ​ Configuration Specify the region during service initialization: Copy Ask AI stt = GoogleSTTService( credentials = credentials, location = "us-central1" , # Use us-central1 endpoint params = GoogleSTTService.InputParams( model = "chirp_2" ) ) ​ Dynamic Region Updates The region can be updated during runtime: Copy Ask AI await stt.update_options( location = "asia" ) ​ Notes The global endpoint is used by default Regional endpoints may provide lower latency for users in those regions Some features or models might only be available in specific regions Regional selection may affect pricing Data residency requirements may dictate region selection ​ Models Model Name Description Best For chirp_2 Google’s latest ASR model General use cases latest_long Latest model optimized for long-form speech Conversations, meetings latest_short Latest model optimized for short-form speech Short messages, notes telephony Optimized for phone calls Call centers medical_dictation Optimized for medical terminology Healthcare dictation medical_conversation Optimized for doctor-patient interactions Medical consultations See Google Cloud’s Speech-to-Text documentation for more details. ​ Language Support Language Code Description Service Codes Language.AF Afrikaans af-ZA Language.SQ Albanian sq-AL Language.AM Amharic am-ET Language.AR Arabic (Default: Egypt) ar-EG Language.AR_AE Arabic (UAE) ar-AE Language.AR_BH Arabic (Bahrain) ar-BH Language.AR_DZ Arabic (Algeria) ar-DZ Language.AR_EG Arabic (Egypt) ar-EG Language.AR_IQ Arabic (Iraq) ar-IQ Language.AR_JO Arabic (Jordan) ar-JO Language.AR_KW Arabic (Kuwait) ar-KW Language.AR_LB Arabic (Lebanon) ar-LB Language.AR_MA Arabic (Morocco) ar-MA Language.AR_OM Arabic (Oman) ar-OM Language.AR_QA Arabic (Qatar) ar-QA Language.AR_SA Arabic (Saudi Arabia) ar-SA Language.AR_SY Arabic (Syria) ar-SY Language.AR_TN Arabic (Tunisia) ar-TN Language.AR_YE Arabic (Yemen) ar-YE Language.HY Armenian hy-AM Language.AZ Azerbaijani az-AZ Language.EU Basque eu-ES Language.BN Bengali (Default: India) bn-IN Language.BN_BD Bengali (Bangladesh) bn-BD Language.BN_IN Bengali (India) bn-IN Language.BS Bosnian bs-BA Language.BG Bulgarian bg-BG Language.MY Burmese my-MM Language.CA Catalan ca-ES Language.ZH Chinese (Default: Simplified) cmn-Hans-CN Language.ZH_CN Chinese (Simplified) cmn-Hans-CN Language.ZH_HK Chinese (Hong Kong) cmn-Hans-HK Language.ZH_TW Chinese (Traditional) cmn-Hant-TW Language.YUE Chinese (Cantonese) yue-Hant-HK Language.HR Croatian hr-HR Language.CS Czech cs-CZ Language.DA Danish da-DK Language.NL Dutch (Default: Netherlands) nl-NL Language.NL_BE Dutch (Belgium) nl-BE Language.NL_NL Dutch (Netherlands) nl-NL Language.EN English (Default: US) en-US Language.EN_AU English (Australia) en-AU Language.EN_CA English (Canada) en-CA Language.EN_GB English (UK) en-GB Language.EN_GH English (Ghana) en-GH Language.EN_HK English (Hong Kong) en-HK Language.EN_IN English (India) en-IN Language.EN_IE English (Ireland) en-IE Language.EN_KE English (Kenya) en-KE Language.EN_NG English (Nigeria) en-NG Language.EN_NZ English (New Zealand) en-NZ Language.EN_PH English (Philippines) en-PH Language.EN_SG English (Singapore) en-SG Language.EN_TZ English (Tanzania) en-TZ Language.EN_US English (US) en-US Language.EN_ZA English (South Africa) en-ZA Language.ET Estonian et-EE Language.FIL Filipino fil-PH Language.FI Finnish fi-FI Language.FR French (Default: France) fr-FR Language.FR_BE French (Belgium) fr-BE Language.FR_CA French (Canada) fr-CA Language.FR_CH French (Switzerland) fr-CH Language.GL Galician gl-ES Language.KA Georgian ka-GE Language.DE German (Default: Germany) de-DE Language.DE_AT German (Austria) de-AT Language.DE_CH German (Switzerland) de-CH Language.EL Greek el-GR Language.GU Gujarati gu-IN Language.HE Hebrew iw-IL Language.HI Hindi hi-IN Language.HU Hungarian hu-HU Language.IS Icelandic is-IS Language.ID Indonesian id-ID Language.IT Italian it-IT Language.IT_CH Italian (Switzerland) it-CH Language.JA Japanese ja-JP Language.JV Javanese jv-ID Language.KN Kannada kn-IN Language.KK Kazakh kk-KZ Language.KM Khmer km-KH Language.KO Korean ko-KR Language.LO Lao lo-LA Language.LV Latvian lv-LV Language.LT Lithuanian lt-LT Language.MK Macedonian mk-MK Language.MS Malay ms-MY Language.ML Malayalam ml-IN Language.MR Marathi mr-IN Language.MN Mongolian mn-MN Language.NE Nepali ne-NP Language.NO Norwegian no-NO Language.FA Persian fa-IR Language.PL Polish pl-PL Language.PT Portuguese (Default: Portugal) pt-PT Language.PT_BR Portuguese (Brazil) pt-BR Language.PT_PT Portuguese (Portugal) pt-PT Language.PA Punjabi pa-Guru-IN Language.RO Romanian ro-RO Language.RU Russian ru-RU Language.SR Serbian sr-RS Language.SI Sinhala si-LK Language.SK Slovak sk-SK Language.SL Slovenian sl-SI Language.ES Spanish (Default: Spain) es-ES Language.ES_AR Spanish (Argentina) es-AR Language.ES_BO Spanish (Bolivia) es-BO Language.ES_CL Spanish (Chile) es-CL Language.ES_CO Spanish (Colombia) es-CO Language.ES_CR Spanish (Costa Rica) es-CR Language.ES_DO Spanish (Dominican Republic) es-DO Language.ES_EC Spanish (Ecuador) es-EC Language.ES_GT Spanish (Guatemala) es-GT Language.ES_HN Spanish (Honduras) es-HN Language.ES_MX Spanish (Mexico) es-MX Language.ES_NI Spanish (Nicaragua) es-NI Language.ES_PA Spanish (Panama) es-PA Language.ES_PE Spanish (Peru) es-PE Language.ES_PR Spanish (Puerto Rico) es-PR Language.ES_PY Spanish (Paraguay) es-PY Language.ES_SV Spanish (El Salvador) es-SV Language.ES_US Spanish (US) es-US Language.ES_UY Spanish (Uruguay) es-UY Language.ES_VE Spanish (Venezuela) es-VE Language.SU Sundanese su-ID Language.SW Swahili (Default: Tanzania) sw-TZ Language.SW_KE Swahili (Kenya) sw-KE Language.SW_TZ Swahili (Tanzania) sw-TZ Language.SV Swedish sv-SE Language.TA Tamil (Default: India) ta-IN Language.TA_IN Tamil (India) ta-IN Language.TA_MY Tamil (Malaysia) ta-MY Language.TA_SG Tamil (Singapore) ta-SG Language.TA_LK Tamil (Sri Lanka) ta-LK Language.TE Telugu te-IN Language.TH Thai th-TH Language.TR Turkish tr-TR Language.UK Ukrainian uk-UA Language.UR Urdu (Default: India) ur-IN Language.UR_IN Urdu (India) ur-IN Language.UR_PK Urdu (Pakistan) ur-PK Language.UZ Uzbek uz-UZ Language.VI Vietnamese vi-VN Language.XH Xhosa xh-ZA Language.ZU Zulu zu-ZA ​ Special Features Supports multiple languages simultaneously Provides regional variants for many languages Handles different Chinese scripts (simplified/traditional) Supports medical-specific models ​ Frame Flow ​ Notes Requires Google Cloud credentials Supports real-time transcription Handles streaming connection management Provides dynamic configuration updates Supports model switching Includes VAD capabilities Manages connection lifecycle Gladia Groq (Whisper) On this page Overview Installation Configuration Constructor Parameters InputParams Input Output Frames TranscriptionFrame InterimTranscriptionFrame Methods Usage Example Regional Support Available Regions Configuration Dynamic Region Updates Notes Models Language Support Special Features Frame Flow Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_google_e339f2ab.txt b/stt_google_e339f2ab.txt
new file mode 100644
index 0000000000000000000000000000000000000000..f1028e16f6fe3bc35e663f12ab30ca574617ed35
--- /dev/null
+++ b/stt_google_e339f2ab.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/google#notes-2
+Title: Google - Pipecat
+==================================================
+
+Google - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text Google Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview GoogleSTTService provides real-time speech-to-text capabilities using Google Cloud’s Speech-to-Text V2 API. It supports interim results, multiple languages, and voice activity detection (VAD). ​ Installation To use GoogleSTTService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[google]" You’ll need Google Cloud credentials either as a JSON string or file. You can obtain Google Cloud credentials by creating a service account in the Google Cloud Console . ​ Configuration ​ Constructor Parameters ​ credentials str Google Cloud service account credentials as JSON string ​ credentials_path str Path to service account credentials JSON file ​ location str default: "global" Google Cloud location for the service ​ sample_rate int Audio sample rate in Hertz ​ params InputParams Configuration parameters for the service ​ InputParams The InputParams class provides configuration options for the Google STT service. ​ languages Language | List[Language] default: "Language.EN_US" Single language or list of recognition languages. First language is primary. Examples: Language.EN_US [Language.EN_US, Language.ES_US] The first language in the list is considered primary. Recognition accuracy may vary with multiple languages. When using multiple languages, list them in order of expected usage frequency for optimal recognition results. ​ model str default: "latest_long" Speech recognition model to use. ​ use_separate_recognition_per_channel bool default: "False" Process each audio channel separately for multi-channel audio. ​ enable_automatic_punctuation bool default: "True" Automatically add punctuation marks to transcriptions. ​ enable_spoken_punctuation bool default: "False" Include spoken punctuation (e.g., “period”, “comma”) in transcript. ​ enable_spoken_emojis bool default: "False" Include spoken emojis (e.g., “smiley face”) in transcript. ​ profanity_filter bool default: "False" Filter profanity from transcriptions. ​ enable_word_time_offsets bool default: "False" Include timing information for each word. ​ enable_word_confidence bool default: "False" Include confidence scores for each word. ​ enable_interim_results bool default: "True" Stream partial recognition results as they become available. ​ enable_voice_activity_events bool default: "False" Enable voice activity detection events. Not all features are available for all models or languages Some combinations of options may affect latency or accuracy Model selection should match your use case for best results ​ Input The service processes raw audio data with: Linear16 PCM encoding 16-bit depth Configurable sample rate Single channel (mono) ​ Output Frames The service produces two types of frames: ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Recognition language ​ InterimTranscriptionFrame Generated during ongoing speech, containing same fields as TranscriptionFrame but with preliminary results. ​ Methods ​ set_languages method Updates the service’s recognition language. Copy Ask AI async def set_languages ( language : List[Language]) -> None Example: Copy Ask AI await service.set_languages([Language. FR_FR ]) ​ set_model method Updates the service’s recognition model. Copy Ask AI async def set_model ( model : str ) -> None Example: Copy Ask AI await service.set_model( "medical_dictation" ) ​ update_options method Updates multiple service options dynamically. Copy Ask AI async def update_options ( * , languages : Optional[List[Language]] = None , model : Optional[ str ] = None , enable_automatic_punctuation : Optional[ bool ] = None , enable_spoken_punctuation : Optional[ bool ] = None , enable_spoken_emojis : Optional[ bool ] = None , profanity_filter : Optional[ bool ] = None , enable_word_time_offsets : Optional[ bool ] = None , enable_word_confidence : Optional[ bool ] = None , enable_interim_results : Optional[ bool ] = None , enable_voice_activity_events : Optional[ bool ] = None , location : Optional[ str ] = None , ) -> None Example: Copy Ask AI await service.update_options( languages = [Language. ES_ES , Language. EN_US ], enable_interim_results = True , profanity_filter = True ) See the STT base class methods for additional functionality. ​ Usage Example Copy Ask AI from pipecat.services.google.stt import GoogleSTTService from pipecat.transcriptions.language import Language # Configure service stt = GoogleSTTService( credentials_path = "path/to/credentials.json" , params = GoogleSTTService.InputParams( languages = Language. EN_US , model = "latest_long" , enable_automatic_punctuation = True , enable_interim_results = True ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, context_aggregator.user(), llm, ... ]) ​ Regional Support Google Cloud Speech-to-Text V2 supports different regional endpoints for improved latency and data residency requirements. ​ Available Regions See supported languages, models, and features for each region in Google’s Speech-to-Text documentation . ​ Configuration Specify the region during service initialization: Copy Ask AI stt = GoogleSTTService( credentials = credentials, location = "us-central1" , # Use us-central1 endpoint params = GoogleSTTService.InputParams( model = "chirp_2" ) ) ​ Dynamic Region Updates The region can be updated during runtime: Copy Ask AI await stt.update_options( location = "asia" ) ​ Notes The global endpoint is used by default Regional endpoints may provide lower latency for users in those regions Some features or models might only be available in specific regions Regional selection may affect pricing Data residency requirements may dictate region selection ​ Models Model Name Description Best For chirp_2 Google’s latest ASR model General use cases latest_long Latest model optimized for long-form speech Conversations, meetings latest_short Latest model optimized for short-form speech Short messages, notes telephony Optimized for phone calls Call centers medical_dictation Optimized for medical terminology Healthcare dictation medical_conversation Optimized for doctor-patient interactions Medical consultations See Google Cloud’s Speech-to-Text documentation for more details. ​ Language Support Language Code Description Service Codes Language.AF Afrikaans af-ZA Language.SQ Albanian sq-AL Language.AM Amharic am-ET Language.AR Arabic (Default: Egypt) ar-EG Language.AR_AE Arabic (UAE) ar-AE Language.AR_BH Arabic (Bahrain) ar-BH Language.AR_DZ Arabic (Algeria) ar-DZ Language.AR_EG Arabic (Egypt) ar-EG Language.AR_IQ Arabic (Iraq) ar-IQ Language.AR_JO Arabic (Jordan) ar-JO Language.AR_KW Arabic (Kuwait) ar-KW Language.AR_LB Arabic (Lebanon) ar-LB Language.AR_MA Arabic (Morocco) ar-MA Language.AR_OM Arabic (Oman) ar-OM Language.AR_QA Arabic (Qatar) ar-QA Language.AR_SA Arabic (Saudi Arabia) ar-SA Language.AR_SY Arabic (Syria) ar-SY Language.AR_TN Arabic (Tunisia) ar-TN Language.AR_YE Arabic (Yemen) ar-YE Language.HY Armenian hy-AM Language.AZ Azerbaijani az-AZ Language.EU Basque eu-ES Language.BN Bengali (Default: India) bn-IN Language.BN_BD Bengali (Bangladesh) bn-BD Language.BN_IN Bengali (India) bn-IN Language.BS Bosnian bs-BA Language.BG Bulgarian bg-BG Language.MY Burmese my-MM Language.CA Catalan ca-ES Language.ZH Chinese (Default: Simplified) cmn-Hans-CN Language.ZH_CN Chinese (Simplified) cmn-Hans-CN Language.ZH_HK Chinese (Hong Kong) cmn-Hans-HK Language.ZH_TW Chinese (Traditional) cmn-Hant-TW Language.YUE Chinese (Cantonese) yue-Hant-HK Language.HR Croatian hr-HR Language.CS Czech cs-CZ Language.DA Danish da-DK Language.NL Dutch (Default: Netherlands) nl-NL Language.NL_BE Dutch (Belgium) nl-BE Language.NL_NL Dutch (Netherlands) nl-NL Language.EN English (Default: US) en-US Language.EN_AU English (Australia) en-AU Language.EN_CA English (Canada) en-CA Language.EN_GB English (UK) en-GB Language.EN_GH English (Ghana) en-GH Language.EN_HK English (Hong Kong) en-HK Language.EN_IN English (India) en-IN Language.EN_IE English (Ireland) en-IE Language.EN_KE English (Kenya) en-KE Language.EN_NG English (Nigeria) en-NG Language.EN_NZ English (New Zealand) en-NZ Language.EN_PH English (Philippines) en-PH Language.EN_SG English (Singapore) en-SG Language.EN_TZ English (Tanzania) en-TZ Language.EN_US English (US) en-US Language.EN_ZA English (South Africa) en-ZA Language.ET Estonian et-EE Language.FIL Filipino fil-PH Language.FI Finnish fi-FI Language.FR French (Default: France) fr-FR Language.FR_BE French (Belgium) fr-BE Language.FR_CA French (Canada) fr-CA Language.FR_CH French (Switzerland) fr-CH Language.GL Galician gl-ES Language.KA Georgian ka-GE Language.DE German (Default: Germany) de-DE Language.DE_AT German (Austria) de-AT Language.DE_CH German (Switzerland) de-CH Language.EL Greek el-GR Language.GU Gujarati gu-IN Language.HE Hebrew iw-IL Language.HI Hindi hi-IN Language.HU Hungarian hu-HU Language.IS Icelandic is-IS Language.ID Indonesian id-ID Language.IT Italian it-IT Language.IT_CH Italian (Switzerland) it-CH Language.JA Japanese ja-JP Language.JV Javanese jv-ID Language.KN Kannada kn-IN Language.KK Kazakh kk-KZ Language.KM Khmer km-KH Language.KO Korean ko-KR Language.LO Lao lo-LA Language.LV Latvian lv-LV Language.LT Lithuanian lt-LT Language.MK Macedonian mk-MK Language.MS Malay ms-MY Language.ML Malayalam ml-IN Language.MR Marathi mr-IN Language.MN Mongolian mn-MN Language.NE Nepali ne-NP Language.NO Norwegian no-NO Language.FA Persian fa-IR Language.PL Polish pl-PL Language.PT Portuguese (Default: Portugal) pt-PT Language.PT_BR Portuguese (Brazil) pt-BR Language.PT_PT Portuguese (Portugal) pt-PT Language.PA Punjabi pa-Guru-IN Language.RO Romanian ro-RO Language.RU Russian ru-RU Language.SR Serbian sr-RS Language.SI Sinhala si-LK Language.SK Slovak sk-SK Language.SL Slovenian sl-SI Language.ES Spanish (Default: Spain) es-ES Language.ES_AR Spanish (Argentina) es-AR Language.ES_BO Spanish (Bolivia) es-BO Language.ES_CL Spanish (Chile) es-CL Language.ES_CO Spanish (Colombia) es-CO Language.ES_CR Spanish (Costa Rica) es-CR Language.ES_DO Spanish (Dominican Republic) es-DO Language.ES_EC Spanish (Ecuador) es-EC Language.ES_GT Spanish (Guatemala) es-GT Language.ES_HN Spanish (Honduras) es-HN Language.ES_MX Spanish (Mexico) es-MX Language.ES_NI Spanish (Nicaragua) es-NI Language.ES_PA Spanish (Panama) es-PA Language.ES_PE Spanish (Peru) es-PE Language.ES_PR Spanish (Puerto Rico) es-PR Language.ES_PY Spanish (Paraguay) es-PY Language.ES_SV Spanish (El Salvador) es-SV Language.ES_US Spanish (US) es-US Language.ES_UY Spanish (Uruguay) es-UY Language.ES_VE Spanish (Venezuela) es-VE Language.SU Sundanese su-ID Language.SW Swahili (Default: Tanzania) sw-TZ Language.SW_KE Swahili (Kenya) sw-KE Language.SW_TZ Swahili (Tanzania) sw-TZ Language.SV Swedish sv-SE Language.TA Tamil (Default: India) ta-IN Language.TA_IN Tamil (India) ta-IN Language.TA_MY Tamil (Malaysia) ta-MY Language.TA_SG Tamil (Singapore) ta-SG Language.TA_LK Tamil (Sri Lanka) ta-LK Language.TE Telugu te-IN Language.TH Thai th-TH Language.TR Turkish tr-TR Language.UK Ukrainian uk-UA Language.UR Urdu (Default: India) ur-IN Language.UR_IN Urdu (India) ur-IN Language.UR_PK Urdu (Pakistan) ur-PK Language.UZ Uzbek uz-UZ Language.VI Vietnamese vi-VN Language.XH Xhosa xh-ZA Language.ZU Zulu zu-ZA ​ Special Features Supports multiple languages simultaneously Provides regional variants for many languages Handles different Chinese scripts (simplified/traditional) Supports medical-specific models ​ Frame Flow ​ Notes Requires Google Cloud credentials Supports real-time transcription Handles streaming connection management Provides dynamic configuration updates Supports model switching Includes VAD capabilities Manages connection lifecycle Gladia Groq (Whisper) On this page Overview Installation Configuration Constructor Parameters InputParams Input Output Frames TranscriptionFrame InterimTranscriptionFrame Methods Usage Example Regional Support Available Regions Configuration Dynamic Region Updates Notes Models Language Support Special Features Frame Flow Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_google_e80a995f.txt b/stt_google_e80a995f.txt
new file mode 100644
index 0000000000000000000000000000000000000000..05c6f807b8f94ca20577c922d67724ae740c0830
--- /dev/null
+++ b/stt_google_e80a995f.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/google#models
+Title: Google - Pipecat
+==================================================
+
+Google - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text Google Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview GoogleSTTService provides real-time speech-to-text capabilities using Google Cloud’s Speech-to-Text V2 API. It supports interim results, multiple languages, and voice activity detection (VAD). ​ Installation To use GoogleSTTService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[google]" You’ll need Google Cloud credentials either as a JSON string or file. You can obtain Google Cloud credentials by creating a service account in the Google Cloud Console . ​ Configuration ​ Constructor Parameters ​ credentials str Google Cloud service account credentials as JSON string ​ credentials_path str Path to service account credentials JSON file ​ location str default: "global" Google Cloud location for the service ​ sample_rate int Audio sample rate in Hertz ​ params InputParams Configuration parameters for the service ​ InputParams The InputParams class provides configuration options for the Google STT service. ​ languages Language | List[Language] default: "Language.EN_US" Single language or list of recognition languages. First language is primary. Examples: Language.EN_US [Language.EN_US, Language.ES_US] The first language in the list is considered primary. Recognition accuracy may vary with multiple languages. When using multiple languages, list them in order of expected usage frequency for optimal recognition results. ​ model str default: "latest_long" Speech recognition model to use. ​ use_separate_recognition_per_channel bool default: "False" Process each audio channel separately for multi-channel audio. ​ enable_automatic_punctuation bool default: "True" Automatically add punctuation marks to transcriptions. ​ enable_spoken_punctuation bool default: "False" Include spoken punctuation (e.g., “period”, “comma”) in transcript. ​ enable_spoken_emojis bool default: "False" Include spoken emojis (e.g., “smiley face”) in transcript. ​ profanity_filter bool default: "False" Filter profanity from transcriptions. ​ enable_word_time_offsets bool default: "False" Include timing information for each word. ​ enable_word_confidence bool default: "False" Include confidence scores for each word. ​ enable_interim_results bool default: "True" Stream partial recognition results as they become available. ​ enable_voice_activity_events bool default: "False" Enable voice activity detection events. Not all features are available for all models or languages Some combinations of options may affect latency or accuracy Model selection should match your use case for best results ​ Input The service processes raw audio data with: Linear16 PCM encoding 16-bit depth Configurable sample rate Single channel (mono) ​ Output Frames The service produces two types of frames: ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Recognition language ​ InterimTranscriptionFrame Generated during ongoing speech, containing same fields as TranscriptionFrame but with preliminary results. ​ Methods ​ set_languages method Updates the service’s recognition language. Copy Ask AI async def set_languages ( language : List[Language]) -> None Example: Copy Ask AI await service.set_languages([Language. FR_FR ]) ​ set_model method Updates the service’s recognition model. Copy Ask AI async def set_model ( model : str ) -> None Example: Copy Ask AI await service.set_model( "medical_dictation" ) ​ update_options method Updates multiple service options dynamically. Copy Ask AI async def update_options ( * , languages : Optional[List[Language]] = None , model : Optional[ str ] = None , enable_automatic_punctuation : Optional[ bool ] = None , enable_spoken_punctuation : Optional[ bool ] = None , enable_spoken_emojis : Optional[ bool ] = None , profanity_filter : Optional[ bool ] = None , enable_word_time_offsets : Optional[ bool ] = None , enable_word_confidence : Optional[ bool ] = None , enable_interim_results : Optional[ bool ] = None , enable_voice_activity_events : Optional[ bool ] = None , location : Optional[ str ] = None , ) -> None Example: Copy Ask AI await service.update_options( languages = [Language. ES_ES , Language. EN_US ], enable_interim_results = True , profanity_filter = True ) See the STT base class methods for additional functionality. ​ Usage Example Copy Ask AI from pipecat.services.google.stt import GoogleSTTService from pipecat.transcriptions.language import Language # Configure service stt = GoogleSTTService( credentials_path = "path/to/credentials.json" , params = GoogleSTTService.InputParams( languages = Language. EN_US , model = "latest_long" , enable_automatic_punctuation = True , enable_interim_results = True ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, context_aggregator.user(), llm, ... ]) ​ Regional Support Google Cloud Speech-to-Text V2 supports different regional endpoints for improved latency and data residency requirements. ​ Available Regions See supported languages, models, and features for each region in Google’s Speech-to-Text documentation . ​ Configuration Specify the region during service initialization: Copy Ask AI stt = GoogleSTTService( credentials = credentials, location = "us-central1" , # Use us-central1 endpoint params = GoogleSTTService.InputParams( model = "chirp_2" ) ) ​ Dynamic Region Updates The region can be updated during runtime: Copy Ask AI await stt.update_options( location = "asia" ) ​ Notes The global endpoint is used by default Regional endpoints may provide lower latency for users in those regions Some features or models might only be available in specific regions Regional selection may affect pricing Data residency requirements may dictate region selection ​ Models Model Name Description Best For chirp_2 Google’s latest ASR model General use cases latest_long Latest model optimized for long-form speech Conversations, meetings latest_short Latest model optimized for short-form speech Short messages, notes telephony Optimized for phone calls Call centers medical_dictation Optimized for medical terminology Healthcare dictation medical_conversation Optimized for doctor-patient interactions Medical consultations See Google Cloud’s Speech-to-Text documentation for more details. ​ Language Support Language Code Description Service Codes Language.AF Afrikaans af-ZA Language.SQ Albanian sq-AL Language.AM Amharic am-ET Language.AR Arabic (Default: Egypt) ar-EG Language.AR_AE Arabic (UAE) ar-AE Language.AR_BH Arabic (Bahrain) ar-BH Language.AR_DZ Arabic (Algeria) ar-DZ Language.AR_EG Arabic (Egypt) ar-EG Language.AR_IQ Arabic (Iraq) ar-IQ Language.AR_JO Arabic (Jordan) ar-JO Language.AR_KW Arabic (Kuwait) ar-KW Language.AR_LB Arabic (Lebanon) ar-LB Language.AR_MA Arabic (Morocco) ar-MA Language.AR_OM Arabic (Oman) ar-OM Language.AR_QA Arabic (Qatar) ar-QA Language.AR_SA Arabic (Saudi Arabia) ar-SA Language.AR_SY Arabic (Syria) ar-SY Language.AR_TN Arabic (Tunisia) ar-TN Language.AR_YE Arabic (Yemen) ar-YE Language.HY Armenian hy-AM Language.AZ Azerbaijani az-AZ Language.EU Basque eu-ES Language.BN Bengali (Default: India) bn-IN Language.BN_BD Bengali (Bangladesh) bn-BD Language.BN_IN Bengali (India) bn-IN Language.BS Bosnian bs-BA Language.BG Bulgarian bg-BG Language.MY Burmese my-MM Language.CA Catalan ca-ES Language.ZH Chinese (Default: Simplified) cmn-Hans-CN Language.ZH_CN Chinese (Simplified) cmn-Hans-CN Language.ZH_HK Chinese (Hong Kong) cmn-Hans-HK Language.ZH_TW Chinese (Traditional) cmn-Hant-TW Language.YUE Chinese (Cantonese) yue-Hant-HK Language.HR Croatian hr-HR Language.CS Czech cs-CZ Language.DA Danish da-DK Language.NL Dutch (Default: Netherlands) nl-NL Language.NL_BE Dutch (Belgium) nl-BE Language.NL_NL Dutch (Netherlands) nl-NL Language.EN English (Default: US) en-US Language.EN_AU English (Australia) en-AU Language.EN_CA English (Canada) en-CA Language.EN_GB English (UK) en-GB Language.EN_GH English (Ghana) en-GH Language.EN_HK English (Hong Kong) en-HK Language.EN_IN English (India) en-IN Language.EN_IE English (Ireland) en-IE Language.EN_KE English (Kenya) en-KE Language.EN_NG English (Nigeria) en-NG Language.EN_NZ English (New Zealand) en-NZ Language.EN_PH English (Philippines) en-PH Language.EN_SG English (Singapore) en-SG Language.EN_TZ English (Tanzania) en-TZ Language.EN_US English (US) en-US Language.EN_ZA English (South Africa) en-ZA Language.ET Estonian et-EE Language.FIL Filipino fil-PH Language.FI Finnish fi-FI Language.FR French (Default: France) fr-FR Language.FR_BE French (Belgium) fr-BE Language.FR_CA French (Canada) fr-CA Language.FR_CH French (Switzerland) fr-CH Language.GL Galician gl-ES Language.KA Georgian ka-GE Language.DE German (Default: Germany) de-DE Language.DE_AT German (Austria) de-AT Language.DE_CH German (Switzerland) de-CH Language.EL Greek el-GR Language.GU Gujarati gu-IN Language.HE Hebrew iw-IL Language.HI Hindi hi-IN Language.HU Hungarian hu-HU Language.IS Icelandic is-IS Language.ID Indonesian id-ID Language.IT Italian it-IT Language.IT_CH Italian (Switzerland) it-CH Language.JA Japanese ja-JP Language.JV Javanese jv-ID Language.KN Kannada kn-IN Language.KK Kazakh kk-KZ Language.KM Khmer km-KH Language.KO Korean ko-KR Language.LO Lao lo-LA Language.LV Latvian lv-LV Language.LT Lithuanian lt-LT Language.MK Macedonian mk-MK Language.MS Malay ms-MY Language.ML Malayalam ml-IN Language.MR Marathi mr-IN Language.MN Mongolian mn-MN Language.NE Nepali ne-NP Language.NO Norwegian no-NO Language.FA Persian fa-IR Language.PL Polish pl-PL Language.PT Portuguese (Default: Portugal) pt-PT Language.PT_BR Portuguese (Brazil) pt-BR Language.PT_PT Portuguese (Portugal) pt-PT Language.PA Punjabi pa-Guru-IN Language.RO Romanian ro-RO Language.RU Russian ru-RU Language.SR Serbian sr-RS Language.SI Sinhala si-LK Language.SK Slovak sk-SK Language.SL Slovenian sl-SI Language.ES Spanish (Default: Spain) es-ES Language.ES_AR Spanish (Argentina) es-AR Language.ES_BO Spanish (Bolivia) es-BO Language.ES_CL Spanish (Chile) es-CL Language.ES_CO Spanish (Colombia) es-CO Language.ES_CR Spanish (Costa Rica) es-CR Language.ES_DO Spanish (Dominican Republic) es-DO Language.ES_EC Spanish (Ecuador) es-EC Language.ES_GT Spanish (Guatemala) es-GT Language.ES_HN Spanish (Honduras) es-HN Language.ES_MX Spanish (Mexico) es-MX Language.ES_NI Spanish (Nicaragua) es-NI Language.ES_PA Spanish (Panama) es-PA Language.ES_PE Spanish (Peru) es-PE Language.ES_PR Spanish (Puerto Rico) es-PR Language.ES_PY Spanish (Paraguay) es-PY Language.ES_SV Spanish (El Salvador) es-SV Language.ES_US Spanish (US) es-US Language.ES_UY Spanish (Uruguay) es-UY Language.ES_VE Spanish (Venezuela) es-VE Language.SU Sundanese su-ID Language.SW Swahili (Default: Tanzania) sw-TZ Language.SW_KE Swahili (Kenya) sw-KE Language.SW_TZ Swahili (Tanzania) sw-TZ Language.SV Swedish sv-SE Language.TA Tamil (Default: India) ta-IN Language.TA_IN Tamil (India) ta-IN Language.TA_MY Tamil (Malaysia) ta-MY Language.TA_SG Tamil (Singapore) ta-SG Language.TA_LK Tamil (Sri Lanka) ta-LK Language.TE Telugu te-IN Language.TH Thai th-TH Language.TR Turkish tr-TR Language.UK Ukrainian uk-UA Language.UR Urdu (Default: India) ur-IN Language.UR_IN Urdu (India) ur-IN Language.UR_PK Urdu (Pakistan) ur-PK Language.UZ Uzbek uz-UZ Language.VI Vietnamese vi-VN Language.XH Xhosa xh-ZA Language.ZU Zulu zu-ZA ​ Special Features Supports multiple languages simultaneously Provides regional variants for many languages Handles different Chinese scripts (simplified/traditional) Supports medical-specific models ​ Frame Flow ​ Notes Requires Google Cloud credentials Supports real-time transcription Handles streaming connection management Provides dynamic configuration updates Supports model switching Includes VAD capabilities Manages connection lifecycle Gladia Groq (Whisper) On this page Overview Installation Configuration Constructor Parameters InputParams Input Output Frames TranscriptionFrame InterimTranscriptionFrame Methods Usage Example Regional Support Available Regions Configuration Dynamic Region Updates Notes Models Language Support Special Features Frame Flow Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_google_f162313d.txt b/stt_google_f162313d.txt
new file mode 100644
index 0000000000000000000000000000000000000000..a69608dd235a32770c5a866ba6cad06349c59312
--- /dev/null
+++ b/stt_google_f162313d.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/google#transcriptionframe
+Title: Google - Pipecat
+==================================================
+
+Google - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text Google Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview GoogleSTTService provides real-time speech-to-text capabilities using Google Cloud’s Speech-to-Text V2 API. It supports interim results, multiple languages, and voice activity detection (VAD). ​ Installation To use GoogleSTTService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[google]" You’ll need Google Cloud credentials either as a JSON string or file. You can obtain Google Cloud credentials by creating a service account in the Google Cloud Console . ​ Configuration ​ Constructor Parameters ​ credentials str Google Cloud service account credentials as JSON string ​ credentials_path str Path to service account credentials JSON file ​ location str default: "global" Google Cloud location for the service ​ sample_rate int Audio sample rate in Hertz ​ params InputParams Configuration parameters for the service ​ InputParams The InputParams class provides configuration options for the Google STT service. ​ languages Language | List[Language] default: "Language.EN_US" Single language or list of recognition languages. First language is primary. Examples: Language.EN_US [Language.EN_US, Language.ES_US] The first language in the list is considered primary. Recognition accuracy may vary with multiple languages. When using multiple languages, list them in order of expected usage frequency for optimal recognition results. ​ model str default: "latest_long" Speech recognition model to use. ​ use_separate_recognition_per_channel bool default: "False" Process each audio channel separately for multi-channel audio. ​ enable_automatic_punctuation bool default: "True" Automatically add punctuation marks to transcriptions. ​ enable_spoken_punctuation bool default: "False" Include spoken punctuation (e.g., “period”, “comma”) in transcript. ​ enable_spoken_emojis bool default: "False" Include spoken emojis (e.g., “smiley face”) in transcript. ​ profanity_filter bool default: "False" Filter profanity from transcriptions. ​ enable_word_time_offsets bool default: "False" Include timing information for each word. ​ enable_word_confidence bool default: "False" Include confidence scores for each word. ​ enable_interim_results bool default: "True" Stream partial recognition results as they become available. ​ enable_voice_activity_events bool default: "False" Enable voice activity detection events. Not all features are available for all models or languages Some combinations of options may affect latency or accuracy Model selection should match your use case for best results ​ Input The service processes raw audio data with: Linear16 PCM encoding 16-bit depth Configurable sample rate Single channel (mono) ​ Output Frames The service produces two types of frames: ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Recognition language ​ InterimTranscriptionFrame Generated during ongoing speech, containing same fields as TranscriptionFrame but with preliminary results. ​ Methods ​ set_languages method Updates the service’s recognition language. Copy Ask AI async def set_languages ( language : List[Language]) -> None Example: Copy Ask AI await service.set_languages([Language. FR_FR ]) ​ set_model method Updates the service’s recognition model. Copy Ask AI async def set_model ( model : str ) -> None Example: Copy Ask AI await service.set_model( "medical_dictation" ) ​ update_options method Updates multiple service options dynamically. Copy Ask AI async def update_options ( * , languages : Optional[List[Language]] = None , model : Optional[ str ] = None , enable_automatic_punctuation : Optional[ bool ] = None , enable_spoken_punctuation : Optional[ bool ] = None , enable_spoken_emojis : Optional[ bool ] = None , profanity_filter : Optional[ bool ] = None , enable_word_time_offsets : Optional[ bool ] = None , enable_word_confidence : Optional[ bool ] = None , enable_interim_results : Optional[ bool ] = None , enable_voice_activity_events : Optional[ bool ] = None , location : Optional[ str ] = None , ) -> None Example: Copy Ask AI await service.update_options( languages = [Language. ES_ES , Language. EN_US ], enable_interim_results = True , profanity_filter = True ) See the STT base class methods for additional functionality. ​ Usage Example Copy Ask AI from pipecat.services.google.stt import GoogleSTTService from pipecat.transcriptions.language import Language # Configure service stt = GoogleSTTService( credentials_path = "path/to/credentials.json" , params = GoogleSTTService.InputParams( languages = Language. EN_US , model = "latest_long" , enable_automatic_punctuation = True , enable_interim_results = True ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, context_aggregator.user(), llm, ... ]) ​ Regional Support Google Cloud Speech-to-Text V2 supports different regional endpoints for improved latency and data residency requirements. ​ Available Regions See supported languages, models, and features for each region in Google’s Speech-to-Text documentation . ​ Configuration Specify the region during service initialization: Copy Ask AI stt = GoogleSTTService( credentials = credentials, location = "us-central1" , # Use us-central1 endpoint params = GoogleSTTService.InputParams( model = "chirp_2" ) ) ​ Dynamic Region Updates The region can be updated during runtime: Copy Ask AI await stt.update_options( location = "asia" ) ​ Notes The global endpoint is used by default Regional endpoints may provide lower latency for users in those regions Some features or models might only be available in specific regions Regional selection may affect pricing Data residency requirements may dictate region selection ​ Models Model Name Description Best For chirp_2 Google’s latest ASR model General use cases latest_long Latest model optimized for long-form speech Conversations, meetings latest_short Latest model optimized for short-form speech Short messages, notes telephony Optimized for phone calls Call centers medical_dictation Optimized for medical terminology Healthcare dictation medical_conversation Optimized for doctor-patient interactions Medical consultations See Google Cloud’s Speech-to-Text documentation for more details. ​ Language Support Language Code Description Service Codes Language.AF Afrikaans af-ZA Language.SQ Albanian sq-AL Language.AM Amharic am-ET Language.AR Arabic (Default: Egypt) ar-EG Language.AR_AE Arabic (UAE) ar-AE Language.AR_BH Arabic (Bahrain) ar-BH Language.AR_DZ Arabic (Algeria) ar-DZ Language.AR_EG Arabic (Egypt) ar-EG Language.AR_IQ Arabic (Iraq) ar-IQ Language.AR_JO Arabic (Jordan) ar-JO Language.AR_KW Arabic (Kuwait) ar-KW Language.AR_LB Arabic (Lebanon) ar-LB Language.AR_MA Arabic (Morocco) ar-MA Language.AR_OM Arabic (Oman) ar-OM Language.AR_QA Arabic (Qatar) ar-QA Language.AR_SA Arabic (Saudi Arabia) ar-SA Language.AR_SY Arabic (Syria) ar-SY Language.AR_TN Arabic (Tunisia) ar-TN Language.AR_YE Arabic (Yemen) ar-YE Language.HY Armenian hy-AM Language.AZ Azerbaijani az-AZ Language.EU Basque eu-ES Language.BN Bengali (Default: India) bn-IN Language.BN_BD Bengali (Bangladesh) bn-BD Language.BN_IN Bengali (India) bn-IN Language.BS Bosnian bs-BA Language.BG Bulgarian bg-BG Language.MY Burmese my-MM Language.CA Catalan ca-ES Language.ZH Chinese (Default: Simplified) cmn-Hans-CN Language.ZH_CN Chinese (Simplified) cmn-Hans-CN Language.ZH_HK Chinese (Hong Kong) cmn-Hans-HK Language.ZH_TW Chinese (Traditional) cmn-Hant-TW Language.YUE Chinese (Cantonese) yue-Hant-HK Language.HR Croatian hr-HR Language.CS Czech cs-CZ Language.DA Danish da-DK Language.NL Dutch (Default: Netherlands) nl-NL Language.NL_BE Dutch (Belgium) nl-BE Language.NL_NL Dutch (Netherlands) nl-NL Language.EN English (Default: US) en-US Language.EN_AU English (Australia) en-AU Language.EN_CA English (Canada) en-CA Language.EN_GB English (UK) en-GB Language.EN_GH English (Ghana) en-GH Language.EN_HK English (Hong Kong) en-HK Language.EN_IN English (India) en-IN Language.EN_IE English (Ireland) en-IE Language.EN_KE English (Kenya) en-KE Language.EN_NG English (Nigeria) en-NG Language.EN_NZ English (New Zealand) en-NZ Language.EN_PH English (Philippines) en-PH Language.EN_SG English (Singapore) en-SG Language.EN_TZ English (Tanzania) en-TZ Language.EN_US English (US) en-US Language.EN_ZA English (South Africa) en-ZA Language.ET Estonian et-EE Language.FIL Filipino fil-PH Language.FI Finnish fi-FI Language.FR French (Default: France) fr-FR Language.FR_BE French (Belgium) fr-BE Language.FR_CA French (Canada) fr-CA Language.FR_CH French (Switzerland) fr-CH Language.GL Galician gl-ES Language.KA Georgian ka-GE Language.DE German (Default: Germany) de-DE Language.DE_AT German (Austria) de-AT Language.DE_CH German (Switzerland) de-CH Language.EL Greek el-GR Language.GU Gujarati gu-IN Language.HE Hebrew iw-IL Language.HI Hindi hi-IN Language.HU Hungarian hu-HU Language.IS Icelandic is-IS Language.ID Indonesian id-ID Language.IT Italian it-IT Language.IT_CH Italian (Switzerland) it-CH Language.JA Japanese ja-JP Language.JV Javanese jv-ID Language.KN Kannada kn-IN Language.KK Kazakh kk-KZ Language.KM Khmer km-KH Language.KO Korean ko-KR Language.LO Lao lo-LA Language.LV Latvian lv-LV Language.LT Lithuanian lt-LT Language.MK Macedonian mk-MK Language.MS Malay ms-MY Language.ML Malayalam ml-IN Language.MR Marathi mr-IN Language.MN Mongolian mn-MN Language.NE Nepali ne-NP Language.NO Norwegian no-NO Language.FA Persian fa-IR Language.PL Polish pl-PL Language.PT Portuguese (Default: Portugal) pt-PT Language.PT_BR Portuguese (Brazil) pt-BR Language.PT_PT Portuguese (Portugal) pt-PT Language.PA Punjabi pa-Guru-IN Language.RO Romanian ro-RO Language.RU Russian ru-RU Language.SR Serbian sr-RS Language.SI Sinhala si-LK Language.SK Slovak sk-SK Language.SL Slovenian sl-SI Language.ES Spanish (Default: Spain) es-ES Language.ES_AR Spanish (Argentina) es-AR Language.ES_BO Spanish (Bolivia) es-BO Language.ES_CL Spanish (Chile) es-CL Language.ES_CO Spanish (Colombia) es-CO Language.ES_CR Spanish (Costa Rica) es-CR Language.ES_DO Spanish (Dominican Republic) es-DO Language.ES_EC Spanish (Ecuador) es-EC Language.ES_GT Spanish (Guatemala) es-GT Language.ES_HN Spanish (Honduras) es-HN Language.ES_MX Spanish (Mexico) es-MX Language.ES_NI Spanish (Nicaragua) es-NI Language.ES_PA Spanish (Panama) es-PA Language.ES_PE Spanish (Peru) es-PE Language.ES_PR Spanish (Puerto Rico) es-PR Language.ES_PY Spanish (Paraguay) es-PY Language.ES_SV Spanish (El Salvador) es-SV Language.ES_US Spanish (US) es-US Language.ES_UY Spanish (Uruguay) es-UY Language.ES_VE Spanish (Venezuela) es-VE Language.SU Sundanese su-ID Language.SW Swahili (Default: Tanzania) sw-TZ Language.SW_KE Swahili (Kenya) sw-KE Language.SW_TZ Swahili (Tanzania) sw-TZ Language.SV Swedish sv-SE Language.TA Tamil (Default: India) ta-IN Language.TA_IN Tamil (India) ta-IN Language.TA_MY Tamil (Malaysia) ta-MY Language.TA_SG Tamil (Singapore) ta-SG Language.TA_LK Tamil (Sri Lanka) ta-LK Language.TE Telugu te-IN Language.TH Thai th-TH Language.TR Turkish tr-TR Language.UK Ukrainian uk-UA Language.UR Urdu (Default: India) ur-IN Language.UR_IN Urdu (India) ur-IN Language.UR_PK Urdu (Pakistan) ur-PK Language.UZ Uzbek uz-UZ Language.VI Vietnamese vi-VN Language.XH Xhosa xh-ZA Language.ZU Zulu zu-ZA ​ Special Features Supports multiple languages simultaneously Provides regional variants for many languages Handles different Chinese scripts (simplified/traditional) Supports medical-specific models ​ Frame Flow ​ Notes Requires Google Cloud credentials Supports real-time transcription Handles streaming connection management Provides dynamic configuration updates Supports model switching Includes VAD capabilities Manages connection lifecycle Gladia Groq (Whisper) On this page Overview Installation Configuration Constructor Parameters InputParams Input Output Frames TranscriptionFrame InterimTranscriptionFrame Methods Usage Example Regional Support Available Regions Configuration Dynamic Region Updates Notes Models Language Support Special Features Frame Flow Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_groq_0444b58d.txt b/stt_groq_0444b58d.txt
new file mode 100644
index 0000000000000000000000000000000000000000..bcc6f5e356b56876c0bc1b13014a86f87b77a188
--- /dev/null
+++ b/stt_groq_0444b58d.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/groq#param-prompt
+Title: Groq (Whisper) - Pipecat
+==================================================
+
+Groq (Whisper) - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text Groq (Whisper) Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview GroqSTTService provides speech-to-text capabilities using Groq’s hosted Whisper API. It offers high-accuracy transcription with minimal setup requirements. The service uses Voice Activity Detection (VAD) to process only speech segments, optimizing API usage and improving response time. ​ Installation To use GroqSTTService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[groq]" You’ll need to set up your Groq API key as an environment variable: GROQ_API_KEY . You can obtain a Groq API key from the Groq Console . ​ Configuration ​ Constructor Parameters ​ model str default: "whisper-large-v3-turbo" Whisper model to use. Currently only “whisper-large-v3-turbo” is available. ​ api_key str Your Groq API key. If not provided, will use environment variable. ​ base_url str default: "https://api.groq.com/openai/v1" Custom API base URL for Groq API requests. ​ language Language default: "Language.EN" Language of the audio input. Defaults to English. ​ prompt str Optional text to guide the model’s style or continue a previous segment. ​ temperature float Sampling temperature between 0 and 1. Lower values are more deterministic, higher values more creative. Defaults to 0.0. ​ sample_rate int Audio sample rate in Hz. If not provided, uses the pipeline’s sample rate. ​ Input The service processes audio data with the following requirements: PCM audio format 16-bit depth Single channel (mono) ​ Output Frames The service produces two types of frames during transcription: ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Detected language (if available) ​ ErrorFrame Generated when transcription errors occur, containing error details. ​ Methods ​ Set Model Copy Ask AI await service.set_model( "whisper-large-v3-turbo" ) See the STT base class methods for additional functionality. ​ Language Support Groq’s Whisper API supports a wide range of languages. The service automatically maps Language enum values to the appropriate Whisper language codes. Language Code Description Whisper Code Language.AF Afrikaans af Language.AR Arabic ar Language.HY Armenian hy Language.AZ Azerbaijani az Language.BE Belarusian be Language.BS Bosnian bs Language.BG Bulgarian bg Language.CA Catalan ca Language.ZH Chinese zh Language.HR Croatian hr Language.CS Czech cs Language.DA Danish da Language.NL Dutch nl Language.EN English en Language.ET Estonian et Language.FI Finnish fi Language.FR French fr Language.GL Galician gl Language.DE German de Language.EL Greek el Language.HE Hebrew he Language.HI Hindi hi Language.HU Hungarian hu Language.IS Icelandic is Language.ID Indonesian id Language.IT Italian it Language.JA Japanese ja Language.KN Kannada kn Language.KK Kazakh kk Language.KO Korean ko Language.LV Latvian lv Language.LT Lithuanian lt Language.MK Macedonian mk Language.MS Malay ms Language.MR Marathi mr Language.MI Maori mi Language.NE Nepali ne Language.NO Norwegian no Language.FA Persian fa Language.PL Polish pl Language.PT Portuguese pt Language.RO Romanian ro Language.RU Russian ru Language.SR Serbian sr Language.SK Slovak sk Language.SL Slovenian sl Language.ES Spanish es Language.SW Swahili sw Language.SV Swedish sv Language.TL Tagalog tl Language.TA Tamil ta Language.TH Thai th Language.TR Turkish tr Language.UK Ukrainian uk Language.UR Urdu ur Language.VI Vietnamese vi Language.CY Welsh cy Groq’s Whisper implementation supports language variants (like en-US , fr-CA ) by mapping them to their base language. For example, Language.EN_US and Language.EN_GB will both map to en . The service will automatically detect the language if none is specified, but specifying the language typically improves transcription accuracy. For the most up-to-date list of supported languages, refer to the Groq documentation. ​ Usage Example Copy Ask AI from pipecat.services.groq.stt import GroqSTTService from pipecat.transcriptions.language import Language # Configure service stt = GroqSTTService( model = "whisper-large-v3-turbo" , api_key = "your-api-key" , language = Language. EN , prompt = "Transcribe the following conversation" , temperature = 0.0 ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ Voice Activity Detection Integration This service inherits from SegmentedSTTService , which uses Voice Activity Detection (VAD) to identify speech segments for processing. This approach: Processes only actual speech, not silence or background noise Maintains a small audio buffer (default 1 second) to capture speech that occurs slightly before VAD detection Receives UserStartedSpeakingFrame and UserStoppedSpeakingFrame from a VAD component in the pipeline Only sends complete utterances to the API when speech has ended Ensure your transport includes a VAD component (like SileroVADAnalyzer ) to properly detect speech segments. ​ Metrics Support The service collects the following metrics: Time to First Byte (TTFB) Processing duration API response time ​ Notes Requires valid Groq API key Uses Groq’s hosted Whisper model Requires VAD component in transport Processes complete utterances, not continuous audio Handles API rate limiting Automatic error handling Thread-safe processing ​ Error Handling The service handles common API errors including: Authentication errors Rate limiting Invalid audio format Network connectivity issues API timeouts Errors are propagated through ErrorFrames with descriptive messages. Google NVIDIA Riva On this page Overview Installation Configuration Constructor Parameters Input Output Frames TranscriptionFrame ErrorFrame Methods Set Model Language Support Usage Example Voice Activity Detection Integration Metrics Support Notes Error Handling Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_groq_2c723ea5.txt b/stt_groq_2c723ea5.txt
new file mode 100644
index 0000000000000000000000000000000000000000..b0277a14700658b436e4862b7c6fd1707b1fd1f0
--- /dev/null
+++ b/stt_groq_2c723ea5.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/groq#overview
+Title: Groq (Whisper) - Pipecat
+==================================================
+
+Groq (Whisper) - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text Groq (Whisper) Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview GroqSTTService provides speech-to-text capabilities using Groq’s hosted Whisper API. It offers high-accuracy transcription with minimal setup requirements. The service uses Voice Activity Detection (VAD) to process only speech segments, optimizing API usage and improving response time. ​ Installation To use GroqSTTService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[groq]" You’ll need to set up your Groq API key as an environment variable: GROQ_API_KEY . You can obtain a Groq API key from the Groq Console . ​ Configuration ​ Constructor Parameters ​ model str default: "whisper-large-v3-turbo" Whisper model to use. Currently only “whisper-large-v3-turbo” is available. ​ api_key str Your Groq API key. If not provided, will use environment variable. ​ base_url str default: "https://api.groq.com/openai/v1" Custom API base URL for Groq API requests. ​ language Language default: "Language.EN" Language of the audio input. Defaults to English. ​ prompt str Optional text to guide the model’s style or continue a previous segment. ​ temperature float Sampling temperature between 0 and 1. Lower values are more deterministic, higher values more creative. Defaults to 0.0. ​ sample_rate int Audio sample rate in Hz. If not provided, uses the pipeline’s sample rate. ​ Input The service processes audio data with the following requirements: PCM audio format 16-bit depth Single channel (mono) ​ Output Frames The service produces two types of frames during transcription: ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Detected language (if available) ​ ErrorFrame Generated when transcription errors occur, containing error details. ​ Methods ​ Set Model Copy Ask AI await service.set_model( "whisper-large-v3-turbo" ) See the STT base class methods for additional functionality. ​ Language Support Groq’s Whisper API supports a wide range of languages. The service automatically maps Language enum values to the appropriate Whisper language codes. Language Code Description Whisper Code Language.AF Afrikaans af Language.AR Arabic ar Language.HY Armenian hy Language.AZ Azerbaijani az Language.BE Belarusian be Language.BS Bosnian bs Language.BG Bulgarian bg Language.CA Catalan ca Language.ZH Chinese zh Language.HR Croatian hr Language.CS Czech cs Language.DA Danish da Language.NL Dutch nl Language.EN English en Language.ET Estonian et Language.FI Finnish fi Language.FR French fr Language.GL Galician gl Language.DE German de Language.EL Greek el Language.HE Hebrew he Language.HI Hindi hi Language.HU Hungarian hu Language.IS Icelandic is Language.ID Indonesian id Language.IT Italian it Language.JA Japanese ja Language.KN Kannada kn Language.KK Kazakh kk Language.KO Korean ko Language.LV Latvian lv Language.LT Lithuanian lt Language.MK Macedonian mk Language.MS Malay ms Language.MR Marathi mr Language.MI Maori mi Language.NE Nepali ne Language.NO Norwegian no Language.FA Persian fa Language.PL Polish pl Language.PT Portuguese pt Language.RO Romanian ro Language.RU Russian ru Language.SR Serbian sr Language.SK Slovak sk Language.SL Slovenian sl Language.ES Spanish es Language.SW Swahili sw Language.SV Swedish sv Language.TL Tagalog tl Language.TA Tamil ta Language.TH Thai th Language.TR Turkish tr Language.UK Ukrainian uk Language.UR Urdu ur Language.VI Vietnamese vi Language.CY Welsh cy Groq’s Whisper implementation supports language variants (like en-US , fr-CA ) by mapping them to their base language. For example, Language.EN_US and Language.EN_GB will both map to en . The service will automatically detect the language if none is specified, but specifying the language typically improves transcription accuracy. For the most up-to-date list of supported languages, refer to the Groq documentation. ​ Usage Example Copy Ask AI from pipecat.services.groq.stt import GroqSTTService from pipecat.transcriptions.language import Language # Configure service stt = GroqSTTService( model = "whisper-large-v3-turbo" , api_key = "your-api-key" , language = Language. EN , prompt = "Transcribe the following conversation" , temperature = 0.0 ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ Voice Activity Detection Integration This service inherits from SegmentedSTTService , which uses Voice Activity Detection (VAD) to identify speech segments for processing. This approach: Processes only actual speech, not silence or background noise Maintains a small audio buffer (default 1 second) to capture speech that occurs slightly before VAD detection Receives UserStartedSpeakingFrame and UserStoppedSpeakingFrame from a VAD component in the pipeline Only sends complete utterances to the API when speech has ended Ensure your transport includes a VAD component (like SileroVADAnalyzer ) to properly detect speech segments. ​ Metrics Support The service collects the following metrics: Time to First Byte (TTFB) Processing duration API response time ​ Notes Requires valid Groq API key Uses Groq’s hosted Whisper model Requires VAD component in transport Processes complete utterances, not continuous audio Handles API rate limiting Automatic error handling Thread-safe processing ​ Error Handling The service handles common API errors including: Authentication errors Rate limiting Invalid audio format Network connectivity issues API timeouts Errors are propagated through ErrorFrames with descriptive messages. Google NVIDIA Riva On this page Overview Installation Configuration Constructor Parameters Input Output Frames TranscriptionFrame ErrorFrame Methods Set Model Language Support Usage Example Voice Activity Detection Integration Metrics Support Notes Error Handling Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_groq_49b8c540.txt b/stt_groq_49b8c540.txt
new file mode 100644
index 0000000000000000000000000000000000000000..bc6c5e0c6a507bd80a57f1a52de11f57cb3b077c
--- /dev/null
+++ b/stt_groq_49b8c540.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/groq#param-temperature
+Title: Groq (Whisper) - Pipecat
+==================================================
+
+Groq (Whisper) - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text Groq (Whisper) Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview GroqSTTService provides speech-to-text capabilities using Groq’s hosted Whisper API. It offers high-accuracy transcription with minimal setup requirements. The service uses Voice Activity Detection (VAD) to process only speech segments, optimizing API usage and improving response time. ​ Installation To use GroqSTTService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[groq]" You’ll need to set up your Groq API key as an environment variable: GROQ_API_KEY . You can obtain a Groq API key from the Groq Console . ​ Configuration ​ Constructor Parameters ​ model str default: "whisper-large-v3-turbo" Whisper model to use. Currently only “whisper-large-v3-turbo” is available. ​ api_key str Your Groq API key. If not provided, will use environment variable. ​ base_url str default: "https://api.groq.com/openai/v1" Custom API base URL for Groq API requests. ​ language Language default: "Language.EN" Language of the audio input. Defaults to English. ​ prompt str Optional text to guide the model’s style or continue a previous segment. ​ temperature float Sampling temperature between 0 and 1. Lower values are more deterministic, higher values more creative. Defaults to 0.0. ​ sample_rate int Audio sample rate in Hz. If not provided, uses the pipeline’s sample rate. ​ Input The service processes audio data with the following requirements: PCM audio format 16-bit depth Single channel (mono) ​ Output Frames The service produces two types of frames during transcription: ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Detected language (if available) ​ ErrorFrame Generated when transcription errors occur, containing error details. ​ Methods ​ Set Model Copy Ask AI await service.set_model( "whisper-large-v3-turbo" ) See the STT base class methods for additional functionality. ​ Language Support Groq’s Whisper API supports a wide range of languages. The service automatically maps Language enum values to the appropriate Whisper language codes. Language Code Description Whisper Code Language.AF Afrikaans af Language.AR Arabic ar Language.HY Armenian hy Language.AZ Azerbaijani az Language.BE Belarusian be Language.BS Bosnian bs Language.BG Bulgarian bg Language.CA Catalan ca Language.ZH Chinese zh Language.HR Croatian hr Language.CS Czech cs Language.DA Danish da Language.NL Dutch nl Language.EN English en Language.ET Estonian et Language.FI Finnish fi Language.FR French fr Language.GL Galician gl Language.DE German de Language.EL Greek el Language.HE Hebrew he Language.HI Hindi hi Language.HU Hungarian hu Language.IS Icelandic is Language.ID Indonesian id Language.IT Italian it Language.JA Japanese ja Language.KN Kannada kn Language.KK Kazakh kk Language.KO Korean ko Language.LV Latvian lv Language.LT Lithuanian lt Language.MK Macedonian mk Language.MS Malay ms Language.MR Marathi mr Language.MI Maori mi Language.NE Nepali ne Language.NO Norwegian no Language.FA Persian fa Language.PL Polish pl Language.PT Portuguese pt Language.RO Romanian ro Language.RU Russian ru Language.SR Serbian sr Language.SK Slovak sk Language.SL Slovenian sl Language.ES Spanish es Language.SW Swahili sw Language.SV Swedish sv Language.TL Tagalog tl Language.TA Tamil ta Language.TH Thai th Language.TR Turkish tr Language.UK Ukrainian uk Language.UR Urdu ur Language.VI Vietnamese vi Language.CY Welsh cy Groq’s Whisper implementation supports language variants (like en-US , fr-CA ) by mapping them to their base language. For example, Language.EN_US and Language.EN_GB will both map to en . The service will automatically detect the language if none is specified, but specifying the language typically improves transcription accuracy. For the most up-to-date list of supported languages, refer to the Groq documentation. ​ Usage Example Copy Ask AI from pipecat.services.groq.stt import GroqSTTService from pipecat.transcriptions.language import Language # Configure service stt = GroqSTTService( model = "whisper-large-v3-turbo" , api_key = "your-api-key" , language = Language. EN , prompt = "Transcribe the following conversation" , temperature = 0.0 ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ Voice Activity Detection Integration This service inherits from SegmentedSTTService , which uses Voice Activity Detection (VAD) to identify speech segments for processing. This approach: Processes only actual speech, not silence or background noise Maintains a small audio buffer (default 1 second) to capture speech that occurs slightly before VAD detection Receives UserStartedSpeakingFrame and UserStoppedSpeakingFrame from a VAD component in the pipeline Only sends complete utterances to the API when speech has ended Ensure your transport includes a VAD component (like SileroVADAnalyzer ) to properly detect speech segments. ​ Metrics Support The service collects the following metrics: Time to First Byte (TTFB) Processing duration API response time ​ Notes Requires valid Groq API key Uses Groq’s hosted Whisper model Requires VAD component in transport Processes complete utterances, not continuous audio Handles API rate limiting Automatic error handling Thread-safe processing ​ Error Handling The service handles common API errors including: Authentication errors Rate limiting Invalid audio format Network connectivity issues API timeouts Errors are propagated through ErrorFrames with descriptive messages. Google NVIDIA Riva On this page Overview Installation Configuration Constructor Parameters Input Output Frames TranscriptionFrame ErrorFrame Methods Set Model Language Support Usage Example Voice Activity Detection Integration Metrics Support Notes Error Handling Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_groq_88d898d0.txt b/stt_groq_88d898d0.txt
new file mode 100644
index 0000000000000000000000000000000000000000..74f8cfdd5e652b91f4d5a21e677e007c25a98f20
--- /dev/null
+++ b/stt_groq_88d898d0.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/groq#configuration
+Title: Groq (Whisper) - Pipecat
+==================================================
+
+Groq (Whisper) - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text Groq (Whisper) Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview GroqSTTService provides speech-to-text capabilities using Groq’s hosted Whisper API. It offers high-accuracy transcription with minimal setup requirements. The service uses Voice Activity Detection (VAD) to process only speech segments, optimizing API usage and improving response time. ​ Installation To use GroqSTTService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[groq]" You’ll need to set up your Groq API key as an environment variable: GROQ_API_KEY . You can obtain a Groq API key from the Groq Console . ​ Configuration ​ Constructor Parameters ​ model str default: "whisper-large-v3-turbo" Whisper model to use. Currently only “whisper-large-v3-turbo” is available. ​ api_key str Your Groq API key. If not provided, will use environment variable. ​ base_url str default: "https://api.groq.com/openai/v1" Custom API base URL for Groq API requests. ​ language Language default: "Language.EN" Language of the audio input. Defaults to English. ​ prompt str Optional text to guide the model’s style or continue a previous segment. ​ temperature float Sampling temperature between 0 and 1. Lower values are more deterministic, higher values more creative. Defaults to 0.0. ​ sample_rate int Audio sample rate in Hz. If not provided, uses the pipeline’s sample rate. ​ Input The service processes audio data with the following requirements: PCM audio format 16-bit depth Single channel (mono) ​ Output Frames The service produces two types of frames during transcription: ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Detected language (if available) ​ ErrorFrame Generated when transcription errors occur, containing error details. ​ Methods ​ Set Model Copy Ask AI await service.set_model( "whisper-large-v3-turbo" ) See the STT base class methods for additional functionality. ​ Language Support Groq’s Whisper API supports a wide range of languages. The service automatically maps Language enum values to the appropriate Whisper language codes. Language Code Description Whisper Code Language.AF Afrikaans af Language.AR Arabic ar Language.HY Armenian hy Language.AZ Azerbaijani az Language.BE Belarusian be Language.BS Bosnian bs Language.BG Bulgarian bg Language.CA Catalan ca Language.ZH Chinese zh Language.HR Croatian hr Language.CS Czech cs Language.DA Danish da Language.NL Dutch nl Language.EN English en Language.ET Estonian et Language.FI Finnish fi Language.FR French fr Language.GL Galician gl Language.DE German de Language.EL Greek el Language.HE Hebrew he Language.HI Hindi hi Language.HU Hungarian hu Language.IS Icelandic is Language.ID Indonesian id Language.IT Italian it Language.JA Japanese ja Language.KN Kannada kn Language.KK Kazakh kk Language.KO Korean ko Language.LV Latvian lv Language.LT Lithuanian lt Language.MK Macedonian mk Language.MS Malay ms Language.MR Marathi mr Language.MI Maori mi Language.NE Nepali ne Language.NO Norwegian no Language.FA Persian fa Language.PL Polish pl Language.PT Portuguese pt Language.RO Romanian ro Language.RU Russian ru Language.SR Serbian sr Language.SK Slovak sk Language.SL Slovenian sl Language.ES Spanish es Language.SW Swahili sw Language.SV Swedish sv Language.TL Tagalog tl Language.TA Tamil ta Language.TH Thai th Language.TR Turkish tr Language.UK Ukrainian uk Language.UR Urdu ur Language.VI Vietnamese vi Language.CY Welsh cy Groq’s Whisper implementation supports language variants (like en-US , fr-CA ) by mapping them to their base language. For example, Language.EN_US and Language.EN_GB will both map to en . The service will automatically detect the language if none is specified, but specifying the language typically improves transcription accuracy. For the most up-to-date list of supported languages, refer to the Groq documentation. ​ Usage Example Copy Ask AI from pipecat.services.groq.stt import GroqSTTService from pipecat.transcriptions.language import Language # Configure service stt = GroqSTTService( model = "whisper-large-v3-turbo" , api_key = "your-api-key" , language = Language. EN , prompt = "Transcribe the following conversation" , temperature = 0.0 ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ Voice Activity Detection Integration This service inherits from SegmentedSTTService , which uses Voice Activity Detection (VAD) to identify speech segments for processing. This approach: Processes only actual speech, not silence or background noise Maintains a small audio buffer (default 1 second) to capture speech that occurs slightly before VAD detection Receives UserStartedSpeakingFrame and UserStoppedSpeakingFrame from a VAD component in the pipeline Only sends complete utterances to the API when speech has ended Ensure your transport includes a VAD component (like SileroVADAnalyzer ) to properly detect speech segments. ​ Metrics Support The service collects the following metrics: Time to First Byte (TTFB) Processing duration API response time ​ Notes Requires valid Groq API key Uses Groq’s hosted Whisper model Requires VAD component in transport Processes complete utterances, not continuous audio Handles API rate limiting Automatic error handling Thread-safe processing ​ Error Handling The service handles common API errors including: Authentication errors Rate limiting Invalid audio format Network connectivity issues API timeouts Errors are propagated through ErrorFrames with descriptive messages. Google NVIDIA Riva On this page Overview Installation Configuration Constructor Parameters Input Output Frames TranscriptionFrame ErrorFrame Methods Set Model Language Support Usage Example Voice Activity Detection Integration Metrics Support Notes Error Handling Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_groq_9fdba157.txt b/stt_groq_9fdba157.txt
new file mode 100644
index 0000000000000000000000000000000000000000..90e7b3b12ec41c792362621e40efa9a4474bfb5f
--- /dev/null
+++ b/stt_groq_9fdba157.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/groq#voice-activity-detection-integration
+Title: Groq (Whisper) - Pipecat
+==================================================
+
+Groq (Whisper) - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text Groq (Whisper) Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview GroqSTTService provides speech-to-text capabilities using Groq’s hosted Whisper API. It offers high-accuracy transcription with minimal setup requirements. The service uses Voice Activity Detection (VAD) to process only speech segments, optimizing API usage and improving response time. ​ Installation To use GroqSTTService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[groq]" You’ll need to set up your Groq API key as an environment variable: GROQ_API_KEY . You can obtain a Groq API key from the Groq Console . ​ Configuration ​ Constructor Parameters ​ model str default: "whisper-large-v3-turbo" Whisper model to use. Currently only “whisper-large-v3-turbo” is available. ​ api_key str Your Groq API key. If not provided, will use environment variable. ​ base_url str default: "https://api.groq.com/openai/v1" Custom API base URL for Groq API requests. ​ language Language default: "Language.EN" Language of the audio input. Defaults to English. ​ prompt str Optional text to guide the model’s style or continue a previous segment. ​ temperature float Sampling temperature between 0 and 1. Lower values are more deterministic, higher values more creative. Defaults to 0.0. ​ sample_rate int Audio sample rate in Hz. If not provided, uses the pipeline’s sample rate. ​ Input The service processes audio data with the following requirements: PCM audio format 16-bit depth Single channel (mono) ​ Output Frames The service produces two types of frames during transcription: ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Detected language (if available) ​ ErrorFrame Generated when transcription errors occur, containing error details. ​ Methods ​ Set Model Copy Ask AI await service.set_model( "whisper-large-v3-turbo" ) See the STT base class methods for additional functionality. ​ Language Support Groq’s Whisper API supports a wide range of languages. The service automatically maps Language enum values to the appropriate Whisper language codes. Language Code Description Whisper Code Language.AF Afrikaans af Language.AR Arabic ar Language.HY Armenian hy Language.AZ Azerbaijani az Language.BE Belarusian be Language.BS Bosnian bs Language.BG Bulgarian bg Language.CA Catalan ca Language.ZH Chinese zh Language.HR Croatian hr Language.CS Czech cs Language.DA Danish da Language.NL Dutch nl Language.EN English en Language.ET Estonian et Language.FI Finnish fi Language.FR French fr Language.GL Galician gl Language.DE German de Language.EL Greek el Language.HE Hebrew he Language.HI Hindi hi Language.HU Hungarian hu Language.IS Icelandic is Language.ID Indonesian id Language.IT Italian it Language.JA Japanese ja Language.KN Kannada kn Language.KK Kazakh kk Language.KO Korean ko Language.LV Latvian lv Language.LT Lithuanian lt Language.MK Macedonian mk Language.MS Malay ms Language.MR Marathi mr Language.MI Maori mi Language.NE Nepali ne Language.NO Norwegian no Language.FA Persian fa Language.PL Polish pl Language.PT Portuguese pt Language.RO Romanian ro Language.RU Russian ru Language.SR Serbian sr Language.SK Slovak sk Language.SL Slovenian sl Language.ES Spanish es Language.SW Swahili sw Language.SV Swedish sv Language.TL Tagalog tl Language.TA Tamil ta Language.TH Thai th Language.TR Turkish tr Language.UK Ukrainian uk Language.UR Urdu ur Language.VI Vietnamese vi Language.CY Welsh cy Groq’s Whisper implementation supports language variants (like en-US , fr-CA ) by mapping them to their base language. For example, Language.EN_US and Language.EN_GB will both map to en . The service will automatically detect the language if none is specified, but specifying the language typically improves transcription accuracy. For the most up-to-date list of supported languages, refer to the Groq documentation. ​ Usage Example Copy Ask AI from pipecat.services.groq.stt import GroqSTTService from pipecat.transcriptions.language import Language # Configure service stt = GroqSTTService( model = "whisper-large-v3-turbo" , api_key = "your-api-key" , language = Language. EN , prompt = "Transcribe the following conversation" , temperature = 0.0 ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ Voice Activity Detection Integration This service inherits from SegmentedSTTService , which uses Voice Activity Detection (VAD) to identify speech segments for processing. This approach: Processes only actual speech, not silence or background noise Maintains a small audio buffer (default 1 second) to capture speech that occurs slightly before VAD detection Receives UserStartedSpeakingFrame and UserStoppedSpeakingFrame from a VAD component in the pipeline Only sends complete utterances to the API when speech has ended Ensure your transport includes a VAD component (like SileroVADAnalyzer ) to properly detect speech segments. ​ Metrics Support The service collects the following metrics: Time to First Byte (TTFB) Processing duration API response time ​ Notes Requires valid Groq API key Uses Groq’s hosted Whisper model Requires VAD component in transport Processes complete utterances, not continuous audio Handles API rate limiting Automatic error handling Thread-safe processing ​ Error Handling The service handles common API errors including: Authentication errors Rate limiting Invalid audio format Network connectivity issues API timeouts Errors are propagated through ErrorFrames with descriptive messages. Google NVIDIA Riva On this page Overview Installation Configuration Constructor Parameters Input Output Frames TranscriptionFrame ErrorFrame Methods Set Model Language Support Usage Example Voice Activity Detection Integration Metrics Support Notes Error Handling Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_groq_a52f2db2.txt b/stt_groq_a52f2db2.txt
new file mode 100644
index 0000000000000000000000000000000000000000..4741a6d399efd825dbb52ae91878999988ad323a
--- /dev/null
+++ b/stt_groq_a52f2db2.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/groq#output-frames
+Title: Groq (Whisper) - Pipecat
+==================================================
+
+Groq (Whisper) - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text Groq (Whisper) Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview GroqSTTService provides speech-to-text capabilities using Groq’s hosted Whisper API. It offers high-accuracy transcription with minimal setup requirements. The service uses Voice Activity Detection (VAD) to process only speech segments, optimizing API usage and improving response time. ​ Installation To use GroqSTTService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[groq]" You’ll need to set up your Groq API key as an environment variable: GROQ_API_KEY . You can obtain a Groq API key from the Groq Console . ​ Configuration ​ Constructor Parameters ​ model str default: "whisper-large-v3-turbo" Whisper model to use. Currently only “whisper-large-v3-turbo” is available. ​ api_key str Your Groq API key. If not provided, will use environment variable. ​ base_url str default: "https://api.groq.com/openai/v1" Custom API base URL for Groq API requests. ​ language Language default: "Language.EN" Language of the audio input. Defaults to English. ​ prompt str Optional text to guide the model’s style or continue a previous segment. ​ temperature float Sampling temperature between 0 and 1. Lower values are more deterministic, higher values more creative. Defaults to 0.0. ​ sample_rate int Audio sample rate in Hz. If not provided, uses the pipeline’s sample rate. ​ Input The service processes audio data with the following requirements: PCM audio format 16-bit depth Single channel (mono) ​ Output Frames The service produces two types of frames during transcription: ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Detected language (if available) ​ ErrorFrame Generated when transcription errors occur, containing error details. ​ Methods ​ Set Model Copy Ask AI await service.set_model( "whisper-large-v3-turbo" ) See the STT base class methods for additional functionality. ​ Language Support Groq’s Whisper API supports a wide range of languages. The service automatically maps Language enum values to the appropriate Whisper language codes. Language Code Description Whisper Code Language.AF Afrikaans af Language.AR Arabic ar Language.HY Armenian hy Language.AZ Azerbaijani az Language.BE Belarusian be Language.BS Bosnian bs Language.BG Bulgarian bg Language.CA Catalan ca Language.ZH Chinese zh Language.HR Croatian hr Language.CS Czech cs Language.DA Danish da Language.NL Dutch nl Language.EN English en Language.ET Estonian et Language.FI Finnish fi Language.FR French fr Language.GL Galician gl Language.DE German de Language.EL Greek el Language.HE Hebrew he Language.HI Hindi hi Language.HU Hungarian hu Language.IS Icelandic is Language.ID Indonesian id Language.IT Italian it Language.JA Japanese ja Language.KN Kannada kn Language.KK Kazakh kk Language.KO Korean ko Language.LV Latvian lv Language.LT Lithuanian lt Language.MK Macedonian mk Language.MS Malay ms Language.MR Marathi mr Language.MI Maori mi Language.NE Nepali ne Language.NO Norwegian no Language.FA Persian fa Language.PL Polish pl Language.PT Portuguese pt Language.RO Romanian ro Language.RU Russian ru Language.SR Serbian sr Language.SK Slovak sk Language.SL Slovenian sl Language.ES Spanish es Language.SW Swahili sw Language.SV Swedish sv Language.TL Tagalog tl Language.TA Tamil ta Language.TH Thai th Language.TR Turkish tr Language.UK Ukrainian uk Language.UR Urdu ur Language.VI Vietnamese vi Language.CY Welsh cy Groq’s Whisper implementation supports language variants (like en-US , fr-CA ) by mapping them to their base language. For example, Language.EN_US and Language.EN_GB will both map to en . The service will automatically detect the language if none is specified, but specifying the language typically improves transcription accuracy. For the most up-to-date list of supported languages, refer to the Groq documentation. ​ Usage Example Copy Ask AI from pipecat.services.groq.stt import GroqSTTService from pipecat.transcriptions.language import Language # Configure service stt = GroqSTTService( model = "whisper-large-v3-turbo" , api_key = "your-api-key" , language = Language. EN , prompt = "Transcribe the following conversation" , temperature = 0.0 ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ Voice Activity Detection Integration This service inherits from SegmentedSTTService , which uses Voice Activity Detection (VAD) to identify speech segments for processing. This approach: Processes only actual speech, not silence or background noise Maintains a small audio buffer (default 1 second) to capture speech that occurs slightly before VAD detection Receives UserStartedSpeakingFrame and UserStoppedSpeakingFrame from a VAD component in the pipeline Only sends complete utterances to the API when speech has ended Ensure your transport includes a VAD component (like SileroVADAnalyzer ) to properly detect speech segments. ​ Metrics Support The service collects the following metrics: Time to First Byte (TTFB) Processing duration API response time ​ Notes Requires valid Groq API key Uses Groq’s hosted Whisper model Requires VAD component in transport Processes complete utterances, not continuous audio Handles API rate limiting Automatic error handling Thread-safe processing ​ Error Handling The service handles common API errors including: Authentication errors Rate limiting Invalid audio format Network connectivity issues API timeouts Errors are propagated through ErrorFrames with descriptive messages. Google NVIDIA Riva On this page Overview Installation Configuration Constructor Parameters Input Output Frames TranscriptionFrame ErrorFrame Methods Set Model Language Support Usage Example Voice Activity Detection Integration Metrics Support Notes Error Handling Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_groq_d03245fc.txt b/stt_groq_d03245fc.txt
new file mode 100644
index 0000000000000000000000000000000000000000..5c437890e7d7c5e084d92753639d3389564a95e4
--- /dev/null
+++ b/stt_groq_d03245fc.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/groq#transcriptionframe
+Title: Groq (Whisper) - Pipecat
+==================================================
+
+Groq (Whisper) - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text Groq (Whisper) Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview GroqSTTService provides speech-to-text capabilities using Groq’s hosted Whisper API. It offers high-accuracy transcription with minimal setup requirements. The service uses Voice Activity Detection (VAD) to process only speech segments, optimizing API usage and improving response time. ​ Installation To use GroqSTTService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[groq]" You’ll need to set up your Groq API key as an environment variable: GROQ_API_KEY . You can obtain a Groq API key from the Groq Console . ​ Configuration ​ Constructor Parameters ​ model str default: "whisper-large-v3-turbo" Whisper model to use. Currently only “whisper-large-v3-turbo” is available. ​ api_key str Your Groq API key. If not provided, will use environment variable. ​ base_url str default: "https://api.groq.com/openai/v1" Custom API base URL for Groq API requests. ​ language Language default: "Language.EN" Language of the audio input. Defaults to English. ​ prompt str Optional text to guide the model’s style or continue a previous segment. ​ temperature float Sampling temperature between 0 and 1. Lower values are more deterministic, higher values more creative. Defaults to 0.0. ​ sample_rate int Audio sample rate in Hz. If not provided, uses the pipeline’s sample rate. ​ Input The service processes audio data with the following requirements: PCM audio format 16-bit depth Single channel (mono) ​ Output Frames The service produces two types of frames during transcription: ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Detected language (if available) ​ ErrorFrame Generated when transcription errors occur, containing error details. ​ Methods ​ Set Model Copy Ask AI await service.set_model( "whisper-large-v3-turbo" ) See the STT base class methods for additional functionality. ​ Language Support Groq’s Whisper API supports a wide range of languages. The service automatically maps Language enum values to the appropriate Whisper language codes. Language Code Description Whisper Code Language.AF Afrikaans af Language.AR Arabic ar Language.HY Armenian hy Language.AZ Azerbaijani az Language.BE Belarusian be Language.BS Bosnian bs Language.BG Bulgarian bg Language.CA Catalan ca Language.ZH Chinese zh Language.HR Croatian hr Language.CS Czech cs Language.DA Danish da Language.NL Dutch nl Language.EN English en Language.ET Estonian et Language.FI Finnish fi Language.FR French fr Language.GL Galician gl Language.DE German de Language.EL Greek el Language.HE Hebrew he Language.HI Hindi hi Language.HU Hungarian hu Language.IS Icelandic is Language.ID Indonesian id Language.IT Italian it Language.JA Japanese ja Language.KN Kannada kn Language.KK Kazakh kk Language.KO Korean ko Language.LV Latvian lv Language.LT Lithuanian lt Language.MK Macedonian mk Language.MS Malay ms Language.MR Marathi mr Language.MI Maori mi Language.NE Nepali ne Language.NO Norwegian no Language.FA Persian fa Language.PL Polish pl Language.PT Portuguese pt Language.RO Romanian ro Language.RU Russian ru Language.SR Serbian sr Language.SK Slovak sk Language.SL Slovenian sl Language.ES Spanish es Language.SW Swahili sw Language.SV Swedish sv Language.TL Tagalog tl Language.TA Tamil ta Language.TH Thai th Language.TR Turkish tr Language.UK Ukrainian uk Language.UR Urdu ur Language.VI Vietnamese vi Language.CY Welsh cy Groq’s Whisper implementation supports language variants (like en-US , fr-CA ) by mapping them to their base language. For example, Language.EN_US and Language.EN_GB will both map to en . The service will automatically detect the language if none is specified, but specifying the language typically improves transcription accuracy. For the most up-to-date list of supported languages, refer to the Groq documentation. ​ Usage Example Copy Ask AI from pipecat.services.groq.stt import GroqSTTService from pipecat.transcriptions.language import Language # Configure service stt = GroqSTTService( model = "whisper-large-v3-turbo" , api_key = "your-api-key" , language = Language. EN , prompt = "Transcribe the following conversation" , temperature = 0.0 ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ Voice Activity Detection Integration This service inherits from SegmentedSTTService , which uses Voice Activity Detection (VAD) to identify speech segments for processing. This approach: Processes only actual speech, not silence or background noise Maintains a small audio buffer (default 1 second) to capture speech that occurs slightly before VAD detection Receives UserStartedSpeakingFrame and UserStoppedSpeakingFrame from a VAD component in the pipeline Only sends complete utterances to the API when speech has ended Ensure your transport includes a VAD component (like SileroVADAnalyzer ) to properly detect speech segments. ​ Metrics Support The service collects the following metrics: Time to First Byte (TTFB) Processing duration API response time ​ Notes Requires valid Groq API key Uses Groq’s hosted Whisper model Requires VAD component in transport Processes complete utterances, not continuous audio Handles API rate limiting Automatic error handling Thread-safe processing ​ Error Handling The service handles common API errors including: Authentication errors Rate limiting Invalid audio format Network connectivity issues API timeouts Errors are propagated through ErrorFrames with descriptive messages. Google NVIDIA Riva On this page Overview Installation Configuration Constructor Parameters Input Output Frames TranscriptionFrame ErrorFrame Methods Set Model Language Support Usage Example Voice Activity Detection Integration Metrics Support Notes Error Handling Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_groq_dd4d7cd1.txt b/stt_groq_dd4d7cd1.txt
new file mode 100644
index 0000000000000000000000000000000000000000..eaf733add48a3fde08bf8ec7457cab125f1fcd4b
--- /dev/null
+++ b/stt_groq_dd4d7cd1.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/groq#param-base-url
+Title: Groq (Whisper) - Pipecat
+==================================================
+
+Groq (Whisper) - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text Groq (Whisper) Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview GroqSTTService provides speech-to-text capabilities using Groq’s hosted Whisper API. It offers high-accuracy transcription with minimal setup requirements. The service uses Voice Activity Detection (VAD) to process only speech segments, optimizing API usage and improving response time. ​ Installation To use GroqSTTService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[groq]" You’ll need to set up your Groq API key as an environment variable: GROQ_API_KEY . You can obtain a Groq API key from the Groq Console . ​ Configuration ​ Constructor Parameters ​ model str default: "whisper-large-v3-turbo" Whisper model to use. Currently only “whisper-large-v3-turbo” is available. ​ api_key str Your Groq API key. If not provided, will use environment variable. ​ base_url str default: "https://api.groq.com/openai/v1" Custom API base URL for Groq API requests. ​ language Language default: "Language.EN" Language of the audio input. Defaults to English. ​ prompt str Optional text to guide the model’s style or continue a previous segment. ​ temperature float Sampling temperature between 0 and 1. Lower values are more deterministic, higher values more creative. Defaults to 0.0. ​ sample_rate int Audio sample rate in Hz. If not provided, uses the pipeline’s sample rate. ​ Input The service processes audio data with the following requirements: PCM audio format 16-bit depth Single channel (mono) ​ Output Frames The service produces two types of frames during transcription: ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Detected language (if available) ​ ErrorFrame Generated when transcription errors occur, containing error details. ​ Methods ​ Set Model Copy Ask AI await service.set_model( "whisper-large-v3-turbo" ) See the STT base class methods for additional functionality. ​ Language Support Groq’s Whisper API supports a wide range of languages. The service automatically maps Language enum values to the appropriate Whisper language codes. Language Code Description Whisper Code Language.AF Afrikaans af Language.AR Arabic ar Language.HY Armenian hy Language.AZ Azerbaijani az Language.BE Belarusian be Language.BS Bosnian bs Language.BG Bulgarian bg Language.CA Catalan ca Language.ZH Chinese zh Language.HR Croatian hr Language.CS Czech cs Language.DA Danish da Language.NL Dutch nl Language.EN English en Language.ET Estonian et Language.FI Finnish fi Language.FR French fr Language.GL Galician gl Language.DE German de Language.EL Greek el Language.HE Hebrew he Language.HI Hindi hi Language.HU Hungarian hu Language.IS Icelandic is Language.ID Indonesian id Language.IT Italian it Language.JA Japanese ja Language.KN Kannada kn Language.KK Kazakh kk Language.KO Korean ko Language.LV Latvian lv Language.LT Lithuanian lt Language.MK Macedonian mk Language.MS Malay ms Language.MR Marathi mr Language.MI Maori mi Language.NE Nepali ne Language.NO Norwegian no Language.FA Persian fa Language.PL Polish pl Language.PT Portuguese pt Language.RO Romanian ro Language.RU Russian ru Language.SR Serbian sr Language.SK Slovak sk Language.SL Slovenian sl Language.ES Spanish es Language.SW Swahili sw Language.SV Swedish sv Language.TL Tagalog tl Language.TA Tamil ta Language.TH Thai th Language.TR Turkish tr Language.UK Ukrainian uk Language.UR Urdu ur Language.VI Vietnamese vi Language.CY Welsh cy Groq’s Whisper implementation supports language variants (like en-US , fr-CA ) by mapping them to their base language. For example, Language.EN_US and Language.EN_GB will both map to en . The service will automatically detect the language if none is specified, but specifying the language typically improves transcription accuracy. For the most up-to-date list of supported languages, refer to the Groq documentation. ​ Usage Example Copy Ask AI from pipecat.services.groq.stt import GroqSTTService from pipecat.transcriptions.language import Language # Configure service stt = GroqSTTService( model = "whisper-large-v3-turbo" , api_key = "your-api-key" , language = Language. EN , prompt = "Transcribe the following conversation" , temperature = 0.0 ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ Voice Activity Detection Integration This service inherits from SegmentedSTTService , which uses Voice Activity Detection (VAD) to identify speech segments for processing. This approach: Processes only actual speech, not silence or background noise Maintains a small audio buffer (default 1 second) to capture speech that occurs slightly before VAD detection Receives UserStartedSpeakingFrame and UserStoppedSpeakingFrame from a VAD component in the pipeline Only sends complete utterances to the API when speech has ended Ensure your transport includes a VAD component (like SileroVADAnalyzer ) to properly detect speech segments. ​ Metrics Support The service collects the following metrics: Time to First Byte (TTFB) Processing duration API response time ​ Notes Requires valid Groq API key Uses Groq’s hosted Whisper model Requires VAD component in transport Processes complete utterances, not continuous audio Handles API rate limiting Automatic error handling Thread-safe processing ​ Error Handling The service handles common API errors including: Authentication errors Rate limiting Invalid audio format Network connectivity issues API timeouts Errors are propagated through ErrorFrames with descriptive messages. Google NVIDIA Riva On this page Overview Installation Configuration Constructor Parameters Input Output Frames TranscriptionFrame ErrorFrame Methods Set Model Language Support Usage Example Voice Activity Detection Integration Metrics Support Notes Error Handling Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_openai_2efa8553.txt b/stt_openai_2efa8553.txt
new file mode 100644
index 0000000000000000000000000000000000000000..acbdcb172e277ea7b7454bef5c66192200f8f2b6
--- /dev/null
+++ b/stt_openai_2efa8553.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/openai#param-base-url
+Title: OpenAI - Pipecat
+==================================================
+
+OpenAI - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text OpenAI Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview OpenAISTTService provides speech-to-text capabilities using OpenAI’s latest models, including the GPT-4o transcription model and the hosted Whisper API. It offers high-accuracy transcription with minimal setup requirements, using Voice Activity Detection (VAD) to process only speech segments. ​ Installation To use OpenAISTTService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[openai]" You’ll need to set up your OpenAI API key as an environment variable: OPENAI_API_KEY . You can obtain an OpenAI API key from the OpenAI platform . ​ Configuration ​ Constructor Parameters ​ model str default: "gpt-4o-transcribe" Model to use. Supported models include “gpt-4o-transcribe” (recommended) and “whisper-1”. ​ api_key str Your OpenAI API key. ​ base_url str Custom API base URL for OpenAI API requests. ​ language Language default: "Language.EN" Language of the audio input. Defaults to English. ​ prompt str Optional text to guide the model’s style or continue a previous segment. ​ temperature float Sampling temperature between 0 and 1. Lower values are more deterministic, higher values more creative. Defaults to 0.0. ​ sample_rate int Audio sample rate in Hz. If not provided, uses the pipeline’s sample rate. ​ Input The service processes audio data with the following requirements: PCM audio format 16-bit depth Single channel (mono) ​ Output Frames The service produces two types of frames during transcription: ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Detected language (if available) ​ ErrorFrame Generated when transcription errors occur, containing error details. ​ Methods ​ Set Model Copy Ask AI await service.set_model( "gpt-4o-transcribe" ) # For the latest GPT-4o transcription model # or await service.set_model( "whisper-1" ) # For the Whisper model See the STT base class methods for additional functionality. ​ Models Model Description Best For gpt-4o-transcribe Latest GPT-4o model fine-tuned for transcription High accuracy, robustness to accents, better context understanding whisper-1 OpenAI’s Whisper model Broad language support, good performance on clean audio ​ Language Support OpenAI’s speech-to-text models support a wide range of languages. The service automatically maps Language enum values to the appropriate language codes. Language Code Description Service Code Language.AF Afrikaans af Language.AR Arabic ar Language.HY Armenian hy Language.AZ Azerbaijani az Language.BE Belarusian be Language.BS Bosnian bs Language.BG Bulgarian bg Language.CA Catalan ca Language.ZH Chinese zh Language.HR Croatian hr Language.CS Czech cs Language.DA Danish da Language.NL Dutch nl Language.EN English en Language.ET Estonian et Language.FI Finnish fi Language.FR French fr Language.GL Galician gl Language.DE German de Language.EL Greek el Language.HE Hebrew he Language.HI Hindi hi Language.HU Hungarian hu Language.IS Icelandic is Language.ID Indonesian id Language.IT Italian it Language.JA Japanese ja Language.KN Kannada kn Language.KK Kazakh kk Language.KO Korean ko Language.LV Latvian lv Language.LT Lithuanian lt Language.MK Macedonian mk Language.MS Malay ms Language.MR Marathi mr Language.MI Maori mi Language.NE Nepali ne Language.NO Norwegian no Language.FA Persian fa Language.PL Polish pl Language.PT Portuguese pt Language.RO Romanian ro Language.RU Russian ru Language.SR Serbian sr Language.SK Slovak sk Language.SL Slovenian sl Language.ES Spanish es Language.SW Swahili sw Language.SV Swedish sv Language.TL Tagalog tl Language.TA Tamil ta Language.TH Thai th Language.TR Turkish tr Language.UK Ukrainian uk Language.UR Urdu ur Language.VI Vietnamese vi Language.CY Welsh cy OpenAI’s models support language variants (like en-US , fr-CA ) by mapping them to their base language. For example, Language.EN_US and Language.EN_GB will both map to en . The service will automatically detect the language if none is specified, but specifying the language typically improves transcription accuracy. ​ Usage Example Copy Ask AI from pipecat.services.openai.stt import OpenAISTTService # Configure service stt = OpenAISTTService( model = "gpt-4o-transcribe" , api_key = "your-api-key" , language = Language. EN , prompt = "Transcribe technical terms accurately. Format numbers as digits rather than words." ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ Voice Activity Detection Integration This service inherits from SegmentedSTTService , which uses Voice Activity Detection (VAD) to identify speech segments for processing. This approach: Processes only actual speech, not silence or background noise Maintains a small audio buffer (default 1 second) to capture speech that occurs slightly before VAD detection Receives UserStartedSpeakingFrame and UserStoppedSpeakingFrame from a VAD component in the pipeline Only sends complete utterances to the API when speech has ended Ensure your transport includes a VAD component (like SileroVADAnalyzer ) to properly detect speech segments. ​ Metrics Support The service collects the following metrics: Time to First Byte (TTFB) Processing duration API response time ​ Notes Requires valid OpenAI API key GPT-4o transcription model offers superior accuracy to Whisper Requires VAD component in transport Handles API rate limiting Automatic error handling Thread-safe processing ​ Error Handling The service handles common API errors including: Authentication errors Rate limiting Invalid audio format Network connectivity issues API timeouts Errors are propagated through ErrorFrames with descriptive messages. NVIDIA Riva SambaNova (Whisper) On this page Overview Installation Configuration Constructor Parameters Input Output Frames TranscriptionFrame ErrorFrame Methods Set Model Models Language Support Usage Example Voice Activity Detection Integration Metrics Support Notes Error Handling Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_openai_372ca5ee.txt b/stt_openai_372ca5ee.txt
new file mode 100644
index 0000000000000000000000000000000000000000..a12b91c3b2c4dfc6ee34102e858d4e2dbb924cc8
--- /dev/null
+++ b/stt_openai_372ca5ee.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/openai#constructor-parameters
+Title: OpenAI - Pipecat
+==================================================
+
+OpenAI - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text OpenAI Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview OpenAISTTService provides speech-to-text capabilities using OpenAI’s latest models, including the GPT-4o transcription model and the hosted Whisper API. It offers high-accuracy transcription with minimal setup requirements, using Voice Activity Detection (VAD) to process only speech segments. ​ Installation To use OpenAISTTService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[openai]" You’ll need to set up your OpenAI API key as an environment variable: OPENAI_API_KEY . You can obtain an OpenAI API key from the OpenAI platform . ​ Configuration ​ Constructor Parameters ​ model str default: "gpt-4o-transcribe" Model to use. Supported models include “gpt-4o-transcribe” (recommended) and “whisper-1”. ​ api_key str Your OpenAI API key. ​ base_url str Custom API base URL for OpenAI API requests. ​ language Language default: "Language.EN" Language of the audio input. Defaults to English. ​ prompt str Optional text to guide the model’s style or continue a previous segment. ​ temperature float Sampling temperature between 0 and 1. Lower values are more deterministic, higher values more creative. Defaults to 0.0. ​ sample_rate int Audio sample rate in Hz. If not provided, uses the pipeline’s sample rate. ​ Input The service processes audio data with the following requirements: PCM audio format 16-bit depth Single channel (mono) ​ Output Frames The service produces two types of frames during transcription: ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Detected language (if available) ​ ErrorFrame Generated when transcription errors occur, containing error details. ​ Methods ​ Set Model Copy Ask AI await service.set_model( "gpt-4o-transcribe" ) # For the latest GPT-4o transcription model # or await service.set_model( "whisper-1" ) # For the Whisper model See the STT base class methods for additional functionality. ​ Models Model Description Best For gpt-4o-transcribe Latest GPT-4o model fine-tuned for transcription High accuracy, robustness to accents, better context understanding whisper-1 OpenAI’s Whisper model Broad language support, good performance on clean audio ​ Language Support OpenAI’s speech-to-text models support a wide range of languages. The service automatically maps Language enum values to the appropriate language codes. Language Code Description Service Code Language.AF Afrikaans af Language.AR Arabic ar Language.HY Armenian hy Language.AZ Azerbaijani az Language.BE Belarusian be Language.BS Bosnian bs Language.BG Bulgarian bg Language.CA Catalan ca Language.ZH Chinese zh Language.HR Croatian hr Language.CS Czech cs Language.DA Danish da Language.NL Dutch nl Language.EN English en Language.ET Estonian et Language.FI Finnish fi Language.FR French fr Language.GL Galician gl Language.DE German de Language.EL Greek el Language.HE Hebrew he Language.HI Hindi hi Language.HU Hungarian hu Language.IS Icelandic is Language.ID Indonesian id Language.IT Italian it Language.JA Japanese ja Language.KN Kannada kn Language.KK Kazakh kk Language.KO Korean ko Language.LV Latvian lv Language.LT Lithuanian lt Language.MK Macedonian mk Language.MS Malay ms Language.MR Marathi mr Language.MI Maori mi Language.NE Nepali ne Language.NO Norwegian no Language.FA Persian fa Language.PL Polish pl Language.PT Portuguese pt Language.RO Romanian ro Language.RU Russian ru Language.SR Serbian sr Language.SK Slovak sk Language.SL Slovenian sl Language.ES Spanish es Language.SW Swahili sw Language.SV Swedish sv Language.TL Tagalog tl Language.TA Tamil ta Language.TH Thai th Language.TR Turkish tr Language.UK Ukrainian uk Language.UR Urdu ur Language.VI Vietnamese vi Language.CY Welsh cy OpenAI’s models support language variants (like en-US , fr-CA ) by mapping them to their base language. For example, Language.EN_US and Language.EN_GB will both map to en . The service will automatically detect the language if none is specified, but specifying the language typically improves transcription accuracy. ​ Usage Example Copy Ask AI from pipecat.services.openai.stt import OpenAISTTService # Configure service stt = OpenAISTTService( model = "gpt-4o-transcribe" , api_key = "your-api-key" , language = Language. EN , prompt = "Transcribe technical terms accurately. Format numbers as digits rather than words." ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ Voice Activity Detection Integration This service inherits from SegmentedSTTService , which uses Voice Activity Detection (VAD) to identify speech segments for processing. This approach: Processes only actual speech, not silence or background noise Maintains a small audio buffer (default 1 second) to capture speech that occurs slightly before VAD detection Receives UserStartedSpeakingFrame and UserStoppedSpeakingFrame from a VAD component in the pipeline Only sends complete utterances to the API when speech has ended Ensure your transport includes a VAD component (like SileroVADAnalyzer ) to properly detect speech segments. ​ Metrics Support The service collects the following metrics: Time to First Byte (TTFB) Processing duration API response time ​ Notes Requires valid OpenAI API key GPT-4o transcription model offers superior accuracy to Whisper Requires VAD component in transport Handles API rate limiting Automatic error handling Thread-safe processing ​ Error Handling The service handles common API errors including: Authentication errors Rate limiting Invalid audio format Network connectivity issues API timeouts Errors are propagated through ErrorFrames with descriptive messages. NVIDIA Riva SambaNova (Whisper) On this page Overview Installation Configuration Constructor Parameters Input Output Frames TranscriptionFrame ErrorFrame Methods Set Model Models Language Support Usage Example Voice Activity Detection Integration Metrics Support Notes Error Handling Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_openai_3bfdfa04.txt b/stt_openai_3bfdfa04.txt
new file mode 100644
index 0000000000000000000000000000000000000000..611e2ccbbecbfeb836f11efb30050fcfbbe5ddf4
--- /dev/null
+++ b/stt_openai_3bfdfa04.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/openai#param-language
+Title: OpenAI - Pipecat
+==================================================
+
+OpenAI - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text OpenAI Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview OpenAISTTService provides speech-to-text capabilities using OpenAI’s latest models, including the GPT-4o transcription model and the hosted Whisper API. It offers high-accuracy transcription with minimal setup requirements, using Voice Activity Detection (VAD) to process only speech segments. ​ Installation To use OpenAISTTService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[openai]" You’ll need to set up your OpenAI API key as an environment variable: OPENAI_API_KEY . You can obtain an OpenAI API key from the OpenAI platform . ​ Configuration ​ Constructor Parameters ​ model str default: "gpt-4o-transcribe" Model to use. Supported models include “gpt-4o-transcribe” (recommended) and “whisper-1”. ​ api_key str Your OpenAI API key. ​ base_url str Custom API base URL for OpenAI API requests. ​ language Language default: "Language.EN" Language of the audio input. Defaults to English. ​ prompt str Optional text to guide the model’s style or continue a previous segment. ​ temperature float Sampling temperature between 0 and 1. Lower values are more deterministic, higher values more creative. Defaults to 0.0. ​ sample_rate int Audio sample rate in Hz. If not provided, uses the pipeline’s sample rate. ​ Input The service processes audio data with the following requirements: PCM audio format 16-bit depth Single channel (mono) ​ Output Frames The service produces two types of frames during transcription: ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Detected language (if available) ​ ErrorFrame Generated when transcription errors occur, containing error details. ​ Methods ​ Set Model Copy Ask AI await service.set_model( "gpt-4o-transcribe" ) # For the latest GPT-4o transcription model # or await service.set_model( "whisper-1" ) # For the Whisper model See the STT base class methods for additional functionality. ​ Models Model Description Best For gpt-4o-transcribe Latest GPT-4o model fine-tuned for transcription High accuracy, robustness to accents, better context understanding whisper-1 OpenAI’s Whisper model Broad language support, good performance on clean audio ​ Language Support OpenAI’s speech-to-text models support a wide range of languages. The service automatically maps Language enum values to the appropriate language codes. Language Code Description Service Code Language.AF Afrikaans af Language.AR Arabic ar Language.HY Armenian hy Language.AZ Azerbaijani az Language.BE Belarusian be Language.BS Bosnian bs Language.BG Bulgarian bg Language.CA Catalan ca Language.ZH Chinese zh Language.HR Croatian hr Language.CS Czech cs Language.DA Danish da Language.NL Dutch nl Language.EN English en Language.ET Estonian et Language.FI Finnish fi Language.FR French fr Language.GL Galician gl Language.DE German de Language.EL Greek el Language.HE Hebrew he Language.HI Hindi hi Language.HU Hungarian hu Language.IS Icelandic is Language.ID Indonesian id Language.IT Italian it Language.JA Japanese ja Language.KN Kannada kn Language.KK Kazakh kk Language.KO Korean ko Language.LV Latvian lv Language.LT Lithuanian lt Language.MK Macedonian mk Language.MS Malay ms Language.MR Marathi mr Language.MI Maori mi Language.NE Nepali ne Language.NO Norwegian no Language.FA Persian fa Language.PL Polish pl Language.PT Portuguese pt Language.RO Romanian ro Language.RU Russian ru Language.SR Serbian sr Language.SK Slovak sk Language.SL Slovenian sl Language.ES Spanish es Language.SW Swahili sw Language.SV Swedish sv Language.TL Tagalog tl Language.TA Tamil ta Language.TH Thai th Language.TR Turkish tr Language.UK Ukrainian uk Language.UR Urdu ur Language.VI Vietnamese vi Language.CY Welsh cy OpenAI’s models support language variants (like en-US , fr-CA ) by mapping them to their base language. For example, Language.EN_US and Language.EN_GB will both map to en . The service will automatically detect the language if none is specified, but specifying the language typically improves transcription accuracy. ​ Usage Example Copy Ask AI from pipecat.services.openai.stt import OpenAISTTService # Configure service stt = OpenAISTTService( model = "gpt-4o-transcribe" , api_key = "your-api-key" , language = Language. EN , prompt = "Transcribe technical terms accurately. Format numbers as digits rather than words." ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ Voice Activity Detection Integration This service inherits from SegmentedSTTService , which uses Voice Activity Detection (VAD) to identify speech segments for processing. This approach: Processes only actual speech, not silence or background noise Maintains a small audio buffer (default 1 second) to capture speech that occurs slightly before VAD detection Receives UserStartedSpeakingFrame and UserStoppedSpeakingFrame from a VAD component in the pipeline Only sends complete utterances to the API when speech has ended Ensure your transport includes a VAD component (like SileroVADAnalyzer ) to properly detect speech segments. ​ Metrics Support The service collects the following metrics: Time to First Byte (TTFB) Processing duration API response time ​ Notes Requires valid OpenAI API key GPT-4o transcription model offers superior accuracy to Whisper Requires VAD component in transport Handles API rate limiting Automatic error handling Thread-safe processing ​ Error Handling The service handles common API errors including: Authentication errors Rate limiting Invalid audio format Network connectivity issues API timeouts Errors are propagated through ErrorFrames with descriptive messages. NVIDIA Riva SambaNova (Whisper) On this page Overview Installation Configuration Constructor Parameters Input Output Frames TranscriptionFrame ErrorFrame Methods Set Model Models Language Support Usage Example Voice Activity Detection Integration Metrics Support Notes Error Handling Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_openai_80c905eb.txt b/stt_openai_80c905eb.txt
new file mode 100644
index 0000000000000000000000000000000000000000..5531c87b78f2bc913bca30eaea718b186ff8493c
--- /dev/null
+++ b/stt_openai_80c905eb.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/openai#param-language-1
+Title: OpenAI - Pipecat
+==================================================
+
+OpenAI - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text OpenAI Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview OpenAISTTService provides speech-to-text capabilities using OpenAI’s latest models, including the GPT-4o transcription model and the hosted Whisper API. It offers high-accuracy transcription with minimal setup requirements, using Voice Activity Detection (VAD) to process only speech segments. ​ Installation To use OpenAISTTService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[openai]" You’ll need to set up your OpenAI API key as an environment variable: OPENAI_API_KEY . You can obtain an OpenAI API key from the OpenAI platform . ​ Configuration ​ Constructor Parameters ​ model str default: "gpt-4o-transcribe" Model to use. Supported models include “gpt-4o-transcribe” (recommended) and “whisper-1”. ​ api_key str Your OpenAI API key. ​ base_url str Custom API base URL for OpenAI API requests. ​ language Language default: "Language.EN" Language of the audio input. Defaults to English. ​ prompt str Optional text to guide the model’s style or continue a previous segment. ​ temperature float Sampling temperature between 0 and 1. Lower values are more deterministic, higher values more creative. Defaults to 0.0. ​ sample_rate int Audio sample rate in Hz. If not provided, uses the pipeline’s sample rate. ​ Input The service processes audio data with the following requirements: PCM audio format 16-bit depth Single channel (mono) ​ Output Frames The service produces two types of frames during transcription: ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Detected language (if available) ​ ErrorFrame Generated when transcription errors occur, containing error details. ​ Methods ​ Set Model Copy Ask AI await service.set_model( "gpt-4o-transcribe" ) # For the latest GPT-4o transcription model # or await service.set_model( "whisper-1" ) # For the Whisper model See the STT base class methods for additional functionality. ​ Models Model Description Best For gpt-4o-transcribe Latest GPT-4o model fine-tuned for transcription High accuracy, robustness to accents, better context understanding whisper-1 OpenAI’s Whisper model Broad language support, good performance on clean audio ​ Language Support OpenAI’s speech-to-text models support a wide range of languages. The service automatically maps Language enum values to the appropriate language codes. Language Code Description Service Code Language.AF Afrikaans af Language.AR Arabic ar Language.HY Armenian hy Language.AZ Azerbaijani az Language.BE Belarusian be Language.BS Bosnian bs Language.BG Bulgarian bg Language.CA Catalan ca Language.ZH Chinese zh Language.HR Croatian hr Language.CS Czech cs Language.DA Danish da Language.NL Dutch nl Language.EN English en Language.ET Estonian et Language.FI Finnish fi Language.FR French fr Language.GL Galician gl Language.DE German de Language.EL Greek el Language.HE Hebrew he Language.HI Hindi hi Language.HU Hungarian hu Language.IS Icelandic is Language.ID Indonesian id Language.IT Italian it Language.JA Japanese ja Language.KN Kannada kn Language.KK Kazakh kk Language.KO Korean ko Language.LV Latvian lv Language.LT Lithuanian lt Language.MK Macedonian mk Language.MS Malay ms Language.MR Marathi mr Language.MI Maori mi Language.NE Nepali ne Language.NO Norwegian no Language.FA Persian fa Language.PL Polish pl Language.PT Portuguese pt Language.RO Romanian ro Language.RU Russian ru Language.SR Serbian sr Language.SK Slovak sk Language.SL Slovenian sl Language.ES Spanish es Language.SW Swahili sw Language.SV Swedish sv Language.TL Tagalog tl Language.TA Tamil ta Language.TH Thai th Language.TR Turkish tr Language.UK Ukrainian uk Language.UR Urdu ur Language.VI Vietnamese vi Language.CY Welsh cy OpenAI’s models support language variants (like en-US , fr-CA ) by mapping them to their base language. For example, Language.EN_US and Language.EN_GB will both map to en . The service will automatically detect the language if none is specified, but specifying the language typically improves transcription accuracy. ​ Usage Example Copy Ask AI from pipecat.services.openai.stt import OpenAISTTService # Configure service stt = OpenAISTTService( model = "gpt-4o-transcribe" , api_key = "your-api-key" , language = Language. EN , prompt = "Transcribe technical terms accurately. Format numbers as digits rather than words." ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ Voice Activity Detection Integration This service inherits from SegmentedSTTService , which uses Voice Activity Detection (VAD) to identify speech segments for processing. This approach: Processes only actual speech, not silence or background noise Maintains a small audio buffer (default 1 second) to capture speech that occurs slightly before VAD detection Receives UserStartedSpeakingFrame and UserStoppedSpeakingFrame from a VAD component in the pipeline Only sends complete utterances to the API when speech has ended Ensure your transport includes a VAD component (like SileroVADAnalyzer ) to properly detect speech segments. ​ Metrics Support The service collects the following metrics: Time to First Byte (TTFB) Processing duration API response time ​ Notes Requires valid OpenAI API key GPT-4o transcription model offers superior accuracy to Whisper Requires VAD component in transport Handles API rate limiting Automatic error handling Thread-safe processing ​ Error Handling The service handles common API errors including: Authentication errors Rate limiting Invalid audio format Network connectivity issues API timeouts Errors are propagated through ErrorFrames with descriptive messages. NVIDIA Riva SambaNova (Whisper) On this page Overview Installation Configuration Constructor Parameters Input Output Frames TranscriptionFrame ErrorFrame Methods Set Model Models Language Support Usage Example Voice Activity Detection Integration Metrics Support Notes Error Handling Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_openai_9d62c1ca.txt b/stt_openai_9d62c1ca.txt
new file mode 100644
index 0000000000000000000000000000000000000000..a650064179452d26fa08bc240bf7a927db6e83ef
--- /dev/null
+++ b/stt_openai_9d62c1ca.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/openai#methods
+Title: OpenAI - Pipecat
+==================================================
+
+OpenAI - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text OpenAI Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview OpenAISTTService provides speech-to-text capabilities using OpenAI’s latest models, including the GPT-4o transcription model and the hosted Whisper API. It offers high-accuracy transcription with minimal setup requirements, using Voice Activity Detection (VAD) to process only speech segments. ​ Installation To use OpenAISTTService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[openai]" You’ll need to set up your OpenAI API key as an environment variable: OPENAI_API_KEY . You can obtain an OpenAI API key from the OpenAI platform . ​ Configuration ​ Constructor Parameters ​ model str default: "gpt-4o-transcribe" Model to use. Supported models include “gpt-4o-transcribe” (recommended) and “whisper-1”. ​ api_key str Your OpenAI API key. ​ base_url str Custom API base URL for OpenAI API requests. ​ language Language default: "Language.EN" Language of the audio input. Defaults to English. ​ prompt str Optional text to guide the model’s style or continue a previous segment. ​ temperature float Sampling temperature between 0 and 1. Lower values are more deterministic, higher values more creative. Defaults to 0.0. ​ sample_rate int Audio sample rate in Hz. If not provided, uses the pipeline’s sample rate. ​ Input The service processes audio data with the following requirements: PCM audio format 16-bit depth Single channel (mono) ​ Output Frames The service produces two types of frames during transcription: ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Detected language (if available) ​ ErrorFrame Generated when transcription errors occur, containing error details. ​ Methods ​ Set Model Copy Ask AI await service.set_model( "gpt-4o-transcribe" ) # For the latest GPT-4o transcription model # or await service.set_model( "whisper-1" ) # For the Whisper model See the STT base class methods for additional functionality. ​ Models Model Description Best For gpt-4o-transcribe Latest GPT-4o model fine-tuned for transcription High accuracy, robustness to accents, better context understanding whisper-1 OpenAI’s Whisper model Broad language support, good performance on clean audio ​ Language Support OpenAI’s speech-to-text models support a wide range of languages. The service automatically maps Language enum values to the appropriate language codes. Language Code Description Service Code Language.AF Afrikaans af Language.AR Arabic ar Language.HY Armenian hy Language.AZ Azerbaijani az Language.BE Belarusian be Language.BS Bosnian bs Language.BG Bulgarian bg Language.CA Catalan ca Language.ZH Chinese zh Language.HR Croatian hr Language.CS Czech cs Language.DA Danish da Language.NL Dutch nl Language.EN English en Language.ET Estonian et Language.FI Finnish fi Language.FR French fr Language.GL Galician gl Language.DE German de Language.EL Greek el Language.HE Hebrew he Language.HI Hindi hi Language.HU Hungarian hu Language.IS Icelandic is Language.ID Indonesian id Language.IT Italian it Language.JA Japanese ja Language.KN Kannada kn Language.KK Kazakh kk Language.KO Korean ko Language.LV Latvian lv Language.LT Lithuanian lt Language.MK Macedonian mk Language.MS Malay ms Language.MR Marathi mr Language.MI Maori mi Language.NE Nepali ne Language.NO Norwegian no Language.FA Persian fa Language.PL Polish pl Language.PT Portuguese pt Language.RO Romanian ro Language.RU Russian ru Language.SR Serbian sr Language.SK Slovak sk Language.SL Slovenian sl Language.ES Spanish es Language.SW Swahili sw Language.SV Swedish sv Language.TL Tagalog tl Language.TA Tamil ta Language.TH Thai th Language.TR Turkish tr Language.UK Ukrainian uk Language.UR Urdu ur Language.VI Vietnamese vi Language.CY Welsh cy OpenAI’s models support language variants (like en-US , fr-CA ) by mapping them to their base language. For example, Language.EN_US and Language.EN_GB will both map to en . The service will automatically detect the language if none is specified, but specifying the language typically improves transcription accuracy. ​ Usage Example Copy Ask AI from pipecat.services.openai.stt import OpenAISTTService # Configure service stt = OpenAISTTService( model = "gpt-4o-transcribe" , api_key = "your-api-key" , language = Language. EN , prompt = "Transcribe technical terms accurately. Format numbers as digits rather than words." ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ Voice Activity Detection Integration This service inherits from SegmentedSTTService , which uses Voice Activity Detection (VAD) to identify speech segments for processing. This approach: Processes only actual speech, not silence or background noise Maintains a small audio buffer (default 1 second) to capture speech that occurs slightly before VAD detection Receives UserStartedSpeakingFrame and UserStoppedSpeakingFrame from a VAD component in the pipeline Only sends complete utterances to the API when speech has ended Ensure your transport includes a VAD component (like SileroVADAnalyzer ) to properly detect speech segments. ​ Metrics Support The service collects the following metrics: Time to First Byte (TTFB) Processing duration API response time ​ Notes Requires valid OpenAI API key GPT-4o transcription model offers superior accuracy to Whisper Requires VAD component in transport Handles API rate limiting Automatic error handling Thread-safe processing ​ Error Handling The service handles common API errors including: Authentication errors Rate limiting Invalid audio format Network connectivity issues API timeouts Errors are propagated through ErrorFrames with descriptive messages. NVIDIA Riva SambaNova (Whisper) On this page Overview Installation Configuration Constructor Parameters Input Output Frames TranscriptionFrame ErrorFrame Methods Set Model Models Language Support Usage Example Voice Activity Detection Integration Metrics Support Notes Error Handling Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_openai_e1cae6e6.txt b/stt_openai_e1cae6e6.txt
new file mode 100644
index 0000000000000000000000000000000000000000..5816037a99fc0bc49b0d69647d749f1b164115e2
--- /dev/null
+++ b/stt_openai_e1cae6e6.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/openai#error-handling
+Title: OpenAI - Pipecat
+==================================================
+
+OpenAI - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text OpenAI Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview OpenAISTTService provides speech-to-text capabilities using OpenAI’s latest models, including the GPT-4o transcription model and the hosted Whisper API. It offers high-accuracy transcription with minimal setup requirements, using Voice Activity Detection (VAD) to process only speech segments. ​ Installation To use OpenAISTTService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[openai]" You’ll need to set up your OpenAI API key as an environment variable: OPENAI_API_KEY . You can obtain an OpenAI API key from the OpenAI platform . ​ Configuration ​ Constructor Parameters ​ model str default: "gpt-4o-transcribe" Model to use. Supported models include “gpt-4o-transcribe” (recommended) and “whisper-1”. ​ api_key str Your OpenAI API key. ​ base_url str Custom API base URL for OpenAI API requests. ​ language Language default: "Language.EN" Language of the audio input. Defaults to English. ​ prompt str Optional text to guide the model’s style or continue a previous segment. ​ temperature float Sampling temperature between 0 and 1. Lower values are more deterministic, higher values more creative. Defaults to 0.0. ​ sample_rate int Audio sample rate in Hz. If not provided, uses the pipeline’s sample rate. ​ Input The service processes audio data with the following requirements: PCM audio format 16-bit depth Single channel (mono) ​ Output Frames The service produces two types of frames during transcription: ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Detected language (if available) ​ ErrorFrame Generated when transcription errors occur, containing error details. ​ Methods ​ Set Model Copy Ask AI await service.set_model( "gpt-4o-transcribe" ) # For the latest GPT-4o transcription model # or await service.set_model( "whisper-1" ) # For the Whisper model See the STT base class methods for additional functionality. ​ Models Model Description Best For gpt-4o-transcribe Latest GPT-4o model fine-tuned for transcription High accuracy, robustness to accents, better context understanding whisper-1 OpenAI’s Whisper model Broad language support, good performance on clean audio ​ Language Support OpenAI’s speech-to-text models support a wide range of languages. The service automatically maps Language enum values to the appropriate language codes. Language Code Description Service Code Language.AF Afrikaans af Language.AR Arabic ar Language.HY Armenian hy Language.AZ Azerbaijani az Language.BE Belarusian be Language.BS Bosnian bs Language.BG Bulgarian bg Language.CA Catalan ca Language.ZH Chinese zh Language.HR Croatian hr Language.CS Czech cs Language.DA Danish da Language.NL Dutch nl Language.EN English en Language.ET Estonian et Language.FI Finnish fi Language.FR French fr Language.GL Galician gl Language.DE German de Language.EL Greek el Language.HE Hebrew he Language.HI Hindi hi Language.HU Hungarian hu Language.IS Icelandic is Language.ID Indonesian id Language.IT Italian it Language.JA Japanese ja Language.KN Kannada kn Language.KK Kazakh kk Language.KO Korean ko Language.LV Latvian lv Language.LT Lithuanian lt Language.MK Macedonian mk Language.MS Malay ms Language.MR Marathi mr Language.MI Maori mi Language.NE Nepali ne Language.NO Norwegian no Language.FA Persian fa Language.PL Polish pl Language.PT Portuguese pt Language.RO Romanian ro Language.RU Russian ru Language.SR Serbian sr Language.SK Slovak sk Language.SL Slovenian sl Language.ES Spanish es Language.SW Swahili sw Language.SV Swedish sv Language.TL Tagalog tl Language.TA Tamil ta Language.TH Thai th Language.TR Turkish tr Language.UK Ukrainian uk Language.UR Urdu ur Language.VI Vietnamese vi Language.CY Welsh cy OpenAI’s models support language variants (like en-US , fr-CA ) by mapping them to their base language. For example, Language.EN_US and Language.EN_GB will both map to en . The service will automatically detect the language if none is specified, but specifying the language typically improves transcription accuracy. ​ Usage Example Copy Ask AI from pipecat.services.openai.stt import OpenAISTTService # Configure service stt = OpenAISTTService( model = "gpt-4o-transcribe" , api_key = "your-api-key" , language = Language. EN , prompt = "Transcribe technical terms accurately. Format numbers as digits rather than words." ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ Voice Activity Detection Integration This service inherits from SegmentedSTTService , which uses Voice Activity Detection (VAD) to identify speech segments for processing. This approach: Processes only actual speech, not silence or background noise Maintains a small audio buffer (default 1 second) to capture speech that occurs slightly before VAD detection Receives UserStartedSpeakingFrame and UserStoppedSpeakingFrame from a VAD component in the pipeline Only sends complete utterances to the API when speech has ended Ensure your transport includes a VAD component (like SileroVADAnalyzer ) to properly detect speech segments. ​ Metrics Support The service collects the following metrics: Time to First Byte (TTFB) Processing duration API response time ​ Notes Requires valid OpenAI API key GPT-4o transcription model offers superior accuracy to Whisper Requires VAD component in transport Handles API rate limiting Automatic error handling Thread-safe processing ​ Error Handling The service handles common API errors including: Authentication errors Rate limiting Invalid audio format Network connectivity issues API timeouts Errors are propagated through ErrorFrames with descriptive messages. NVIDIA Riva SambaNova (Whisper) On this page Overview Installation Configuration Constructor Parameters Input Output Frames TranscriptionFrame ErrorFrame Methods Set Model Models Language Support Usage Example Voice Activity Detection Integration Metrics Support Notes Error Handling Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_openai_e67b23e9.txt b/stt_openai_e67b23e9.txt
new file mode 100644
index 0000000000000000000000000000000000000000..c1b5442eb85b5501076759732f6833ef0afc49a5
--- /dev/null
+++ b/stt_openai_e67b23e9.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/openai#configuration
+Title: OpenAI - Pipecat
+==================================================
+
+OpenAI - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text OpenAI Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview OpenAISTTService provides speech-to-text capabilities using OpenAI’s latest models, including the GPT-4o transcription model and the hosted Whisper API. It offers high-accuracy transcription with minimal setup requirements, using Voice Activity Detection (VAD) to process only speech segments. ​ Installation To use OpenAISTTService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[openai]" You’ll need to set up your OpenAI API key as an environment variable: OPENAI_API_KEY . You can obtain an OpenAI API key from the OpenAI platform . ​ Configuration ​ Constructor Parameters ​ model str default: "gpt-4o-transcribe" Model to use. Supported models include “gpt-4o-transcribe” (recommended) and “whisper-1”. ​ api_key str Your OpenAI API key. ​ base_url str Custom API base URL for OpenAI API requests. ​ language Language default: "Language.EN" Language of the audio input. Defaults to English. ​ prompt str Optional text to guide the model’s style or continue a previous segment. ​ temperature float Sampling temperature between 0 and 1. Lower values are more deterministic, higher values more creative. Defaults to 0.0. ​ sample_rate int Audio sample rate in Hz. If not provided, uses the pipeline’s sample rate. ​ Input The service processes audio data with the following requirements: PCM audio format 16-bit depth Single channel (mono) ​ Output Frames The service produces two types of frames during transcription: ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Detected language (if available) ​ ErrorFrame Generated when transcription errors occur, containing error details. ​ Methods ​ Set Model Copy Ask AI await service.set_model( "gpt-4o-transcribe" ) # For the latest GPT-4o transcription model # or await service.set_model( "whisper-1" ) # For the Whisper model See the STT base class methods for additional functionality. ​ Models Model Description Best For gpt-4o-transcribe Latest GPT-4o model fine-tuned for transcription High accuracy, robustness to accents, better context understanding whisper-1 OpenAI’s Whisper model Broad language support, good performance on clean audio ​ Language Support OpenAI’s speech-to-text models support a wide range of languages. The service automatically maps Language enum values to the appropriate language codes. Language Code Description Service Code Language.AF Afrikaans af Language.AR Arabic ar Language.HY Armenian hy Language.AZ Azerbaijani az Language.BE Belarusian be Language.BS Bosnian bs Language.BG Bulgarian bg Language.CA Catalan ca Language.ZH Chinese zh Language.HR Croatian hr Language.CS Czech cs Language.DA Danish da Language.NL Dutch nl Language.EN English en Language.ET Estonian et Language.FI Finnish fi Language.FR French fr Language.GL Galician gl Language.DE German de Language.EL Greek el Language.HE Hebrew he Language.HI Hindi hi Language.HU Hungarian hu Language.IS Icelandic is Language.ID Indonesian id Language.IT Italian it Language.JA Japanese ja Language.KN Kannada kn Language.KK Kazakh kk Language.KO Korean ko Language.LV Latvian lv Language.LT Lithuanian lt Language.MK Macedonian mk Language.MS Malay ms Language.MR Marathi mr Language.MI Maori mi Language.NE Nepali ne Language.NO Norwegian no Language.FA Persian fa Language.PL Polish pl Language.PT Portuguese pt Language.RO Romanian ro Language.RU Russian ru Language.SR Serbian sr Language.SK Slovak sk Language.SL Slovenian sl Language.ES Spanish es Language.SW Swahili sw Language.SV Swedish sv Language.TL Tagalog tl Language.TA Tamil ta Language.TH Thai th Language.TR Turkish tr Language.UK Ukrainian uk Language.UR Urdu ur Language.VI Vietnamese vi Language.CY Welsh cy OpenAI’s models support language variants (like en-US , fr-CA ) by mapping them to their base language. For example, Language.EN_US and Language.EN_GB will both map to en . The service will automatically detect the language if none is specified, but specifying the language typically improves transcription accuracy. ​ Usage Example Copy Ask AI from pipecat.services.openai.stt import OpenAISTTService # Configure service stt = OpenAISTTService( model = "gpt-4o-transcribe" , api_key = "your-api-key" , language = Language. EN , prompt = "Transcribe technical terms accurately. Format numbers as digits rather than words." ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ Voice Activity Detection Integration This service inherits from SegmentedSTTService , which uses Voice Activity Detection (VAD) to identify speech segments for processing. This approach: Processes only actual speech, not silence or background noise Maintains a small audio buffer (default 1 second) to capture speech that occurs slightly before VAD detection Receives UserStartedSpeakingFrame and UserStoppedSpeakingFrame from a VAD component in the pipeline Only sends complete utterances to the API when speech has ended Ensure your transport includes a VAD component (like SileroVADAnalyzer ) to properly detect speech segments. ​ Metrics Support The service collects the following metrics: Time to First Byte (TTFB) Processing duration API response time ​ Notes Requires valid OpenAI API key GPT-4o transcription model offers superior accuracy to Whisper Requires VAD component in transport Handles API rate limiting Automatic error handling Thread-safe processing ​ Error Handling The service handles common API errors including: Authentication errors Rate limiting Invalid audio format Network connectivity issues API timeouts Errors are propagated through ErrorFrames with descriptive messages. NVIDIA Riva SambaNova (Whisper) On this page Overview Installation Configuration Constructor Parameters Input Output Frames TranscriptionFrame ErrorFrame Methods Set Model Models Language Support Usage Example Voice Activity Detection Integration Metrics Support Notes Error Handling Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_openai_ee937c1b.txt b/stt_openai_ee937c1b.txt
new file mode 100644
index 0000000000000000000000000000000000000000..3809d06dfb5f7bcc00b7657019c3cb78adf241de
--- /dev/null
+++ b/stt_openai_ee937c1b.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/openai#errorframe
+Title: OpenAI - Pipecat
+==================================================
+
+OpenAI - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text OpenAI Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview OpenAISTTService provides speech-to-text capabilities using OpenAI’s latest models, including the GPT-4o transcription model and the hosted Whisper API. It offers high-accuracy transcription with minimal setup requirements, using Voice Activity Detection (VAD) to process only speech segments. ​ Installation To use OpenAISTTService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[openai]" You’ll need to set up your OpenAI API key as an environment variable: OPENAI_API_KEY . You can obtain an OpenAI API key from the OpenAI platform . ​ Configuration ​ Constructor Parameters ​ model str default: "gpt-4o-transcribe" Model to use. Supported models include “gpt-4o-transcribe” (recommended) and “whisper-1”. ​ api_key str Your OpenAI API key. ​ base_url str Custom API base URL for OpenAI API requests. ​ language Language default: "Language.EN" Language of the audio input. Defaults to English. ​ prompt str Optional text to guide the model’s style or continue a previous segment. ​ temperature float Sampling temperature between 0 and 1. Lower values are more deterministic, higher values more creative. Defaults to 0.0. ​ sample_rate int Audio sample rate in Hz. If not provided, uses the pipeline’s sample rate. ​ Input The service processes audio data with the following requirements: PCM audio format 16-bit depth Single channel (mono) ​ Output Frames The service produces two types of frames during transcription: ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Detected language (if available) ​ ErrorFrame Generated when transcription errors occur, containing error details. ​ Methods ​ Set Model Copy Ask AI await service.set_model( "gpt-4o-transcribe" ) # For the latest GPT-4o transcription model # or await service.set_model( "whisper-1" ) # For the Whisper model See the STT base class methods for additional functionality. ​ Models Model Description Best For gpt-4o-transcribe Latest GPT-4o model fine-tuned for transcription High accuracy, robustness to accents, better context understanding whisper-1 OpenAI’s Whisper model Broad language support, good performance on clean audio ​ Language Support OpenAI’s speech-to-text models support a wide range of languages. The service automatically maps Language enum values to the appropriate language codes. Language Code Description Service Code Language.AF Afrikaans af Language.AR Arabic ar Language.HY Armenian hy Language.AZ Azerbaijani az Language.BE Belarusian be Language.BS Bosnian bs Language.BG Bulgarian bg Language.CA Catalan ca Language.ZH Chinese zh Language.HR Croatian hr Language.CS Czech cs Language.DA Danish da Language.NL Dutch nl Language.EN English en Language.ET Estonian et Language.FI Finnish fi Language.FR French fr Language.GL Galician gl Language.DE German de Language.EL Greek el Language.HE Hebrew he Language.HI Hindi hi Language.HU Hungarian hu Language.IS Icelandic is Language.ID Indonesian id Language.IT Italian it Language.JA Japanese ja Language.KN Kannada kn Language.KK Kazakh kk Language.KO Korean ko Language.LV Latvian lv Language.LT Lithuanian lt Language.MK Macedonian mk Language.MS Malay ms Language.MR Marathi mr Language.MI Maori mi Language.NE Nepali ne Language.NO Norwegian no Language.FA Persian fa Language.PL Polish pl Language.PT Portuguese pt Language.RO Romanian ro Language.RU Russian ru Language.SR Serbian sr Language.SK Slovak sk Language.SL Slovenian sl Language.ES Spanish es Language.SW Swahili sw Language.SV Swedish sv Language.TL Tagalog tl Language.TA Tamil ta Language.TH Thai th Language.TR Turkish tr Language.UK Ukrainian uk Language.UR Urdu ur Language.VI Vietnamese vi Language.CY Welsh cy OpenAI’s models support language variants (like en-US , fr-CA ) by mapping them to their base language. For example, Language.EN_US and Language.EN_GB will both map to en . The service will automatically detect the language if none is specified, but specifying the language typically improves transcription accuracy. ​ Usage Example Copy Ask AI from pipecat.services.openai.stt import OpenAISTTService # Configure service stt = OpenAISTTService( model = "gpt-4o-transcribe" , api_key = "your-api-key" , language = Language. EN , prompt = "Transcribe technical terms accurately. Format numbers as digits rather than words." ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ Voice Activity Detection Integration This service inherits from SegmentedSTTService , which uses Voice Activity Detection (VAD) to identify speech segments for processing. This approach: Processes only actual speech, not silence or background noise Maintains a small audio buffer (default 1 second) to capture speech that occurs slightly before VAD detection Receives UserStartedSpeakingFrame and UserStoppedSpeakingFrame from a VAD component in the pipeline Only sends complete utterances to the API when speech has ended Ensure your transport includes a VAD component (like SileroVADAnalyzer ) to properly detect speech segments. ​ Metrics Support The service collects the following metrics: Time to First Byte (TTFB) Processing duration API response time ​ Notes Requires valid OpenAI API key GPT-4o transcription model offers superior accuracy to Whisper Requires VAD component in transport Handles API rate limiting Automatic error handling Thread-safe processing ​ Error Handling The service handles common API errors including: Authentication errors Rate limiting Invalid audio format Network connectivity issues API timeouts Errors are propagated through ErrorFrames with descriptive messages. NVIDIA Riva SambaNova (Whisper) On this page Overview Installation Configuration Constructor Parameters Input Output Frames TranscriptionFrame ErrorFrame Methods Set Model Models Language Support Usage Example Voice Activity Detection Integration Metrics Support Notes Error Handling Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_riva_0f6d62e6.txt b/stt_riva_0f6d62e6.txt
new file mode 100644
index 0000000000000000000000000000000000000000..6d72e1cbaff559bcda171c128611d1028d1ac67e
--- /dev/null
+++ b/stt_riva_0f6d62e6.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/riva#overview
+Title: NVIDIA Riva - Pipecat
+==================================================
+
+NVIDIA Riva - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text NVIDIA Riva Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview RivaSTTService provides real-time speech-to-text capabilities using NVIDIA’s Riva Parakeet model. It supports interim results and configurable recognition parameters for enhanced accuracy. RivaSegmentedSTTService provides speech-to-text capabilities via NVIDIA’s Riva Canary model. ​ Installation To use RivaSTTService or RivaSegmentedSTTService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[riva]" You’ll also need to set up your NVIDIA API key as an environment variable: NVIDIA_API_KEY . You can obtain an NVIDIA API key by signing up through NVIDIA’s developer portal . ​ RivaSTTService ​ Configuration ​ api_key str required Your NVIDIA API key ​ server str default: "grpc.nvcf.nvidia.com:443" NVIDIA Riva server address ​ model_function_map Mapping [str, str] A mapping of the NVIDIA function identifier for the STT service with the model name. ​ sample_rate int default: "None" Audio sample rate in Hz ​ params InputParams default: "InputParams()" Additional configuration parameters ​ InputParams ​ language Language default: "Language.EN_US" The language for speech recognition ​ Input The service processes audio frames containing: Raw PCM audio data 16-bit depth Single channel (mono) ​ Output Frames ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Language used for transcription ​ InterimTranscriptionFrame Generated during ongoing speech, containing same fields as TranscriptionFrame but with preliminary results. ​ RivaSegmentedSTTService ​ Configuration ​ api_key str required Your NVIDIA API key ​ server str default: "grpc.nvcf.nvidia.com:443" NVIDIA Riva server address ​ model_function_map Mapping [str, str] A mapping of the NVIDIA function identifier for the STT service with the model name. ​ sample_rate int default: "None" Audio sample rate in Hz ​ params InputParams default: "InputParams()" Additional configuration parameters ​ InputParams ​ language Language default: "Language.EN_US" The language for speech recognition ​ Input The service processes audio frames containing: Raw audio bytes in WAV format ​ Output Frames ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Language used for transcription ​ InterimTranscriptionFrame Generated during ongoing speech, containing same fields as TranscriptionFrame but with preliminary results. ​ Methods See the STT base class methods for additional functionality. ​ Models Model Pipecat Class Model Card Link parakeet-ctc-1.1b-asr RivaSTTService NVIDIA Model Card canary-1b-asr RivaSegmentedSTTService NVIDIA Model Card ​ Usage Examples ​ RivaSTTService Copy Ask AI from pipecat.services.riva.stt import RivaSTTService from pipecat.transcriptions.language import Language # Configure service stt = RivaSTTService( api_key = "your-nvidia-api-key" , params = RivaSTTService.InputParams( language = Language. EN_US ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ RivaSegmentedSTTService Copy Ask AI from pipecat.services.riva.stt import RivaSegmentedSTTService from pipecat.transcriptions.language import Language # Configure service stt = RivaSegmentedSTTService( api_key = "your-nvidia-api-key" , params = RivaSegmentedSTTService.InputParams( language = Language. EN_US ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ Language Support Riva model parakeet-ctc-1.1b-asr (default) primarily supports English with various regional accents: Language Code Description Service Codes Language.EN_US English (US) en-US ​ Frame Flow ​ Advanced Configuration The service supports several advanced configuration options that can be adjusted: ​ _profanity_filter bool default: "False" Filter profanity from transcription ​ _automatic_punctuation bool default: "False" Automatically add punctuation ​ _no_verbatim_transcripts bool default: "False" Whether to disable verbatim transcripts ​ _boosted_lm_words list default: "None" List of words to boost in the language model ​ _boosted_lm_score float default: "4.0" Score applied to boosted words ​ Example with Advanced Configuration Copy Ask AI # Configure service with advanced parameters stt = RivaSTTService( api_key = "your-nvidia-api-key" , params = RivaSTTService.InputParams( language = Language. EN_US ) ) # Configure advanced options stt._profanity_filter = True stt._automatic_punctuation = True stt._boosted_lm_words = [ "PipeCat" , "AI" , "speech" ] ​ Notes Uses NVIDIA’s Riva AI Services platform Handles streaming audio input Provides real-time transcription results Manages connection lifecycle Uses asyncio for asynchronous processing Automatically cleans up resources on stop/cancel Groq (Whisper) OpenAI On this page Overview Installation RivaSTTService Configuration InputParams Input Output Frames TranscriptionFrame InterimTranscriptionFrame RivaSegmentedSTTService Configuration InputParams Input Output Frames TranscriptionFrame InterimTranscriptionFrame Methods Models Usage Examples RivaSTTService RivaSegmentedSTTService Language Support Frame Flow Advanced Configuration Example with Advanced Configuration Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_riva_1ab753f3.txt b/stt_riva_1ab753f3.txt
new file mode 100644
index 0000000000000000000000000000000000000000..d5d4a092b50757abe98bc09b0ce2d7f6a32d5560
--- /dev/null
+++ b/stt_riva_1ab753f3.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/riva#param-language-1
+Title: NVIDIA Riva - Pipecat
+==================================================
+
+NVIDIA Riva - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text NVIDIA Riva Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview RivaSTTService provides real-time speech-to-text capabilities using NVIDIA’s Riva Parakeet model. It supports interim results and configurable recognition parameters for enhanced accuracy. RivaSegmentedSTTService provides speech-to-text capabilities via NVIDIA’s Riva Canary model. ​ Installation To use RivaSTTService or RivaSegmentedSTTService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[riva]" You’ll also need to set up your NVIDIA API key as an environment variable: NVIDIA_API_KEY . You can obtain an NVIDIA API key by signing up through NVIDIA’s developer portal . ​ RivaSTTService ​ Configuration ​ api_key str required Your NVIDIA API key ​ server str default: "grpc.nvcf.nvidia.com:443" NVIDIA Riva server address ​ model_function_map Mapping [str, str] A mapping of the NVIDIA function identifier for the STT service with the model name. ​ sample_rate int default: "None" Audio sample rate in Hz ​ params InputParams default: "InputParams()" Additional configuration parameters ​ InputParams ​ language Language default: "Language.EN_US" The language for speech recognition ​ Input The service processes audio frames containing: Raw PCM audio data 16-bit depth Single channel (mono) ​ Output Frames ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Language used for transcription ​ InterimTranscriptionFrame Generated during ongoing speech, containing same fields as TranscriptionFrame but with preliminary results. ​ RivaSegmentedSTTService ​ Configuration ​ api_key str required Your NVIDIA API key ​ server str default: "grpc.nvcf.nvidia.com:443" NVIDIA Riva server address ​ model_function_map Mapping [str, str] A mapping of the NVIDIA function identifier for the STT service with the model name. ​ sample_rate int default: "None" Audio sample rate in Hz ​ params InputParams default: "InputParams()" Additional configuration parameters ​ InputParams ​ language Language default: "Language.EN_US" The language for speech recognition ​ Input The service processes audio frames containing: Raw audio bytes in WAV format ​ Output Frames ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Language used for transcription ​ InterimTranscriptionFrame Generated during ongoing speech, containing same fields as TranscriptionFrame but with preliminary results. ​ Methods See the STT base class methods for additional functionality. ​ Models Model Pipecat Class Model Card Link parakeet-ctc-1.1b-asr RivaSTTService NVIDIA Model Card canary-1b-asr RivaSegmentedSTTService NVIDIA Model Card ​ Usage Examples ​ RivaSTTService Copy Ask AI from pipecat.services.riva.stt import RivaSTTService from pipecat.transcriptions.language import Language # Configure service stt = RivaSTTService( api_key = "your-nvidia-api-key" , params = RivaSTTService.InputParams( language = Language. EN_US ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ RivaSegmentedSTTService Copy Ask AI from pipecat.services.riva.stt import RivaSegmentedSTTService from pipecat.transcriptions.language import Language # Configure service stt = RivaSegmentedSTTService( api_key = "your-nvidia-api-key" , params = RivaSegmentedSTTService.InputParams( language = Language. EN_US ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ Language Support Riva model parakeet-ctc-1.1b-asr (default) primarily supports English with various regional accents: Language Code Description Service Codes Language.EN_US English (US) en-US ​ Frame Flow ​ Advanced Configuration The service supports several advanced configuration options that can be adjusted: ​ _profanity_filter bool default: "False" Filter profanity from transcription ​ _automatic_punctuation bool default: "False" Automatically add punctuation ​ _no_verbatim_transcripts bool default: "False" Whether to disable verbatim transcripts ​ _boosted_lm_words list default: "None" List of words to boost in the language model ​ _boosted_lm_score float default: "4.0" Score applied to boosted words ​ Example with Advanced Configuration Copy Ask AI # Configure service with advanced parameters stt = RivaSTTService( api_key = "your-nvidia-api-key" , params = RivaSTTService.InputParams( language = Language. EN_US ) ) # Configure advanced options stt._profanity_filter = True stt._automatic_punctuation = True stt._boosted_lm_words = [ "PipeCat" , "AI" , "speech" ] ​ Notes Uses NVIDIA’s Riva AI Services platform Handles streaming audio input Provides real-time transcription results Manages connection lifecycle Uses asyncio for asynchronous processing Automatically cleans up resources on stop/cancel Groq (Whisper) OpenAI On this page Overview Installation RivaSTTService Configuration InputParams Input Output Frames TranscriptionFrame InterimTranscriptionFrame RivaSegmentedSTTService Configuration InputParams Input Output Frames TranscriptionFrame InterimTranscriptionFrame Methods Models Usage Examples RivaSTTService RivaSegmentedSTTService Language Support Frame Flow Advanced Configuration Example with Advanced Configuration Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_riva_3a0a5211.txt b/stt_riva_3a0a5211.txt
new file mode 100644
index 0000000000000000000000000000000000000000..f1c8e0a3daa86a751fbdec284a4868e81e2cea73
--- /dev/null
+++ b/stt_riva_3a0a5211.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/riva#param-server
+Title: NVIDIA Riva - Pipecat
+==================================================
+
+NVIDIA Riva - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text NVIDIA Riva Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview RivaSTTService provides real-time speech-to-text capabilities using NVIDIA’s Riva Parakeet model. It supports interim results and configurable recognition parameters for enhanced accuracy. RivaSegmentedSTTService provides speech-to-text capabilities via NVIDIA’s Riva Canary model. ​ Installation To use RivaSTTService or RivaSegmentedSTTService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[riva]" You’ll also need to set up your NVIDIA API key as an environment variable: NVIDIA_API_KEY . You can obtain an NVIDIA API key by signing up through NVIDIA’s developer portal . ​ RivaSTTService ​ Configuration ​ api_key str required Your NVIDIA API key ​ server str default: "grpc.nvcf.nvidia.com:443" NVIDIA Riva server address ​ model_function_map Mapping [str, str] A mapping of the NVIDIA function identifier for the STT service with the model name. ​ sample_rate int default: "None" Audio sample rate in Hz ​ params InputParams default: "InputParams()" Additional configuration parameters ​ InputParams ​ language Language default: "Language.EN_US" The language for speech recognition ​ Input The service processes audio frames containing: Raw PCM audio data 16-bit depth Single channel (mono) ​ Output Frames ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Language used for transcription ​ InterimTranscriptionFrame Generated during ongoing speech, containing same fields as TranscriptionFrame but with preliminary results. ​ RivaSegmentedSTTService ​ Configuration ​ api_key str required Your NVIDIA API key ​ server str default: "grpc.nvcf.nvidia.com:443" NVIDIA Riva server address ​ model_function_map Mapping [str, str] A mapping of the NVIDIA function identifier for the STT service with the model name. ​ sample_rate int default: "None" Audio sample rate in Hz ​ params InputParams default: "InputParams()" Additional configuration parameters ​ InputParams ​ language Language default: "Language.EN_US" The language for speech recognition ​ Input The service processes audio frames containing: Raw audio bytes in WAV format ​ Output Frames ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Language used for transcription ​ InterimTranscriptionFrame Generated during ongoing speech, containing same fields as TranscriptionFrame but with preliminary results. ​ Methods See the STT base class methods for additional functionality. ​ Models Model Pipecat Class Model Card Link parakeet-ctc-1.1b-asr RivaSTTService NVIDIA Model Card canary-1b-asr RivaSegmentedSTTService NVIDIA Model Card ​ Usage Examples ​ RivaSTTService Copy Ask AI from pipecat.services.riva.stt import RivaSTTService from pipecat.transcriptions.language import Language # Configure service stt = RivaSTTService( api_key = "your-nvidia-api-key" , params = RivaSTTService.InputParams( language = Language. EN_US ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ RivaSegmentedSTTService Copy Ask AI from pipecat.services.riva.stt import RivaSegmentedSTTService from pipecat.transcriptions.language import Language # Configure service stt = RivaSegmentedSTTService( api_key = "your-nvidia-api-key" , params = RivaSegmentedSTTService.InputParams( language = Language. EN_US ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ Language Support Riva model parakeet-ctc-1.1b-asr (default) primarily supports English with various regional accents: Language Code Description Service Codes Language.EN_US English (US) en-US ​ Frame Flow ​ Advanced Configuration The service supports several advanced configuration options that can be adjusted: ​ _profanity_filter bool default: "False" Filter profanity from transcription ​ _automatic_punctuation bool default: "False" Automatically add punctuation ​ _no_verbatim_transcripts bool default: "False" Whether to disable verbatim transcripts ​ _boosted_lm_words list default: "None" List of words to boost in the language model ​ _boosted_lm_score float default: "4.0" Score applied to boosted words ​ Example with Advanced Configuration Copy Ask AI # Configure service with advanced parameters stt = RivaSTTService( api_key = "your-nvidia-api-key" , params = RivaSTTService.InputParams( language = Language. EN_US ) ) # Configure advanced options stt._profanity_filter = True stt._automatic_punctuation = True stt._boosted_lm_words = [ "PipeCat" , "AI" , "speech" ] ​ Notes Uses NVIDIA’s Riva AI Services platform Handles streaming audio input Provides real-time transcription results Manages connection lifecycle Uses asyncio for asynchronous processing Automatically cleans up resources on stop/cancel Groq (Whisper) OpenAI On this page Overview Installation RivaSTTService Configuration InputParams Input Output Frames TranscriptionFrame InterimTranscriptionFrame RivaSegmentedSTTService Configuration InputParams Input Output Frames TranscriptionFrame InterimTranscriptionFrame Methods Models Usage Examples RivaSTTService RivaSegmentedSTTService Language Support Frame Flow Advanced Configuration Example with Advanced Configuration Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_riva_4057e242.txt b/stt_riva_4057e242.txt
new file mode 100644
index 0000000000000000000000000000000000000000..967e92483e27cef8d65c62df5a33c8534264253b
--- /dev/null
+++ b/stt_riva_4057e242.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/riva#models
+Title: NVIDIA Riva - Pipecat
+==================================================
+
+NVIDIA Riva - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text NVIDIA Riva Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview RivaSTTService provides real-time speech-to-text capabilities using NVIDIA’s Riva Parakeet model. It supports interim results and configurable recognition parameters for enhanced accuracy. RivaSegmentedSTTService provides speech-to-text capabilities via NVIDIA’s Riva Canary model. ​ Installation To use RivaSTTService or RivaSegmentedSTTService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[riva]" You’ll also need to set up your NVIDIA API key as an environment variable: NVIDIA_API_KEY . You can obtain an NVIDIA API key by signing up through NVIDIA’s developer portal . ​ RivaSTTService ​ Configuration ​ api_key str required Your NVIDIA API key ​ server str default: "grpc.nvcf.nvidia.com:443" NVIDIA Riva server address ​ model_function_map Mapping [str, str] A mapping of the NVIDIA function identifier for the STT service with the model name. ​ sample_rate int default: "None" Audio sample rate in Hz ​ params InputParams default: "InputParams()" Additional configuration parameters ​ InputParams ​ language Language default: "Language.EN_US" The language for speech recognition ​ Input The service processes audio frames containing: Raw PCM audio data 16-bit depth Single channel (mono) ​ Output Frames ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Language used for transcription ​ InterimTranscriptionFrame Generated during ongoing speech, containing same fields as TranscriptionFrame but with preliminary results. ​ RivaSegmentedSTTService ​ Configuration ​ api_key str required Your NVIDIA API key ​ server str default: "grpc.nvcf.nvidia.com:443" NVIDIA Riva server address ​ model_function_map Mapping [str, str] A mapping of the NVIDIA function identifier for the STT service with the model name. ​ sample_rate int default: "None" Audio sample rate in Hz ​ params InputParams default: "InputParams()" Additional configuration parameters ​ InputParams ​ language Language default: "Language.EN_US" The language for speech recognition ​ Input The service processes audio frames containing: Raw audio bytes in WAV format ​ Output Frames ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Language used for transcription ​ InterimTranscriptionFrame Generated during ongoing speech, containing same fields as TranscriptionFrame but with preliminary results. ​ Methods See the STT base class methods for additional functionality. ​ Models Model Pipecat Class Model Card Link parakeet-ctc-1.1b-asr RivaSTTService NVIDIA Model Card canary-1b-asr RivaSegmentedSTTService NVIDIA Model Card ​ Usage Examples ​ RivaSTTService Copy Ask AI from pipecat.services.riva.stt import RivaSTTService from pipecat.transcriptions.language import Language # Configure service stt = RivaSTTService( api_key = "your-nvidia-api-key" , params = RivaSTTService.InputParams( language = Language. EN_US ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ RivaSegmentedSTTService Copy Ask AI from pipecat.services.riva.stt import RivaSegmentedSTTService from pipecat.transcriptions.language import Language # Configure service stt = RivaSegmentedSTTService( api_key = "your-nvidia-api-key" , params = RivaSegmentedSTTService.InputParams( language = Language. EN_US ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ Language Support Riva model parakeet-ctc-1.1b-asr (default) primarily supports English with various regional accents: Language Code Description Service Codes Language.EN_US English (US) en-US ​ Frame Flow ​ Advanced Configuration The service supports several advanced configuration options that can be adjusted: ​ _profanity_filter bool default: "False" Filter profanity from transcription ​ _automatic_punctuation bool default: "False" Automatically add punctuation ​ _no_verbatim_transcripts bool default: "False" Whether to disable verbatim transcripts ​ _boosted_lm_words list default: "None" List of words to boost in the language model ​ _boosted_lm_score float default: "4.0" Score applied to boosted words ​ Example with Advanced Configuration Copy Ask AI # Configure service with advanced parameters stt = RivaSTTService( api_key = "your-nvidia-api-key" , params = RivaSTTService.InputParams( language = Language. EN_US ) ) # Configure advanced options stt._profanity_filter = True stt._automatic_punctuation = True stt._boosted_lm_words = [ "PipeCat" , "AI" , "speech" ] ​ Notes Uses NVIDIA’s Riva AI Services platform Handles streaming audio input Provides real-time transcription results Manages connection lifecycle Uses asyncio for asynchronous processing Automatically cleans up resources on stop/cancel Groq (Whisper) OpenAI On this page Overview Installation RivaSTTService Configuration InputParams Input Output Frames TranscriptionFrame InterimTranscriptionFrame RivaSegmentedSTTService Configuration InputParams Input Output Frames TranscriptionFrame InterimTranscriptionFrame Methods Models Usage Examples RivaSTTService RivaSegmentedSTTService Language Support Frame Flow Advanced Configuration Example with Advanced Configuration Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_riva_4c104bf4.txt b/stt_riva_4c104bf4.txt
new file mode 100644
index 0000000000000000000000000000000000000000..43afb3a5dc4fb95195e8640676fadc107becf9de
--- /dev/null
+++ b/stt_riva_4c104bf4.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/riva#param-language-3
+Title: NVIDIA Riva - Pipecat
+==================================================
+
+NVIDIA Riva - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text NVIDIA Riva Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview RivaSTTService provides real-time speech-to-text capabilities using NVIDIA’s Riva Parakeet model. It supports interim results and configurable recognition parameters for enhanced accuracy. RivaSegmentedSTTService provides speech-to-text capabilities via NVIDIA’s Riva Canary model. ​ Installation To use RivaSTTService or RivaSegmentedSTTService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[riva]" You’ll also need to set up your NVIDIA API key as an environment variable: NVIDIA_API_KEY . You can obtain an NVIDIA API key by signing up through NVIDIA’s developer portal . ​ RivaSTTService ​ Configuration ​ api_key str required Your NVIDIA API key ​ server str default: "grpc.nvcf.nvidia.com:443" NVIDIA Riva server address ​ model_function_map Mapping [str, str] A mapping of the NVIDIA function identifier for the STT service with the model name. ​ sample_rate int default: "None" Audio sample rate in Hz ​ params InputParams default: "InputParams()" Additional configuration parameters ​ InputParams ​ language Language default: "Language.EN_US" The language for speech recognition ​ Input The service processes audio frames containing: Raw PCM audio data 16-bit depth Single channel (mono) ​ Output Frames ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Language used for transcription ​ InterimTranscriptionFrame Generated during ongoing speech, containing same fields as TranscriptionFrame but with preliminary results. ​ RivaSegmentedSTTService ​ Configuration ​ api_key str required Your NVIDIA API key ​ server str default: "grpc.nvcf.nvidia.com:443" NVIDIA Riva server address ​ model_function_map Mapping [str, str] A mapping of the NVIDIA function identifier for the STT service with the model name. ​ sample_rate int default: "None" Audio sample rate in Hz ​ params InputParams default: "InputParams()" Additional configuration parameters ​ InputParams ​ language Language default: "Language.EN_US" The language for speech recognition ​ Input The service processes audio frames containing: Raw audio bytes in WAV format ​ Output Frames ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Language used for transcription ​ InterimTranscriptionFrame Generated during ongoing speech, containing same fields as TranscriptionFrame but with preliminary results. ​ Methods See the STT base class methods for additional functionality. ​ Models Model Pipecat Class Model Card Link parakeet-ctc-1.1b-asr RivaSTTService NVIDIA Model Card canary-1b-asr RivaSegmentedSTTService NVIDIA Model Card ​ Usage Examples ​ RivaSTTService Copy Ask AI from pipecat.services.riva.stt import RivaSTTService from pipecat.transcriptions.language import Language # Configure service stt = RivaSTTService( api_key = "your-nvidia-api-key" , params = RivaSTTService.InputParams( language = Language. EN_US ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ RivaSegmentedSTTService Copy Ask AI from pipecat.services.riva.stt import RivaSegmentedSTTService from pipecat.transcriptions.language import Language # Configure service stt = RivaSegmentedSTTService( api_key = "your-nvidia-api-key" , params = RivaSegmentedSTTService.InputParams( language = Language. EN_US ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ Language Support Riva model parakeet-ctc-1.1b-asr (default) primarily supports English with various regional accents: Language Code Description Service Codes Language.EN_US English (US) en-US ​ Frame Flow ​ Advanced Configuration The service supports several advanced configuration options that can be adjusted: ​ _profanity_filter bool default: "False" Filter profanity from transcription ​ _automatic_punctuation bool default: "False" Automatically add punctuation ​ _no_verbatim_transcripts bool default: "False" Whether to disable verbatim transcripts ​ _boosted_lm_words list default: "None" List of words to boost in the language model ​ _boosted_lm_score float default: "4.0" Score applied to boosted words ​ Example with Advanced Configuration Copy Ask AI # Configure service with advanced parameters stt = RivaSTTService( api_key = "your-nvidia-api-key" , params = RivaSTTService.InputParams( language = Language. EN_US ) ) # Configure advanced options stt._profanity_filter = True stt._automatic_punctuation = True stt._boosted_lm_words = [ "PipeCat" , "AI" , "speech" ] ​ Notes Uses NVIDIA’s Riva AI Services platform Handles streaming audio input Provides real-time transcription results Manages connection lifecycle Uses asyncio for asynchronous processing Automatically cleans up resources on stop/cancel Groq (Whisper) OpenAI On this page Overview Installation RivaSTTService Configuration InputParams Input Output Frames TranscriptionFrame InterimTranscriptionFrame RivaSegmentedSTTService Configuration InputParams Input Output Frames TranscriptionFrame InterimTranscriptionFrame Methods Models Usage Examples RivaSTTService RivaSegmentedSTTService Language Support Frame Flow Advanced Configuration Example with Advanced Configuration Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_riva_5bf58f28.txt b/stt_riva_5bf58f28.txt
new file mode 100644
index 0000000000000000000000000000000000000000..3859e3047ca98c4186eea3cf9dce593f8b1afc43
--- /dev/null
+++ b/stt_riva_5bf58f28.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/riva#rivasegmentedsttservice
+Title: NVIDIA Riva - Pipecat
+==================================================
+
+NVIDIA Riva - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text NVIDIA Riva Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview RivaSTTService provides real-time speech-to-text capabilities using NVIDIA’s Riva Parakeet model. It supports interim results and configurable recognition parameters for enhanced accuracy. RivaSegmentedSTTService provides speech-to-text capabilities via NVIDIA’s Riva Canary model. ​ Installation To use RivaSTTService or RivaSegmentedSTTService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[riva]" You’ll also need to set up your NVIDIA API key as an environment variable: NVIDIA_API_KEY . You can obtain an NVIDIA API key by signing up through NVIDIA’s developer portal . ​ RivaSTTService ​ Configuration ​ api_key str required Your NVIDIA API key ​ server str default: "grpc.nvcf.nvidia.com:443" NVIDIA Riva server address ​ model_function_map Mapping [str, str] A mapping of the NVIDIA function identifier for the STT service with the model name. ​ sample_rate int default: "None" Audio sample rate in Hz ​ params InputParams default: "InputParams()" Additional configuration parameters ​ InputParams ​ language Language default: "Language.EN_US" The language for speech recognition ​ Input The service processes audio frames containing: Raw PCM audio data 16-bit depth Single channel (mono) ​ Output Frames ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Language used for transcription ​ InterimTranscriptionFrame Generated during ongoing speech, containing same fields as TranscriptionFrame but with preliminary results. ​ RivaSegmentedSTTService ​ Configuration ​ api_key str required Your NVIDIA API key ​ server str default: "grpc.nvcf.nvidia.com:443" NVIDIA Riva server address ​ model_function_map Mapping [str, str] A mapping of the NVIDIA function identifier for the STT service with the model name. ​ sample_rate int default: "None" Audio sample rate in Hz ​ params InputParams default: "InputParams()" Additional configuration parameters ​ InputParams ​ language Language default: "Language.EN_US" The language for speech recognition ​ Input The service processes audio frames containing: Raw audio bytes in WAV format ​ Output Frames ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Language used for transcription ​ InterimTranscriptionFrame Generated during ongoing speech, containing same fields as TranscriptionFrame but with preliminary results. ​ Methods See the STT base class methods for additional functionality. ​ Models Model Pipecat Class Model Card Link parakeet-ctc-1.1b-asr RivaSTTService NVIDIA Model Card canary-1b-asr RivaSegmentedSTTService NVIDIA Model Card ​ Usage Examples ​ RivaSTTService Copy Ask AI from pipecat.services.riva.stt import RivaSTTService from pipecat.transcriptions.language import Language # Configure service stt = RivaSTTService( api_key = "your-nvidia-api-key" , params = RivaSTTService.InputParams( language = Language. EN_US ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ RivaSegmentedSTTService Copy Ask AI from pipecat.services.riva.stt import RivaSegmentedSTTService from pipecat.transcriptions.language import Language # Configure service stt = RivaSegmentedSTTService( api_key = "your-nvidia-api-key" , params = RivaSegmentedSTTService.InputParams( language = Language. EN_US ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ Language Support Riva model parakeet-ctc-1.1b-asr (default) primarily supports English with various regional accents: Language Code Description Service Codes Language.EN_US English (US) en-US ​ Frame Flow ​ Advanced Configuration The service supports several advanced configuration options that can be adjusted: ​ _profanity_filter bool default: "False" Filter profanity from transcription ​ _automatic_punctuation bool default: "False" Automatically add punctuation ​ _no_verbatim_transcripts bool default: "False" Whether to disable verbatim transcripts ​ _boosted_lm_words list default: "None" List of words to boost in the language model ​ _boosted_lm_score float default: "4.0" Score applied to boosted words ​ Example with Advanced Configuration Copy Ask AI # Configure service with advanced parameters stt = RivaSTTService( api_key = "your-nvidia-api-key" , params = RivaSTTService.InputParams( language = Language. EN_US ) ) # Configure advanced options stt._profanity_filter = True stt._automatic_punctuation = True stt._boosted_lm_words = [ "PipeCat" , "AI" , "speech" ] ​ Notes Uses NVIDIA’s Riva AI Services platform Handles streaming audio input Provides real-time transcription results Manages connection lifecycle Uses asyncio for asynchronous processing Automatically cleans up resources on stop/cancel Groq (Whisper) OpenAI On this page Overview Installation RivaSTTService Configuration InputParams Input Output Frames TranscriptionFrame InterimTranscriptionFrame RivaSegmentedSTTService Configuration InputParams Input Output Frames TranscriptionFrame InterimTranscriptionFrame Methods Models Usage Examples RivaSTTService RivaSegmentedSTTService Language Support Frame Flow Advanced Configuration Example with Advanced Configuration Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_riva_6e13ccfd.txt b/stt_riva_6e13ccfd.txt
new file mode 100644
index 0000000000000000000000000000000000000000..cf697a8ed482657c7de4b9b8ad6a227c2e8691a8
--- /dev/null
+++ b/stt_riva_6e13ccfd.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/riva
+Title: NVIDIA Riva - Pipecat
+==================================================
+
+NVIDIA Riva - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text NVIDIA Riva Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview RivaSTTService provides real-time speech-to-text capabilities using NVIDIA’s Riva Parakeet model. It supports interim results and configurable recognition parameters for enhanced accuracy. RivaSegmentedSTTService provides speech-to-text capabilities via NVIDIA’s Riva Canary model. ​ Installation To use RivaSTTService or RivaSegmentedSTTService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[riva]" You’ll also need to set up your NVIDIA API key as an environment variable: NVIDIA_API_KEY . You can obtain an NVIDIA API key by signing up through NVIDIA’s developer portal . ​ RivaSTTService ​ Configuration ​ api_key str required Your NVIDIA API key ​ server str default: "grpc.nvcf.nvidia.com:443" NVIDIA Riva server address ​ model_function_map Mapping [str, str] A mapping of the NVIDIA function identifier for the STT service with the model name. ​ sample_rate int default: "None" Audio sample rate in Hz ​ params InputParams default: "InputParams()" Additional configuration parameters ​ InputParams ​ language Language default: "Language.EN_US" The language for speech recognition ​ Input The service processes audio frames containing: Raw PCM audio data 16-bit depth Single channel (mono) ​ Output Frames ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Language used for transcription ​ InterimTranscriptionFrame Generated during ongoing speech, containing same fields as TranscriptionFrame but with preliminary results. ​ RivaSegmentedSTTService ​ Configuration ​ api_key str required Your NVIDIA API key ​ server str default: "grpc.nvcf.nvidia.com:443" NVIDIA Riva server address ​ model_function_map Mapping [str, str] A mapping of the NVIDIA function identifier for the STT service with the model name. ​ sample_rate int default: "None" Audio sample rate in Hz ​ params InputParams default: "InputParams()" Additional configuration parameters ​ InputParams ​ language Language default: "Language.EN_US" The language for speech recognition ​ Input The service processes audio frames containing: Raw audio bytes in WAV format ​ Output Frames ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Language used for transcription ​ InterimTranscriptionFrame Generated during ongoing speech, containing same fields as TranscriptionFrame but with preliminary results. ​ Methods See the STT base class methods for additional functionality. ​ Models Model Pipecat Class Model Card Link parakeet-ctc-1.1b-asr RivaSTTService NVIDIA Model Card canary-1b-asr RivaSegmentedSTTService NVIDIA Model Card ​ Usage Examples ​ RivaSTTService Copy Ask AI from pipecat.services.riva.stt import RivaSTTService from pipecat.transcriptions.language import Language # Configure service stt = RivaSTTService( api_key = "your-nvidia-api-key" , params = RivaSTTService.InputParams( language = Language. EN_US ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ RivaSegmentedSTTService Copy Ask AI from pipecat.services.riva.stt import RivaSegmentedSTTService from pipecat.transcriptions.language import Language # Configure service stt = RivaSegmentedSTTService( api_key = "your-nvidia-api-key" , params = RivaSegmentedSTTService.InputParams( language = Language. EN_US ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ Language Support Riva model parakeet-ctc-1.1b-asr (default) primarily supports English with various regional accents: Language Code Description Service Codes Language.EN_US English (US) en-US ​ Frame Flow ​ Advanced Configuration The service supports several advanced configuration options that can be adjusted: ​ _profanity_filter bool default: "False" Filter profanity from transcription ​ _automatic_punctuation bool default: "False" Automatically add punctuation ​ _no_verbatim_transcripts bool default: "False" Whether to disable verbatim transcripts ​ _boosted_lm_words list default: "None" List of words to boost in the language model ​ _boosted_lm_score float default: "4.0" Score applied to boosted words ​ Example with Advanced Configuration Copy Ask AI # Configure service with advanced parameters stt = RivaSTTService( api_key = "your-nvidia-api-key" , params = RivaSTTService.InputParams( language = Language. EN_US ) ) # Configure advanced options stt._profanity_filter = True stt._automatic_punctuation = True stt._boosted_lm_words = [ "PipeCat" , "AI" , "speech" ] ​ Notes Uses NVIDIA’s Riva AI Services platform Handles streaming audio input Provides real-time transcription results Manages connection lifecycle Uses asyncio for asynchronous processing Automatically cleans up resources on stop/cancel Groq (Whisper) OpenAI On this page Overview Installation RivaSTTService Configuration InputParams Input Output Frames TranscriptionFrame InterimTranscriptionFrame RivaSegmentedSTTService Configuration InputParams Input Output Frames TranscriptionFrame InterimTranscriptionFrame Methods Models Usage Examples RivaSTTService RivaSegmentedSTTService Language Support Frame Flow Advanced Configuration Example with Advanced Configuration Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_riva_7ce8ff26.txt b/stt_riva_7ce8ff26.txt
new file mode 100644
index 0000000000000000000000000000000000000000..be55d8cfac83b0109bb7702fe37c2346c317658f
--- /dev/null
+++ b/stt_riva_7ce8ff26.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/riva#param-no-verbatim-transcripts
+Title: NVIDIA Riva - Pipecat
+==================================================
+
+NVIDIA Riva - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text NVIDIA Riva Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview RivaSTTService provides real-time speech-to-text capabilities using NVIDIA’s Riva Parakeet model. It supports interim results and configurable recognition parameters for enhanced accuracy. RivaSegmentedSTTService provides speech-to-text capabilities via NVIDIA’s Riva Canary model. ​ Installation To use RivaSTTService or RivaSegmentedSTTService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[riva]" You’ll also need to set up your NVIDIA API key as an environment variable: NVIDIA_API_KEY . You can obtain an NVIDIA API key by signing up through NVIDIA’s developer portal . ​ RivaSTTService ​ Configuration ​ api_key str required Your NVIDIA API key ​ server str default: "grpc.nvcf.nvidia.com:443" NVIDIA Riva server address ​ model_function_map Mapping [str, str] A mapping of the NVIDIA function identifier for the STT service with the model name. ​ sample_rate int default: "None" Audio sample rate in Hz ​ params InputParams default: "InputParams()" Additional configuration parameters ​ InputParams ​ language Language default: "Language.EN_US" The language for speech recognition ​ Input The service processes audio frames containing: Raw PCM audio data 16-bit depth Single channel (mono) ​ Output Frames ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Language used for transcription ​ InterimTranscriptionFrame Generated during ongoing speech, containing same fields as TranscriptionFrame but with preliminary results. ​ RivaSegmentedSTTService ​ Configuration ​ api_key str required Your NVIDIA API key ​ server str default: "grpc.nvcf.nvidia.com:443" NVIDIA Riva server address ​ model_function_map Mapping [str, str] A mapping of the NVIDIA function identifier for the STT service with the model name. ​ sample_rate int default: "None" Audio sample rate in Hz ​ params InputParams default: "InputParams()" Additional configuration parameters ​ InputParams ​ language Language default: "Language.EN_US" The language for speech recognition ​ Input The service processes audio frames containing: Raw audio bytes in WAV format ​ Output Frames ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Language used for transcription ​ InterimTranscriptionFrame Generated during ongoing speech, containing same fields as TranscriptionFrame but with preliminary results. ​ Methods See the STT base class methods for additional functionality. ​ Models Model Pipecat Class Model Card Link parakeet-ctc-1.1b-asr RivaSTTService NVIDIA Model Card canary-1b-asr RivaSegmentedSTTService NVIDIA Model Card ​ Usage Examples ​ RivaSTTService Copy Ask AI from pipecat.services.riva.stt import RivaSTTService from pipecat.transcriptions.language import Language # Configure service stt = RivaSTTService( api_key = "your-nvidia-api-key" , params = RivaSTTService.InputParams( language = Language. EN_US ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ RivaSegmentedSTTService Copy Ask AI from pipecat.services.riva.stt import RivaSegmentedSTTService from pipecat.transcriptions.language import Language # Configure service stt = RivaSegmentedSTTService( api_key = "your-nvidia-api-key" , params = RivaSegmentedSTTService.InputParams( language = Language. EN_US ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ Language Support Riva model parakeet-ctc-1.1b-asr (default) primarily supports English with various regional accents: Language Code Description Service Codes Language.EN_US English (US) en-US ​ Frame Flow ​ Advanced Configuration The service supports several advanced configuration options that can be adjusted: ​ _profanity_filter bool default: "False" Filter profanity from transcription ​ _automatic_punctuation bool default: "False" Automatically add punctuation ​ _no_verbatim_transcripts bool default: "False" Whether to disable verbatim transcripts ​ _boosted_lm_words list default: "None" List of words to boost in the language model ​ _boosted_lm_score float default: "4.0" Score applied to boosted words ​ Example with Advanced Configuration Copy Ask AI # Configure service with advanced parameters stt = RivaSTTService( api_key = "your-nvidia-api-key" , params = RivaSTTService.InputParams( language = Language. EN_US ) ) # Configure advanced options stt._profanity_filter = True stt._automatic_punctuation = True stt._boosted_lm_words = [ "PipeCat" , "AI" , "speech" ] ​ Notes Uses NVIDIA’s Riva AI Services platform Handles streaming audio input Provides real-time transcription results Manages connection lifecycle Uses asyncio for asynchronous processing Automatically cleans up resources on stop/cancel Groq (Whisper) OpenAI On this page Overview Installation RivaSTTService Configuration InputParams Input Output Frames TranscriptionFrame InterimTranscriptionFrame RivaSegmentedSTTService Configuration InputParams Input Output Frames TranscriptionFrame InterimTranscriptionFrame Methods Models Usage Examples RivaSTTService RivaSegmentedSTTService Language Support Frame Flow Advanced Configuration Example with Advanced Configuration Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_riva_818bc6c3.txt b/stt_riva_818bc6c3.txt
new file mode 100644
index 0000000000000000000000000000000000000000..5d6dfadf9df520968bec38bafe59a5fd24aa5848
--- /dev/null
+++ b/stt_riva_818bc6c3.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/riva#notes
+Title: NVIDIA Riva - Pipecat
+==================================================
+
+NVIDIA Riva - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text NVIDIA Riva Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview RivaSTTService provides real-time speech-to-text capabilities using NVIDIA’s Riva Parakeet model. It supports interim results and configurable recognition parameters for enhanced accuracy. RivaSegmentedSTTService provides speech-to-text capabilities via NVIDIA’s Riva Canary model. ​ Installation To use RivaSTTService or RivaSegmentedSTTService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[riva]" You’ll also need to set up your NVIDIA API key as an environment variable: NVIDIA_API_KEY . You can obtain an NVIDIA API key by signing up through NVIDIA’s developer portal . ​ RivaSTTService ​ Configuration ​ api_key str required Your NVIDIA API key ​ server str default: "grpc.nvcf.nvidia.com:443" NVIDIA Riva server address ​ model_function_map Mapping [str, str] A mapping of the NVIDIA function identifier for the STT service with the model name. ​ sample_rate int default: "None" Audio sample rate in Hz ​ params InputParams default: "InputParams()" Additional configuration parameters ​ InputParams ​ language Language default: "Language.EN_US" The language for speech recognition ​ Input The service processes audio frames containing: Raw PCM audio data 16-bit depth Single channel (mono) ​ Output Frames ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Language used for transcription ​ InterimTranscriptionFrame Generated during ongoing speech, containing same fields as TranscriptionFrame but with preliminary results. ​ RivaSegmentedSTTService ​ Configuration ​ api_key str required Your NVIDIA API key ​ server str default: "grpc.nvcf.nvidia.com:443" NVIDIA Riva server address ​ model_function_map Mapping [str, str] A mapping of the NVIDIA function identifier for the STT service with the model name. ​ sample_rate int default: "None" Audio sample rate in Hz ​ params InputParams default: "InputParams()" Additional configuration parameters ​ InputParams ​ language Language default: "Language.EN_US" The language for speech recognition ​ Input The service processes audio frames containing: Raw audio bytes in WAV format ​ Output Frames ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Language used for transcription ​ InterimTranscriptionFrame Generated during ongoing speech, containing same fields as TranscriptionFrame but with preliminary results. ​ Methods See the STT base class methods for additional functionality. ​ Models Model Pipecat Class Model Card Link parakeet-ctc-1.1b-asr RivaSTTService NVIDIA Model Card canary-1b-asr RivaSegmentedSTTService NVIDIA Model Card ​ Usage Examples ​ RivaSTTService Copy Ask AI from pipecat.services.riva.stt import RivaSTTService from pipecat.transcriptions.language import Language # Configure service stt = RivaSTTService( api_key = "your-nvidia-api-key" , params = RivaSTTService.InputParams( language = Language. EN_US ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ RivaSegmentedSTTService Copy Ask AI from pipecat.services.riva.stt import RivaSegmentedSTTService from pipecat.transcriptions.language import Language # Configure service stt = RivaSegmentedSTTService( api_key = "your-nvidia-api-key" , params = RivaSegmentedSTTService.InputParams( language = Language. EN_US ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ Language Support Riva model parakeet-ctc-1.1b-asr (default) primarily supports English with various regional accents: Language Code Description Service Codes Language.EN_US English (US) en-US ​ Frame Flow ​ Advanced Configuration The service supports several advanced configuration options that can be adjusted: ​ _profanity_filter bool default: "False" Filter profanity from transcription ​ _automatic_punctuation bool default: "False" Automatically add punctuation ​ _no_verbatim_transcripts bool default: "False" Whether to disable verbatim transcripts ​ _boosted_lm_words list default: "None" List of words to boost in the language model ​ _boosted_lm_score float default: "4.0" Score applied to boosted words ​ Example with Advanced Configuration Copy Ask AI # Configure service with advanced parameters stt = RivaSTTService( api_key = "your-nvidia-api-key" , params = RivaSTTService.InputParams( language = Language. EN_US ) ) # Configure advanced options stt._profanity_filter = True stt._automatic_punctuation = True stt._boosted_lm_words = [ "PipeCat" , "AI" , "speech" ] ​ Notes Uses NVIDIA’s Riva AI Services platform Handles streaming audio input Provides real-time transcription results Manages connection lifecycle Uses asyncio for asynchronous processing Automatically cleans up resources on stop/cancel Groq (Whisper) OpenAI On this page Overview Installation RivaSTTService Configuration InputParams Input Output Frames TranscriptionFrame InterimTranscriptionFrame RivaSegmentedSTTService Configuration InputParams Input Output Frames TranscriptionFrame InterimTranscriptionFrame Methods Models Usage Examples RivaSTTService RivaSegmentedSTTService Language Support Frame Flow Advanced Configuration Example with Advanced Configuration Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_riva_9a3d68a6.txt b/stt_riva_9a3d68a6.txt
new file mode 100644
index 0000000000000000000000000000000000000000..c8f76b69c46c9c09cea838b95641b9b33bee856c
--- /dev/null
+++ b/stt_riva_9a3d68a6.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/riva#param-timestamp-1
+Title: NVIDIA Riva - Pipecat
+==================================================
+
+NVIDIA Riva - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text NVIDIA Riva Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview RivaSTTService provides real-time speech-to-text capabilities using NVIDIA’s Riva Parakeet model. It supports interim results and configurable recognition parameters for enhanced accuracy. RivaSegmentedSTTService provides speech-to-text capabilities via NVIDIA’s Riva Canary model. ​ Installation To use RivaSTTService or RivaSegmentedSTTService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[riva]" You’ll also need to set up your NVIDIA API key as an environment variable: NVIDIA_API_KEY . You can obtain an NVIDIA API key by signing up through NVIDIA’s developer portal . ​ RivaSTTService ​ Configuration ​ api_key str required Your NVIDIA API key ​ server str default: "grpc.nvcf.nvidia.com:443" NVIDIA Riva server address ​ model_function_map Mapping [str, str] A mapping of the NVIDIA function identifier for the STT service with the model name. ​ sample_rate int default: "None" Audio sample rate in Hz ​ params InputParams default: "InputParams()" Additional configuration parameters ​ InputParams ​ language Language default: "Language.EN_US" The language for speech recognition ​ Input The service processes audio frames containing: Raw PCM audio data 16-bit depth Single channel (mono) ​ Output Frames ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Language used for transcription ​ InterimTranscriptionFrame Generated during ongoing speech, containing same fields as TranscriptionFrame but with preliminary results. ​ RivaSegmentedSTTService ​ Configuration ​ api_key str required Your NVIDIA API key ​ server str default: "grpc.nvcf.nvidia.com:443" NVIDIA Riva server address ​ model_function_map Mapping [str, str] A mapping of the NVIDIA function identifier for the STT service with the model name. ​ sample_rate int default: "None" Audio sample rate in Hz ​ params InputParams default: "InputParams()" Additional configuration parameters ​ InputParams ​ language Language default: "Language.EN_US" The language for speech recognition ​ Input The service processes audio frames containing: Raw audio bytes in WAV format ​ Output Frames ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Language used for transcription ​ InterimTranscriptionFrame Generated during ongoing speech, containing same fields as TranscriptionFrame but with preliminary results. ​ Methods See the STT base class methods for additional functionality. ​ Models Model Pipecat Class Model Card Link parakeet-ctc-1.1b-asr RivaSTTService NVIDIA Model Card canary-1b-asr RivaSegmentedSTTService NVIDIA Model Card ​ Usage Examples ​ RivaSTTService Copy Ask AI from pipecat.services.riva.stt import RivaSTTService from pipecat.transcriptions.language import Language # Configure service stt = RivaSTTService( api_key = "your-nvidia-api-key" , params = RivaSTTService.InputParams( language = Language. EN_US ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ RivaSegmentedSTTService Copy Ask AI from pipecat.services.riva.stt import RivaSegmentedSTTService from pipecat.transcriptions.language import Language # Configure service stt = RivaSegmentedSTTService( api_key = "your-nvidia-api-key" , params = RivaSegmentedSTTService.InputParams( language = Language. EN_US ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ Language Support Riva model parakeet-ctc-1.1b-asr (default) primarily supports English with various regional accents: Language Code Description Service Codes Language.EN_US English (US) en-US ​ Frame Flow ​ Advanced Configuration The service supports several advanced configuration options that can be adjusted: ​ _profanity_filter bool default: "False" Filter profanity from transcription ​ _automatic_punctuation bool default: "False" Automatically add punctuation ​ _no_verbatim_transcripts bool default: "False" Whether to disable verbatim transcripts ​ _boosted_lm_words list default: "None" List of words to boost in the language model ​ _boosted_lm_score float default: "4.0" Score applied to boosted words ​ Example with Advanced Configuration Copy Ask AI # Configure service with advanced parameters stt = RivaSTTService( api_key = "your-nvidia-api-key" , params = RivaSTTService.InputParams( language = Language. EN_US ) ) # Configure advanced options stt._profanity_filter = True stt._automatic_punctuation = True stt._boosted_lm_words = [ "PipeCat" , "AI" , "speech" ] ​ Notes Uses NVIDIA’s Riva AI Services platform Handles streaming audio input Provides real-time transcription results Manages connection lifecycle Uses asyncio for asynchronous processing Automatically cleans up resources on stop/cancel Groq (Whisper) OpenAI On this page Overview Installation RivaSTTService Configuration InputParams Input Output Frames TranscriptionFrame InterimTranscriptionFrame RivaSegmentedSTTService Configuration InputParams Input Output Frames TranscriptionFrame InterimTranscriptionFrame Methods Models Usage Examples RivaSTTService RivaSegmentedSTTService Language Support Frame Flow Advanced Configuration Example with Advanced Configuration Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_riva_c743c0a0.txt b/stt_riva_c743c0a0.txt
new file mode 100644
index 0000000000000000000000000000000000000000..5f12a2bc42542cf1c5d20dc547a8ff9cf1b3421a
--- /dev/null
+++ b/stt_riva_c743c0a0.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/riva#param-sample-rate-1
+Title: NVIDIA Riva - Pipecat
+==================================================
+
+NVIDIA Riva - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text NVIDIA Riva Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview RivaSTTService provides real-time speech-to-text capabilities using NVIDIA’s Riva Parakeet model. It supports interim results and configurable recognition parameters for enhanced accuracy. RivaSegmentedSTTService provides speech-to-text capabilities via NVIDIA’s Riva Canary model. ​ Installation To use RivaSTTService or RivaSegmentedSTTService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[riva]" You’ll also need to set up your NVIDIA API key as an environment variable: NVIDIA_API_KEY . You can obtain an NVIDIA API key by signing up through NVIDIA’s developer portal . ​ RivaSTTService ​ Configuration ​ api_key str required Your NVIDIA API key ​ server str default: "grpc.nvcf.nvidia.com:443" NVIDIA Riva server address ​ model_function_map Mapping [str, str] A mapping of the NVIDIA function identifier for the STT service with the model name. ​ sample_rate int default: "None" Audio sample rate in Hz ​ params InputParams default: "InputParams()" Additional configuration parameters ​ InputParams ​ language Language default: "Language.EN_US" The language for speech recognition ​ Input The service processes audio frames containing: Raw PCM audio data 16-bit depth Single channel (mono) ​ Output Frames ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Language used for transcription ​ InterimTranscriptionFrame Generated during ongoing speech, containing same fields as TranscriptionFrame but with preliminary results. ​ RivaSegmentedSTTService ​ Configuration ​ api_key str required Your NVIDIA API key ​ server str default: "grpc.nvcf.nvidia.com:443" NVIDIA Riva server address ​ model_function_map Mapping [str, str] A mapping of the NVIDIA function identifier for the STT service with the model name. ​ sample_rate int default: "None" Audio sample rate in Hz ​ params InputParams default: "InputParams()" Additional configuration parameters ​ InputParams ​ language Language default: "Language.EN_US" The language for speech recognition ​ Input The service processes audio frames containing: Raw audio bytes in WAV format ​ Output Frames ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Language used for transcription ​ InterimTranscriptionFrame Generated during ongoing speech, containing same fields as TranscriptionFrame but with preliminary results. ​ Methods See the STT base class methods for additional functionality. ​ Models Model Pipecat Class Model Card Link parakeet-ctc-1.1b-asr RivaSTTService NVIDIA Model Card canary-1b-asr RivaSegmentedSTTService NVIDIA Model Card ​ Usage Examples ​ RivaSTTService Copy Ask AI from pipecat.services.riva.stt import RivaSTTService from pipecat.transcriptions.language import Language # Configure service stt = RivaSTTService( api_key = "your-nvidia-api-key" , params = RivaSTTService.InputParams( language = Language. EN_US ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ RivaSegmentedSTTService Copy Ask AI from pipecat.services.riva.stt import RivaSegmentedSTTService from pipecat.transcriptions.language import Language # Configure service stt = RivaSegmentedSTTService( api_key = "your-nvidia-api-key" , params = RivaSegmentedSTTService.InputParams( language = Language. EN_US ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ Language Support Riva model parakeet-ctc-1.1b-asr (default) primarily supports English with various regional accents: Language Code Description Service Codes Language.EN_US English (US) en-US ​ Frame Flow ​ Advanced Configuration The service supports several advanced configuration options that can be adjusted: ​ _profanity_filter bool default: "False" Filter profanity from transcription ​ _automatic_punctuation bool default: "False" Automatically add punctuation ​ _no_verbatim_transcripts bool default: "False" Whether to disable verbatim transcripts ​ _boosted_lm_words list default: "None" List of words to boost in the language model ​ _boosted_lm_score float default: "4.0" Score applied to boosted words ​ Example with Advanced Configuration Copy Ask AI # Configure service with advanced parameters stt = RivaSTTService( api_key = "your-nvidia-api-key" , params = RivaSTTService.InputParams( language = Language. EN_US ) ) # Configure advanced options stt._profanity_filter = True stt._automatic_punctuation = True stt._boosted_lm_words = [ "PipeCat" , "AI" , "speech" ] ​ Notes Uses NVIDIA’s Riva AI Services platform Handles streaming audio input Provides real-time transcription results Manages connection lifecycle Uses asyncio for asynchronous processing Automatically cleans up resources on stop/cancel Groq (Whisper) OpenAI On this page Overview Installation RivaSTTService Configuration InputParams Input Output Frames TranscriptionFrame InterimTranscriptionFrame RivaSegmentedSTTService Configuration InputParams Input Output Frames TranscriptionFrame InterimTranscriptionFrame Methods Models Usage Examples RivaSTTService RivaSegmentedSTTService Language Support Frame Flow Advanced Configuration Example with Advanced Configuration Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_riva_f2733c26.txt b/stt_riva_f2733c26.txt
new file mode 100644
index 0000000000000000000000000000000000000000..032ae7cee372372b3ee7652772525a8324a985e1
--- /dev/null
+++ b/stt_riva_f2733c26.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/riva#frame-flow
+Title: NVIDIA Riva - Pipecat
+==================================================
+
+NVIDIA Riva - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text NVIDIA Riva Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview RivaSTTService provides real-time speech-to-text capabilities using NVIDIA’s Riva Parakeet model. It supports interim results and configurable recognition parameters for enhanced accuracy. RivaSegmentedSTTService provides speech-to-text capabilities via NVIDIA’s Riva Canary model. ​ Installation To use RivaSTTService or RivaSegmentedSTTService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[riva]" You’ll also need to set up your NVIDIA API key as an environment variable: NVIDIA_API_KEY . You can obtain an NVIDIA API key by signing up through NVIDIA’s developer portal . ​ RivaSTTService ​ Configuration ​ api_key str required Your NVIDIA API key ​ server str default: "grpc.nvcf.nvidia.com:443" NVIDIA Riva server address ​ model_function_map Mapping [str, str] A mapping of the NVIDIA function identifier for the STT service with the model name. ​ sample_rate int default: "None" Audio sample rate in Hz ​ params InputParams default: "InputParams()" Additional configuration parameters ​ InputParams ​ language Language default: "Language.EN_US" The language for speech recognition ​ Input The service processes audio frames containing: Raw PCM audio data 16-bit depth Single channel (mono) ​ Output Frames ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Language used for transcription ​ InterimTranscriptionFrame Generated during ongoing speech, containing same fields as TranscriptionFrame but with preliminary results. ​ RivaSegmentedSTTService ​ Configuration ​ api_key str required Your NVIDIA API key ​ server str default: "grpc.nvcf.nvidia.com:443" NVIDIA Riva server address ​ model_function_map Mapping [str, str] A mapping of the NVIDIA function identifier for the STT service with the model name. ​ sample_rate int default: "None" Audio sample rate in Hz ​ params InputParams default: "InputParams()" Additional configuration parameters ​ InputParams ​ language Language default: "Language.EN_US" The language for speech recognition ​ Input The service processes audio frames containing: Raw audio bytes in WAV format ​ Output Frames ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Language used for transcription ​ InterimTranscriptionFrame Generated during ongoing speech, containing same fields as TranscriptionFrame but with preliminary results. ​ Methods See the STT base class methods for additional functionality. ​ Models Model Pipecat Class Model Card Link parakeet-ctc-1.1b-asr RivaSTTService NVIDIA Model Card canary-1b-asr RivaSegmentedSTTService NVIDIA Model Card ​ Usage Examples ​ RivaSTTService Copy Ask AI from pipecat.services.riva.stt import RivaSTTService from pipecat.transcriptions.language import Language # Configure service stt = RivaSTTService( api_key = "your-nvidia-api-key" , params = RivaSTTService.InputParams( language = Language. EN_US ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ RivaSegmentedSTTService Copy Ask AI from pipecat.services.riva.stt import RivaSegmentedSTTService from pipecat.transcriptions.language import Language # Configure service stt = RivaSegmentedSTTService( api_key = "your-nvidia-api-key" , params = RivaSegmentedSTTService.InputParams( language = Language. EN_US ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ Language Support Riva model parakeet-ctc-1.1b-asr (default) primarily supports English with various regional accents: Language Code Description Service Codes Language.EN_US English (US) en-US ​ Frame Flow ​ Advanced Configuration The service supports several advanced configuration options that can be adjusted: ​ _profanity_filter bool default: "False" Filter profanity from transcription ​ _automatic_punctuation bool default: "False" Automatically add punctuation ​ _no_verbatim_transcripts bool default: "False" Whether to disable verbatim transcripts ​ _boosted_lm_words list default: "None" List of words to boost in the language model ​ _boosted_lm_score float default: "4.0" Score applied to boosted words ​ Example with Advanced Configuration Copy Ask AI # Configure service with advanced parameters stt = RivaSTTService( api_key = "your-nvidia-api-key" , params = RivaSTTService.InputParams( language = Language. EN_US ) ) # Configure advanced options stt._profanity_filter = True stt._automatic_punctuation = True stt._boosted_lm_words = [ "PipeCat" , "AI" , "speech" ] ​ Notes Uses NVIDIA’s Riva AI Services platform Handles streaming audio input Provides real-time transcription results Manages connection lifecycle Uses asyncio for asynchronous processing Automatically cleans up resources on stop/cancel Groq (Whisper) OpenAI On this page Overview Installation RivaSTTService Configuration InputParams Input Output Frames TranscriptionFrame InterimTranscriptionFrame RivaSegmentedSTTService Configuration InputParams Input Output Frames TranscriptionFrame InterimTranscriptionFrame Methods Models Usage Examples RivaSTTService RivaSegmentedSTTService Language Support Frame Flow Advanced Configuration Example with Advanced Configuration Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_riva_fb8614af.txt b/stt_riva_fb8614af.txt
new file mode 100644
index 0000000000000000000000000000000000000000..61fdffdda86a4e3aca67f6bc55e0cf524c9961ac
--- /dev/null
+++ b/stt_riva_fb8614af.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/riva#output-frames
+Title: NVIDIA Riva - Pipecat
+==================================================
+
+NVIDIA Riva - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text NVIDIA Riva Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview RivaSTTService provides real-time speech-to-text capabilities using NVIDIA’s Riva Parakeet model. It supports interim results and configurable recognition parameters for enhanced accuracy. RivaSegmentedSTTService provides speech-to-text capabilities via NVIDIA’s Riva Canary model. ​ Installation To use RivaSTTService or RivaSegmentedSTTService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[riva]" You’ll also need to set up your NVIDIA API key as an environment variable: NVIDIA_API_KEY . You can obtain an NVIDIA API key by signing up through NVIDIA’s developer portal . ​ RivaSTTService ​ Configuration ​ api_key str required Your NVIDIA API key ​ server str default: "grpc.nvcf.nvidia.com:443" NVIDIA Riva server address ​ model_function_map Mapping [str, str] A mapping of the NVIDIA function identifier for the STT service with the model name. ​ sample_rate int default: "None" Audio sample rate in Hz ​ params InputParams default: "InputParams()" Additional configuration parameters ​ InputParams ​ language Language default: "Language.EN_US" The language for speech recognition ​ Input The service processes audio frames containing: Raw PCM audio data 16-bit depth Single channel (mono) ​ Output Frames ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Language used for transcription ​ InterimTranscriptionFrame Generated during ongoing speech, containing same fields as TranscriptionFrame but with preliminary results. ​ RivaSegmentedSTTService ​ Configuration ​ api_key str required Your NVIDIA API key ​ server str default: "grpc.nvcf.nvidia.com:443" NVIDIA Riva server address ​ model_function_map Mapping [str, str] A mapping of the NVIDIA function identifier for the STT service with the model name. ​ sample_rate int default: "None" Audio sample rate in Hz ​ params InputParams default: "InputParams()" Additional configuration parameters ​ InputParams ​ language Language default: "Language.EN_US" The language for speech recognition ​ Input The service processes audio frames containing: Raw audio bytes in WAV format ​ Output Frames ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Language used for transcription ​ InterimTranscriptionFrame Generated during ongoing speech, containing same fields as TranscriptionFrame but with preliminary results. ​ Methods See the STT base class methods for additional functionality. ​ Models Model Pipecat Class Model Card Link parakeet-ctc-1.1b-asr RivaSTTService NVIDIA Model Card canary-1b-asr RivaSegmentedSTTService NVIDIA Model Card ​ Usage Examples ​ RivaSTTService Copy Ask AI from pipecat.services.riva.stt import RivaSTTService from pipecat.transcriptions.language import Language # Configure service stt = RivaSTTService( api_key = "your-nvidia-api-key" , params = RivaSTTService.InputParams( language = Language. EN_US ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ RivaSegmentedSTTService Copy Ask AI from pipecat.services.riva.stt import RivaSegmentedSTTService from pipecat.transcriptions.language import Language # Configure service stt = RivaSegmentedSTTService( api_key = "your-nvidia-api-key" , params = RivaSegmentedSTTService.InputParams( language = Language. EN_US ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ Language Support Riva model parakeet-ctc-1.1b-asr (default) primarily supports English with various regional accents: Language Code Description Service Codes Language.EN_US English (US) en-US ​ Frame Flow ​ Advanced Configuration The service supports several advanced configuration options that can be adjusted: ​ _profanity_filter bool default: "False" Filter profanity from transcription ​ _automatic_punctuation bool default: "False" Automatically add punctuation ​ _no_verbatim_transcripts bool default: "False" Whether to disable verbatim transcripts ​ _boosted_lm_words list default: "None" List of words to boost in the language model ​ _boosted_lm_score float default: "4.0" Score applied to boosted words ​ Example with Advanced Configuration Copy Ask AI # Configure service with advanced parameters stt = RivaSTTService( api_key = "your-nvidia-api-key" , params = RivaSTTService.InputParams( language = Language. EN_US ) ) # Configure advanced options stt._profanity_filter = True stt._automatic_punctuation = True stt._boosted_lm_words = [ "PipeCat" , "AI" , "speech" ] ​ Notes Uses NVIDIA’s Riva AI Services platform Handles streaming audio input Provides real-time transcription results Manages connection lifecycle Uses asyncio for asynchronous processing Automatically cleans up resources on stop/cancel Groq (Whisper) OpenAI On this page Overview Installation RivaSTTService Configuration InputParams Input Output Frames TranscriptionFrame InterimTranscriptionFrame RivaSegmentedSTTService Configuration InputParams Input Output Frames TranscriptionFrame InterimTranscriptionFrame Methods Models Usage Examples RivaSTTService RivaSegmentedSTTService Language Support Frame Flow Advanced Configuration Example with Advanced Configuration Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_sambanova_1b8d106e.txt b/stt_sambanova_1b8d106e.txt
new file mode 100644
index 0000000000000000000000000000000000000000..1c6cfa3962cb85909b50314a9f1c05f37dfd1765
--- /dev/null
+++ b/stt_sambanova_1b8d106e.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/sambanova#input
+Title: SambaNova (Whisper) - Pipecat
+==================================================
+
+SambaNova (Whisper) - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text SambaNova (Whisper) Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview SambaNovaSTTService provides speech-to-text capabilities using SambaNova’s hosted Whisper API. It offers high-accuracy transcription with minimal setup requirements. The service uses Voice Activity Detection (VAD) to process only speech segments, optimizing API usage and improving response time. ​ Installation To use SambaNovaSTTService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[sambanova]" You need to set up your SambaNova API key as an environment variable: SAMBANOVA_API_KEY . Get your SambaNova API key here . ​ Configuration ​ Constructor Parameters ​ model str default: "Whisper-Large-v3" Whisper model to use. Currently only “Whisper-Large-v3” is available. ​ api_key str Your SambaNova API key. If not provided, will use environment variable. ​ base_url str default: "https://api.sambanova.ai/v1" Custom API base URL for SambaNova API requests. ​ language Language default: "Language.EN" Language of the audio input. Defaults to English. ​ prompt str Optional text to guide the model’s style or continue a previous segment. ​ temperature float Sampling temperature between 0 and 1. Lower values are more deterministic, higher values more creative. Defaults to 0.0. ​ sample_rate int Audio sample rate in Hz. If not provided, uses the pipeline’s sample rate. ​ Input The service processes audio data with the following requirements: PCM audio format. 16-bit depth. Single channel (mono). ​ Output Frames The service produces two types of frames during transcription: ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Detected language (if available) ​ ErrorFrame Generated when transcription errors occur, containing error details. ​ Methods ​ Set Model Copy Ask AI await service.set_model( "Whisper-Large-v3" ) See the STT base class methods for additional functionality. ​ Language Support SambaNova’s Whisper API supports a wide range of languages. The service automatically maps Language enum values to the appropriate Whisper language codes. Language Code Description Whisper Code Language.AF Afrikaans af Language.AR Arabic ar Language.HY Armenian hy Language.AZ Azerbaijani az Language.BE Belarusian be Language.BS Bosnian bs Language.BG Bulgarian bg Language.CA Catalan ca Language.ZH Chinese zh Language.HR Croatian hr Language.CS Czech cs Language.DA Danish da Language.NL Dutch nl Language.EN English en Language.ET Estonian et Language.FI Finnish fi Language.FR French fr Language.GL Galician gl Language.DE German de Language.EL Greek el Language.HE Hebrew he Language.HI Hindi hi Language.HU Hungarian hu Language.IS Icelandic is Language.ID Indonesian id Language.IT Italian it Language.JA Japanese ja Language.KN Kannada kn Language.KK Kazakh kk Language.KO Korean ko Language.LV Latvian lv Language.LT Lithuanian lt Language.MK Macedonian mk Language.MS Malay ms Language.MR Marathi mr Language.MI Maori mi Language.NE Nepali ne Language.NO Norwegian no Language.FA Persian fa Language.PL Polish pl Language.PT Portuguese pt Language.RO Romanian ro Language.RU Russian ru Language.SR Serbian sr Language.SK Slovak sk Language.SL Slovenian sl Language.ES Spanish es Language.SW Swahili sw Language.SV Swedish sv Language.TL Tagalog tl Language.TA Tamil ta Language.TH Thai th Language.TR Turkish tr Language.UK Ukrainian uk Language.UR Urdu ur Language.VI Vietnamese vi Language.CY Welsh cy SambaNova’s Whisper implementation supports language variants (like en-US , fr-CA ) by mapping them to their base language. For example, Language.EN_US and Language.EN_GB will both map to en . The service will automatically detect the language if none is specified, but specifying the language typically improves transcription accuracy. For the most up-to-date list of supported languages, refer to the SambaNova’s docs . ​ Usage Example Copy Ask AI from pipecat.services.sambanova.stt import SambaNovaSTTService from pipecat.transcriptions.language import Language # Configure service stt = SambaNovaSTTService( model = "Whisper-Large-v3" , api_key = "your-sambanova-api-key" , language = Language. EN , prompt = "Transcribe the following conversation" , temperature = 0.0 ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ Voice Activity Detection Integration This service inherits from SegmentedSTTService , which uses Voice Activity Detection (VAD) to identify speech segments for processing. This approach: Processes only actual speech, not silence or background noise. Maintains a small audio buffer (default 1 second) to capture speech that occurs slightly before VAD detection. Receives UserStartedSpeakingFrame and UserStoppedSpeakingFrame from a VAD component in the pipeline. Only sends complete utterances to the API when speech has ended. Ensure your transport includes a VAD component (like SileroVADAnalyzer ) to properly detect speech segments. ​ Metrics Support The service collects the following metrics: Time to First Byte (TTFB). Processing duration. API response time. ​ Notes Requires valid SambaNova API key. Uses SambaNova’s hosted Whisper model. Requires VAD component in transport. Processes complete utterances, not continuous audio. Handles API rate limiting. Automatic error handling. Thread-safe processing. ​ Error Handling The service handles common API errors including: Authentication errors. Rate limiting. Invalid audio format. Network connectivity issues. API timeouts. Errors are propagated through ErrorFrames with descriptive messages. OpenAI Speechmatics On this page Overview Installation Configuration Constructor Parameters Input Output Frames TranscriptionFrame ErrorFrame Methods Set Model Language Support Usage Example Voice Activity Detection Integration Metrics Support Notes Error Handling Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_sambanova_299d8708.txt b/stt_sambanova_299d8708.txt
new file mode 100644
index 0000000000000000000000000000000000000000..11d58b49b581c89ff9d7678cb1109ce30115fa3d
--- /dev/null
+++ b/stt_sambanova_299d8708.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/sambanova#notes
+Title: SambaNova (Whisper) - Pipecat
+==================================================
+
+SambaNova (Whisper) - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text SambaNova (Whisper) Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview SambaNovaSTTService provides speech-to-text capabilities using SambaNova’s hosted Whisper API. It offers high-accuracy transcription with minimal setup requirements. The service uses Voice Activity Detection (VAD) to process only speech segments, optimizing API usage and improving response time. ​ Installation To use SambaNovaSTTService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[sambanova]" You need to set up your SambaNova API key as an environment variable: SAMBANOVA_API_KEY . Get your SambaNova API key here . ​ Configuration ​ Constructor Parameters ​ model str default: "Whisper-Large-v3" Whisper model to use. Currently only “Whisper-Large-v3” is available. ​ api_key str Your SambaNova API key. If not provided, will use environment variable. ​ base_url str default: "https://api.sambanova.ai/v1" Custom API base URL for SambaNova API requests. ​ language Language default: "Language.EN" Language of the audio input. Defaults to English. ​ prompt str Optional text to guide the model’s style or continue a previous segment. ​ temperature float Sampling temperature between 0 and 1. Lower values are more deterministic, higher values more creative. Defaults to 0.0. ​ sample_rate int Audio sample rate in Hz. If not provided, uses the pipeline’s sample rate. ​ Input The service processes audio data with the following requirements: PCM audio format. 16-bit depth. Single channel (mono). ​ Output Frames The service produces two types of frames during transcription: ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Detected language (if available) ​ ErrorFrame Generated when transcription errors occur, containing error details. ​ Methods ​ Set Model Copy Ask AI await service.set_model( "Whisper-Large-v3" ) See the STT base class methods for additional functionality. ​ Language Support SambaNova’s Whisper API supports a wide range of languages. The service automatically maps Language enum values to the appropriate Whisper language codes. Language Code Description Whisper Code Language.AF Afrikaans af Language.AR Arabic ar Language.HY Armenian hy Language.AZ Azerbaijani az Language.BE Belarusian be Language.BS Bosnian bs Language.BG Bulgarian bg Language.CA Catalan ca Language.ZH Chinese zh Language.HR Croatian hr Language.CS Czech cs Language.DA Danish da Language.NL Dutch nl Language.EN English en Language.ET Estonian et Language.FI Finnish fi Language.FR French fr Language.GL Galician gl Language.DE German de Language.EL Greek el Language.HE Hebrew he Language.HI Hindi hi Language.HU Hungarian hu Language.IS Icelandic is Language.ID Indonesian id Language.IT Italian it Language.JA Japanese ja Language.KN Kannada kn Language.KK Kazakh kk Language.KO Korean ko Language.LV Latvian lv Language.LT Lithuanian lt Language.MK Macedonian mk Language.MS Malay ms Language.MR Marathi mr Language.MI Maori mi Language.NE Nepali ne Language.NO Norwegian no Language.FA Persian fa Language.PL Polish pl Language.PT Portuguese pt Language.RO Romanian ro Language.RU Russian ru Language.SR Serbian sr Language.SK Slovak sk Language.SL Slovenian sl Language.ES Spanish es Language.SW Swahili sw Language.SV Swedish sv Language.TL Tagalog tl Language.TA Tamil ta Language.TH Thai th Language.TR Turkish tr Language.UK Ukrainian uk Language.UR Urdu ur Language.VI Vietnamese vi Language.CY Welsh cy SambaNova’s Whisper implementation supports language variants (like en-US , fr-CA ) by mapping them to their base language. For example, Language.EN_US and Language.EN_GB will both map to en . The service will automatically detect the language if none is specified, but specifying the language typically improves transcription accuracy. For the most up-to-date list of supported languages, refer to the SambaNova’s docs . ​ Usage Example Copy Ask AI from pipecat.services.sambanova.stt import SambaNovaSTTService from pipecat.transcriptions.language import Language # Configure service stt = SambaNovaSTTService( model = "Whisper-Large-v3" , api_key = "your-sambanova-api-key" , language = Language. EN , prompt = "Transcribe the following conversation" , temperature = 0.0 ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ Voice Activity Detection Integration This service inherits from SegmentedSTTService , which uses Voice Activity Detection (VAD) to identify speech segments for processing. This approach: Processes only actual speech, not silence or background noise. Maintains a small audio buffer (default 1 second) to capture speech that occurs slightly before VAD detection. Receives UserStartedSpeakingFrame and UserStoppedSpeakingFrame from a VAD component in the pipeline. Only sends complete utterances to the API when speech has ended. Ensure your transport includes a VAD component (like SileroVADAnalyzer ) to properly detect speech segments. ​ Metrics Support The service collects the following metrics: Time to First Byte (TTFB). Processing duration. API response time. ​ Notes Requires valid SambaNova API key. Uses SambaNova’s hosted Whisper model. Requires VAD component in transport. Processes complete utterances, not continuous audio. Handles API rate limiting. Automatic error handling. Thread-safe processing. ​ Error Handling The service handles common API errors including: Authentication errors. Rate limiting. Invalid audio format. Network connectivity issues. API timeouts. Errors are propagated through ErrorFrames with descriptive messages. OpenAI Speechmatics On this page Overview Installation Configuration Constructor Parameters Input Output Frames TranscriptionFrame ErrorFrame Methods Set Model Language Support Usage Example Voice Activity Detection Integration Metrics Support Notes Error Handling Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_sambanova_426bb875.txt b/stt_sambanova_426bb875.txt
new file mode 100644
index 0000000000000000000000000000000000000000..d5bd75f5434fd1449d95dcca5b29938afd9a379f
--- /dev/null
+++ b/stt_sambanova_426bb875.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/sambanova#output-frames
+Title: SambaNova (Whisper) - Pipecat
+==================================================
+
+SambaNova (Whisper) - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text SambaNova (Whisper) Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview SambaNovaSTTService provides speech-to-text capabilities using SambaNova’s hosted Whisper API. It offers high-accuracy transcription with minimal setup requirements. The service uses Voice Activity Detection (VAD) to process only speech segments, optimizing API usage and improving response time. ​ Installation To use SambaNovaSTTService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[sambanova]" You need to set up your SambaNova API key as an environment variable: SAMBANOVA_API_KEY . Get your SambaNova API key here . ​ Configuration ​ Constructor Parameters ​ model str default: "Whisper-Large-v3" Whisper model to use. Currently only “Whisper-Large-v3” is available. ​ api_key str Your SambaNova API key. If not provided, will use environment variable. ​ base_url str default: "https://api.sambanova.ai/v1" Custom API base URL for SambaNova API requests. ​ language Language default: "Language.EN" Language of the audio input. Defaults to English. ​ prompt str Optional text to guide the model’s style or continue a previous segment. ​ temperature float Sampling temperature between 0 and 1. Lower values are more deterministic, higher values more creative. Defaults to 0.0. ​ sample_rate int Audio sample rate in Hz. If not provided, uses the pipeline’s sample rate. ​ Input The service processes audio data with the following requirements: PCM audio format. 16-bit depth. Single channel (mono). ​ Output Frames The service produces two types of frames during transcription: ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Detected language (if available) ​ ErrorFrame Generated when transcription errors occur, containing error details. ​ Methods ​ Set Model Copy Ask AI await service.set_model( "Whisper-Large-v3" ) See the STT base class methods for additional functionality. ​ Language Support SambaNova’s Whisper API supports a wide range of languages. The service automatically maps Language enum values to the appropriate Whisper language codes. Language Code Description Whisper Code Language.AF Afrikaans af Language.AR Arabic ar Language.HY Armenian hy Language.AZ Azerbaijani az Language.BE Belarusian be Language.BS Bosnian bs Language.BG Bulgarian bg Language.CA Catalan ca Language.ZH Chinese zh Language.HR Croatian hr Language.CS Czech cs Language.DA Danish da Language.NL Dutch nl Language.EN English en Language.ET Estonian et Language.FI Finnish fi Language.FR French fr Language.GL Galician gl Language.DE German de Language.EL Greek el Language.HE Hebrew he Language.HI Hindi hi Language.HU Hungarian hu Language.IS Icelandic is Language.ID Indonesian id Language.IT Italian it Language.JA Japanese ja Language.KN Kannada kn Language.KK Kazakh kk Language.KO Korean ko Language.LV Latvian lv Language.LT Lithuanian lt Language.MK Macedonian mk Language.MS Malay ms Language.MR Marathi mr Language.MI Maori mi Language.NE Nepali ne Language.NO Norwegian no Language.FA Persian fa Language.PL Polish pl Language.PT Portuguese pt Language.RO Romanian ro Language.RU Russian ru Language.SR Serbian sr Language.SK Slovak sk Language.SL Slovenian sl Language.ES Spanish es Language.SW Swahili sw Language.SV Swedish sv Language.TL Tagalog tl Language.TA Tamil ta Language.TH Thai th Language.TR Turkish tr Language.UK Ukrainian uk Language.UR Urdu ur Language.VI Vietnamese vi Language.CY Welsh cy SambaNova’s Whisper implementation supports language variants (like en-US , fr-CA ) by mapping them to their base language. For example, Language.EN_US and Language.EN_GB will both map to en . The service will automatically detect the language if none is specified, but specifying the language typically improves transcription accuracy. For the most up-to-date list of supported languages, refer to the SambaNova’s docs . ​ Usage Example Copy Ask AI from pipecat.services.sambanova.stt import SambaNovaSTTService from pipecat.transcriptions.language import Language # Configure service stt = SambaNovaSTTService( model = "Whisper-Large-v3" , api_key = "your-sambanova-api-key" , language = Language. EN , prompt = "Transcribe the following conversation" , temperature = 0.0 ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ Voice Activity Detection Integration This service inherits from SegmentedSTTService , which uses Voice Activity Detection (VAD) to identify speech segments for processing. This approach: Processes only actual speech, not silence or background noise. Maintains a small audio buffer (default 1 second) to capture speech that occurs slightly before VAD detection. Receives UserStartedSpeakingFrame and UserStoppedSpeakingFrame from a VAD component in the pipeline. Only sends complete utterances to the API when speech has ended. Ensure your transport includes a VAD component (like SileroVADAnalyzer ) to properly detect speech segments. ​ Metrics Support The service collects the following metrics: Time to First Byte (TTFB). Processing duration. API response time. ​ Notes Requires valid SambaNova API key. Uses SambaNova’s hosted Whisper model. Requires VAD component in transport. Processes complete utterances, not continuous audio. Handles API rate limiting. Automatic error handling. Thread-safe processing. ​ Error Handling The service handles common API errors including: Authentication errors. Rate limiting. Invalid audio format. Network connectivity issues. API timeouts. Errors are propagated through ErrorFrames with descriptive messages. OpenAI Speechmatics On this page Overview Installation Configuration Constructor Parameters Input Output Frames TranscriptionFrame ErrorFrame Methods Set Model Language Support Usage Example Voice Activity Detection Integration Metrics Support Notes Error Handling Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_sambanova_6bccf807.txt b/stt_sambanova_6bccf807.txt
new file mode 100644
index 0000000000000000000000000000000000000000..5d6fd830a1880947daa8afc57e9e1ee2bcfb6397
--- /dev/null
+++ b/stt_sambanova_6bccf807.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/sambanova#param-language-1
+Title: SambaNova (Whisper) - Pipecat
+==================================================
+
+SambaNova (Whisper) - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text SambaNova (Whisper) Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview SambaNovaSTTService provides speech-to-text capabilities using SambaNova’s hosted Whisper API. It offers high-accuracy transcription with minimal setup requirements. The service uses Voice Activity Detection (VAD) to process only speech segments, optimizing API usage and improving response time. ​ Installation To use SambaNovaSTTService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[sambanova]" You need to set up your SambaNova API key as an environment variable: SAMBANOVA_API_KEY . Get your SambaNova API key here . ​ Configuration ​ Constructor Parameters ​ model str default: "Whisper-Large-v3" Whisper model to use. Currently only “Whisper-Large-v3” is available. ​ api_key str Your SambaNova API key. If not provided, will use environment variable. ​ base_url str default: "https://api.sambanova.ai/v1" Custom API base URL for SambaNova API requests. ​ language Language default: "Language.EN" Language of the audio input. Defaults to English. ​ prompt str Optional text to guide the model’s style or continue a previous segment. ​ temperature float Sampling temperature between 0 and 1. Lower values are more deterministic, higher values more creative. Defaults to 0.0. ​ sample_rate int Audio sample rate in Hz. If not provided, uses the pipeline’s sample rate. ​ Input The service processes audio data with the following requirements: PCM audio format. 16-bit depth. Single channel (mono). ​ Output Frames The service produces two types of frames during transcription: ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Detected language (if available) ​ ErrorFrame Generated when transcription errors occur, containing error details. ​ Methods ​ Set Model Copy Ask AI await service.set_model( "Whisper-Large-v3" ) See the STT base class methods for additional functionality. ​ Language Support SambaNova’s Whisper API supports a wide range of languages. The service automatically maps Language enum values to the appropriate Whisper language codes. Language Code Description Whisper Code Language.AF Afrikaans af Language.AR Arabic ar Language.HY Armenian hy Language.AZ Azerbaijani az Language.BE Belarusian be Language.BS Bosnian bs Language.BG Bulgarian bg Language.CA Catalan ca Language.ZH Chinese zh Language.HR Croatian hr Language.CS Czech cs Language.DA Danish da Language.NL Dutch nl Language.EN English en Language.ET Estonian et Language.FI Finnish fi Language.FR French fr Language.GL Galician gl Language.DE German de Language.EL Greek el Language.HE Hebrew he Language.HI Hindi hi Language.HU Hungarian hu Language.IS Icelandic is Language.ID Indonesian id Language.IT Italian it Language.JA Japanese ja Language.KN Kannada kn Language.KK Kazakh kk Language.KO Korean ko Language.LV Latvian lv Language.LT Lithuanian lt Language.MK Macedonian mk Language.MS Malay ms Language.MR Marathi mr Language.MI Maori mi Language.NE Nepali ne Language.NO Norwegian no Language.FA Persian fa Language.PL Polish pl Language.PT Portuguese pt Language.RO Romanian ro Language.RU Russian ru Language.SR Serbian sr Language.SK Slovak sk Language.SL Slovenian sl Language.ES Spanish es Language.SW Swahili sw Language.SV Swedish sv Language.TL Tagalog tl Language.TA Tamil ta Language.TH Thai th Language.TR Turkish tr Language.UK Ukrainian uk Language.UR Urdu ur Language.VI Vietnamese vi Language.CY Welsh cy SambaNova’s Whisper implementation supports language variants (like en-US , fr-CA ) by mapping them to their base language. For example, Language.EN_US and Language.EN_GB will both map to en . The service will automatically detect the language if none is specified, but specifying the language typically improves transcription accuracy. For the most up-to-date list of supported languages, refer to the SambaNova’s docs . ​ Usage Example Copy Ask AI from pipecat.services.sambanova.stt import SambaNovaSTTService from pipecat.transcriptions.language import Language # Configure service stt = SambaNovaSTTService( model = "Whisper-Large-v3" , api_key = "your-sambanova-api-key" , language = Language. EN , prompt = "Transcribe the following conversation" , temperature = 0.0 ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ Voice Activity Detection Integration This service inherits from SegmentedSTTService , which uses Voice Activity Detection (VAD) to identify speech segments for processing. This approach: Processes only actual speech, not silence or background noise. Maintains a small audio buffer (default 1 second) to capture speech that occurs slightly before VAD detection. Receives UserStartedSpeakingFrame and UserStoppedSpeakingFrame from a VAD component in the pipeline. Only sends complete utterances to the API when speech has ended. Ensure your transport includes a VAD component (like SileroVADAnalyzer ) to properly detect speech segments. ​ Metrics Support The service collects the following metrics: Time to First Byte (TTFB). Processing duration. API response time. ​ Notes Requires valid SambaNova API key. Uses SambaNova’s hosted Whisper model. Requires VAD component in transport. Processes complete utterances, not continuous audio. Handles API rate limiting. Automatic error handling. Thread-safe processing. ​ Error Handling The service handles common API errors including: Authentication errors. Rate limiting. Invalid audio format. Network connectivity issues. API timeouts. Errors are propagated through ErrorFrames with descriptive messages. OpenAI Speechmatics On this page Overview Installation Configuration Constructor Parameters Input Output Frames TranscriptionFrame ErrorFrame Methods Set Model Language Support Usage Example Voice Activity Detection Integration Metrics Support Notes Error Handling Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_sambanova_7e5a0d0b.txt b/stt_sambanova_7e5a0d0b.txt
new file mode 100644
index 0000000000000000000000000000000000000000..0f9881ed8b41d10f5459ad5c7bf313068a9899fe
--- /dev/null
+++ b/stt_sambanova_7e5a0d0b.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/sambanova#param-sample-rate
+Title: SambaNova (Whisper) - Pipecat
+==================================================
+
+SambaNova (Whisper) - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text SambaNova (Whisper) Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview SambaNovaSTTService provides speech-to-text capabilities using SambaNova’s hosted Whisper API. It offers high-accuracy transcription with minimal setup requirements. The service uses Voice Activity Detection (VAD) to process only speech segments, optimizing API usage and improving response time. ​ Installation To use SambaNovaSTTService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[sambanova]" You need to set up your SambaNova API key as an environment variable: SAMBANOVA_API_KEY . Get your SambaNova API key here . ​ Configuration ​ Constructor Parameters ​ model str default: "Whisper-Large-v3" Whisper model to use. Currently only “Whisper-Large-v3” is available. ​ api_key str Your SambaNova API key. If not provided, will use environment variable. ​ base_url str default: "https://api.sambanova.ai/v1" Custom API base URL for SambaNova API requests. ​ language Language default: "Language.EN" Language of the audio input. Defaults to English. ​ prompt str Optional text to guide the model’s style or continue a previous segment. ​ temperature float Sampling temperature between 0 and 1. Lower values are more deterministic, higher values more creative. Defaults to 0.0. ​ sample_rate int Audio sample rate in Hz. If not provided, uses the pipeline’s sample rate. ​ Input The service processes audio data with the following requirements: PCM audio format. 16-bit depth. Single channel (mono). ​ Output Frames The service produces two types of frames during transcription: ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Detected language (if available) ​ ErrorFrame Generated when transcription errors occur, containing error details. ​ Methods ​ Set Model Copy Ask AI await service.set_model( "Whisper-Large-v3" ) See the STT base class methods for additional functionality. ​ Language Support SambaNova’s Whisper API supports a wide range of languages. The service automatically maps Language enum values to the appropriate Whisper language codes. Language Code Description Whisper Code Language.AF Afrikaans af Language.AR Arabic ar Language.HY Armenian hy Language.AZ Azerbaijani az Language.BE Belarusian be Language.BS Bosnian bs Language.BG Bulgarian bg Language.CA Catalan ca Language.ZH Chinese zh Language.HR Croatian hr Language.CS Czech cs Language.DA Danish da Language.NL Dutch nl Language.EN English en Language.ET Estonian et Language.FI Finnish fi Language.FR French fr Language.GL Galician gl Language.DE German de Language.EL Greek el Language.HE Hebrew he Language.HI Hindi hi Language.HU Hungarian hu Language.IS Icelandic is Language.ID Indonesian id Language.IT Italian it Language.JA Japanese ja Language.KN Kannada kn Language.KK Kazakh kk Language.KO Korean ko Language.LV Latvian lv Language.LT Lithuanian lt Language.MK Macedonian mk Language.MS Malay ms Language.MR Marathi mr Language.MI Maori mi Language.NE Nepali ne Language.NO Norwegian no Language.FA Persian fa Language.PL Polish pl Language.PT Portuguese pt Language.RO Romanian ro Language.RU Russian ru Language.SR Serbian sr Language.SK Slovak sk Language.SL Slovenian sl Language.ES Spanish es Language.SW Swahili sw Language.SV Swedish sv Language.TL Tagalog tl Language.TA Tamil ta Language.TH Thai th Language.TR Turkish tr Language.UK Ukrainian uk Language.UR Urdu ur Language.VI Vietnamese vi Language.CY Welsh cy SambaNova’s Whisper implementation supports language variants (like en-US , fr-CA ) by mapping them to their base language. For example, Language.EN_US and Language.EN_GB will both map to en . The service will automatically detect the language if none is specified, but specifying the language typically improves transcription accuracy. For the most up-to-date list of supported languages, refer to the SambaNova’s docs . ​ Usage Example Copy Ask AI from pipecat.services.sambanova.stt import SambaNovaSTTService from pipecat.transcriptions.language import Language # Configure service stt = SambaNovaSTTService( model = "Whisper-Large-v3" , api_key = "your-sambanova-api-key" , language = Language. EN , prompt = "Transcribe the following conversation" , temperature = 0.0 ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ Voice Activity Detection Integration This service inherits from SegmentedSTTService , which uses Voice Activity Detection (VAD) to identify speech segments for processing. This approach: Processes only actual speech, not silence or background noise. Maintains a small audio buffer (default 1 second) to capture speech that occurs slightly before VAD detection. Receives UserStartedSpeakingFrame and UserStoppedSpeakingFrame from a VAD component in the pipeline. Only sends complete utterances to the API when speech has ended. Ensure your transport includes a VAD component (like SileroVADAnalyzer ) to properly detect speech segments. ​ Metrics Support The service collects the following metrics: Time to First Byte (TTFB). Processing duration. API response time. ​ Notes Requires valid SambaNova API key. Uses SambaNova’s hosted Whisper model. Requires VAD component in transport. Processes complete utterances, not continuous audio. Handles API rate limiting. Automatic error handling. Thread-safe processing. ​ Error Handling The service handles common API errors including: Authentication errors. Rate limiting. Invalid audio format. Network connectivity issues. API timeouts. Errors are propagated through ErrorFrames with descriptive messages. OpenAI Speechmatics On this page Overview Installation Configuration Constructor Parameters Input Output Frames TranscriptionFrame ErrorFrame Methods Set Model Language Support Usage Example Voice Activity Detection Integration Metrics Support Notes Error Handling Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_sambanova_93e2170f.txt b/stt_sambanova_93e2170f.txt
new file mode 100644
index 0000000000000000000000000000000000000000..fd8e1301d3c9a3213d53ca465e3987d318703b23
--- /dev/null
+++ b/stt_sambanova_93e2170f.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/sambanova#param-language
+Title: SambaNova (Whisper) - Pipecat
+==================================================
+
+SambaNova (Whisper) - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text SambaNova (Whisper) Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview SambaNovaSTTService provides speech-to-text capabilities using SambaNova’s hosted Whisper API. It offers high-accuracy transcription with minimal setup requirements. The service uses Voice Activity Detection (VAD) to process only speech segments, optimizing API usage and improving response time. ​ Installation To use SambaNovaSTTService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[sambanova]" You need to set up your SambaNova API key as an environment variable: SAMBANOVA_API_KEY . Get your SambaNova API key here . ​ Configuration ​ Constructor Parameters ​ model str default: "Whisper-Large-v3" Whisper model to use. Currently only “Whisper-Large-v3” is available. ​ api_key str Your SambaNova API key. If not provided, will use environment variable. ​ base_url str default: "https://api.sambanova.ai/v1" Custom API base URL for SambaNova API requests. ​ language Language default: "Language.EN" Language of the audio input. Defaults to English. ​ prompt str Optional text to guide the model’s style or continue a previous segment. ​ temperature float Sampling temperature between 0 and 1. Lower values are more deterministic, higher values more creative. Defaults to 0.0. ​ sample_rate int Audio sample rate in Hz. If not provided, uses the pipeline’s sample rate. ​ Input The service processes audio data with the following requirements: PCM audio format. 16-bit depth. Single channel (mono). ​ Output Frames The service produces two types of frames during transcription: ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Detected language (if available) ​ ErrorFrame Generated when transcription errors occur, containing error details. ​ Methods ​ Set Model Copy Ask AI await service.set_model( "Whisper-Large-v3" ) See the STT base class methods for additional functionality. ​ Language Support SambaNova’s Whisper API supports a wide range of languages. The service automatically maps Language enum values to the appropriate Whisper language codes. Language Code Description Whisper Code Language.AF Afrikaans af Language.AR Arabic ar Language.HY Armenian hy Language.AZ Azerbaijani az Language.BE Belarusian be Language.BS Bosnian bs Language.BG Bulgarian bg Language.CA Catalan ca Language.ZH Chinese zh Language.HR Croatian hr Language.CS Czech cs Language.DA Danish da Language.NL Dutch nl Language.EN English en Language.ET Estonian et Language.FI Finnish fi Language.FR French fr Language.GL Galician gl Language.DE German de Language.EL Greek el Language.HE Hebrew he Language.HI Hindi hi Language.HU Hungarian hu Language.IS Icelandic is Language.ID Indonesian id Language.IT Italian it Language.JA Japanese ja Language.KN Kannada kn Language.KK Kazakh kk Language.KO Korean ko Language.LV Latvian lv Language.LT Lithuanian lt Language.MK Macedonian mk Language.MS Malay ms Language.MR Marathi mr Language.MI Maori mi Language.NE Nepali ne Language.NO Norwegian no Language.FA Persian fa Language.PL Polish pl Language.PT Portuguese pt Language.RO Romanian ro Language.RU Russian ru Language.SR Serbian sr Language.SK Slovak sk Language.SL Slovenian sl Language.ES Spanish es Language.SW Swahili sw Language.SV Swedish sv Language.TL Tagalog tl Language.TA Tamil ta Language.TH Thai th Language.TR Turkish tr Language.UK Ukrainian uk Language.UR Urdu ur Language.VI Vietnamese vi Language.CY Welsh cy SambaNova’s Whisper implementation supports language variants (like en-US , fr-CA ) by mapping them to their base language. For example, Language.EN_US and Language.EN_GB will both map to en . The service will automatically detect the language if none is specified, but specifying the language typically improves transcription accuracy. For the most up-to-date list of supported languages, refer to the SambaNova’s docs . ​ Usage Example Copy Ask AI from pipecat.services.sambanova.stt import SambaNovaSTTService from pipecat.transcriptions.language import Language # Configure service stt = SambaNovaSTTService( model = "Whisper-Large-v3" , api_key = "your-sambanova-api-key" , language = Language. EN , prompt = "Transcribe the following conversation" , temperature = 0.0 ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ Voice Activity Detection Integration This service inherits from SegmentedSTTService , which uses Voice Activity Detection (VAD) to identify speech segments for processing. This approach: Processes only actual speech, not silence or background noise. Maintains a small audio buffer (default 1 second) to capture speech that occurs slightly before VAD detection. Receives UserStartedSpeakingFrame and UserStoppedSpeakingFrame from a VAD component in the pipeline. Only sends complete utterances to the API when speech has ended. Ensure your transport includes a VAD component (like SileroVADAnalyzer ) to properly detect speech segments. ​ Metrics Support The service collects the following metrics: Time to First Byte (TTFB). Processing duration. API response time. ​ Notes Requires valid SambaNova API key. Uses SambaNova’s hosted Whisper model. Requires VAD component in transport. Processes complete utterances, not continuous audio. Handles API rate limiting. Automatic error handling. Thread-safe processing. ​ Error Handling The service handles common API errors including: Authentication errors. Rate limiting. Invalid audio format. Network connectivity issues. API timeouts. Errors are propagated through ErrorFrames with descriptive messages. OpenAI Speechmatics On this page Overview Installation Configuration Constructor Parameters Input Output Frames TranscriptionFrame ErrorFrame Methods Set Model Language Support Usage Example Voice Activity Detection Integration Metrics Support Notes Error Handling Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_sambanova_96876355.txt b/stt_sambanova_96876355.txt
new file mode 100644
index 0000000000000000000000000000000000000000..929d95315d3649ae22c4fa5c6536b4979570eed9
--- /dev/null
+++ b/stt_sambanova_96876355.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/sambanova#param-temperature
+Title: SambaNova (Whisper) - Pipecat
+==================================================
+
+SambaNova (Whisper) - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text SambaNova (Whisper) Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview SambaNovaSTTService provides speech-to-text capabilities using SambaNova’s hosted Whisper API. It offers high-accuracy transcription with minimal setup requirements. The service uses Voice Activity Detection (VAD) to process only speech segments, optimizing API usage and improving response time. ​ Installation To use SambaNovaSTTService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[sambanova]" You need to set up your SambaNova API key as an environment variable: SAMBANOVA_API_KEY . Get your SambaNova API key here . ​ Configuration ​ Constructor Parameters ​ model str default: "Whisper-Large-v3" Whisper model to use. Currently only “Whisper-Large-v3” is available. ​ api_key str Your SambaNova API key. If not provided, will use environment variable. ​ base_url str default: "https://api.sambanova.ai/v1" Custom API base URL for SambaNova API requests. ​ language Language default: "Language.EN" Language of the audio input. Defaults to English. ​ prompt str Optional text to guide the model’s style or continue a previous segment. ​ temperature float Sampling temperature between 0 and 1. Lower values are more deterministic, higher values more creative. Defaults to 0.0. ​ sample_rate int Audio sample rate in Hz. If not provided, uses the pipeline’s sample rate. ​ Input The service processes audio data with the following requirements: PCM audio format. 16-bit depth. Single channel (mono). ​ Output Frames The service produces two types of frames during transcription: ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Detected language (if available) ​ ErrorFrame Generated when transcription errors occur, containing error details. ​ Methods ​ Set Model Copy Ask AI await service.set_model( "Whisper-Large-v3" ) See the STT base class methods for additional functionality. ​ Language Support SambaNova’s Whisper API supports a wide range of languages. The service automatically maps Language enum values to the appropriate Whisper language codes. Language Code Description Whisper Code Language.AF Afrikaans af Language.AR Arabic ar Language.HY Armenian hy Language.AZ Azerbaijani az Language.BE Belarusian be Language.BS Bosnian bs Language.BG Bulgarian bg Language.CA Catalan ca Language.ZH Chinese zh Language.HR Croatian hr Language.CS Czech cs Language.DA Danish da Language.NL Dutch nl Language.EN English en Language.ET Estonian et Language.FI Finnish fi Language.FR French fr Language.GL Galician gl Language.DE German de Language.EL Greek el Language.HE Hebrew he Language.HI Hindi hi Language.HU Hungarian hu Language.IS Icelandic is Language.ID Indonesian id Language.IT Italian it Language.JA Japanese ja Language.KN Kannada kn Language.KK Kazakh kk Language.KO Korean ko Language.LV Latvian lv Language.LT Lithuanian lt Language.MK Macedonian mk Language.MS Malay ms Language.MR Marathi mr Language.MI Maori mi Language.NE Nepali ne Language.NO Norwegian no Language.FA Persian fa Language.PL Polish pl Language.PT Portuguese pt Language.RO Romanian ro Language.RU Russian ru Language.SR Serbian sr Language.SK Slovak sk Language.SL Slovenian sl Language.ES Spanish es Language.SW Swahili sw Language.SV Swedish sv Language.TL Tagalog tl Language.TA Tamil ta Language.TH Thai th Language.TR Turkish tr Language.UK Ukrainian uk Language.UR Urdu ur Language.VI Vietnamese vi Language.CY Welsh cy SambaNova’s Whisper implementation supports language variants (like en-US , fr-CA ) by mapping them to their base language. For example, Language.EN_US and Language.EN_GB will both map to en . The service will automatically detect the language if none is specified, but specifying the language typically improves transcription accuracy. For the most up-to-date list of supported languages, refer to the SambaNova’s docs . ​ Usage Example Copy Ask AI from pipecat.services.sambanova.stt import SambaNovaSTTService from pipecat.transcriptions.language import Language # Configure service stt = SambaNovaSTTService( model = "Whisper-Large-v3" , api_key = "your-sambanova-api-key" , language = Language. EN , prompt = "Transcribe the following conversation" , temperature = 0.0 ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ Voice Activity Detection Integration This service inherits from SegmentedSTTService , which uses Voice Activity Detection (VAD) to identify speech segments for processing. This approach: Processes only actual speech, not silence or background noise. Maintains a small audio buffer (default 1 second) to capture speech that occurs slightly before VAD detection. Receives UserStartedSpeakingFrame and UserStoppedSpeakingFrame from a VAD component in the pipeline. Only sends complete utterances to the API when speech has ended. Ensure your transport includes a VAD component (like SileroVADAnalyzer ) to properly detect speech segments. ​ Metrics Support The service collects the following metrics: Time to First Byte (TTFB). Processing duration. API response time. ​ Notes Requires valid SambaNova API key. Uses SambaNova’s hosted Whisper model. Requires VAD component in transport. Processes complete utterances, not continuous audio. Handles API rate limiting. Automatic error handling. Thread-safe processing. ​ Error Handling The service handles common API errors including: Authentication errors. Rate limiting. Invalid audio format. Network connectivity issues. API timeouts. Errors are propagated through ErrorFrames with descriptive messages. OpenAI Speechmatics On this page Overview Installation Configuration Constructor Parameters Input Output Frames TranscriptionFrame ErrorFrame Methods Set Model Language Support Usage Example Voice Activity Detection Integration Metrics Support Notes Error Handling Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_sambanova_a5d1563a.txt b/stt_sambanova_a5d1563a.txt
new file mode 100644
index 0000000000000000000000000000000000000000..f68b31286b7f9019415255e5c47357581d18da5b
--- /dev/null
+++ b/stt_sambanova_a5d1563a.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/sambanova#methods
+Title: SambaNova (Whisper) - Pipecat
+==================================================
+
+SambaNova (Whisper) - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text SambaNova (Whisper) Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview SambaNovaSTTService provides speech-to-text capabilities using SambaNova’s hosted Whisper API. It offers high-accuracy transcription with minimal setup requirements. The service uses Voice Activity Detection (VAD) to process only speech segments, optimizing API usage and improving response time. ​ Installation To use SambaNovaSTTService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[sambanova]" You need to set up your SambaNova API key as an environment variable: SAMBANOVA_API_KEY . Get your SambaNova API key here . ​ Configuration ​ Constructor Parameters ​ model str default: "Whisper-Large-v3" Whisper model to use. Currently only “Whisper-Large-v3” is available. ​ api_key str Your SambaNova API key. If not provided, will use environment variable. ​ base_url str default: "https://api.sambanova.ai/v1" Custom API base URL for SambaNova API requests. ​ language Language default: "Language.EN" Language of the audio input. Defaults to English. ​ prompt str Optional text to guide the model’s style or continue a previous segment. ​ temperature float Sampling temperature between 0 and 1. Lower values are more deterministic, higher values more creative. Defaults to 0.0. ​ sample_rate int Audio sample rate in Hz. If not provided, uses the pipeline’s sample rate. ​ Input The service processes audio data with the following requirements: PCM audio format. 16-bit depth. Single channel (mono). ​ Output Frames The service produces two types of frames during transcription: ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Detected language (if available) ​ ErrorFrame Generated when transcription errors occur, containing error details. ​ Methods ​ Set Model Copy Ask AI await service.set_model( "Whisper-Large-v3" ) See the STT base class methods for additional functionality. ​ Language Support SambaNova’s Whisper API supports a wide range of languages. The service automatically maps Language enum values to the appropriate Whisper language codes. Language Code Description Whisper Code Language.AF Afrikaans af Language.AR Arabic ar Language.HY Armenian hy Language.AZ Azerbaijani az Language.BE Belarusian be Language.BS Bosnian bs Language.BG Bulgarian bg Language.CA Catalan ca Language.ZH Chinese zh Language.HR Croatian hr Language.CS Czech cs Language.DA Danish da Language.NL Dutch nl Language.EN English en Language.ET Estonian et Language.FI Finnish fi Language.FR French fr Language.GL Galician gl Language.DE German de Language.EL Greek el Language.HE Hebrew he Language.HI Hindi hi Language.HU Hungarian hu Language.IS Icelandic is Language.ID Indonesian id Language.IT Italian it Language.JA Japanese ja Language.KN Kannada kn Language.KK Kazakh kk Language.KO Korean ko Language.LV Latvian lv Language.LT Lithuanian lt Language.MK Macedonian mk Language.MS Malay ms Language.MR Marathi mr Language.MI Maori mi Language.NE Nepali ne Language.NO Norwegian no Language.FA Persian fa Language.PL Polish pl Language.PT Portuguese pt Language.RO Romanian ro Language.RU Russian ru Language.SR Serbian sr Language.SK Slovak sk Language.SL Slovenian sl Language.ES Spanish es Language.SW Swahili sw Language.SV Swedish sv Language.TL Tagalog tl Language.TA Tamil ta Language.TH Thai th Language.TR Turkish tr Language.UK Ukrainian uk Language.UR Urdu ur Language.VI Vietnamese vi Language.CY Welsh cy SambaNova’s Whisper implementation supports language variants (like en-US , fr-CA ) by mapping them to their base language. For example, Language.EN_US and Language.EN_GB will both map to en . The service will automatically detect the language if none is specified, but specifying the language typically improves transcription accuracy. For the most up-to-date list of supported languages, refer to the SambaNova’s docs . ​ Usage Example Copy Ask AI from pipecat.services.sambanova.stt import SambaNovaSTTService from pipecat.transcriptions.language import Language # Configure service stt = SambaNovaSTTService( model = "Whisper-Large-v3" , api_key = "your-sambanova-api-key" , language = Language. EN , prompt = "Transcribe the following conversation" , temperature = 0.0 ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ Voice Activity Detection Integration This service inherits from SegmentedSTTService , which uses Voice Activity Detection (VAD) to identify speech segments for processing. This approach: Processes only actual speech, not silence or background noise. Maintains a small audio buffer (default 1 second) to capture speech that occurs slightly before VAD detection. Receives UserStartedSpeakingFrame and UserStoppedSpeakingFrame from a VAD component in the pipeline. Only sends complete utterances to the API when speech has ended. Ensure your transport includes a VAD component (like SileroVADAnalyzer ) to properly detect speech segments. ​ Metrics Support The service collects the following metrics: Time to First Byte (TTFB). Processing duration. API response time. ​ Notes Requires valid SambaNova API key. Uses SambaNova’s hosted Whisper model. Requires VAD component in transport. Processes complete utterances, not continuous audio. Handles API rate limiting. Automatic error handling. Thread-safe processing. ​ Error Handling The service handles common API errors including: Authentication errors. Rate limiting. Invalid audio format. Network connectivity issues. API timeouts. Errors are propagated through ErrorFrames with descriptive messages. OpenAI Speechmatics On this page Overview Installation Configuration Constructor Parameters Input Output Frames TranscriptionFrame ErrorFrame Methods Set Model Language Support Usage Example Voice Activity Detection Integration Metrics Support Notes Error Handling Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_sambanova_a88c4080.txt b/stt_sambanova_a88c4080.txt
new file mode 100644
index 0000000000000000000000000000000000000000..f7c5dd0d6b5983333a8e9b2418a2b6a04593a73a
--- /dev/null
+++ b/stt_sambanova_a88c4080.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/sambanova#set-model
+Title: SambaNova (Whisper) - Pipecat
+==================================================
+
+SambaNova (Whisper) - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text SambaNova (Whisper) Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview SambaNovaSTTService provides speech-to-text capabilities using SambaNova’s hosted Whisper API. It offers high-accuracy transcription with minimal setup requirements. The service uses Voice Activity Detection (VAD) to process only speech segments, optimizing API usage and improving response time. ​ Installation To use SambaNovaSTTService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[sambanova]" You need to set up your SambaNova API key as an environment variable: SAMBANOVA_API_KEY . Get your SambaNova API key here . ​ Configuration ​ Constructor Parameters ​ model str default: "Whisper-Large-v3" Whisper model to use. Currently only “Whisper-Large-v3” is available. ​ api_key str Your SambaNova API key. If not provided, will use environment variable. ​ base_url str default: "https://api.sambanova.ai/v1" Custom API base URL for SambaNova API requests. ​ language Language default: "Language.EN" Language of the audio input. Defaults to English. ​ prompt str Optional text to guide the model’s style or continue a previous segment. ​ temperature float Sampling temperature between 0 and 1. Lower values are more deterministic, higher values more creative. Defaults to 0.0. ​ sample_rate int Audio sample rate in Hz. If not provided, uses the pipeline’s sample rate. ​ Input The service processes audio data with the following requirements: PCM audio format. 16-bit depth. Single channel (mono). ​ Output Frames The service produces two types of frames during transcription: ​ TranscriptionFrame Generated for final transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ language Language Detected language (if available) ​ ErrorFrame Generated when transcription errors occur, containing error details. ​ Methods ​ Set Model Copy Ask AI await service.set_model( "Whisper-Large-v3" ) See the STT base class methods for additional functionality. ​ Language Support SambaNova’s Whisper API supports a wide range of languages. The service automatically maps Language enum values to the appropriate Whisper language codes. Language Code Description Whisper Code Language.AF Afrikaans af Language.AR Arabic ar Language.HY Armenian hy Language.AZ Azerbaijani az Language.BE Belarusian be Language.BS Bosnian bs Language.BG Bulgarian bg Language.CA Catalan ca Language.ZH Chinese zh Language.HR Croatian hr Language.CS Czech cs Language.DA Danish da Language.NL Dutch nl Language.EN English en Language.ET Estonian et Language.FI Finnish fi Language.FR French fr Language.GL Galician gl Language.DE German de Language.EL Greek el Language.HE Hebrew he Language.HI Hindi hi Language.HU Hungarian hu Language.IS Icelandic is Language.ID Indonesian id Language.IT Italian it Language.JA Japanese ja Language.KN Kannada kn Language.KK Kazakh kk Language.KO Korean ko Language.LV Latvian lv Language.LT Lithuanian lt Language.MK Macedonian mk Language.MS Malay ms Language.MR Marathi mr Language.MI Maori mi Language.NE Nepali ne Language.NO Norwegian no Language.FA Persian fa Language.PL Polish pl Language.PT Portuguese pt Language.RO Romanian ro Language.RU Russian ru Language.SR Serbian sr Language.SK Slovak sk Language.SL Slovenian sl Language.ES Spanish es Language.SW Swahili sw Language.SV Swedish sv Language.TL Tagalog tl Language.TA Tamil ta Language.TH Thai th Language.TR Turkish tr Language.UK Ukrainian uk Language.UR Urdu ur Language.VI Vietnamese vi Language.CY Welsh cy SambaNova’s Whisper implementation supports language variants (like en-US , fr-CA ) by mapping them to their base language. For example, Language.EN_US and Language.EN_GB will both map to en . The service will automatically detect the language if none is specified, but specifying the language typically improves transcription accuracy. For the most up-to-date list of supported languages, refer to the SambaNova’s docs . ​ Usage Example Copy Ask AI from pipecat.services.sambanova.stt import SambaNovaSTTService from pipecat.transcriptions.language import Language # Configure service stt = SambaNovaSTTService( model = "Whisper-Large-v3" , api_key = "your-sambanova-api-key" , language = Language. EN , prompt = "Transcribe the following conversation" , temperature = 0.0 ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ Voice Activity Detection Integration This service inherits from SegmentedSTTService , which uses Voice Activity Detection (VAD) to identify speech segments for processing. This approach: Processes only actual speech, not silence or background noise. Maintains a small audio buffer (default 1 second) to capture speech that occurs slightly before VAD detection. Receives UserStartedSpeakingFrame and UserStoppedSpeakingFrame from a VAD component in the pipeline. Only sends complete utterances to the API when speech has ended. Ensure your transport includes a VAD component (like SileroVADAnalyzer ) to properly detect speech segments. ​ Metrics Support The service collects the following metrics: Time to First Byte (TTFB). Processing duration. API response time. ​ Notes Requires valid SambaNova API key. Uses SambaNova’s hosted Whisper model. Requires VAD component in transport. Processes complete utterances, not continuous audio. Handles API rate limiting. Automatic error handling. Thread-safe processing. ​ Error Handling The service handles common API errors including: Authentication errors. Rate limiting. Invalid audio format. Network connectivity issues. API timeouts. Errors are propagated through ErrorFrames with descriptive messages. OpenAI Speechmatics On this page Overview Installation Configuration Constructor Parameters Input Output Frames TranscriptionFrame ErrorFrame Methods Set Model Language Support Usage Example Voice Activity Detection Integration Metrics Support Notes Error Handling Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_speechmatics_0729e31f.txt b/stt_speechmatics_0729e31f.txt
new file mode 100644
index 0000000000000000000000000000000000000000..70da92503fd03e9db4c6dfbb5adb7ded45e58dc0
--- /dev/null
+++ b/stt_speechmatics_0729e31f.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/speechmatics#language-support
+Title: Speechmatics - Pipecat
+==================================================
+
+Speechmatics - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text Speechmatics Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview SpeechmaticsSTTService enables real-time speech transcription using Speechmatics’ WebSocket API with partial + final results, speaker diarization, and end of utterance detection (VAD). API Reference Complete API documentation and method details Speechmatics Docs Official Speechmatics documentation and features Speaker Diarization Separating out different speakers in the audio Example Code Working example with interruption handling ​ Installation To use SpeechmaticsSTTService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[speechmatics]" You’ll also need to set up your Speechmatics API key as an environment variable: SPEECHMATICS_API_KEY . Get your API key from the Speechmatics Portal . ​ Frames ​ Input InputAudioRawFrame - Raw PCM audio data (16-bit, 16kHz, mono) STTUpdateSettingsFrame - Runtime transcription configuration updates STTMuteFrame - Mute audio input for transcription ​ Output InterimTranscriptionFrame - Real-time transcription updates TranscriptionFrame - Final transcription results ErrorFrame - Connection or processing errors ​ Endpoints Speechmatics STT supports the following endpoints (defaults to EU2 ): Region Environment STT Endpoint EU EU1 wss://eu1.rt.speechmatics.com/ EU EU2 wss://eu2.rt.speechmatics.com/ US US1 wss://us1.rt.speechmatics.com/ ​ Feature Discovery To check the languages and features supported by Speechmatics STT, you can use the following code: Copy Ask AI curl "https://eu2.rt.speechmatics.com/v1/discovery/features" ​ Language Support Refer to the Speechmatics docs for more information on supported languages. Speechmatics STT supports the following languages and regional variants. Setting a language can be done using the language parameter when creating the STT object. The exception to this is English / Mandarin which has the code cmn_en and must be set using the language_code parameter. Language Code Description Locales Domain Options Language.AR Arabic - - Language.BA Bashkir - - Language.EU Basque - - Language.BE Belarusian - - Language.BG Bulgarian - - Language.BN Bengali - - Language.YUE Cantonese - - Language.CA Catalan - - Language.HR Croatian - - Language.CS Czech - - Language.DA Danish - - Language.NL Dutch - - Language.EN English en-US , en-GB , en-AU finance Language.EO Esperanto - - Language.ET Estonian - - Language.FA Persian - - Language.FI Finnish - - Language.FR French - - Language.GL Galician - - Language.DE German - - Language.EL Greek - - Language.HE Hebrew - - Language.HI Hindi - - Language.HU Hungarian - - Language.IA Interlingua - - Language.IT Italian - - Language.ID Indonesian - - Language.GA Irish - - Language.JA Japanese - - Language.KO Korean - - Language.LV Latvian - - Language.LT Lithuanian - - Language.MS Malay - - Language.MT Maltese - - Language.CMN Mandarin cmn-Hans , cmn-Hant - cmn_en English / Mandarin - - Language.MR Marathi - - Language.MN Mongolian - - Language.NO Norwegian - - Language.PL Polish - - Language.PT Portuguese - - Language.RO Romanian - - Language.RU Russian - - Language.SK Slovakian - - Language.SL Slovenian - - Language.ES Spanish - bilingual-en Language.SV Swedish - - Language.SW Swahili - - Language.TA Tamil - - Language.TH Thai - - Language.TR Turkish - - Language.UG Uyghur - - Language.UK Ukrainian - - Language.UR Urdu - - Language.VI Vietnamese - - Language.CY Welsh - - ​ Translation Support Speechmatics supports the translation of transcribed output into the following languages: Language Code Description Language.BG Bulgarian Language.CA Catalan Language.CMN Mandarin Language.CS Czech Language.DA Danish Language.DE German Language.EL Greek Language.EN English Language.ES Spanish Language.ET Estonian Language.FI Finnish Language.FR French Language.GL Galician Language.HI Hindi Language.HR Croatian Language.HU Hungarian Language.ID Indonesian Language.IT Italian Language.JA Japanese Language.KO Korean Language.LT Lithuanian Language.LV Latvian Language.MS Malay Language.NL Dutch Language.NO Norwegian Language.PL Polish Language.PT Portuguese Language.RO Romanian Language.RU Russian Language.SK Slovakian Language.SL Slovenian Language.SV Swedish Language.TR Turkish Language.UK Ukrainian Language.VI Vietnamese ​ Speaker Diarization Speechmatics STT supports speaker diarization, which separates out different speakers in the audio. The identity of each speaker is returned in the TranscriptionFrame objects in the user_id attribute. To enable this feature, set enable_speaker_diarization to True . Additionally, if a text_format is provided, then the text output for the TranscriptionFrame will be formatted to this specification. Your system context can then be updated to include information about this format to understand which speaker spoke which words. For example, if you have text_format = <{speaker_id}>{text}</{speaker_id}> , then the output would be <S1>Good morning.</S1> . ​ Available attributes Attribute Description Example speaker_id The ID of the speaker S1 text The transcribed text Good morning. ​ Usage Example Copy Ask AI from pipecat.services.speechmatics.stt import SpeechmaticsSTTService from pipecat.transcriptions.language import Language # Configure service stt = SpeechmaticsSTTService( api_key = "your-api-key" , language = Language. FR , ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, tts, transport.output() ]) ​ Additional Notes Connection Management : Automatically handles WebSocket connections and reconnections Sample Rate : The default sample rate of 16000 in pcm_s16le format VAD Integration : Supports Speechmatics’ built-in VAD and end of utterance detection SambaNova (Whisper) Ultravox On this page Overview Installation Frames Input Output Endpoints Feature Discovery Language Support Translation Support Speaker Diarization Available attributes Usage Example Additional Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_speechmatics_deb1327d.txt b/stt_speechmatics_deb1327d.txt
new file mode 100644
index 0000000000000000000000000000000000000000..f83ec3dd25efec0307fd0d744381799161ed2c55
--- /dev/null
+++ b/stt_speechmatics_deb1327d.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/speechmatics#overview
+Title: Speechmatics - Pipecat
+==================================================
+
+Speechmatics - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text Speechmatics Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview SpeechmaticsSTTService enables real-time speech transcription using Speechmatics’ WebSocket API with partial + final results, speaker diarization, and end of utterance detection (VAD). API Reference Complete API documentation and method details Speechmatics Docs Official Speechmatics documentation and features Speaker Diarization Separating out different speakers in the audio Example Code Working example with interruption handling ​ Installation To use SpeechmaticsSTTService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[speechmatics]" You’ll also need to set up your Speechmatics API key as an environment variable: SPEECHMATICS_API_KEY . Get your API key from the Speechmatics Portal . ​ Frames ​ Input InputAudioRawFrame - Raw PCM audio data (16-bit, 16kHz, mono) STTUpdateSettingsFrame - Runtime transcription configuration updates STTMuteFrame - Mute audio input for transcription ​ Output InterimTranscriptionFrame - Real-time transcription updates TranscriptionFrame - Final transcription results ErrorFrame - Connection or processing errors ​ Endpoints Speechmatics STT supports the following endpoints (defaults to EU2 ): Region Environment STT Endpoint EU EU1 wss://eu1.rt.speechmatics.com/ EU EU2 wss://eu2.rt.speechmatics.com/ US US1 wss://us1.rt.speechmatics.com/ ​ Feature Discovery To check the languages and features supported by Speechmatics STT, you can use the following code: Copy Ask AI curl "https://eu2.rt.speechmatics.com/v1/discovery/features" ​ Language Support Refer to the Speechmatics docs for more information on supported languages. Speechmatics STT supports the following languages and regional variants. Setting a language can be done using the language parameter when creating the STT object. The exception to this is English / Mandarin which has the code cmn_en and must be set using the language_code parameter. Language Code Description Locales Domain Options Language.AR Arabic - - Language.BA Bashkir - - Language.EU Basque - - Language.BE Belarusian - - Language.BG Bulgarian - - Language.BN Bengali - - Language.YUE Cantonese - - Language.CA Catalan - - Language.HR Croatian - - Language.CS Czech - - Language.DA Danish - - Language.NL Dutch - - Language.EN English en-US , en-GB , en-AU finance Language.EO Esperanto - - Language.ET Estonian - - Language.FA Persian - - Language.FI Finnish - - Language.FR French - - Language.GL Galician - - Language.DE German - - Language.EL Greek - - Language.HE Hebrew - - Language.HI Hindi - - Language.HU Hungarian - - Language.IA Interlingua - - Language.IT Italian - - Language.ID Indonesian - - Language.GA Irish - - Language.JA Japanese - - Language.KO Korean - - Language.LV Latvian - - Language.LT Lithuanian - - Language.MS Malay - - Language.MT Maltese - - Language.CMN Mandarin cmn-Hans , cmn-Hant - cmn_en English / Mandarin - - Language.MR Marathi - - Language.MN Mongolian - - Language.NO Norwegian - - Language.PL Polish - - Language.PT Portuguese - - Language.RO Romanian - - Language.RU Russian - - Language.SK Slovakian - - Language.SL Slovenian - - Language.ES Spanish - bilingual-en Language.SV Swedish - - Language.SW Swahili - - Language.TA Tamil - - Language.TH Thai - - Language.TR Turkish - - Language.UG Uyghur - - Language.UK Ukrainian - - Language.UR Urdu - - Language.VI Vietnamese - - Language.CY Welsh - - ​ Translation Support Speechmatics supports the translation of transcribed output into the following languages: Language Code Description Language.BG Bulgarian Language.CA Catalan Language.CMN Mandarin Language.CS Czech Language.DA Danish Language.DE German Language.EL Greek Language.EN English Language.ES Spanish Language.ET Estonian Language.FI Finnish Language.FR French Language.GL Galician Language.HI Hindi Language.HR Croatian Language.HU Hungarian Language.ID Indonesian Language.IT Italian Language.JA Japanese Language.KO Korean Language.LT Lithuanian Language.LV Latvian Language.MS Malay Language.NL Dutch Language.NO Norwegian Language.PL Polish Language.PT Portuguese Language.RO Romanian Language.RU Russian Language.SK Slovakian Language.SL Slovenian Language.SV Swedish Language.TR Turkish Language.UK Ukrainian Language.VI Vietnamese ​ Speaker Diarization Speechmatics STT supports speaker diarization, which separates out different speakers in the audio. The identity of each speaker is returned in the TranscriptionFrame objects in the user_id attribute. To enable this feature, set enable_speaker_diarization to True . Additionally, if a text_format is provided, then the text output for the TranscriptionFrame will be formatted to this specification. Your system context can then be updated to include information about this format to understand which speaker spoke which words. For example, if you have text_format = <{speaker_id}>{text}</{speaker_id}> , then the output would be <S1>Good morning.</S1> . ​ Available attributes Attribute Description Example speaker_id The ID of the speaker S1 text The transcribed text Good morning. ​ Usage Example Copy Ask AI from pipecat.services.speechmatics.stt import SpeechmaticsSTTService from pipecat.transcriptions.language import Language # Configure service stt = SpeechmaticsSTTService( api_key = "your-api-key" , language = Language. FR , ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, tts, transport.output() ]) ​ Additional Notes Connection Management : Automatically handles WebSocket connections and reconnections Sample Rate : The default sample rate of 16000 in pcm_s16le format VAD Integration : Supports Speechmatics’ built-in VAD and end of utterance detection SambaNova (Whisper) Ultravox On this page Overview Installation Frames Input Output Endpoints Feature Discovery Language Support Translation Support Speaker Diarization Available attributes Usage Example Additional Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_ultravox_30410814.txt b/stt_ultravox_30410814.txt
new file mode 100644
index 0000000000000000000000000000000000000000..a2ea74a52d8d79aaaf5275b536305649f0e10403
--- /dev/null
+++ b/stt_ultravox_30410814.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/ultravox
+Title: Ultravox - Pipecat
+==================================================
+
+Ultravox - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text Ultravox Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview UltravoxSTTService provides real-time speech-to-text capabilities using the Ultravox multimodal model running locally. Ultravox directly encodes audio into the LLM’s embedding space, eliminating the need for a separate ASR component and providing faster, more efficient transcription. ​ Installation To use UltravoxSTTService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[ultravox]" ​ Configuration ​ Constructor Parameters ​ model_size str default: "fixie-ai/ultravox-v0_4_1-llama-3_1-8b" The Ultravox model to use. Defaults to the 8B parameter model based on Llama 3.1. ​ hf_token str Your Hugging Face token for accessing the model. Will use the HF_TOKEN environment variable if not provided. ​ temperature float default: "0.7" Temperature for text generation, controlling creativity vs. determinism. ​ max_tokens int default: "100" Maximum number of tokens to generate for each transcription. ​ Input The service processes AudioRawFrame instances containing: Raw PCM audio data 16-bit depth (int16) 16kHz sample rate (recommended) Single channel (mono) The service intelligently buffers audio between UserStartedSpeakingFrame and UserStoppedSpeakingFrame events. ​ Output Frames The service produces the following frame during transcription: ​ TranscriptionFrame Generated when speech processing is complete, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ Usage Example Copy Ask AI from pipecat.services.ultravox.stt import UltravoxSTTService import os # Configure service stt = UltravoxSTTService( model_size = "fixie-ai/ultravox-v0_4_1-llama-3_1-8b" , hf_token = os.environ.get( "HF_TOKEN" ), temperature = 0.5 , max_tokens = 150 , ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ Internal Architecture UltravoxSTTService uses the following internal components: ​ AudioBuffer Collects audio frames during speech events for batch processing: Manages frames between UserStartedSpeakingFrame and UserStoppedSpeakingFrame Tracks speech start timestamp Prevents concurrent processing with status flag ​ UltravoxModel Handles loading and running the Ultravox model: Initializes vLLM engine for optimal GPU inference Manages model tokenization and prompt formatting Provides streaming text generation from audio input ​ Frame Flow ​ Metrics Support The service supports metrics collection: Time to First Byte (TTFB) Total processing duration Buffer collection time ​ Error Handling The service produces ErrorFrames in the following scenarios: Empty audio buffer Invalid audio data Model inference errors Timeout during processing ​ Available Models Model ID Parameters Base LLM Min VRAM Languages fixie-ai/ultravox-v0_4_1-llama-3_1-8b 8B Llama 3.1 16GB English, limited multilingual fixie-ai/ultravox-v0_4_1-llama-3_1-70b 70B Llama 3.1 80GB+ English, broader multilingual fixie-ai/ultravox-v0_4-mistral-7b 7B Mistral 14GB English See the Fixie.ai Hugging Face page for the latest model availability. Ensure your environment has sufficient GPU resources for the selected model. ​ Memory Considerations The Ultravox model requires GPU resources: For the 8B parameter model ( fixie-ai/ultravox-v0_4_1-llama-3_1-8b ): Benchmarked on A100-40GB GPUs Can run on consumer GPUs with appropriate memory (at least 8GB VRAM, but performance may vary) For larger variants like the 70B parameter model: Requires significantly more memory (40GB+ VRAM recommended) Audio buffer size grows with speech duration Processing occurs in batches, not streaming According to the Hugging Face card, the 8B model achieves a time-to-first-token of approximately 150ms and generates 50-100 tokens per second on an A100-40GB GPU. Speechmatics Whisper On this page Overview Installation Configuration Constructor Parameters Input Output Frames TranscriptionFrame Usage Example Internal Architecture AudioBuffer UltravoxModel Frame Flow Metrics Support Error Handling Available Models Memory Considerations Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_ultravox_710e291e.txt b/stt_ultravox_710e291e.txt
new file mode 100644
index 0000000000000000000000000000000000000000..fc3af0c3c1f7fcb73071dc722fc74d8c228dc03f
--- /dev/null
+++ b/stt_ultravox_710e291e.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/ultravox#configuration
+Title: Ultravox - Pipecat
+==================================================
+
+Ultravox - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text Ultravox Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview UltravoxSTTService provides real-time speech-to-text capabilities using the Ultravox multimodal model running locally. Ultravox directly encodes audio into the LLM’s embedding space, eliminating the need for a separate ASR component and providing faster, more efficient transcription. ​ Installation To use UltravoxSTTService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[ultravox]" ​ Configuration ​ Constructor Parameters ​ model_size str default: "fixie-ai/ultravox-v0_4_1-llama-3_1-8b" The Ultravox model to use. Defaults to the 8B parameter model based on Llama 3.1. ​ hf_token str Your Hugging Face token for accessing the model. Will use the HF_TOKEN environment variable if not provided. ​ temperature float default: "0.7" Temperature for text generation, controlling creativity vs. determinism. ​ max_tokens int default: "100" Maximum number of tokens to generate for each transcription. ​ Input The service processes AudioRawFrame instances containing: Raw PCM audio data 16-bit depth (int16) 16kHz sample rate (recommended) Single channel (mono) The service intelligently buffers audio between UserStartedSpeakingFrame and UserStoppedSpeakingFrame events. ​ Output Frames The service produces the following frame during transcription: ​ TranscriptionFrame Generated when speech processing is complete, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ Usage Example Copy Ask AI from pipecat.services.ultravox.stt import UltravoxSTTService import os # Configure service stt = UltravoxSTTService( model_size = "fixie-ai/ultravox-v0_4_1-llama-3_1-8b" , hf_token = os.environ.get( "HF_TOKEN" ), temperature = 0.5 , max_tokens = 150 , ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ Internal Architecture UltravoxSTTService uses the following internal components: ​ AudioBuffer Collects audio frames during speech events for batch processing: Manages frames between UserStartedSpeakingFrame and UserStoppedSpeakingFrame Tracks speech start timestamp Prevents concurrent processing with status flag ​ UltravoxModel Handles loading and running the Ultravox model: Initializes vLLM engine for optimal GPU inference Manages model tokenization and prompt formatting Provides streaming text generation from audio input ​ Frame Flow ​ Metrics Support The service supports metrics collection: Time to First Byte (TTFB) Total processing duration Buffer collection time ​ Error Handling The service produces ErrorFrames in the following scenarios: Empty audio buffer Invalid audio data Model inference errors Timeout during processing ​ Available Models Model ID Parameters Base LLM Min VRAM Languages fixie-ai/ultravox-v0_4_1-llama-3_1-8b 8B Llama 3.1 16GB English, limited multilingual fixie-ai/ultravox-v0_4_1-llama-3_1-70b 70B Llama 3.1 80GB+ English, broader multilingual fixie-ai/ultravox-v0_4-mistral-7b 7B Mistral 14GB English See the Fixie.ai Hugging Face page for the latest model availability. Ensure your environment has sufficient GPU resources for the selected model. ​ Memory Considerations The Ultravox model requires GPU resources: For the 8B parameter model ( fixie-ai/ultravox-v0_4_1-llama-3_1-8b ): Benchmarked on A100-40GB GPUs Can run on consumer GPUs with appropriate memory (at least 8GB VRAM, but performance may vary) For larger variants like the 70B parameter model: Requires significantly more memory (40GB+ VRAM recommended) Audio buffer size grows with speech duration Processing occurs in batches, not streaming According to the Hugging Face card, the 8B model achieves a time-to-first-token of approximately 150ms and generates 50-100 tokens per second on an A100-40GB GPU. Speechmatics Whisper On this page Overview Installation Configuration Constructor Parameters Input Output Frames TranscriptionFrame Usage Example Internal Architecture AudioBuffer UltravoxModel Frame Flow Metrics Support Error Handling Available Models Memory Considerations Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_whisper_202b9f71.txt b/stt_whisper_202b9f71.txt
new file mode 100644
index 0000000000000000000000000000000000000000..a78f77b99165408a96f787b001de1aacf4f93be4
--- /dev/null
+++ b/stt_whisper_202b9f71.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/whisper#constructor-parameters
+Title: Whisper - Pipecat
+==================================================
+
+Whisper - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text Whisper Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview WhisperSTTService provides speech-to-text capabilities using OpenAI’s Whisper models running locally. It supports multiple model sizes and configurations for offline transcription. ​ Installation To use WhisperSTTService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[whisper]" ​ Configuration ​ Constructor Parameters ​ model str | Model default: "Model.DISTIL_MEDIUM_EN" Whisper model to use. Can be a string or Model enum value ​ device str default: "auto" Device to run the model on (‘cpu’, ‘cuda’, or ‘auto’) ​ compute_type str default: "default" Computation type for model inference ​ no_speech_prob float default: "0.4" Threshold for filtering out non-speech segments ​ Available Models Copy Ask AI class Model ( Enum ): TINY = "tiny" # Smallest, fastest model BASE = "base" # Basic model MEDIUM = "medium" # Medium-sized model LARGE = "large-v3" # Largest, most accurate model DISTIL_LARGE_V2 = "Systran/faster-distil-whisper-large-v2" DISTIL_MEDIUM_EN = "Systran/faster-distil-whisper-medium.en" ​ Input The service processes raw audio data with the following requirements: PCM audio format 16-bit depth Single channel (mono) Normalized to float32 range [-1.0, 1.0] ​ Output Frames ​ TranscriptionFrame Generated for transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ ErrorFrame Generated when transcription errors occur, containing error details. ​ Usage Example Copy Ask AI from pipecat.services.whisper.stt import WhisperSTTService, Model # Configure service with default model stt = WhisperSTTService( model = Model. DISTIL_MEDIUM_EN , device = "cuda" , no_speech_prob = 0.4 ) # Or use a custom model path stt = WhisperSTTService( model = "path/to/custom/model" , device = "cpu" ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ Methods See the STT base class methods for additional functionality. ​ Model Selection Guide Model Size Speed Accuracy Memory Usage TINY 39M Fastest Basic Minimal BASE 74M Fast Good Low MEDIUM 769M Medium Better Moderate LARGE 1.5GB Slow Best High DISTIL_MEDIUM_EN ~400M Fast Good (English) Moderate DISTIL_LARGE_V2 ~750M Medium Better Moderate ​ Frame Flow ​ Metrics Support The service collects processing metrics: Time to First Byte (TTFB) Processing duration Model loading time Inference time ​ Notes Runs completely offline after model download First run requires model download Supports CPU and CUDA acceleration Processes audio in segments Filters out non-speech segments Thread-safe processing Automatic error handling Ultravox Anthropic On this page Overview Installation Configuration Constructor Parameters Available Models Input Output Frames TranscriptionFrame ErrorFrame Usage Example Methods Model Selection Guide Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_whisper_4c065530.txt b/stt_whisper_4c065530.txt
new file mode 100644
index 0000000000000000000000000000000000000000..271873e5ec43104f3e094eb7306e622c594d6311
--- /dev/null
+++ b/stt_whisper_4c065530.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/whisper#param-no-speech-prob
+Title: Whisper - Pipecat
+==================================================
+
+Whisper - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text Whisper Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview WhisperSTTService provides speech-to-text capabilities using OpenAI’s Whisper models running locally. It supports multiple model sizes and configurations for offline transcription. ​ Installation To use WhisperSTTService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[whisper]" ​ Configuration ​ Constructor Parameters ​ model str | Model default: "Model.DISTIL_MEDIUM_EN" Whisper model to use. Can be a string or Model enum value ​ device str default: "auto" Device to run the model on (‘cpu’, ‘cuda’, or ‘auto’) ​ compute_type str default: "default" Computation type for model inference ​ no_speech_prob float default: "0.4" Threshold for filtering out non-speech segments ​ Available Models Copy Ask AI class Model ( Enum ): TINY = "tiny" # Smallest, fastest model BASE = "base" # Basic model MEDIUM = "medium" # Medium-sized model LARGE = "large-v3" # Largest, most accurate model DISTIL_LARGE_V2 = "Systran/faster-distil-whisper-large-v2" DISTIL_MEDIUM_EN = "Systran/faster-distil-whisper-medium.en" ​ Input The service processes raw audio data with the following requirements: PCM audio format 16-bit depth Single channel (mono) Normalized to float32 range [-1.0, 1.0] ​ Output Frames ​ TranscriptionFrame Generated for transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ ErrorFrame Generated when transcription errors occur, containing error details. ​ Usage Example Copy Ask AI from pipecat.services.whisper.stt import WhisperSTTService, Model # Configure service with default model stt = WhisperSTTService( model = Model. DISTIL_MEDIUM_EN , device = "cuda" , no_speech_prob = 0.4 ) # Or use a custom model path stt = WhisperSTTService( model = "path/to/custom/model" , device = "cpu" ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ Methods See the STT base class methods for additional functionality. ​ Model Selection Guide Model Size Speed Accuracy Memory Usage TINY 39M Fastest Basic Minimal BASE 74M Fast Good Low MEDIUM 769M Medium Better Moderate LARGE 1.5GB Slow Best High DISTIL_MEDIUM_EN ~400M Fast Good (English) Moderate DISTIL_LARGE_V2 ~750M Medium Better Moderate ​ Frame Flow ​ Metrics Support The service collects processing metrics: Time to First Byte (TTFB) Processing duration Model loading time Inference time ​ Notes Runs completely offline after model download First run requires model download Supports CPU and CUDA acceleration Processes audio in segments Filters out non-speech segments Thread-safe processing Automatic error handling Ultravox Anthropic On this page Overview Installation Configuration Constructor Parameters Available Models Input Output Frames TranscriptionFrame ErrorFrame Usage Example Methods Model Selection Guide Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_whisper_a203943e.txt b/stt_whisper_a203943e.txt
new file mode 100644
index 0000000000000000000000000000000000000000..6372d466f972b0aea16b10604579229a871dd1da
--- /dev/null
+++ b/stt_whisper_a203943e.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/whisper#param-text
+Title: Whisper - Pipecat
+==================================================
+
+Whisper - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text Whisper Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview WhisperSTTService provides speech-to-text capabilities using OpenAI’s Whisper models running locally. It supports multiple model sizes and configurations for offline transcription. ​ Installation To use WhisperSTTService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[whisper]" ​ Configuration ​ Constructor Parameters ​ model str | Model default: "Model.DISTIL_MEDIUM_EN" Whisper model to use. Can be a string or Model enum value ​ device str default: "auto" Device to run the model on (‘cpu’, ‘cuda’, or ‘auto’) ​ compute_type str default: "default" Computation type for model inference ​ no_speech_prob float default: "0.4" Threshold for filtering out non-speech segments ​ Available Models Copy Ask AI class Model ( Enum ): TINY = "tiny" # Smallest, fastest model BASE = "base" # Basic model MEDIUM = "medium" # Medium-sized model LARGE = "large-v3" # Largest, most accurate model DISTIL_LARGE_V2 = "Systran/faster-distil-whisper-large-v2" DISTIL_MEDIUM_EN = "Systran/faster-distil-whisper-medium.en" ​ Input The service processes raw audio data with the following requirements: PCM audio format 16-bit depth Single channel (mono) Normalized to float32 range [-1.0, 1.0] ​ Output Frames ​ TranscriptionFrame Generated for transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ ErrorFrame Generated when transcription errors occur, containing error details. ​ Usage Example Copy Ask AI from pipecat.services.whisper.stt import WhisperSTTService, Model # Configure service with default model stt = WhisperSTTService( model = Model. DISTIL_MEDIUM_EN , device = "cuda" , no_speech_prob = 0.4 ) # Or use a custom model path stt = WhisperSTTService( model = "path/to/custom/model" , device = "cpu" ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ Methods See the STT base class methods for additional functionality. ​ Model Selection Guide Model Size Speed Accuracy Memory Usage TINY 39M Fastest Basic Minimal BASE 74M Fast Good Low MEDIUM 769M Medium Better Moderate LARGE 1.5GB Slow Best High DISTIL_MEDIUM_EN ~400M Fast Good (English) Moderate DISTIL_LARGE_V2 ~750M Medium Better Moderate ​ Frame Flow ​ Metrics Support The service collects processing metrics: Time to First Byte (TTFB) Processing duration Model loading time Inference time ​ Notes Runs completely offline after model download First run requires model download Supports CPU and CUDA acceleration Processes audio in segments Filters out non-speech segments Thread-safe processing Automatic error handling Ultravox Anthropic On this page Overview Installation Configuration Constructor Parameters Available Models Input Output Frames TranscriptionFrame ErrorFrame Usage Example Methods Model Selection Guide Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/stt_whisper_f4bb9209.txt b/stt_whisper_f4bb9209.txt
new file mode 100644
index 0000000000000000000000000000000000000000..851a6ba2e614796c8f5236d5a9836cae85773b9a
--- /dev/null
+++ b/stt_whisper_f4bb9209.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/stt/whisper#model-selection-guide
+Title: Whisper - Pipecat
+==================================================
+
+Whisper - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Speech-to-Text Whisper Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text AssemblyAI AWS Transcribe Azure Cartesia Deepgram Fal (Wizper) Gladia Google Groq (Whisper) NVIDIA Riva OpenAI SambaNova (Whisper) Speechmatics Ultravox Whisper LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview WhisperSTTService provides speech-to-text capabilities using OpenAI’s Whisper models running locally. It supports multiple model sizes and configurations for offline transcription. ​ Installation To use WhisperSTTService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[whisper]" ​ Configuration ​ Constructor Parameters ​ model str | Model default: "Model.DISTIL_MEDIUM_EN" Whisper model to use. Can be a string or Model enum value ​ device str default: "auto" Device to run the model on (‘cpu’, ‘cuda’, or ‘auto’) ​ compute_type str default: "default" Computation type for model inference ​ no_speech_prob float default: "0.4" Threshold for filtering out non-speech segments ​ Available Models Copy Ask AI class Model ( Enum ): TINY = "tiny" # Smallest, fastest model BASE = "base" # Basic model MEDIUM = "medium" # Medium-sized model LARGE = "large-v3" # Largest, most accurate model DISTIL_LARGE_V2 = "Systran/faster-distil-whisper-large-v2" DISTIL_MEDIUM_EN = "Systran/faster-distil-whisper-medium.en" ​ Input The service processes raw audio data with the following requirements: PCM audio format 16-bit depth Single channel (mono) Normalized to float32 range [-1.0, 1.0] ​ Output Frames ​ TranscriptionFrame Generated for transcriptions, containing: ​ text string Transcribed text ​ user_id string User identifier ​ timestamp string ISO 8601 formatted timestamp ​ ErrorFrame Generated when transcription errors occur, containing error details. ​ Usage Example Copy Ask AI from pipecat.services.whisper.stt import WhisperSTTService, Model # Configure service with default model stt = WhisperSTTService( model = Model. DISTIL_MEDIUM_EN , device = "cuda" , no_speech_prob = 0.4 ) # Or use a custom model path stt = WhisperSTTService( model = "path/to/custom/model" , device = "cpu" ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, ... ]) ​ Methods See the STT base class methods for additional functionality. ​ Model Selection Guide Model Size Speed Accuracy Memory Usage TINY 39M Fastest Basic Minimal BASE 74M Fast Good Low MEDIUM 769M Medium Better Moderate LARGE 1.5GB Slow Best High DISTIL_MEDIUM_EN ~400M Fast Good (English) Moderate DISTIL_LARGE_V2 ~750M Medium Better Moderate ​ Frame Flow ​ Metrics Support The service collects processing metrics: Time to First Byte (TTFB) Processing duration Model loading time Inference time ​ Notes Runs completely offline after model download First run requires model download Supports CPU and CUDA acceleration Processes audio in segments Filters out non-speech segments Thread-safe processing Automatic error handling Ultravox Anthropic On this page Overview Installation Configuration Constructor Parameters Available Models Input Output Frames TranscriptionFrame ErrorFrame Usage Example Methods Model Selection Guide Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/telephony_daily-webrtc_83779e5a.txt b/telephony_daily-webrtc_83779e5a.txt
new file mode 100644
index 0000000000000000000000000000000000000000..b7e1fe62cb8441f685f5b1b35e1c82324ea961b4
--- /dev/null
+++ b/telephony_daily-webrtc_83779e5a.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/guides/telephony/daily-webrtc#orchestrating-incoming-calls
+Title: Dial-in: WebRTC (Daily) - Pipecat
+==================================================
+
+Dial-in: WebRTC (Daily) - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Telephony Dial-in: WebRTC (Daily) Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Guides Fundamentals Context Management Custom FrameProcessor Detecting Idle Users Ending a Pipeline Function Calling Muting User Input Recording Audio Recording Transcripts Features Gemini Multimodal Live Metrics Noise cancellation with Krisp OpenAI Audio Models and APIs Pipecat Flows Telephony Overview Dial-in: WebRTC (Daily) Dial-in: WebRTC (Twilio + Daily) Dial-in: Twilio (Media Streams) Dialout: WebRTC (Daily) Deploying your bot Overview Deployment pattern Example: Pipecat Cloud Example: Fly.io Example: Cerebrium Example: Modal ​ Things you’ll need An active Daily developer key. One or more Daily provisioned phone numbers (covered below). Prefer to look at code? See the example project! We have a complete dialin-ready project using Daily as both a transport and PSTN/SIP provider in the Pipecat repo. This guide referencse the project and steps through the important parts that make dial-in work. ​ Do I need to provision my phone numbers through Daily? You can use Daily solely as a transport if you prefer. This is particularly useful if you already have Twilio-provisioned numbers and workflows. In that case, you can configure Twilio to forward calls to your Pipecat agents and join a Daily WebRTC call. More details on using Twilio with Daily as a transport can be found here . If you’re starting from scratch, using everything on one platform offers some convenience. By provisioning your phone numbers through Daily and using Daily as the transport layer, you won’t need to worry about initial call routing. ​ Purchasing a phone number You can purchase a number via the Daily REST API Purchase a random number Purchase specific number List numbers Copy Ask AI curl --request POST \ --url 'https://api.daily.co/v1/buy-phone-number' \ --header 'Authorization: Bearer [YOUR_DAILY_API_KEY]' \ --header 'Content-Type: application/json' ​ Configuring your bot runner You’ll need a HTTP service that can receive incoming call hooks and trigger a new agent session. We discussed the concept of a bot runner in the deployment section, which we’ll build on here to add support for incoming phone calls. Within the start_bot method, we’ll need to grab both callId and callDomain from the incoming web request that is triggered by Daily when someone dials the number: bot_daily.py Copy Ask AI # Get the dial-in properties from the request try : data = await request.json() callId = data.get( "callId" ) callDomain = data.get( "callDomain" ) except Exception : raise HTTPException( status_code = 500 , detail = "Missing properties 'callId' or 'callDomain'" ) Full bot source code here ​ Orchestrating incoming calls Daily needs a URL / webhook endpoint it can trigger when a user dials the phone number. We can configure this by assigning the number to an endpoint via their REST API. Here is an example: Copy Ask AI curl --location 'https://api.daily.co/v1' \ --header 'Content-Type: application/json' \ --header 'Authorization: Bearer [DAILY API TOKEN HERE]' \ --data '{ "properties": { "pinless_dialin": [ { "phone_number": "[DAILY PROVISIONED NUMBER HERE]", "room_creation_api": "[BOT RUNNER URL]/start_bot" } ] } }' If you want to test locally, you can expose your web method using a service such as ngrok . Example ngrok tunnel Copy Ask AI python bot_runner.py --host localhost --port 7860 --reload ngrok http localhost:7860 # E.g: https://123.ngrok.app/start_bot ​ Creating a new SIP-enabled room We’ll need to configure the Daily room to be setup to receive SIP connections. daily-helpers.py included in Pipecat has some useful imports that make this easy. We just need to pass through new SIP parameters as part of room creation: bot_runner.py Copy Ask AI from pipecat.transports.services.helpers.daily_rest import DailyRoomParams, DailyRoomProperties, DailyRoomSipParams params = DailyRoomParams( properties = DailyRoomProperties( sip = DailyRoomSipParams( display_name = "sip-dialin" video = False sip_mode = "dial-in" num_endpoints = 1 ) ) ) # Create sip-enabled Daily room via REST try : room: DailyRoomObject = daily_rest_helper.create_room( params = params) except Exception as e: raise HTTPException( status_code = 500 , detail = f "Unable to provision room { e } " ) print ( f "Daily room returned { room.url } { room.config.sip_endpoint } " ) Incoming calls will include both callId and callDomain properties in the body of the request; we’ll need to pass to the Pipecat agent. For simplicity, our agents are spawned as sub-processes of the bot runner, so we’ll pass the callId and callDomain through as command line arguments: bot_runner.py Copy Ask AI proc = subprocess.Popen( [ f "python3 -m bot_daily -u { room.url } -t { token } -i { callId } -d { callDomain } " ], shell = True , bufsize = 1 , cwd = os.path.dirname(os.path.abspath( __file__ )) ) That’s all the configuration we need in our bot_runner.py . ​ Configuring your Pipecat bot Let’s take a look at bot_daily.py and step through the differences from other examples. First, it’s setup to receive additional command line parameters which are passed through to the DailyTransport object: bot_daily.py Copy Ask AI # ... async def main ( room_url : str , token : str , callId : str , callDomain : str ): async with aiohttp.ClientSession() as session: diallin_settings = DailyDialinSettings( call_id = callId, call_domain = callDomain ) transport = DailyTransport( room_url, token, "Chatbot" , DailyParams( api_url = daily_api_url, api_key = daily_api_key, dialin_settings = diallin_settings, audio_in_enabled = True , audio_out_enabled = True , video_out_enabled = False , vad_analyzer = SileroVADAnalyzer(), transcription_enabled = True , ) ) # ... your bot code if __name__ == "__main__" : parser = argparse.ArgumentParser( description = "Pipecat Simple ChatBot" ) parser.add_argument( "-u" , type = str , help = "Room URL" ) parser.add_argument( "-t" , type = str , help = "Token" ) parser.add_argument( "-i" , type = str , help = "Call ID" ) parser.add_argument( "-d" , type = str , help = "Call Domain" ) config = parser.parse_args() asyncio.run(main(config.u, config.t, config.i, config.d)) Optionally, we can listen and respond to the on_dialin_ready event manually. This is useful if you have specific scenarios in whih you want to indicates that the SIP worker and is ready to be forwarded to the call. This would stop any hold music and connect the end-user to our Pipecat bot. Copy Ask AI @transport.event_handler ( "on_dialin_ready" ) async def on_dialin_ready ( transport , cdata ): print ( f "on_dialin_ready" , cdata) Since we’re using Daily as a phone vendor, this method is handled internally by the Pipecat Daily service. It can, however, be useful to override this default behaviour if you want to configure your bot in a certain way as soon as the call is ready. Typically, however, initial setup is done in the on_first_participant_joined event after the user has joined the session. Copy Ask AI @transport.event_handler ( "on_first_participant_joined" ) async def on_first_participant_joined ( transport , participant ): transport.capture_participant_transcription(participant[ "id" ]) await task.queue_frames([LLMMessagesFrame(messages)]) Overview Dial-in: WebRTC (Twilio + Daily) On this page Things you’ll need Do I need to provision my phone numbers through Daily? Purchasing a phone number Configuring your bot runner Orchestrating incoming calls Creating a new SIP-enabled room Configuring your Pipecat bot Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/telephony_dialout_78b5a0f5.txt b/telephony_dialout_78b5a0f5.txt
new file mode 100644
index 0000000000000000000000000000000000000000..965da1825823e5490a4b3bb1b275735cb5f97178
--- /dev/null
+++ b/telephony_dialout_78b5a0f5.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/guides/telephony/dialout
+Title: Dialout: WebRTC (Daily) - Pipecat
+==================================================
+
+Dialout: WebRTC (Daily) - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Telephony Dialout: WebRTC (Daily) Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Guides Fundamentals Context Management Custom FrameProcessor Detecting Idle Users Ending a Pipeline Function Calling Muting User Input Recording Audio Recording Transcripts Features Gemini Multimodal Live Metrics Noise cancellation with Krisp OpenAI Audio Models and APIs Pipecat Flows Telephony Overview Dial-in: WebRTC (Daily) Dial-in: WebRTC (Twilio + Daily) Dial-in: Twilio (Media Streams) Dialout: WebRTC (Daily) Deploying your bot Overview Deployment pattern Example: Pipecat Cloud Example: Fly.io Example: Cerebrium Example: Modal We’re working on this page and will update it soon. In the meantime, you can follow the instructions in the Quickstart guide to get started. Dial-in: Twilio (Media Streams) Overview Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/telephony_overview_fdab87ae.txt b/telephony_overview_fdab87ae.txt
new file mode 100644
index 0000000000000000000000000000000000000000..599360c3adabe5dbb367cbe59ef1dc6d2112fa6b
--- /dev/null
+++ b/telephony_overview_fdab87ae.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/guides/telephony/overview#what-are-pstn-and-sip%3F-what-are-the-differences%3F
+Title: Overview - Pipecat
+==================================================
+
+Overview - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Telephony Overview Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Guides Fundamentals Context Management Custom FrameProcessor Detecting Idle Users Ending a Pipeline Function Calling Muting User Input Recording Audio Recording Transcripts Features Gemini Multimodal Live Metrics Noise cancellation with Krisp OpenAI Audio Models and APIs Pipecat Flows Telephony Overview Dial-in: WebRTC (Daily) Dial-in: WebRTC (Twilio + Daily) Dial-in: Twilio (Media Streams) Dialout: WebRTC (Daily) Deploying your bot Overview Deployment pattern Example: Pipecat Cloud Example: Fly.io Example: Cerebrium Example: Modal ​ Introduction You can dial-in to your Pipecat bots, and have them dial-out too, across both PSTN and SIP. The technical implementation will depend on your chosen transport and phone number vendor; each will likely have their own methods and events to consider. ​ Which transport should I use? This really depends on your project. We have examples that cover both WebRTC (Daily) and Twilio (WebSockets), and Pipecat supports multiple different types of media transport: local, WebSockets, WebRTC etc. Use Pipecat’s native Twilio WebSockets integration for simple workflows that are only telephony-based. The call is managed by Twilio (or similar telephony provider), which means that the bot is not able to perform any form of complex call control. Typically Twilio-specific APIs need to be implemented (for example, when you’re already using Twilio Studio, Twilio Flex, etc). We strongly recommend against using WebSockets for non-telephony use cases (mobile apps, web browsers, etc.). See below. You must use SIP for use cases like the below. These require SIP-based call control: Multi-agents or multi-party calls Connect to legacy call centers powered by open source or cloud Forwarding calls, agent assist/co-pilot, warm transfers, etc. Supporting different telephony vendors without having telephony platform-specific code We strongly recommend using WebRTC for non telephony use cases — ie, mobile apps, web-based experiences. WebRTC is designed to support users on devices with varying network conditions at scale. Learn more in the Voice AI & Voice Agents Illustrated Primer here Please note: you can configure your Pipecat bots to handle multiple vendors. You could,for example, use both Daily and Twilio as phone number vendors concurrently. ​ What are PSTN and SIP? What are the differences? PSTN is an abbreviation for traditional phone networks, consisting of the physical phone lines, cables and transmission links. One of the main differences to consider between these two forms of telephony is PSTN operates on a one user per line basis while SIP can have multiple users per line. Depending on your use-case, you may or may not want to have a single phone number that routes users to a specific bot session, something we’ll cover in the following guides. Pipecat Flows Dial-in: WebRTC (Daily) On this page Introduction Which transport should I use? What are PSTN and SIP? What are the differences? Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/telephony_twilio-daily-webrtc_59dd0501.txt b/telephony_twilio-daily-webrtc_59dd0501.txt
new file mode 100644
index 0000000000000000000000000000000000000000..a2d5d81bceec962855ee3883750a2640662e8b5d
--- /dev/null
+++ b/telephony_twilio-daily-webrtc_59dd0501.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/guides/telephony/twilio-daily-webrtc#how-it-works
+Title: Dial-in: WebRTC (Twilio + Daily) - Pipecat
+==================================================
+
+Dial-in: WebRTC (Twilio + Daily) - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Telephony Dial-in: WebRTC (Twilio + Daily) Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Guides Fundamentals Context Management Custom FrameProcessor Detecting Idle Users Ending a Pipeline Function Calling Muting User Input Recording Audio Recording Transcripts Features Gemini Multimodal Live Metrics Noise cancellation with Krisp OpenAI Audio Models and APIs Pipecat Flows Telephony Overview Dial-in: WebRTC (Daily) Dial-in: WebRTC (Twilio + Daily) Dial-in: Twilio (Media Streams) Dialout: WebRTC (Daily) Deploying your bot Overview Deployment pattern Example: Pipecat Cloud Example: Fly.io Example: Cerebrium Example: Modal ​ Things you’ll need An active Twilio developer key One or more Twilio provisioned phone numbers (covered below) The Twilio Python client library ( pip install twilio ) Prefer to look at code? See the example project! We have a complete example project using Daily as a transport with Twilio as a phone provider. This guide walks through the important components and best practices when integrating voice calls. ​ How It Works Here’s the sequence of events when someone calls your Twilio number: Twilio receives an incoming call to your phone number Twilio calls your webhook server ( /call endpoint) Your server creates a Daily room with SIP capabilities Your server starts the bot process with the room details Your server responds to Twilio with TwiML that puts the caller on hold with music Upon receiving the on_dialin_ready event, the bot forwards the call to the Daily SIP endpoint The caller and bot are connected, and the bot handles the conversation ​ Getting a phone number Visit console.twilio.com and purchase a new phone number (or via the API) Ensure your purchased number supports Voice capabilities Ensure your purchased number appears in your ‘active numbers’ list ​ Project setup You’ll need to set two environment variables for your project: TWILIO_ACCOUNT_SID and TWILIO_AUTH_TOKEN , which you can obtain from the Twilio console. .env Copy Ask AI DAILY_API_KEY = ... DAILY_API_URL = https://api.daily.co/v1 TWILIO_ACCOUNT_SID = ... TWILIO_AUTH_TOKEN = ... OPENAI_API_KEY = ... CARTESIA_API_KEY = ... ​ Configuring your bot runner You’ll need a HTTP service that can receive incoming call hooks and trigger a new agent session. We discussed the concept of a bot runner in the deployment section, which we’ll build on here to add support for incoming phone calls. Here’s how to implement this with FastAPI: server.py Copy Ask AI @app.post ( "/call" , response_class = PlainTextResponse) async def handle_call ( request : Request): """Handle incoming Twilio call webhook.""" print ( "Received call webhook from Twilio" ) try : # Get form data from Twilio webhook form_data = await request.form() data = dict (form_data) # Extract call ID (required to forward the call later) call_sid = data.get( "CallSid" ) if not call_sid: raise HTTPException( status_code = 400 , detail = "Missing CallSid in request" ) print ( f "Processing call with ID: { call_sid } " ) # 1. Create a Daily room with SIP capabilities # More on this in the next section room_details = await create_sip_room(request.app.session) room_url = room_details[ "room_url" ] token = room_details[ "token" ] sip_endpoint = room_details[ "sip_endpoint" ] # 2. Start the bot process with the necessary parameters bot_cmd = f "python bot.py -u { room_url } -t { token } -i { call_sid } -s { sip_endpoint } " subprocess.Popen(shlex.split(bot_cmd)) print ( f "Started bot process with command: { bot_cmd } " ) # 3. IMPORTANT: Put the caller on hold with music while the bot initializes # This is critical to avoid timing issues with Daily SIP initialization resp = VoiceResponse() resp.play( url = "https://your-hold-music.mp3" , # Your custom hold music URL loop = 10 , # Loop the music until the bot is ready ) return str (resp) except Exception as e: print ( f "Unexpected error: { str (e) } " ) raise HTTPException( status_code = 500 , detail = f "Server error: { str (e) } " ) ​ Creating a SIP-enabled room We’ll need to configure the Daily room to be setup to receive SIP connections. In our example project, utils/daily-helpers.py demonstrates how to set up the Daily room using the Daily REST helpers available in Pipecat. We just need to pass through new SIP parameters as part of room creation: utils/daily_helpers.py Copy Ask AI async def create_sip_room ( session : Optional[aiohttp.ClientSession] = None ) -> Dict[ str , str ]: """Create a Daily room with SIP capabilities for phone calls.""" daily_helper = await get_daily_helper(session) # Configure SIP parameters sip_params = DailyRoomSipParams( display_name = "phone-user" , # This can be customized with caller info video = False , # Audio-only call sip_mode = "dial-in" , # For receiving calls num_endpoints = 1 , # Number of SIP endpoints needed ) # Create room properties with SIP enabled properties = DailyRoomProperties( sip = sip_params, enable_dialout = True , # For future expansion if needed start_video_off = True , # Voice only ) # Create and return the room room = await daily_helper.create_room(DailyRoomParams( properties = properties)) token = await daily_helper.get_token(room.url, 24 * 60 * 60 ) # 24 hours validity return { "room_url" : room.url, "token" : token, "sip_endpoint" : room.config.sip_endpoint } ​ Configuring your bot Your bot needs to handle the on_dialin_ready event to forward the call at the right time: bot.py Copy Ask AI async def run_bot ( room_url : str , token : str , call_id : str , sip_uri : str ) -> None : """Run the voice bot with the given parameters.""" logger.info( f "Starting bot with room: { room_url } " ) logger.info( f "SIP endpoint: { sip_uri } " ) # IMPORTANT: Track if call has been forwarded to avoid multiple forwards call_already_forwarded = False # Setup the Daily transport transport = DailyTransport( room_url, token, "Phone Bot" , DailyParams( audio_in_enabled = True , audio_out_enabled = True , transcription_enabled = True , vad_analyzer = SileroVADAnalyzer(), ), ) # ... rest of your bot setup code ... # Handle call ready to forward @transport.event_handler ( "on_dialin_ready" ) async def on_dialin_ready ( transport , cdata ): nonlocal call_already_forwarded # We only want to forward the call once # The on_dialin_ready event will be triggered for each SIP endpoint if call_already_forwarded: logger.warning( "Call already forwarded, ignoring this event." ) return logger.info( f "Forwarding call { call_id } to { sip_uri } " ) try : # Update the Twilio call with TwiML to forward to the Daily SIP endpoint twilio_client.calls(call_id).update( twiml = f "<Response><Dial timeout= \" 30 \" ><Sip> { sip_uri } </Sip></Dial></Response>" ) logger.info( "Call forwarded successfully" ) call_already_forwarded = True except Exception as e: logger.error( f "Failed to forward call: { str (e) } " ) raise ​ Setting up the Twilio webhook Configure your Twilio phone number to use your server’s webhook URL: Go to the Twilio Console Navigate to Phone Numbers → Manage → Active Numbers Click on your phone number Under “Configure”, set “A Call Comes In” to: Webhook: https://your-server.com/call (your server’s URL) HTTP Method: POST ​ Testing locally For local development, you can use ngrok to expose your local server: Copy Ask AI # Start your server python server.py # In another terminal, start ngrok ngrok http 8000 # Use the ngrok URL (e.g., https://a1b2c3.ngrok.io/call) as your webhook ​ Best Practices and Common Pitfalls ​ ✅ Best Practice: Put the call on hold Always respond to Twilio’s initial webhook with hold music. This gives your bot time to initialize and the Daily SIP endpoint to become ready. Copy Ask AI resp = VoiceResponse() resp.play( url = "https://your-hold-music.mp3" , loop = 10 ) return str (resp) ​ ❌ Pitfall: Using <Pause> instead of hold music Don’t use the TwiML <Pause> element to wait for the bot to initialize: Copy Ask AI # DON'T DO THIS resp = VoiceResponse() resp.pause( length = 10 ) # This can cause connection issues Twilio can only pause a call for a short duration (~5 seconds), which may not be enough time for the Daily SIP setup to complete. ​ ✅ Best Practice: Handle multiple on_dialin_ready events If your Daily room has multiple SIP endpoints, use a flag to ensure you only forward the call once: Copy Ask AI call_already_forwarded = False @transport.event_handler ( "on_dialin_ready" ) async def on_dialin_ready ( transport , cdata ): nonlocal call_already_forwarded if call_already_forwarded: logger.info( "Call already forwarded, ignoring this event." ) return # Forward the call... call_already_forwarded = True A single SIP endpoint is sufficient for the initial connection. A second SIP endpoint is needed only if you plan to forward the call. Dial-in: WebRTC (Daily) Dial-in: Twilio (Media Streams) On this page Things you’ll need How It Works Getting a phone number Project setup Configuring your bot runner Creating a SIP-enabled room Configuring your bot Setting up the Twilio webhook Testing locally Best Practices and Common Pitfalls ✅ Best Practice: Put the call on hold ❌ Pitfall: Using <Pause> instead of hold music ✅ Best Practice: Handle multiple on_dialin_ready events Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/telephony_twilio-daily-webrtc_79996e1b.txt b/telephony_twilio-daily-webrtc_79996e1b.txt
new file mode 100644
index 0000000000000000000000000000000000000000..7f3dc086812b0b2ad25eea7eb4ae85c659bd8f15
--- /dev/null
+++ b/telephony_twilio-daily-webrtc_79996e1b.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/guides/telephony/twilio-daily-webrtc#configuring-your-bot-runner
+Title: Dial-in: WebRTC (Twilio + Daily) - Pipecat
+==================================================
+
+Dial-in: WebRTC (Twilio + Daily) - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Telephony Dial-in: WebRTC (Twilio + Daily) Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Guides Fundamentals Context Management Custom FrameProcessor Detecting Idle Users Ending a Pipeline Function Calling Muting User Input Recording Audio Recording Transcripts Features Gemini Multimodal Live Metrics Noise cancellation with Krisp OpenAI Audio Models and APIs Pipecat Flows Telephony Overview Dial-in: WebRTC (Daily) Dial-in: WebRTC (Twilio + Daily) Dial-in: Twilio (Media Streams) Dialout: WebRTC (Daily) Deploying your bot Overview Deployment pattern Example: Pipecat Cloud Example: Fly.io Example: Cerebrium Example: Modal ​ Things you’ll need An active Twilio developer key One or more Twilio provisioned phone numbers (covered below) The Twilio Python client library ( pip install twilio ) Prefer to look at code? See the example project! We have a complete example project using Daily as a transport with Twilio as a phone provider. This guide walks through the important components and best practices when integrating voice calls. ​ How It Works Here’s the sequence of events when someone calls your Twilio number: Twilio receives an incoming call to your phone number Twilio calls your webhook server ( /call endpoint) Your server creates a Daily room with SIP capabilities Your server starts the bot process with the room details Your server responds to Twilio with TwiML that puts the caller on hold with music Upon receiving the on_dialin_ready event, the bot forwards the call to the Daily SIP endpoint The caller and bot are connected, and the bot handles the conversation ​ Getting a phone number Visit console.twilio.com and purchase a new phone number (or via the API) Ensure your purchased number supports Voice capabilities Ensure your purchased number appears in your ‘active numbers’ list ​ Project setup You’ll need to set two environment variables for your project: TWILIO_ACCOUNT_SID and TWILIO_AUTH_TOKEN , which you can obtain from the Twilio console. .env Copy Ask AI DAILY_API_KEY = ... DAILY_API_URL = https://api.daily.co/v1 TWILIO_ACCOUNT_SID = ... TWILIO_AUTH_TOKEN = ... OPENAI_API_KEY = ... CARTESIA_API_KEY = ... ​ Configuring your bot runner You’ll need a HTTP service that can receive incoming call hooks and trigger a new agent session. We discussed the concept of a bot runner in the deployment section, which we’ll build on here to add support for incoming phone calls. Here’s how to implement this with FastAPI: server.py Copy Ask AI @app.post ( "/call" , response_class = PlainTextResponse) async def handle_call ( request : Request): """Handle incoming Twilio call webhook.""" print ( "Received call webhook from Twilio" ) try : # Get form data from Twilio webhook form_data = await request.form() data = dict (form_data) # Extract call ID (required to forward the call later) call_sid = data.get( "CallSid" ) if not call_sid: raise HTTPException( status_code = 400 , detail = "Missing CallSid in request" ) print ( f "Processing call with ID: { call_sid } " ) # 1. Create a Daily room with SIP capabilities # More on this in the next section room_details = await create_sip_room(request.app.session) room_url = room_details[ "room_url" ] token = room_details[ "token" ] sip_endpoint = room_details[ "sip_endpoint" ] # 2. Start the bot process with the necessary parameters bot_cmd = f "python bot.py -u { room_url } -t { token } -i { call_sid } -s { sip_endpoint } " subprocess.Popen(shlex.split(bot_cmd)) print ( f "Started bot process with command: { bot_cmd } " ) # 3. IMPORTANT: Put the caller on hold with music while the bot initializes # This is critical to avoid timing issues with Daily SIP initialization resp = VoiceResponse() resp.play( url = "https://your-hold-music.mp3" , # Your custom hold music URL loop = 10 , # Loop the music until the bot is ready ) return str (resp) except Exception as e: print ( f "Unexpected error: { str (e) } " ) raise HTTPException( status_code = 500 , detail = f "Server error: { str (e) } " ) ​ Creating a SIP-enabled room We’ll need to configure the Daily room to be setup to receive SIP connections. In our example project, utils/daily-helpers.py demonstrates how to set up the Daily room using the Daily REST helpers available in Pipecat. We just need to pass through new SIP parameters as part of room creation: utils/daily_helpers.py Copy Ask AI async def create_sip_room ( session : Optional[aiohttp.ClientSession] = None ) -> Dict[ str , str ]: """Create a Daily room with SIP capabilities for phone calls.""" daily_helper = await get_daily_helper(session) # Configure SIP parameters sip_params = DailyRoomSipParams( display_name = "phone-user" , # This can be customized with caller info video = False , # Audio-only call sip_mode = "dial-in" , # For receiving calls num_endpoints = 1 , # Number of SIP endpoints needed ) # Create room properties with SIP enabled properties = DailyRoomProperties( sip = sip_params, enable_dialout = True , # For future expansion if needed start_video_off = True , # Voice only ) # Create and return the room room = await daily_helper.create_room(DailyRoomParams( properties = properties)) token = await daily_helper.get_token(room.url, 24 * 60 * 60 ) # 24 hours validity return { "room_url" : room.url, "token" : token, "sip_endpoint" : room.config.sip_endpoint } ​ Configuring your bot Your bot needs to handle the on_dialin_ready event to forward the call at the right time: bot.py Copy Ask AI async def run_bot ( room_url : str , token : str , call_id : str , sip_uri : str ) -> None : """Run the voice bot with the given parameters.""" logger.info( f "Starting bot with room: { room_url } " ) logger.info( f "SIP endpoint: { sip_uri } " ) # IMPORTANT: Track if call has been forwarded to avoid multiple forwards call_already_forwarded = False # Setup the Daily transport transport = DailyTransport( room_url, token, "Phone Bot" , DailyParams( audio_in_enabled = True , audio_out_enabled = True , transcription_enabled = True , vad_analyzer = SileroVADAnalyzer(), ), ) # ... rest of your bot setup code ... # Handle call ready to forward @transport.event_handler ( "on_dialin_ready" ) async def on_dialin_ready ( transport , cdata ): nonlocal call_already_forwarded # We only want to forward the call once # The on_dialin_ready event will be triggered for each SIP endpoint if call_already_forwarded: logger.warning( "Call already forwarded, ignoring this event." ) return logger.info( f "Forwarding call { call_id } to { sip_uri } " ) try : # Update the Twilio call with TwiML to forward to the Daily SIP endpoint twilio_client.calls(call_id).update( twiml = f "<Response><Dial timeout= \" 30 \" ><Sip> { sip_uri } </Sip></Dial></Response>" ) logger.info( "Call forwarded successfully" ) call_already_forwarded = True except Exception as e: logger.error( f "Failed to forward call: { str (e) } " ) raise ​ Setting up the Twilio webhook Configure your Twilio phone number to use your server’s webhook URL: Go to the Twilio Console Navigate to Phone Numbers → Manage → Active Numbers Click on your phone number Under “Configure”, set “A Call Comes In” to: Webhook: https://your-server.com/call (your server’s URL) HTTP Method: POST ​ Testing locally For local development, you can use ngrok to expose your local server: Copy Ask AI # Start your server python server.py # In another terminal, start ngrok ngrok http 8000 # Use the ngrok URL (e.g., https://a1b2c3.ngrok.io/call) as your webhook ​ Best Practices and Common Pitfalls ​ ✅ Best Practice: Put the call on hold Always respond to Twilio’s initial webhook with hold music. This gives your bot time to initialize and the Daily SIP endpoint to become ready. Copy Ask AI resp = VoiceResponse() resp.play( url = "https://your-hold-music.mp3" , loop = 10 ) return str (resp) ​ ❌ Pitfall: Using <Pause> instead of hold music Don’t use the TwiML <Pause> element to wait for the bot to initialize: Copy Ask AI # DON'T DO THIS resp = VoiceResponse() resp.pause( length = 10 ) # This can cause connection issues Twilio can only pause a call for a short duration (~5 seconds), which may not be enough time for the Daily SIP setup to complete. ​ ✅ Best Practice: Handle multiple on_dialin_ready events If your Daily room has multiple SIP endpoints, use a flag to ensure you only forward the call once: Copy Ask AI call_already_forwarded = False @transport.event_handler ( "on_dialin_ready" ) async def on_dialin_ready ( transport , cdata ): nonlocal call_already_forwarded if call_already_forwarded: logger.info( "Call already forwarded, ignoring this event." ) return # Forward the call... call_already_forwarded = True A single SIP endpoint is sufficient for the initial connection. A second SIP endpoint is needed only if you plan to forward the call. Dial-in: WebRTC (Daily) Dial-in: Twilio (Media Streams) On this page Things you’ll need How It Works Getting a phone number Project setup Configuring your bot runner Creating a SIP-enabled room Configuring your bot Setting up the Twilio webhook Testing locally Best Practices and Common Pitfalls ✅ Best Practice: Put the call on hold ❌ Pitfall: Using <Pause> instead of hold music ✅ Best Practice: Handle multiple on_dialin_ready events Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/telephony_twilio-daily-webrtc_b485e11f.txt b/telephony_twilio-daily-webrtc_b485e11f.txt
new file mode 100644
index 0000000000000000000000000000000000000000..334acf6215275b4e1c90257dcdd1cc3ea1e41321
--- /dev/null
+++ b/telephony_twilio-daily-webrtc_b485e11f.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/telephony/twilio-daily-webrtc#join-our-community
+Title: Overview - Pipecat
+==================================================
+
+Overview - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Get Started Overview Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Get Started Overview Installation & Setup Quickstart Core Concepts Next Steps & Examples Pipecat is an open source Python framework that handles the complex orchestration of AI services, network transport, audio processing, and multimodal interactions. “Multimodal” means you can use any combination of audio, video, images, and/or text in your interactions. And “real-time” means that things are happening quickly enough that it feels conversational—a “back-and-forth” with a bot, not submitting a query and waiting for results. ​ What You Can Build Voice Assistants Natural, real-time conversations with AI using speech recognition and synthesis Interactive Agents Personal coaches and meeting assistants that can understand context and provide guidance Multimodal Apps Applications that combine voice, video, images, and text for rich interactions Creative Tools Storytelling experiences and social companions that engage users Business Solutions Customer intake flows and support bots for automated business processes Complex Flows Structured conversations using Pipecat Flows for managing complex interactions ​ How It Works The flow of interactions in a Pipecat application is typically straightforward: The bot says something The user says something The bot says something The user says something This continues until the conversation naturally ends. While this flow seems simple, making it feel natural requires sophisticated real-time processing. ​ Real-time Processing Pipecat’s pipeline architecture handles both simple voice interactions and complex multimodal processing. Let’s look at how data flows through the system: Voice app Multimodal app 1 Send Audio Transmit and capture streamed audio from the user 2 Transcribe Speech Convert speech to text as the user is talking 3 Process with LLM Generate responses using a large language model 4 Convert to Speech Transform text responses into natural speech 5 Play Audio Stream the audio response back to the user 1 Send Audio Transmit and capture streamed audio from the user 2 Transcribe Speech Convert speech to text as the user is talking 3 Process with LLM Generate responses using a large language model 4 Convert to Speech Transform text responses into natural speech 5 Play Audio Stream the audio response back to the user 1 Send Audio and Video Transmit and capture audio, video, and image inputs simultaneously 2 Process Streams Handle multiple input streams in parallel 3 Model Processing Send combined inputs to multimodal models (like GPT-4V) 4 Generate Outputs Create various outputs (text, images, audio, etc.) 5 Coordinate Presentation Synchronize and present multiple output types In both cases, Pipecat: Processes responses as they stream in Handles multiple input/output modalities concurrently Manages resource allocation and synchronization Coordinates parallel processing tasks This architecture creates fluid, natural interactions without noticeable delays, whether you’re building a simple voice assistant or a complex multimodal application. Pipecat’s pipeline architecture is particularly valuable for managing the complexity of real-time, multimodal interactions, ensuring smooth data flow and proper synchronization regardless of the input/output types involved. Pipecat handles all this complexity for you, letting you focus on building your application rather than managing the underlying infrastructure. ​ Next Steps Ready to build your first Pipecat application? Installation & Setup Prepare your environment and install required dependencies Quickstart Build and run your first Pipecat application Core Concepts Learn about pipelines, frames, and real-time processing Use Cases Explore example implementations and patterns ​ Join Our Community Discord Community Connect with other developers, share your projects, and get support from the Pipecat team. Installation & Setup On this page What You Can Build How It Works Real-time Processing Next Steps Join Our Community Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/telephony_twilio-websockets_00f81d45.txt b/telephony_twilio-websockets_00f81d45.txt
new file mode 100644
index 0000000000000000000000000000000000000000..c1849153b663c5e09db05a6a723c07015ece73c6
--- /dev/null
+++ b/telephony_twilio-websockets_00f81d45.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/guides/telephony/twilio-websockets#audio-format-and-sample-rate
+Title: Dial-in: Twilio (Media Streams) - Pipecat
+==================================================
+
+Dial-in: Twilio (Media Streams) - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Telephony Dial-in: Twilio (Media Streams) Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Guides Fundamentals Context Management Custom FrameProcessor Detecting Idle Users Ending a Pipeline Function Calling Muting User Input Recording Audio Recording Transcripts Features Gemini Multimodal Live Metrics Noise cancellation with Krisp OpenAI Audio Models and APIs Pipecat Flows Telephony Overview Dial-in: WebRTC (Daily) Dial-in: WebRTC (Twilio + Daily) Dial-in: Twilio (Media Streams) Dialout: WebRTC (Daily) Deploying your bot Overview Deployment pattern Example: Pipecat Cloud Example: Fly.io Example: Cerebrium Example: Modal This guide walks through creating a voice AI agent that users can reach by dialing a Twilio phone number. We’ll integrate Twilio Media Streams via WebSockets with a Pipecat pipeline to create a fully functional voice AI experience. ​ Things you’ll need An active Twilio account with at least one phone number A public-facing server or a tunneling service like ngrok API keys for speech-to-text, text-to-speech, and LLM services Complete example code on GitHub The full source code for this example is available in the Pipecat repository. ​ Architecture Overview When a user dials your Twilio phone number: Twilio calls your server’s endpoint, which returns TwiML that establishes a WebSocket connection Twilio sends real-time audio data over the WebSocket, and your server processes it using the Pipecat pipeline Your Pipecat agent processes the audio which is sent to the pipeline, which then outputs audio data back to Twilio Twilio plays this audio to the caller in real-time ​ Server Implementation The server needs to: Serve TwiML in response to Twilio’s HTTP requests Handle WebSocket connections Process audio with the Pipecat pipeline Let’s look at the key components: ​ FastAPI Server Setup Copy Ask AI from fastapi import FastAPI, WebSocket from starlette.responses import HTMLResponse app = FastAPI() @app.post ( "/" ) async def start_call (): return HTMLResponse( content = open ( "templates/streams.xml" ).read(), media_type = "application/xml" ) @app.websocket ( "/ws" ) async def websocket_endpoint ( websocket : WebSocket): await websocket.accept() # Read initial WebSocket messages start_data = ws.iter_text() await start_data. __anext__ () # Second message contains the call details call_data = json.loads( await start_data. __anext__ ()) # Extract both StreamSid and CallSid stream_sid = call_data[ "start" ][ "streamSid" ] call_sid = call_data[ "start" ][ "callSid" ] # Run your Pipecat bot await run_bot(websocket, stream_sid, call_sid, app.state.testing) This server has two main endpoints: POST / - Returns TwiML instructions to Twilio WebSocket /ws - Handles the WebSocket connection for real-time audio ​ TwiML Configuration The TwiML tells Twilio to establish a WebSocket connection with your server: Copy Ask AI <? xml version = "1.0" encoding = "UTF-8" ?> < Response > < Connect > < Stream url = "wss://<your server url>/ws" ></ Stream > </ Connect > < Pause length = "40" /> </ Response > Replace <your server url> with your server’s publicly accessible domain. The Pause element keeps the call alive for a maximum of 40 seconds. Adjust this value based on your expected conversation length. ​ Pipecat Bot Implementation The run_bot function creates and connects all the components in the Pipecat pipeline: Copy Ask AI async def run_bot ( websocket_client : WebSocket, stream_sid : str , call_sid : str , testing : bool ): # Initialize the Twilio serializer with stream and call SIDs serializer = TwilioFrameSerializer( stream_sid = stream_sid, call_sid = call_sid, account_sid = os.getenv( "TWILIO_ACCOUNT_SID" , "" ), auth_token = os.getenv( "TWILIO_AUTH_TOKEN" , "" ), ) # Create the WebSocket transport transport = FastAPIWebsocketTransport( websocket = websocket_client, params = FastAPIWebsocketParams( audio_in_enabled = True , audio_out_enabled = True , add_wav_header = False , vad_analyzer = SileroVADAnalyzer(), serializer = serializer, ), ) # Initialize AI services llm = OpenAILLMService( api_key = os.getenv( "OPENAI_API_KEY" )) stt = DeepgramSTTService( api_key = os.getenv( "DEEPGRAM_API_KEY" ), audio_passthrough = True ) tts = CartesiaTTSService( api_key = os.getenv( "CARTESIA_API_KEY" ), voice_id = "71a7ad14-091c-4e8e-a314-022ece01c121" , # British Reading Lady push_silence_after_stop = testing, ) # Create the initial conversation prompt messages = [ { "role" : "system" , "content" : "You are an elementary teacher in an audio call. Your output will be converted to audio so don't include special characters in your answers. Respond to what the student said in a short short sentence." , }, ] # Setup the context aggregator context = OpenAILLMContext(messages) context_aggregator = llm.create_context_aggregator(context) # Create an audio buffer processor to capture conversation audio audiobuffer = AudioBufferProcessor( user_continuous_stream = not testing) # Build the pipeline pipeline = Pipeline( [ transport.input(), # Websocket input from client stt, # Speech-To-Text context_aggregator.user(), llm, # LLM tts, # Text-To-Speech transport.output(), # Websocket output to client audiobuffer, # Used to buffer the audio in the pipeline context_aggregator.assistant(), ] ) # Create a pipeline task with appropriate parameters task = PipelineTask( pipeline, params = PipelineParams( audio_in_sample_rate = 8000 , # Twilio uses 8kHz audio audio_out_sample_rate = 8000 , allow_interruptions = True , ), ) # Setup event handlers @transport.event_handler ( "on_client_connected" ) async def on_client_connected ( transport , client ): # Start recording await audiobuffer.start_recording() # Kickstart the conversation messages.append({ "role" : "system" , "content" : "Please introduce yourself to the user." }) await task.queue_frames([context_aggregator.user().get_context_frame()]) @transport.event_handler ( "on_client_disconnected" ) async def on_client_disconnected ( transport , client ): await task.cancel() @audiobuffer.event_handler ( "on_audio_data" ) async def on_audio_data ( buffer , audio , sample_rate , num_channels ): server_name = f "server_ { websocket_client.client.port } " await save_audio(server_name, audio, sample_rate, num_channels) # Run the pipeline runner = PipelineRunner( handle_sigint = False , force_gc = True ) await runner.run(task) ​ Key Technical Considerations ​ Audio Format and Sample Rate Twilio Media Streams uses 8kHz mono audio with 16-bit PCM encoding. Make sure your pipeline is configured correctly: Copy Ask AI task = PipelineTask( pipeline, params = PipelineParams( audio_in_sample_rate = 8000 , # Twilio's audio format audio_out_sample_rate = 8000 , allow_interruptions = True , ), ) ​ Serialization and Call Control The TwilioFrameSerializer handles the protocol specifics for communicating with Twilio’s Media Streams: Copy Ask AI serializer = TwilioFrameSerializer( stream_sid = stream_sid, call_sid = call_sid, account_sid = os.getenv( "TWILIO_ACCOUNT_SID" , "" ), auth_token = os.getenv( "TWILIO_AUTH_TOKEN" , "" ), ) When you provide the account_sid and auth_token to the TwilioFrameSerializer , it will automatically end the call via Twilio’s REST API when the pipeline ends. This ensures clean call termination when your bot finishes its conversation. ​ Voice Activity Detection The SileroVAD analyzer helps determine when a user has finished speaking: Copy Ask AI params = FastAPIWebsocketParams( audio_in_enabled = True , audio_out_enabled = True , add_wav_header = False , vad_analyzer = SileroVADAnalyzer(), serializer = serializer, ) ​ Configuring Twilio To set up your Twilio phone number to use your server: Purchase a phone number in your Twilio account Navigate to the Phone Numbers section and select your number Under “Voice & Fax”, set the webhook for “A Call Comes In” to your server’s URL (e.g., https://your-server.com/ ) Make sure the request type is set to HTTP POST Save your changes If you’re using ngrok for local development, your webhook URL will look like https://abc123.ngrok.io/ . Remember to update your TwiML template with the correct WebSocket URL as well. ​ Testing Your Implementation ​ Local Testing Without Phone Calls The example includes a test client that can simulate phone calls without actually using Twilio: Copy Ask AI python server.py - t # Start server in testing mode python client.py - u http: // localhost: 8765 - c 2 # Start 2 test clients The -t flag puts the server in testing mode, and the test client creates virtual clients that communicate with your server as if they were Twilio Media Streams. ​ Using the Phone To test with an actual phone call: Make sure your server is running and accessible via the internet Configure Twilio as described above Dial your Twilio phone number from any phone You should hear your AI agent respond! ​ Ending a Conversation There are two primary ways to end a conversation: Automatic termination : If you provided Twilio credentials to the TwilioFrameSerializer , the call will be ended automatically when your pipeline ends: Copy Ask AI # End the conversation programmatically await task.queue_frame(EndFrame()) Manual termination : You can also end the call explicitly through Twilio’s REST API. This is useful for implementing custom hang-up logic: Copy Ask AI from twilio.rest import Client client = Client(account_sid, auth_token) client.calls(call_sid).update( status = "completed" ) ​ Scaling Considerations For production deployments, consider: Multiple concurrent calls : Each bot instance should run in its own process to handle concurrent calls efficiently. The example server can spawn individual bot processes for each call: Copy Ask AI proc = subprocess.Popen( [ f "python3 -m bot_twilio --stream-sid { stream_sid } --call-sid { call_sid } " ], shell = True , bufsize = 1 , cwd = os.path.dirname(os.path.abspath( __file__ )) ) Error handling : Add robust error handling for network issues, service outages, etc. Logging and monitoring : Implement detailed logging to track call quality and agent performance. Security : Add authentication to your endpoints and use environment variables for all sensitive credentials. ​ Next Steps Try the complete example Clone the full working Twilio chatbot example from GitHub and run it yourself. Deploy to Pipecat Cloud Learn how to deploy your Twilio voice AI agent to production with Pipecat Cloud’s managed infrastructure. Dial-in: WebRTC (Twilio + Daily) Dialout: WebRTC (Daily) On this page Things you’ll need Architecture Overview Server Implementation FastAPI Server Setup TwiML Configuration Pipecat Bot Implementation Key Technical Considerations Audio Format and Sample Rate Serialization and Call Control Voice Activity Detection Configuring Twilio Testing Your Implementation Local Testing Without Phone Calls Using the Phone Ending a Conversation Scaling Considerations Next Steps Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/telephony_twilio-websockets_044ff3a6.txt b/telephony_twilio-websockets_044ff3a6.txt
new file mode 100644
index 0000000000000000000000000000000000000000..0e5debb136d20bb5da43d2e9d28192e0cbd1a25a
--- /dev/null
+++ b/telephony_twilio-websockets_044ff3a6.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/guides/telephony/twilio-websockets#configuring-twilio
+Title: Dial-in: Twilio (Media Streams) - Pipecat
+==================================================
+
+Dial-in: Twilio (Media Streams) - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Telephony Dial-in: Twilio (Media Streams) Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Guides Fundamentals Context Management Custom FrameProcessor Detecting Idle Users Ending a Pipeline Function Calling Muting User Input Recording Audio Recording Transcripts Features Gemini Multimodal Live Metrics Noise cancellation with Krisp OpenAI Audio Models and APIs Pipecat Flows Telephony Overview Dial-in: WebRTC (Daily) Dial-in: WebRTC (Twilio + Daily) Dial-in: Twilio (Media Streams) Dialout: WebRTC (Daily) Deploying your bot Overview Deployment pattern Example: Pipecat Cloud Example: Fly.io Example: Cerebrium Example: Modal This guide walks through creating a voice AI agent that users can reach by dialing a Twilio phone number. We’ll integrate Twilio Media Streams via WebSockets with a Pipecat pipeline to create a fully functional voice AI experience. ​ Things you’ll need An active Twilio account with at least one phone number A public-facing server or a tunneling service like ngrok API keys for speech-to-text, text-to-speech, and LLM services Complete example code on GitHub The full source code for this example is available in the Pipecat repository. ​ Architecture Overview When a user dials your Twilio phone number: Twilio calls your server’s endpoint, which returns TwiML that establishes a WebSocket connection Twilio sends real-time audio data over the WebSocket, and your server processes it using the Pipecat pipeline Your Pipecat agent processes the audio which is sent to the pipeline, which then outputs audio data back to Twilio Twilio plays this audio to the caller in real-time ​ Server Implementation The server needs to: Serve TwiML in response to Twilio’s HTTP requests Handle WebSocket connections Process audio with the Pipecat pipeline Let’s look at the key components: ​ FastAPI Server Setup Copy Ask AI from fastapi import FastAPI, WebSocket from starlette.responses import HTMLResponse app = FastAPI() @app.post ( "/" ) async def start_call (): return HTMLResponse( content = open ( "templates/streams.xml" ).read(), media_type = "application/xml" ) @app.websocket ( "/ws" ) async def websocket_endpoint ( websocket : WebSocket): await websocket.accept() # Read initial WebSocket messages start_data = ws.iter_text() await start_data. __anext__ () # Second message contains the call details call_data = json.loads( await start_data. __anext__ ()) # Extract both StreamSid and CallSid stream_sid = call_data[ "start" ][ "streamSid" ] call_sid = call_data[ "start" ][ "callSid" ] # Run your Pipecat bot await run_bot(websocket, stream_sid, call_sid, app.state.testing) This server has two main endpoints: POST / - Returns TwiML instructions to Twilio WebSocket /ws - Handles the WebSocket connection for real-time audio ​ TwiML Configuration The TwiML tells Twilio to establish a WebSocket connection with your server: Copy Ask AI <? xml version = "1.0" encoding = "UTF-8" ?> < Response > < Connect > < Stream url = "wss://<your server url>/ws" ></ Stream > </ Connect > < Pause length = "40" /> </ Response > Replace <your server url> with your server’s publicly accessible domain. The Pause element keeps the call alive for a maximum of 40 seconds. Adjust this value based on your expected conversation length. ​ Pipecat Bot Implementation The run_bot function creates and connects all the components in the Pipecat pipeline: Copy Ask AI async def run_bot ( websocket_client : WebSocket, stream_sid : str , call_sid : str , testing : bool ): # Initialize the Twilio serializer with stream and call SIDs serializer = TwilioFrameSerializer( stream_sid = stream_sid, call_sid = call_sid, account_sid = os.getenv( "TWILIO_ACCOUNT_SID" , "" ), auth_token = os.getenv( "TWILIO_AUTH_TOKEN" , "" ), ) # Create the WebSocket transport transport = FastAPIWebsocketTransport( websocket = websocket_client, params = FastAPIWebsocketParams( audio_in_enabled = True , audio_out_enabled = True , add_wav_header = False , vad_analyzer = SileroVADAnalyzer(), serializer = serializer, ), ) # Initialize AI services llm = OpenAILLMService( api_key = os.getenv( "OPENAI_API_KEY" )) stt = DeepgramSTTService( api_key = os.getenv( "DEEPGRAM_API_KEY" ), audio_passthrough = True ) tts = CartesiaTTSService( api_key = os.getenv( "CARTESIA_API_KEY" ), voice_id = "71a7ad14-091c-4e8e-a314-022ece01c121" , # British Reading Lady push_silence_after_stop = testing, ) # Create the initial conversation prompt messages = [ { "role" : "system" , "content" : "You are an elementary teacher in an audio call. Your output will be converted to audio so don't include special characters in your answers. Respond to what the student said in a short short sentence." , }, ] # Setup the context aggregator context = OpenAILLMContext(messages) context_aggregator = llm.create_context_aggregator(context) # Create an audio buffer processor to capture conversation audio audiobuffer = AudioBufferProcessor( user_continuous_stream = not testing) # Build the pipeline pipeline = Pipeline( [ transport.input(), # Websocket input from client stt, # Speech-To-Text context_aggregator.user(), llm, # LLM tts, # Text-To-Speech transport.output(), # Websocket output to client audiobuffer, # Used to buffer the audio in the pipeline context_aggregator.assistant(), ] ) # Create a pipeline task with appropriate parameters task = PipelineTask( pipeline, params = PipelineParams( audio_in_sample_rate = 8000 , # Twilio uses 8kHz audio audio_out_sample_rate = 8000 , allow_interruptions = True , ), ) # Setup event handlers @transport.event_handler ( "on_client_connected" ) async def on_client_connected ( transport , client ): # Start recording await audiobuffer.start_recording() # Kickstart the conversation messages.append({ "role" : "system" , "content" : "Please introduce yourself to the user." }) await task.queue_frames([context_aggregator.user().get_context_frame()]) @transport.event_handler ( "on_client_disconnected" ) async def on_client_disconnected ( transport , client ): await task.cancel() @audiobuffer.event_handler ( "on_audio_data" ) async def on_audio_data ( buffer , audio , sample_rate , num_channels ): server_name = f "server_ { websocket_client.client.port } " await save_audio(server_name, audio, sample_rate, num_channels) # Run the pipeline runner = PipelineRunner( handle_sigint = False , force_gc = True ) await runner.run(task) ​ Key Technical Considerations ​ Audio Format and Sample Rate Twilio Media Streams uses 8kHz mono audio with 16-bit PCM encoding. Make sure your pipeline is configured correctly: Copy Ask AI task = PipelineTask( pipeline, params = PipelineParams( audio_in_sample_rate = 8000 , # Twilio's audio format audio_out_sample_rate = 8000 , allow_interruptions = True , ), ) ​ Serialization and Call Control The TwilioFrameSerializer handles the protocol specifics for communicating with Twilio’s Media Streams: Copy Ask AI serializer = TwilioFrameSerializer( stream_sid = stream_sid, call_sid = call_sid, account_sid = os.getenv( "TWILIO_ACCOUNT_SID" , "" ), auth_token = os.getenv( "TWILIO_AUTH_TOKEN" , "" ), ) When you provide the account_sid and auth_token to the TwilioFrameSerializer , it will automatically end the call via Twilio’s REST API when the pipeline ends. This ensures clean call termination when your bot finishes its conversation. ​ Voice Activity Detection The SileroVAD analyzer helps determine when a user has finished speaking: Copy Ask AI params = FastAPIWebsocketParams( audio_in_enabled = True , audio_out_enabled = True , add_wav_header = False , vad_analyzer = SileroVADAnalyzer(), serializer = serializer, ) ​ Configuring Twilio To set up your Twilio phone number to use your server: Purchase a phone number in your Twilio account Navigate to the Phone Numbers section and select your number Under “Voice & Fax”, set the webhook for “A Call Comes In” to your server’s URL (e.g., https://your-server.com/ ) Make sure the request type is set to HTTP POST Save your changes If you’re using ngrok for local development, your webhook URL will look like https://abc123.ngrok.io/ . Remember to update your TwiML template with the correct WebSocket URL as well. ​ Testing Your Implementation ​ Local Testing Without Phone Calls The example includes a test client that can simulate phone calls without actually using Twilio: Copy Ask AI python server.py - t # Start server in testing mode python client.py - u http: // localhost: 8765 - c 2 # Start 2 test clients The -t flag puts the server in testing mode, and the test client creates virtual clients that communicate with your server as if they were Twilio Media Streams. ​ Using the Phone To test with an actual phone call: Make sure your server is running and accessible via the internet Configure Twilio as described above Dial your Twilio phone number from any phone You should hear your AI agent respond! ​ Ending a Conversation There are two primary ways to end a conversation: Automatic termination : If you provided Twilio credentials to the TwilioFrameSerializer , the call will be ended automatically when your pipeline ends: Copy Ask AI # End the conversation programmatically await task.queue_frame(EndFrame()) Manual termination : You can also end the call explicitly through Twilio’s REST API. This is useful for implementing custom hang-up logic: Copy Ask AI from twilio.rest import Client client = Client(account_sid, auth_token) client.calls(call_sid).update( status = "completed" ) ​ Scaling Considerations For production deployments, consider: Multiple concurrent calls : Each bot instance should run in its own process to handle concurrent calls efficiently. The example server can spawn individual bot processes for each call: Copy Ask AI proc = subprocess.Popen( [ f "python3 -m bot_twilio --stream-sid { stream_sid } --call-sid { call_sid } " ], shell = True , bufsize = 1 , cwd = os.path.dirname(os.path.abspath( __file__ )) ) Error handling : Add robust error handling for network issues, service outages, etc. Logging and monitoring : Implement detailed logging to track call quality and agent performance. Security : Add authentication to your endpoints and use environment variables for all sensitive credentials. ​ Next Steps Try the complete example Clone the full working Twilio chatbot example from GitHub and run it yourself. Deploy to Pipecat Cloud Learn how to deploy your Twilio voice AI agent to production with Pipecat Cloud’s managed infrastructure. Dial-in: WebRTC (Twilio + Daily) Dialout: WebRTC (Daily) On this page Things you’ll need Architecture Overview Server Implementation FastAPI Server Setup TwiML Configuration Pipecat Bot Implementation Key Technical Considerations Audio Format and Sample Rate Serialization and Call Control Voice Activity Detection Configuring Twilio Testing Your Implementation Local Testing Without Phone Calls Using the Phone Ending a Conversation Scaling Considerations Next Steps Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/telephony_twilio-websockets_29b698cc.txt b/telephony_twilio-websockets_29b698cc.txt
new file mode 100644
index 0000000000000000000000000000000000000000..abad10422bcacb73004ce69d7f721047fab7cdb9
--- /dev/null
+++ b/telephony_twilio-websockets_29b698cc.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/guides/telephony/twilio-websockets#server-implementation
+Title: Dial-in: Twilio (Media Streams) - Pipecat
+==================================================
+
+Dial-in: Twilio (Media Streams) - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Telephony Dial-in: Twilio (Media Streams) Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Guides Fundamentals Context Management Custom FrameProcessor Detecting Idle Users Ending a Pipeline Function Calling Muting User Input Recording Audio Recording Transcripts Features Gemini Multimodal Live Metrics Noise cancellation with Krisp OpenAI Audio Models and APIs Pipecat Flows Telephony Overview Dial-in: WebRTC (Daily) Dial-in: WebRTC (Twilio + Daily) Dial-in: Twilio (Media Streams) Dialout: WebRTC (Daily) Deploying your bot Overview Deployment pattern Example: Pipecat Cloud Example: Fly.io Example: Cerebrium Example: Modal This guide walks through creating a voice AI agent that users can reach by dialing a Twilio phone number. We’ll integrate Twilio Media Streams via WebSockets with a Pipecat pipeline to create a fully functional voice AI experience. ​ Things you’ll need An active Twilio account with at least one phone number A public-facing server or a tunneling service like ngrok API keys for speech-to-text, text-to-speech, and LLM services Complete example code on GitHub The full source code for this example is available in the Pipecat repository. ​ Architecture Overview When a user dials your Twilio phone number: Twilio calls your server’s endpoint, which returns TwiML that establishes a WebSocket connection Twilio sends real-time audio data over the WebSocket, and your server processes it using the Pipecat pipeline Your Pipecat agent processes the audio which is sent to the pipeline, which then outputs audio data back to Twilio Twilio plays this audio to the caller in real-time ​ Server Implementation The server needs to: Serve TwiML in response to Twilio’s HTTP requests Handle WebSocket connections Process audio with the Pipecat pipeline Let’s look at the key components: ​ FastAPI Server Setup Copy Ask AI from fastapi import FastAPI, WebSocket from starlette.responses import HTMLResponse app = FastAPI() @app.post ( "/" ) async def start_call (): return HTMLResponse( content = open ( "templates/streams.xml" ).read(), media_type = "application/xml" ) @app.websocket ( "/ws" ) async def websocket_endpoint ( websocket : WebSocket): await websocket.accept() # Read initial WebSocket messages start_data = ws.iter_text() await start_data. __anext__ () # Second message contains the call details call_data = json.loads( await start_data. __anext__ ()) # Extract both StreamSid and CallSid stream_sid = call_data[ "start" ][ "streamSid" ] call_sid = call_data[ "start" ][ "callSid" ] # Run your Pipecat bot await run_bot(websocket, stream_sid, call_sid, app.state.testing) This server has two main endpoints: POST / - Returns TwiML instructions to Twilio WebSocket /ws - Handles the WebSocket connection for real-time audio ​ TwiML Configuration The TwiML tells Twilio to establish a WebSocket connection with your server: Copy Ask AI <? xml version = "1.0" encoding = "UTF-8" ?> < Response > < Connect > < Stream url = "wss://<your server url>/ws" ></ Stream > </ Connect > < Pause length = "40" /> </ Response > Replace <your server url> with your server’s publicly accessible domain. The Pause element keeps the call alive for a maximum of 40 seconds. Adjust this value based on your expected conversation length. ​ Pipecat Bot Implementation The run_bot function creates and connects all the components in the Pipecat pipeline: Copy Ask AI async def run_bot ( websocket_client : WebSocket, stream_sid : str , call_sid : str , testing : bool ): # Initialize the Twilio serializer with stream and call SIDs serializer = TwilioFrameSerializer( stream_sid = stream_sid, call_sid = call_sid, account_sid = os.getenv( "TWILIO_ACCOUNT_SID" , "" ), auth_token = os.getenv( "TWILIO_AUTH_TOKEN" , "" ), ) # Create the WebSocket transport transport = FastAPIWebsocketTransport( websocket = websocket_client, params = FastAPIWebsocketParams( audio_in_enabled = True , audio_out_enabled = True , add_wav_header = False , vad_analyzer = SileroVADAnalyzer(), serializer = serializer, ), ) # Initialize AI services llm = OpenAILLMService( api_key = os.getenv( "OPENAI_API_KEY" )) stt = DeepgramSTTService( api_key = os.getenv( "DEEPGRAM_API_KEY" ), audio_passthrough = True ) tts = CartesiaTTSService( api_key = os.getenv( "CARTESIA_API_KEY" ), voice_id = "71a7ad14-091c-4e8e-a314-022ece01c121" , # British Reading Lady push_silence_after_stop = testing, ) # Create the initial conversation prompt messages = [ { "role" : "system" , "content" : "You are an elementary teacher in an audio call. Your output will be converted to audio so don't include special characters in your answers. Respond to what the student said in a short short sentence." , }, ] # Setup the context aggregator context = OpenAILLMContext(messages) context_aggregator = llm.create_context_aggregator(context) # Create an audio buffer processor to capture conversation audio audiobuffer = AudioBufferProcessor( user_continuous_stream = not testing) # Build the pipeline pipeline = Pipeline( [ transport.input(), # Websocket input from client stt, # Speech-To-Text context_aggregator.user(), llm, # LLM tts, # Text-To-Speech transport.output(), # Websocket output to client audiobuffer, # Used to buffer the audio in the pipeline context_aggregator.assistant(), ] ) # Create a pipeline task with appropriate parameters task = PipelineTask( pipeline, params = PipelineParams( audio_in_sample_rate = 8000 , # Twilio uses 8kHz audio audio_out_sample_rate = 8000 , allow_interruptions = True , ), ) # Setup event handlers @transport.event_handler ( "on_client_connected" ) async def on_client_connected ( transport , client ): # Start recording await audiobuffer.start_recording() # Kickstart the conversation messages.append({ "role" : "system" , "content" : "Please introduce yourself to the user." }) await task.queue_frames([context_aggregator.user().get_context_frame()]) @transport.event_handler ( "on_client_disconnected" ) async def on_client_disconnected ( transport , client ): await task.cancel() @audiobuffer.event_handler ( "on_audio_data" ) async def on_audio_data ( buffer , audio , sample_rate , num_channels ): server_name = f "server_ { websocket_client.client.port } " await save_audio(server_name, audio, sample_rate, num_channels) # Run the pipeline runner = PipelineRunner( handle_sigint = False , force_gc = True ) await runner.run(task) ​ Key Technical Considerations ​ Audio Format and Sample Rate Twilio Media Streams uses 8kHz mono audio with 16-bit PCM encoding. Make sure your pipeline is configured correctly: Copy Ask AI task = PipelineTask( pipeline, params = PipelineParams( audio_in_sample_rate = 8000 , # Twilio's audio format audio_out_sample_rate = 8000 , allow_interruptions = True , ), ) ​ Serialization and Call Control The TwilioFrameSerializer handles the protocol specifics for communicating with Twilio’s Media Streams: Copy Ask AI serializer = TwilioFrameSerializer( stream_sid = stream_sid, call_sid = call_sid, account_sid = os.getenv( "TWILIO_ACCOUNT_SID" , "" ), auth_token = os.getenv( "TWILIO_AUTH_TOKEN" , "" ), ) When you provide the account_sid and auth_token to the TwilioFrameSerializer , it will automatically end the call via Twilio’s REST API when the pipeline ends. This ensures clean call termination when your bot finishes its conversation. ​ Voice Activity Detection The SileroVAD analyzer helps determine when a user has finished speaking: Copy Ask AI params = FastAPIWebsocketParams( audio_in_enabled = True , audio_out_enabled = True , add_wav_header = False , vad_analyzer = SileroVADAnalyzer(), serializer = serializer, ) ​ Configuring Twilio To set up your Twilio phone number to use your server: Purchase a phone number in your Twilio account Navigate to the Phone Numbers section and select your number Under “Voice & Fax”, set the webhook for “A Call Comes In” to your server’s URL (e.g., https://your-server.com/ ) Make sure the request type is set to HTTP POST Save your changes If you’re using ngrok for local development, your webhook URL will look like https://abc123.ngrok.io/ . Remember to update your TwiML template with the correct WebSocket URL as well. ​ Testing Your Implementation ​ Local Testing Without Phone Calls The example includes a test client that can simulate phone calls without actually using Twilio: Copy Ask AI python server.py - t # Start server in testing mode python client.py - u http: // localhost: 8765 - c 2 # Start 2 test clients The -t flag puts the server in testing mode, and the test client creates virtual clients that communicate with your server as if they were Twilio Media Streams. ​ Using the Phone To test with an actual phone call: Make sure your server is running and accessible via the internet Configure Twilio as described above Dial your Twilio phone number from any phone You should hear your AI agent respond! ​ Ending a Conversation There are two primary ways to end a conversation: Automatic termination : If you provided Twilio credentials to the TwilioFrameSerializer , the call will be ended automatically when your pipeline ends: Copy Ask AI # End the conversation programmatically await task.queue_frame(EndFrame()) Manual termination : You can also end the call explicitly through Twilio’s REST API. This is useful for implementing custom hang-up logic: Copy Ask AI from twilio.rest import Client client = Client(account_sid, auth_token) client.calls(call_sid).update( status = "completed" ) ​ Scaling Considerations For production deployments, consider: Multiple concurrent calls : Each bot instance should run in its own process to handle concurrent calls efficiently. The example server can spawn individual bot processes for each call: Copy Ask AI proc = subprocess.Popen( [ f "python3 -m bot_twilio --stream-sid { stream_sid } --call-sid { call_sid } " ], shell = True , bufsize = 1 , cwd = os.path.dirname(os.path.abspath( __file__ )) ) Error handling : Add robust error handling for network issues, service outages, etc. Logging and monitoring : Implement detailed logging to track call quality and agent performance. Security : Add authentication to your endpoints and use environment variables for all sensitive credentials. ​ Next Steps Try the complete example Clone the full working Twilio chatbot example from GitHub and run it yourself. Deploy to Pipecat Cloud Learn how to deploy your Twilio voice AI agent to production with Pipecat Cloud’s managed infrastructure. Dial-in: WebRTC (Twilio + Daily) Dialout: WebRTC (Daily) On this page Things you’ll need Architecture Overview Server Implementation FastAPI Server Setup TwiML Configuration Pipecat Bot Implementation Key Technical Considerations Audio Format and Sample Rate Serialization and Call Control Voice Activity Detection Configuring Twilio Testing Your Implementation Local Testing Without Phone Calls Using the Phone Ending a Conversation Scaling Considerations Next Steps Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/telephony_twilio-websockets_89e0bace.txt b/telephony_twilio-websockets_89e0bace.txt
new file mode 100644
index 0000000000000000000000000000000000000000..2559f298996dad5c396b4298591c141f8e347d2d
--- /dev/null
+++ b/telephony_twilio-websockets_89e0bace.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/guides/telephony/twilio-websockets#ending-a-conversation
+Title: Dial-in: Twilio (Media Streams) - Pipecat
+==================================================
+
+Dial-in: Twilio (Media Streams) - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Telephony Dial-in: Twilio (Media Streams) Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Guides Fundamentals Context Management Custom FrameProcessor Detecting Idle Users Ending a Pipeline Function Calling Muting User Input Recording Audio Recording Transcripts Features Gemini Multimodal Live Metrics Noise cancellation with Krisp OpenAI Audio Models and APIs Pipecat Flows Telephony Overview Dial-in: WebRTC (Daily) Dial-in: WebRTC (Twilio + Daily) Dial-in: Twilio (Media Streams) Dialout: WebRTC (Daily) Deploying your bot Overview Deployment pattern Example: Pipecat Cloud Example: Fly.io Example: Cerebrium Example: Modal This guide walks through creating a voice AI agent that users can reach by dialing a Twilio phone number. We’ll integrate Twilio Media Streams via WebSockets with a Pipecat pipeline to create a fully functional voice AI experience. ​ Things you’ll need An active Twilio account with at least one phone number A public-facing server or a tunneling service like ngrok API keys for speech-to-text, text-to-speech, and LLM services Complete example code on GitHub The full source code for this example is available in the Pipecat repository. ​ Architecture Overview When a user dials your Twilio phone number: Twilio calls your server’s endpoint, which returns TwiML that establishes a WebSocket connection Twilio sends real-time audio data over the WebSocket, and your server processes it using the Pipecat pipeline Your Pipecat agent processes the audio which is sent to the pipeline, which then outputs audio data back to Twilio Twilio plays this audio to the caller in real-time ​ Server Implementation The server needs to: Serve TwiML in response to Twilio’s HTTP requests Handle WebSocket connections Process audio with the Pipecat pipeline Let’s look at the key components: ​ FastAPI Server Setup Copy Ask AI from fastapi import FastAPI, WebSocket from starlette.responses import HTMLResponse app = FastAPI() @app.post ( "/" ) async def start_call (): return HTMLResponse( content = open ( "templates/streams.xml" ).read(), media_type = "application/xml" ) @app.websocket ( "/ws" ) async def websocket_endpoint ( websocket : WebSocket): await websocket.accept() # Read initial WebSocket messages start_data = ws.iter_text() await start_data. __anext__ () # Second message contains the call details call_data = json.loads( await start_data. __anext__ ()) # Extract both StreamSid and CallSid stream_sid = call_data[ "start" ][ "streamSid" ] call_sid = call_data[ "start" ][ "callSid" ] # Run your Pipecat bot await run_bot(websocket, stream_sid, call_sid, app.state.testing) This server has two main endpoints: POST / - Returns TwiML instructions to Twilio WebSocket /ws - Handles the WebSocket connection for real-time audio ​ TwiML Configuration The TwiML tells Twilio to establish a WebSocket connection with your server: Copy Ask AI <? xml version = "1.0" encoding = "UTF-8" ?> < Response > < Connect > < Stream url = "wss://<your server url>/ws" ></ Stream > </ Connect > < Pause length = "40" /> </ Response > Replace <your server url> with your server’s publicly accessible domain. The Pause element keeps the call alive for a maximum of 40 seconds. Adjust this value based on your expected conversation length. ​ Pipecat Bot Implementation The run_bot function creates and connects all the components in the Pipecat pipeline: Copy Ask AI async def run_bot ( websocket_client : WebSocket, stream_sid : str , call_sid : str , testing : bool ): # Initialize the Twilio serializer with stream and call SIDs serializer = TwilioFrameSerializer( stream_sid = stream_sid, call_sid = call_sid, account_sid = os.getenv( "TWILIO_ACCOUNT_SID" , "" ), auth_token = os.getenv( "TWILIO_AUTH_TOKEN" , "" ), ) # Create the WebSocket transport transport = FastAPIWebsocketTransport( websocket = websocket_client, params = FastAPIWebsocketParams( audio_in_enabled = True , audio_out_enabled = True , add_wav_header = False , vad_analyzer = SileroVADAnalyzer(), serializer = serializer, ), ) # Initialize AI services llm = OpenAILLMService( api_key = os.getenv( "OPENAI_API_KEY" )) stt = DeepgramSTTService( api_key = os.getenv( "DEEPGRAM_API_KEY" ), audio_passthrough = True ) tts = CartesiaTTSService( api_key = os.getenv( "CARTESIA_API_KEY" ), voice_id = "71a7ad14-091c-4e8e-a314-022ece01c121" , # British Reading Lady push_silence_after_stop = testing, ) # Create the initial conversation prompt messages = [ { "role" : "system" , "content" : "You are an elementary teacher in an audio call. Your output will be converted to audio so don't include special characters in your answers. Respond to what the student said in a short short sentence." , }, ] # Setup the context aggregator context = OpenAILLMContext(messages) context_aggregator = llm.create_context_aggregator(context) # Create an audio buffer processor to capture conversation audio audiobuffer = AudioBufferProcessor( user_continuous_stream = not testing) # Build the pipeline pipeline = Pipeline( [ transport.input(), # Websocket input from client stt, # Speech-To-Text context_aggregator.user(), llm, # LLM tts, # Text-To-Speech transport.output(), # Websocket output to client audiobuffer, # Used to buffer the audio in the pipeline context_aggregator.assistant(), ] ) # Create a pipeline task with appropriate parameters task = PipelineTask( pipeline, params = PipelineParams( audio_in_sample_rate = 8000 , # Twilio uses 8kHz audio audio_out_sample_rate = 8000 , allow_interruptions = True , ), ) # Setup event handlers @transport.event_handler ( "on_client_connected" ) async def on_client_connected ( transport , client ): # Start recording await audiobuffer.start_recording() # Kickstart the conversation messages.append({ "role" : "system" , "content" : "Please introduce yourself to the user." }) await task.queue_frames([context_aggregator.user().get_context_frame()]) @transport.event_handler ( "on_client_disconnected" ) async def on_client_disconnected ( transport , client ): await task.cancel() @audiobuffer.event_handler ( "on_audio_data" ) async def on_audio_data ( buffer , audio , sample_rate , num_channels ): server_name = f "server_ { websocket_client.client.port } " await save_audio(server_name, audio, sample_rate, num_channels) # Run the pipeline runner = PipelineRunner( handle_sigint = False , force_gc = True ) await runner.run(task) ​ Key Technical Considerations ​ Audio Format and Sample Rate Twilio Media Streams uses 8kHz mono audio with 16-bit PCM encoding. Make sure your pipeline is configured correctly: Copy Ask AI task = PipelineTask( pipeline, params = PipelineParams( audio_in_sample_rate = 8000 , # Twilio's audio format audio_out_sample_rate = 8000 , allow_interruptions = True , ), ) ​ Serialization and Call Control The TwilioFrameSerializer handles the protocol specifics for communicating with Twilio’s Media Streams: Copy Ask AI serializer = TwilioFrameSerializer( stream_sid = stream_sid, call_sid = call_sid, account_sid = os.getenv( "TWILIO_ACCOUNT_SID" , "" ), auth_token = os.getenv( "TWILIO_AUTH_TOKEN" , "" ), ) When you provide the account_sid and auth_token to the TwilioFrameSerializer , it will automatically end the call via Twilio’s REST API when the pipeline ends. This ensures clean call termination when your bot finishes its conversation. ​ Voice Activity Detection The SileroVAD analyzer helps determine when a user has finished speaking: Copy Ask AI params = FastAPIWebsocketParams( audio_in_enabled = True , audio_out_enabled = True , add_wav_header = False , vad_analyzer = SileroVADAnalyzer(), serializer = serializer, ) ​ Configuring Twilio To set up your Twilio phone number to use your server: Purchase a phone number in your Twilio account Navigate to the Phone Numbers section and select your number Under “Voice & Fax”, set the webhook for “A Call Comes In” to your server’s URL (e.g., https://your-server.com/ ) Make sure the request type is set to HTTP POST Save your changes If you’re using ngrok for local development, your webhook URL will look like https://abc123.ngrok.io/ . Remember to update your TwiML template with the correct WebSocket URL as well. ​ Testing Your Implementation ​ Local Testing Without Phone Calls The example includes a test client that can simulate phone calls without actually using Twilio: Copy Ask AI python server.py - t # Start server in testing mode python client.py - u http: // localhost: 8765 - c 2 # Start 2 test clients The -t flag puts the server in testing mode, and the test client creates virtual clients that communicate with your server as if they were Twilio Media Streams. ​ Using the Phone To test with an actual phone call: Make sure your server is running and accessible via the internet Configure Twilio as described above Dial your Twilio phone number from any phone You should hear your AI agent respond! ​ Ending a Conversation There are two primary ways to end a conversation: Automatic termination : If you provided Twilio credentials to the TwilioFrameSerializer , the call will be ended automatically when your pipeline ends: Copy Ask AI # End the conversation programmatically await task.queue_frame(EndFrame()) Manual termination : You can also end the call explicitly through Twilio’s REST API. This is useful for implementing custom hang-up logic: Copy Ask AI from twilio.rest import Client client = Client(account_sid, auth_token) client.calls(call_sid).update( status = "completed" ) ​ Scaling Considerations For production deployments, consider: Multiple concurrent calls : Each bot instance should run in its own process to handle concurrent calls efficiently. The example server can spawn individual bot processes for each call: Copy Ask AI proc = subprocess.Popen( [ f "python3 -m bot_twilio --stream-sid { stream_sid } --call-sid { call_sid } " ], shell = True , bufsize = 1 , cwd = os.path.dirname(os.path.abspath( __file__ )) ) Error handling : Add robust error handling for network issues, service outages, etc. Logging and monitoring : Implement detailed logging to track call quality and agent performance. Security : Add authentication to your endpoints and use environment variables for all sensitive credentials. ​ Next Steps Try the complete example Clone the full working Twilio chatbot example from GitHub and run it yourself. Deploy to Pipecat Cloud Learn how to deploy your Twilio voice AI agent to production with Pipecat Cloud’s managed infrastructure. Dial-in: WebRTC (Twilio + Daily) Dialout: WebRTC (Daily) On this page Things you’ll need Architecture Overview Server Implementation FastAPI Server Setup TwiML Configuration Pipecat Bot Implementation Key Technical Considerations Audio Format and Sample Rate Serialization and Call Control Voice Activity Detection Configuring Twilio Testing Your Implementation Local Testing Without Phone Calls Using the Phone Ending a Conversation Scaling Considerations Next Steps Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/telephony_twilio-websockets_96db3736.txt b/telephony_twilio-websockets_96db3736.txt
new file mode 100644
index 0000000000000000000000000000000000000000..9111512be29c08d56b02ef2da44220014fdcfe3a
--- /dev/null
+++ b/telephony_twilio-websockets_96db3736.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/guides/telephony/twilio-websockets#twiml-configuration
+Title: Dial-in: Twilio (Media Streams) - Pipecat
+==================================================
+
+Dial-in: Twilio (Media Streams) - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Telephony Dial-in: Twilio (Media Streams) Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Guides Fundamentals Context Management Custom FrameProcessor Detecting Idle Users Ending a Pipeline Function Calling Muting User Input Recording Audio Recording Transcripts Features Gemini Multimodal Live Metrics Noise cancellation with Krisp OpenAI Audio Models and APIs Pipecat Flows Telephony Overview Dial-in: WebRTC (Daily) Dial-in: WebRTC (Twilio + Daily) Dial-in: Twilio (Media Streams) Dialout: WebRTC (Daily) Deploying your bot Overview Deployment pattern Example: Pipecat Cloud Example: Fly.io Example: Cerebrium Example: Modal This guide walks through creating a voice AI agent that users can reach by dialing a Twilio phone number. We’ll integrate Twilio Media Streams via WebSockets with a Pipecat pipeline to create a fully functional voice AI experience. ​ Things you’ll need An active Twilio account with at least one phone number A public-facing server or a tunneling service like ngrok API keys for speech-to-text, text-to-speech, and LLM services Complete example code on GitHub The full source code for this example is available in the Pipecat repository. ​ Architecture Overview When a user dials your Twilio phone number: Twilio calls your server’s endpoint, which returns TwiML that establishes a WebSocket connection Twilio sends real-time audio data over the WebSocket, and your server processes it using the Pipecat pipeline Your Pipecat agent processes the audio which is sent to the pipeline, which then outputs audio data back to Twilio Twilio plays this audio to the caller in real-time ​ Server Implementation The server needs to: Serve TwiML in response to Twilio’s HTTP requests Handle WebSocket connections Process audio with the Pipecat pipeline Let’s look at the key components: ​ FastAPI Server Setup Copy Ask AI from fastapi import FastAPI, WebSocket from starlette.responses import HTMLResponse app = FastAPI() @app.post ( "/" ) async def start_call (): return HTMLResponse( content = open ( "templates/streams.xml" ).read(), media_type = "application/xml" ) @app.websocket ( "/ws" ) async def websocket_endpoint ( websocket : WebSocket): await websocket.accept() # Read initial WebSocket messages start_data = ws.iter_text() await start_data. __anext__ () # Second message contains the call details call_data = json.loads( await start_data. __anext__ ()) # Extract both StreamSid and CallSid stream_sid = call_data[ "start" ][ "streamSid" ] call_sid = call_data[ "start" ][ "callSid" ] # Run your Pipecat bot await run_bot(websocket, stream_sid, call_sid, app.state.testing) This server has two main endpoints: POST / - Returns TwiML instructions to Twilio WebSocket /ws - Handles the WebSocket connection for real-time audio ​ TwiML Configuration The TwiML tells Twilio to establish a WebSocket connection with your server: Copy Ask AI <? xml version = "1.0" encoding = "UTF-8" ?> < Response > < Connect > < Stream url = "wss://<your server url>/ws" ></ Stream > </ Connect > < Pause length = "40" /> </ Response > Replace <your server url> with your server’s publicly accessible domain. The Pause element keeps the call alive for a maximum of 40 seconds. Adjust this value based on your expected conversation length. ​ Pipecat Bot Implementation The run_bot function creates and connects all the components in the Pipecat pipeline: Copy Ask AI async def run_bot ( websocket_client : WebSocket, stream_sid : str , call_sid : str , testing : bool ): # Initialize the Twilio serializer with stream and call SIDs serializer = TwilioFrameSerializer( stream_sid = stream_sid, call_sid = call_sid, account_sid = os.getenv( "TWILIO_ACCOUNT_SID" , "" ), auth_token = os.getenv( "TWILIO_AUTH_TOKEN" , "" ), ) # Create the WebSocket transport transport = FastAPIWebsocketTransport( websocket = websocket_client, params = FastAPIWebsocketParams( audio_in_enabled = True , audio_out_enabled = True , add_wav_header = False , vad_analyzer = SileroVADAnalyzer(), serializer = serializer, ), ) # Initialize AI services llm = OpenAILLMService( api_key = os.getenv( "OPENAI_API_KEY" )) stt = DeepgramSTTService( api_key = os.getenv( "DEEPGRAM_API_KEY" ), audio_passthrough = True ) tts = CartesiaTTSService( api_key = os.getenv( "CARTESIA_API_KEY" ), voice_id = "71a7ad14-091c-4e8e-a314-022ece01c121" , # British Reading Lady push_silence_after_stop = testing, ) # Create the initial conversation prompt messages = [ { "role" : "system" , "content" : "You are an elementary teacher in an audio call. Your output will be converted to audio so don't include special characters in your answers. Respond to what the student said in a short short sentence." , }, ] # Setup the context aggregator context = OpenAILLMContext(messages) context_aggregator = llm.create_context_aggregator(context) # Create an audio buffer processor to capture conversation audio audiobuffer = AudioBufferProcessor( user_continuous_stream = not testing) # Build the pipeline pipeline = Pipeline( [ transport.input(), # Websocket input from client stt, # Speech-To-Text context_aggregator.user(), llm, # LLM tts, # Text-To-Speech transport.output(), # Websocket output to client audiobuffer, # Used to buffer the audio in the pipeline context_aggregator.assistant(), ] ) # Create a pipeline task with appropriate parameters task = PipelineTask( pipeline, params = PipelineParams( audio_in_sample_rate = 8000 , # Twilio uses 8kHz audio audio_out_sample_rate = 8000 , allow_interruptions = True , ), ) # Setup event handlers @transport.event_handler ( "on_client_connected" ) async def on_client_connected ( transport , client ): # Start recording await audiobuffer.start_recording() # Kickstart the conversation messages.append({ "role" : "system" , "content" : "Please introduce yourself to the user." }) await task.queue_frames([context_aggregator.user().get_context_frame()]) @transport.event_handler ( "on_client_disconnected" ) async def on_client_disconnected ( transport , client ): await task.cancel() @audiobuffer.event_handler ( "on_audio_data" ) async def on_audio_data ( buffer , audio , sample_rate , num_channels ): server_name = f "server_ { websocket_client.client.port } " await save_audio(server_name, audio, sample_rate, num_channels) # Run the pipeline runner = PipelineRunner( handle_sigint = False , force_gc = True ) await runner.run(task) ​ Key Technical Considerations ​ Audio Format and Sample Rate Twilio Media Streams uses 8kHz mono audio with 16-bit PCM encoding. Make sure your pipeline is configured correctly: Copy Ask AI task = PipelineTask( pipeline, params = PipelineParams( audio_in_sample_rate = 8000 , # Twilio's audio format audio_out_sample_rate = 8000 , allow_interruptions = True , ), ) ​ Serialization and Call Control The TwilioFrameSerializer handles the protocol specifics for communicating with Twilio’s Media Streams: Copy Ask AI serializer = TwilioFrameSerializer( stream_sid = stream_sid, call_sid = call_sid, account_sid = os.getenv( "TWILIO_ACCOUNT_SID" , "" ), auth_token = os.getenv( "TWILIO_AUTH_TOKEN" , "" ), ) When you provide the account_sid and auth_token to the TwilioFrameSerializer , it will automatically end the call via Twilio’s REST API when the pipeline ends. This ensures clean call termination when your bot finishes its conversation. ​ Voice Activity Detection The SileroVAD analyzer helps determine when a user has finished speaking: Copy Ask AI params = FastAPIWebsocketParams( audio_in_enabled = True , audio_out_enabled = True , add_wav_header = False , vad_analyzer = SileroVADAnalyzer(), serializer = serializer, ) ​ Configuring Twilio To set up your Twilio phone number to use your server: Purchase a phone number in your Twilio account Navigate to the Phone Numbers section and select your number Under “Voice & Fax”, set the webhook for “A Call Comes In” to your server’s URL (e.g., https://your-server.com/ ) Make sure the request type is set to HTTP POST Save your changes If you’re using ngrok for local development, your webhook URL will look like https://abc123.ngrok.io/ . Remember to update your TwiML template with the correct WebSocket URL as well. ​ Testing Your Implementation ​ Local Testing Without Phone Calls The example includes a test client that can simulate phone calls without actually using Twilio: Copy Ask AI python server.py - t # Start server in testing mode python client.py - u http: // localhost: 8765 - c 2 # Start 2 test clients The -t flag puts the server in testing mode, and the test client creates virtual clients that communicate with your server as if they were Twilio Media Streams. ​ Using the Phone To test with an actual phone call: Make sure your server is running and accessible via the internet Configure Twilio as described above Dial your Twilio phone number from any phone You should hear your AI agent respond! ​ Ending a Conversation There are two primary ways to end a conversation: Automatic termination : If you provided Twilio credentials to the TwilioFrameSerializer , the call will be ended automatically when your pipeline ends: Copy Ask AI # End the conversation programmatically await task.queue_frame(EndFrame()) Manual termination : You can also end the call explicitly through Twilio’s REST API. This is useful for implementing custom hang-up logic: Copy Ask AI from twilio.rest import Client client = Client(account_sid, auth_token) client.calls(call_sid).update( status = "completed" ) ​ Scaling Considerations For production deployments, consider: Multiple concurrent calls : Each bot instance should run in its own process to handle concurrent calls efficiently. The example server can spawn individual bot processes for each call: Copy Ask AI proc = subprocess.Popen( [ f "python3 -m bot_twilio --stream-sid { stream_sid } --call-sid { call_sid } " ], shell = True , bufsize = 1 , cwd = os.path.dirname(os.path.abspath( __file__ )) ) Error handling : Add robust error handling for network issues, service outages, etc. Logging and monitoring : Implement detailed logging to track call quality and agent performance. Security : Add authentication to your endpoints and use environment variables for all sensitive credentials. ​ Next Steps Try the complete example Clone the full working Twilio chatbot example from GitHub and run it yourself. Deploy to Pipecat Cloud Learn how to deploy your Twilio voice AI agent to production with Pipecat Cloud’s managed infrastructure. Dial-in: WebRTC (Twilio + Daily) Dialout: WebRTC (Daily) On this page Things you’ll need Architecture Overview Server Implementation FastAPI Server Setup TwiML Configuration Pipecat Bot Implementation Key Technical Considerations Audio Format and Sample Rate Serialization and Call Control Voice Activity Detection Configuring Twilio Testing Your Implementation Local Testing Without Phone Calls Using the Phone Ending a Conversation Scaling Considerations Next Steps Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/telephony_twilio-websockets_982a5f17.txt b/telephony_twilio-websockets_982a5f17.txt
new file mode 100644
index 0000000000000000000000000000000000000000..d21038e9c899381e9d1c80c587c20b323b8d564b
--- /dev/null
+++ b/telephony_twilio-websockets_982a5f17.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/guides/telephony/twilio-websockets#architecture-overview
+Title: Dial-in: Twilio (Media Streams) - Pipecat
+==================================================
+
+Dial-in: Twilio (Media Streams) - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Telephony Dial-in: Twilio (Media Streams) Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Guides Fundamentals Context Management Custom FrameProcessor Detecting Idle Users Ending a Pipeline Function Calling Muting User Input Recording Audio Recording Transcripts Features Gemini Multimodal Live Metrics Noise cancellation with Krisp OpenAI Audio Models and APIs Pipecat Flows Telephony Overview Dial-in: WebRTC (Daily) Dial-in: WebRTC (Twilio + Daily) Dial-in: Twilio (Media Streams) Dialout: WebRTC (Daily) Deploying your bot Overview Deployment pattern Example: Pipecat Cloud Example: Fly.io Example: Cerebrium Example: Modal This guide walks through creating a voice AI agent that users can reach by dialing a Twilio phone number. We’ll integrate Twilio Media Streams via WebSockets with a Pipecat pipeline to create a fully functional voice AI experience. ​ Things you’ll need An active Twilio account with at least one phone number A public-facing server or a tunneling service like ngrok API keys for speech-to-text, text-to-speech, and LLM services Complete example code on GitHub The full source code for this example is available in the Pipecat repository. ​ Architecture Overview When a user dials your Twilio phone number: Twilio calls your server’s endpoint, which returns TwiML that establishes a WebSocket connection Twilio sends real-time audio data over the WebSocket, and your server processes it using the Pipecat pipeline Your Pipecat agent processes the audio which is sent to the pipeline, which then outputs audio data back to Twilio Twilio plays this audio to the caller in real-time ​ Server Implementation The server needs to: Serve TwiML in response to Twilio’s HTTP requests Handle WebSocket connections Process audio with the Pipecat pipeline Let’s look at the key components: ​ FastAPI Server Setup Copy Ask AI from fastapi import FastAPI, WebSocket from starlette.responses import HTMLResponse app = FastAPI() @app.post ( "/" ) async def start_call (): return HTMLResponse( content = open ( "templates/streams.xml" ).read(), media_type = "application/xml" ) @app.websocket ( "/ws" ) async def websocket_endpoint ( websocket : WebSocket): await websocket.accept() # Read initial WebSocket messages start_data = ws.iter_text() await start_data. __anext__ () # Second message contains the call details call_data = json.loads( await start_data. __anext__ ()) # Extract both StreamSid and CallSid stream_sid = call_data[ "start" ][ "streamSid" ] call_sid = call_data[ "start" ][ "callSid" ] # Run your Pipecat bot await run_bot(websocket, stream_sid, call_sid, app.state.testing) This server has two main endpoints: POST / - Returns TwiML instructions to Twilio WebSocket /ws - Handles the WebSocket connection for real-time audio ​ TwiML Configuration The TwiML tells Twilio to establish a WebSocket connection with your server: Copy Ask AI <? xml version = "1.0" encoding = "UTF-8" ?> < Response > < Connect > < Stream url = "wss://<your server url>/ws" ></ Stream > </ Connect > < Pause length = "40" /> </ Response > Replace <your server url> with your server’s publicly accessible domain. The Pause element keeps the call alive for a maximum of 40 seconds. Adjust this value based on your expected conversation length. ​ Pipecat Bot Implementation The run_bot function creates and connects all the components in the Pipecat pipeline: Copy Ask AI async def run_bot ( websocket_client : WebSocket, stream_sid : str , call_sid : str , testing : bool ): # Initialize the Twilio serializer with stream and call SIDs serializer = TwilioFrameSerializer( stream_sid = stream_sid, call_sid = call_sid, account_sid = os.getenv( "TWILIO_ACCOUNT_SID" , "" ), auth_token = os.getenv( "TWILIO_AUTH_TOKEN" , "" ), ) # Create the WebSocket transport transport = FastAPIWebsocketTransport( websocket = websocket_client, params = FastAPIWebsocketParams( audio_in_enabled = True , audio_out_enabled = True , add_wav_header = False , vad_analyzer = SileroVADAnalyzer(), serializer = serializer, ), ) # Initialize AI services llm = OpenAILLMService( api_key = os.getenv( "OPENAI_API_KEY" )) stt = DeepgramSTTService( api_key = os.getenv( "DEEPGRAM_API_KEY" ), audio_passthrough = True ) tts = CartesiaTTSService( api_key = os.getenv( "CARTESIA_API_KEY" ), voice_id = "71a7ad14-091c-4e8e-a314-022ece01c121" , # British Reading Lady push_silence_after_stop = testing, ) # Create the initial conversation prompt messages = [ { "role" : "system" , "content" : "You are an elementary teacher in an audio call. Your output will be converted to audio so don't include special characters in your answers. Respond to what the student said in a short short sentence." , }, ] # Setup the context aggregator context = OpenAILLMContext(messages) context_aggregator = llm.create_context_aggregator(context) # Create an audio buffer processor to capture conversation audio audiobuffer = AudioBufferProcessor( user_continuous_stream = not testing) # Build the pipeline pipeline = Pipeline( [ transport.input(), # Websocket input from client stt, # Speech-To-Text context_aggregator.user(), llm, # LLM tts, # Text-To-Speech transport.output(), # Websocket output to client audiobuffer, # Used to buffer the audio in the pipeline context_aggregator.assistant(), ] ) # Create a pipeline task with appropriate parameters task = PipelineTask( pipeline, params = PipelineParams( audio_in_sample_rate = 8000 , # Twilio uses 8kHz audio audio_out_sample_rate = 8000 , allow_interruptions = True , ), ) # Setup event handlers @transport.event_handler ( "on_client_connected" ) async def on_client_connected ( transport , client ): # Start recording await audiobuffer.start_recording() # Kickstart the conversation messages.append({ "role" : "system" , "content" : "Please introduce yourself to the user." }) await task.queue_frames([context_aggregator.user().get_context_frame()]) @transport.event_handler ( "on_client_disconnected" ) async def on_client_disconnected ( transport , client ): await task.cancel() @audiobuffer.event_handler ( "on_audio_data" ) async def on_audio_data ( buffer , audio , sample_rate , num_channels ): server_name = f "server_ { websocket_client.client.port } " await save_audio(server_name, audio, sample_rate, num_channels) # Run the pipeline runner = PipelineRunner( handle_sigint = False , force_gc = True ) await runner.run(task) ​ Key Technical Considerations ​ Audio Format and Sample Rate Twilio Media Streams uses 8kHz mono audio with 16-bit PCM encoding. Make sure your pipeline is configured correctly: Copy Ask AI task = PipelineTask( pipeline, params = PipelineParams( audio_in_sample_rate = 8000 , # Twilio's audio format audio_out_sample_rate = 8000 , allow_interruptions = True , ), ) ​ Serialization and Call Control The TwilioFrameSerializer handles the protocol specifics for communicating with Twilio’s Media Streams: Copy Ask AI serializer = TwilioFrameSerializer( stream_sid = stream_sid, call_sid = call_sid, account_sid = os.getenv( "TWILIO_ACCOUNT_SID" , "" ), auth_token = os.getenv( "TWILIO_AUTH_TOKEN" , "" ), ) When you provide the account_sid and auth_token to the TwilioFrameSerializer , it will automatically end the call via Twilio’s REST API when the pipeline ends. This ensures clean call termination when your bot finishes its conversation. ​ Voice Activity Detection The SileroVAD analyzer helps determine when a user has finished speaking: Copy Ask AI params = FastAPIWebsocketParams( audio_in_enabled = True , audio_out_enabled = True , add_wav_header = False , vad_analyzer = SileroVADAnalyzer(), serializer = serializer, ) ​ Configuring Twilio To set up your Twilio phone number to use your server: Purchase a phone number in your Twilio account Navigate to the Phone Numbers section and select your number Under “Voice & Fax”, set the webhook for “A Call Comes In” to your server’s URL (e.g., https://your-server.com/ ) Make sure the request type is set to HTTP POST Save your changes If you’re using ngrok for local development, your webhook URL will look like https://abc123.ngrok.io/ . Remember to update your TwiML template with the correct WebSocket URL as well. ​ Testing Your Implementation ​ Local Testing Without Phone Calls The example includes a test client that can simulate phone calls without actually using Twilio: Copy Ask AI python server.py - t # Start server in testing mode python client.py - u http: // localhost: 8765 - c 2 # Start 2 test clients The -t flag puts the server in testing mode, and the test client creates virtual clients that communicate with your server as if they were Twilio Media Streams. ​ Using the Phone To test with an actual phone call: Make sure your server is running and accessible via the internet Configure Twilio as described above Dial your Twilio phone number from any phone You should hear your AI agent respond! ​ Ending a Conversation There are two primary ways to end a conversation: Automatic termination : If you provided Twilio credentials to the TwilioFrameSerializer , the call will be ended automatically when your pipeline ends: Copy Ask AI # End the conversation programmatically await task.queue_frame(EndFrame()) Manual termination : You can also end the call explicitly through Twilio’s REST API. This is useful for implementing custom hang-up logic: Copy Ask AI from twilio.rest import Client client = Client(account_sid, auth_token) client.calls(call_sid).update( status = "completed" ) ​ Scaling Considerations For production deployments, consider: Multiple concurrent calls : Each bot instance should run in its own process to handle concurrent calls efficiently. The example server can spawn individual bot processes for each call: Copy Ask AI proc = subprocess.Popen( [ f "python3 -m bot_twilio --stream-sid { stream_sid } --call-sid { call_sid } " ], shell = True , bufsize = 1 , cwd = os.path.dirname(os.path.abspath( __file__ )) ) Error handling : Add robust error handling for network issues, service outages, etc. Logging and monitoring : Implement detailed logging to track call quality and agent performance. Security : Add authentication to your endpoints and use environment variables for all sensitive credentials. ​ Next Steps Try the complete example Clone the full working Twilio chatbot example from GitHub and run it yourself. Deploy to Pipecat Cloud Learn how to deploy your Twilio voice AI agent to production with Pipecat Cloud’s managed infrastructure. Dial-in: WebRTC (Twilio + Daily) Dialout: WebRTC (Daily) On this page Things you’ll need Architecture Overview Server Implementation FastAPI Server Setup TwiML Configuration Pipecat Bot Implementation Key Technical Considerations Audio Format and Sample Rate Serialization and Call Control Voice Activity Detection Configuring Twilio Testing Your Implementation Local Testing Without Phone Calls Using the Phone Ending a Conversation Scaling Considerations Next Steps Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/telephony_twilio-websockets_a702a6dc.txt b/telephony_twilio-websockets_a702a6dc.txt
new file mode 100644
index 0000000000000000000000000000000000000000..20c0a6f6f8b0dc985ae9f28ac40cffe23be59fbc
--- /dev/null
+++ b/telephony_twilio-websockets_a702a6dc.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/guides/telephony/twilio-websockets#pipecat-bot-implementation
+Title: Dial-in: Twilio (Media Streams) - Pipecat
+==================================================
+
+Dial-in: Twilio (Media Streams) - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Telephony Dial-in: Twilio (Media Streams) Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Guides Fundamentals Context Management Custom FrameProcessor Detecting Idle Users Ending a Pipeline Function Calling Muting User Input Recording Audio Recording Transcripts Features Gemini Multimodal Live Metrics Noise cancellation with Krisp OpenAI Audio Models and APIs Pipecat Flows Telephony Overview Dial-in: WebRTC (Daily) Dial-in: WebRTC (Twilio + Daily) Dial-in: Twilio (Media Streams) Dialout: WebRTC (Daily) Deploying your bot Overview Deployment pattern Example: Pipecat Cloud Example: Fly.io Example: Cerebrium Example: Modal This guide walks through creating a voice AI agent that users can reach by dialing a Twilio phone number. We’ll integrate Twilio Media Streams via WebSockets with a Pipecat pipeline to create a fully functional voice AI experience. ​ Things you’ll need An active Twilio account with at least one phone number A public-facing server or a tunneling service like ngrok API keys for speech-to-text, text-to-speech, and LLM services Complete example code on GitHub The full source code for this example is available in the Pipecat repository. ​ Architecture Overview When a user dials your Twilio phone number: Twilio calls your server’s endpoint, which returns TwiML that establishes a WebSocket connection Twilio sends real-time audio data over the WebSocket, and your server processes it using the Pipecat pipeline Your Pipecat agent processes the audio which is sent to the pipeline, which then outputs audio data back to Twilio Twilio plays this audio to the caller in real-time ​ Server Implementation The server needs to: Serve TwiML in response to Twilio’s HTTP requests Handle WebSocket connections Process audio with the Pipecat pipeline Let’s look at the key components: ​ FastAPI Server Setup Copy Ask AI from fastapi import FastAPI, WebSocket from starlette.responses import HTMLResponse app = FastAPI() @app.post ( "/" ) async def start_call (): return HTMLResponse( content = open ( "templates/streams.xml" ).read(), media_type = "application/xml" ) @app.websocket ( "/ws" ) async def websocket_endpoint ( websocket : WebSocket): await websocket.accept() # Read initial WebSocket messages start_data = ws.iter_text() await start_data. __anext__ () # Second message contains the call details call_data = json.loads( await start_data. __anext__ ()) # Extract both StreamSid and CallSid stream_sid = call_data[ "start" ][ "streamSid" ] call_sid = call_data[ "start" ][ "callSid" ] # Run your Pipecat bot await run_bot(websocket, stream_sid, call_sid, app.state.testing) This server has two main endpoints: POST / - Returns TwiML instructions to Twilio WebSocket /ws - Handles the WebSocket connection for real-time audio ​ TwiML Configuration The TwiML tells Twilio to establish a WebSocket connection with your server: Copy Ask AI <? xml version = "1.0" encoding = "UTF-8" ?> < Response > < Connect > < Stream url = "wss://<your server url>/ws" ></ Stream > </ Connect > < Pause length = "40" /> </ Response > Replace <your server url> with your server’s publicly accessible domain. The Pause element keeps the call alive for a maximum of 40 seconds. Adjust this value based on your expected conversation length. ​ Pipecat Bot Implementation The run_bot function creates and connects all the components in the Pipecat pipeline: Copy Ask AI async def run_bot ( websocket_client : WebSocket, stream_sid : str , call_sid : str , testing : bool ): # Initialize the Twilio serializer with stream and call SIDs serializer = TwilioFrameSerializer( stream_sid = stream_sid, call_sid = call_sid, account_sid = os.getenv( "TWILIO_ACCOUNT_SID" , "" ), auth_token = os.getenv( "TWILIO_AUTH_TOKEN" , "" ), ) # Create the WebSocket transport transport = FastAPIWebsocketTransport( websocket = websocket_client, params = FastAPIWebsocketParams( audio_in_enabled = True , audio_out_enabled = True , add_wav_header = False , vad_analyzer = SileroVADAnalyzer(), serializer = serializer, ), ) # Initialize AI services llm = OpenAILLMService( api_key = os.getenv( "OPENAI_API_KEY" )) stt = DeepgramSTTService( api_key = os.getenv( "DEEPGRAM_API_KEY" ), audio_passthrough = True ) tts = CartesiaTTSService( api_key = os.getenv( "CARTESIA_API_KEY" ), voice_id = "71a7ad14-091c-4e8e-a314-022ece01c121" , # British Reading Lady push_silence_after_stop = testing, ) # Create the initial conversation prompt messages = [ { "role" : "system" , "content" : "You are an elementary teacher in an audio call. Your output will be converted to audio so don't include special characters in your answers. Respond to what the student said in a short short sentence." , }, ] # Setup the context aggregator context = OpenAILLMContext(messages) context_aggregator = llm.create_context_aggregator(context) # Create an audio buffer processor to capture conversation audio audiobuffer = AudioBufferProcessor( user_continuous_stream = not testing) # Build the pipeline pipeline = Pipeline( [ transport.input(), # Websocket input from client stt, # Speech-To-Text context_aggregator.user(), llm, # LLM tts, # Text-To-Speech transport.output(), # Websocket output to client audiobuffer, # Used to buffer the audio in the pipeline context_aggregator.assistant(), ] ) # Create a pipeline task with appropriate parameters task = PipelineTask( pipeline, params = PipelineParams( audio_in_sample_rate = 8000 , # Twilio uses 8kHz audio audio_out_sample_rate = 8000 , allow_interruptions = True , ), ) # Setup event handlers @transport.event_handler ( "on_client_connected" ) async def on_client_connected ( transport , client ): # Start recording await audiobuffer.start_recording() # Kickstart the conversation messages.append({ "role" : "system" , "content" : "Please introduce yourself to the user." }) await task.queue_frames([context_aggregator.user().get_context_frame()]) @transport.event_handler ( "on_client_disconnected" ) async def on_client_disconnected ( transport , client ): await task.cancel() @audiobuffer.event_handler ( "on_audio_data" ) async def on_audio_data ( buffer , audio , sample_rate , num_channels ): server_name = f "server_ { websocket_client.client.port } " await save_audio(server_name, audio, sample_rate, num_channels) # Run the pipeline runner = PipelineRunner( handle_sigint = False , force_gc = True ) await runner.run(task) ​ Key Technical Considerations ​ Audio Format and Sample Rate Twilio Media Streams uses 8kHz mono audio with 16-bit PCM encoding. Make sure your pipeline is configured correctly: Copy Ask AI task = PipelineTask( pipeline, params = PipelineParams( audio_in_sample_rate = 8000 , # Twilio's audio format audio_out_sample_rate = 8000 , allow_interruptions = True , ), ) ​ Serialization and Call Control The TwilioFrameSerializer handles the protocol specifics for communicating with Twilio’s Media Streams: Copy Ask AI serializer = TwilioFrameSerializer( stream_sid = stream_sid, call_sid = call_sid, account_sid = os.getenv( "TWILIO_ACCOUNT_SID" , "" ), auth_token = os.getenv( "TWILIO_AUTH_TOKEN" , "" ), ) When you provide the account_sid and auth_token to the TwilioFrameSerializer , it will automatically end the call via Twilio’s REST API when the pipeline ends. This ensures clean call termination when your bot finishes its conversation. ​ Voice Activity Detection The SileroVAD analyzer helps determine when a user has finished speaking: Copy Ask AI params = FastAPIWebsocketParams( audio_in_enabled = True , audio_out_enabled = True , add_wav_header = False , vad_analyzer = SileroVADAnalyzer(), serializer = serializer, ) ​ Configuring Twilio To set up your Twilio phone number to use your server: Purchase a phone number in your Twilio account Navigate to the Phone Numbers section and select your number Under “Voice & Fax”, set the webhook for “A Call Comes In” to your server’s URL (e.g., https://your-server.com/ ) Make sure the request type is set to HTTP POST Save your changes If you’re using ngrok for local development, your webhook URL will look like https://abc123.ngrok.io/ . Remember to update your TwiML template with the correct WebSocket URL as well. ​ Testing Your Implementation ​ Local Testing Without Phone Calls The example includes a test client that can simulate phone calls without actually using Twilio: Copy Ask AI python server.py - t # Start server in testing mode python client.py - u http: // localhost: 8765 - c 2 # Start 2 test clients The -t flag puts the server in testing mode, and the test client creates virtual clients that communicate with your server as if they were Twilio Media Streams. ​ Using the Phone To test with an actual phone call: Make sure your server is running and accessible via the internet Configure Twilio as described above Dial your Twilio phone number from any phone You should hear your AI agent respond! ​ Ending a Conversation There are two primary ways to end a conversation: Automatic termination : If you provided Twilio credentials to the TwilioFrameSerializer , the call will be ended automatically when your pipeline ends: Copy Ask AI # End the conversation programmatically await task.queue_frame(EndFrame()) Manual termination : You can also end the call explicitly through Twilio’s REST API. This is useful for implementing custom hang-up logic: Copy Ask AI from twilio.rest import Client client = Client(account_sid, auth_token) client.calls(call_sid).update( status = "completed" ) ​ Scaling Considerations For production deployments, consider: Multiple concurrent calls : Each bot instance should run in its own process to handle concurrent calls efficiently. The example server can spawn individual bot processes for each call: Copy Ask AI proc = subprocess.Popen( [ f "python3 -m bot_twilio --stream-sid { stream_sid } --call-sid { call_sid } " ], shell = True , bufsize = 1 , cwd = os.path.dirname(os.path.abspath( __file__ )) ) Error handling : Add robust error handling for network issues, service outages, etc. Logging and monitoring : Implement detailed logging to track call quality and agent performance. Security : Add authentication to your endpoints and use environment variables for all sensitive credentials. ​ Next Steps Try the complete example Clone the full working Twilio chatbot example from GitHub and run it yourself. Deploy to Pipecat Cloud Learn how to deploy your Twilio voice AI agent to production with Pipecat Cloud’s managed infrastructure. Dial-in: WebRTC (Twilio + Daily) Dialout: WebRTC (Daily) On this page Things you’ll need Architecture Overview Server Implementation FastAPI Server Setup TwiML Configuration Pipecat Bot Implementation Key Technical Considerations Audio Format and Sample Rate Serialization and Call Control Voice Activity Detection Configuring Twilio Testing Your Implementation Local Testing Without Phone Calls Using the Phone Ending a Conversation Scaling Considerations Next Steps Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/telephony_twilio-websockets_d1507c8f.txt b/telephony_twilio-websockets_d1507c8f.txt
new file mode 100644
index 0000000000000000000000000000000000000000..1af5b31e14f63393baf38f7765212de9f96ec73a
--- /dev/null
+++ b/telephony_twilio-websockets_d1507c8f.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/guides/telephony/twilio-websockets#serialization-and-call-control
+Title: Dial-in: Twilio (Media Streams) - Pipecat
+==================================================
+
+Dial-in: Twilio (Media Streams) - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Telephony Dial-in: Twilio (Media Streams) Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Guides Fundamentals Context Management Custom FrameProcessor Detecting Idle Users Ending a Pipeline Function Calling Muting User Input Recording Audio Recording Transcripts Features Gemini Multimodal Live Metrics Noise cancellation with Krisp OpenAI Audio Models and APIs Pipecat Flows Telephony Overview Dial-in: WebRTC (Daily) Dial-in: WebRTC (Twilio + Daily) Dial-in: Twilio (Media Streams) Dialout: WebRTC (Daily) Deploying your bot Overview Deployment pattern Example: Pipecat Cloud Example: Fly.io Example: Cerebrium Example: Modal This guide walks through creating a voice AI agent that users can reach by dialing a Twilio phone number. We’ll integrate Twilio Media Streams via WebSockets with a Pipecat pipeline to create a fully functional voice AI experience. ​ Things you’ll need An active Twilio account with at least one phone number A public-facing server or a tunneling service like ngrok API keys for speech-to-text, text-to-speech, and LLM services Complete example code on GitHub The full source code for this example is available in the Pipecat repository. ​ Architecture Overview When a user dials your Twilio phone number: Twilio calls your server’s endpoint, which returns TwiML that establishes a WebSocket connection Twilio sends real-time audio data over the WebSocket, and your server processes it using the Pipecat pipeline Your Pipecat agent processes the audio which is sent to the pipeline, which then outputs audio data back to Twilio Twilio plays this audio to the caller in real-time ​ Server Implementation The server needs to: Serve TwiML in response to Twilio’s HTTP requests Handle WebSocket connections Process audio with the Pipecat pipeline Let’s look at the key components: ​ FastAPI Server Setup Copy Ask AI from fastapi import FastAPI, WebSocket from starlette.responses import HTMLResponse app = FastAPI() @app.post ( "/" ) async def start_call (): return HTMLResponse( content = open ( "templates/streams.xml" ).read(), media_type = "application/xml" ) @app.websocket ( "/ws" ) async def websocket_endpoint ( websocket : WebSocket): await websocket.accept() # Read initial WebSocket messages start_data = ws.iter_text() await start_data. __anext__ () # Second message contains the call details call_data = json.loads( await start_data. __anext__ ()) # Extract both StreamSid and CallSid stream_sid = call_data[ "start" ][ "streamSid" ] call_sid = call_data[ "start" ][ "callSid" ] # Run your Pipecat bot await run_bot(websocket, stream_sid, call_sid, app.state.testing) This server has two main endpoints: POST / - Returns TwiML instructions to Twilio WebSocket /ws - Handles the WebSocket connection for real-time audio ​ TwiML Configuration The TwiML tells Twilio to establish a WebSocket connection with your server: Copy Ask AI <? xml version = "1.0" encoding = "UTF-8" ?> < Response > < Connect > < Stream url = "wss://<your server url>/ws" ></ Stream > </ Connect > < Pause length = "40" /> </ Response > Replace <your server url> with your server’s publicly accessible domain. The Pause element keeps the call alive for a maximum of 40 seconds. Adjust this value based on your expected conversation length. ​ Pipecat Bot Implementation The run_bot function creates and connects all the components in the Pipecat pipeline: Copy Ask AI async def run_bot ( websocket_client : WebSocket, stream_sid : str , call_sid : str , testing : bool ): # Initialize the Twilio serializer with stream and call SIDs serializer = TwilioFrameSerializer( stream_sid = stream_sid, call_sid = call_sid, account_sid = os.getenv( "TWILIO_ACCOUNT_SID" , "" ), auth_token = os.getenv( "TWILIO_AUTH_TOKEN" , "" ), ) # Create the WebSocket transport transport = FastAPIWebsocketTransport( websocket = websocket_client, params = FastAPIWebsocketParams( audio_in_enabled = True , audio_out_enabled = True , add_wav_header = False , vad_analyzer = SileroVADAnalyzer(), serializer = serializer, ), ) # Initialize AI services llm = OpenAILLMService( api_key = os.getenv( "OPENAI_API_KEY" )) stt = DeepgramSTTService( api_key = os.getenv( "DEEPGRAM_API_KEY" ), audio_passthrough = True ) tts = CartesiaTTSService( api_key = os.getenv( "CARTESIA_API_KEY" ), voice_id = "71a7ad14-091c-4e8e-a314-022ece01c121" , # British Reading Lady push_silence_after_stop = testing, ) # Create the initial conversation prompt messages = [ { "role" : "system" , "content" : "You are an elementary teacher in an audio call. Your output will be converted to audio so don't include special characters in your answers. Respond to what the student said in a short short sentence." , }, ] # Setup the context aggregator context = OpenAILLMContext(messages) context_aggregator = llm.create_context_aggregator(context) # Create an audio buffer processor to capture conversation audio audiobuffer = AudioBufferProcessor( user_continuous_stream = not testing) # Build the pipeline pipeline = Pipeline( [ transport.input(), # Websocket input from client stt, # Speech-To-Text context_aggregator.user(), llm, # LLM tts, # Text-To-Speech transport.output(), # Websocket output to client audiobuffer, # Used to buffer the audio in the pipeline context_aggregator.assistant(), ] ) # Create a pipeline task with appropriate parameters task = PipelineTask( pipeline, params = PipelineParams( audio_in_sample_rate = 8000 , # Twilio uses 8kHz audio audio_out_sample_rate = 8000 , allow_interruptions = True , ), ) # Setup event handlers @transport.event_handler ( "on_client_connected" ) async def on_client_connected ( transport , client ): # Start recording await audiobuffer.start_recording() # Kickstart the conversation messages.append({ "role" : "system" , "content" : "Please introduce yourself to the user." }) await task.queue_frames([context_aggregator.user().get_context_frame()]) @transport.event_handler ( "on_client_disconnected" ) async def on_client_disconnected ( transport , client ): await task.cancel() @audiobuffer.event_handler ( "on_audio_data" ) async def on_audio_data ( buffer , audio , sample_rate , num_channels ): server_name = f "server_ { websocket_client.client.port } " await save_audio(server_name, audio, sample_rate, num_channels) # Run the pipeline runner = PipelineRunner( handle_sigint = False , force_gc = True ) await runner.run(task) ​ Key Technical Considerations ​ Audio Format and Sample Rate Twilio Media Streams uses 8kHz mono audio with 16-bit PCM encoding. Make sure your pipeline is configured correctly: Copy Ask AI task = PipelineTask( pipeline, params = PipelineParams( audio_in_sample_rate = 8000 , # Twilio's audio format audio_out_sample_rate = 8000 , allow_interruptions = True , ), ) ​ Serialization and Call Control The TwilioFrameSerializer handles the protocol specifics for communicating with Twilio’s Media Streams: Copy Ask AI serializer = TwilioFrameSerializer( stream_sid = stream_sid, call_sid = call_sid, account_sid = os.getenv( "TWILIO_ACCOUNT_SID" , "" ), auth_token = os.getenv( "TWILIO_AUTH_TOKEN" , "" ), ) When you provide the account_sid and auth_token to the TwilioFrameSerializer , it will automatically end the call via Twilio’s REST API when the pipeline ends. This ensures clean call termination when your bot finishes its conversation. ​ Voice Activity Detection The SileroVAD analyzer helps determine when a user has finished speaking: Copy Ask AI params = FastAPIWebsocketParams( audio_in_enabled = True , audio_out_enabled = True , add_wav_header = False , vad_analyzer = SileroVADAnalyzer(), serializer = serializer, ) ​ Configuring Twilio To set up your Twilio phone number to use your server: Purchase a phone number in your Twilio account Navigate to the Phone Numbers section and select your number Under “Voice & Fax”, set the webhook for “A Call Comes In” to your server’s URL (e.g., https://your-server.com/ ) Make sure the request type is set to HTTP POST Save your changes If you’re using ngrok for local development, your webhook URL will look like https://abc123.ngrok.io/ . Remember to update your TwiML template with the correct WebSocket URL as well. ​ Testing Your Implementation ​ Local Testing Without Phone Calls The example includes a test client that can simulate phone calls without actually using Twilio: Copy Ask AI python server.py - t # Start server in testing mode python client.py - u http: // localhost: 8765 - c 2 # Start 2 test clients The -t flag puts the server in testing mode, and the test client creates virtual clients that communicate with your server as if they were Twilio Media Streams. ​ Using the Phone To test with an actual phone call: Make sure your server is running and accessible via the internet Configure Twilio as described above Dial your Twilio phone number from any phone You should hear your AI agent respond! ​ Ending a Conversation There are two primary ways to end a conversation: Automatic termination : If you provided Twilio credentials to the TwilioFrameSerializer , the call will be ended automatically when your pipeline ends: Copy Ask AI # End the conversation programmatically await task.queue_frame(EndFrame()) Manual termination : You can also end the call explicitly through Twilio’s REST API. This is useful for implementing custom hang-up logic: Copy Ask AI from twilio.rest import Client client = Client(account_sid, auth_token) client.calls(call_sid).update( status = "completed" ) ​ Scaling Considerations For production deployments, consider: Multiple concurrent calls : Each bot instance should run in its own process to handle concurrent calls efficiently. The example server can spawn individual bot processes for each call: Copy Ask AI proc = subprocess.Popen( [ f "python3 -m bot_twilio --stream-sid { stream_sid } --call-sid { call_sid } " ], shell = True , bufsize = 1 , cwd = os.path.dirname(os.path.abspath( __file__ )) ) Error handling : Add robust error handling for network issues, service outages, etc. Logging and monitoring : Implement detailed logging to track call quality and agent performance. Security : Add authentication to your endpoints and use environment variables for all sensitive credentials. ​ Next Steps Try the complete example Clone the full working Twilio chatbot example from GitHub and run it yourself. Deploy to Pipecat Cloud Learn how to deploy your Twilio voice AI agent to production with Pipecat Cloud’s managed infrastructure. Dial-in: WebRTC (Twilio + Daily) Dialout: WebRTC (Daily) On this page Things you’ll need Architecture Overview Server Implementation FastAPI Server Setup TwiML Configuration Pipecat Bot Implementation Key Technical Considerations Audio Format and Sample Rate Serialization and Call Control Voice Activity Detection Configuring Twilio Testing Your Implementation Local Testing Without Phone Calls Using the Phone Ending a Conversation Scaling Considerations Next Steps Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/telephony_twilio-websockets_ef9465da.txt b/telephony_twilio-websockets_ef9465da.txt
new file mode 100644
index 0000000000000000000000000000000000000000..e4b8870ee569d72949c3818c605ba39121de6132
--- /dev/null
+++ b/telephony_twilio-websockets_ef9465da.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/guides/telephony/twilio-websockets#local-testing-without-phone-calls
+Title: Dial-in: Twilio (Media Streams) - Pipecat
+==================================================
+
+Dial-in: Twilio (Media Streams) - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Telephony Dial-in: Twilio (Media Streams) Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Guides Fundamentals Context Management Custom FrameProcessor Detecting Idle Users Ending a Pipeline Function Calling Muting User Input Recording Audio Recording Transcripts Features Gemini Multimodal Live Metrics Noise cancellation with Krisp OpenAI Audio Models and APIs Pipecat Flows Telephony Overview Dial-in: WebRTC (Daily) Dial-in: WebRTC (Twilio + Daily) Dial-in: Twilio (Media Streams) Dialout: WebRTC (Daily) Deploying your bot Overview Deployment pattern Example: Pipecat Cloud Example: Fly.io Example: Cerebrium Example: Modal This guide walks through creating a voice AI agent that users can reach by dialing a Twilio phone number. We’ll integrate Twilio Media Streams via WebSockets with a Pipecat pipeline to create a fully functional voice AI experience. ​ Things you’ll need An active Twilio account with at least one phone number A public-facing server or a tunneling service like ngrok API keys for speech-to-text, text-to-speech, and LLM services Complete example code on GitHub The full source code for this example is available in the Pipecat repository. ​ Architecture Overview When a user dials your Twilio phone number: Twilio calls your server’s endpoint, which returns TwiML that establishes a WebSocket connection Twilio sends real-time audio data over the WebSocket, and your server processes it using the Pipecat pipeline Your Pipecat agent processes the audio which is sent to the pipeline, which then outputs audio data back to Twilio Twilio plays this audio to the caller in real-time ​ Server Implementation The server needs to: Serve TwiML in response to Twilio’s HTTP requests Handle WebSocket connections Process audio with the Pipecat pipeline Let’s look at the key components: ​ FastAPI Server Setup Copy Ask AI from fastapi import FastAPI, WebSocket from starlette.responses import HTMLResponse app = FastAPI() @app.post ( "/" ) async def start_call (): return HTMLResponse( content = open ( "templates/streams.xml" ).read(), media_type = "application/xml" ) @app.websocket ( "/ws" ) async def websocket_endpoint ( websocket : WebSocket): await websocket.accept() # Read initial WebSocket messages start_data = ws.iter_text() await start_data. __anext__ () # Second message contains the call details call_data = json.loads( await start_data. __anext__ ()) # Extract both StreamSid and CallSid stream_sid = call_data[ "start" ][ "streamSid" ] call_sid = call_data[ "start" ][ "callSid" ] # Run your Pipecat bot await run_bot(websocket, stream_sid, call_sid, app.state.testing) This server has two main endpoints: POST / - Returns TwiML instructions to Twilio WebSocket /ws - Handles the WebSocket connection for real-time audio ​ TwiML Configuration The TwiML tells Twilio to establish a WebSocket connection with your server: Copy Ask AI <? xml version = "1.0" encoding = "UTF-8" ?> < Response > < Connect > < Stream url = "wss://<your server url>/ws" ></ Stream > </ Connect > < Pause length = "40" /> </ Response > Replace <your server url> with your server’s publicly accessible domain. The Pause element keeps the call alive for a maximum of 40 seconds. Adjust this value based on your expected conversation length. ​ Pipecat Bot Implementation The run_bot function creates and connects all the components in the Pipecat pipeline: Copy Ask AI async def run_bot ( websocket_client : WebSocket, stream_sid : str , call_sid : str , testing : bool ): # Initialize the Twilio serializer with stream and call SIDs serializer = TwilioFrameSerializer( stream_sid = stream_sid, call_sid = call_sid, account_sid = os.getenv( "TWILIO_ACCOUNT_SID" , "" ), auth_token = os.getenv( "TWILIO_AUTH_TOKEN" , "" ), ) # Create the WebSocket transport transport = FastAPIWebsocketTransport( websocket = websocket_client, params = FastAPIWebsocketParams( audio_in_enabled = True , audio_out_enabled = True , add_wav_header = False , vad_analyzer = SileroVADAnalyzer(), serializer = serializer, ), ) # Initialize AI services llm = OpenAILLMService( api_key = os.getenv( "OPENAI_API_KEY" )) stt = DeepgramSTTService( api_key = os.getenv( "DEEPGRAM_API_KEY" ), audio_passthrough = True ) tts = CartesiaTTSService( api_key = os.getenv( "CARTESIA_API_KEY" ), voice_id = "71a7ad14-091c-4e8e-a314-022ece01c121" , # British Reading Lady push_silence_after_stop = testing, ) # Create the initial conversation prompt messages = [ { "role" : "system" , "content" : "You are an elementary teacher in an audio call. Your output will be converted to audio so don't include special characters in your answers. Respond to what the student said in a short short sentence." , }, ] # Setup the context aggregator context = OpenAILLMContext(messages) context_aggregator = llm.create_context_aggregator(context) # Create an audio buffer processor to capture conversation audio audiobuffer = AudioBufferProcessor( user_continuous_stream = not testing) # Build the pipeline pipeline = Pipeline( [ transport.input(), # Websocket input from client stt, # Speech-To-Text context_aggregator.user(), llm, # LLM tts, # Text-To-Speech transport.output(), # Websocket output to client audiobuffer, # Used to buffer the audio in the pipeline context_aggregator.assistant(), ] ) # Create a pipeline task with appropriate parameters task = PipelineTask( pipeline, params = PipelineParams( audio_in_sample_rate = 8000 , # Twilio uses 8kHz audio audio_out_sample_rate = 8000 , allow_interruptions = True , ), ) # Setup event handlers @transport.event_handler ( "on_client_connected" ) async def on_client_connected ( transport , client ): # Start recording await audiobuffer.start_recording() # Kickstart the conversation messages.append({ "role" : "system" , "content" : "Please introduce yourself to the user." }) await task.queue_frames([context_aggregator.user().get_context_frame()]) @transport.event_handler ( "on_client_disconnected" ) async def on_client_disconnected ( transport , client ): await task.cancel() @audiobuffer.event_handler ( "on_audio_data" ) async def on_audio_data ( buffer , audio , sample_rate , num_channels ): server_name = f "server_ { websocket_client.client.port } " await save_audio(server_name, audio, sample_rate, num_channels) # Run the pipeline runner = PipelineRunner( handle_sigint = False , force_gc = True ) await runner.run(task) ​ Key Technical Considerations ​ Audio Format and Sample Rate Twilio Media Streams uses 8kHz mono audio with 16-bit PCM encoding. Make sure your pipeline is configured correctly: Copy Ask AI task = PipelineTask( pipeline, params = PipelineParams( audio_in_sample_rate = 8000 , # Twilio's audio format audio_out_sample_rate = 8000 , allow_interruptions = True , ), ) ​ Serialization and Call Control The TwilioFrameSerializer handles the protocol specifics for communicating with Twilio’s Media Streams: Copy Ask AI serializer = TwilioFrameSerializer( stream_sid = stream_sid, call_sid = call_sid, account_sid = os.getenv( "TWILIO_ACCOUNT_SID" , "" ), auth_token = os.getenv( "TWILIO_AUTH_TOKEN" , "" ), ) When you provide the account_sid and auth_token to the TwilioFrameSerializer , it will automatically end the call via Twilio’s REST API when the pipeline ends. This ensures clean call termination when your bot finishes its conversation. ​ Voice Activity Detection The SileroVAD analyzer helps determine when a user has finished speaking: Copy Ask AI params = FastAPIWebsocketParams( audio_in_enabled = True , audio_out_enabled = True , add_wav_header = False , vad_analyzer = SileroVADAnalyzer(), serializer = serializer, ) ​ Configuring Twilio To set up your Twilio phone number to use your server: Purchase a phone number in your Twilio account Navigate to the Phone Numbers section and select your number Under “Voice & Fax”, set the webhook for “A Call Comes In” to your server’s URL (e.g., https://your-server.com/ ) Make sure the request type is set to HTTP POST Save your changes If you’re using ngrok for local development, your webhook URL will look like https://abc123.ngrok.io/ . Remember to update your TwiML template with the correct WebSocket URL as well. ​ Testing Your Implementation ​ Local Testing Without Phone Calls The example includes a test client that can simulate phone calls without actually using Twilio: Copy Ask AI python server.py - t # Start server in testing mode python client.py - u http: // localhost: 8765 - c 2 # Start 2 test clients The -t flag puts the server in testing mode, and the test client creates virtual clients that communicate with your server as if they were Twilio Media Streams. ​ Using the Phone To test with an actual phone call: Make sure your server is running and accessible via the internet Configure Twilio as described above Dial your Twilio phone number from any phone You should hear your AI agent respond! ​ Ending a Conversation There are two primary ways to end a conversation: Automatic termination : If you provided Twilio credentials to the TwilioFrameSerializer , the call will be ended automatically when your pipeline ends: Copy Ask AI # End the conversation programmatically await task.queue_frame(EndFrame()) Manual termination : You can also end the call explicitly through Twilio’s REST API. This is useful for implementing custom hang-up logic: Copy Ask AI from twilio.rest import Client client = Client(account_sid, auth_token) client.calls(call_sid).update( status = "completed" ) ​ Scaling Considerations For production deployments, consider: Multiple concurrent calls : Each bot instance should run in its own process to handle concurrent calls efficiently. The example server can spawn individual bot processes for each call: Copy Ask AI proc = subprocess.Popen( [ f "python3 -m bot_twilio --stream-sid { stream_sid } --call-sid { call_sid } " ], shell = True , bufsize = 1 , cwd = os.path.dirname(os.path.abspath( __file__ )) ) Error handling : Add robust error handling for network issues, service outages, etc. Logging and monitoring : Implement detailed logging to track call quality and agent performance. Security : Add authentication to your endpoints and use environment variables for all sensitive credentials. ​ Next Steps Try the complete example Clone the full working Twilio chatbot example from GitHub and run it yourself. Deploy to Pipecat Cloud Learn how to deploy your Twilio voice AI agent to production with Pipecat Cloud’s managed infrastructure. Dial-in: WebRTC (Twilio + Daily) Dialout: WebRTC (Daily) On this page Things you’ll need Architecture Overview Server Implementation FastAPI Server Setup TwiML Configuration Pipecat Bot Implementation Key Technical Considerations Audio Format and Sample Rate Serialization and Call Control Voice Activity Detection Configuring Twilio Testing Your Implementation Local Testing Without Phone Calls Using the Phone Ending a Conversation Scaling Considerations Next Steps Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/text_markdown-text-filter_09471d40.txt b/text_markdown-text-filter_09471d40.txt
new file mode 100644
index 0000000000000000000000000000000000000000..c1adf4acaebb66f675a0364af74c6bcfda7b642a
--- /dev/null
+++ b/text_markdown-text-filter_09471d40.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/text/markdown-text-filter#param-params
+Title: MarkdownTextFilter - Pipecat
+==================================================
+
+MarkdownTextFilter - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text Aggregators and Filters MarkdownTextFilter Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters MarkdownTextFilter PatternPairAggregator User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview MarkdownTextFilter transforms Markdown-formatted text into plain text that’s suitable for text-to-speech (TTS) systems. It intelligently removes formatting elements while preserving the content structure, including proper spacing and list formatting. This filter is especially valuable for LLM-generated content, which often includes Markdown formatting that would sound unnatural if read aloud by a TTS system. ​ Constructor Copy Ask AI filter = MarkdownTextFilter( params = InputParams()) ​ params InputParams Configuration parameters for the filter ​ Input Parameters Configure the filter behavior with these options: ​ enable_text_filter bool default: "True" Whether the filter is active (when False, text passes through unchanged) ​ filter_code bool default: "False" Whether to remove code blocks from the output ​ filter_tables bool default: "False" Whether to remove Markdown tables from the output ​ Features The filter handles these Markdown elements: Basic Formatting : Removes *italic* , **bold** , and other formatting markers Code : Removes inline code ticks and optionally removes code blocks Lists : Preserves numbered lists while removing Markdown formatting Tables : Optionally removes Markdown tables Whitespace : Carefully preserves meaningful whitespace for natural speech HTML : Removes HTML tags and converts entities to their plain text equivalents ​ Usage Examples ​ Basic Usage with TTS Service Copy Ask AI from pipecat.utils.text.markdown_text_filter import MarkdownTextFilter from pipecat.services.cartesia.tts import CartesiaTTSService # Create the filter md_filter = MarkdownTextFilter() # Use with TTS service tts = CartesiaTTSService( api_key = os.getenv( "CARTESIA_API_KEY" ), voice_id = "voice_id_here" , text_filter = md_filter ) ​ Custom Configuration Copy Ask AI # Create filter that removes code blocks and tables md_filter = MarkdownTextFilter( params = MarkdownTextFilter.InputParams( filter_code = True , filter_tables = True ) ) ​ What Gets Removed Markdown Feature Example Result Bold **important** important Italic *emphasized* emphasized Headers ## Section Section Code (inline) `code` code Code blocks (when enabled) ```python\ncode\n``` Tables (when enabled) |A|B|\n|--|--| HTML tags <em>text</em> text Repeated characters !!!!!!! ! ​ Notes Preserves sentence structure and readability Maintains whitespace that affects speech prosody Handles streaming text with partial Markdown elements Efficiently converts HTML entities to plain text characters Smart handling of code blocks and tables with state tracking Integrates directly with TTS services in the Pipecat pipeline DTMFAggregator PatternPairAggregator On this page Overview Constructor Input Parameters Features Usage Examples Basic Usage with TTS Service Custom Configuration What Gets Removed Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/text_markdown-text-filter_21b59db0.txt b/text_markdown-text-filter_21b59db0.txt
new file mode 100644
index 0000000000000000000000000000000000000000..2a47fec1ad37c5b49b7a31dd5126e1bc22fde0a5
--- /dev/null
+++ b/text_markdown-text-filter_21b59db0.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/text/markdown-text-filter#param-filter-code
+Title: MarkdownTextFilter - Pipecat
+==================================================
+
+MarkdownTextFilter - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text Aggregators and Filters MarkdownTextFilter Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters MarkdownTextFilter PatternPairAggregator User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview MarkdownTextFilter transforms Markdown-formatted text into plain text that’s suitable for text-to-speech (TTS) systems. It intelligently removes formatting elements while preserving the content structure, including proper spacing and list formatting. This filter is especially valuable for LLM-generated content, which often includes Markdown formatting that would sound unnatural if read aloud by a TTS system. ​ Constructor Copy Ask AI filter = MarkdownTextFilter( params = InputParams()) ​ params InputParams Configuration parameters for the filter ​ Input Parameters Configure the filter behavior with these options: ​ enable_text_filter bool default: "True" Whether the filter is active (when False, text passes through unchanged) ​ filter_code bool default: "False" Whether to remove code blocks from the output ​ filter_tables bool default: "False" Whether to remove Markdown tables from the output ​ Features The filter handles these Markdown elements: Basic Formatting : Removes *italic* , **bold** , and other formatting markers Code : Removes inline code ticks and optionally removes code blocks Lists : Preserves numbered lists while removing Markdown formatting Tables : Optionally removes Markdown tables Whitespace : Carefully preserves meaningful whitespace for natural speech HTML : Removes HTML tags and converts entities to their plain text equivalents ​ Usage Examples ​ Basic Usage with TTS Service Copy Ask AI from pipecat.utils.text.markdown_text_filter import MarkdownTextFilter from pipecat.services.cartesia.tts import CartesiaTTSService # Create the filter md_filter = MarkdownTextFilter() # Use with TTS service tts = CartesiaTTSService( api_key = os.getenv( "CARTESIA_API_KEY" ), voice_id = "voice_id_here" , text_filter = md_filter ) ​ Custom Configuration Copy Ask AI # Create filter that removes code blocks and tables md_filter = MarkdownTextFilter( params = MarkdownTextFilter.InputParams( filter_code = True , filter_tables = True ) ) ​ What Gets Removed Markdown Feature Example Result Bold **important** important Italic *emphasized* emphasized Headers ## Section Section Code (inline) `code` code Code blocks (when enabled) ```python\ncode\n``` Tables (when enabled) |A|B|\n|--|--| HTML tags <em>text</em> text Repeated characters !!!!!!! ! ​ Notes Preserves sentence structure and readability Maintains whitespace that affects speech prosody Handles streaming text with partial Markdown elements Efficiently converts HTML entities to plain text characters Smart handling of code blocks and tables with state tracking Integrates directly with TTS services in the Pipecat pipeline DTMFAggregator PatternPairAggregator On this page Overview Constructor Input Parameters Features Usage Examples Basic Usage with TTS Service Custom Configuration What Gets Removed Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/text_markdown-text-filter_324c98ce.txt b/text_markdown-text-filter_324c98ce.txt
new file mode 100644
index 0000000000000000000000000000000000000000..a1a7e288a41959e504f214d2b62df35a0f849a26
--- /dev/null
+++ b/text_markdown-text-filter_324c98ce.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/text/markdown-text-filter#input-parameters
+Title: MarkdownTextFilter - Pipecat
+==================================================
+
+MarkdownTextFilter - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text Aggregators and Filters MarkdownTextFilter Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters MarkdownTextFilter PatternPairAggregator User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview MarkdownTextFilter transforms Markdown-formatted text into plain text that’s suitable for text-to-speech (TTS) systems. It intelligently removes formatting elements while preserving the content structure, including proper spacing and list formatting. This filter is especially valuable for LLM-generated content, which often includes Markdown formatting that would sound unnatural if read aloud by a TTS system. ​ Constructor Copy Ask AI filter = MarkdownTextFilter( params = InputParams()) ​ params InputParams Configuration parameters for the filter ​ Input Parameters Configure the filter behavior with these options: ​ enable_text_filter bool default: "True" Whether the filter is active (when False, text passes through unchanged) ​ filter_code bool default: "False" Whether to remove code blocks from the output ​ filter_tables bool default: "False" Whether to remove Markdown tables from the output ​ Features The filter handles these Markdown elements: Basic Formatting : Removes *italic* , **bold** , and other formatting markers Code : Removes inline code ticks and optionally removes code blocks Lists : Preserves numbered lists while removing Markdown formatting Tables : Optionally removes Markdown tables Whitespace : Carefully preserves meaningful whitespace for natural speech HTML : Removes HTML tags and converts entities to their plain text equivalents ​ Usage Examples ​ Basic Usage with TTS Service Copy Ask AI from pipecat.utils.text.markdown_text_filter import MarkdownTextFilter from pipecat.services.cartesia.tts import CartesiaTTSService # Create the filter md_filter = MarkdownTextFilter() # Use with TTS service tts = CartesiaTTSService( api_key = os.getenv( "CARTESIA_API_KEY" ), voice_id = "voice_id_here" , text_filter = md_filter ) ​ Custom Configuration Copy Ask AI # Create filter that removes code blocks and tables md_filter = MarkdownTextFilter( params = MarkdownTextFilter.InputParams( filter_code = True , filter_tables = True ) ) ​ What Gets Removed Markdown Feature Example Result Bold **important** important Italic *emphasized* emphasized Headers ## Section Section Code (inline) `code` code Code blocks (when enabled) ```python\ncode\n``` Tables (when enabled) |A|B|\n|--|--| HTML tags <em>text</em> text Repeated characters !!!!!!! ! ​ Notes Preserves sentence structure and readability Maintains whitespace that affects speech prosody Handles streaming text with partial Markdown elements Efficiently converts HTML entities to plain text characters Smart handling of code blocks and tables with state tracking Integrates directly with TTS services in the Pipecat pipeline DTMFAggregator PatternPairAggregator On this page Overview Constructor Input Parameters Features Usage Examples Basic Usage with TTS Service Custom Configuration What Gets Removed Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/text_markdown-text-filter_36c0f2c5.txt b/text_markdown-text-filter_36c0f2c5.txt
new file mode 100644
index 0000000000000000000000000000000000000000..6d4fe2c8251f875d8347f0119b2f10a0fc3f203b
--- /dev/null
+++ b/text_markdown-text-filter_36c0f2c5.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/text/markdown-text-filter#param-enable-text-filter
+Title: MarkdownTextFilter - Pipecat
+==================================================
+
+MarkdownTextFilter - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text Aggregators and Filters MarkdownTextFilter Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters MarkdownTextFilter PatternPairAggregator User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview MarkdownTextFilter transforms Markdown-formatted text into plain text that’s suitable for text-to-speech (TTS) systems. It intelligently removes formatting elements while preserving the content structure, including proper spacing and list formatting. This filter is especially valuable for LLM-generated content, which often includes Markdown formatting that would sound unnatural if read aloud by a TTS system. ​ Constructor Copy Ask AI filter = MarkdownTextFilter( params = InputParams()) ​ params InputParams Configuration parameters for the filter ​ Input Parameters Configure the filter behavior with these options: ​ enable_text_filter bool default: "True" Whether the filter is active (when False, text passes through unchanged) ​ filter_code bool default: "False" Whether to remove code blocks from the output ​ filter_tables bool default: "False" Whether to remove Markdown tables from the output ​ Features The filter handles these Markdown elements: Basic Formatting : Removes *italic* , **bold** , and other formatting markers Code : Removes inline code ticks and optionally removes code blocks Lists : Preserves numbered lists while removing Markdown formatting Tables : Optionally removes Markdown tables Whitespace : Carefully preserves meaningful whitespace for natural speech HTML : Removes HTML tags and converts entities to their plain text equivalents ​ Usage Examples ​ Basic Usage with TTS Service Copy Ask AI from pipecat.utils.text.markdown_text_filter import MarkdownTextFilter from pipecat.services.cartesia.tts import CartesiaTTSService # Create the filter md_filter = MarkdownTextFilter() # Use with TTS service tts = CartesiaTTSService( api_key = os.getenv( "CARTESIA_API_KEY" ), voice_id = "voice_id_here" , text_filter = md_filter ) ​ Custom Configuration Copy Ask AI # Create filter that removes code blocks and tables md_filter = MarkdownTextFilter( params = MarkdownTextFilter.InputParams( filter_code = True , filter_tables = True ) ) ​ What Gets Removed Markdown Feature Example Result Bold **important** important Italic *emphasized* emphasized Headers ## Section Section Code (inline) `code` code Code blocks (when enabled) ```python\ncode\n``` Tables (when enabled) |A|B|\n|--|--| HTML tags <em>text</em> text Repeated characters !!!!!!! ! ​ Notes Preserves sentence structure and readability Maintains whitespace that affects speech prosody Handles streaming text with partial Markdown elements Efficiently converts HTML entities to plain text characters Smart handling of code blocks and tables with state tracking Integrates directly with TTS services in the Pipecat pipeline DTMFAggregator PatternPairAggregator On this page Overview Constructor Input Parameters Features Usage Examples Basic Usage with TTS Service Custom Configuration What Gets Removed Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/text_markdown-text-filter_827e9113.txt b/text_markdown-text-filter_827e9113.txt
new file mode 100644
index 0000000000000000000000000000000000000000..c9e0aa47ee3b1c2a5df6748fab60ac275e804256
--- /dev/null
+++ b/text_markdown-text-filter_827e9113.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/text/markdown-text-filter#overview
+Title: MarkdownTextFilter - Pipecat
+==================================================
+
+MarkdownTextFilter - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text Aggregators and Filters MarkdownTextFilter Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters MarkdownTextFilter PatternPairAggregator User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview MarkdownTextFilter transforms Markdown-formatted text into plain text that’s suitable for text-to-speech (TTS) systems. It intelligently removes formatting elements while preserving the content structure, including proper spacing and list formatting. This filter is especially valuable for LLM-generated content, which often includes Markdown formatting that would sound unnatural if read aloud by a TTS system. ​ Constructor Copy Ask AI filter = MarkdownTextFilter( params = InputParams()) ​ params InputParams Configuration parameters for the filter ​ Input Parameters Configure the filter behavior with these options: ​ enable_text_filter bool default: "True" Whether the filter is active (when False, text passes through unchanged) ​ filter_code bool default: "False" Whether to remove code blocks from the output ​ filter_tables bool default: "False" Whether to remove Markdown tables from the output ​ Features The filter handles these Markdown elements: Basic Formatting : Removes *italic* , **bold** , and other formatting markers Code : Removes inline code ticks and optionally removes code blocks Lists : Preserves numbered lists while removing Markdown formatting Tables : Optionally removes Markdown tables Whitespace : Carefully preserves meaningful whitespace for natural speech HTML : Removes HTML tags and converts entities to their plain text equivalents ​ Usage Examples ​ Basic Usage with TTS Service Copy Ask AI from pipecat.utils.text.markdown_text_filter import MarkdownTextFilter from pipecat.services.cartesia.tts import CartesiaTTSService # Create the filter md_filter = MarkdownTextFilter() # Use with TTS service tts = CartesiaTTSService( api_key = os.getenv( "CARTESIA_API_KEY" ), voice_id = "voice_id_here" , text_filter = md_filter ) ​ Custom Configuration Copy Ask AI # Create filter that removes code blocks and tables md_filter = MarkdownTextFilter( params = MarkdownTextFilter.InputParams( filter_code = True , filter_tables = True ) ) ​ What Gets Removed Markdown Feature Example Result Bold **important** important Italic *emphasized* emphasized Headers ## Section Section Code (inline) `code` code Code blocks (when enabled) ```python\ncode\n``` Tables (when enabled) |A|B|\n|--|--| HTML tags <em>text</em> text Repeated characters !!!!!!! ! ​ Notes Preserves sentence structure and readability Maintains whitespace that affects speech prosody Handles streaming text with partial Markdown elements Efficiently converts HTML entities to plain text characters Smart handling of code blocks and tables with state tracking Integrates directly with TTS services in the Pipecat pipeline DTMFAggregator PatternPairAggregator On this page Overview Constructor Input Parameters Features Usage Examples Basic Usage with TTS Service Custom Configuration What Gets Removed Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/text_pattern-pair-aggregator_373a1629.txt b/text_pattern-pair-aggregator_373a1629.txt
new file mode 100644
index 0000000000000000000000000000000000000000..d6052c8aafb173939724d7c4ae876540eda04bdc
--- /dev/null
+++ b/text_pattern-pair-aggregator_373a1629.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/text/pattern-pair-aggregator#param-pattern-id-2
+Title: PatternPairAggregator - Pipecat
+==================================================
+
+PatternPairAggregator - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text Aggregators and Filters PatternPairAggregator Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters MarkdownTextFilter PatternPairAggregator User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview PatternPairAggregator is a specialized text aggregator that buffers streaming text until it can identify complete pattern pairs (like XML tags, markdown formatting, or custom delimiters). It processes the content between these patterns using registered handlers and returns text at sentence boundaries (therefore allowing normal TTS processing to occur). This aggregator is particularly useful for applications like voice switching, structured content processing, and extracting metadata from LLM outputs, ensuring that patterns spanning multiple text chunks are correctly identified. Want to see it in action? Check out the voice switching demo . ​ Constructor Copy Ask AI aggregator = PatternPairAggregator() No parameters are required for initialization. The aggregator starts with an empty buffer and no registered patterns. ​ Methods ​ add_pattern_pair Copy Ask AI aggregator.add_pattern_pair(pattern_id, start_pattern, end_pattern, remove_match = True ) Registers a new pattern pair to detect in the text. ​ pattern_id str required Unique identifier for this pattern pair ​ start_pattern str required Pattern that marks the beginning of content ​ end_pattern str required Pattern that marks the end of content ​ remove_match bool default: "True" Whether to remove the matched patterns from the output text ​ Returns Self for method chaining ​ on_pattern_match Copy Ask AI aggregator.on_pattern_match(pattern_id, handler) Registers a handler function to be called when a specific pattern pair is matched. ​ pattern_id str required ID of the pattern pair to match ​ handler Callable[[PatternMatch], None] required Function to call when the pattern is matched. The function should accept a PatternMatch object. ​ Returns Self for method chaining ​ Pattern Match Object When a pattern is matched, the handler function receives a PatternMatch object with these attributes: ​ pattern_id str The identifier of the matched pattern pair ​ full_match str The complete text including start and end patterns ​ content str The text content between the start and end patterns ​ Usage Examples ​ Voice Switching in TTS Copy Ask AI # Define voice IDs VOICE_IDS = { "narrator" : "c45bc5ec-dc68-4feb-8829-6e6b2748095d" , "female" : "71a7ad14-091c-4e8e-a314-022ece01c121" , "male" : "7cf0e2b1-8daf-4fe4-89ad-f6039398f359" , } # Create pattern aggregator pattern_aggregator = PatternPairAggregator() # Add pattern for voice tags pattern_aggregator.add_pattern_pair( pattern_id = "voice_tag" , start_pattern = "<voice>" , end_pattern = "</voice>" , remove_match = True ) # Register handler for voice switching def on_voice_tag ( match : PatternMatch): voice_name = match.content.strip().lower() if voice_name in VOICE_IDS : voice_id = VOICE_IDS [voice_name] tts.set_voice(voice_id) logger.info( f "Switched to { voice_name } voice" ) pattern_aggregator.on_pattern_match( "voice_tag" , on_voice_tag) # Set the aggregator on a TTS service tts = CartesiaTTSService( api_key = os.getenv( "CARTESIA_API_KEY" ), voice_id = VOICE_IDS [ "narrator" ], text_aggregator = pattern_aggregator ) ​ Extracting Structured Data from LLM Outputs Copy Ask AI # Create pattern aggregator data_extractor = PatternPairAggregator() # Add pattern for JSON data data_extractor.add_pattern_pair( pattern_id = "json_data" , start_pattern = "```json" , end_pattern = "```" , remove_match = True ) # Track extracted data extracted_data = {} # Register handler for JSON data def on_json_data ( match : PatternMatch): try : data = json.loads(match.content) extracted_data.update(data) except json.JSONDecodeError as e: logger.error( f "Failed to parse JSON: { e } " ) data_extractor.on_pattern_match( "json_data" , on_json_data) ​ Concept Explanation with Multiple Patterns Copy Ask AI # Create pattern aggregator concept_aggregator = PatternPairAggregator() # Add patterns for different parts of an explanation concept_aggregator.add_pattern_pair( pattern_id = "definition" , start_pattern = "<definition>" , end_pattern = "</definition>" , remove_match = False # Keep the tags in the output ) concept_aggregator.add_pattern_pair( pattern_id = "example" , start_pattern = "<example>" , end_pattern = "</example>" , remove_match = False ) # Register handlers def on_definition ( match : PatternMatch): logger.info( f "Found definition: { match.content } " ) # Could format differently, store for later, etc. def on_example ( match : PatternMatch): logger.info( f "Found example: { match.content } " ) # Could create a visual representation, etc. concept_aggregator.on_pattern_match( "definition" , on_definition) concept_aggregator.on_pattern_match( "example" , on_example) ​ How It Works ​ Notes Patterns are processed in the order they appear in the text Handlers are called when complete patterns are found Patterns can span multiple sentences of text, but be aware that encoding many “reasoning” tokens may slow down the LLM response MarkdownTextFilter TranscriptProcessor On this page Overview Constructor Methods add_pattern_pair on_pattern_match Pattern Match Object Usage Examples Voice Switching in TTS Extracting Structured Data from LLM Outputs Concept Explanation with Multiple Patterns How It Works Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/text_pattern-pair-aggregator_3bcd7b4c.txt b/text_pattern-pair-aggregator_3bcd7b4c.txt
new file mode 100644
index 0000000000000000000000000000000000000000..c3896347b2a86e196ae9df21a8da03b2781842eb
--- /dev/null
+++ b/text_pattern-pair-aggregator_3bcd7b4c.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/text/pattern-pair-aggregator#concept-explanation-with-multiple-patterns
+Title: PatternPairAggregator - Pipecat
+==================================================
+
+PatternPairAggregator - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text Aggregators and Filters PatternPairAggregator Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters MarkdownTextFilter PatternPairAggregator User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview PatternPairAggregator is a specialized text aggregator that buffers streaming text until it can identify complete pattern pairs (like XML tags, markdown formatting, or custom delimiters). It processes the content between these patterns using registered handlers and returns text at sentence boundaries (therefore allowing normal TTS processing to occur). This aggregator is particularly useful for applications like voice switching, structured content processing, and extracting metadata from LLM outputs, ensuring that patterns spanning multiple text chunks are correctly identified. Want to see it in action? Check out the voice switching demo . ​ Constructor Copy Ask AI aggregator = PatternPairAggregator() No parameters are required for initialization. The aggregator starts with an empty buffer and no registered patterns. ​ Methods ​ add_pattern_pair Copy Ask AI aggregator.add_pattern_pair(pattern_id, start_pattern, end_pattern, remove_match = True ) Registers a new pattern pair to detect in the text. ​ pattern_id str required Unique identifier for this pattern pair ​ start_pattern str required Pattern that marks the beginning of content ​ end_pattern str required Pattern that marks the end of content ​ remove_match bool default: "True" Whether to remove the matched patterns from the output text ​ Returns Self for method chaining ​ on_pattern_match Copy Ask AI aggregator.on_pattern_match(pattern_id, handler) Registers a handler function to be called when a specific pattern pair is matched. ​ pattern_id str required ID of the pattern pair to match ​ handler Callable[[PatternMatch], None] required Function to call when the pattern is matched. The function should accept a PatternMatch object. ​ Returns Self for method chaining ​ Pattern Match Object When a pattern is matched, the handler function receives a PatternMatch object with these attributes: ​ pattern_id str The identifier of the matched pattern pair ​ full_match str The complete text including start and end patterns ​ content str The text content between the start and end patterns ​ Usage Examples ​ Voice Switching in TTS Copy Ask AI # Define voice IDs VOICE_IDS = { "narrator" : "c45bc5ec-dc68-4feb-8829-6e6b2748095d" , "female" : "71a7ad14-091c-4e8e-a314-022ece01c121" , "male" : "7cf0e2b1-8daf-4fe4-89ad-f6039398f359" , } # Create pattern aggregator pattern_aggregator = PatternPairAggregator() # Add pattern for voice tags pattern_aggregator.add_pattern_pair( pattern_id = "voice_tag" , start_pattern = "<voice>" , end_pattern = "</voice>" , remove_match = True ) # Register handler for voice switching def on_voice_tag ( match : PatternMatch): voice_name = match.content.strip().lower() if voice_name in VOICE_IDS : voice_id = VOICE_IDS [voice_name] tts.set_voice(voice_id) logger.info( f "Switched to { voice_name } voice" ) pattern_aggregator.on_pattern_match( "voice_tag" , on_voice_tag) # Set the aggregator on a TTS service tts = CartesiaTTSService( api_key = os.getenv( "CARTESIA_API_KEY" ), voice_id = VOICE_IDS [ "narrator" ], text_aggregator = pattern_aggregator ) ​ Extracting Structured Data from LLM Outputs Copy Ask AI # Create pattern aggregator data_extractor = PatternPairAggregator() # Add pattern for JSON data data_extractor.add_pattern_pair( pattern_id = "json_data" , start_pattern = "```json" , end_pattern = "```" , remove_match = True ) # Track extracted data extracted_data = {} # Register handler for JSON data def on_json_data ( match : PatternMatch): try : data = json.loads(match.content) extracted_data.update(data) except json.JSONDecodeError as e: logger.error( f "Failed to parse JSON: { e } " ) data_extractor.on_pattern_match( "json_data" , on_json_data) ​ Concept Explanation with Multiple Patterns Copy Ask AI # Create pattern aggregator concept_aggregator = PatternPairAggregator() # Add patterns for different parts of an explanation concept_aggregator.add_pattern_pair( pattern_id = "definition" , start_pattern = "<definition>" , end_pattern = "</definition>" , remove_match = False # Keep the tags in the output ) concept_aggregator.add_pattern_pair( pattern_id = "example" , start_pattern = "<example>" , end_pattern = "</example>" , remove_match = False ) # Register handlers def on_definition ( match : PatternMatch): logger.info( f "Found definition: { match.content } " ) # Could format differently, store for later, etc. def on_example ( match : PatternMatch): logger.info( f "Found example: { match.content } " ) # Could create a visual representation, etc. concept_aggregator.on_pattern_match( "definition" , on_definition) concept_aggregator.on_pattern_match( "example" , on_example) ​ How It Works ​ Notes Patterns are processed in the order they appear in the text Handlers are called when complete patterns are found Patterns can span multiple sentences of text, but be aware that encoding many “reasoning” tokens may slow down the LLM response MarkdownTextFilter TranscriptProcessor On this page Overview Constructor Methods add_pattern_pair on_pattern_match Pattern Match Object Usage Examples Voice Switching in TTS Extracting Structured Data from LLM Outputs Concept Explanation with Multiple Patterns How It Works Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/text_pattern-pair-aggregator_40872c5a.txt b/text_pattern-pair-aggregator_40872c5a.txt
new file mode 100644
index 0000000000000000000000000000000000000000..a4adb4ac0d47437e71d0f58cab79eaf831d12aa7
--- /dev/null
+++ b/text_pattern-pair-aggregator_40872c5a.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/text/pattern-pair-aggregator#notes
+Title: PatternPairAggregator - Pipecat
+==================================================
+
+PatternPairAggregator - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text Aggregators and Filters PatternPairAggregator Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters MarkdownTextFilter PatternPairAggregator User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview PatternPairAggregator is a specialized text aggregator that buffers streaming text until it can identify complete pattern pairs (like XML tags, markdown formatting, or custom delimiters). It processes the content between these patterns using registered handlers and returns text at sentence boundaries (therefore allowing normal TTS processing to occur). This aggregator is particularly useful for applications like voice switching, structured content processing, and extracting metadata from LLM outputs, ensuring that patterns spanning multiple text chunks are correctly identified. Want to see it in action? Check out the voice switching demo . ​ Constructor Copy Ask AI aggregator = PatternPairAggregator() No parameters are required for initialization. The aggregator starts with an empty buffer and no registered patterns. ​ Methods ​ add_pattern_pair Copy Ask AI aggregator.add_pattern_pair(pattern_id, start_pattern, end_pattern, remove_match = True ) Registers a new pattern pair to detect in the text. ​ pattern_id str required Unique identifier for this pattern pair ​ start_pattern str required Pattern that marks the beginning of content ​ end_pattern str required Pattern that marks the end of content ​ remove_match bool default: "True" Whether to remove the matched patterns from the output text ​ Returns Self for method chaining ​ on_pattern_match Copy Ask AI aggregator.on_pattern_match(pattern_id, handler) Registers a handler function to be called when a specific pattern pair is matched. ​ pattern_id str required ID of the pattern pair to match ​ handler Callable[[PatternMatch], None] required Function to call when the pattern is matched. The function should accept a PatternMatch object. ​ Returns Self for method chaining ​ Pattern Match Object When a pattern is matched, the handler function receives a PatternMatch object with these attributes: ​ pattern_id str The identifier of the matched pattern pair ​ full_match str The complete text including start and end patterns ​ content str The text content between the start and end patterns ​ Usage Examples ​ Voice Switching in TTS Copy Ask AI # Define voice IDs VOICE_IDS = { "narrator" : "c45bc5ec-dc68-4feb-8829-6e6b2748095d" , "female" : "71a7ad14-091c-4e8e-a314-022ece01c121" , "male" : "7cf0e2b1-8daf-4fe4-89ad-f6039398f359" , } # Create pattern aggregator pattern_aggregator = PatternPairAggregator() # Add pattern for voice tags pattern_aggregator.add_pattern_pair( pattern_id = "voice_tag" , start_pattern = "<voice>" , end_pattern = "</voice>" , remove_match = True ) # Register handler for voice switching def on_voice_tag ( match : PatternMatch): voice_name = match.content.strip().lower() if voice_name in VOICE_IDS : voice_id = VOICE_IDS [voice_name] tts.set_voice(voice_id) logger.info( f "Switched to { voice_name } voice" ) pattern_aggregator.on_pattern_match( "voice_tag" , on_voice_tag) # Set the aggregator on a TTS service tts = CartesiaTTSService( api_key = os.getenv( "CARTESIA_API_KEY" ), voice_id = VOICE_IDS [ "narrator" ], text_aggregator = pattern_aggregator ) ​ Extracting Structured Data from LLM Outputs Copy Ask AI # Create pattern aggregator data_extractor = PatternPairAggregator() # Add pattern for JSON data data_extractor.add_pattern_pair( pattern_id = "json_data" , start_pattern = "```json" , end_pattern = "```" , remove_match = True ) # Track extracted data extracted_data = {} # Register handler for JSON data def on_json_data ( match : PatternMatch): try : data = json.loads(match.content) extracted_data.update(data) except json.JSONDecodeError as e: logger.error( f "Failed to parse JSON: { e } " ) data_extractor.on_pattern_match( "json_data" , on_json_data) ​ Concept Explanation with Multiple Patterns Copy Ask AI # Create pattern aggregator concept_aggregator = PatternPairAggregator() # Add patterns for different parts of an explanation concept_aggregator.add_pattern_pair( pattern_id = "definition" , start_pattern = "<definition>" , end_pattern = "</definition>" , remove_match = False # Keep the tags in the output ) concept_aggregator.add_pattern_pair( pattern_id = "example" , start_pattern = "<example>" , end_pattern = "</example>" , remove_match = False ) # Register handlers def on_definition ( match : PatternMatch): logger.info( f "Found definition: { match.content } " ) # Could format differently, store for later, etc. def on_example ( match : PatternMatch): logger.info( f "Found example: { match.content } " ) # Could create a visual representation, etc. concept_aggregator.on_pattern_match( "definition" , on_definition) concept_aggregator.on_pattern_match( "example" , on_example) ​ How It Works ​ Notes Patterns are processed in the order they appear in the text Handlers are called when complete patterns are found Patterns can span multiple sentences of text, but be aware that encoding many “reasoning” tokens may slow down the LLM response MarkdownTextFilter TranscriptProcessor On this page Overview Constructor Methods add_pattern_pair on_pattern_match Pattern Match Object Usage Examples Voice Switching in TTS Extracting Structured Data from LLM Outputs Concept Explanation with Multiple Patterns How It Works Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/text_pattern-pair-aggregator_ce893ed4.txt b/text_pattern-pair-aggregator_ce893ed4.txt
new file mode 100644
index 0000000000000000000000000000000000000000..f056b11e26fe00af6621f281978c69a91536b453
--- /dev/null
+++ b/text_pattern-pair-aggregator_ce893ed4.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/text/pattern-pair-aggregator#param-start-pattern
+Title: PatternPairAggregator - Pipecat
+==================================================
+
+PatternPairAggregator - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text Aggregators and Filters PatternPairAggregator Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters MarkdownTextFilter PatternPairAggregator User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview PatternPairAggregator is a specialized text aggregator that buffers streaming text until it can identify complete pattern pairs (like XML tags, markdown formatting, or custom delimiters). It processes the content between these patterns using registered handlers and returns text at sentence boundaries (therefore allowing normal TTS processing to occur). This aggregator is particularly useful for applications like voice switching, structured content processing, and extracting metadata from LLM outputs, ensuring that patterns spanning multiple text chunks are correctly identified. Want to see it in action? Check out the voice switching demo . ​ Constructor Copy Ask AI aggregator = PatternPairAggregator() No parameters are required for initialization. The aggregator starts with an empty buffer and no registered patterns. ​ Methods ​ add_pattern_pair Copy Ask AI aggregator.add_pattern_pair(pattern_id, start_pattern, end_pattern, remove_match = True ) Registers a new pattern pair to detect in the text. ​ pattern_id str required Unique identifier for this pattern pair ​ start_pattern str required Pattern that marks the beginning of content ​ end_pattern str required Pattern that marks the end of content ​ remove_match bool default: "True" Whether to remove the matched patterns from the output text ​ Returns Self for method chaining ​ on_pattern_match Copy Ask AI aggregator.on_pattern_match(pattern_id, handler) Registers a handler function to be called when a specific pattern pair is matched. ​ pattern_id str required ID of the pattern pair to match ​ handler Callable[[PatternMatch], None] required Function to call when the pattern is matched. The function should accept a PatternMatch object. ​ Returns Self for method chaining ​ Pattern Match Object When a pattern is matched, the handler function receives a PatternMatch object with these attributes: ​ pattern_id str The identifier of the matched pattern pair ​ full_match str The complete text including start and end patterns ​ content str The text content between the start and end patterns ​ Usage Examples ​ Voice Switching in TTS Copy Ask AI # Define voice IDs VOICE_IDS = { "narrator" : "c45bc5ec-dc68-4feb-8829-6e6b2748095d" , "female" : "71a7ad14-091c-4e8e-a314-022ece01c121" , "male" : "7cf0e2b1-8daf-4fe4-89ad-f6039398f359" , } # Create pattern aggregator pattern_aggregator = PatternPairAggregator() # Add pattern for voice tags pattern_aggregator.add_pattern_pair( pattern_id = "voice_tag" , start_pattern = "<voice>" , end_pattern = "</voice>" , remove_match = True ) # Register handler for voice switching def on_voice_tag ( match : PatternMatch): voice_name = match.content.strip().lower() if voice_name in VOICE_IDS : voice_id = VOICE_IDS [voice_name] tts.set_voice(voice_id) logger.info( f "Switched to { voice_name } voice" ) pattern_aggregator.on_pattern_match( "voice_tag" , on_voice_tag) # Set the aggregator on a TTS service tts = CartesiaTTSService( api_key = os.getenv( "CARTESIA_API_KEY" ), voice_id = VOICE_IDS [ "narrator" ], text_aggregator = pattern_aggregator ) ​ Extracting Structured Data from LLM Outputs Copy Ask AI # Create pattern aggregator data_extractor = PatternPairAggregator() # Add pattern for JSON data data_extractor.add_pattern_pair( pattern_id = "json_data" , start_pattern = "```json" , end_pattern = "```" , remove_match = True ) # Track extracted data extracted_data = {} # Register handler for JSON data def on_json_data ( match : PatternMatch): try : data = json.loads(match.content) extracted_data.update(data) except json.JSONDecodeError as e: logger.error( f "Failed to parse JSON: { e } " ) data_extractor.on_pattern_match( "json_data" , on_json_data) ​ Concept Explanation with Multiple Patterns Copy Ask AI # Create pattern aggregator concept_aggregator = PatternPairAggregator() # Add patterns for different parts of an explanation concept_aggregator.add_pattern_pair( pattern_id = "definition" , start_pattern = "<definition>" , end_pattern = "</definition>" , remove_match = False # Keep the tags in the output ) concept_aggregator.add_pattern_pair( pattern_id = "example" , start_pattern = "<example>" , end_pattern = "</example>" , remove_match = False ) # Register handlers def on_definition ( match : PatternMatch): logger.info( f "Found definition: { match.content } " ) # Could format differently, store for later, etc. def on_example ( match : PatternMatch): logger.info( f "Found example: { match.content } " ) # Could create a visual representation, etc. concept_aggregator.on_pattern_match( "definition" , on_definition) concept_aggregator.on_pattern_match( "example" , on_example) ​ How It Works ​ Notes Patterns are processed in the order they appear in the text Handlers are called when complete patterns are found Patterns can span multiple sentences of text, but be aware that encoding many “reasoning” tokens may slow down the LLM response MarkdownTextFilter TranscriptProcessor On this page Overview Constructor Methods add_pattern_pair on_pattern_match Pattern Match Object Usage Examples Voice Switching in TTS Extracting Structured Data from LLM Outputs Concept Explanation with Multiple Patterns How It Works Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/transport_daily_08f16e9a.txt b/transport_daily_08f16e9a.txt
new file mode 100644
index 0000000000000000000000000000000000000000..0c7a7dcfc649a438fe28ecb3e07ec1b0b7a151b8
--- /dev/null
+++ b/transport_daily_08f16e9a.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/transport/daily#param-on-dialout-error
+Title: Daily WebRTC - Pipecat
+==================================================
+
+Daily WebRTC - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Transport Daily WebRTC Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Daily WebRTC SmallWebRTCTransport FastAPI WebSocket WebSocket Server Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview DailyTransport provides real-time audio and video communication capabilities using Daily’s WebRTC platform. It supports bidirectional audio/video streams, transcription, voice activity detection (VAD), and participant management. ​ Installation To use DailyTransport , install the required dependencies: Copy Ask AI pip install "pipecat-ai[daily]" You’ll also need to set up your Daily API key as an environment variable: DAILY_API_KEY . You can obtain a Daily API key by signing up at Daily . ​ Configuration ​ Constructor Parameters ​ room_url str required Daily room URL ​ token str | None Daily room token ​ bot_name str required Name of the bot in the room ​ params DailyParams default: "DailyParams()" Transport configuration parameters ​ DailyParams Configuration ​ api_url str default: "https://api.daily.co/v1" Daily API endpoint URL ​ api_key str default: "" Daily API key for authentication ​ Audio Output Configuration ​ audio_out_enabled bool default: "False" Enable audio output capabilities ​ audio_out_is_live bool default: "False" Enable live audio streaming mode ​ audio_out_sample_rate int default: "None" Audio output sample rate in Hz ​ audio_out_channels int default: "1" Number of audio output channels ​ audio_out_bitrate int default: "96000" Audio output bitrate in bits per second ​ Audio Input Configuration ​ audio_in_enabled bool default: "False" Enable audio input capabilities ​ audio_in_passthrough bool default: "False" When enabled, incoming audio frames are pushed downstream ​ audio_in_sample_rate int default: "None" Audio input sample rate in Hz ​ audio_in_channels int default: "1" Number of audio input channels ​ audio_in_filter Optional[BaseAudioFilter] default: "None" Audio filter for input processing. Supported filters are: KrispFilter() and NoisereduceFilter() . See the KrispFilter() reference docs and NoisereduceFilter() reference docs ` for more information. ​ Video Output Configuration ​ video_out_enabled bool default: "False" Enable video output capabilities ​ video_out_is_live bool default: "False" Enable live video streaming mode ​ video_out_width int default: "1024" Video output width in pixels ​ video_out_height int default: "768" Video output height in pixels ​ video_out_bitrate int default: "800000" Video output bitrate in bits per second ​ video_out_framerate int default: "30" Video output frame rate ​ video_out_color_format str default: "RGB" Video color format (RGB, BGR, etc.) ​ Voice Activity Detection (VAD) ​ vad_analyzer VADAnalyzer | None default: "None" Voice Activity Detection analyzer. You can set this to either SileroVADAnalyzer() or WebRTCVADAnalyzer() . SileroVADAnalyzer is the recommended option. Learn more about the SileroVADAnalyzer . ​ Feature Settings ​ dialin_settings Optional[DailyDialinSettings] default: "None" Configuration for dial-in functionality ​ transcription_enabled bool default: "False" Enable real-time transcription ​ transcription_settings DailyTranscriptionSettings Configuration for transcription features Copy Ask AI class DailyTranscriptionSettings ( BaseModel ): language: str = "en" # Default language model: str = "nova-2-general" # Transcription model profanity_filter: bool = True # Filter profanity redact: bool = False # Redact sensitive information endpointing: bool = True # Enable speech endpointing punctuate: bool = True # Add punctuation includeRawResponse: bool = True # Include raw API response extra: Mapping[ str , Any] = { "interim_results" : True # Provide any Deepgram-specific settings } ​ Basic Usage Copy Ask AI from pipecat.transports.services.daily import DailyTransport, DailyParams from pipecat.audio.vad.silero import SileroVADAnalyzer # Configure transport transport = DailyTransport( room_url = "https://your-domain.daily.co/room-name" , token = "your-room-token" , bot_name = "AI Assistant" , params = DailyParams( audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer( params = VADParams( stop_secs = 0.5 )), ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), # Handle incoming audio/video stt, # Speech-to-text llm, # Language model tts, # Text-to-speech transport.output() # Handle outgoing audio/video ]) # Use transport event handlers to manage the call lifecycle # Event handler for when the user joins the room: @transport.event_handler ( "on_first_participant_joined" ) async def on_first_participant_joined ( transport , participant ): # Start transcription for the user await transport.capture_participant_transcription(participant[ "id" ]) # Kick off the conversation. await task.queue_frames([context_aggregator.user().get_context_frame()]) # Event handler for when the user leaves the room: @transport.event_handler ( "on_participant_left" ) async def on_participant_left ( transport , participant , reason ): # Cancel the pipeline, which stops processing and removes the bot from the room await task.cancel() ​ Event Callbacks DailyTransport provides a comprehensive callback system for handling various events. Register callbacks using the @transport.event_handler() decorator. ​ Connection Events ​ on_joined async callback Called when the bot successfully joins the room. Parameters: transport : The DailyTransport instance data : Dictionary containing room join information Copy Ask AI @transport.event_handler ( "on_joined" ) async def on_joined ( transport , data : Dict[ str , Any]): logger.info( f "Joined room with data: { data } " ) ​ on_left async callback Called when the bot leaves the room. Parameters: transport : The DailyTransport instance Copy Ask AI @transport.event_handler ( "on_left" ) async def on_left ( transport ): logger.info( "Left room" ) ​ Participant Events ​ on_first_participant_joined async callback Called when the first participant joins the room. Useful for initializing conversations. Parameters: transport : The DailyTransport instance participant : Dictionary containing participant information Copy Ask AI @transport.event_handler ( "on_first_participant_joined" ) async def on_first_participant_joined ( transport , participant : Dict[ str , Any]): await transport.capture_participant_transcription(participant[ "id" ]) await task.queue_frames([LLMMessagesFrame(initial_messages)]) ​ on_participant_joined async callback Called when any participant joins the room. Parameters: transport : The DailyTransport instance participant : Dictionary containing participant information Copy Ask AI @transport.event_handler ( "on_participant_joined" ) async def on_participant_joined ( transport , participant : Dict[ str , Any]): logger.info( f "Participant joined: { participant[ 'id' ] } " ) ​ on_participant_left async callback Called when a participant leaves the room. Parameters: transport : The DailyTransport instance participant : Dictionary containing participant information reason : String describing why the participant left, “leftCall” | “hidden” Copy Ask AI @transport.event_handler ( "on_participant_left" ) async def on_participant_left ( transport , participant : Dict[ str , Any], reason : str ): logger.info( f "Participant { participant[ 'id' ] } left: { reason } " ) # Cancel the pipeline task to stop processing and remove the bot from the Daily room await task.cancel() ​ on_participant_updated async callback Event emitted when a participant is updated. This can mean either the participant’s metadata was updated, or the tracks belonging to the participant changed. Parameters: transport : The DailyTransport instance participant : Dictionary containing updated participant information Copy Ask AI @transport.event_handler ( "on_participant_updated" ) async def on_participant_updated ( transport , participant : Dict[ str , Any]): logger.info( f "Participant updated: { participant } " ) ​ Communication Events ​ on_app_message async callback Event emitted when a custom app message is received from another participant or via the REST API. Parameters: transport : The DailyTransport instance message : The message content (any type) sender : String identifier of the message sender Copy Ask AI @transport.event_handler ( "on_app_message" ) async def on_app_message ( transport , message : Any, sender : str ): logger.info( f "Message from { sender } : { message } " ) ​ on_call_state_updated async callback Event emitted when the call state changes, normally as a consequence of joining or leaving the call. Parameters: transport : The DailyTransport instance state : String representing the new call state. Learn more about call states . Copy Ask AI @transport.event_handler ( "on_call_state_updated" ) async def on_call_state_updated ( transport , state : str ): logger.info( f "Call state updated: { state } " ) ​ Dial Events ​ Dial-in ​ on_dialin_ready async callback Event emitted when dial-in is ready. This happens after the room has connected to the SIP endpoint and the system is ready to receive dial-in calls. Parameters: transport : The DailyTransport instance sip_endpoint : String containing the SIP endpoint information Copy Ask AI @transport.event_handler ( "on_dialin_ready" ) async def on_dialin_ready ( transport , sip_endpoint : str ): logger.info( f "Dial-in ready at: { sip_endpoint } " ) ​ on_dialin_connected async callback Event emitted when the session with the dial-in remote end is established (i.e. SIP endpoint or PSTN are connectd to the Daily room). Note: connected does not mean media (audio or video) has started flowing between the room and PSTN, it means the room received the connection request and both endpoints are negotiating the media flow. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. See DialinConnectedEvent . Copy Ask AI @transport.event_handler ( "on_dialin_connected" ) async def on_dialin_connected ( transport , data : Any): logger.info( f "Dial-in connected: { data } " ) ​ on_dialin_stopped async callback Event emitted when the dial-in remote end disconnects the call. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. See DialinStoppedEvent . Copy Ask AI @transport.event_handler ( "on_dialin_stopped" ) async def on_dialin_stopped ( transport , data : Any): logger.info( f "Dial-in stopped: { data } " ) ​ on_dialin_error async callback Event emitted in the case of dial-in errors which are fatal and the service cannot proceed. For example, an error in SDP negotiation is fatal to the media/SIP pipeline and will result in dialin-error being triggered. Parameters: transport : The DailyTransport instance data : Dictionary containing error information. See DialinEvent . Copy Ask AI @transport.event_handler ( "on_dialin_error" ) async def on_dialin_error ( transport , data : Any): logger.error( f "Dial-in error: { data } " ) ​ on_dialin_warning async callback Event emitted there is a dial-in non-fatal error, such as the selected codec not being used and a fallback codec being utilized. Parameters: transport : The DailyTransport instance data : Dictionary containing warning information. See DialinEvent . Copy Ask AI @transport.event_handler ( "on_dialin_warning" ) async def on_dialin_warning ( transport , data : Any): logger.warning( f "Dial-in warning: { data } " ) ​ Dial-out ​ on_dialout_answered async callback Event emitted when the session with the dial-out remote end is answered. Parameters: transport : The DailyTransport instance data : Dictionary containing call information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_answered" ) async def on_dialout_answered ( transport , data : Any): logger.info( f "Dial-out answered: { data } " ) ​ on_dialout_connected async callback Event emitted when the session with the dial-out remote end is established. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_connected" ) async def on_dialout_connected ( transport , data : Any): logger.info( f "Dial-out connected: { data } " ) ​ on_dialout_stopped async callback Event emitted when the dial-out session is stopped. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_stopped" ) async def on_dialout_stopped ( transport , data : Any): logger.info( f "Dial-out stopped: { data } " ) ​ on_dialout_error async callback Event emitted in the case of dial-out errors which are fatal and the service cannot proceed. For example, an error in SDP negotiation is fatal to the media/SIP pipeline and will result in dialout-error being triggered. Parameters: transport : The DailyTransport instance data : Dictionary containing error information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_error" ) async def on_dialout_error ( transport , data : Any): logger.error( f "Dial-out error: { data } " ) ​ on_dialout_warning async callback Event emitted there is a dial-out non-fatal error, such as the selected codec not being used and a fallback codec being utilized. Parameters: transport : The DailyTransport instance data : Dictionary containing warning information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_warning" ) async def on_dialout_warning ( transport , data : Any): logger.warning( f "Dial-out warning: { data } " ) ​ Transcription Events ​ on_transcription_message async callback Called when a transcription message is received. This includes both interim and final transcriptions. Parameters: transport : The DailyTransport instance message : Dictionary containing transcription data including. Learn more . Copy Ask AI @transport.event_handler ( "on_transcription_message" ) async def on_transcription_message ( transport , message : Dict[ str , Any]): participant_id = message.get( "participantId" ) text = message.get( "text" ) is_final = message[ "rawResponse" ][ "is_final" ] logger.info( f "Transcription from { participant_id } : { text } (final: { is_final } )" ) ​ Recording Events ​ on_recording_started async callback Called when a room recording starts successfully. Parameters: transport : The DailyTransport instance status : Dictionary containing recording status information. Learn more . Copy Ask AI @transport.event_handler ( "on_recording_started" ) async def on_recording_started ( transport , status : Dict[ str , Any]): logger.info( f "Recording started with status: { status } " ) ​ on_recording_stopped async callback Called when a room recording stops. Parameters: transport : The DailyTransport instance stream_id : String identifier of the stopped recording stream Copy Ask AI @transport.event_handler ( "on_recording_stopped" ) async def on_recording_stopped ( transport , stream_id : str ): logger.info( f "Recording stopped for stream: { stream_id } " ) ​ on_recording_error async callback Called when an error occurs during recording. Parameters: transport : The DailyTransport instance stream_id : String identifier of the recording stream message : Error message describing what went wrong Copy Ask AI @transport.event_handler ( "on_recording_error" ) async def on_recording_error ( transport , stream_id : str , message : str ): logger.error( f "Recording error for stream { stream_id } : { message } " ) ​ Error Events ​ on_error async callback Called when a transport error occurs. Parameters: transport : The DailyTransport instance error : String containing error details Copy Ask AI @transport.event_handler ( "on_error" ) async def on_error ( transport , error : str ): logger.error( f "Transport error: { error } " ) ​ Notes on Callbacks All callbacks are asynchronous Callbacks are executed in the order they were registered Multiple handlers can be registered for the same event Exceptions in callbacks are logged but don’t stop the transport Callbacks should be lightweight to avoid blocking the event loop Heavy processing should be offloaded to separate tasks ​ Frame Types ​ Input Frames ​ InputAudioRawFrame Frame Raw audio data from participants ​ UserImageRawFrame Frame Video frames from participants ​ Output Frames ​ OutputAudioRawFrame Frame Audio data to be sent ​ OutputImageRawFrame Frame Video frames to be sent ​ Methods ​ Room Management ​ participants method Returns a dictionary of all participants currently in the room. Copy Ask AI def participants () -> Dict[ str , Any] See participants for more details. ​ participant_counts method Returns participant count statistics for the room. Copy Ask AI def participant_counts () -> Dict[ str , int ] See participant_counts for more details. ​ send_message async method Sends a message to participants in the Daily room. Messages can be directed to all participants or targeted to a specific participant. Copy Ask AI async def send_message ( frame : TransportMessageFrame | TransportMessageUrgentFrame) Parameters: frame : A transport message frame containing the message to send. Can be either: TransportMessageFrame : Standard message TransportMessageUrgentFrame : High-priority message Message Structure: Messages should conform to the expected format for the receiving system. For RTVI messages, use: Copy Ask AI { "label" : "rtvi-ai" , # Required for RTVI protocol "type" : "message-type" , # Defines how the client will process it "id" : "unique-id" , # Unique identifier for the message "data" : { # Custom payload # Your custom data fields } } Example - Standard Message: Copy Ask AI import uuid # Send a standard message to all participants message_data = { "label" : "rtvi-ai" , "type" : "custom-message" , "id" : str (uuid.uuid4()), "data" : { "message" : "Hello everyone!" , "timestamp" : datetime.datetime.now().isoformat() } } await transport.send_message(TransportMessageFrame( message = message_data)) Example - Urgent Message to Specific Participant: Copy Ask AI # Send urgent message to specific participant urgent_data = { "label" : "rtvi-ai" , "type" : "server-message" , "id" : str (uuid.uuid4()), "data" : { "message" : "Private message" , "importance" : "high" } } await transport.send_message( DailyTransportMessageUrgentFrame( message = urgent_data, participant_id = "participant-123" ) ) The message will be delivered to: All participants if no specific participant_id is set Only the specified participant if participant_id is set Through the Daily app messaging system Can be received by registering an on_app_message handler Use TransportMessageUrgentFrame for high-priority messages that need immediate delivery. ​ Media Control ​ send_image async method Sends an image frame to the room. Copy Ask AI async def send_image ( frame : OutputImageRawFrame | SpriteFrame) -> None Parameters: frame : Image frame to send, either raw image data or sprite animation ​ send_audio async method Sends an audio frame to the room. Copy Ask AI async def send_audio ( frame : OutputAudioRawFrame) -> None Parameters: frame : Audio frame to send ​ Video Management ​ capture_participant_video async method Starts capturing video from a specific participant. Copy Ask AI async def capture_participant_video ( participant_id : str , framerate : int = 30 , video_source : str = "camera" , color_format : str = "RGB" ) -> None Parameters: participant_id : ID of the participant to capture framerate : Target frame rate (default: 30) video_source : Video source type (default: “camera”) color_format : Color format of the video (default: “RGB”) To request an image from the camera stream, set the framerate to 0 and push a UserImageRequestFrame . ​ Transcription Control ​ capture_participant_transcription async method Starts capturing and transcribing audio from a specific participant. Copy Ask AI async def capture_participant_transcription ( participant_id : str ) -> None Parameters: participant_id : ID of the participant to transcribe ​ update_transcription async method Updates the transcription configuration for specific participants or instances. Copy Ask AI async def update_transcription ( participants : List[ str ] | None = None , instance_id : str | None = None ) -> None Parameters: participants : Optional list of participant IDs to transcribe. If None, transcribes all participants. instance_id : Optional specific transcription instance ID to update Example: Copy Ask AI # Update transcription for specific participants await transport.update_transcription( participants = [ "participant-123" , "participant-456" ]) # Update specific transcription instance await transport.update_transcription( instance_id = "transcription-789" ) ​ Recording Control ​ start_recording async method Starts recording the room session. Copy Ask AI async def start_recording ( streaming_settings : Dict = None , stream_id : str = None , force_new : bool = None ) -> None Parameters: streaming_settings : Recording configuration settings stream_id : Optional stream identifier force_new : Force start a new recording See start_recording for more details. ​ stop_recording async method Stops an active recording. Copy Ask AI async def stop_recording ( stream_id : str = None ) -> None Parameters: stream_id : Optional stream identifier to stop specific recording See stop_recording for more details. ​ Dial-out Control ​ start_dialout async method Initiates a dial-out call. Copy Ask AI async def start_dialout ( settings : Dict = None ) -> None Parameters: settings : Dial-out configuration settings See start_dialout for more details. ​ stop_dialout async method Stops an active dial-out call. Copy Ask AI async def stop_dialout ( participant_id : str ) -> None Parameters: participant_id : ID of the participant to disconnect See stop_dialout for more details. ​ send_dtmf async method Sends DTMF tones in an existing dial-out session. Copy Ask AI async def send_dtmf ( settings : Dict = None ) -> None Parameters: settings : DTMF settings See send_dtmf for more details. ​ Subscription Management ​ update_subscriptions async method Updates subscriptions and subscription profiles. This function allows you to update subscription profiles and at the same time assign specific subscription profiles to a participant and even change specific settings for some participants. Copy Ask AI async def update_subscriptions ( participant_settings : Dict = None , profile_settings : Dict = None ) -> None Parameters: participant_settings : Per-participant subscription settings profile_settings : Global subscription profile settings See update_subscriptions for more details. ​ Participant Admin When using the DailyTransport, if your bot is given a meeting token with owner or participant admin permissions, it can remote-control other participants’ capabilities and permissions. With this control, the bot can modify settings for how others send and receive media, how they are seen in a session, and whether they have administrative controls. ​ update_remote_participants async method Updates permissions and settings for specific participants in the room. Copy Ask AI async def update_remote_participants ( remote_participants : Mapping[ str , Any] = None ) -> None Parameters: remote_participants : Dictionary mapping participant IDs to their updated settings This method allows fine-grained control over participant capabilities and permissions in the Daily room. Examples: Copy Ask AI # Mute a participant's microphone await transport.update_remote_participants({ "participant-123" : { "inputsEnabled" : { "microphone" : False } } }) # Revoke send permissions (mutes the participant and prevents them from unmuting) await transport.update_remote_participants({ "participant-123" : { "permissions" : { "canSend" : [] # Empty list revokes all send permissions } } }) # Configure selective audio routing await transport.update_remote_participants({ "participant-123" : { # the customer "permissions" : { "canReceive" : { "base" : False , # Don't receive from anyone by default "byUserId" : { "hold-music" : True # Only receive from hold music player } } } } }) # Set up user-to-user permissions mapping (e.g., when transferring calls) # Assumes customer was previously unable to hear anyone (they were on hold) and agent was speaking to bot await transport.update_remote_participants({ "participant-123" : { # the customer "permissions" : { "base" : False , "canReceive" : { "byUserId" : { "agent" : True }} } }, "participant-456" : { # the agent "permissions" : { "base" : False , "canReceive" : { "byUserId" : { "customer" : True }} } } }) # Promote a participant to be a participant admin await transport.update_remote_participants({ "participant-123" : { "permissions" : { "canAdmin" : [ "participants" ] } } }) Permissions Settings: For comprehensive details on permissions settings, see the Daily API documentation: daily-python’s update_remote_participants() documentation daily-js’s updateParticipant() -> updatePermission and participants().permissions ​ Chat messages ​ send_prebuilt_chat_message async method When testing with Daily Prebuilt, you can send messages to the Prebuilt chat using this method. Parameters: message (str): The chat message to send user_name (str): The name of the user sending the message See send_prebuilt_chat_message for more details. ​ Properties ​ participant_id string Returns the transport’s participant ID in the room. ​ room_url string Returns the transport’s room URL for the room currently in use. ​ Notes Supports real-time audio/video communication Handles WebRTC connection management Provides voice activity detection Includes transcription capabilities Manages participant interactions Supports recording and streaming Thread-safe processing All callbacks are asynchronous Multiple handlers can be registered for the same event Exceptions in callbacks are logged but don’t stop the transport Supported Services SmallWebRTCTransport On this page Overview Installation Configuration Constructor Parameters DailyParams Configuration Audio Output Configuration Audio Input Configuration Video Output Configuration Voice Activity Detection (VAD) Feature Settings Basic Usage Event Callbacks Connection Events Participant Events Communication Events Dial Events Dial-in Dial-out Transcription Events Recording Events Error Events Notes on Callbacks Frame Types Input Frames Output Frames Methods Room Management Media Control Video Management Transcription Control Recording Control Dial-out Control Subscription Management Participant Admin Chat messages Properties Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/transport_daily_0b416dd5.txt b/transport_daily_0b416dd5.txt
new file mode 100644
index 0000000000000000000000000000000000000000..b827e0088c5f9ce061a51f2f362debc597ab94d4
--- /dev/null
+++ b/transport_daily_0b416dd5.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/transport/daily#video-management
+Title: Daily WebRTC - Pipecat
+==================================================
+
+Daily WebRTC - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Transport Daily WebRTC Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Daily WebRTC SmallWebRTCTransport FastAPI WebSocket WebSocket Server Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview DailyTransport provides real-time audio and video communication capabilities using Daily’s WebRTC platform. It supports bidirectional audio/video streams, transcription, voice activity detection (VAD), and participant management. ​ Installation To use DailyTransport , install the required dependencies: Copy Ask AI pip install "pipecat-ai[daily]" You’ll also need to set up your Daily API key as an environment variable: DAILY_API_KEY . You can obtain a Daily API key by signing up at Daily . ​ Configuration ​ Constructor Parameters ​ room_url str required Daily room URL ​ token str | None Daily room token ​ bot_name str required Name of the bot in the room ​ params DailyParams default: "DailyParams()" Transport configuration parameters ​ DailyParams Configuration ​ api_url str default: "https://api.daily.co/v1" Daily API endpoint URL ​ api_key str default: "" Daily API key for authentication ​ Audio Output Configuration ​ audio_out_enabled bool default: "False" Enable audio output capabilities ​ audio_out_is_live bool default: "False" Enable live audio streaming mode ​ audio_out_sample_rate int default: "None" Audio output sample rate in Hz ​ audio_out_channels int default: "1" Number of audio output channels ​ audio_out_bitrate int default: "96000" Audio output bitrate in bits per second ​ Audio Input Configuration ​ audio_in_enabled bool default: "False" Enable audio input capabilities ​ audio_in_passthrough bool default: "False" When enabled, incoming audio frames are pushed downstream ​ audio_in_sample_rate int default: "None" Audio input sample rate in Hz ​ audio_in_channels int default: "1" Number of audio input channels ​ audio_in_filter Optional[BaseAudioFilter] default: "None" Audio filter for input processing. Supported filters are: KrispFilter() and NoisereduceFilter() . See the KrispFilter() reference docs and NoisereduceFilter() reference docs ` for more information. ​ Video Output Configuration ​ video_out_enabled bool default: "False" Enable video output capabilities ​ video_out_is_live bool default: "False" Enable live video streaming mode ​ video_out_width int default: "1024" Video output width in pixels ​ video_out_height int default: "768" Video output height in pixels ​ video_out_bitrate int default: "800000" Video output bitrate in bits per second ​ video_out_framerate int default: "30" Video output frame rate ​ video_out_color_format str default: "RGB" Video color format (RGB, BGR, etc.) ​ Voice Activity Detection (VAD) ​ vad_analyzer VADAnalyzer | None default: "None" Voice Activity Detection analyzer. You can set this to either SileroVADAnalyzer() or WebRTCVADAnalyzer() . SileroVADAnalyzer is the recommended option. Learn more about the SileroVADAnalyzer . ​ Feature Settings ​ dialin_settings Optional[DailyDialinSettings] default: "None" Configuration for dial-in functionality ​ transcription_enabled bool default: "False" Enable real-time transcription ​ transcription_settings DailyTranscriptionSettings Configuration for transcription features Copy Ask AI class DailyTranscriptionSettings ( BaseModel ): language: str = "en" # Default language model: str = "nova-2-general" # Transcription model profanity_filter: bool = True # Filter profanity redact: bool = False # Redact sensitive information endpointing: bool = True # Enable speech endpointing punctuate: bool = True # Add punctuation includeRawResponse: bool = True # Include raw API response extra: Mapping[ str , Any] = { "interim_results" : True # Provide any Deepgram-specific settings } ​ Basic Usage Copy Ask AI from pipecat.transports.services.daily import DailyTransport, DailyParams from pipecat.audio.vad.silero import SileroVADAnalyzer # Configure transport transport = DailyTransport( room_url = "https://your-domain.daily.co/room-name" , token = "your-room-token" , bot_name = "AI Assistant" , params = DailyParams( audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer( params = VADParams( stop_secs = 0.5 )), ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), # Handle incoming audio/video stt, # Speech-to-text llm, # Language model tts, # Text-to-speech transport.output() # Handle outgoing audio/video ]) # Use transport event handlers to manage the call lifecycle # Event handler for when the user joins the room: @transport.event_handler ( "on_first_participant_joined" ) async def on_first_participant_joined ( transport , participant ): # Start transcription for the user await transport.capture_participant_transcription(participant[ "id" ]) # Kick off the conversation. await task.queue_frames([context_aggregator.user().get_context_frame()]) # Event handler for when the user leaves the room: @transport.event_handler ( "on_participant_left" ) async def on_participant_left ( transport , participant , reason ): # Cancel the pipeline, which stops processing and removes the bot from the room await task.cancel() ​ Event Callbacks DailyTransport provides a comprehensive callback system for handling various events. Register callbacks using the @transport.event_handler() decorator. ​ Connection Events ​ on_joined async callback Called when the bot successfully joins the room. Parameters: transport : The DailyTransport instance data : Dictionary containing room join information Copy Ask AI @transport.event_handler ( "on_joined" ) async def on_joined ( transport , data : Dict[ str , Any]): logger.info( f "Joined room with data: { data } " ) ​ on_left async callback Called when the bot leaves the room. Parameters: transport : The DailyTransport instance Copy Ask AI @transport.event_handler ( "on_left" ) async def on_left ( transport ): logger.info( "Left room" ) ​ Participant Events ​ on_first_participant_joined async callback Called when the first participant joins the room. Useful for initializing conversations. Parameters: transport : The DailyTransport instance participant : Dictionary containing participant information Copy Ask AI @transport.event_handler ( "on_first_participant_joined" ) async def on_first_participant_joined ( transport , participant : Dict[ str , Any]): await transport.capture_participant_transcription(participant[ "id" ]) await task.queue_frames([LLMMessagesFrame(initial_messages)]) ​ on_participant_joined async callback Called when any participant joins the room. Parameters: transport : The DailyTransport instance participant : Dictionary containing participant information Copy Ask AI @transport.event_handler ( "on_participant_joined" ) async def on_participant_joined ( transport , participant : Dict[ str , Any]): logger.info( f "Participant joined: { participant[ 'id' ] } " ) ​ on_participant_left async callback Called when a participant leaves the room. Parameters: transport : The DailyTransport instance participant : Dictionary containing participant information reason : String describing why the participant left, “leftCall” | “hidden” Copy Ask AI @transport.event_handler ( "on_participant_left" ) async def on_participant_left ( transport , participant : Dict[ str , Any], reason : str ): logger.info( f "Participant { participant[ 'id' ] } left: { reason } " ) # Cancel the pipeline task to stop processing and remove the bot from the Daily room await task.cancel() ​ on_participant_updated async callback Event emitted when a participant is updated. This can mean either the participant’s metadata was updated, or the tracks belonging to the participant changed. Parameters: transport : The DailyTransport instance participant : Dictionary containing updated participant information Copy Ask AI @transport.event_handler ( "on_participant_updated" ) async def on_participant_updated ( transport , participant : Dict[ str , Any]): logger.info( f "Participant updated: { participant } " ) ​ Communication Events ​ on_app_message async callback Event emitted when a custom app message is received from another participant or via the REST API. Parameters: transport : The DailyTransport instance message : The message content (any type) sender : String identifier of the message sender Copy Ask AI @transport.event_handler ( "on_app_message" ) async def on_app_message ( transport , message : Any, sender : str ): logger.info( f "Message from { sender } : { message } " ) ​ on_call_state_updated async callback Event emitted when the call state changes, normally as a consequence of joining or leaving the call. Parameters: transport : The DailyTransport instance state : String representing the new call state. Learn more about call states . Copy Ask AI @transport.event_handler ( "on_call_state_updated" ) async def on_call_state_updated ( transport , state : str ): logger.info( f "Call state updated: { state } " ) ​ Dial Events ​ Dial-in ​ on_dialin_ready async callback Event emitted when dial-in is ready. This happens after the room has connected to the SIP endpoint and the system is ready to receive dial-in calls. Parameters: transport : The DailyTransport instance sip_endpoint : String containing the SIP endpoint information Copy Ask AI @transport.event_handler ( "on_dialin_ready" ) async def on_dialin_ready ( transport , sip_endpoint : str ): logger.info( f "Dial-in ready at: { sip_endpoint } " ) ​ on_dialin_connected async callback Event emitted when the session with the dial-in remote end is established (i.e. SIP endpoint or PSTN are connectd to the Daily room). Note: connected does not mean media (audio or video) has started flowing between the room and PSTN, it means the room received the connection request and both endpoints are negotiating the media flow. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. See DialinConnectedEvent . Copy Ask AI @transport.event_handler ( "on_dialin_connected" ) async def on_dialin_connected ( transport , data : Any): logger.info( f "Dial-in connected: { data } " ) ​ on_dialin_stopped async callback Event emitted when the dial-in remote end disconnects the call. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. See DialinStoppedEvent . Copy Ask AI @transport.event_handler ( "on_dialin_stopped" ) async def on_dialin_stopped ( transport , data : Any): logger.info( f "Dial-in stopped: { data } " ) ​ on_dialin_error async callback Event emitted in the case of dial-in errors which are fatal and the service cannot proceed. For example, an error in SDP negotiation is fatal to the media/SIP pipeline and will result in dialin-error being triggered. Parameters: transport : The DailyTransport instance data : Dictionary containing error information. See DialinEvent . Copy Ask AI @transport.event_handler ( "on_dialin_error" ) async def on_dialin_error ( transport , data : Any): logger.error( f "Dial-in error: { data } " ) ​ on_dialin_warning async callback Event emitted there is a dial-in non-fatal error, such as the selected codec not being used and a fallback codec being utilized. Parameters: transport : The DailyTransport instance data : Dictionary containing warning information. See DialinEvent . Copy Ask AI @transport.event_handler ( "on_dialin_warning" ) async def on_dialin_warning ( transport , data : Any): logger.warning( f "Dial-in warning: { data } " ) ​ Dial-out ​ on_dialout_answered async callback Event emitted when the session with the dial-out remote end is answered. Parameters: transport : The DailyTransport instance data : Dictionary containing call information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_answered" ) async def on_dialout_answered ( transport , data : Any): logger.info( f "Dial-out answered: { data } " ) ​ on_dialout_connected async callback Event emitted when the session with the dial-out remote end is established. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_connected" ) async def on_dialout_connected ( transport , data : Any): logger.info( f "Dial-out connected: { data } " ) ​ on_dialout_stopped async callback Event emitted when the dial-out session is stopped. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_stopped" ) async def on_dialout_stopped ( transport , data : Any): logger.info( f "Dial-out stopped: { data } " ) ​ on_dialout_error async callback Event emitted in the case of dial-out errors which are fatal and the service cannot proceed. For example, an error in SDP negotiation is fatal to the media/SIP pipeline and will result in dialout-error being triggered. Parameters: transport : The DailyTransport instance data : Dictionary containing error information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_error" ) async def on_dialout_error ( transport , data : Any): logger.error( f "Dial-out error: { data } " ) ​ on_dialout_warning async callback Event emitted there is a dial-out non-fatal error, such as the selected codec not being used and a fallback codec being utilized. Parameters: transport : The DailyTransport instance data : Dictionary containing warning information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_warning" ) async def on_dialout_warning ( transport , data : Any): logger.warning( f "Dial-out warning: { data } " ) ​ Transcription Events ​ on_transcription_message async callback Called when a transcription message is received. This includes both interim and final transcriptions. Parameters: transport : The DailyTransport instance message : Dictionary containing transcription data including. Learn more . Copy Ask AI @transport.event_handler ( "on_transcription_message" ) async def on_transcription_message ( transport , message : Dict[ str , Any]): participant_id = message.get( "participantId" ) text = message.get( "text" ) is_final = message[ "rawResponse" ][ "is_final" ] logger.info( f "Transcription from { participant_id } : { text } (final: { is_final } )" ) ​ Recording Events ​ on_recording_started async callback Called when a room recording starts successfully. Parameters: transport : The DailyTransport instance status : Dictionary containing recording status information. Learn more . Copy Ask AI @transport.event_handler ( "on_recording_started" ) async def on_recording_started ( transport , status : Dict[ str , Any]): logger.info( f "Recording started with status: { status } " ) ​ on_recording_stopped async callback Called when a room recording stops. Parameters: transport : The DailyTransport instance stream_id : String identifier of the stopped recording stream Copy Ask AI @transport.event_handler ( "on_recording_stopped" ) async def on_recording_stopped ( transport , stream_id : str ): logger.info( f "Recording stopped for stream: { stream_id } " ) ​ on_recording_error async callback Called when an error occurs during recording. Parameters: transport : The DailyTransport instance stream_id : String identifier of the recording stream message : Error message describing what went wrong Copy Ask AI @transport.event_handler ( "on_recording_error" ) async def on_recording_error ( transport , stream_id : str , message : str ): logger.error( f "Recording error for stream { stream_id } : { message } " ) ​ Error Events ​ on_error async callback Called when a transport error occurs. Parameters: transport : The DailyTransport instance error : String containing error details Copy Ask AI @transport.event_handler ( "on_error" ) async def on_error ( transport , error : str ): logger.error( f "Transport error: { error } " ) ​ Notes on Callbacks All callbacks are asynchronous Callbacks are executed in the order they were registered Multiple handlers can be registered for the same event Exceptions in callbacks are logged but don’t stop the transport Callbacks should be lightweight to avoid blocking the event loop Heavy processing should be offloaded to separate tasks ​ Frame Types ​ Input Frames ​ InputAudioRawFrame Frame Raw audio data from participants ​ UserImageRawFrame Frame Video frames from participants ​ Output Frames ​ OutputAudioRawFrame Frame Audio data to be sent ​ OutputImageRawFrame Frame Video frames to be sent ​ Methods ​ Room Management ​ participants method Returns a dictionary of all participants currently in the room. Copy Ask AI def participants () -> Dict[ str , Any] See participants for more details. ​ participant_counts method Returns participant count statistics for the room. Copy Ask AI def participant_counts () -> Dict[ str , int ] See participant_counts for more details. ​ send_message async method Sends a message to participants in the Daily room. Messages can be directed to all participants or targeted to a specific participant. Copy Ask AI async def send_message ( frame : TransportMessageFrame | TransportMessageUrgentFrame) Parameters: frame : A transport message frame containing the message to send. Can be either: TransportMessageFrame : Standard message TransportMessageUrgentFrame : High-priority message Message Structure: Messages should conform to the expected format for the receiving system. For RTVI messages, use: Copy Ask AI { "label" : "rtvi-ai" , # Required for RTVI protocol "type" : "message-type" , # Defines how the client will process it "id" : "unique-id" , # Unique identifier for the message "data" : { # Custom payload # Your custom data fields } } Example - Standard Message: Copy Ask AI import uuid # Send a standard message to all participants message_data = { "label" : "rtvi-ai" , "type" : "custom-message" , "id" : str (uuid.uuid4()), "data" : { "message" : "Hello everyone!" , "timestamp" : datetime.datetime.now().isoformat() } } await transport.send_message(TransportMessageFrame( message = message_data)) Example - Urgent Message to Specific Participant: Copy Ask AI # Send urgent message to specific participant urgent_data = { "label" : "rtvi-ai" , "type" : "server-message" , "id" : str (uuid.uuid4()), "data" : { "message" : "Private message" , "importance" : "high" } } await transport.send_message( DailyTransportMessageUrgentFrame( message = urgent_data, participant_id = "participant-123" ) ) The message will be delivered to: All participants if no specific participant_id is set Only the specified participant if participant_id is set Through the Daily app messaging system Can be received by registering an on_app_message handler Use TransportMessageUrgentFrame for high-priority messages that need immediate delivery. ​ Media Control ​ send_image async method Sends an image frame to the room. Copy Ask AI async def send_image ( frame : OutputImageRawFrame | SpriteFrame) -> None Parameters: frame : Image frame to send, either raw image data or sprite animation ​ send_audio async method Sends an audio frame to the room. Copy Ask AI async def send_audio ( frame : OutputAudioRawFrame) -> None Parameters: frame : Audio frame to send ​ Video Management ​ capture_participant_video async method Starts capturing video from a specific participant. Copy Ask AI async def capture_participant_video ( participant_id : str , framerate : int = 30 , video_source : str = "camera" , color_format : str = "RGB" ) -> None Parameters: participant_id : ID of the participant to capture framerate : Target frame rate (default: 30) video_source : Video source type (default: “camera”) color_format : Color format of the video (default: “RGB”) To request an image from the camera stream, set the framerate to 0 and push a UserImageRequestFrame . ​ Transcription Control ​ capture_participant_transcription async method Starts capturing and transcribing audio from a specific participant. Copy Ask AI async def capture_participant_transcription ( participant_id : str ) -> None Parameters: participant_id : ID of the participant to transcribe ​ update_transcription async method Updates the transcription configuration for specific participants or instances. Copy Ask AI async def update_transcription ( participants : List[ str ] | None = None , instance_id : str | None = None ) -> None Parameters: participants : Optional list of participant IDs to transcribe. If None, transcribes all participants. instance_id : Optional specific transcription instance ID to update Example: Copy Ask AI # Update transcription for specific participants await transport.update_transcription( participants = [ "participant-123" , "participant-456" ]) # Update specific transcription instance await transport.update_transcription( instance_id = "transcription-789" ) ​ Recording Control ​ start_recording async method Starts recording the room session. Copy Ask AI async def start_recording ( streaming_settings : Dict = None , stream_id : str = None , force_new : bool = None ) -> None Parameters: streaming_settings : Recording configuration settings stream_id : Optional stream identifier force_new : Force start a new recording See start_recording for more details. ​ stop_recording async method Stops an active recording. Copy Ask AI async def stop_recording ( stream_id : str = None ) -> None Parameters: stream_id : Optional stream identifier to stop specific recording See stop_recording for more details. ​ Dial-out Control ​ start_dialout async method Initiates a dial-out call. Copy Ask AI async def start_dialout ( settings : Dict = None ) -> None Parameters: settings : Dial-out configuration settings See start_dialout for more details. ​ stop_dialout async method Stops an active dial-out call. Copy Ask AI async def stop_dialout ( participant_id : str ) -> None Parameters: participant_id : ID of the participant to disconnect See stop_dialout for more details. ​ send_dtmf async method Sends DTMF tones in an existing dial-out session. Copy Ask AI async def send_dtmf ( settings : Dict = None ) -> None Parameters: settings : DTMF settings See send_dtmf for more details. ​ Subscription Management ​ update_subscriptions async method Updates subscriptions and subscription profiles. This function allows you to update subscription profiles and at the same time assign specific subscription profiles to a participant and even change specific settings for some participants. Copy Ask AI async def update_subscriptions ( participant_settings : Dict = None , profile_settings : Dict = None ) -> None Parameters: participant_settings : Per-participant subscription settings profile_settings : Global subscription profile settings See update_subscriptions for more details. ​ Participant Admin When using the DailyTransport, if your bot is given a meeting token with owner or participant admin permissions, it can remote-control other participants’ capabilities and permissions. With this control, the bot can modify settings for how others send and receive media, how they are seen in a session, and whether they have administrative controls. ​ update_remote_participants async method Updates permissions and settings for specific participants in the room. Copy Ask AI async def update_remote_participants ( remote_participants : Mapping[ str , Any] = None ) -> None Parameters: remote_participants : Dictionary mapping participant IDs to their updated settings This method allows fine-grained control over participant capabilities and permissions in the Daily room. Examples: Copy Ask AI # Mute a participant's microphone await transport.update_remote_participants({ "participant-123" : { "inputsEnabled" : { "microphone" : False } } }) # Revoke send permissions (mutes the participant and prevents them from unmuting) await transport.update_remote_participants({ "participant-123" : { "permissions" : { "canSend" : [] # Empty list revokes all send permissions } } }) # Configure selective audio routing await transport.update_remote_participants({ "participant-123" : { # the customer "permissions" : { "canReceive" : { "base" : False , # Don't receive from anyone by default "byUserId" : { "hold-music" : True # Only receive from hold music player } } } } }) # Set up user-to-user permissions mapping (e.g., when transferring calls) # Assumes customer was previously unable to hear anyone (they were on hold) and agent was speaking to bot await transport.update_remote_participants({ "participant-123" : { # the customer "permissions" : { "base" : False , "canReceive" : { "byUserId" : { "agent" : True }} } }, "participant-456" : { # the agent "permissions" : { "base" : False , "canReceive" : { "byUserId" : { "customer" : True }} } } }) # Promote a participant to be a participant admin await transport.update_remote_participants({ "participant-123" : { "permissions" : { "canAdmin" : [ "participants" ] } } }) Permissions Settings: For comprehensive details on permissions settings, see the Daily API documentation: daily-python’s update_remote_participants() documentation daily-js’s updateParticipant() -> updatePermission and participants().permissions ​ Chat messages ​ send_prebuilt_chat_message async method When testing with Daily Prebuilt, you can send messages to the Prebuilt chat using this method. Parameters: message (str): The chat message to send user_name (str): The name of the user sending the message See send_prebuilt_chat_message for more details. ​ Properties ​ participant_id string Returns the transport’s participant ID in the room. ​ room_url string Returns the transport’s room URL for the room currently in use. ​ Notes Supports real-time audio/video communication Handles WebRTC connection management Provides voice activity detection Includes transcription capabilities Manages participant interactions Supports recording and streaming Thread-safe processing All callbacks are asynchronous Multiple handlers can be registered for the same event Exceptions in callbacks are logged but don’t stop the transport Supported Services SmallWebRTCTransport On this page Overview Installation Configuration Constructor Parameters DailyParams Configuration Audio Output Configuration Audio Input Configuration Video Output Configuration Voice Activity Detection (VAD) Feature Settings Basic Usage Event Callbacks Connection Events Participant Events Communication Events Dial Events Dial-in Dial-out Transcription Events Recording Events Error Events Notes on Callbacks Frame Types Input Frames Output Frames Methods Room Management Media Control Video Management Transcription Control Recording Control Dial-out Control Subscription Management Participant Admin Chat messages Properties Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/transport_daily_0c820754.txt b/transport_daily_0c820754.txt
new file mode 100644
index 0000000000000000000000000000000000000000..e00baafa971da99af2006f5de8e07cec9e2a0aa7
--- /dev/null
+++ b/transport_daily_0c820754.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/transport/daily#param-participant-counts
+Title: Daily WebRTC - Pipecat
+==================================================
+
+Daily WebRTC - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Transport Daily WebRTC Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Daily WebRTC SmallWebRTCTransport FastAPI WebSocket WebSocket Server Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview DailyTransport provides real-time audio and video communication capabilities using Daily’s WebRTC platform. It supports bidirectional audio/video streams, transcription, voice activity detection (VAD), and participant management. ​ Installation To use DailyTransport , install the required dependencies: Copy Ask AI pip install "pipecat-ai[daily]" You’ll also need to set up your Daily API key as an environment variable: DAILY_API_KEY . You can obtain a Daily API key by signing up at Daily . ​ Configuration ​ Constructor Parameters ​ room_url str required Daily room URL ​ token str | None Daily room token ​ bot_name str required Name of the bot in the room ​ params DailyParams default: "DailyParams()" Transport configuration parameters ​ DailyParams Configuration ​ api_url str default: "https://api.daily.co/v1" Daily API endpoint URL ​ api_key str default: "" Daily API key for authentication ​ Audio Output Configuration ​ audio_out_enabled bool default: "False" Enable audio output capabilities ​ audio_out_is_live bool default: "False" Enable live audio streaming mode ​ audio_out_sample_rate int default: "None" Audio output sample rate in Hz ​ audio_out_channels int default: "1" Number of audio output channels ​ audio_out_bitrate int default: "96000" Audio output bitrate in bits per second ​ Audio Input Configuration ​ audio_in_enabled bool default: "False" Enable audio input capabilities ​ audio_in_passthrough bool default: "False" When enabled, incoming audio frames are pushed downstream ​ audio_in_sample_rate int default: "None" Audio input sample rate in Hz ​ audio_in_channels int default: "1" Number of audio input channels ​ audio_in_filter Optional[BaseAudioFilter] default: "None" Audio filter for input processing. Supported filters are: KrispFilter() and NoisereduceFilter() . See the KrispFilter() reference docs and NoisereduceFilter() reference docs ` for more information. ​ Video Output Configuration ​ video_out_enabled bool default: "False" Enable video output capabilities ​ video_out_is_live bool default: "False" Enable live video streaming mode ​ video_out_width int default: "1024" Video output width in pixels ​ video_out_height int default: "768" Video output height in pixels ​ video_out_bitrate int default: "800000" Video output bitrate in bits per second ​ video_out_framerate int default: "30" Video output frame rate ​ video_out_color_format str default: "RGB" Video color format (RGB, BGR, etc.) ​ Voice Activity Detection (VAD) ​ vad_analyzer VADAnalyzer | None default: "None" Voice Activity Detection analyzer. You can set this to either SileroVADAnalyzer() or WebRTCVADAnalyzer() . SileroVADAnalyzer is the recommended option. Learn more about the SileroVADAnalyzer . ​ Feature Settings ​ dialin_settings Optional[DailyDialinSettings] default: "None" Configuration for dial-in functionality ​ transcription_enabled bool default: "False" Enable real-time transcription ​ transcription_settings DailyTranscriptionSettings Configuration for transcription features Copy Ask AI class DailyTranscriptionSettings ( BaseModel ): language: str = "en" # Default language model: str = "nova-2-general" # Transcription model profanity_filter: bool = True # Filter profanity redact: bool = False # Redact sensitive information endpointing: bool = True # Enable speech endpointing punctuate: bool = True # Add punctuation includeRawResponse: bool = True # Include raw API response extra: Mapping[ str , Any] = { "interim_results" : True # Provide any Deepgram-specific settings } ​ Basic Usage Copy Ask AI from pipecat.transports.services.daily import DailyTransport, DailyParams from pipecat.audio.vad.silero import SileroVADAnalyzer # Configure transport transport = DailyTransport( room_url = "https://your-domain.daily.co/room-name" , token = "your-room-token" , bot_name = "AI Assistant" , params = DailyParams( audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer( params = VADParams( stop_secs = 0.5 )), ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), # Handle incoming audio/video stt, # Speech-to-text llm, # Language model tts, # Text-to-speech transport.output() # Handle outgoing audio/video ]) # Use transport event handlers to manage the call lifecycle # Event handler for when the user joins the room: @transport.event_handler ( "on_first_participant_joined" ) async def on_first_participant_joined ( transport , participant ): # Start transcription for the user await transport.capture_participant_transcription(participant[ "id" ]) # Kick off the conversation. await task.queue_frames([context_aggregator.user().get_context_frame()]) # Event handler for when the user leaves the room: @transport.event_handler ( "on_participant_left" ) async def on_participant_left ( transport , participant , reason ): # Cancel the pipeline, which stops processing and removes the bot from the room await task.cancel() ​ Event Callbacks DailyTransport provides a comprehensive callback system for handling various events. Register callbacks using the @transport.event_handler() decorator. ​ Connection Events ​ on_joined async callback Called when the bot successfully joins the room. Parameters: transport : The DailyTransport instance data : Dictionary containing room join information Copy Ask AI @transport.event_handler ( "on_joined" ) async def on_joined ( transport , data : Dict[ str , Any]): logger.info( f "Joined room with data: { data } " ) ​ on_left async callback Called when the bot leaves the room. Parameters: transport : The DailyTransport instance Copy Ask AI @transport.event_handler ( "on_left" ) async def on_left ( transport ): logger.info( "Left room" ) ​ Participant Events ​ on_first_participant_joined async callback Called when the first participant joins the room. Useful for initializing conversations. Parameters: transport : The DailyTransport instance participant : Dictionary containing participant information Copy Ask AI @transport.event_handler ( "on_first_participant_joined" ) async def on_first_participant_joined ( transport , participant : Dict[ str , Any]): await transport.capture_participant_transcription(participant[ "id" ]) await task.queue_frames([LLMMessagesFrame(initial_messages)]) ​ on_participant_joined async callback Called when any participant joins the room. Parameters: transport : The DailyTransport instance participant : Dictionary containing participant information Copy Ask AI @transport.event_handler ( "on_participant_joined" ) async def on_participant_joined ( transport , participant : Dict[ str , Any]): logger.info( f "Participant joined: { participant[ 'id' ] } " ) ​ on_participant_left async callback Called when a participant leaves the room. Parameters: transport : The DailyTransport instance participant : Dictionary containing participant information reason : String describing why the participant left, “leftCall” | “hidden” Copy Ask AI @transport.event_handler ( "on_participant_left" ) async def on_participant_left ( transport , participant : Dict[ str , Any], reason : str ): logger.info( f "Participant { participant[ 'id' ] } left: { reason } " ) # Cancel the pipeline task to stop processing and remove the bot from the Daily room await task.cancel() ​ on_participant_updated async callback Event emitted when a participant is updated. This can mean either the participant’s metadata was updated, or the tracks belonging to the participant changed. Parameters: transport : The DailyTransport instance participant : Dictionary containing updated participant information Copy Ask AI @transport.event_handler ( "on_participant_updated" ) async def on_participant_updated ( transport , participant : Dict[ str , Any]): logger.info( f "Participant updated: { participant } " ) ​ Communication Events ​ on_app_message async callback Event emitted when a custom app message is received from another participant or via the REST API. Parameters: transport : The DailyTransport instance message : The message content (any type) sender : String identifier of the message sender Copy Ask AI @transport.event_handler ( "on_app_message" ) async def on_app_message ( transport , message : Any, sender : str ): logger.info( f "Message from { sender } : { message } " ) ​ on_call_state_updated async callback Event emitted when the call state changes, normally as a consequence of joining or leaving the call. Parameters: transport : The DailyTransport instance state : String representing the new call state. Learn more about call states . Copy Ask AI @transport.event_handler ( "on_call_state_updated" ) async def on_call_state_updated ( transport , state : str ): logger.info( f "Call state updated: { state } " ) ​ Dial Events ​ Dial-in ​ on_dialin_ready async callback Event emitted when dial-in is ready. This happens after the room has connected to the SIP endpoint and the system is ready to receive dial-in calls. Parameters: transport : The DailyTransport instance sip_endpoint : String containing the SIP endpoint information Copy Ask AI @transport.event_handler ( "on_dialin_ready" ) async def on_dialin_ready ( transport , sip_endpoint : str ): logger.info( f "Dial-in ready at: { sip_endpoint } " ) ​ on_dialin_connected async callback Event emitted when the session with the dial-in remote end is established (i.e. SIP endpoint or PSTN are connectd to the Daily room). Note: connected does not mean media (audio or video) has started flowing between the room and PSTN, it means the room received the connection request and both endpoints are negotiating the media flow. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. See DialinConnectedEvent . Copy Ask AI @transport.event_handler ( "on_dialin_connected" ) async def on_dialin_connected ( transport , data : Any): logger.info( f "Dial-in connected: { data } " ) ​ on_dialin_stopped async callback Event emitted when the dial-in remote end disconnects the call. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. See DialinStoppedEvent . Copy Ask AI @transport.event_handler ( "on_dialin_stopped" ) async def on_dialin_stopped ( transport , data : Any): logger.info( f "Dial-in stopped: { data } " ) ​ on_dialin_error async callback Event emitted in the case of dial-in errors which are fatal and the service cannot proceed. For example, an error in SDP negotiation is fatal to the media/SIP pipeline and will result in dialin-error being triggered. Parameters: transport : The DailyTransport instance data : Dictionary containing error information. See DialinEvent . Copy Ask AI @transport.event_handler ( "on_dialin_error" ) async def on_dialin_error ( transport , data : Any): logger.error( f "Dial-in error: { data } " ) ​ on_dialin_warning async callback Event emitted there is a dial-in non-fatal error, such as the selected codec not being used and a fallback codec being utilized. Parameters: transport : The DailyTransport instance data : Dictionary containing warning information. See DialinEvent . Copy Ask AI @transport.event_handler ( "on_dialin_warning" ) async def on_dialin_warning ( transport , data : Any): logger.warning( f "Dial-in warning: { data } " ) ​ Dial-out ​ on_dialout_answered async callback Event emitted when the session with the dial-out remote end is answered. Parameters: transport : The DailyTransport instance data : Dictionary containing call information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_answered" ) async def on_dialout_answered ( transport , data : Any): logger.info( f "Dial-out answered: { data } " ) ​ on_dialout_connected async callback Event emitted when the session with the dial-out remote end is established. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_connected" ) async def on_dialout_connected ( transport , data : Any): logger.info( f "Dial-out connected: { data } " ) ​ on_dialout_stopped async callback Event emitted when the dial-out session is stopped. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_stopped" ) async def on_dialout_stopped ( transport , data : Any): logger.info( f "Dial-out stopped: { data } " ) ​ on_dialout_error async callback Event emitted in the case of dial-out errors which are fatal and the service cannot proceed. For example, an error in SDP negotiation is fatal to the media/SIP pipeline and will result in dialout-error being triggered. Parameters: transport : The DailyTransport instance data : Dictionary containing error information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_error" ) async def on_dialout_error ( transport , data : Any): logger.error( f "Dial-out error: { data } " ) ​ on_dialout_warning async callback Event emitted there is a dial-out non-fatal error, such as the selected codec not being used and a fallback codec being utilized. Parameters: transport : The DailyTransport instance data : Dictionary containing warning information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_warning" ) async def on_dialout_warning ( transport , data : Any): logger.warning( f "Dial-out warning: { data } " ) ​ Transcription Events ​ on_transcription_message async callback Called when a transcription message is received. This includes both interim and final transcriptions. Parameters: transport : The DailyTransport instance message : Dictionary containing transcription data including. Learn more . Copy Ask AI @transport.event_handler ( "on_transcription_message" ) async def on_transcription_message ( transport , message : Dict[ str , Any]): participant_id = message.get( "participantId" ) text = message.get( "text" ) is_final = message[ "rawResponse" ][ "is_final" ] logger.info( f "Transcription from { participant_id } : { text } (final: { is_final } )" ) ​ Recording Events ​ on_recording_started async callback Called when a room recording starts successfully. Parameters: transport : The DailyTransport instance status : Dictionary containing recording status information. Learn more . Copy Ask AI @transport.event_handler ( "on_recording_started" ) async def on_recording_started ( transport , status : Dict[ str , Any]): logger.info( f "Recording started with status: { status } " ) ​ on_recording_stopped async callback Called when a room recording stops. Parameters: transport : The DailyTransport instance stream_id : String identifier of the stopped recording stream Copy Ask AI @transport.event_handler ( "on_recording_stopped" ) async def on_recording_stopped ( transport , stream_id : str ): logger.info( f "Recording stopped for stream: { stream_id } " ) ​ on_recording_error async callback Called when an error occurs during recording. Parameters: transport : The DailyTransport instance stream_id : String identifier of the recording stream message : Error message describing what went wrong Copy Ask AI @transport.event_handler ( "on_recording_error" ) async def on_recording_error ( transport , stream_id : str , message : str ): logger.error( f "Recording error for stream { stream_id } : { message } " ) ​ Error Events ​ on_error async callback Called when a transport error occurs. Parameters: transport : The DailyTransport instance error : String containing error details Copy Ask AI @transport.event_handler ( "on_error" ) async def on_error ( transport , error : str ): logger.error( f "Transport error: { error } " ) ​ Notes on Callbacks All callbacks are asynchronous Callbacks are executed in the order they were registered Multiple handlers can be registered for the same event Exceptions in callbacks are logged but don’t stop the transport Callbacks should be lightweight to avoid blocking the event loop Heavy processing should be offloaded to separate tasks ​ Frame Types ​ Input Frames ​ InputAudioRawFrame Frame Raw audio data from participants ​ UserImageRawFrame Frame Video frames from participants ​ Output Frames ​ OutputAudioRawFrame Frame Audio data to be sent ​ OutputImageRawFrame Frame Video frames to be sent ​ Methods ​ Room Management ​ participants method Returns a dictionary of all participants currently in the room. Copy Ask AI def participants () -> Dict[ str , Any] See participants for more details. ​ participant_counts method Returns participant count statistics for the room. Copy Ask AI def participant_counts () -> Dict[ str , int ] See participant_counts for more details. ​ send_message async method Sends a message to participants in the Daily room. Messages can be directed to all participants or targeted to a specific participant. Copy Ask AI async def send_message ( frame : TransportMessageFrame | TransportMessageUrgentFrame) Parameters: frame : A transport message frame containing the message to send. Can be either: TransportMessageFrame : Standard message TransportMessageUrgentFrame : High-priority message Message Structure: Messages should conform to the expected format for the receiving system. For RTVI messages, use: Copy Ask AI { "label" : "rtvi-ai" , # Required for RTVI protocol "type" : "message-type" , # Defines how the client will process it "id" : "unique-id" , # Unique identifier for the message "data" : { # Custom payload # Your custom data fields } } Example - Standard Message: Copy Ask AI import uuid # Send a standard message to all participants message_data = { "label" : "rtvi-ai" , "type" : "custom-message" , "id" : str (uuid.uuid4()), "data" : { "message" : "Hello everyone!" , "timestamp" : datetime.datetime.now().isoformat() } } await transport.send_message(TransportMessageFrame( message = message_data)) Example - Urgent Message to Specific Participant: Copy Ask AI # Send urgent message to specific participant urgent_data = { "label" : "rtvi-ai" , "type" : "server-message" , "id" : str (uuid.uuid4()), "data" : { "message" : "Private message" , "importance" : "high" } } await transport.send_message( DailyTransportMessageUrgentFrame( message = urgent_data, participant_id = "participant-123" ) ) The message will be delivered to: All participants if no specific participant_id is set Only the specified participant if participant_id is set Through the Daily app messaging system Can be received by registering an on_app_message handler Use TransportMessageUrgentFrame for high-priority messages that need immediate delivery. ​ Media Control ​ send_image async method Sends an image frame to the room. Copy Ask AI async def send_image ( frame : OutputImageRawFrame | SpriteFrame) -> None Parameters: frame : Image frame to send, either raw image data or sprite animation ​ send_audio async method Sends an audio frame to the room. Copy Ask AI async def send_audio ( frame : OutputAudioRawFrame) -> None Parameters: frame : Audio frame to send ​ Video Management ​ capture_participant_video async method Starts capturing video from a specific participant. Copy Ask AI async def capture_participant_video ( participant_id : str , framerate : int = 30 , video_source : str = "camera" , color_format : str = "RGB" ) -> None Parameters: participant_id : ID of the participant to capture framerate : Target frame rate (default: 30) video_source : Video source type (default: “camera”) color_format : Color format of the video (default: “RGB”) To request an image from the camera stream, set the framerate to 0 and push a UserImageRequestFrame . ​ Transcription Control ​ capture_participant_transcription async method Starts capturing and transcribing audio from a specific participant. Copy Ask AI async def capture_participant_transcription ( participant_id : str ) -> None Parameters: participant_id : ID of the participant to transcribe ​ update_transcription async method Updates the transcription configuration for specific participants or instances. Copy Ask AI async def update_transcription ( participants : List[ str ] | None = None , instance_id : str | None = None ) -> None Parameters: participants : Optional list of participant IDs to transcribe. If None, transcribes all participants. instance_id : Optional specific transcription instance ID to update Example: Copy Ask AI # Update transcription for specific participants await transport.update_transcription( participants = [ "participant-123" , "participant-456" ]) # Update specific transcription instance await transport.update_transcription( instance_id = "transcription-789" ) ​ Recording Control ​ start_recording async method Starts recording the room session. Copy Ask AI async def start_recording ( streaming_settings : Dict = None , stream_id : str = None , force_new : bool = None ) -> None Parameters: streaming_settings : Recording configuration settings stream_id : Optional stream identifier force_new : Force start a new recording See start_recording for more details. ​ stop_recording async method Stops an active recording. Copy Ask AI async def stop_recording ( stream_id : str = None ) -> None Parameters: stream_id : Optional stream identifier to stop specific recording See stop_recording for more details. ​ Dial-out Control ​ start_dialout async method Initiates a dial-out call. Copy Ask AI async def start_dialout ( settings : Dict = None ) -> None Parameters: settings : Dial-out configuration settings See start_dialout for more details. ​ stop_dialout async method Stops an active dial-out call. Copy Ask AI async def stop_dialout ( participant_id : str ) -> None Parameters: participant_id : ID of the participant to disconnect See stop_dialout for more details. ​ send_dtmf async method Sends DTMF tones in an existing dial-out session. Copy Ask AI async def send_dtmf ( settings : Dict = None ) -> None Parameters: settings : DTMF settings See send_dtmf for more details. ​ Subscription Management ​ update_subscriptions async method Updates subscriptions and subscription profiles. This function allows you to update subscription profiles and at the same time assign specific subscription profiles to a participant and even change specific settings for some participants. Copy Ask AI async def update_subscriptions ( participant_settings : Dict = None , profile_settings : Dict = None ) -> None Parameters: participant_settings : Per-participant subscription settings profile_settings : Global subscription profile settings See update_subscriptions for more details. ​ Participant Admin When using the DailyTransport, if your bot is given a meeting token with owner or participant admin permissions, it can remote-control other participants’ capabilities and permissions. With this control, the bot can modify settings for how others send and receive media, how they are seen in a session, and whether they have administrative controls. ​ update_remote_participants async method Updates permissions and settings for specific participants in the room. Copy Ask AI async def update_remote_participants ( remote_participants : Mapping[ str , Any] = None ) -> None Parameters: remote_participants : Dictionary mapping participant IDs to their updated settings This method allows fine-grained control over participant capabilities and permissions in the Daily room. Examples: Copy Ask AI # Mute a participant's microphone await transport.update_remote_participants({ "participant-123" : { "inputsEnabled" : { "microphone" : False } } }) # Revoke send permissions (mutes the participant and prevents them from unmuting) await transport.update_remote_participants({ "participant-123" : { "permissions" : { "canSend" : [] # Empty list revokes all send permissions } } }) # Configure selective audio routing await transport.update_remote_participants({ "participant-123" : { # the customer "permissions" : { "canReceive" : { "base" : False , # Don't receive from anyone by default "byUserId" : { "hold-music" : True # Only receive from hold music player } } } } }) # Set up user-to-user permissions mapping (e.g., when transferring calls) # Assumes customer was previously unable to hear anyone (they were on hold) and agent was speaking to bot await transport.update_remote_participants({ "participant-123" : { # the customer "permissions" : { "base" : False , "canReceive" : { "byUserId" : { "agent" : True }} } }, "participant-456" : { # the agent "permissions" : { "base" : False , "canReceive" : { "byUserId" : { "customer" : True }} } } }) # Promote a participant to be a participant admin await transport.update_remote_participants({ "participant-123" : { "permissions" : { "canAdmin" : [ "participants" ] } } }) Permissions Settings: For comprehensive details on permissions settings, see the Daily API documentation: daily-python’s update_remote_participants() documentation daily-js’s updateParticipant() -> updatePermission and participants().permissions ​ Chat messages ​ send_prebuilt_chat_message async method When testing with Daily Prebuilt, you can send messages to the Prebuilt chat using this method. Parameters: message (str): The chat message to send user_name (str): The name of the user sending the message See send_prebuilt_chat_message for more details. ​ Properties ​ participant_id string Returns the transport’s participant ID in the room. ​ room_url string Returns the transport’s room URL for the room currently in use. ​ Notes Supports real-time audio/video communication Handles WebRTC connection management Provides voice activity detection Includes transcription capabilities Manages participant interactions Supports recording and streaming Thread-safe processing All callbacks are asynchronous Multiple handlers can be registered for the same event Exceptions in callbacks are logged but don’t stop the transport Supported Services SmallWebRTCTransport On this page Overview Installation Configuration Constructor Parameters DailyParams Configuration Audio Output Configuration Audio Input Configuration Video Output Configuration Voice Activity Detection (VAD) Feature Settings Basic Usage Event Callbacks Connection Events Participant Events Communication Events Dial Events Dial-in Dial-out Transcription Events Recording Events Error Events Notes on Callbacks Frame Types Input Frames Output Frames Methods Room Management Media Control Video Management Transcription Control Recording Control Dial-out Control Subscription Management Participant Admin Chat messages Properties Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/transport_daily_156d1549.txt b/transport_daily_156d1549.txt
new file mode 100644
index 0000000000000000000000000000000000000000..0195c14462f230092ec539bfa4b236ddca12e2bc
--- /dev/null
+++ b/transport_daily_156d1549.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/transport/daily#param-participants
+Title: Daily WebRTC - Pipecat
+==================================================
+
+Daily WebRTC - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Transport Daily WebRTC Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Daily WebRTC SmallWebRTCTransport FastAPI WebSocket WebSocket Server Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview DailyTransport provides real-time audio and video communication capabilities using Daily’s WebRTC platform. It supports bidirectional audio/video streams, transcription, voice activity detection (VAD), and participant management. ​ Installation To use DailyTransport , install the required dependencies: Copy Ask AI pip install "pipecat-ai[daily]" You’ll also need to set up your Daily API key as an environment variable: DAILY_API_KEY . You can obtain a Daily API key by signing up at Daily . ​ Configuration ​ Constructor Parameters ​ room_url str required Daily room URL ​ token str | None Daily room token ​ bot_name str required Name of the bot in the room ​ params DailyParams default: "DailyParams()" Transport configuration parameters ​ DailyParams Configuration ​ api_url str default: "https://api.daily.co/v1" Daily API endpoint URL ​ api_key str default: "" Daily API key for authentication ​ Audio Output Configuration ​ audio_out_enabled bool default: "False" Enable audio output capabilities ​ audio_out_is_live bool default: "False" Enable live audio streaming mode ​ audio_out_sample_rate int default: "None" Audio output sample rate in Hz ​ audio_out_channels int default: "1" Number of audio output channels ​ audio_out_bitrate int default: "96000" Audio output bitrate in bits per second ​ Audio Input Configuration ​ audio_in_enabled bool default: "False" Enable audio input capabilities ​ audio_in_passthrough bool default: "False" When enabled, incoming audio frames are pushed downstream ​ audio_in_sample_rate int default: "None" Audio input sample rate in Hz ​ audio_in_channels int default: "1" Number of audio input channels ​ audio_in_filter Optional[BaseAudioFilter] default: "None" Audio filter for input processing. Supported filters are: KrispFilter() and NoisereduceFilter() . See the KrispFilter() reference docs and NoisereduceFilter() reference docs ` for more information. ​ Video Output Configuration ​ video_out_enabled bool default: "False" Enable video output capabilities ​ video_out_is_live bool default: "False" Enable live video streaming mode ​ video_out_width int default: "1024" Video output width in pixels ​ video_out_height int default: "768" Video output height in pixels ​ video_out_bitrate int default: "800000" Video output bitrate in bits per second ​ video_out_framerate int default: "30" Video output frame rate ​ video_out_color_format str default: "RGB" Video color format (RGB, BGR, etc.) ​ Voice Activity Detection (VAD) ​ vad_analyzer VADAnalyzer | None default: "None" Voice Activity Detection analyzer. You can set this to either SileroVADAnalyzer() or WebRTCVADAnalyzer() . SileroVADAnalyzer is the recommended option. Learn more about the SileroVADAnalyzer . ​ Feature Settings ​ dialin_settings Optional[DailyDialinSettings] default: "None" Configuration for dial-in functionality ​ transcription_enabled bool default: "False" Enable real-time transcription ​ transcription_settings DailyTranscriptionSettings Configuration for transcription features Copy Ask AI class DailyTranscriptionSettings ( BaseModel ): language: str = "en" # Default language model: str = "nova-2-general" # Transcription model profanity_filter: bool = True # Filter profanity redact: bool = False # Redact sensitive information endpointing: bool = True # Enable speech endpointing punctuate: bool = True # Add punctuation includeRawResponse: bool = True # Include raw API response extra: Mapping[ str , Any] = { "interim_results" : True # Provide any Deepgram-specific settings } ​ Basic Usage Copy Ask AI from pipecat.transports.services.daily import DailyTransport, DailyParams from pipecat.audio.vad.silero import SileroVADAnalyzer # Configure transport transport = DailyTransport( room_url = "https://your-domain.daily.co/room-name" , token = "your-room-token" , bot_name = "AI Assistant" , params = DailyParams( audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer( params = VADParams( stop_secs = 0.5 )), ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), # Handle incoming audio/video stt, # Speech-to-text llm, # Language model tts, # Text-to-speech transport.output() # Handle outgoing audio/video ]) # Use transport event handlers to manage the call lifecycle # Event handler for when the user joins the room: @transport.event_handler ( "on_first_participant_joined" ) async def on_first_participant_joined ( transport , participant ): # Start transcription for the user await transport.capture_participant_transcription(participant[ "id" ]) # Kick off the conversation. await task.queue_frames([context_aggregator.user().get_context_frame()]) # Event handler for when the user leaves the room: @transport.event_handler ( "on_participant_left" ) async def on_participant_left ( transport , participant , reason ): # Cancel the pipeline, which stops processing and removes the bot from the room await task.cancel() ​ Event Callbacks DailyTransport provides a comprehensive callback system for handling various events. Register callbacks using the @transport.event_handler() decorator. ​ Connection Events ​ on_joined async callback Called when the bot successfully joins the room. Parameters: transport : The DailyTransport instance data : Dictionary containing room join information Copy Ask AI @transport.event_handler ( "on_joined" ) async def on_joined ( transport , data : Dict[ str , Any]): logger.info( f "Joined room with data: { data } " ) ​ on_left async callback Called when the bot leaves the room. Parameters: transport : The DailyTransport instance Copy Ask AI @transport.event_handler ( "on_left" ) async def on_left ( transport ): logger.info( "Left room" ) ​ Participant Events ​ on_first_participant_joined async callback Called when the first participant joins the room. Useful for initializing conversations. Parameters: transport : The DailyTransport instance participant : Dictionary containing participant information Copy Ask AI @transport.event_handler ( "on_first_participant_joined" ) async def on_first_participant_joined ( transport , participant : Dict[ str , Any]): await transport.capture_participant_transcription(participant[ "id" ]) await task.queue_frames([LLMMessagesFrame(initial_messages)]) ​ on_participant_joined async callback Called when any participant joins the room. Parameters: transport : The DailyTransport instance participant : Dictionary containing participant information Copy Ask AI @transport.event_handler ( "on_participant_joined" ) async def on_participant_joined ( transport , participant : Dict[ str , Any]): logger.info( f "Participant joined: { participant[ 'id' ] } " ) ​ on_participant_left async callback Called when a participant leaves the room. Parameters: transport : The DailyTransport instance participant : Dictionary containing participant information reason : String describing why the participant left, “leftCall” | “hidden” Copy Ask AI @transport.event_handler ( "on_participant_left" ) async def on_participant_left ( transport , participant : Dict[ str , Any], reason : str ): logger.info( f "Participant { participant[ 'id' ] } left: { reason } " ) # Cancel the pipeline task to stop processing and remove the bot from the Daily room await task.cancel() ​ on_participant_updated async callback Event emitted when a participant is updated. This can mean either the participant’s metadata was updated, or the tracks belonging to the participant changed. Parameters: transport : The DailyTransport instance participant : Dictionary containing updated participant information Copy Ask AI @transport.event_handler ( "on_participant_updated" ) async def on_participant_updated ( transport , participant : Dict[ str , Any]): logger.info( f "Participant updated: { participant } " ) ​ Communication Events ​ on_app_message async callback Event emitted when a custom app message is received from another participant or via the REST API. Parameters: transport : The DailyTransport instance message : The message content (any type) sender : String identifier of the message sender Copy Ask AI @transport.event_handler ( "on_app_message" ) async def on_app_message ( transport , message : Any, sender : str ): logger.info( f "Message from { sender } : { message } " ) ​ on_call_state_updated async callback Event emitted when the call state changes, normally as a consequence of joining or leaving the call. Parameters: transport : The DailyTransport instance state : String representing the new call state. Learn more about call states . Copy Ask AI @transport.event_handler ( "on_call_state_updated" ) async def on_call_state_updated ( transport , state : str ): logger.info( f "Call state updated: { state } " ) ​ Dial Events ​ Dial-in ​ on_dialin_ready async callback Event emitted when dial-in is ready. This happens after the room has connected to the SIP endpoint and the system is ready to receive dial-in calls. Parameters: transport : The DailyTransport instance sip_endpoint : String containing the SIP endpoint information Copy Ask AI @transport.event_handler ( "on_dialin_ready" ) async def on_dialin_ready ( transport , sip_endpoint : str ): logger.info( f "Dial-in ready at: { sip_endpoint } " ) ​ on_dialin_connected async callback Event emitted when the session with the dial-in remote end is established (i.e. SIP endpoint or PSTN are connectd to the Daily room). Note: connected does not mean media (audio or video) has started flowing between the room and PSTN, it means the room received the connection request and both endpoints are negotiating the media flow. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. See DialinConnectedEvent . Copy Ask AI @transport.event_handler ( "on_dialin_connected" ) async def on_dialin_connected ( transport , data : Any): logger.info( f "Dial-in connected: { data } " ) ​ on_dialin_stopped async callback Event emitted when the dial-in remote end disconnects the call. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. See DialinStoppedEvent . Copy Ask AI @transport.event_handler ( "on_dialin_stopped" ) async def on_dialin_stopped ( transport , data : Any): logger.info( f "Dial-in stopped: { data } " ) ​ on_dialin_error async callback Event emitted in the case of dial-in errors which are fatal and the service cannot proceed. For example, an error in SDP negotiation is fatal to the media/SIP pipeline and will result in dialin-error being triggered. Parameters: transport : The DailyTransport instance data : Dictionary containing error information. See DialinEvent . Copy Ask AI @transport.event_handler ( "on_dialin_error" ) async def on_dialin_error ( transport , data : Any): logger.error( f "Dial-in error: { data } " ) ​ on_dialin_warning async callback Event emitted there is a dial-in non-fatal error, such as the selected codec not being used and a fallback codec being utilized. Parameters: transport : The DailyTransport instance data : Dictionary containing warning information. See DialinEvent . Copy Ask AI @transport.event_handler ( "on_dialin_warning" ) async def on_dialin_warning ( transport , data : Any): logger.warning( f "Dial-in warning: { data } " ) ​ Dial-out ​ on_dialout_answered async callback Event emitted when the session with the dial-out remote end is answered. Parameters: transport : The DailyTransport instance data : Dictionary containing call information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_answered" ) async def on_dialout_answered ( transport , data : Any): logger.info( f "Dial-out answered: { data } " ) ​ on_dialout_connected async callback Event emitted when the session with the dial-out remote end is established. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_connected" ) async def on_dialout_connected ( transport , data : Any): logger.info( f "Dial-out connected: { data } " ) ​ on_dialout_stopped async callback Event emitted when the dial-out session is stopped. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_stopped" ) async def on_dialout_stopped ( transport , data : Any): logger.info( f "Dial-out stopped: { data } " ) ​ on_dialout_error async callback Event emitted in the case of dial-out errors which are fatal and the service cannot proceed. For example, an error in SDP negotiation is fatal to the media/SIP pipeline and will result in dialout-error being triggered. Parameters: transport : The DailyTransport instance data : Dictionary containing error information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_error" ) async def on_dialout_error ( transport , data : Any): logger.error( f "Dial-out error: { data } " ) ​ on_dialout_warning async callback Event emitted there is a dial-out non-fatal error, such as the selected codec not being used and a fallback codec being utilized. Parameters: transport : The DailyTransport instance data : Dictionary containing warning information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_warning" ) async def on_dialout_warning ( transport , data : Any): logger.warning( f "Dial-out warning: { data } " ) ​ Transcription Events ​ on_transcription_message async callback Called when a transcription message is received. This includes both interim and final transcriptions. Parameters: transport : The DailyTransport instance message : Dictionary containing transcription data including. Learn more . Copy Ask AI @transport.event_handler ( "on_transcription_message" ) async def on_transcription_message ( transport , message : Dict[ str , Any]): participant_id = message.get( "participantId" ) text = message.get( "text" ) is_final = message[ "rawResponse" ][ "is_final" ] logger.info( f "Transcription from { participant_id } : { text } (final: { is_final } )" ) ​ Recording Events ​ on_recording_started async callback Called when a room recording starts successfully. Parameters: transport : The DailyTransport instance status : Dictionary containing recording status information. Learn more . Copy Ask AI @transport.event_handler ( "on_recording_started" ) async def on_recording_started ( transport , status : Dict[ str , Any]): logger.info( f "Recording started with status: { status } " ) ​ on_recording_stopped async callback Called when a room recording stops. Parameters: transport : The DailyTransport instance stream_id : String identifier of the stopped recording stream Copy Ask AI @transport.event_handler ( "on_recording_stopped" ) async def on_recording_stopped ( transport , stream_id : str ): logger.info( f "Recording stopped for stream: { stream_id } " ) ​ on_recording_error async callback Called when an error occurs during recording. Parameters: transport : The DailyTransport instance stream_id : String identifier of the recording stream message : Error message describing what went wrong Copy Ask AI @transport.event_handler ( "on_recording_error" ) async def on_recording_error ( transport , stream_id : str , message : str ): logger.error( f "Recording error for stream { stream_id } : { message } " ) ​ Error Events ​ on_error async callback Called when a transport error occurs. Parameters: transport : The DailyTransport instance error : String containing error details Copy Ask AI @transport.event_handler ( "on_error" ) async def on_error ( transport , error : str ): logger.error( f "Transport error: { error } " ) ​ Notes on Callbacks All callbacks are asynchronous Callbacks are executed in the order they were registered Multiple handlers can be registered for the same event Exceptions in callbacks are logged but don’t stop the transport Callbacks should be lightweight to avoid blocking the event loop Heavy processing should be offloaded to separate tasks ​ Frame Types ​ Input Frames ​ InputAudioRawFrame Frame Raw audio data from participants ​ UserImageRawFrame Frame Video frames from participants ​ Output Frames ​ OutputAudioRawFrame Frame Audio data to be sent ​ OutputImageRawFrame Frame Video frames to be sent ​ Methods ​ Room Management ​ participants method Returns a dictionary of all participants currently in the room. Copy Ask AI def participants () -> Dict[ str , Any] See participants for more details. ​ participant_counts method Returns participant count statistics for the room. Copy Ask AI def participant_counts () -> Dict[ str , int ] See participant_counts for more details. ​ send_message async method Sends a message to participants in the Daily room. Messages can be directed to all participants or targeted to a specific participant. Copy Ask AI async def send_message ( frame : TransportMessageFrame | TransportMessageUrgentFrame) Parameters: frame : A transport message frame containing the message to send. Can be either: TransportMessageFrame : Standard message TransportMessageUrgentFrame : High-priority message Message Structure: Messages should conform to the expected format for the receiving system. For RTVI messages, use: Copy Ask AI { "label" : "rtvi-ai" , # Required for RTVI protocol "type" : "message-type" , # Defines how the client will process it "id" : "unique-id" , # Unique identifier for the message "data" : { # Custom payload # Your custom data fields } } Example - Standard Message: Copy Ask AI import uuid # Send a standard message to all participants message_data = { "label" : "rtvi-ai" , "type" : "custom-message" , "id" : str (uuid.uuid4()), "data" : { "message" : "Hello everyone!" , "timestamp" : datetime.datetime.now().isoformat() } } await transport.send_message(TransportMessageFrame( message = message_data)) Example - Urgent Message to Specific Participant: Copy Ask AI # Send urgent message to specific participant urgent_data = { "label" : "rtvi-ai" , "type" : "server-message" , "id" : str (uuid.uuid4()), "data" : { "message" : "Private message" , "importance" : "high" } } await transport.send_message( DailyTransportMessageUrgentFrame( message = urgent_data, participant_id = "participant-123" ) ) The message will be delivered to: All participants if no specific participant_id is set Only the specified participant if participant_id is set Through the Daily app messaging system Can be received by registering an on_app_message handler Use TransportMessageUrgentFrame for high-priority messages that need immediate delivery. ​ Media Control ​ send_image async method Sends an image frame to the room. Copy Ask AI async def send_image ( frame : OutputImageRawFrame | SpriteFrame) -> None Parameters: frame : Image frame to send, either raw image data or sprite animation ​ send_audio async method Sends an audio frame to the room. Copy Ask AI async def send_audio ( frame : OutputAudioRawFrame) -> None Parameters: frame : Audio frame to send ​ Video Management ​ capture_participant_video async method Starts capturing video from a specific participant. Copy Ask AI async def capture_participant_video ( participant_id : str , framerate : int = 30 , video_source : str = "camera" , color_format : str = "RGB" ) -> None Parameters: participant_id : ID of the participant to capture framerate : Target frame rate (default: 30) video_source : Video source type (default: “camera”) color_format : Color format of the video (default: “RGB”) To request an image from the camera stream, set the framerate to 0 and push a UserImageRequestFrame . ​ Transcription Control ​ capture_participant_transcription async method Starts capturing and transcribing audio from a specific participant. Copy Ask AI async def capture_participant_transcription ( participant_id : str ) -> None Parameters: participant_id : ID of the participant to transcribe ​ update_transcription async method Updates the transcription configuration for specific participants or instances. Copy Ask AI async def update_transcription ( participants : List[ str ] | None = None , instance_id : str | None = None ) -> None Parameters: participants : Optional list of participant IDs to transcribe. If None, transcribes all participants. instance_id : Optional specific transcription instance ID to update Example: Copy Ask AI # Update transcription for specific participants await transport.update_transcription( participants = [ "participant-123" , "participant-456" ]) # Update specific transcription instance await transport.update_transcription( instance_id = "transcription-789" ) ​ Recording Control ​ start_recording async method Starts recording the room session. Copy Ask AI async def start_recording ( streaming_settings : Dict = None , stream_id : str = None , force_new : bool = None ) -> None Parameters: streaming_settings : Recording configuration settings stream_id : Optional stream identifier force_new : Force start a new recording See start_recording for more details. ​ stop_recording async method Stops an active recording. Copy Ask AI async def stop_recording ( stream_id : str = None ) -> None Parameters: stream_id : Optional stream identifier to stop specific recording See stop_recording for more details. ​ Dial-out Control ​ start_dialout async method Initiates a dial-out call. Copy Ask AI async def start_dialout ( settings : Dict = None ) -> None Parameters: settings : Dial-out configuration settings See start_dialout for more details. ​ stop_dialout async method Stops an active dial-out call. Copy Ask AI async def stop_dialout ( participant_id : str ) -> None Parameters: participant_id : ID of the participant to disconnect See stop_dialout for more details. ​ send_dtmf async method Sends DTMF tones in an existing dial-out session. Copy Ask AI async def send_dtmf ( settings : Dict = None ) -> None Parameters: settings : DTMF settings See send_dtmf for more details. ​ Subscription Management ​ update_subscriptions async method Updates subscriptions and subscription profiles. This function allows you to update subscription profiles and at the same time assign specific subscription profiles to a participant and even change specific settings for some participants. Copy Ask AI async def update_subscriptions ( participant_settings : Dict = None , profile_settings : Dict = None ) -> None Parameters: participant_settings : Per-participant subscription settings profile_settings : Global subscription profile settings See update_subscriptions for more details. ​ Participant Admin When using the DailyTransport, if your bot is given a meeting token with owner or participant admin permissions, it can remote-control other participants’ capabilities and permissions. With this control, the bot can modify settings for how others send and receive media, how they are seen in a session, and whether they have administrative controls. ​ update_remote_participants async method Updates permissions and settings for specific participants in the room. Copy Ask AI async def update_remote_participants ( remote_participants : Mapping[ str , Any] = None ) -> None Parameters: remote_participants : Dictionary mapping participant IDs to their updated settings This method allows fine-grained control over participant capabilities and permissions in the Daily room. Examples: Copy Ask AI # Mute a participant's microphone await transport.update_remote_participants({ "participant-123" : { "inputsEnabled" : { "microphone" : False } } }) # Revoke send permissions (mutes the participant and prevents them from unmuting) await transport.update_remote_participants({ "participant-123" : { "permissions" : { "canSend" : [] # Empty list revokes all send permissions } } }) # Configure selective audio routing await transport.update_remote_participants({ "participant-123" : { # the customer "permissions" : { "canReceive" : { "base" : False , # Don't receive from anyone by default "byUserId" : { "hold-music" : True # Only receive from hold music player } } } } }) # Set up user-to-user permissions mapping (e.g., when transferring calls) # Assumes customer was previously unable to hear anyone (they were on hold) and agent was speaking to bot await transport.update_remote_participants({ "participant-123" : { # the customer "permissions" : { "base" : False , "canReceive" : { "byUserId" : { "agent" : True }} } }, "participant-456" : { # the agent "permissions" : { "base" : False , "canReceive" : { "byUserId" : { "customer" : True }} } } }) # Promote a participant to be a participant admin await transport.update_remote_participants({ "participant-123" : { "permissions" : { "canAdmin" : [ "participants" ] } } }) Permissions Settings: For comprehensive details on permissions settings, see the Daily API documentation: daily-python’s update_remote_participants() documentation daily-js’s updateParticipant() -> updatePermission and participants().permissions ​ Chat messages ​ send_prebuilt_chat_message async method When testing with Daily Prebuilt, you can send messages to the Prebuilt chat using this method. Parameters: message (str): The chat message to send user_name (str): The name of the user sending the message See send_prebuilt_chat_message for more details. ​ Properties ​ participant_id string Returns the transport’s participant ID in the room. ​ room_url string Returns the transport’s room URL for the room currently in use. ​ Notes Supports real-time audio/video communication Handles WebRTC connection management Provides voice activity detection Includes transcription capabilities Manages participant interactions Supports recording and streaming Thread-safe processing All callbacks are asynchronous Multiple handlers can be registered for the same event Exceptions in callbacks are logged but don’t stop the transport Supported Services SmallWebRTCTransport On this page Overview Installation Configuration Constructor Parameters DailyParams Configuration Audio Output Configuration Audio Input Configuration Video Output Configuration Voice Activity Detection (VAD) Feature Settings Basic Usage Event Callbacks Connection Events Participant Events Communication Events Dial Events Dial-in Dial-out Transcription Events Recording Events Error Events Notes on Callbacks Frame Types Input Frames Output Frames Methods Room Management Media Control Video Management Transcription Control Recording Control Dial-out Control Subscription Management Participant Admin Chat messages Properties Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/transport_daily_18b88004.txt b/transport_daily_18b88004.txt
new file mode 100644
index 0000000000000000000000000000000000000000..d12e81bdfd35e84edd4e77e254759164e3f567c4
--- /dev/null
+++ b/transport_daily_18b88004.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/transport/daily#param-on-call-state-updated
+Title: Daily WebRTC - Pipecat
+==================================================
+
+Daily WebRTC - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Transport Daily WebRTC Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Daily WebRTC SmallWebRTCTransport FastAPI WebSocket WebSocket Server Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview DailyTransport provides real-time audio and video communication capabilities using Daily’s WebRTC platform. It supports bidirectional audio/video streams, transcription, voice activity detection (VAD), and participant management. ​ Installation To use DailyTransport , install the required dependencies: Copy Ask AI pip install "pipecat-ai[daily]" You’ll also need to set up your Daily API key as an environment variable: DAILY_API_KEY . You can obtain a Daily API key by signing up at Daily . ​ Configuration ​ Constructor Parameters ​ room_url str required Daily room URL ​ token str | None Daily room token ​ bot_name str required Name of the bot in the room ​ params DailyParams default: "DailyParams()" Transport configuration parameters ​ DailyParams Configuration ​ api_url str default: "https://api.daily.co/v1" Daily API endpoint URL ​ api_key str default: "" Daily API key for authentication ​ Audio Output Configuration ​ audio_out_enabled bool default: "False" Enable audio output capabilities ​ audio_out_is_live bool default: "False" Enable live audio streaming mode ​ audio_out_sample_rate int default: "None" Audio output sample rate in Hz ​ audio_out_channels int default: "1" Number of audio output channels ​ audio_out_bitrate int default: "96000" Audio output bitrate in bits per second ​ Audio Input Configuration ​ audio_in_enabled bool default: "False" Enable audio input capabilities ​ audio_in_passthrough bool default: "False" When enabled, incoming audio frames are pushed downstream ​ audio_in_sample_rate int default: "None" Audio input sample rate in Hz ​ audio_in_channels int default: "1" Number of audio input channels ​ audio_in_filter Optional[BaseAudioFilter] default: "None" Audio filter for input processing. Supported filters are: KrispFilter() and NoisereduceFilter() . See the KrispFilter() reference docs and NoisereduceFilter() reference docs ` for more information. ​ Video Output Configuration ​ video_out_enabled bool default: "False" Enable video output capabilities ​ video_out_is_live bool default: "False" Enable live video streaming mode ​ video_out_width int default: "1024" Video output width in pixels ​ video_out_height int default: "768" Video output height in pixels ​ video_out_bitrate int default: "800000" Video output bitrate in bits per second ​ video_out_framerate int default: "30" Video output frame rate ​ video_out_color_format str default: "RGB" Video color format (RGB, BGR, etc.) ​ Voice Activity Detection (VAD) ​ vad_analyzer VADAnalyzer | None default: "None" Voice Activity Detection analyzer. You can set this to either SileroVADAnalyzer() or WebRTCVADAnalyzer() . SileroVADAnalyzer is the recommended option. Learn more about the SileroVADAnalyzer . ​ Feature Settings ​ dialin_settings Optional[DailyDialinSettings] default: "None" Configuration for dial-in functionality ​ transcription_enabled bool default: "False" Enable real-time transcription ​ transcription_settings DailyTranscriptionSettings Configuration for transcription features Copy Ask AI class DailyTranscriptionSettings ( BaseModel ): language: str = "en" # Default language model: str = "nova-2-general" # Transcription model profanity_filter: bool = True # Filter profanity redact: bool = False # Redact sensitive information endpointing: bool = True # Enable speech endpointing punctuate: bool = True # Add punctuation includeRawResponse: bool = True # Include raw API response extra: Mapping[ str , Any] = { "interim_results" : True # Provide any Deepgram-specific settings } ​ Basic Usage Copy Ask AI from pipecat.transports.services.daily import DailyTransport, DailyParams from pipecat.audio.vad.silero import SileroVADAnalyzer # Configure transport transport = DailyTransport( room_url = "https://your-domain.daily.co/room-name" , token = "your-room-token" , bot_name = "AI Assistant" , params = DailyParams( audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer( params = VADParams( stop_secs = 0.5 )), ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), # Handle incoming audio/video stt, # Speech-to-text llm, # Language model tts, # Text-to-speech transport.output() # Handle outgoing audio/video ]) # Use transport event handlers to manage the call lifecycle # Event handler for when the user joins the room: @transport.event_handler ( "on_first_participant_joined" ) async def on_first_participant_joined ( transport , participant ): # Start transcription for the user await transport.capture_participant_transcription(participant[ "id" ]) # Kick off the conversation. await task.queue_frames([context_aggregator.user().get_context_frame()]) # Event handler for when the user leaves the room: @transport.event_handler ( "on_participant_left" ) async def on_participant_left ( transport , participant , reason ): # Cancel the pipeline, which stops processing and removes the bot from the room await task.cancel() ​ Event Callbacks DailyTransport provides a comprehensive callback system for handling various events. Register callbacks using the @transport.event_handler() decorator. ​ Connection Events ​ on_joined async callback Called when the bot successfully joins the room. Parameters: transport : The DailyTransport instance data : Dictionary containing room join information Copy Ask AI @transport.event_handler ( "on_joined" ) async def on_joined ( transport , data : Dict[ str , Any]): logger.info( f "Joined room with data: { data } " ) ​ on_left async callback Called when the bot leaves the room. Parameters: transport : The DailyTransport instance Copy Ask AI @transport.event_handler ( "on_left" ) async def on_left ( transport ): logger.info( "Left room" ) ​ Participant Events ​ on_first_participant_joined async callback Called when the first participant joins the room. Useful for initializing conversations. Parameters: transport : The DailyTransport instance participant : Dictionary containing participant information Copy Ask AI @transport.event_handler ( "on_first_participant_joined" ) async def on_first_participant_joined ( transport , participant : Dict[ str , Any]): await transport.capture_participant_transcription(participant[ "id" ]) await task.queue_frames([LLMMessagesFrame(initial_messages)]) ​ on_participant_joined async callback Called when any participant joins the room. Parameters: transport : The DailyTransport instance participant : Dictionary containing participant information Copy Ask AI @transport.event_handler ( "on_participant_joined" ) async def on_participant_joined ( transport , participant : Dict[ str , Any]): logger.info( f "Participant joined: { participant[ 'id' ] } " ) ​ on_participant_left async callback Called when a participant leaves the room. Parameters: transport : The DailyTransport instance participant : Dictionary containing participant information reason : String describing why the participant left, “leftCall” | “hidden” Copy Ask AI @transport.event_handler ( "on_participant_left" ) async def on_participant_left ( transport , participant : Dict[ str , Any], reason : str ): logger.info( f "Participant { participant[ 'id' ] } left: { reason } " ) # Cancel the pipeline task to stop processing and remove the bot from the Daily room await task.cancel() ​ on_participant_updated async callback Event emitted when a participant is updated. This can mean either the participant’s metadata was updated, or the tracks belonging to the participant changed. Parameters: transport : The DailyTransport instance participant : Dictionary containing updated participant information Copy Ask AI @transport.event_handler ( "on_participant_updated" ) async def on_participant_updated ( transport , participant : Dict[ str , Any]): logger.info( f "Participant updated: { participant } " ) ​ Communication Events ​ on_app_message async callback Event emitted when a custom app message is received from another participant or via the REST API. Parameters: transport : The DailyTransport instance message : The message content (any type) sender : String identifier of the message sender Copy Ask AI @transport.event_handler ( "on_app_message" ) async def on_app_message ( transport , message : Any, sender : str ): logger.info( f "Message from { sender } : { message } " ) ​ on_call_state_updated async callback Event emitted when the call state changes, normally as a consequence of joining or leaving the call. Parameters: transport : The DailyTransport instance state : String representing the new call state. Learn more about call states . Copy Ask AI @transport.event_handler ( "on_call_state_updated" ) async def on_call_state_updated ( transport , state : str ): logger.info( f "Call state updated: { state } " ) ​ Dial Events ​ Dial-in ​ on_dialin_ready async callback Event emitted when dial-in is ready. This happens after the room has connected to the SIP endpoint and the system is ready to receive dial-in calls. Parameters: transport : The DailyTransport instance sip_endpoint : String containing the SIP endpoint information Copy Ask AI @transport.event_handler ( "on_dialin_ready" ) async def on_dialin_ready ( transport , sip_endpoint : str ): logger.info( f "Dial-in ready at: { sip_endpoint } " ) ​ on_dialin_connected async callback Event emitted when the session with the dial-in remote end is established (i.e. SIP endpoint or PSTN are connectd to the Daily room). Note: connected does not mean media (audio or video) has started flowing between the room and PSTN, it means the room received the connection request and both endpoints are negotiating the media flow. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. See DialinConnectedEvent . Copy Ask AI @transport.event_handler ( "on_dialin_connected" ) async def on_dialin_connected ( transport , data : Any): logger.info( f "Dial-in connected: { data } " ) ​ on_dialin_stopped async callback Event emitted when the dial-in remote end disconnects the call. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. See DialinStoppedEvent . Copy Ask AI @transport.event_handler ( "on_dialin_stopped" ) async def on_dialin_stopped ( transport , data : Any): logger.info( f "Dial-in stopped: { data } " ) ​ on_dialin_error async callback Event emitted in the case of dial-in errors which are fatal and the service cannot proceed. For example, an error in SDP negotiation is fatal to the media/SIP pipeline and will result in dialin-error being triggered. Parameters: transport : The DailyTransport instance data : Dictionary containing error information. See DialinEvent . Copy Ask AI @transport.event_handler ( "on_dialin_error" ) async def on_dialin_error ( transport , data : Any): logger.error( f "Dial-in error: { data } " ) ​ on_dialin_warning async callback Event emitted there is a dial-in non-fatal error, such as the selected codec not being used and a fallback codec being utilized. Parameters: transport : The DailyTransport instance data : Dictionary containing warning information. See DialinEvent . Copy Ask AI @transport.event_handler ( "on_dialin_warning" ) async def on_dialin_warning ( transport , data : Any): logger.warning( f "Dial-in warning: { data } " ) ​ Dial-out ​ on_dialout_answered async callback Event emitted when the session with the dial-out remote end is answered. Parameters: transport : The DailyTransport instance data : Dictionary containing call information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_answered" ) async def on_dialout_answered ( transport , data : Any): logger.info( f "Dial-out answered: { data } " ) ​ on_dialout_connected async callback Event emitted when the session with the dial-out remote end is established. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_connected" ) async def on_dialout_connected ( transport , data : Any): logger.info( f "Dial-out connected: { data } " ) ​ on_dialout_stopped async callback Event emitted when the dial-out session is stopped. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_stopped" ) async def on_dialout_stopped ( transport , data : Any): logger.info( f "Dial-out stopped: { data } " ) ​ on_dialout_error async callback Event emitted in the case of dial-out errors which are fatal and the service cannot proceed. For example, an error in SDP negotiation is fatal to the media/SIP pipeline and will result in dialout-error being triggered. Parameters: transport : The DailyTransport instance data : Dictionary containing error information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_error" ) async def on_dialout_error ( transport , data : Any): logger.error( f "Dial-out error: { data } " ) ​ on_dialout_warning async callback Event emitted there is a dial-out non-fatal error, such as the selected codec not being used and a fallback codec being utilized. Parameters: transport : The DailyTransport instance data : Dictionary containing warning information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_warning" ) async def on_dialout_warning ( transport , data : Any): logger.warning( f "Dial-out warning: { data } " ) ​ Transcription Events ​ on_transcription_message async callback Called when a transcription message is received. This includes both interim and final transcriptions. Parameters: transport : The DailyTransport instance message : Dictionary containing transcription data including. Learn more . Copy Ask AI @transport.event_handler ( "on_transcription_message" ) async def on_transcription_message ( transport , message : Dict[ str , Any]): participant_id = message.get( "participantId" ) text = message.get( "text" ) is_final = message[ "rawResponse" ][ "is_final" ] logger.info( f "Transcription from { participant_id } : { text } (final: { is_final } )" ) ​ Recording Events ​ on_recording_started async callback Called when a room recording starts successfully. Parameters: transport : The DailyTransport instance status : Dictionary containing recording status information. Learn more . Copy Ask AI @transport.event_handler ( "on_recording_started" ) async def on_recording_started ( transport , status : Dict[ str , Any]): logger.info( f "Recording started with status: { status } " ) ​ on_recording_stopped async callback Called when a room recording stops. Parameters: transport : The DailyTransport instance stream_id : String identifier of the stopped recording stream Copy Ask AI @transport.event_handler ( "on_recording_stopped" ) async def on_recording_stopped ( transport , stream_id : str ): logger.info( f "Recording stopped for stream: { stream_id } " ) ​ on_recording_error async callback Called when an error occurs during recording. Parameters: transport : The DailyTransport instance stream_id : String identifier of the recording stream message : Error message describing what went wrong Copy Ask AI @transport.event_handler ( "on_recording_error" ) async def on_recording_error ( transport , stream_id : str , message : str ): logger.error( f "Recording error for stream { stream_id } : { message } " ) ​ Error Events ​ on_error async callback Called when a transport error occurs. Parameters: transport : The DailyTransport instance error : String containing error details Copy Ask AI @transport.event_handler ( "on_error" ) async def on_error ( transport , error : str ): logger.error( f "Transport error: { error } " ) ​ Notes on Callbacks All callbacks are asynchronous Callbacks are executed in the order they were registered Multiple handlers can be registered for the same event Exceptions in callbacks are logged but don’t stop the transport Callbacks should be lightweight to avoid blocking the event loop Heavy processing should be offloaded to separate tasks ​ Frame Types ​ Input Frames ​ InputAudioRawFrame Frame Raw audio data from participants ​ UserImageRawFrame Frame Video frames from participants ​ Output Frames ​ OutputAudioRawFrame Frame Audio data to be sent ​ OutputImageRawFrame Frame Video frames to be sent ​ Methods ​ Room Management ​ participants method Returns a dictionary of all participants currently in the room. Copy Ask AI def participants () -> Dict[ str , Any] See participants for more details. ​ participant_counts method Returns participant count statistics for the room. Copy Ask AI def participant_counts () -> Dict[ str , int ] See participant_counts for more details. ​ send_message async method Sends a message to participants in the Daily room. Messages can be directed to all participants or targeted to a specific participant. Copy Ask AI async def send_message ( frame : TransportMessageFrame | TransportMessageUrgentFrame) Parameters: frame : A transport message frame containing the message to send. Can be either: TransportMessageFrame : Standard message TransportMessageUrgentFrame : High-priority message Message Structure: Messages should conform to the expected format for the receiving system. For RTVI messages, use: Copy Ask AI { "label" : "rtvi-ai" , # Required for RTVI protocol "type" : "message-type" , # Defines how the client will process it "id" : "unique-id" , # Unique identifier for the message "data" : { # Custom payload # Your custom data fields } } Example - Standard Message: Copy Ask AI import uuid # Send a standard message to all participants message_data = { "label" : "rtvi-ai" , "type" : "custom-message" , "id" : str (uuid.uuid4()), "data" : { "message" : "Hello everyone!" , "timestamp" : datetime.datetime.now().isoformat() } } await transport.send_message(TransportMessageFrame( message = message_data)) Example - Urgent Message to Specific Participant: Copy Ask AI # Send urgent message to specific participant urgent_data = { "label" : "rtvi-ai" , "type" : "server-message" , "id" : str (uuid.uuid4()), "data" : { "message" : "Private message" , "importance" : "high" } } await transport.send_message( DailyTransportMessageUrgentFrame( message = urgent_data, participant_id = "participant-123" ) ) The message will be delivered to: All participants if no specific participant_id is set Only the specified participant if participant_id is set Through the Daily app messaging system Can be received by registering an on_app_message handler Use TransportMessageUrgentFrame for high-priority messages that need immediate delivery. ​ Media Control ​ send_image async method Sends an image frame to the room. Copy Ask AI async def send_image ( frame : OutputImageRawFrame | SpriteFrame) -> None Parameters: frame : Image frame to send, either raw image data or sprite animation ​ send_audio async method Sends an audio frame to the room. Copy Ask AI async def send_audio ( frame : OutputAudioRawFrame) -> None Parameters: frame : Audio frame to send ​ Video Management ​ capture_participant_video async method Starts capturing video from a specific participant. Copy Ask AI async def capture_participant_video ( participant_id : str , framerate : int = 30 , video_source : str = "camera" , color_format : str = "RGB" ) -> None Parameters: participant_id : ID of the participant to capture framerate : Target frame rate (default: 30) video_source : Video source type (default: “camera”) color_format : Color format of the video (default: “RGB”) To request an image from the camera stream, set the framerate to 0 and push a UserImageRequestFrame . ​ Transcription Control ​ capture_participant_transcription async method Starts capturing and transcribing audio from a specific participant. Copy Ask AI async def capture_participant_transcription ( participant_id : str ) -> None Parameters: participant_id : ID of the participant to transcribe ​ update_transcription async method Updates the transcription configuration for specific participants or instances. Copy Ask AI async def update_transcription ( participants : List[ str ] | None = None , instance_id : str | None = None ) -> None Parameters: participants : Optional list of participant IDs to transcribe. If None, transcribes all participants. instance_id : Optional specific transcription instance ID to update Example: Copy Ask AI # Update transcription for specific participants await transport.update_transcription( participants = [ "participant-123" , "participant-456" ]) # Update specific transcription instance await transport.update_transcription( instance_id = "transcription-789" ) ​ Recording Control ​ start_recording async method Starts recording the room session. Copy Ask AI async def start_recording ( streaming_settings : Dict = None , stream_id : str = None , force_new : bool = None ) -> None Parameters: streaming_settings : Recording configuration settings stream_id : Optional stream identifier force_new : Force start a new recording See start_recording for more details. ​ stop_recording async method Stops an active recording. Copy Ask AI async def stop_recording ( stream_id : str = None ) -> None Parameters: stream_id : Optional stream identifier to stop specific recording See stop_recording for more details. ​ Dial-out Control ​ start_dialout async method Initiates a dial-out call. Copy Ask AI async def start_dialout ( settings : Dict = None ) -> None Parameters: settings : Dial-out configuration settings See start_dialout for more details. ​ stop_dialout async method Stops an active dial-out call. Copy Ask AI async def stop_dialout ( participant_id : str ) -> None Parameters: participant_id : ID of the participant to disconnect See stop_dialout for more details. ​ send_dtmf async method Sends DTMF tones in an existing dial-out session. Copy Ask AI async def send_dtmf ( settings : Dict = None ) -> None Parameters: settings : DTMF settings See send_dtmf for more details. ​ Subscription Management ​ update_subscriptions async method Updates subscriptions and subscription profiles. This function allows you to update subscription profiles and at the same time assign specific subscription profiles to a participant and even change specific settings for some participants. Copy Ask AI async def update_subscriptions ( participant_settings : Dict = None , profile_settings : Dict = None ) -> None Parameters: participant_settings : Per-participant subscription settings profile_settings : Global subscription profile settings See update_subscriptions for more details. ​ Participant Admin When using the DailyTransport, if your bot is given a meeting token with owner or participant admin permissions, it can remote-control other participants’ capabilities and permissions. With this control, the bot can modify settings for how others send and receive media, how they are seen in a session, and whether they have administrative controls. ​ update_remote_participants async method Updates permissions and settings for specific participants in the room. Copy Ask AI async def update_remote_participants ( remote_participants : Mapping[ str , Any] = None ) -> None Parameters: remote_participants : Dictionary mapping participant IDs to their updated settings This method allows fine-grained control over participant capabilities and permissions in the Daily room. Examples: Copy Ask AI # Mute a participant's microphone await transport.update_remote_participants({ "participant-123" : { "inputsEnabled" : { "microphone" : False } } }) # Revoke send permissions (mutes the participant and prevents them from unmuting) await transport.update_remote_participants({ "participant-123" : { "permissions" : { "canSend" : [] # Empty list revokes all send permissions } } }) # Configure selective audio routing await transport.update_remote_participants({ "participant-123" : { # the customer "permissions" : { "canReceive" : { "base" : False , # Don't receive from anyone by default "byUserId" : { "hold-music" : True # Only receive from hold music player } } } } }) # Set up user-to-user permissions mapping (e.g., when transferring calls) # Assumes customer was previously unable to hear anyone (they were on hold) and agent was speaking to bot await transport.update_remote_participants({ "participant-123" : { # the customer "permissions" : { "base" : False , "canReceive" : { "byUserId" : { "agent" : True }} } }, "participant-456" : { # the agent "permissions" : { "base" : False , "canReceive" : { "byUserId" : { "customer" : True }} } } }) # Promote a participant to be a participant admin await transport.update_remote_participants({ "participant-123" : { "permissions" : { "canAdmin" : [ "participants" ] } } }) Permissions Settings: For comprehensive details on permissions settings, see the Daily API documentation: daily-python’s update_remote_participants() documentation daily-js’s updateParticipant() -> updatePermission and participants().permissions ​ Chat messages ​ send_prebuilt_chat_message async method When testing with Daily Prebuilt, you can send messages to the Prebuilt chat using this method. Parameters: message (str): The chat message to send user_name (str): The name of the user sending the message See send_prebuilt_chat_message for more details. ​ Properties ​ participant_id string Returns the transport’s participant ID in the room. ​ room_url string Returns the transport’s room URL for the room currently in use. ​ Notes Supports real-time audio/video communication Handles WebRTC connection management Provides voice activity detection Includes transcription capabilities Manages participant interactions Supports recording and streaming Thread-safe processing All callbacks are asynchronous Multiple handlers can be registered for the same event Exceptions in callbacks are logged but don’t stop the transport Supported Services SmallWebRTCTransport On this page Overview Installation Configuration Constructor Parameters DailyParams Configuration Audio Output Configuration Audio Input Configuration Video Output Configuration Voice Activity Detection (VAD) Feature Settings Basic Usage Event Callbacks Connection Events Participant Events Communication Events Dial Events Dial-in Dial-out Transcription Events Recording Events Error Events Notes on Callbacks Frame Types Input Frames Output Frames Methods Room Management Media Control Video Management Transcription Control Recording Control Dial-out Control Subscription Management Participant Admin Chat messages Properties Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/transport_daily_22dbb191.txt b/transport_daily_22dbb191.txt
new file mode 100644
index 0000000000000000000000000000000000000000..24285296c23d8d7f8897298f77a796685bdeebd9
--- /dev/null
+++ b/transport_daily_22dbb191.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/transport/daily#param-api-key
+Title: Daily WebRTC - Pipecat
+==================================================
+
+Daily WebRTC - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Transport Daily WebRTC Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Daily WebRTC SmallWebRTCTransport FastAPI WebSocket WebSocket Server Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview DailyTransport provides real-time audio and video communication capabilities using Daily’s WebRTC platform. It supports bidirectional audio/video streams, transcription, voice activity detection (VAD), and participant management. ​ Installation To use DailyTransport , install the required dependencies: Copy Ask AI pip install "pipecat-ai[daily]" You’ll also need to set up your Daily API key as an environment variable: DAILY_API_KEY . You can obtain a Daily API key by signing up at Daily . ​ Configuration ​ Constructor Parameters ​ room_url str required Daily room URL ​ token str | None Daily room token ​ bot_name str required Name of the bot in the room ​ params DailyParams default: "DailyParams()" Transport configuration parameters ​ DailyParams Configuration ​ api_url str default: "https://api.daily.co/v1" Daily API endpoint URL ​ api_key str default: "" Daily API key for authentication ​ Audio Output Configuration ​ audio_out_enabled bool default: "False" Enable audio output capabilities ​ audio_out_is_live bool default: "False" Enable live audio streaming mode ​ audio_out_sample_rate int default: "None" Audio output sample rate in Hz ​ audio_out_channels int default: "1" Number of audio output channels ​ audio_out_bitrate int default: "96000" Audio output bitrate in bits per second ​ Audio Input Configuration ​ audio_in_enabled bool default: "False" Enable audio input capabilities ​ audio_in_passthrough bool default: "False" When enabled, incoming audio frames are pushed downstream ​ audio_in_sample_rate int default: "None" Audio input sample rate in Hz ​ audio_in_channels int default: "1" Number of audio input channels ​ audio_in_filter Optional[BaseAudioFilter] default: "None" Audio filter for input processing. Supported filters are: KrispFilter() and NoisereduceFilter() . See the KrispFilter() reference docs and NoisereduceFilter() reference docs ` for more information. ​ Video Output Configuration ​ video_out_enabled bool default: "False" Enable video output capabilities ​ video_out_is_live bool default: "False" Enable live video streaming mode ​ video_out_width int default: "1024" Video output width in pixels ​ video_out_height int default: "768" Video output height in pixels ​ video_out_bitrate int default: "800000" Video output bitrate in bits per second ​ video_out_framerate int default: "30" Video output frame rate ​ video_out_color_format str default: "RGB" Video color format (RGB, BGR, etc.) ​ Voice Activity Detection (VAD) ​ vad_analyzer VADAnalyzer | None default: "None" Voice Activity Detection analyzer. You can set this to either SileroVADAnalyzer() or WebRTCVADAnalyzer() . SileroVADAnalyzer is the recommended option. Learn more about the SileroVADAnalyzer . ​ Feature Settings ​ dialin_settings Optional[DailyDialinSettings] default: "None" Configuration for dial-in functionality ​ transcription_enabled bool default: "False" Enable real-time transcription ​ transcription_settings DailyTranscriptionSettings Configuration for transcription features Copy Ask AI class DailyTranscriptionSettings ( BaseModel ): language: str = "en" # Default language model: str = "nova-2-general" # Transcription model profanity_filter: bool = True # Filter profanity redact: bool = False # Redact sensitive information endpointing: bool = True # Enable speech endpointing punctuate: bool = True # Add punctuation includeRawResponse: bool = True # Include raw API response extra: Mapping[ str , Any] = { "interim_results" : True # Provide any Deepgram-specific settings } ​ Basic Usage Copy Ask AI from pipecat.transports.services.daily import DailyTransport, DailyParams from pipecat.audio.vad.silero import SileroVADAnalyzer # Configure transport transport = DailyTransport( room_url = "https://your-domain.daily.co/room-name" , token = "your-room-token" , bot_name = "AI Assistant" , params = DailyParams( audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer( params = VADParams( stop_secs = 0.5 )), ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), # Handle incoming audio/video stt, # Speech-to-text llm, # Language model tts, # Text-to-speech transport.output() # Handle outgoing audio/video ]) # Use transport event handlers to manage the call lifecycle # Event handler for when the user joins the room: @transport.event_handler ( "on_first_participant_joined" ) async def on_first_participant_joined ( transport , participant ): # Start transcription for the user await transport.capture_participant_transcription(participant[ "id" ]) # Kick off the conversation. await task.queue_frames([context_aggregator.user().get_context_frame()]) # Event handler for when the user leaves the room: @transport.event_handler ( "on_participant_left" ) async def on_participant_left ( transport , participant , reason ): # Cancel the pipeline, which stops processing and removes the bot from the room await task.cancel() ​ Event Callbacks DailyTransport provides a comprehensive callback system for handling various events. Register callbacks using the @transport.event_handler() decorator. ​ Connection Events ​ on_joined async callback Called when the bot successfully joins the room. Parameters: transport : The DailyTransport instance data : Dictionary containing room join information Copy Ask AI @transport.event_handler ( "on_joined" ) async def on_joined ( transport , data : Dict[ str , Any]): logger.info( f "Joined room with data: { data } " ) ​ on_left async callback Called when the bot leaves the room. Parameters: transport : The DailyTransport instance Copy Ask AI @transport.event_handler ( "on_left" ) async def on_left ( transport ): logger.info( "Left room" ) ​ Participant Events ​ on_first_participant_joined async callback Called when the first participant joins the room. Useful for initializing conversations. Parameters: transport : The DailyTransport instance participant : Dictionary containing participant information Copy Ask AI @transport.event_handler ( "on_first_participant_joined" ) async def on_first_participant_joined ( transport , participant : Dict[ str , Any]): await transport.capture_participant_transcription(participant[ "id" ]) await task.queue_frames([LLMMessagesFrame(initial_messages)]) ​ on_participant_joined async callback Called when any participant joins the room. Parameters: transport : The DailyTransport instance participant : Dictionary containing participant information Copy Ask AI @transport.event_handler ( "on_participant_joined" ) async def on_participant_joined ( transport , participant : Dict[ str , Any]): logger.info( f "Participant joined: { participant[ 'id' ] } " ) ​ on_participant_left async callback Called when a participant leaves the room. Parameters: transport : The DailyTransport instance participant : Dictionary containing participant information reason : String describing why the participant left, “leftCall” | “hidden” Copy Ask AI @transport.event_handler ( "on_participant_left" ) async def on_participant_left ( transport , participant : Dict[ str , Any], reason : str ): logger.info( f "Participant { participant[ 'id' ] } left: { reason } " ) # Cancel the pipeline task to stop processing and remove the bot from the Daily room await task.cancel() ​ on_participant_updated async callback Event emitted when a participant is updated. This can mean either the participant’s metadata was updated, or the tracks belonging to the participant changed. Parameters: transport : The DailyTransport instance participant : Dictionary containing updated participant information Copy Ask AI @transport.event_handler ( "on_participant_updated" ) async def on_participant_updated ( transport , participant : Dict[ str , Any]): logger.info( f "Participant updated: { participant } " ) ​ Communication Events ​ on_app_message async callback Event emitted when a custom app message is received from another participant or via the REST API. Parameters: transport : The DailyTransport instance message : The message content (any type) sender : String identifier of the message sender Copy Ask AI @transport.event_handler ( "on_app_message" ) async def on_app_message ( transport , message : Any, sender : str ): logger.info( f "Message from { sender } : { message } " ) ​ on_call_state_updated async callback Event emitted when the call state changes, normally as a consequence of joining or leaving the call. Parameters: transport : The DailyTransport instance state : String representing the new call state. Learn more about call states . Copy Ask AI @transport.event_handler ( "on_call_state_updated" ) async def on_call_state_updated ( transport , state : str ): logger.info( f "Call state updated: { state } " ) ​ Dial Events ​ Dial-in ​ on_dialin_ready async callback Event emitted when dial-in is ready. This happens after the room has connected to the SIP endpoint and the system is ready to receive dial-in calls. Parameters: transport : The DailyTransport instance sip_endpoint : String containing the SIP endpoint information Copy Ask AI @transport.event_handler ( "on_dialin_ready" ) async def on_dialin_ready ( transport , sip_endpoint : str ): logger.info( f "Dial-in ready at: { sip_endpoint } " ) ​ on_dialin_connected async callback Event emitted when the session with the dial-in remote end is established (i.e. SIP endpoint or PSTN are connectd to the Daily room). Note: connected does not mean media (audio or video) has started flowing between the room and PSTN, it means the room received the connection request and both endpoints are negotiating the media flow. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. See DialinConnectedEvent . Copy Ask AI @transport.event_handler ( "on_dialin_connected" ) async def on_dialin_connected ( transport , data : Any): logger.info( f "Dial-in connected: { data } " ) ​ on_dialin_stopped async callback Event emitted when the dial-in remote end disconnects the call. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. See DialinStoppedEvent . Copy Ask AI @transport.event_handler ( "on_dialin_stopped" ) async def on_dialin_stopped ( transport , data : Any): logger.info( f "Dial-in stopped: { data } " ) ​ on_dialin_error async callback Event emitted in the case of dial-in errors which are fatal and the service cannot proceed. For example, an error in SDP negotiation is fatal to the media/SIP pipeline and will result in dialin-error being triggered. Parameters: transport : The DailyTransport instance data : Dictionary containing error information. See DialinEvent . Copy Ask AI @transport.event_handler ( "on_dialin_error" ) async def on_dialin_error ( transport , data : Any): logger.error( f "Dial-in error: { data } " ) ​ on_dialin_warning async callback Event emitted there is a dial-in non-fatal error, such as the selected codec not being used and a fallback codec being utilized. Parameters: transport : The DailyTransport instance data : Dictionary containing warning information. See DialinEvent . Copy Ask AI @transport.event_handler ( "on_dialin_warning" ) async def on_dialin_warning ( transport , data : Any): logger.warning( f "Dial-in warning: { data } " ) ​ Dial-out ​ on_dialout_answered async callback Event emitted when the session with the dial-out remote end is answered. Parameters: transport : The DailyTransport instance data : Dictionary containing call information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_answered" ) async def on_dialout_answered ( transport , data : Any): logger.info( f "Dial-out answered: { data } " ) ​ on_dialout_connected async callback Event emitted when the session with the dial-out remote end is established. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_connected" ) async def on_dialout_connected ( transport , data : Any): logger.info( f "Dial-out connected: { data } " ) ​ on_dialout_stopped async callback Event emitted when the dial-out session is stopped. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_stopped" ) async def on_dialout_stopped ( transport , data : Any): logger.info( f "Dial-out stopped: { data } " ) ​ on_dialout_error async callback Event emitted in the case of dial-out errors which are fatal and the service cannot proceed. For example, an error in SDP negotiation is fatal to the media/SIP pipeline and will result in dialout-error being triggered. Parameters: transport : The DailyTransport instance data : Dictionary containing error information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_error" ) async def on_dialout_error ( transport , data : Any): logger.error( f "Dial-out error: { data } " ) ​ on_dialout_warning async callback Event emitted there is a dial-out non-fatal error, such as the selected codec not being used and a fallback codec being utilized. Parameters: transport : The DailyTransport instance data : Dictionary containing warning information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_warning" ) async def on_dialout_warning ( transport , data : Any): logger.warning( f "Dial-out warning: { data } " ) ​ Transcription Events ​ on_transcription_message async callback Called when a transcription message is received. This includes both interim and final transcriptions. Parameters: transport : The DailyTransport instance message : Dictionary containing transcription data including. Learn more . Copy Ask AI @transport.event_handler ( "on_transcription_message" ) async def on_transcription_message ( transport , message : Dict[ str , Any]): participant_id = message.get( "participantId" ) text = message.get( "text" ) is_final = message[ "rawResponse" ][ "is_final" ] logger.info( f "Transcription from { participant_id } : { text } (final: { is_final } )" ) ​ Recording Events ​ on_recording_started async callback Called when a room recording starts successfully. Parameters: transport : The DailyTransport instance status : Dictionary containing recording status information. Learn more . Copy Ask AI @transport.event_handler ( "on_recording_started" ) async def on_recording_started ( transport , status : Dict[ str , Any]): logger.info( f "Recording started with status: { status } " ) ​ on_recording_stopped async callback Called when a room recording stops. Parameters: transport : The DailyTransport instance stream_id : String identifier of the stopped recording stream Copy Ask AI @transport.event_handler ( "on_recording_stopped" ) async def on_recording_stopped ( transport , stream_id : str ): logger.info( f "Recording stopped for stream: { stream_id } " ) ​ on_recording_error async callback Called when an error occurs during recording. Parameters: transport : The DailyTransport instance stream_id : String identifier of the recording stream message : Error message describing what went wrong Copy Ask AI @transport.event_handler ( "on_recording_error" ) async def on_recording_error ( transport , stream_id : str , message : str ): logger.error( f "Recording error for stream { stream_id } : { message } " ) ​ Error Events ​ on_error async callback Called when a transport error occurs. Parameters: transport : The DailyTransport instance error : String containing error details Copy Ask AI @transport.event_handler ( "on_error" ) async def on_error ( transport , error : str ): logger.error( f "Transport error: { error } " ) ​ Notes on Callbacks All callbacks are asynchronous Callbacks are executed in the order they were registered Multiple handlers can be registered for the same event Exceptions in callbacks are logged but don’t stop the transport Callbacks should be lightweight to avoid blocking the event loop Heavy processing should be offloaded to separate tasks ​ Frame Types ​ Input Frames ​ InputAudioRawFrame Frame Raw audio data from participants ​ UserImageRawFrame Frame Video frames from participants ​ Output Frames ​ OutputAudioRawFrame Frame Audio data to be sent ​ OutputImageRawFrame Frame Video frames to be sent ​ Methods ​ Room Management ​ participants method Returns a dictionary of all participants currently in the room. Copy Ask AI def participants () -> Dict[ str , Any] See participants for more details. ​ participant_counts method Returns participant count statistics for the room. Copy Ask AI def participant_counts () -> Dict[ str , int ] See participant_counts for more details. ​ send_message async method Sends a message to participants in the Daily room. Messages can be directed to all participants or targeted to a specific participant. Copy Ask AI async def send_message ( frame : TransportMessageFrame | TransportMessageUrgentFrame) Parameters: frame : A transport message frame containing the message to send. Can be either: TransportMessageFrame : Standard message TransportMessageUrgentFrame : High-priority message Message Structure: Messages should conform to the expected format for the receiving system. For RTVI messages, use: Copy Ask AI { "label" : "rtvi-ai" , # Required for RTVI protocol "type" : "message-type" , # Defines how the client will process it "id" : "unique-id" , # Unique identifier for the message "data" : { # Custom payload # Your custom data fields } } Example - Standard Message: Copy Ask AI import uuid # Send a standard message to all participants message_data = { "label" : "rtvi-ai" , "type" : "custom-message" , "id" : str (uuid.uuid4()), "data" : { "message" : "Hello everyone!" , "timestamp" : datetime.datetime.now().isoformat() } } await transport.send_message(TransportMessageFrame( message = message_data)) Example - Urgent Message to Specific Participant: Copy Ask AI # Send urgent message to specific participant urgent_data = { "label" : "rtvi-ai" , "type" : "server-message" , "id" : str (uuid.uuid4()), "data" : { "message" : "Private message" , "importance" : "high" } } await transport.send_message( DailyTransportMessageUrgentFrame( message = urgent_data, participant_id = "participant-123" ) ) The message will be delivered to: All participants if no specific participant_id is set Only the specified participant if participant_id is set Through the Daily app messaging system Can be received by registering an on_app_message handler Use TransportMessageUrgentFrame for high-priority messages that need immediate delivery. ​ Media Control ​ send_image async method Sends an image frame to the room. Copy Ask AI async def send_image ( frame : OutputImageRawFrame | SpriteFrame) -> None Parameters: frame : Image frame to send, either raw image data or sprite animation ​ send_audio async method Sends an audio frame to the room. Copy Ask AI async def send_audio ( frame : OutputAudioRawFrame) -> None Parameters: frame : Audio frame to send ​ Video Management ​ capture_participant_video async method Starts capturing video from a specific participant. Copy Ask AI async def capture_participant_video ( participant_id : str , framerate : int = 30 , video_source : str = "camera" , color_format : str = "RGB" ) -> None Parameters: participant_id : ID of the participant to capture framerate : Target frame rate (default: 30) video_source : Video source type (default: “camera”) color_format : Color format of the video (default: “RGB”) To request an image from the camera stream, set the framerate to 0 and push a UserImageRequestFrame . ​ Transcription Control ​ capture_participant_transcription async method Starts capturing and transcribing audio from a specific participant. Copy Ask AI async def capture_participant_transcription ( participant_id : str ) -> None Parameters: participant_id : ID of the participant to transcribe ​ update_transcription async method Updates the transcription configuration for specific participants or instances. Copy Ask AI async def update_transcription ( participants : List[ str ] | None = None , instance_id : str | None = None ) -> None Parameters: participants : Optional list of participant IDs to transcribe. If None, transcribes all participants. instance_id : Optional specific transcription instance ID to update Example: Copy Ask AI # Update transcription for specific participants await transport.update_transcription( participants = [ "participant-123" , "participant-456" ]) # Update specific transcription instance await transport.update_transcription( instance_id = "transcription-789" ) ​ Recording Control ​ start_recording async method Starts recording the room session. Copy Ask AI async def start_recording ( streaming_settings : Dict = None , stream_id : str = None , force_new : bool = None ) -> None Parameters: streaming_settings : Recording configuration settings stream_id : Optional stream identifier force_new : Force start a new recording See start_recording for more details. ​ stop_recording async method Stops an active recording. Copy Ask AI async def stop_recording ( stream_id : str = None ) -> None Parameters: stream_id : Optional stream identifier to stop specific recording See stop_recording for more details. ​ Dial-out Control ​ start_dialout async method Initiates a dial-out call. Copy Ask AI async def start_dialout ( settings : Dict = None ) -> None Parameters: settings : Dial-out configuration settings See start_dialout for more details. ​ stop_dialout async method Stops an active dial-out call. Copy Ask AI async def stop_dialout ( participant_id : str ) -> None Parameters: participant_id : ID of the participant to disconnect See stop_dialout for more details. ​ send_dtmf async method Sends DTMF tones in an existing dial-out session. Copy Ask AI async def send_dtmf ( settings : Dict = None ) -> None Parameters: settings : DTMF settings See send_dtmf for more details. ​ Subscription Management ​ update_subscriptions async method Updates subscriptions and subscription profiles. This function allows you to update subscription profiles and at the same time assign specific subscription profiles to a participant and even change specific settings for some participants. Copy Ask AI async def update_subscriptions ( participant_settings : Dict = None , profile_settings : Dict = None ) -> None Parameters: participant_settings : Per-participant subscription settings profile_settings : Global subscription profile settings See update_subscriptions for more details. ​ Participant Admin When using the DailyTransport, if your bot is given a meeting token with owner or participant admin permissions, it can remote-control other participants’ capabilities and permissions. With this control, the bot can modify settings for how others send and receive media, how they are seen in a session, and whether they have administrative controls. ​ update_remote_participants async method Updates permissions and settings for specific participants in the room. Copy Ask AI async def update_remote_participants ( remote_participants : Mapping[ str , Any] = None ) -> None Parameters: remote_participants : Dictionary mapping participant IDs to their updated settings This method allows fine-grained control over participant capabilities and permissions in the Daily room. Examples: Copy Ask AI # Mute a participant's microphone await transport.update_remote_participants({ "participant-123" : { "inputsEnabled" : { "microphone" : False } } }) # Revoke send permissions (mutes the participant and prevents them from unmuting) await transport.update_remote_participants({ "participant-123" : { "permissions" : { "canSend" : [] # Empty list revokes all send permissions } } }) # Configure selective audio routing await transport.update_remote_participants({ "participant-123" : { # the customer "permissions" : { "canReceive" : { "base" : False , # Don't receive from anyone by default "byUserId" : { "hold-music" : True # Only receive from hold music player } } } } }) # Set up user-to-user permissions mapping (e.g., when transferring calls) # Assumes customer was previously unable to hear anyone (they were on hold) and agent was speaking to bot await transport.update_remote_participants({ "participant-123" : { # the customer "permissions" : { "base" : False , "canReceive" : { "byUserId" : { "agent" : True }} } }, "participant-456" : { # the agent "permissions" : { "base" : False , "canReceive" : { "byUserId" : { "customer" : True }} } } }) # Promote a participant to be a participant admin await transport.update_remote_participants({ "participant-123" : { "permissions" : { "canAdmin" : [ "participants" ] } } }) Permissions Settings: For comprehensive details on permissions settings, see the Daily API documentation: daily-python’s update_remote_participants() documentation daily-js’s updateParticipant() -> updatePermission and participants().permissions ​ Chat messages ​ send_prebuilt_chat_message async method When testing with Daily Prebuilt, you can send messages to the Prebuilt chat using this method. Parameters: message (str): The chat message to send user_name (str): The name of the user sending the message See send_prebuilt_chat_message for more details. ​ Properties ​ participant_id string Returns the transport’s participant ID in the room. ​ room_url string Returns the transport’s room URL for the room currently in use. ​ Notes Supports real-time audio/video communication Handles WebRTC connection management Provides voice activity detection Includes transcription capabilities Manages participant interactions Supports recording and streaming Thread-safe processing All callbacks are asynchronous Multiple handlers can be registered for the same event Exceptions in callbacks are logged but don’t stop the transport Supported Services SmallWebRTCTransport On this page Overview Installation Configuration Constructor Parameters DailyParams Configuration Audio Output Configuration Audio Input Configuration Video Output Configuration Voice Activity Detection (VAD) Feature Settings Basic Usage Event Callbacks Connection Events Participant Events Communication Events Dial Events Dial-in Dial-out Transcription Events Recording Events Error Events Notes on Callbacks Frame Types Input Frames Output Frames Methods Room Management Media Control Video Management Transcription Control Recording Control Dial-out Control Subscription Management Participant Admin Chat messages Properties Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/transport_daily_23ee3384.txt b/transport_daily_23ee3384.txt
new file mode 100644
index 0000000000000000000000000000000000000000..cbd009b8915140fc775dfc9515e6fd4899bf3f8d
--- /dev/null
+++ b/transport_daily_23ee3384.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/transport/daily#param-on-dialin-stopped
+Title: Daily WebRTC - Pipecat
+==================================================
+
+Daily WebRTC - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Transport Daily WebRTC Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Daily WebRTC SmallWebRTCTransport FastAPI WebSocket WebSocket Server Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview DailyTransport provides real-time audio and video communication capabilities using Daily’s WebRTC platform. It supports bidirectional audio/video streams, transcription, voice activity detection (VAD), and participant management. ​ Installation To use DailyTransport , install the required dependencies: Copy Ask AI pip install "pipecat-ai[daily]" You’ll also need to set up your Daily API key as an environment variable: DAILY_API_KEY . You can obtain a Daily API key by signing up at Daily . ​ Configuration ​ Constructor Parameters ​ room_url str required Daily room URL ​ token str | None Daily room token ​ bot_name str required Name of the bot in the room ​ params DailyParams default: "DailyParams()" Transport configuration parameters ​ DailyParams Configuration ​ api_url str default: "https://api.daily.co/v1" Daily API endpoint URL ​ api_key str default: "" Daily API key for authentication ​ Audio Output Configuration ​ audio_out_enabled bool default: "False" Enable audio output capabilities ​ audio_out_is_live bool default: "False" Enable live audio streaming mode ​ audio_out_sample_rate int default: "None" Audio output sample rate in Hz ​ audio_out_channels int default: "1" Number of audio output channels ​ audio_out_bitrate int default: "96000" Audio output bitrate in bits per second ​ Audio Input Configuration ​ audio_in_enabled bool default: "False" Enable audio input capabilities ​ audio_in_passthrough bool default: "False" When enabled, incoming audio frames are pushed downstream ​ audio_in_sample_rate int default: "None" Audio input sample rate in Hz ​ audio_in_channels int default: "1" Number of audio input channels ​ audio_in_filter Optional[BaseAudioFilter] default: "None" Audio filter for input processing. Supported filters are: KrispFilter() and NoisereduceFilter() . See the KrispFilter() reference docs and NoisereduceFilter() reference docs ` for more information. ​ Video Output Configuration ​ video_out_enabled bool default: "False" Enable video output capabilities ​ video_out_is_live bool default: "False" Enable live video streaming mode ​ video_out_width int default: "1024" Video output width in pixels ​ video_out_height int default: "768" Video output height in pixels ​ video_out_bitrate int default: "800000" Video output bitrate in bits per second ​ video_out_framerate int default: "30" Video output frame rate ​ video_out_color_format str default: "RGB" Video color format (RGB, BGR, etc.) ​ Voice Activity Detection (VAD) ​ vad_analyzer VADAnalyzer | None default: "None" Voice Activity Detection analyzer. You can set this to either SileroVADAnalyzer() or WebRTCVADAnalyzer() . SileroVADAnalyzer is the recommended option. Learn more about the SileroVADAnalyzer . ​ Feature Settings ​ dialin_settings Optional[DailyDialinSettings] default: "None" Configuration for dial-in functionality ​ transcription_enabled bool default: "False" Enable real-time transcription ​ transcription_settings DailyTranscriptionSettings Configuration for transcription features Copy Ask AI class DailyTranscriptionSettings ( BaseModel ): language: str = "en" # Default language model: str = "nova-2-general" # Transcription model profanity_filter: bool = True # Filter profanity redact: bool = False # Redact sensitive information endpointing: bool = True # Enable speech endpointing punctuate: bool = True # Add punctuation includeRawResponse: bool = True # Include raw API response extra: Mapping[ str , Any] = { "interim_results" : True # Provide any Deepgram-specific settings } ​ Basic Usage Copy Ask AI from pipecat.transports.services.daily import DailyTransport, DailyParams from pipecat.audio.vad.silero import SileroVADAnalyzer # Configure transport transport = DailyTransport( room_url = "https://your-domain.daily.co/room-name" , token = "your-room-token" , bot_name = "AI Assistant" , params = DailyParams( audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer( params = VADParams( stop_secs = 0.5 )), ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), # Handle incoming audio/video stt, # Speech-to-text llm, # Language model tts, # Text-to-speech transport.output() # Handle outgoing audio/video ]) # Use transport event handlers to manage the call lifecycle # Event handler for when the user joins the room: @transport.event_handler ( "on_first_participant_joined" ) async def on_first_participant_joined ( transport , participant ): # Start transcription for the user await transport.capture_participant_transcription(participant[ "id" ]) # Kick off the conversation. await task.queue_frames([context_aggregator.user().get_context_frame()]) # Event handler for when the user leaves the room: @transport.event_handler ( "on_participant_left" ) async def on_participant_left ( transport , participant , reason ): # Cancel the pipeline, which stops processing and removes the bot from the room await task.cancel() ​ Event Callbacks DailyTransport provides a comprehensive callback system for handling various events. Register callbacks using the @transport.event_handler() decorator. ​ Connection Events ​ on_joined async callback Called when the bot successfully joins the room. Parameters: transport : The DailyTransport instance data : Dictionary containing room join information Copy Ask AI @transport.event_handler ( "on_joined" ) async def on_joined ( transport , data : Dict[ str , Any]): logger.info( f "Joined room with data: { data } " ) ​ on_left async callback Called when the bot leaves the room. Parameters: transport : The DailyTransport instance Copy Ask AI @transport.event_handler ( "on_left" ) async def on_left ( transport ): logger.info( "Left room" ) ​ Participant Events ​ on_first_participant_joined async callback Called when the first participant joins the room. Useful for initializing conversations. Parameters: transport : The DailyTransport instance participant : Dictionary containing participant information Copy Ask AI @transport.event_handler ( "on_first_participant_joined" ) async def on_first_participant_joined ( transport , participant : Dict[ str , Any]): await transport.capture_participant_transcription(participant[ "id" ]) await task.queue_frames([LLMMessagesFrame(initial_messages)]) ​ on_participant_joined async callback Called when any participant joins the room. Parameters: transport : The DailyTransport instance participant : Dictionary containing participant information Copy Ask AI @transport.event_handler ( "on_participant_joined" ) async def on_participant_joined ( transport , participant : Dict[ str , Any]): logger.info( f "Participant joined: { participant[ 'id' ] } " ) ​ on_participant_left async callback Called when a participant leaves the room. Parameters: transport : The DailyTransport instance participant : Dictionary containing participant information reason : String describing why the participant left, “leftCall” | “hidden” Copy Ask AI @transport.event_handler ( "on_participant_left" ) async def on_participant_left ( transport , participant : Dict[ str , Any], reason : str ): logger.info( f "Participant { participant[ 'id' ] } left: { reason } " ) # Cancel the pipeline task to stop processing and remove the bot from the Daily room await task.cancel() ​ on_participant_updated async callback Event emitted when a participant is updated. This can mean either the participant’s metadata was updated, or the tracks belonging to the participant changed. Parameters: transport : The DailyTransport instance participant : Dictionary containing updated participant information Copy Ask AI @transport.event_handler ( "on_participant_updated" ) async def on_participant_updated ( transport , participant : Dict[ str , Any]): logger.info( f "Participant updated: { participant } " ) ​ Communication Events ​ on_app_message async callback Event emitted when a custom app message is received from another participant or via the REST API. Parameters: transport : The DailyTransport instance message : The message content (any type) sender : String identifier of the message sender Copy Ask AI @transport.event_handler ( "on_app_message" ) async def on_app_message ( transport , message : Any, sender : str ): logger.info( f "Message from { sender } : { message } " ) ​ on_call_state_updated async callback Event emitted when the call state changes, normally as a consequence of joining or leaving the call. Parameters: transport : The DailyTransport instance state : String representing the new call state. Learn more about call states . Copy Ask AI @transport.event_handler ( "on_call_state_updated" ) async def on_call_state_updated ( transport , state : str ): logger.info( f "Call state updated: { state } " ) ​ Dial Events ​ Dial-in ​ on_dialin_ready async callback Event emitted when dial-in is ready. This happens after the room has connected to the SIP endpoint and the system is ready to receive dial-in calls. Parameters: transport : The DailyTransport instance sip_endpoint : String containing the SIP endpoint information Copy Ask AI @transport.event_handler ( "on_dialin_ready" ) async def on_dialin_ready ( transport , sip_endpoint : str ): logger.info( f "Dial-in ready at: { sip_endpoint } " ) ​ on_dialin_connected async callback Event emitted when the session with the dial-in remote end is established (i.e. SIP endpoint or PSTN are connectd to the Daily room). Note: connected does not mean media (audio or video) has started flowing between the room and PSTN, it means the room received the connection request and both endpoints are negotiating the media flow. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. See DialinConnectedEvent . Copy Ask AI @transport.event_handler ( "on_dialin_connected" ) async def on_dialin_connected ( transport , data : Any): logger.info( f "Dial-in connected: { data } " ) ​ on_dialin_stopped async callback Event emitted when the dial-in remote end disconnects the call. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. See DialinStoppedEvent . Copy Ask AI @transport.event_handler ( "on_dialin_stopped" ) async def on_dialin_stopped ( transport , data : Any): logger.info( f "Dial-in stopped: { data } " ) ​ on_dialin_error async callback Event emitted in the case of dial-in errors which are fatal and the service cannot proceed. For example, an error in SDP negotiation is fatal to the media/SIP pipeline and will result in dialin-error being triggered. Parameters: transport : The DailyTransport instance data : Dictionary containing error information. See DialinEvent . Copy Ask AI @transport.event_handler ( "on_dialin_error" ) async def on_dialin_error ( transport , data : Any): logger.error( f "Dial-in error: { data } " ) ​ on_dialin_warning async callback Event emitted there is a dial-in non-fatal error, such as the selected codec not being used and a fallback codec being utilized. Parameters: transport : The DailyTransport instance data : Dictionary containing warning information. See DialinEvent . Copy Ask AI @transport.event_handler ( "on_dialin_warning" ) async def on_dialin_warning ( transport , data : Any): logger.warning( f "Dial-in warning: { data } " ) ​ Dial-out ​ on_dialout_answered async callback Event emitted when the session with the dial-out remote end is answered. Parameters: transport : The DailyTransport instance data : Dictionary containing call information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_answered" ) async def on_dialout_answered ( transport , data : Any): logger.info( f "Dial-out answered: { data } " ) ​ on_dialout_connected async callback Event emitted when the session with the dial-out remote end is established. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_connected" ) async def on_dialout_connected ( transport , data : Any): logger.info( f "Dial-out connected: { data } " ) ​ on_dialout_stopped async callback Event emitted when the dial-out session is stopped. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_stopped" ) async def on_dialout_stopped ( transport , data : Any): logger.info( f "Dial-out stopped: { data } " ) ​ on_dialout_error async callback Event emitted in the case of dial-out errors which are fatal and the service cannot proceed. For example, an error in SDP negotiation is fatal to the media/SIP pipeline and will result in dialout-error being triggered. Parameters: transport : The DailyTransport instance data : Dictionary containing error information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_error" ) async def on_dialout_error ( transport , data : Any): logger.error( f "Dial-out error: { data } " ) ​ on_dialout_warning async callback Event emitted there is a dial-out non-fatal error, such as the selected codec not being used and a fallback codec being utilized. Parameters: transport : The DailyTransport instance data : Dictionary containing warning information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_warning" ) async def on_dialout_warning ( transport , data : Any): logger.warning( f "Dial-out warning: { data } " ) ​ Transcription Events ​ on_transcription_message async callback Called when a transcription message is received. This includes both interim and final transcriptions. Parameters: transport : The DailyTransport instance message : Dictionary containing transcription data including. Learn more . Copy Ask AI @transport.event_handler ( "on_transcription_message" ) async def on_transcription_message ( transport , message : Dict[ str , Any]): participant_id = message.get( "participantId" ) text = message.get( "text" ) is_final = message[ "rawResponse" ][ "is_final" ] logger.info( f "Transcription from { participant_id } : { text } (final: { is_final } )" ) ​ Recording Events ​ on_recording_started async callback Called when a room recording starts successfully. Parameters: transport : The DailyTransport instance status : Dictionary containing recording status information. Learn more . Copy Ask AI @transport.event_handler ( "on_recording_started" ) async def on_recording_started ( transport , status : Dict[ str , Any]): logger.info( f "Recording started with status: { status } " ) ​ on_recording_stopped async callback Called when a room recording stops. Parameters: transport : The DailyTransport instance stream_id : String identifier of the stopped recording stream Copy Ask AI @transport.event_handler ( "on_recording_stopped" ) async def on_recording_stopped ( transport , stream_id : str ): logger.info( f "Recording stopped for stream: { stream_id } " ) ​ on_recording_error async callback Called when an error occurs during recording. Parameters: transport : The DailyTransport instance stream_id : String identifier of the recording stream message : Error message describing what went wrong Copy Ask AI @transport.event_handler ( "on_recording_error" ) async def on_recording_error ( transport , stream_id : str , message : str ): logger.error( f "Recording error for stream { stream_id } : { message } " ) ​ Error Events ​ on_error async callback Called when a transport error occurs. Parameters: transport : The DailyTransport instance error : String containing error details Copy Ask AI @transport.event_handler ( "on_error" ) async def on_error ( transport , error : str ): logger.error( f "Transport error: { error } " ) ​ Notes on Callbacks All callbacks are asynchronous Callbacks are executed in the order they were registered Multiple handlers can be registered for the same event Exceptions in callbacks are logged but don’t stop the transport Callbacks should be lightweight to avoid blocking the event loop Heavy processing should be offloaded to separate tasks ​ Frame Types ​ Input Frames ​ InputAudioRawFrame Frame Raw audio data from participants ​ UserImageRawFrame Frame Video frames from participants ​ Output Frames ​ OutputAudioRawFrame Frame Audio data to be sent ​ OutputImageRawFrame Frame Video frames to be sent ​ Methods ​ Room Management ​ participants method Returns a dictionary of all participants currently in the room. Copy Ask AI def participants () -> Dict[ str , Any] See participants for more details. ​ participant_counts method Returns participant count statistics for the room. Copy Ask AI def participant_counts () -> Dict[ str , int ] See participant_counts for more details. ​ send_message async method Sends a message to participants in the Daily room. Messages can be directed to all participants or targeted to a specific participant. Copy Ask AI async def send_message ( frame : TransportMessageFrame | TransportMessageUrgentFrame) Parameters: frame : A transport message frame containing the message to send. Can be either: TransportMessageFrame : Standard message TransportMessageUrgentFrame : High-priority message Message Structure: Messages should conform to the expected format for the receiving system. For RTVI messages, use: Copy Ask AI { "label" : "rtvi-ai" , # Required for RTVI protocol "type" : "message-type" , # Defines how the client will process it "id" : "unique-id" , # Unique identifier for the message "data" : { # Custom payload # Your custom data fields } } Example - Standard Message: Copy Ask AI import uuid # Send a standard message to all participants message_data = { "label" : "rtvi-ai" , "type" : "custom-message" , "id" : str (uuid.uuid4()), "data" : { "message" : "Hello everyone!" , "timestamp" : datetime.datetime.now().isoformat() } } await transport.send_message(TransportMessageFrame( message = message_data)) Example - Urgent Message to Specific Participant: Copy Ask AI # Send urgent message to specific participant urgent_data = { "label" : "rtvi-ai" , "type" : "server-message" , "id" : str (uuid.uuid4()), "data" : { "message" : "Private message" , "importance" : "high" } } await transport.send_message( DailyTransportMessageUrgentFrame( message = urgent_data, participant_id = "participant-123" ) ) The message will be delivered to: All participants if no specific participant_id is set Only the specified participant if participant_id is set Through the Daily app messaging system Can be received by registering an on_app_message handler Use TransportMessageUrgentFrame for high-priority messages that need immediate delivery. ​ Media Control ​ send_image async method Sends an image frame to the room. Copy Ask AI async def send_image ( frame : OutputImageRawFrame | SpriteFrame) -> None Parameters: frame : Image frame to send, either raw image data or sprite animation ​ send_audio async method Sends an audio frame to the room. Copy Ask AI async def send_audio ( frame : OutputAudioRawFrame) -> None Parameters: frame : Audio frame to send ​ Video Management ​ capture_participant_video async method Starts capturing video from a specific participant. Copy Ask AI async def capture_participant_video ( participant_id : str , framerate : int = 30 , video_source : str = "camera" , color_format : str = "RGB" ) -> None Parameters: participant_id : ID of the participant to capture framerate : Target frame rate (default: 30) video_source : Video source type (default: “camera”) color_format : Color format of the video (default: “RGB”) To request an image from the camera stream, set the framerate to 0 and push a UserImageRequestFrame . ​ Transcription Control ​ capture_participant_transcription async method Starts capturing and transcribing audio from a specific participant. Copy Ask AI async def capture_participant_transcription ( participant_id : str ) -> None Parameters: participant_id : ID of the participant to transcribe ​ update_transcription async method Updates the transcription configuration for specific participants or instances. Copy Ask AI async def update_transcription ( participants : List[ str ] | None = None , instance_id : str | None = None ) -> None Parameters: participants : Optional list of participant IDs to transcribe. If None, transcribes all participants. instance_id : Optional specific transcription instance ID to update Example: Copy Ask AI # Update transcription for specific participants await transport.update_transcription( participants = [ "participant-123" , "participant-456" ]) # Update specific transcription instance await transport.update_transcription( instance_id = "transcription-789" ) ​ Recording Control ​ start_recording async method Starts recording the room session. Copy Ask AI async def start_recording ( streaming_settings : Dict = None , stream_id : str = None , force_new : bool = None ) -> None Parameters: streaming_settings : Recording configuration settings stream_id : Optional stream identifier force_new : Force start a new recording See start_recording for more details. ​ stop_recording async method Stops an active recording. Copy Ask AI async def stop_recording ( stream_id : str = None ) -> None Parameters: stream_id : Optional stream identifier to stop specific recording See stop_recording for more details. ​ Dial-out Control ​ start_dialout async method Initiates a dial-out call. Copy Ask AI async def start_dialout ( settings : Dict = None ) -> None Parameters: settings : Dial-out configuration settings See start_dialout for more details. ​ stop_dialout async method Stops an active dial-out call. Copy Ask AI async def stop_dialout ( participant_id : str ) -> None Parameters: participant_id : ID of the participant to disconnect See stop_dialout for more details. ​ send_dtmf async method Sends DTMF tones in an existing dial-out session. Copy Ask AI async def send_dtmf ( settings : Dict = None ) -> None Parameters: settings : DTMF settings See send_dtmf for more details. ​ Subscription Management ​ update_subscriptions async method Updates subscriptions and subscription profiles. This function allows you to update subscription profiles and at the same time assign specific subscription profiles to a participant and even change specific settings for some participants. Copy Ask AI async def update_subscriptions ( participant_settings : Dict = None , profile_settings : Dict = None ) -> None Parameters: participant_settings : Per-participant subscription settings profile_settings : Global subscription profile settings See update_subscriptions for more details. ​ Participant Admin When using the DailyTransport, if your bot is given a meeting token with owner or participant admin permissions, it can remote-control other participants’ capabilities and permissions. With this control, the bot can modify settings for how others send and receive media, how they are seen in a session, and whether they have administrative controls. ​ update_remote_participants async method Updates permissions and settings for specific participants in the room. Copy Ask AI async def update_remote_participants ( remote_participants : Mapping[ str , Any] = None ) -> None Parameters: remote_participants : Dictionary mapping participant IDs to their updated settings This method allows fine-grained control over participant capabilities and permissions in the Daily room. Examples: Copy Ask AI # Mute a participant's microphone await transport.update_remote_participants({ "participant-123" : { "inputsEnabled" : { "microphone" : False } } }) # Revoke send permissions (mutes the participant and prevents them from unmuting) await transport.update_remote_participants({ "participant-123" : { "permissions" : { "canSend" : [] # Empty list revokes all send permissions } } }) # Configure selective audio routing await transport.update_remote_participants({ "participant-123" : { # the customer "permissions" : { "canReceive" : { "base" : False , # Don't receive from anyone by default "byUserId" : { "hold-music" : True # Only receive from hold music player } } } } }) # Set up user-to-user permissions mapping (e.g., when transferring calls) # Assumes customer was previously unable to hear anyone (they were on hold) and agent was speaking to bot await transport.update_remote_participants({ "participant-123" : { # the customer "permissions" : { "base" : False , "canReceive" : { "byUserId" : { "agent" : True }} } }, "participant-456" : { # the agent "permissions" : { "base" : False , "canReceive" : { "byUserId" : { "customer" : True }} } } }) # Promote a participant to be a participant admin await transport.update_remote_participants({ "participant-123" : { "permissions" : { "canAdmin" : [ "participants" ] } } }) Permissions Settings: For comprehensive details on permissions settings, see the Daily API documentation: daily-python’s update_remote_participants() documentation daily-js’s updateParticipant() -> updatePermission and participants().permissions ​ Chat messages ​ send_prebuilt_chat_message async method When testing with Daily Prebuilt, you can send messages to the Prebuilt chat using this method. Parameters: message (str): The chat message to send user_name (str): The name of the user sending the message See send_prebuilt_chat_message for more details. ​ Properties ​ participant_id string Returns the transport’s participant ID in the room. ​ room_url string Returns the transport’s room URL for the room currently in use. ​ Notes Supports real-time audio/video communication Handles WebRTC connection management Provides voice activity detection Includes transcription capabilities Manages participant interactions Supports recording and streaming Thread-safe processing All callbacks are asynchronous Multiple handlers can be registered for the same event Exceptions in callbacks are logged but don’t stop the transport Supported Services SmallWebRTCTransport On this page Overview Installation Configuration Constructor Parameters DailyParams Configuration Audio Output Configuration Audio Input Configuration Video Output Configuration Voice Activity Detection (VAD) Feature Settings Basic Usage Event Callbacks Connection Events Participant Events Communication Events Dial Events Dial-in Dial-out Transcription Events Recording Events Error Events Notes on Callbacks Frame Types Input Frames Output Frames Methods Room Management Media Control Video Management Transcription Control Recording Control Dial-out Control Subscription Management Participant Admin Chat messages Properties Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/transport_daily_2eabacf7.txt b/transport_daily_2eabacf7.txt
new file mode 100644
index 0000000000000000000000000000000000000000..4401fdd53bfcfcc38b40c360fb4dc622dc48d98e
--- /dev/null
+++ b/transport_daily_2eabacf7.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/transport/daily#param-start-dialout
+Title: Daily WebRTC - Pipecat
+==================================================
+
+Daily WebRTC - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Transport Daily WebRTC Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Daily WebRTC SmallWebRTCTransport FastAPI WebSocket WebSocket Server Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview DailyTransport provides real-time audio and video communication capabilities using Daily’s WebRTC platform. It supports bidirectional audio/video streams, transcription, voice activity detection (VAD), and participant management. ​ Installation To use DailyTransport , install the required dependencies: Copy Ask AI pip install "pipecat-ai[daily]" You’ll also need to set up your Daily API key as an environment variable: DAILY_API_KEY . You can obtain a Daily API key by signing up at Daily . ​ Configuration ​ Constructor Parameters ​ room_url str required Daily room URL ​ token str | None Daily room token ​ bot_name str required Name of the bot in the room ​ params DailyParams default: "DailyParams()" Transport configuration parameters ​ DailyParams Configuration ​ api_url str default: "https://api.daily.co/v1" Daily API endpoint URL ​ api_key str default: "" Daily API key for authentication ​ Audio Output Configuration ​ audio_out_enabled bool default: "False" Enable audio output capabilities ​ audio_out_is_live bool default: "False" Enable live audio streaming mode ​ audio_out_sample_rate int default: "None" Audio output sample rate in Hz ​ audio_out_channels int default: "1" Number of audio output channels ​ audio_out_bitrate int default: "96000" Audio output bitrate in bits per second ​ Audio Input Configuration ​ audio_in_enabled bool default: "False" Enable audio input capabilities ​ audio_in_passthrough bool default: "False" When enabled, incoming audio frames are pushed downstream ​ audio_in_sample_rate int default: "None" Audio input sample rate in Hz ​ audio_in_channels int default: "1" Number of audio input channels ​ audio_in_filter Optional[BaseAudioFilter] default: "None" Audio filter for input processing. Supported filters are: KrispFilter() and NoisereduceFilter() . See the KrispFilter() reference docs and NoisereduceFilter() reference docs ` for more information. ​ Video Output Configuration ​ video_out_enabled bool default: "False" Enable video output capabilities ​ video_out_is_live bool default: "False" Enable live video streaming mode ​ video_out_width int default: "1024" Video output width in pixels ​ video_out_height int default: "768" Video output height in pixels ​ video_out_bitrate int default: "800000" Video output bitrate in bits per second ​ video_out_framerate int default: "30" Video output frame rate ​ video_out_color_format str default: "RGB" Video color format (RGB, BGR, etc.) ​ Voice Activity Detection (VAD) ​ vad_analyzer VADAnalyzer | None default: "None" Voice Activity Detection analyzer. You can set this to either SileroVADAnalyzer() or WebRTCVADAnalyzer() . SileroVADAnalyzer is the recommended option. Learn more about the SileroVADAnalyzer . ​ Feature Settings ​ dialin_settings Optional[DailyDialinSettings] default: "None" Configuration for dial-in functionality ​ transcription_enabled bool default: "False" Enable real-time transcription ​ transcription_settings DailyTranscriptionSettings Configuration for transcription features Copy Ask AI class DailyTranscriptionSettings ( BaseModel ): language: str = "en" # Default language model: str = "nova-2-general" # Transcription model profanity_filter: bool = True # Filter profanity redact: bool = False # Redact sensitive information endpointing: bool = True # Enable speech endpointing punctuate: bool = True # Add punctuation includeRawResponse: bool = True # Include raw API response extra: Mapping[ str , Any] = { "interim_results" : True # Provide any Deepgram-specific settings } ​ Basic Usage Copy Ask AI from pipecat.transports.services.daily import DailyTransport, DailyParams from pipecat.audio.vad.silero import SileroVADAnalyzer # Configure transport transport = DailyTransport( room_url = "https://your-domain.daily.co/room-name" , token = "your-room-token" , bot_name = "AI Assistant" , params = DailyParams( audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer( params = VADParams( stop_secs = 0.5 )), ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), # Handle incoming audio/video stt, # Speech-to-text llm, # Language model tts, # Text-to-speech transport.output() # Handle outgoing audio/video ]) # Use transport event handlers to manage the call lifecycle # Event handler for when the user joins the room: @transport.event_handler ( "on_first_participant_joined" ) async def on_first_participant_joined ( transport , participant ): # Start transcription for the user await transport.capture_participant_transcription(participant[ "id" ]) # Kick off the conversation. await task.queue_frames([context_aggregator.user().get_context_frame()]) # Event handler for when the user leaves the room: @transport.event_handler ( "on_participant_left" ) async def on_participant_left ( transport , participant , reason ): # Cancel the pipeline, which stops processing and removes the bot from the room await task.cancel() ​ Event Callbacks DailyTransport provides a comprehensive callback system for handling various events. Register callbacks using the @transport.event_handler() decorator. ​ Connection Events ​ on_joined async callback Called when the bot successfully joins the room. Parameters: transport : The DailyTransport instance data : Dictionary containing room join information Copy Ask AI @transport.event_handler ( "on_joined" ) async def on_joined ( transport , data : Dict[ str , Any]): logger.info( f "Joined room with data: { data } " ) ​ on_left async callback Called when the bot leaves the room. Parameters: transport : The DailyTransport instance Copy Ask AI @transport.event_handler ( "on_left" ) async def on_left ( transport ): logger.info( "Left room" ) ​ Participant Events ​ on_first_participant_joined async callback Called when the first participant joins the room. Useful for initializing conversations. Parameters: transport : The DailyTransport instance participant : Dictionary containing participant information Copy Ask AI @transport.event_handler ( "on_first_participant_joined" ) async def on_first_participant_joined ( transport , participant : Dict[ str , Any]): await transport.capture_participant_transcription(participant[ "id" ]) await task.queue_frames([LLMMessagesFrame(initial_messages)]) ​ on_participant_joined async callback Called when any participant joins the room. Parameters: transport : The DailyTransport instance participant : Dictionary containing participant information Copy Ask AI @transport.event_handler ( "on_participant_joined" ) async def on_participant_joined ( transport , participant : Dict[ str , Any]): logger.info( f "Participant joined: { participant[ 'id' ] } " ) ​ on_participant_left async callback Called when a participant leaves the room. Parameters: transport : The DailyTransport instance participant : Dictionary containing participant information reason : String describing why the participant left, “leftCall” | “hidden” Copy Ask AI @transport.event_handler ( "on_participant_left" ) async def on_participant_left ( transport , participant : Dict[ str , Any], reason : str ): logger.info( f "Participant { participant[ 'id' ] } left: { reason } " ) # Cancel the pipeline task to stop processing and remove the bot from the Daily room await task.cancel() ​ on_participant_updated async callback Event emitted when a participant is updated. This can mean either the participant’s metadata was updated, or the tracks belonging to the participant changed. Parameters: transport : The DailyTransport instance participant : Dictionary containing updated participant information Copy Ask AI @transport.event_handler ( "on_participant_updated" ) async def on_participant_updated ( transport , participant : Dict[ str , Any]): logger.info( f "Participant updated: { participant } " ) ​ Communication Events ​ on_app_message async callback Event emitted when a custom app message is received from another participant or via the REST API. Parameters: transport : The DailyTransport instance message : The message content (any type) sender : String identifier of the message sender Copy Ask AI @transport.event_handler ( "on_app_message" ) async def on_app_message ( transport , message : Any, sender : str ): logger.info( f "Message from { sender } : { message } " ) ​ on_call_state_updated async callback Event emitted when the call state changes, normally as a consequence of joining or leaving the call. Parameters: transport : The DailyTransport instance state : String representing the new call state. Learn more about call states . Copy Ask AI @transport.event_handler ( "on_call_state_updated" ) async def on_call_state_updated ( transport , state : str ): logger.info( f "Call state updated: { state } " ) ​ Dial Events ​ Dial-in ​ on_dialin_ready async callback Event emitted when dial-in is ready. This happens after the room has connected to the SIP endpoint and the system is ready to receive dial-in calls. Parameters: transport : The DailyTransport instance sip_endpoint : String containing the SIP endpoint information Copy Ask AI @transport.event_handler ( "on_dialin_ready" ) async def on_dialin_ready ( transport , sip_endpoint : str ): logger.info( f "Dial-in ready at: { sip_endpoint } " ) ​ on_dialin_connected async callback Event emitted when the session with the dial-in remote end is established (i.e. SIP endpoint or PSTN are connectd to the Daily room). Note: connected does not mean media (audio or video) has started flowing between the room and PSTN, it means the room received the connection request and both endpoints are negotiating the media flow. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. See DialinConnectedEvent . Copy Ask AI @transport.event_handler ( "on_dialin_connected" ) async def on_dialin_connected ( transport , data : Any): logger.info( f "Dial-in connected: { data } " ) ​ on_dialin_stopped async callback Event emitted when the dial-in remote end disconnects the call. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. See DialinStoppedEvent . Copy Ask AI @transport.event_handler ( "on_dialin_stopped" ) async def on_dialin_stopped ( transport , data : Any): logger.info( f "Dial-in stopped: { data } " ) ​ on_dialin_error async callback Event emitted in the case of dial-in errors which are fatal and the service cannot proceed. For example, an error in SDP negotiation is fatal to the media/SIP pipeline and will result in dialin-error being triggered. Parameters: transport : The DailyTransport instance data : Dictionary containing error information. See DialinEvent . Copy Ask AI @transport.event_handler ( "on_dialin_error" ) async def on_dialin_error ( transport , data : Any): logger.error( f "Dial-in error: { data } " ) ​ on_dialin_warning async callback Event emitted there is a dial-in non-fatal error, such as the selected codec not being used and a fallback codec being utilized. Parameters: transport : The DailyTransport instance data : Dictionary containing warning information. See DialinEvent . Copy Ask AI @transport.event_handler ( "on_dialin_warning" ) async def on_dialin_warning ( transport , data : Any): logger.warning( f "Dial-in warning: { data } " ) ​ Dial-out ​ on_dialout_answered async callback Event emitted when the session with the dial-out remote end is answered. Parameters: transport : The DailyTransport instance data : Dictionary containing call information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_answered" ) async def on_dialout_answered ( transport , data : Any): logger.info( f "Dial-out answered: { data } " ) ​ on_dialout_connected async callback Event emitted when the session with the dial-out remote end is established. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_connected" ) async def on_dialout_connected ( transport , data : Any): logger.info( f "Dial-out connected: { data } " ) ​ on_dialout_stopped async callback Event emitted when the dial-out session is stopped. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_stopped" ) async def on_dialout_stopped ( transport , data : Any): logger.info( f "Dial-out stopped: { data } " ) ​ on_dialout_error async callback Event emitted in the case of dial-out errors which are fatal and the service cannot proceed. For example, an error in SDP negotiation is fatal to the media/SIP pipeline and will result in dialout-error being triggered. Parameters: transport : The DailyTransport instance data : Dictionary containing error information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_error" ) async def on_dialout_error ( transport , data : Any): logger.error( f "Dial-out error: { data } " ) ​ on_dialout_warning async callback Event emitted there is a dial-out non-fatal error, such as the selected codec not being used and a fallback codec being utilized. Parameters: transport : The DailyTransport instance data : Dictionary containing warning information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_warning" ) async def on_dialout_warning ( transport , data : Any): logger.warning( f "Dial-out warning: { data } " ) ​ Transcription Events ​ on_transcription_message async callback Called when a transcription message is received. This includes both interim and final transcriptions. Parameters: transport : The DailyTransport instance message : Dictionary containing transcription data including. Learn more . Copy Ask AI @transport.event_handler ( "on_transcription_message" ) async def on_transcription_message ( transport , message : Dict[ str , Any]): participant_id = message.get( "participantId" ) text = message.get( "text" ) is_final = message[ "rawResponse" ][ "is_final" ] logger.info( f "Transcription from { participant_id } : { text } (final: { is_final } )" ) ​ Recording Events ​ on_recording_started async callback Called when a room recording starts successfully. Parameters: transport : The DailyTransport instance status : Dictionary containing recording status information. Learn more . Copy Ask AI @transport.event_handler ( "on_recording_started" ) async def on_recording_started ( transport , status : Dict[ str , Any]): logger.info( f "Recording started with status: { status } " ) ​ on_recording_stopped async callback Called when a room recording stops. Parameters: transport : The DailyTransport instance stream_id : String identifier of the stopped recording stream Copy Ask AI @transport.event_handler ( "on_recording_stopped" ) async def on_recording_stopped ( transport , stream_id : str ): logger.info( f "Recording stopped for stream: { stream_id } " ) ​ on_recording_error async callback Called when an error occurs during recording. Parameters: transport : The DailyTransport instance stream_id : String identifier of the recording stream message : Error message describing what went wrong Copy Ask AI @transport.event_handler ( "on_recording_error" ) async def on_recording_error ( transport , stream_id : str , message : str ): logger.error( f "Recording error for stream { stream_id } : { message } " ) ​ Error Events ​ on_error async callback Called when a transport error occurs. Parameters: transport : The DailyTransport instance error : String containing error details Copy Ask AI @transport.event_handler ( "on_error" ) async def on_error ( transport , error : str ): logger.error( f "Transport error: { error } " ) ​ Notes on Callbacks All callbacks are asynchronous Callbacks are executed in the order they were registered Multiple handlers can be registered for the same event Exceptions in callbacks are logged but don’t stop the transport Callbacks should be lightweight to avoid blocking the event loop Heavy processing should be offloaded to separate tasks ​ Frame Types ​ Input Frames ​ InputAudioRawFrame Frame Raw audio data from participants ​ UserImageRawFrame Frame Video frames from participants ​ Output Frames ​ OutputAudioRawFrame Frame Audio data to be sent ​ OutputImageRawFrame Frame Video frames to be sent ​ Methods ​ Room Management ​ participants method Returns a dictionary of all participants currently in the room. Copy Ask AI def participants () -> Dict[ str , Any] See participants for more details. ​ participant_counts method Returns participant count statistics for the room. Copy Ask AI def participant_counts () -> Dict[ str , int ] See participant_counts for more details. ​ send_message async method Sends a message to participants in the Daily room. Messages can be directed to all participants or targeted to a specific participant. Copy Ask AI async def send_message ( frame : TransportMessageFrame | TransportMessageUrgentFrame) Parameters: frame : A transport message frame containing the message to send. Can be either: TransportMessageFrame : Standard message TransportMessageUrgentFrame : High-priority message Message Structure: Messages should conform to the expected format for the receiving system. For RTVI messages, use: Copy Ask AI { "label" : "rtvi-ai" , # Required for RTVI protocol "type" : "message-type" , # Defines how the client will process it "id" : "unique-id" , # Unique identifier for the message "data" : { # Custom payload # Your custom data fields } } Example - Standard Message: Copy Ask AI import uuid # Send a standard message to all participants message_data = { "label" : "rtvi-ai" , "type" : "custom-message" , "id" : str (uuid.uuid4()), "data" : { "message" : "Hello everyone!" , "timestamp" : datetime.datetime.now().isoformat() } } await transport.send_message(TransportMessageFrame( message = message_data)) Example - Urgent Message to Specific Participant: Copy Ask AI # Send urgent message to specific participant urgent_data = { "label" : "rtvi-ai" , "type" : "server-message" , "id" : str (uuid.uuid4()), "data" : { "message" : "Private message" , "importance" : "high" } } await transport.send_message( DailyTransportMessageUrgentFrame( message = urgent_data, participant_id = "participant-123" ) ) The message will be delivered to: All participants if no specific participant_id is set Only the specified participant if participant_id is set Through the Daily app messaging system Can be received by registering an on_app_message handler Use TransportMessageUrgentFrame for high-priority messages that need immediate delivery. ​ Media Control ​ send_image async method Sends an image frame to the room. Copy Ask AI async def send_image ( frame : OutputImageRawFrame | SpriteFrame) -> None Parameters: frame : Image frame to send, either raw image data or sprite animation ​ send_audio async method Sends an audio frame to the room. Copy Ask AI async def send_audio ( frame : OutputAudioRawFrame) -> None Parameters: frame : Audio frame to send ​ Video Management ​ capture_participant_video async method Starts capturing video from a specific participant. Copy Ask AI async def capture_participant_video ( participant_id : str , framerate : int = 30 , video_source : str = "camera" , color_format : str = "RGB" ) -> None Parameters: participant_id : ID of the participant to capture framerate : Target frame rate (default: 30) video_source : Video source type (default: “camera”) color_format : Color format of the video (default: “RGB”) To request an image from the camera stream, set the framerate to 0 and push a UserImageRequestFrame . ​ Transcription Control ​ capture_participant_transcription async method Starts capturing and transcribing audio from a specific participant. Copy Ask AI async def capture_participant_transcription ( participant_id : str ) -> None Parameters: participant_id : ID of the participant to transcribe ​ update_transcription async method Updates the transcription configuration for specific participants or instances. Copy Ask AI async def update_transcription ( participants : List[ str ] | None = None , instance_id : str | None = None ) -> None Parameters: participants : Optional list of participant IDs to transcribe. If None, transcribes all participants. instance_id : Optional specific transcription instance ID to update Example: Copy Ask AI # Update transcription for specific participants await transport.update_transcription( participants = [ "participant-123" , "participant-456" ]) # Update specific transcription instance await transport.update_transcription( instance_id = "transcription-789" ) ​ Recording Control ​ start_recording async method Starts recording the room session. Copy Ask AI async def start_recording ( streaming_settings : Dict = None , stream_id : str = None , force_new : bool = None ) -> None Parameters: streaming_settings : Recording configuration settings stream_id : Optional stream identifier force_new : Force start a new recording See start_recording for more details. ​ stop_recording async method Stops an active recording. Copy Ask AI async def stop_recording ( stream_id : str = None ) -> None Parameters: stream_id : Optional stream identifier to stop specific recording See stop_recording for more details. ​ Dial-out Control ​ start_dialout async method Initiates a dial-out call. Copy Ask AI async def start_dialout ( settings : Dict = None ) -> None Parameters: settings : Dial-out configuration settings See start_dialout for more details. ​ stop_dialout async method Stops an active dial-out call. Copy Ask AI async def stop_dialout ( participant_id : str ) -> None Parameters: participant_id : ID of the participant to disconnect See stop_dialout for more details. ​ send_dtmf async method Sends DTMF tones in an existing dial-out session. Copy Ask AI async def send_dtmf ( settings : Dict = None ) -> None Parameters: settings : DTMF settings See send_dtmf for more details. ​ Subscription Management ​ update_subscriptions async method Updates subscriptions and subscription profiles. This function allows you to update subscription profiles and at the same time assign specific subscription profiles to a participant and even change specific settings for some participants. Copy Ask AI async def update_subscriptions ( participant_settings : Dict = None , profile_settings : Dict = None ) -> None Parameters: participant_settings : Per-participant subscription settings profile_settings : Global subscription profile settings See update_subscriptions for more details. ​ Participant Admin When using the DailyTransport, if your bot is given a meeting token with owner or participant admin permissions, it can remote-control other participants’ capabilities and permissions. With this control, the bot can modify settings for how others send and receive media, how they are seen in a session, and whether they have administrative controls. ​ update_remote_participants async method Updates permissions and settings for specific participants in the room. Copy Ask AI async def update_remote_participants ( remote_participants : Mapping[ str , Any] = None ) -> None Parameters: remote_participants : Dictionary mapping participant IDs to their updated settings This method allows fine-grained control over participant capabilities and permissions in the Daily room. Examples: Copy Ask AI # Mute a participant's microphone await transport.update_remote_participants({ "participant-123" : { "inputsEnabled" : { "microphone" : False } } }) # Revoke send permissions (mutes the participant and prevents them from unmuting) await transport.update_remote_participants({ "participant-123" : { "permissions" : { "canSend" : [] # Empty list revokes all send permissions } } }) # Configure selective audio routing await transport.update_remote_participants({ "participant-123" : { # the customer "permissions" : { "canReceive" : { "base" : False , # Don't receive from anyone by default "byUserId" : { "hold-music" : True # Only receive from hold music player } } } } }) # Set up user-to-user permissions mapping (e.g., when transferring calls) # Assumes customer was previously unable to hear anyone (they were on hold) and agent was speaking to bot await transport.update_remote_participants({ "participant-123" : { # the customer "permissions" : { "base" : False , "canReceive" : { "byUserId" : { "agent" : True }} } }, "participant-456" : { # the agent "permissions" : { "base" : False , "canReceive" : { "byUserId" : { "customer" : True }} } } }) # Promote a participant to be a participant admin await transport.update_remote_participants({ "participant-123" : { "permissions" : { "canAdmin" : [ "participants" ] } } }) Permissions Settings: For comprehensive details on permissions settings, see the Daily API documentation: daily-python’s update_remote_participants() documentation daily-js’s updateParticipant() -> updatePermission and participants().permissions ​ Chat messages ​ send_prebuilt_chat_message async method When testing with Daily Prebuilt, you can send messages to the Prebuilt chat using this method. Parameters: message (str): The chat message to send user_name (str): The name of the user sending the message See send_prebuilt_chat_message for more details. ​ Properties ​ participant_id string Returns the transport’s participant ID in the room. ​ room_url string Returns the transport’s room URL for the room currently in use. ​ Notes Supports real-time audio/video communication Handles WebRTC connection management Provides voice activity detection Includes transcription capabilities Manages participant interactions Supports recording and streaming Thread-safe processing All callbacks are asynchronous Multiple handlers can be registered for the same event Exceptions in callbacks are logged but don’t stop the transport Supported Services SmallWebRTCTransport On this page Overview Installation Configuration Constructor Parameters DailyParams Configuration Audio Output Configuration Audio Input Configuration Video Output Configuration Voice Activity Detection (VAD) Feature Settings Basic Usage Event Callbacks Connection Events Participant Events Communication Events Dial Events Dial-in Dial-out Transcription Events Recording Events Error Events Notes on Callbacks Frame Types Input Frames Output Frames Methods Room Management Media Control Video Management Transcription Control Recording Control Dial-out Control Subscription Management Participant Admin Chat messages Properties Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/transport_daily_58c14b51.txt b/transport_daily_58c14b51.txt
new file mode 100644
index 0000000000000000000000000000000000000000..5fd5e63323767c7a1037ee1919f64f8b5b64f84a
--- /dev/null
+++ b/transport_daily_58c14b51.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/transport/daily#param-audio-out-enabled
+Title: Daily WebRTC - Pipecat
+==================================================
+
+Daily WebRTC - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Transport Daily WebRTC Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Daily WebRTC SmallWebRTCTransport FastAPI WebSocket WebSocket Server Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview DailyTransport provides real-time audio and video communication capabilities using Daily’s WebRTC platform. It supports bidirectional audio/video streams, transcription, voice activity detection (VAD), and participant management. ​ Installation To use DailyTransport , install the required dependencies: Copy Ask AI pip install "pipecat-ai[daily]" You’ll also need to set up your Daily API key as an environment variable: DAILY_API_KEY . You can obtain a Daily API key by signing up at Daily . ​ Configuration ​ Constructor Parameters ​ room_url str required Daily room URL ​ token str | None Daily room token ​ bot_name str required Name of the bot in the room ​ params DailyParams default: "DailyParams()" Transport configuration parameters ​ DailyParams Configuration ​ api_url str default: "https://api.daily.co/v1" Daily API endpoint URL ​ api_key str default: "" Daily API key for authentication ​ Audio Output Configuration ​ audio_out_enabled bool default: "False" Enable audio output capabilities ​ audio_out_is_live bool default: "False" Enable live audio streaming mode ​ audio_out_sample_rate int default: "None" Audio output sample rate in Hz ​ audio_out_channels int default: "1" Number of audio output channels ​ audio_out_bitrate int default: "96000" Audio output bitrate in bits per second ​ Audio Input Configuration ​ audio_in_enabled bool default: "False" Enable audio input capabilities ​ audio_in_passthrough bool default: "False" When enabled, incoming audio frames are pushed downstream ​ audio_in_sample_rate int default: "None" Audio input sample rate in Hz ​ audio_in_channels int default: "1" Number of audio input channels ​ audio_in_filter Optional[BaseAudioFilter] default: "None" Audio filter for input processing. Supported filters are: KrispFilter() and NoisereduceFilter() . See the KrispFilter() reference docs and NoisereduceFilter() reference docs ` for more information. ​ Video Output Configuration ​ video_out_enabled bool default: "False" Enable video output capabilities ​ video_out_is_live bool default: "False" Enable live video streaming mode ​ video_out_width int default: "1024" Video output width in pixels ​ video_out_height int default: "768" Video output height in pixels ​ video_out_bitrate int default: "800000" Video output bitrate in bits per second ​ video_out_framerate int default: "30" Video output frame rate ​ video_out_color_format str default: "RGB" Video color format (RGB, BGR, etc.) ​ Voice Activity Detection (VAD) ​ vad_analyzer VADAnalyzer | None default: "None" Voice Activity Detection analyzer. You can set this to either SileroVADAnalyzer() or WebRTCVADAnalyzer() . SileroVADAnalyzer is the recommended option. Learn more about the SileroVADAnalyzer . ​ Feature Settings ​ dialin_settings Optional[DailyDialinSettings] default: "None" Configuration for dial-in functionality ​ transcription_enabled bool default: "False" Enable real-time transcription ​ transcription_settings DailyTranscriptionSettings Configuration for transcription features Copy Ask AI class DailyTranscriptionSettings ( BaseModel ): language: str = "en" # Default language model: str = "nova-2-general" # Transcription model profanity_filter: bool = True # Filter profanity redact: bool = False # Redact sensitive information endpointing: bool = True # Enable speech endpointing punctuate: bool = True # Add punctuation includeRawResponse: bool = True # Include raw API response extra: Mapping[ str , Any] = { "interim_results" : True # Provide any Deepgram-specific settings } ​ Basic Usage Copy Ask AI from pipecat.transports.services.daily import DailyTransport, DailyParams from pipecat.audio.vad.silero import SileroVADAnalyzer # Configure transport transport = DailyTransport( room_url = "https://your-domain.daily.co/room-name" , token = "your-room-token" , bot_name = "AI Assistant" , params = DailyParams( audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer( params = VADParams( stop_secs = 0.5 )), ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), # Handle incoming audio/video stt, # Speech-to-text llm, # Language model tts, # Text-to-speech transport.output() # Handle outgoing audio/video ]) # Use transport event handlers to manage the call lifecycle # Event handler for when the user joins the room: @transport.event_handler ( "on_first_participant_joined" ) async def on_first_participant_joined ( transport , participant ): # Start transcription for the user await transport.capture_participant_transcription(participant[ "id" ]) # Kick off the conversation. await task.queue_frames([context_aggregator.user().get_context_frame()]) # Event handler for when the user leaves the room: @transport.event_handler ( "on_participant_left" ) async def on_participant_left ( transport , participant , reason ): # Cancel the pipeline, which stops processing and removes the bot from the room await task.cancel() ​ Event Callbacks DailyTransport provides a comprehensive callback system for handling various events. Register callbacks using the @transport.event_handler() decorator. ​ Connection Events ​ on_joined async callback Called when the bot successfully joins the room. Parameters: transport : The DailyTransport instance data : Dictionary containing room join information Copy Ask AI @transport.event_handler ( "on_joined" ) async def on_joined ( transport , data : Dict[ str , Any]): logger.info( f "Joined room with data: { data } " ) ​ on_left async callback Called when the bot leaves the room. Parameters: transport : The DailyTransport instance Copy Ask AI @transport.event_handler ( "on_left" ) async def on_left ( transport ): logger.info( "Left room" ) ​ Participant Events ​ on_first_participant_joined async callback Called when the first participant joins the room. Useful for initializing conversations. Parameters: transport : The DailyTransport instance participant : Dictionary containing participant information Copy Ask AI @transport.event_handler ( "on_first_participant_joined" ) async def on_first_participant_joined ( transport , participant : Dict[ str , Any]): await transport.capture_participant_transcription(participant[ "id" ]) await task.queue_frames([LLMMessagesFrame(initial_messages)]) ​ on_participant_joined async callback Called when any participant joins the room. Parameters: transport : The DailyTransport instance participant : Dictionary containing participant information Copy Ask AI @transport.event_handler ( "on_participant_joined" ) async def on_participant_joined ( transport , participant : Dict[ str , Any]): logger.info( f "Participant joined: { participant[ 'id' ] } " ) ​ on_participant_left async callback Called when a participant leaves the room. Parameters: transport : The DailyTransport instance participant : Dictionary containing participant information reason : String describing why the participant left, “leftCall” | “hidden” Copy Ask AI @transport.event_handler ( "on_participant_left" ) async def on_participant_left ( transport , participant : Dict[ str , Any], reason : str ): logger.info( f "Participant { participant[ 'id' ] } left: { reason } " ) # Cancel the pipeline task to stop processing and remove the bot from the Daily room await task.cancel() ​ on_participant_updated async callback Event emitted when a participant is updated. This can mean either the participant’s metadata was updated, or the tracks belonging to the participant changed. Parameters: transport : The DailyTransport instance participant : Dictionary containing updated participant information Copy Ask AI @transport.event_handler ( "on_participant_updated" ) async def on_participant_updated ( transport , participant : Dict[ str , Any]): logger.info( f "Participant updated: { participant } " ) ​ Communication Events ​ on_app_message async callback Event emitted when a custom app message is received from another participant or via the REST API. Parameters: transport : The DailyTransport instance message : The message content (any type) sender : String identifier of the message sender Copy Ask AI @transport.event_handler ( "on_app_message" ) async def on_app_message ( transport , message : Any, sender : str ): logger.info( f "Message from { sender } : { message } " ) ​ on_call_state_updated async callback Event emitted when the call state changes, normally as a consequence of joining or leaving the call. Parameters: transport : The DailyTransport instance state : String representing the new call state. Learn more about call states . Copy Ask AI @transport.event_handler ( "on_call_state_updated" ) async def on_call_state_updated ( transport , state : str ): logger.info( f "Call state updated: { state } " ) ​ Dial Events ​ Dial-in ​ on_dialin_ready async callback Event emitted when dial-in is ready. This happens after the room has connected to the SIP endpoint and the system is ready to receive dial-in calls. Parameters: transport : The DailyTransport instance sip_endpoint : String containing the SIP endpoint information Copy Ask AI @transport.event_handler ( "on_dialin_ready" ) async def on_dialin_ready ( transport , sip_endpoint : str ): logger.info( f "Dial-in ready at: { sip_endpoint } " ) ​ on_dialin_connected async callback Event emitted when the session with the dial-in remote end is established (i.e. SIP endpoint or PSTN are connectd to the Daily room). Note: connected does not mean media (audio or video) has started flowing between the room and PSTN, it means the room received the connection request and both endpoints are negotiating the media flow. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. See DialinConnectedEvent . Copy Ask AI @transport.event_handler ( "on_dialin_connected" ) async def on_dialin_connected ( transport , data : Any): logger.info( f "Dial-in connected: { data } " ) ​ on_dialin_stopped async callback Event emitted when the dial-in remote end disconnects the call. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. See DialinStoppedEvent . Copy Ask AI @transport.event_handler ( "on_dialin_stopped" ) async def on_dialin_stopped ( transport , data : Any): logger.info( f "Dial-in stopped: { data } " ) ​ on_dialin_error async callback Event emitted in the case of dial-in errors which are fatal and the service cannot proceed. For example, an error in SDP negotiation is fatal to the media/SIP pipeline and will result in dialin-error being triggered. Parameters: transport : The DailyTransport instance data : Dictionary containing error information. See DialinEvent . Copy Ask AI @transport.event_handler ( "on_dialin_error" ) async def on_dialin_error ( transport , data : Any): logger.error( f "Dial-in error: { data } " ) ​ on_dialin_warning async callback Event emitted there is a dial-in non-fatal error, such as the selected codec not being used and a fallback codec being utilized. Parameters: transport : The DailyTransport instance data : Dictionary containing warning information. See DialinEvent . Copy Ask AI @transport.event_handler ( "on_dialin_warning" ) async def on_dialin_warning ( transport , data : Any): logger.warning( f "Dial-in warning: { data } " ) ​ Dial-out ​ on_dialout_answered async callback Event emitted when the session with the dial-out remote end is answered. Parameters: transport : The DailyTransport instance data : Dictionary containing call information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_answered" ) async def on_dialout_answered ( transport , data : Any): logger.info( f "Dial-out answered: { data } " ) ​ on_dialout_connected async callback Event emitted when the session with the dial-out remote end is established. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_connected" ) async def on_dialout_connected ( transport , data : Any): logger.info( f "Dial-out connected: { data } " ) ​ on_dialout_stopped async callback Event emitted when the dial-out session is stopped. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_stopped" ) async def on_dialout_stopped ( transport , data : Any): logger.info( f "Dial-out stopped: { data } " ) ​ on_dialout_error async callback Event emitted in the case of dial-out errors which are fatal and the service cannot proceed. For example, an error in SDP negotiation is fatal to the media/SIP pipeline and will result in dialout-error being triggered. Parameters: transport : The DailyTransport instance data : Dictionary containing error information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_error" ) async def on_dialout_error ( transport , data : Any): logger.error( f "Dial-out error: { data } " ) ​ on_dialout_warning async callback Event emitted there is a dial-out non-fatal error, such as the selected codec not being used and a fallback codec being utilized. Parameters: transport : The DailyTransport instance data : Dictionary containing warning information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_warning" ) async def on_dialout_warning ( transport , data : Any): logger.warning( f "Dial-out warning: { data } " ) ​ Transcription Events ​ on_transcription_message async callback Called when a transcription message is received. This includes both interim and final transcriptions. Parameters: transport : The DailyTransport instance message : Dictionary containing transcription data including. Learn more . Copy Ask AI @transport.event_handler ( "on_transcription_message" ) async def on_transcription_message ( transport , message : Dict[ str , Any]): participant_id = message.get( "participantId" ) text = message.get( "text" ) is_final = message[ "rawResponse" ][ "is_final" ] logger.info( f "Transcription from { participant_id } : { text } (final: { is_final } )" ) ​ Recording Events ​ on_recording_started async callback Called when a room recording starts successfully. Parameters: transport : The DailyTransport instance status : Dictionary containing recording status information. Learn more . Copy Ask AI @transport.event_handler ( "on_recording_started" ) async def on_recording_started ( transport , status : Dict[ str , Any]): logger.info( f "Recording started with status: { status } " ) ​ on_recording_stopped async callback Called when a room recording stops. Parameters: transport : The DailyTransport instance stream_id : String identifier of the stopped recording stream Copy Ask AI @transport.event_handler ( "on_recording_stopped" ) async def on_recording_stopped ( transport , stream_id : str ): logger.info( f "Recording stopped for stream: { stream_id } " ) ​ on_recording_error async callback Called when an error occurs during recording. Parameters: transport : The DailyTransport instance stream_id : String identifier of the recording stream message : Error message describing what went wrong Copy Ask AI @transport.event_handler ( "on_recording_error" ) async def on_recording_error ( transport , stream_id : str , message : str ): logger.error( f "Recording error for stream { stream_id } : { message } " ) ​ Error Events ​ on_error async callback Called when a transport error occurs. Parameters: transport : The DailyTransport instance error : String containing error details Copy Ask AI @transport.event_handler ( "on_error" ) async def on_error ( transport , error : str ): logger.error( f "Transport error: { error } " ) ​ Notes on Callbacks All callbacks are asynchronous Callbacks are executed in the order they were registered Multiple handlers can be registered for the same event Exceptions in callbacks are logged but don’t stop the transport Callbacks should be lightweight to avoid blocking the event loop Heavy processing should be offloaded to separate tasks ​ Frame Types ​ Input Frames ​ InputAudioRawFrame Frame Raw audio data from participants ​ UserImageRawFrame Frame Video frames from participants ​ Output Frames ​ OutputAudioRawFrame Frame Audio data to be sent ​ OutputImageRawFrame Frame Video frames to be sent ​ Methods ​ Room Management ​ participants method Returns a dictionary of all participants currently in the room. Copy Ask AI def participants () -> Dict[ str , Any] See participants for more details. ​ participant_counts method Returns participant count statistics for the room. Copy Ask AI def participant_counts () -> Dict[ str , int ] See participant_counts for more details. ​ send_message async method Sends a message to participants in the Daily room. Messages can be directed to all participants or targeted to a specific participant. Copy Ask AI async def send_message ( frame : TransportMessageFrame | TransportMessageUrgentFrame) Parameters: frame : A transport message frame containing the message to send. Can be either: TransportMessageFrame : Standard message TransportMessageUrgentFrame : High-priority message Message Structure: Messages should conform to the expected format for the receiving system. For RTVI messages, use: Copy Ask AI { "label" : "rtvi-ai" , # Required for RTVI protocol "type" : "message-type" , # Defines how the client will process it "id" : "unique-id" , # Unique identifier for the message "data" : { # Custom payload # Your custom data fields } } Example - Standard Message: Copy Ask AI import uuid # Send a standard message to all participants message_data = { "label" : "rtvi-ai" , "type" : "custom-message" , "id" : str (uuid.uuid4()), "data" : { "message" : "Hello everyone!" , "timestamp" : datetime.datetime.now().isoformat() } } await transport.send_message(TransportMessageFrame( message = message_data)) Example - Urgent Message to Specific Participant: Copy Ask AI # Send urgent message to specific participant urgent_data = { "label" : "rtvi-ai" , "type" : "server-message" , "id" : str (uuid.uuid4()), "data" : { "message" : "Private message" , "importance" : "high" } } await transport.send_message( DailyTransportMessageUrgentFrame( message = urgent_data, participant_id = "participant-123" ) ) The message will be delivered to: All participants if no specific participant_id is set Only the specified participant if participant_id is set Through the Daily app messaging system Can be received by registering an on_app_message handler Use TransportMessageUrgentFrame for high-priority messages that need immediate delivery. ​ Media Control ​ send_image async method Sends an image frame to the room. Copy Ask AI async def send_image ( frame : OutputImageRawFrame | SpriteFrame) -> None Parameters: frame : Image frame to send, either raw image data or sprite animation ​ send_audio async method Sends an audio frame to the room. Copy Ask AI async def send_audio ( frame : OutputAudioRawFrame) -> None Parameters: frame : Audio frame to send ​ Video Management ​ capture_participant_video async method Starts capturing video from a specific participant. Copy Ask AI async def capture_participant_video ( participant_id : str , framerate : int = 30 , video_source : str = "camera" , color_format : str = "RGB" ) -> None Parameters: participant_id : ID of the participant to capture framerate : Target frame rate (default: 30) video_source : Video source type (default: “camera”) color_format : Color format of the video (default: “RGB”) To request an image from the camera stream, set the framerate to 0 and push a UserImageRequestFrame . ​ Transcription Control ​ capture_participant_transcription async method Starts capturing and transcribing audio from a specific participant. Copy Ask AI async def capture_participant_transcription ( participant_id : str ) -> None Parameters: participant_id : ID of the participant to transcribe ​ update_transcription async method Updates the transcription configuration for specific participants or instances. Copy Ask AI async def update_transcription ( participants : List[ str ] | None = None , instance_id : str | None = None ) -> None Parameters: participants : Optional list of participant IDs to transcribe. If None, transcribes all participants. instance_id : Optional specific transcription instance ID to update Example: Copy Ask AI # Update transcription for specific participants await transport.update_transcription( participants = [ "participant-123" , "participant-456" ]) # Update specific transcription instance await transport.update_transcription( instance_id = "transcription-789" ) ​ Recording Control ​ start_recording async method Starts recording the room session. Copy Ask AI async def start_recording ( streaming_settings : Dict = None , stream_id : str = None , force_new : bool = None ) -> None Parameters: streaming_settings : Recording configuration settings stream_id : Optional stream identifier force_new : Force start a new recording See start_recording for more details. ​ stop_recording async method Stops an active recording. Copy Ask AI async def stop_recording ( stream_id : str = None ) -> None Parameters: stream_id : Optional stream identifier to stop specific recording See stop_recording for more details. ​ Dial-out Control ​ start_dialout async method Initiates a dial-out call. Copy Ask AI async def start_dialout ( settings : Dict = None ) -> None Parameters: settings : Dial-out configuration settings See start_dialout for more details. ​ stop_dialout async method Stops an active dial-out call. Copy Ask AI async def stop_dialout ( participant_id : str ) -> None Parameters: participant_id : ID of the participant to disconnect See stop_dialout for more details. ​ send_dtmf async method Sends DTMF tones in an existing dial-out session. Copy Ask AI async def send_dtmf ( settings : Dict = None ) -> None Parameters: settings : DTMF settings See send_dtmf for more details. ​ Subscription Management ​ update_subscriptions async method Updates subscriptions and subscription profiles. This function allows you to update subscription profiles and at the same time assign specific subscription profiles to a participant and even change specific settings for some participants. Copy Ask AI async def update_subscriptions ( participant_settings : Dict = None , profile_settings : Dict = None ) -> None Parameters: participant_settings : Per-participant subscription settings profile_settings : Global subscription profile settings See update_subscriptions for more details. ​ Participant Admin When using the DailyTransport, if your bot is given a meeting token with owner or participant admin permissions, it can remote-control other participants’ capabilities and permissions. With this control, the bot can modify settings for how others send and receive media, how they are seen in a session, and whether they have administrative controls. ​ update_remote_participants async method Updates permissions and settings for specific participants in the room. Copy Ask AI async def update_remote_participants ( remote_participants : Mapping[ str , Any] = None ) -> None Parameters: remote_participants : Dictionary mapping participant IDs to their updated settings This method allows fine-grained control over participant capabilities and permissions in the Daily room. Examples: Copy Ask AI # Mute a participant's microphone await transport.update_remote_participants({ "participant-123" : { "inputsEnabled" : { "microphone" : False } } }) # Revoke send permissions (mutes the participant and prevents them from unmuting) await transport.update_remote_participants({ "participant-123" : { "permissions" : { "canSend" : [] # Empty list revokes all send permissions } } }) # Configure selective audio routing await transport.update_remote_participants({ "participant-123" : { # the customer "permissions" : { "canReceive" : { "base" : False , # Don't receive from anyone by default "byUserId" : { "hold-music" : True # Only receive from hold music player } } } } }) # Set up user-to-user permissions mapping (e.g., when transferring calls) # Assumes customer was previously unable to hear anyone (they were on hold) and agent was speaking to bot await transport.update_remote_participants({ "participant-123" : { # the customer "permissions" : { "base" : False , "canReceive" : { "byUserId" : { "agent" : True }} } }, "participant-456" : { # the agent "permissions" : { "base" : False , "canReceive" : { "byUserId" : { "customer" : True }} } } }) # Promote a participant to be a participant admin await transport.update_remote_participants({ "participant-123" : { "permissions" : { "canAdmin" : [ "participants" ] } } }) Permissions Settings: For comprehensive details on permissions settings, see the Daily API documentation: daily-python’s update_remote_participants() documentation daily-js’s updateParticipant() -> updatePermission and participants().permissions ​ Chat messages ​ send_prebuilt_chat_message async method When testing with Daily Prebuilt, you can send messages to the Prebuilt chat using this method. Parameters: message (str): The chat message to send user_name (str): The name of the user sending the message See send_prebuilt_chat_message for more details. ​ Properties ​ participant_id string Returns the transport’s participant ID in the room. ​ room_url string Returns the transport’s room URL for the room currently in use. ​ Notes Supports real-time audio/video communication Handles WebRTC connection management Provides voice activity detection Includes transcription capabilities Manages participant interactions Supports recording and streaming Thread-safe processing All callbacks are asynchronous Multiple handlers can be registered for the same event Exceptions in callbacks are logged but don’t stop the transport Supported Services SmallWebRTCTransport On this page Overview Installation Configuration Constructor Parameters DailyParams Configuration Audio Output Configuration Audio Input Configuration Video Output Configuration Voice Activity Detection (VAD) Feature Settings Basic Usage Event Callbacks Connection Events Participant Events Communication Events Dial Events Dial-in Dial-out Transcription Events Recording Events Error Events Notes on Callbacks Frame Types Input Frames Output Frames Methods Room Management Media Control Video Management Transcription Control Recording Control Dial-out Control Subscription Management Participant Admin Chat messages Properties Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/transport_daily_5a4c8af2.txt b/transport_daily_5a4c8af2.txt
new file mode 100644
index 0000000000000000000000000000000000000000..748f0d05ffa7a271737806b2cf4f13788340d403
--- /dev/null
+++ b/transport_daily_5a4c8af2.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/transport/daily#param-on-error
+Title: Daily WebRTC - Pipecat
+==================================================
+
+Daily WebRTC - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Transport Daily WebRTC Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Daily WebRTC SmallWebRTCTransport FastAPI WebSocket WebSocket Server Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview DailyTransport provides real-time audio and video communication capabilities using Daily’s WebRTC platform. It supports bidirectional audio/video streams, transcription, voice activity detection (VAD), and participant management. ​ Installation To use DailyTransport , install the required dependencies: Copy Ask AI pip install "pipecat-ai[daily]" You’ll also need to set up your Daily API key as an environment variable: DAILY_API_KEY . You can obtain a Daily API key by signing up at Daily . ​ Configuration ​ Constructor Parameters ​ room_url str required Daily room URL ​ token str | None Daily room token ​ bot_name str required Name of the bot in the room ​ params DailyParams default: "DailyParams()" Transport configuration parameters ​ DailyParams Configuration ​ api_url str default: "https://api.daily.co/v1" Daily API endpoint URL ​ api_key str default: "" Daily API key for authentication ​ Audio Output Configuration ​ audio_out_enabled bool default: "False" Enable audio output capabilities ​ audio_out_is_live bool default: "False" Enable live audio streaming mode ​ audio_out_sample_rate int default: "None" Audio output sample rate in Hz ​ audio_out_channels int default: "1" Number of audio output channels ​ audio_out_bitrate int default: "96000" Audio output bitrate in bits per second ​ Audio Input Configuration ​ audio_in_enabled bool default: "False" Enable audio input capabilities ​ audio_in_passthrough bool default: "False" When enabled, incoming audio frames are pushed downstream ​ audio_in_sample_rate int default: "None" Audio input sample rate in Hz ​ audio_in_channels int default: "1" Number of audio input channels ​ audio_in_filter Optional[BaseAudioFilter] default: "None" Audio filter for input processing. Supported filters are: KrispFilter() and NoisereduceFilter() . See the KrispFilter() reference docs and NoisereduceFilter() reference docs ` for more information. ​ Video Output Configuration ​ video_out_enabled bool default: "False" Enable video output capabilities ​ video_out_is_live bool default: "False" Enable live video streaming mode ​ video_out_width int default: "1024" Video output width in pixels ​ video_out_height int default: "768" Video output height in pixels ​ video_out_bitrate int default: "800000" Video output bitrate in bits per second ​ video_out_framerate int default: "30" Video output frame rate ​ video_out_color_format str default: "RGB" Video color format (RGB, BGR, etc.) ​ Voice Activity Detection (VAD) ​ vad_analyzer VADAnalyzer | None default: "None" Voice Activity Detection analyzer. You can set this to either SileroVADAnalyzer() or WebRTCVADAnalyzer() . SileroVADAnalyzer is the recommended option. Learn more about the SileroVADAnalyzer . ​ Feature Settings ​ dialin_settings Optional[DailyDialinSettings] default: "None" Configuration for dial-in functionality ​ transcription_enabled bool default: "False" Enable real-time transcription ​ transcription_settings DailyTranscriptionSettings Configuration for transcription features Copy Ask AI class DailyTranscriptionSettings ( BaseModel ): language: str = "en" # Default language model: str = "nova-2-general" # Transcription model profanity_filter: bool = True # Filter profanity redact: bool = False # Redact sensitive information endpointing: bool = True # Enable speech endpointing punctuate: bool = True # Add punctuation includeRawResponse: bool = True # Include raw API response extra: Mapping[ str , Any] = { "interim_results" : True # Provide any Deepgram-specific settings } ​ Basic Usage Copy Ask AI from pipecat.transports.services.daily import DailyTransport, DailyParams from pipecat.audio.vad.silero import SileroVADAnalyzer # Configure transport transport = DailyTransport( room_url = "https://your-domain.daily.co/room-name" , token = "your-room-token" , bot_name = "AI Assistant" , params = DailyParams( audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer( params = VADParams( stop_secs = 0.5 )), ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), # Handle incoming audio/video stt, # Speech-to-text llm, # Language model tts, # Text-to-speech transport.output() # Handle outgoing audio/video ]) # Use transport event handlers to manage the call lifecycle # Event handler for when the user joins the room: @transport.event_handler ( "on_first_participant_joined" ) async def on_first_participant_joined ( transport , participant ): # Start transcription for the user await transport.capture_participant_transcription(participant[ "id" ]) # Kick off the conversation. await task.queue_frames([context_aggregator.user().get_context_frame()]) # Event handler for when the user leaves the room: @transport.event_handler ( "on_participant_left" ) async def on_participant_left ( transport , participant , reason ): # Cancel the pipeline, which stops processing and removes the bot from the room await task.cancel() ​ Event Callbacks DailyTransport provides a comprehensive callback system for handling various events. Register callbacks using the @transport.event_handler() decorator. ​ Connection Events ​ on_joined async callback Called when the bot successfully joins the room. Parameters: transport : The DailyTransport instance data : Dictionary containing room join information Copy Ask AI @transport.event_handler ( "on_joined" ) async def on_joined ( transport , data : Dict[ str , Any]): logger.info( f "Joined room with data: { data } " ) ​ on_left async callback Called when the bot leaves the room. Parameters: transport : The DailyTransport instance Copy Ask AI @transport.event_handler ( "on_left" ) async def on_left ( transport ): logger.info( "Left room" ) ​ Participant Events ​ on_first_participant_joined async callback Called when the first participant joins the room. Useful for initializing conversations. Parameters: transport : The DailyTransport instance participant : Dictionary containing participant information Copy Ask AI @transport.event_handler ( "on_first_participant_joined" ) async def on_first_participant_joined ( transport , participant : Dict[ str , Any]): await transport.capture_participant_transcription(participant[ "id" ]) await task.queue_frames([LLMMessagesFrame(initial_messages)]) ​ on_participant_joined async callback Called when any participant joins the room. Parameters: transport : The DailyTransport instance participant : Dictionary containing participant information Copy Ask AI @transport.event_handler ( "on_participant_joined" ) async def on_participant_joined ( transport , participant : Dict[ str , Any]): logger.info( f "Participant joined: { participant[ 'id' ] } " ) ​ on_participant_left async callback Called when a participant leaves the room. Parameters: transport : The DailyTransport instance participant : Dictionary containing participant information reason : String describing why the participant left, “leftCall” | “hidden” Copy Ask AI @transport.event_handler ( "on_participant_left" ) async def on_participant_left ( transport , participant : Dict[ str , Any], reason : str ): logger.info( f "Participant { participant[ 'id' ] } left: { reason } " ) # Cancel the pipeline task to stop processing and remove the bot from the Daily room await task.cancel() ​ on_participant_updated async callback Event emitted when a participant is updated. This can mean either the participant’s metadata was updated, or the tracks belonging to the participant changed. Parameters: transport : The DailyTransport instance participant : Dictionary containing updated participant information Copy Ask AI @transport.event_handler ( "on_participant_updated" ) async def on_participant_updated ( transport , participant : Dict[ str , Any]): logger.info( f "Participant updated: { participant } " ) ​ Communication Events ​ on_app_message async callback Event emitted when a custom app message is received from another participant or via the REST API. Parameters: transport : The DailyTransport instance message : The message content (any type) sender : String identifier of the message sender Copy Ask AI @transport.event_handler ( "on_app_message" ) async def on_app_message ( transport , message : Any, sender : str ): logger.info( f "Message from { sender } : { message } " ) ​ on_call_state_updated async callback Event emitted when the call state changes, normally as a consequence of joining or leaving the call. Parameters: transport : The DailyTransport instance state : String representing the new call state. Learn more about call states . Copy Ask AI @transport.event_handler ( "on_call_state_updated" ) async def on_call_state_updated ( transport , state : str ): logger.info( f "Call state updated: { state } " ) ​ Dial Events ​ Dial-in ​ on_dialin_ready async callback Event emitted when dial-in is ready. This happens after the room has connected to the SIP endpoint and the system is ready to receive dial-in calls. Parameters: transport : The DailyTransport instance sip_endpoint : String containing the SIP endpoint information Copy Ask AI @transport.event_handler ( "on_dialin_ready" ) async def on_dialin_ready ( transport , sip_endpoint : str ): logger.info( f "Dial-in ready at: { sip_endpoint } " ) ​ on_dialin_connected async callback Event emitted when the session with the dial-in remote end is established (i.e. SIP endpoint or PSTN are connectd to the Daily room). Note: connected does not mean media (audio or video) has started flowing between the room and PSTN, it means the room received the connection request and both endpoints are negotiating the media flow. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. See DialinConnectedEvent . Copy Ask AI @transport.event_handler ( "on_dialin_connected" ) async def on_dialin_connected ( transport , data : Any): logger.info( f "Dial-in connected: { data } " ) ​ on_dialin_stopped async callback Event emitted when the dial-in remote end disconnects the call. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. See DialinStoppedEvent . Copy Ask AI @transport.event_handler ( "on_dialin_stopped" ) async def on_dialin_stopped ( transport , data : Any): logger.info( f "Dial-in stopped: { data } " ) ​ on_dialin_error async callback Event emitted in the case of dial-in errors which are fatal and the service cannot proceed. For example, an error in SDP negotiation is fatal to the media/SIP pipeline and will result in dialin-error being triggered. Parameters: transport : The DailyTransport instance data : Dictionary containing error information. See DialinEvent . Copy Ask AI @transport.event_handler ( "on_dialin_error" ) async def on_dialin_error ( transport , data : Any): logger.error( f "Dial-in error: { data } " ) ​ on_dialin_warning async callback Event emitted there is a dial-in non-fatal error, such as the selected codec not being used and a fallback codec being utilized. Parameters: transport : The DailyTransport instance data : Dictionary containing warning information. See DialinEvent . Copy Ask AI @transport.event_handler ( "on_dialin_warning" ) async def on_dialin_warning ( transport , data : Any): logger.warning( f "Dial-in warning: { data } " ) ​ Dial-out ​ on_dialout_answered async callback Event emitted when the session with the dial-out remote end is answered. Parameters: transport : The DailyTransport instance data : Dictionary containing call information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_answered" ) async def on_dialout_answered ( transport , data : Any): logger.info( f "Dial-out answered: { data } " ) ​ on_dialout_connected async callback Event emitted when the session with the dial-out remote end is established. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_connected" ) async def on_dialout_connected ( transport , data : Any): logger.info( f "Dial-out connected: { data } " ) ​ on_dialout_stopped async callback Event emitted when the dial-out session is stopped. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_stopped" ) async def on_dialout_stopped ( transport , data : Any): logger.info( f "Dial-out stopped: { data } " ) ​ on_dialout_error async callback Event emitted in the case of dial-out errors which are fatal and the service cannot proceed. For example, an error in SDP negotiation is fatal to the media/SIP pipeline and will result in dialout-error being triggered. Parameters: transport : The DailyTransport instance data : Dictionary containing error information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_error" ) async def on_dialout_error ( transport , data : Any): logger.error( f "Dial-out error: { data } " ) ​ on_dialout_warning async callback Event emitted there is a dial-out non-fatal error, such as the selected codec not being used and a fallback codec being utilized. Parameters: transport : The DailyTransport instance data : Dictionary containing warning information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_warning" ) async def on_dialout_warning ( transport , data : Any): logger.warning( f "Dial-out warning: { data } " ) ​ Transcription Events ​ on_transcription_message async callback Called when a transcription message is received. This includes both interim and final transcriptions. Parameters: transport : The DailyTransport instance message : Dictionary containing transcription data including. Learn more . Copy Ask AI @transport.event_handler ( "on_transcription_message" ) async def on_transcription_message ( transport , message : Dict[ str , Any]): participant_id = message.get( "participantId" ) text = message.get( "text" ) is_final = message[ "rawResponse" ][ "is_final" ] logger.info( f "Transcription from { participant_id } : { text } (final: { is_final } )" ) ​ Recording Events ​ on_recording_started async callback Called when a room recording starts successfully. Parameters: transport : The DailyTransport instance status : Dictionary containing recording status information. Learn more . Copy Ask AI @transport.event_handler ( "on_recording_started" ) async def on_recording_started ( transport , status : Dict[ str , Any]): logger.info( f "Recording started with status: { status } " ) ​ on_recording_stopped async callback Called when a room recording stops. Parameters: transport : The DailyTransport instance stream_id : String identifier of the stopped recording stream Copy Ask AI @transport.event_handler ( "on_recording_stopped" ) async def on_recording_stopped ( transport , stream_id : str ): logger.info( f "Recording stopped for stream: { stream_id } " ) ​ on_recording_error async callback Called when an error occurs during recording. Parameters: transport : The DailyTransport instance stream_id : String identifier of the recording stream message : Error message describing what went wrong Copy Ask AI @transport.event_handler ( "on_recording_error" ) async def on_recording_error ( transport , stream_id : str , message : str ): logger.error( f "Recording error for stream { stream_id } : { message } " ) ​ Error Events ​ on_error async callback Called when a transport error occurs. Parameters: transport : The DailyTransport instance error : String containing error details Copy Ask AI @transport.event_handler ( "on_error" ) async def on_error ( transport , error : str ): logger.error( f "Transport error: { error } " ) ​ Notes on Callbacks All callbacks are asynchronous Callbacks are executed in the order they were registered Multiple handlers can be registered for the same event Exceptions in callbacks are logged but don’t stop the transport Callbacks should be lightweight to avoid blocking the event loop Heavy processing should be offloaded to separate tasks ​ Frame Types ​ Input Frames ​ InputAudioRawFrame Frame Raw audio data from participants ​ UserImageRawFrame Frame Video frames from participants ​ Output Frames ​ OutputAudioRawFrame Frame Audio data to be sent ​ OutputImageRawFrame Frame Video frames to be sent ​ Methods ​ Room Management ​ participants method Returns a dictionary of all participants currently in the room. Copy Ask AI def participants () -> Dict[ str , Any] See participants for more details. ​ participant_counts method Returns participant count statistics for the room. Copy Ask AI def participant_counts () -> Dict[ str , int ] See participant_counts for more details. ​ send_message async method Sends a message to participants in the Daily room. Messages can be directed to all participants or targeted to a specific participant. Copy Ask AI async def send_message ( frame : TransportMessageFrame | TransportMessageUrgentFrame) Parameters: frame : A transport message frame containing the message to send. Can be either: TransportMessageFrame : Standard message TransportMessageUrgentFrame : High-priority message Message Structure: Messages should conform to the expected format for the receiving system. For RTVI messages, use: Copy Ask AI { "label" : "rtvi-ai" , # Required for RTVI protocol "type" : "message-type" , # Defines how the client will process it "id" : "unique-id" , # Unique identifier for the message "data" : { # Custom payload # Your custom data fields } } Example - Standard Message: Copy Ask AI import uuid # Send a standard message to all participants message_data = { "label" : "rtvi-ai" , "type" : "custom-message" , "id" : str (uuid.uuid4()), "data" : { "message" : "Hello everyone!" , "timestamp" : datetime.datetime.now().isoformat() } } await transport.send_message(TransportMessageFrame( message = message_data)) Example - Urgent Message to Specific Participant: Copy Ask AI # Send urgent message to specific participant urgent_data = { "label" : "rtvi-ai" , "type" : "server-message" , "id" : str (uuid.uuid4()), "data" : { "message" : "Private message" , "importance" : "high" } } await transport.send_message( DailyTransportMessageUrgentFrame( message = urgent_data, participant_id = "participant-123" ) ) The message will be delivered to: All participants if no specific participant_id is set Only the specified participant if participant_id is set Through the Daily app messaging system Can be received by registering an on_app_message handler Use TransportMessageUrgentFrame for high-priority messages that need immediate delivery. ​ Media Control ​ send_image async method Sends an image frame to the room. Copy Ask AI async def send_image ( frame : OutputImageRawFrame | SpriteFrame) -> None Parameters: frame : Image frame to send, either raw image data or sprite animation ​ send_audio async method Sends an audio frame to the room. Copy Ask AI async def send_audio ( frame : OutputAudioRawFrame) -> None Parameters: frame : Audio frame to send ​ Video Management ​ capture_participant_video async method Starts capturing video from a specific participant. Copy Ask AI async def capture_participant_video ( participant_id : str , framerate : int = 30 , video_source : str = "camera" , color_format : str = "RGB" ) -> None Parameters: participant_id : ID of the participant to capture framerate : Target frame rate (default: 30) video_source : Video source type (default: “camera”) color_format : Color format of the video (default: “RGB”) To request an image from the camera stream, set the framerate to 0 and push a UserImageRequestFrame . ​ Transcription Control ​ capture_participant_transcription async method Starts capturing and transcribing audio from a specific participant. Copy Ask AI async def capture_participant_transcription ( participant_id : str ) -> None Parameters: participant_id : ID of the participant to transcribe ​ update_transcription async method Updates the transcription configuration for specific participants or instances. Copy Ask AI async def update_transcription ( participants : List[ str ] | None = None , instance_id : str | None = None ) -> None Parameters: participants : Optional list of participant IDs to transcribe. If None, transcribes all participants. instance_id : Optional specific transcription instance ID to update Example: Copy Ask AI # Update transcription for specific participants await transport.update_transcription( participants = [ "participant-123" , "participant-456" ]) # Update specific transcription instance await transport.update_transcription( instance_id = "transcription-789" ) ​ Recording Control ​ start_recording async method Starts recording the room session. Copy Ask AI async def start_recording ( streaming_settings : Dict = None , stream_id : str = None , force_new : bool = None ) -> None Parameters: streaming_settings : Recording configuration settings stream_id : Optional stream identifier force_new : Force start a new recording See start_recording for more details. ​ stop_recording async method Stops an active recording. Copy Ask AI async def stop_recording ( stream_id : str = None ) -> None Parameters: stream_id : Optional stream identifier to stop specific recording See stop_recording for more details. ​ Dial-out Control ​ start_dialout async method Initiates a dial-out call. Copy Ask AI async def start_dialout ( settings : Dict = None ) -> None Parameters: settings : Dial-out configuration settings See start_dialout for more details. ​ stop_dialout async method Stops an active dial-out call. Copy Ask AI async def stop_dialout ( participant_id : str ) -> None Parameters: participant_id : ID of the participant to disconnect See stop_dialout for more details. ​ send_dtmf async method Sends DTMF tones in an existing dial-out session. Copy Ask AI async def send_dtmf ( settings : Dict = None ) -> None Parameters: settings : DTMF settings See send_dtmf for more details. ​ Subscription Management ​ update_subscriptions async method Updates subscriptions and subscription profiles. This function allows you to update subscription profiles and at the same time assign specific subscription profiles to a participant and even change specific settings for some participants. Copy Ask AI async def update_subscriptions ( participant_settings : Dict = None , profile_settings : Dict = None ) -> None Parameters: participant_settings : Per-participant subscription settings profile_settings : Global subscription profile settings See update_subscriptions for more details. ​ Participant Admin When using the DailyTransport, if your bot is given a meeting token with owner or participant admin permissions, it can remote-control other participants’ capabilities and permissions. With this control, the bot can modify settings for how others send and receive media, how they are seen in a session, and whether they have administrative controls. ​ update_remote_participants async method Updates permissions and settings for specific participants in the room. Copy Ask AI async def update_remote_participants ( remote_participants : Mapping[ str , Any] = None ) -> None Parameters: remote_participants : Dictionary mapping participant IDs to their updated settings This method allows fine-grained control over participant capabilities and permissions in the Daily room. Examples: Copy Ask AI # Mute a participant's microphone await transport.update_remote_participants({ "participant-123" : { "inputsEnabled" : { "microphone" : False } } }) # Revoke send permissions (mutes the participant and prevents them from unmuting) await transport.update_remote_participants({ "participant-123" : { "permissions" : { "canSend" : [] # Empty list revokes all send permissions } } }) # Configure selective audio routing await transport.update_remote_participants({ "participant-123" : { # the customer "permissions" : { "canReceive" : { "base" : False , # Don't receive from anyone by default "byUserId" : { "hold-music" : True # Only receive from hold music player } } } } }) # Set up user-to-user permissions mapping (e.g., when transferring calls) # Assumes customer was previously unable to hear anyone (they were on hold) and agent was speaking to bot await transport.update_remote_participants({ "participant-123" : { # the customer "permissions" : { "base" : False , "canReceive" : { "byUserId" : { "agent" : True }} } }, "participant-456" : { # the agent "permissions" : { "base" : False , "canReceive" : { "byUserId" : { "customer" : True }} } } }) # Promote a participant to be a participant admin await transport.update_remote_participants({ "participant-123" : { "permissions" : { "canAdmin" : [ "participants" ] } } }) Permissions Settings: For comprehensive details on permissions settings, see the Daily API documentation: daily-python’s update_remote_participants() documentation daily-js’s updateParticipant() -> updatePermission and participants().permissions ​ Chat messages ​ send_prebuilt_chat_message async method When testing with Daily Prebuilt, you can send messages to the Prebuilt chat using this method. Parameters: message (str): The chat message to send user_name (str): The name of the user sending the message See send_prebuilt_chat_message for more details. ​ Properties ​ participant_id string Returns the transport’s participant ID in the room. ​ room_url string Returns the transport’s room URL for the room currently in use. ​ Notes Supports real-time audio/video communication Handles WebRTC connection management Provides voice activity detection Includes transcription capabilities Manages participant interactions Supports recording and streaming Thread-safe processing All callbacks are asynchronous Multiple handlers can be registered for the same event Exceptions in callbacks are logged but don’t stop the transport Supported Services SmallWebRTCTransport On this page Overview Installation Configuration Constructor Parameters DailyParams Configuration Audio Output Configuration Audio Input Configuration Video Output Configuration Voice Activity Detection (VAD) Feature Settings Basic Usage Event Callbacks Connection Events Participant Events Communication Events Dial Events Dial-in Dial-out Transcription Events Recording Events Error Events Notes on Callbacks Frame Types Input Frames Output Frames Methods Room Management Media Control Video Management Transcription Control Recording Control Dial-out Control Subscription Management Participant Admin Chat messages Properties Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/transport_daily_5c321c99.txt b/transport_daily_5c321c99.txt
new file mode 100644
index 0000000000000000000000000000000000000000..1a3a7beecaa74f74521cbb9901cf07e05a05c0be
--- /dev/null
+++ b/transport_daily_5c321c99.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/transport/daily#param-capture-participant-transcription
+Title: Daily WebRTC - Pipecat
+==================================================
+
+Daily WebRTC - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Transport Daily WebRTC Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Daily WebRTC SmallWebRTCTransport FastAPI WebSocket WebSocket Server Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview DailyTransport provides real-time audio and video communication capabilities using Daily’s WebRTC platform. It supports bidirectional audio/video streams, transcription, voice activity detection (VAD), and participant management. ​ Installation To use DailyTransport , install the required dependencies: Copy Ask AI pip install "pipecat-ai[daily]" You’ll also need to set up your Daily API key as an environment variable: DAILY_API_KEY . You can obtain a Daily API key by signing up at Daily . ​ Configuration ​ Constructor Parameters ​ room_url str required Daily room URL ​ token str | None Daily room token ​ bot_name str required Name of the bot in the room ​ params DailyParams default: "DailyParams()" Transport configuration parameters ​ DailyParams Configuration ​ api_url str default: "https://api.daily.co/v1" Daily API endpoint URL ​ api_key str default: "" Daily API key for authentication ​ Audio Output Configuration ​ audio_out_enabled bool default: "False" Enable audio output capabilities ​ audio_out_is_live bool default: "False" Enable live audio streaming mode ​ audio_out_sample_rate int default: "None" Audio output sample rate in Hz ​ audio_out_channels int default: "1" Number of audio output channels ​ audio_out_bitrate int default: "96000" Audio output bitrate in bits per second ​ Audio Input Configuration ​ audio_in_enabled bool default: "False" Enable audio input capabilities ​ audio_in_passthrough bool default: "False" When enabled, incoming audio frames are pushed downstream ​ audio_in_sample_rate int default: "None" Audio input sample rate in Hz ​ audio_in_channels int default: "1" Number of audio input channels ​ audio_in_filter Optional[BaseAudioFilter] default: "None" Audio filter for input processing. Supported filters are: KrispFilter() and NoisereduceFilter() . See the KrispFilter() reference docs and NoisereduceFilter() reference docs ` for more information. ​ Video Output Configuration ​ video_out_enabled bool default: "False" Enable video output capabilities ​ video_out_is_live bool default: "False" Enable live video streaming mode ​ video_out_width int default: "1024" Video output width in pixels ​ video_out_height int default: "768" Video output height in pixels ​ video_out_bitrate int default: "800000" Video output bitrate in bits per second ​ video_out_framerate int default: "30" Video output frame rate ​ video_out_color_format str default: "RGB" Video color format (RGB, BGR, etc.) ​ Voice Activity Detection (VAD) ​ vad_analyzer VADAnalyzer | None default: "None" Voice Activity Detection analyzer. You can set this to either SileroVADAnalyzer() or WebRTCVADAnalyzer() . SileroVADAnalyzer is the recommended option. Learn more about the SileroVADAnalyzer . ​ Feature Settings ​ dialin_settings Optional[DailyDialinSettings] default: "None" Configuration for dial-in functionality ​ transcription_enabled bool default: "False" Enable real-time transcription ​ transcription_settings DailyTranscriptionSettings Configuration for transcription features Copy Ask AI class DailyTranscriptionSettings ( BaseModel ): language: str = "en" # Default language model: str = "nova-2-general" # Transcription model profanity_filter: bool = True # Filter profanity redact: bool = False # Redact sensitive information endpointing: bool = True # Enable speech endpointing punctuate: bool = True # Add punctuation includeRawResponse: bool = True # Include raw API response extra: Mapping[ str , Any] = { "interim_results" : True # Provide any Deepgram-specific settings } ​ Basic Usage Copy Ask AI from pipecat.transports.services.daily import DailyTransport, DailyParams from pipecat.audio.vad.silero import SileroVADAnalyzer # Configure transport transport = DailyTransport( room_url = "https://your-domain.daily.co/room-name" , token = "your-room-token" , bot_name = "AI Assistant" , params = DailyParams( audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer( params = VADParams( stop_secs = 0.5 )), ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), # Handle incoming audio/video stt, # Speech-to-text llm, # Language model tts, # Text-to-speech transport.output() # Handle outgoing audio/video ]) # Use transport event handlers to manage the call lifecycle # Event handler for when the user joins the room: @transport.event_handler ( "on_first_participant_joined" ) async def on_first_participant_joined ( transport , participant ): # Start transcription for the user await transport.capture_participant_transcription(participant[ "id" ]) # Kick off the conversation. await task.queue_frames([context_aggregator.user().get_context_frame()]) # Event handler for when the user leaves the room: @transport.event_handler ( "on_participant_left" ) async def on_participant_left ( transport , participant , reason ): # Cancel the pipeline, which stops processing and removes the bot from the room await task.cancel() ​ Event Callbacks DailyTransport provides a comprehensive callback system for handling various events. Register callbacks using the @transport.event_handler() decorator. ​ Connection Events ​ on_joined async callback Called when the bot successfully joins the room. Parameters: transport : The DailyTransport instance data : Dictionary containing room join information Copy Ask AI @transport.event_handler ( "on_joined" ) async def on_joined ( transport , data : Dict[ str , Any]): logger.info( f "Joined room with data: { data } " ) ​ on_left async callback Called when the bot leaves the room. Parameters: transport : The DailyTransport instance Copy Ask AI @transport.event_handler ( "on_left" ) async def on_left ( transport ): logger.info( "Left room" ) ​ Participant Events ​ on_first_participant_joined async callback Called when the first participant joins the room. Useful for initializing conversations. Parameters: transport : The DailyTransport instance participant : Dictionary containing participant information Copy Ask AI @transport.event_handler ( "on_first_participant_joined" ) async def on_first_participant_joined ( transport , participant : Dict[ str , Any]): await transport.capture_participant_transcription(participant[ "id" ]) await task.queue_frames([LLMMessagesFrame(initial_messages)]) ​ on_participant_joined async callback Called when any participant joins the room. Parameters: transport : The DailyTransport instance participant : Dictionary containing participant information Copy Ask AI @transport.event_handler ( "on_participant_joined" ) async def on_participant_joined ( transport , participant : Dict[ str , Any]): logger.info( f "Participant joined: { participant[ 'id' ] } " ) ​ on_participant_left async callback Called when a participant leaves the room. Parameters: transport : The DailyTransport instance participant : Dictionary containing participant information reason : String describing why the participant left, “leftCall” | “hidden” Copy Ask AI @transport.event_handler ( "on_participant_left" ) async def on_participant_left ( transport , participant : Dict[ str , Any], reason : str ): logger.info( f "Participant { participant[ 'id' ] } left: { reason } " ) # Cancel the pipeline task to stop processing and remove the bot from the Daily room await task.cancel() ​ on_participant_updated async callback Event emitted when a participant is updated. This can mean either the participant’s metadata was updated, or the tracks belonging to the participant changed. Parameters: transport : The DailyTransport instance participant : Dictionary containing updated participant information Copy Ask AI @transport.event_handler ( "on_participant_updated" ) async def on_participant_updated ( transport , participant : Dict[ str , Any]): logger.info( f "Participant updated: { participant } " ) ​ Communication Events ​ on_app_message async callback Event emitted when a custom app message is received from another participant or via the REST API. Parameters: transport : The DailyTransport instance message : The message content (any type) sender : String identifier of the message sender Copy Ask AI @transport.event_handler ( "on_app_message" ) async def on_app_message ( transport , message : Any, sender : str ): logger.info( f "Message from { sender } : { message } " ) ​ on_call_state_updated async callback Event emitted when the call state changes, normally as a consequence of joining or leaving the call. Parameters: transport : The DailyTransport instance state : String representing the new call state. Learn more about call states . Copy Ask AI @transport.event_handler ( "on_call_state_updated" ) async def on_call_state_updated ( transport , state : str ): logger.info( f "Call state updated: { state } " ) ​ Dial Events ​ Dial-in ​ on_dialin_ready async callback Event emitted when dial-in is ready. This happens after the room has connected to the SIP endpoint and the system is ready to receive dial-in calls. Parameters: transport : The DailyTransport instance sip_endpoint : String containing the SIP endpoint information Copy Ask AI @transport.event_handler ( "on_dialin_ready" ) async def on_dialin_ready ( transport , sip_endpoint : str ): logger.info( f "Dial-in ready at: { sip_endpoint } " ) ​ on_dialin_connected async callback Event emitted when the session with the dial-in remote end is established (i.e. SIP endpoint or PSTN are connectd to the Daily room). Note: connected does not mean media (audio or video) has started flowing between the room and PSTN, it means the room received the connection request and both endpoints are negotiating the media flow. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. See DialinConnectedEvent . Copy Ask AI @transport.event_handler ( "on_dialin_connected" ) async def on_dialin_connected ( transport , data : Any): logger.info( f "Dial-in connected: { data } " ) ​ on_dialin_stopped async callback Event emitted when the dial-in remote end disconnects the call. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. See DialinStoppedEvent . Copy Ask AI @transport.event_handler ( "on_dialin_stopped" ) async def on_dialin_stopped ( transport , data : Any): logger.info( f "Dial-in stopped: { data } " ) ​ on_dialin_error async callback Event emitted in the case of dial-in errors which are fatal and the service cannot proceed. For example, an error in SDP negotiation is fatal to the media/SIP pipeline and will result in dialin-error being triggered. Parameters: transport : The DailyTransport instance data : Dictionary containing error information. See DialinEvent . Copy Ask AI @transport.event_handler ( "on_dialin_error" ) async def on_dialin_error ( transport , data : Any): logger.error( f "Dial-in error: { data } " ) ​ on_dialin_warning async callback Event emitted there is a dial-in non-fatal error, such as the selected codec not being used and a fallback codec being utilized. Parameters: transport : The DailyTransport instance data : Dictionary containing warning information. See DialinEvent . Copy Ask AI @transport.event_handler ( "on_dialin_warning" ) async def on_dialin_warning ( transport , data : Any): logger.warning( f "Dial-in warning: { data } " ) ​ Dial-out ​ on_dialout_answered async callback Event emitted when the session with the dial-out remote end is answered. Parameters: transport : The DailyTransport instance data : Dictionary containing call information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_answered" ) async def on_dialout_answered ( transport , data : Any): logger.info( f "Dial-out answered: { data } " ) ​ on_dialout_connected async callback Event emitted when the session with the dial-out remote end is established. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_connected" ) async def on_dialout_connected ( transport , data : Any): logger.info( f "Dial-out connected: { data } " ) ​ on_dialout_stopped async callback Event emitted when the dial-out session is stopped. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_stopped" ) async def on_dialout_stopped ( transport , data : Any): logger.info( f "Dial-out stopped: { data } " ) ​ on_dialout_error async callback Event emitted in the case of dial-out errors which are fatal and the service cannot proceed. For example, an error in SDP negotiation is fatal to the media/SIP pipeline and will result in dialout-error being triggered. Parameters: transport : The DailyTransport instance data : Dictionary containing error information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_error" ) async def on_dialout_error ( transport , data : Any): logger.error( f "Dial-out error: { data } " ) ​ on_dialout_warning async callback Event emitted there is a dial-out non-fatal error, such as the selected codec not being used and a fallback codec being utilized. Parameters: transport : The DailyTransport instance data : Dictionary containing warning information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_warning" ) async def on_dialout_warning ( transport , data : Any): logger.warning( f "Dial-out warning: { data } " ) ​ Transcription Events ​ on_transcription_message async callback Called when a transcription message is received. This includes both interim and final transcriptions. Parameters: transport : The DailyTransport instance message : Dictionary containing transcription data including. Learn more . Copy Ask AI @transport.event_handler ( "on_transcription_message" ) async def on_transcription_message ( transport , message : Dict[ str , Any]): participant_id = message.get( "participantId" ) text = message.get( "text" ) is_final = message[ "rawResponse" ][ "is_final" ] logger.info( f "Transcription from { participant_id } : { text } (final: { is_final } )" ) ​ Recording Events ​ on_recording_started async callback Called when a room recording starts successfully. Parameters: transport : The DailyTransport instance status : Dictionary containing recording status information. Learn more . Copy Ask AI @transport.event_handler ( "on_recording_started" ) async def on_recording_started ( transport , status : Dict[ str , Any]): logger.info( f "Recording started with status: { status } " ) ​ on_recording_stopped async callback Called when a room recording stops. Parameters: transport : The DailyTransport instance stream_id : String identifier of the stopped recording stream Copy Ask AI @transport.event_handler ( "on_recording_stopped" ) async def on_recording_stopped ( transport , stream_id : str ): logger.info( f "Recording stopped for stream: { stream_id } " ) ​ on_recording_error async callback Called when an error occurs during recording. Parameters: transport : The DailyTransport instance stream_id : String identifier of the recording stream message : Error message describing what went wrong Copy Ask AI @transport.event_handler ( "on_recording_error" ) async def on_recording_error ( transport , stream_id : str , message : str ): logger.error( f "Recording error for stream { stream_id } : { message } " ) ​ Error Events ​ on_error async callback Called when a transport error occurs. Parameters: transport : The DailyTransport instance error : String containing error details Copy Ask AI @transport.event_handler ( "on_error" ) async def on_error ( transport , error : str ): logger.error( f "Transport error: { error } " ) ​ Notes on Callbacks All callbacks are asynchronous Callbacks are executed in the order they were registered Multiple handlers can be registered for the same event Exceptions in callbacks are logged but don’t stop the transport Callbacks should be lightweight to avoid blocking the event loop Heavy processing should be offloaded to separate tasks ​ Frame Types ​ Input Frames ​ InputAudioRawFrame Frame Raw audio data from participants ​ UserImageRawFrame Frame Video frames from participants ​ Output Frames ​ OutputAudioRawFrame Frame Audio data to be sent ​ OutputImageRawFrame Frame Video frames to be sent ​ Methods ​ Room Management ​ participants method Returns a dictionary of all participants currently in the room. Copy Ask AI def participants () -> Dict[ str , Any] See participants for more details. ​ participant_counts method Returns participant count statistics for the room. Copy Ask AI def participant_counts () -> Dict[ str , int ] See participant_counts for more details. ​ send_message async method Sends a message to participants in the Daily room. Messages can be directed to all participants or targeted to a specific participant. Copy Ask AI async def send_message ( frame : TransportMessageFrame | TransportMessageUrgentFrame) Parameters: frame : A transport message frame containing the message to send. Can be either: TransportMessageFrame : Standard message TransportMessageUrgentFrame : High-priority message Message Structure: Messages should conform to the expected format for the receiving system. For RTVI messages, use: Copy Ask AI { "label" : "rtvi-ai" , # Required for RTVI protocol "type" : "message-type" , # Defines how the client will process it "id" : "unique-id" , # Unique identifier for the message "data" : { # Custom payload # Your custom data fields } } Example - Standard Message: Copy Ask AI import uuid # Send a standard message to all participants message_data = { "label" : "rtvi-ai" , "type" : "custom-message" , "id" : str (uuid.uuid4()), "data" : { "message" : "Hello everyone!" , "timestamp" : datetime.datetime.now().isoformat() } } await transport.send_message(TransportMessageFrame( message = message_data)) Example - Urgent Message to Specific Participant: Copy Ask AI # Send urgent message to specific participant urgent_data = { "label" : "rtvi-ai" , "type" : "server-message" , "id" : str (uuid.uuid4()), "data" : { "message" : "Private message" , "importance" : "high" } } await transport.send_message( DailyTransportMessageUrgentFrame( message = urgent_data, participant_id = "participant-123" ) ) The message will be delivered to: All participants if no specific participant_id is set Only the specified participant if participant_id is set Through the Daily app messaging system Can be received by registering an on_app_message handler Use TransportMessageUrgentFrame for high-priority messages that need immediate delivery. ​ Media Control ​ send_image async method Sends an image frame to the room. Copy Ask AI async def send_image ( frame : OutputImageRawFrame | SpriteFrame) -> None Parameters: frame : Image frame to send, either raw image data or sprite animation ​ send_audio async method Sends an audio frame to the room. Copy Ask AI async def send_audio ( frame : OutputAudioRawFrame) -> None Parameters: frame : Audio frame to send ​ Video Management ​ capture_participant_video async method Starts capturing video from a specific participant. Copy Ask AI async def capture_participant_video ( participant_id : str , framerate : int = 30 , video_source : str = "camera" , color_format : str = "RGB" ) -> None Parameters: participant_id : ID of the participant to capture framerate : Target frame rate (default: 30) video_source : Video source type (default: “camera”) color_format : Color format of the video (default: “RGB”) To request an image from the camera stream, set the framerate to 0 and push a UserImageRequestFrame . ​ Transcription Control ​ capture_participant_transcription async method Starts capturing and transcribing audio from a specific participant. Copy Ask AI async def capture_participant_transcription ( participant_id : str ) -> None Parameters: participant_id : ID of the participant to transcribe ​ update_transcription async method Updates the transcription configuration for specific participants or instances. Copy Ask AI async def update_transcription ( participants : List[ str ] | None = None , instance_id : str | None = None ) -> None Parameters: participants : Optional list of participant IDs to transcribe. If None, transcribes all participants. instance_id : Optional specific transcription instance ID to update Example: Copy Ask AI # Update transcription for specific participants await transport.update_transcription( participants = [ "participant-123" , "participant-456" ]) # Update specific transcription instance await transport.update_transcription( instance_id = "transcription-789" ) ​ Recording Control ​ start_recording async method Starts recording the room session. Copy Ask AI async def start_recording ( streaming_settings : Dict = None , stream_id : str = None , force_new : bool = None ) -> None Parameters: streaming_settings : Recording configuration settings stream_id : Optional stream identifier force_new : Force start a new recording See start_recording for more details. ​ stop_recording async method Stops an active recording. Copy Ask AI async def stop_recording ( stream_id : str = None ) -> None Parameters: stream_id : Optional stream identifier to stop specific recording See stop_recording for more details. ​ Dial-out Control ​ start_dialout async method Initiates a dial-out call. Copy Ask AI async def start_dialout ( settings : Dict = None ) -> None Parameters: settings : Dial-out configuration settings See start_dialout for more details. ​ stop_dialout async method Stops an active dial-out call. Copy Ask AI async def stop_dialout ( participant_id : str ) -> None Parameters: participant_id : ID of the participant to disconnect See stop_dialout for more details. ​ send_dtmf async method Sends DTMF tones in an existing dial-out session. Copy Ask AI async def send_dtmf ( settings : Dict = None ) -> None Parameters: settings : DTMF settings See send_dtmf for more details. ​ Subscription Management ​ update_subscriptions async method Updates subscriptions and subscription profiles. This function allows you to update subscription profiles and at the same time assign specific subscription profiles to a participant and even change specific settings for some participants. Copy Ask AI async def update_subscriptions ( participant_settings : Dict = None , profile_settings : Dict = None ) -> None Parameters: participant_settings : Per-participant subscription settings profile_settings : Global subscription profile settings See update_subscriptions for more details. ​ Participant Admin When using the DailyTransport, if your bot is given a meeting token with owner or participant admin permissions, it can remote-control other participants’ capabilities and permissions. With this control, the bot can modify settings for how others send and receive media, how they are seen in a session, and whether they have administrative controls. ​ update_remote_participants async method Updates permissions and settings for specific participants in the room. Copy Ask AI async def update_remote_participants ( remote_participants : Mapping[ str , Any] = None ) -> None Parameters: remote_participants : Dictionary mapping participant IDs to their updated settings This method allows fine-grained control over participant capabilities and permissions in the Daily room. Examples: Copy Ask AI # Mute a participant's microphone await transport.update_remote_participants({ "participant-123" : { "inputsEnabled" : { "microphone" : False } } }) # Revoke send permissions (mutes the participant and prevents them from unmuting) await transport.update_remote_participants({ "participant-123" : { "permissions" : { "canSend" : [] # Empty list revokes all send permissions } } }) # Configure selective audio routing await transport.update_remote_participants({ "participant-123" : { # the customer "permissions" : { "canReceive" : { "base" : False , # Don't receive from anyone by default "byUserId" : { "hold-music" : True # Only receive from hold music player } } } } }) # Set up user-to-user permissions mapping (e.g., when transferring calls) # Assumes customer was previously unable to hear anyone (they were on hold) and agent was speaking to bot await transport.update_remote_participants({ "participant-123" : { # the customer "permissions" : { "base" : False , "canReceive" : { "byUserId" : { "agent" : True }} } }, "participant-456" : { # the agent "permissions" : { "base" : False , "canReceive" : { "byUserId" : { "customer" : True }} } } }) # Promote a participant to be a participant admin await transport.update_remote_participants({ "participant-123" : { "permissions" : { "canAdmin" : [ "participants" ] } } }) Permissions Settings: For comprehensive details on permissions settings, see the Daily API documentation: daily-python’s update_remote_participants() documentation daily-js’s updateParticipant() -> updatePermission and participants().permissions ​ Chat messages ​ send_prebuilt_chat_message async method When testing with Daily Prebuilt, you can send messages to the Prebuilt chat using this method. Parameters: message (str): The chat message to send user_name (str): The name of the user sending the message See send_prebuilt_chat_message for more details. ​ Properties ​ participant_id string Returns the transport’s participant ID in the room. ​ room_url string Returns the transport’s room URL for the room currently in use. ​ Notes Supports real-time audio/video communication Handles WebRTC connection management Provides voice activity detection Includes transcription capabilities Manages participant interactions Supports recording and streaming Thread-safe processing All callbacks are asynchronous Multiple handlers can be registered for the same event Exceptions in callbacks are logged but don’t stop the transport Supported Services SmallWebRTCTransport On this page Overview Installation Configuration Constructor Parameters DailyParams Configuration Audio Output Configuration Audio Input Configuration Video Output Configuration Voice Activity Detection (VAD) Feature Settings Basic Usage Event Callbacks Connection Events Participant Events Communication Events Dial Events Dial-in Dial-out Transcription Events Recording Events Error Events Notes on Callbacks Frame Types Input Frames Output Frames Methods Room Management Media Control Video Management Transcription Control Recording Control Dial-out Control Subscription Management Participant Admin Chat messages Properties Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/transport_daily_5cfc0393.txt b/transport_daily_5cfc0393.txt
new file mode 100644
index 0000000000000000000000000000000000000000..128cf014e89af0617affcb1b938b87c76d9219d6
--- /dev/null
+++ b/transport_daily_5cfc0393.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/transport/daily#event-callbacks
+Title: Daily WebRTC - Pipecat
+==================================================
+
+Daily WebRTC - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Transport Daily WebRTC Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Daily WebRTC SmallWebRTCTransport FastAPI WebSocket WebSocket Server Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview DailyTransport provides real-time audio and video communication capabilities using Daily’s WebRTC platform. It supports bidirectional audio/video streams, transcription, voice activity detection (VAD), and participant management. ​ Installation To use DailyTransport , install the required dependencies: Copy Ask AI pip install "pipecat-ai[daily]" You’ll also need to set up your Daily API key as an environment variable: DAILY_API_KEY . You can obtain a Daily API key by signing up at Daily . ​ Configuration ​ Constructor Parameters ​ room_url str required Daily room URL ​ token str | None Daily room token ​ bot_name str required Name of the bot in the room ​ params DailyParams default: "DailyParams()" Transport configuration parameters ​ DailyParams Configuration ​ api_url str default: "https://api.daily.co/v1" Daily API endpoint URL ​ api_key str default: "" Daily API key for authentication ​ Audio Output Configuration ​ audio_out_enabled bool default: "False" Enable audio output capabilities ​ audio_out_is_live bool default: "False" Enable live audio streaming mode ​ audio_out_sample_rate int default: "None" Audio output sample rate in Hz ​ audio_out_channels int default: "1" Number of audio output channels ​ audio_out_bitrate int default: "96000" Audio output bitrate in bits per second ​ Audio Input Configuration ​ audio_in_enabled bool default: "False" Enable audio input capabilities ​ audio_in_passthrough bool default: "False" When enabled, incoming audio frames are pushed downstream ​ audio_in_sample_rate int default: "None" Audio input sample rate in Hz ​ audio_in_channels int default: "1" Number of audio input channels ​ audio_in_filter Optional[BaseAudioFilter] default: "None" Audio filter for input processing. Supported filters are: KrispFilter() and NoisereduceFilter() . See the KrispFilter() reference docs and NoisereduceFilter() reference docs ` for more information. ​ Video Output Configuration ​ video_out_enabled bool default: "False" Enable video output capabilities ​ video_out_is_live bool default: "False" Enable live video streaming mode ​ video_out_width int default: "1024" Video output width in pixels ​ video_out_height int default: "768" Video output height in pixels ​ video_out_bitrate int default: "800000" Video output bitrate in bits per second ​ video_out_framerate int default: "30" Video output frame rate ​ video_out_color_format str default: "RGB" Video color format (RGB, BGR, etc.) ​ Voice Activity Detection (VAD) ​ vad_analyzer VADAnalyzer | None default: "None" Voice Activity Detection analyzer. You can set this to either SileroVADAnalyzer() or WebRTCVADAnalyzer() . SileroVADAnalyzer is the recommended option. Learn more about the SileroVADAnalyzer . ​ Feature Settings ​ dialin_settings Optional[DailyDialinSettings] default: "None" Configuration for dial-in functionality ​ transcription_enabled bool default: "False" Enable real-time transcription ​ transcription_settings DailyTranscriptionSettings Configuration for transcription features Copy Ask AI class DailyTranscriptionSettings ( BaseModel ): language: str = "en" # Default language model: str = "nova-2-general" # Transcription model profanity_filter: bool = True # Filter profanity redact: bool = False # Redact sensitive information endpointing: bool = True # Enable speech endpointing punctuate: bool = True # Add punctuation includeRawResponse: bool = True # Include raw API response extra: Mapping[ str , Any] = { "interim_results" : True # Provide any Deepgram-specific settings } ​ Basic Usage Copy Ask AI from pipecat.transports.services.daily import DailyTransport, DailyParams from pipecat.audio.vad.silero import SileroVADAnalyzer # Configure transport transport = DailyTransport( room_url = "https://your-domain.daily.co/room-name" , token = "your-room-token" , bot_name = "AI Assistant" , params = DailyParams( audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer( params = VADParams( stop_secs = 0.5 )), ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), # Handle incoming audio/video stt, # Speech-to-text llm, # Language model tts, # Text-to-speech transport.output() # Handle outgoing audio/video ]) # Use transport event handlers to manage the call lifecycle # Event handler for when the user joins the room: @transport.event_handler ( "on_first_participant_joined" ) async def on_first_participant_joined ( transport , participant ): # Start transcription for the user await transport.capture_participant_transcription(participant[ "id" ]) # Kick off the conversation. await task.queue_frames([context_aggregator.user().get_context_frame()]) # Event handler for when the user leaves the room: @transport.event_handler ( "on_participant_left" ) async def on_participant_left ( transport , participant , reason ): # Cancel the pipeline, which stops processing and removes the bot from the room await task.cancel() ​ Event Callbacks DailyTransport provides a comprehensive callback system for handling various events. Register callbacks using the @transport.event_handler() decorator. ​ Connection Events ​ on_joined async callback Called when the bot successfully joins the room. Parameters: transport : The DailyTransport instance data : Dictionary containing room join information Copy Ask AI @transport.event_handler ( "on_joined" ) async def on_joined ( transport , data : Dict[ str , Any]): logger.info( f "Joined room with data: { data } " ) ​ on_left async callback Called when the bot leaves the room. Parameters: transport : The DailyTransport instance Copy Ask AI @transport.event_handler ( "on_left" ) async def on_left ( transport ): logger.info( "Left room" ) ​ Participant Events ​ on_first_participant_joined async callback Called when the first participant joins the room. Useful for initializing conversations. Parameters: transport : The DailyTransport instance participant : Dictionary containing participant information Copy Ask AI @transport.event_handler ( "on_first_participant_joined" ) async def on_first_participant_joined ( transport , participant : Dict[ str , Any]): await transport.capture_participant_transcription(participant[ "id" ]) await task.queue_frames([LLMMessagesFrame(initial_messages)]) ​ on_participant_joined async callback Called when any participant joins the room. Parameters: transport : The DailyTransport instance participant : Dictionary containing participant information Copy Ask AI @transport.event_handler ( "on_participant_joined" ) async def on_participant_joined ( transport , participant : Dict[ str , Any]): logger.info( f "Participant joined: { participant[ 'id' ] } " ) ​ on_participant_left async callback Called when a participant leaves the room. Parameters: transport : The DailyTransport instance participant : Dictionary containing participant information reason : String describing why the participant left, “leftCall” | “hidden” Copy Ask AI @transport.event_handler ( "on_participant_left" ) async def on_participant_left ( transport , participant : Dict[ str , Any], reason : str ): logger.info( f "Participant { participant[ 'id' ] } left: { reason } " ) # Cancel the pipeline task to stop processing and remove the bot from the Daily room await task.cancel() ​ on_participant_updated async callback Event emitted when a participant is updated. This can mean either the participant’s metadata was updated, or the tracks belonging to the participant changed. Parameters: transport : The DailyTransport instance participant : Dictionary containing updated participant information Copy Ask AI @transport.event_handler ( "on_participant_updated" ) async def on_participant_updated ( transport , participant : Dict[ str , Any]): logger.info( f "Participant updated: { participant } " ) ​ Communication Events ​ on_app_message async callback Event emitted when a custom app message is received from another participant or via the REST API. Parameters: transport : The DailyTransport instance message : The message content (any type) sender : String identifier of the message sender Copy Ask AI @transport.event_handler ( "on_app_message" ) async def on_app_message ( transport , message : Any, sender : str ): logger.info( f "Message from { sender } : { message } " ) ​ on_call_state_updated async callback Event emitted when the call state changes, normally as a consequence of joining or leaving the call. Parameters: transport : The DailyTransport instance state : String representing the new call state. Learn more about call states . Copy Ask AI @transport.event_handler ( "on_call_state_updated" ) async def on_call_state_updated ( transport , state : str ): logger.info( f "Call state updated: { state } " ) ​ Dial Events ​ Dial-in ​ on_dialin_ready async callback Event emitted when dial-in is ready. This happens after the room has connected to the SIP endpoint and the system is ready to receive dial-in calls. Parameters: transport : The DailyTransport instance sip_endpoint : String containing the SIP endpoint information Copy Ask AI @transport.event_handler ( "on_dialin_ready" ) async def on_dialin_ready ( transport , sip_endpoint : str ): logger.info( f "Dial-in ready at: { sip_endpoint } " ) ​ on_dialin_connected async callback Event emitted when the session with the dial-in remote end is established (i.e. SIP endpoint or PSTN are connectd to the Daily room). Note: connected does not mean media (audio or video) has started flowing between the room and PSTN, it means the room received the connection request and both endpoints are negotiating the media flow. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. See DialinConnectedEvent . Copy Ask AI @transport.event_handler ( "on_dialin_connected" ) async def on_dialin_connected ( transport , data : Any): logger.info( f "Dial-in connected: { data } " ) ​ on_dialin_stopped async callback Event emitted when the dial-in remote end disconnects the call. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. See DialinStoppedEvent . Copy Ask AI @transport.event_handler ( "on_dialin_stopped" ) async def on_dialin_stopped ( transport , data : Any): logger.info( f "Dial-in stopped: { data } " ) ​ on_dialin_error async callback Event emitted in the case of dial-in errors which are fatal and the service cannot proceed. For example, an error in SDP negotiation is fatal to the media/SIP pipeline and will result in dialin-error being triggered. Parameters: transport : The DailyTransport instance data : Dictionary containing error information. See DialinEvent . Copy Ask AI @transport.event_handler ( "on_dialin_error" ) async def on_dialin_error ( transport , data : Any): logger.error( f "Dial-in error: { data } " ) ​ on_dialin_warning async callback Event emitted there is a dial-in non-fatal error, such as the selected codec not being used and a fallback codec being utilized. Parameters: transport : The DailyTransport instance data : Dictionary containing warning information. See DialinEvent . Copy Ask AI @transport.event_handler ( "on_dialin_warning" ) async def on_dialin_warning ( transport , data : Any): logger.warning( f "Dial-in warning: { data } " ) ​ Dial-out ​ on_dialout_answered async callback Event emitted when the session with the dial-out remote end is answered. Parameters: transport : The DailyTransport instance data : Dictionary containing call information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_answered" ) async def on_dialout_answered ( transport , data : Any): logger.info( f "Dial-out answered: { data } " ) ​ on_dialout_connected async callback Event emitted when the session with the dial-out remote end is established. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_connected" ) async def on_dialout_connected ( transport , data : Any): logger.info( f "Dial-out connected: { data } " ) ​ on_dialout_stopped async callback Event emitted when the dial-out session is stopped. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_stopped" ) async def on_dialout_stopped ( transport , data : Any): logger.info( f "Dial-out stopped: { data } " ) ​ on_dialout_error async callback Event emitted in the case of dial-out errors which are fatal and the service cannot proceed. For example, an error in SDP negotiation is fatal to the media/SIP pipeline and will result in dialout-error being triggered. Parameters: transport : The DailyTransport instance data : Dictionary containing error information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_error" ) async def on_dialout_error ( transport , data : Any): logger.error( f "Dial-out error: { data } " ) ​ on_dialout_warning async callback Event emitted there is a dial-out non-fatal error, such as the selected codec not being used and a fallback codec being utilized. Parameters: transport : The DailyTransport instance data : Dictionary containing warning information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_warning" ) async def on_dialout_warning ( transport , data : Any): logger.warning( f "Dial-out warning: { data } " ) ​ Transcription Events ​ on_transcription_message async callback Called when a transcription message is received. This includes both interim and final transcriptions. Parameters: transport : The DailyTransport instance message : Dictionary containing transcription data including. Learn more . Copy Ask AI @transport.event_handler ( "on_transcription_message" ) async def on_transcription_message ( transport , message : Dict[ str , Any]): participant_id = message.get( "participantId" ) text = message.get( "text" ) is_final = message[ "rawResponse" ][ "is_final" ] logger.info( f "Transcription from { participant_id } : { text } (final: { is_final } )" ) ​ Recording Events ​ on_recording_started async callback Called when a room recording starts successfully. Parameters: transport : The DailyTransport instance status : Dictionary containing recording status information. Learn more . Copy Ask AI @transport.event_handler ( "on_recording_started" ) async def on_recording_started ( transport , status : Dict[ str , Any]): logger.info( f "Recording started with status: { status } " ) ​ on_recording_stopped async callback Called when a room recording stops. Parameters: transport : The DailyTransport instance stream_id : String identifier of the stopped recording stream Copy Ask AI @transport.event_handler ( "on_recording_stopped" ) async def on_recording_stopped ( transport , stream_id : str ): logger.info( f "Recording stopped for stream: { stream_id } " ) ​ on_recording_error async callback Called when an error occurs during recording. Parameters: transport : The DailyTransport instance stream_id : String identifier of the recording stream message : Error message describing what went wrong Copy Ask AI @transport.event_handler ( "on_recording_error" ) async def on_recording_error ( transport , stream_id : str , message : str ): logger.error( f "Recording error for stream { stream_id } : { message } " ) ​ Error Events ​ on_error async callback Called when a transport error occurs. Parameters: transport : The DailyTransport instance error : String containing error details Copy Ask AI @transport.event_handler ( "on_error" ) async def on_error ( transport , error : str ): logger.error( f "Transport error: { error } " ) ​ Notes on Callbacks All callbacks are asynchronous Callbacks are executed in the order they were registered Multiple handlers can be registered for the same event Exceptions in callbacks are logged but don’t stop the transport Callbacks should be lightweight to avoid blocking the event loop Heavy processing should be offloaded to separate tasks ​ Frame Types ​ Input Frames ​ InputAudioRawFrame Frame Raw audio data from participants ​ UserImageRawFrame Frame Video frames from participants ​ Output Frames ​ OutputAudioRawFrame Frame Audio data to be sent ​ OutputImageRawFrame Frame Video frames to be sent ​ Methods ​ Room Management ​ participants method Returns a dictionary of all participants currently in the room. Copy Ask AI def participants () -> Dict[ str , Any] See participants for more details. ​ participant_counts method Returns participant count statistics for the room. Copy Ask AI def participant_counts () -> Dict[ str , int ] See participant_counts for more details. ​ send_message async method Sends a message to participants in the Daily room. Messages can be directed to all participants or targeted to a specific participant. Copy Ask AI async def send_message ( frame : TransportMessageFrame | TransportMessageUrgentFrame) Parameters: frame : A transport message frame containing the message to send. Can be either: TransportMessageFrame : Standard message TransportMessageUrgentFrame : High-priority message Message Structure: Messages should conform to the expected format for the receiving system. For RTVI messages, use: Copy Ask AI { "label" : "rtvi-ai" , # Required for RTVI protocol "type" : "message-type" , # Defines how the client will process it "id" : "unique-id" , # Unique identifier for the message "data" : { # Custom payload # Your custom data fields } } Example - Standard Message: Copy Ask AI import uuid # Send a standard message to all participants message_data = { "label" : "rtvi-ai" , "type" : "custom-message" , "id" : str (uuid.uuid4()), "data" : { "message" : "Hello everyone!" , "timestamp" : datetime.datetime.now().isoformat() } } await transport.send_message(TransportMessageFrame( message = message_data)) Example - Urgent Message to Specific Participant: Copy Ask AI # Send urgent message to specific participant urgent_data = { "label" : "rtvi-ai" , "type" : "server-message" , "id" : str (uuid.uuid4()), "data" : { "message" : "Private message" , "importance" : "high" } } await transport.send_message( DailyTransportMessageUrgentFrame( message = urgent_data, participant_id = "participant-123" ) ) The message will be delivered to: All participants if no specific participant_id is set Only the specified participant if participant_id is set Through the Daily app messaging system Can be received by registering an on_app_message handler Use TransportMessageUrgentFrame for high-priority messages that need immediate delivery. ​ Media Control ​ send_image async method Sends an image frame to the room. Copy Ask AI async def send_image ( frame : OutputImageRawFrame | SpriteFrame) -> None Parameters: frame : Image frame to send, either raw image data or sprite animation ​ send_audio async method Sends an audio frame to the room. Copy Ask AI async def send_audio ( frame : OutputAudioRawFrame) -> None Parameters: frame : Audio frame to send ​ Video Management ​ capture_participant_video async method Starts capturing video from a specific participant. Copy Ask AI async def capture_participant_video ( participant_id : str , framerate : int = 30 , video_source : str = "camera" , color_format : str = "RGB" ) -> None Parameters: participant_id : ID of the participant to capture framerate : Target frame rate (default: 30) video_source : Video source type (default: “camera”) color_format : Color format of the video (default: “RGB”) To request an image from the camera stream, set the framerate to 0 and push a UserImageRequestFrame . ​ Transcription Control ​ capture_participant_transcription async method Starts capturing and transcribing audio from a specific participant. Copy Ask AI async def capture_participant_transcription ( participant_id : str ) -> None Parameters: participant_id : ID of the participant to transcribe ​ update_transcription async method Updates the transcription configuration for specific participants or instances. Copy Ask AI async def update_transcription ( participants : List[ str ] | None = None , instance_id : str | None = None ) -> None Parameters: participants : Optional list of participant IDs to transcribe. If None, transcribes all participants. instance_id : Optional specific transcription instance ID to update Example: Copy Ask AI # Update transcription for specific participants await transport.update_transcription( participants = [ "participant-123" , "participant-456" ]) # Update specific transcription instance await transport.update_transcription( instance_id = "transcription-789" ) ​ Recording Control ​ start_recording async method Starts recording the room session. Copy Ask AI async def start_recording ( streaming_settings : Dict = None , stream_id : str = None , force_new : bool = None ) -> None Parameters: streaming_settings : Recording configuration settings stream_id : Optional stream identifier force_new : Force start a new recording See start_recording for more details. ​ stop_recording async method Stops an active recording. Copy Ask AI async def stop_recording ( stream_id : str = None ) -> None Parameters: stream_id : Optional stream identifier to stop specific recording See stop_recording for more details. ​ Dial-out Control ​ start_dialout async method Initiates a dial-out call. Copy Ask AI async def start_dialout ( settings : Dict = None ) -> None Parameters: settings : Dial-out configuration settings See start_dialout for more details. ​ stop_dialout async method Stops an active dial-out call. Copy Ask AI async def stop_dialout ( participant_id : str ) -> None Parameters: participant_id : ID of the participant to disconnect See stop_dialout for more details. ​ send_dtmf async method Sends DTMF tones in an existing dial-out session. Copy Ask AI async def send_dtmf ( settings : Dict = None ) -> None Parameters: settings : DTMF settings See send_dtmf for more details. ​ Subscription Management ​ update_subscriptions async method Updates subscriptions and subscription profiles. This function allows you to update subscription profiles and at the same time assign specific subscription profiles to a participant and even change specific settings for some participants. Copy Ask AI async def update_subscriptions ( participant_settings : Dict = None , profile_settings : Dict = None ) -> None Parameters: participant_settings : Per-participant subscription settings profile_settings : Global subscription profile settings See update_subscriptions for more details. ​ Participant Admin When using the DailyTransport, if your bot is given a meeting token with owner or participant admin permissions, it can remote-control other participants’ capabilities and permissions. With this control, the bot can modify settings for how others send and receive media, how they are seen in a session, and whether they have administrative controls. ​ update_remote_participants async method Updates permissions and settings for specific participants in the room. Copy Ask AI async def update_remote_participants ( remote_participants : Mapping[ str , Any] = None ) -> None Parameters: remote_participants : Dictionary mapping participant IDs to their updated settings This method allows fine-grained control over participant capabilities and permissions in the Daily room. Examples: Copy Ask AI # Mute a participant's microphone await transport.update_remote_participants({ "participant-123" : { "inputsEnabled" : { "microphone" : False } } }) # Revoke send permissions (mutes the participant and prevents them from unmuting) await transport.update_remote_participants({ "participant-123" : { "permissions" : { "canSend" : [] # Empty list revokes all send permissions } } }) # Configure selective audio routing await transport.update_remote_participants({ "participant-123" : { # the customer "permissions" : { "canReceive" : { "base" : False , # Don't receive from anyone by default "byUserId" : { "hold-music" : True # Only receive from hold music player } } } } }) # Set up user-to-user permissions mapping (e.g., when transferring calls) # Assumes customer was previously unable to hear anyone (they were on hold) and agent was speaking to bot await transport.update_remote_participants({ "participant-123" : { # the customer "permissions" : { "base" : False , "canReceive" : { "byUserId" : { "agent" : True }} } }, "participant-456" : { # the agent "permissions" : { "base" : False , "canReceive" : { "byUserId" : { "customer" : True }} } } }) # Promote a participant to be a participant admin await transport.update_remote_participants({ "participant-123" : { "permissions" : { "canAdmin" : [ "participants" ] } } }) Permissions Settings: For comprehensive details on permissions settings, see the Daily API documentation: daily-python’s update_remote_participants() documentation daily-js’s updateParticipant() -> updatePermission and participants().permissions ​ Chat messages ​ send_prebuilt_chat_message async method When testing with Daily Prebuilt, you can send messages to the Prebuilt chat using this method. Parameters: message (str): The chat message to send user_name (str): The name of the user sending the message See send_prebuilt_chat_message for more details. ​ Properties ​ participant_id string Returns the transport’s participant ID in the room. ​ room_url string Returns the transport’s room URL for the room currently in use. ​ Notes Supports real-time audio/video communication Handles WebRTC connection management Provides voice activity detection Includes transcription capabilities Manages participant interactions Supports recording and streaming Thread-safe processing All callbacks are asynchronous Multiple handlers can be registered for the same event Exceptions in callbacks are logged but don’t stop the transport Supported Services SmallWebRTCTransport On this page Overview Installation Configuration Constructor Parameters DailyParams Configuration Audio Output Configuration Audio Input Configuration Video Output Configuration Voice Activity Detection (VAD) Feature Settings Basic Usage Event Callbacks Connection Events Participant Events Communication Events Dial Events Dial-in Dial-out Transcription Events Recording Events Error Events Notes on Callbacks Frame Types Input Frames Output Frames Methods Room Management Media Control Video Management Transcription Control Recording Control Dial-out Control Subscription Management Participant Admin Chat messages Properties Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/transport_daily_5d45c5d0.txt b/transport_daily_5d45c5d0.txt
new file mode 100644
index 0000000000000000000000000000000000000000..bd6b51df015fb07d661440a7e1743f309a82cf19
--- /dev/null
+++ b/transport_daily_5d45c5d0.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/transport/daily#param-vad-analyzer
+Title: Daily WebRTC - Pipecat
+==================================================
+
+Daily WebRTC - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Transport Daily WebRTC Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Daily WebRTC SmallWebRTCTransport FastAPI WebSocket WebSocket Server Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview DailyTransport provides real-time audio and video communication capabilities using Daily’s WebRTC platform. It supports bidirectional audio/video streams, transcription, voice activity detection (VAD), and participant management. ​ Installation To use DailyTransport , install the required dependencies: Copy Ask AI pip install "pipecat-ai[daily]" You’ll also need to set up your Daily API key as an environment variable: DAILY_API_KEY . You can obtain a Daily API key by signing up at Daily . ​ Configuration ​ Constructor Parameters ​ room_url str required Daily room URL ​ token str | None Daily room token ​ bot_name str required Name of the bot in the room ​ params DailyParams default: "DailyParams()" Transport configuration parameters ​ DailyParams Configuration ​ api_url str default: "https://api.daily.co/v1" Daily API endpoint URL ​ api_key str default: "" Daily API key for authentication ​ Audio Output Configuration ​ audio_out_enabled bool default: "False" Enable audio output capabilities ​ audio_out_is_live bool default: "False" Enable live audio streaming mode ​ audio_out_sample_rate int default: "None" Audio output sample rate in Hz ​ audio_out_channels int default: "1" Number of audio output channels ​ audio_out_bitrate int default: "96000" Audio output bitrate in bits per second ​ Audio Input Configuration ​ audio_in_enabled bool default: "False" Enable audio input capabilities ​ audio_in_passthrough bool default: "False" When enabled, incoming audio frames are pushed downstream ​ audio_in_sample_rate int default: "None" Audio input sample rate in Hz ​ audio_in_channels int default: "1" Number of audio input channels ​ audio_in_filter Optional[BaseAudioFilter] default: "None" Audio filter for input processing. Supported filters are: KrispFilter() and NoisereduceFilter() . See the KrispFilter() reference docs and NoisereduceFilter() reference docs ` for more information. ​ Video Output Configuration ​ video_out_enabled bool default: "False" Enable video output capabilities ​ video_out_is_live bool default: "False" Enable live video streaming mode ​ video_out_width int default: "1024" Video output width in pixels ​ video_out_height int default: "768" Video output height in pixels ​ video_out_bitrate int default: "800000" Video output bitrate in bits per second ​ video_out_framerate int default: "30" Video output frame rate ​ video_out_color_format str default: "RGB" Video color format (RGB, BGR, etc.) ​ Voice Activity Detection (VAD) ​ vad_analyzer VADAnalyzer | None default: "None" Voice Activity Detection analyzer. You can set this to either SileroVADAnalyzer() or WebRTCVADAnalyzer() . SileroVADAnalyzer is the recommended option. Learn more about the SileroVADAnalyzer . ​ Feature Settings ​ dialin_settings Optional[DailyDialinSettings] default: "None" Configuration for dial-in functionality ​ transcription_enabled bool default: "False" Enable real-time transcription ​ transcription_settings DailyTranscriptionSettings Configuration for transcription features Copy Ask AI class DailyTranscriptionSettings ( BaseModel ): language: str = "en" # Default language model: str = "nova-2-general" # Transcription model profanity_filter: bool = True # Filter profanity redact: bool = False # Redact sensitive information endpointing: bool = True # Enable speech endpointing punctuate: bool = True # Add punctuation includeRawResponse: bool = True # Include raw API response extra: Mapping[ str , Any] = { "interim_results" : True # Provide any Deepgram-specific settings } ​ Basic Usage Copy Ask AI from pipecat.transports.services.daily import DailyTransport, DailyParams from pipecat.audio.vad.silero import SileroVADAnalyzer # Configure transport transport = DailyTransport( room_url = "https://your-domain.daily.co/room-name" , token = "your-room-token" , bot_name = "AI Assistant" , params = DailyParams( audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer( params = VADParams( stop_secs = 0.5 )), ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), # Handle incoming audio/video stt, # Speech-to-text llm, # Language model tts, # Text-to-speech transport.output() # Handle outgoing audio/video ]) # Use transport event handlers to manage the call lifecycle # Event handler for when the user joins the room: @transport.event_handler ( "on_first_participant_joined" ) async def on_first_participant_joined ( transport , participant ): # Start transcription for the user await transport.capture_participant_transcription(participant[ "id" ]) # Kick off the conversation. await task.queue_frames([context_aggregator.user().get_context_frame()]) # Event handler for when the user leaves the room: @transport.event_handler ( "on_participant_left" ) async def on_participant_left ( transport , participant , reason ): # Cancel the pipeline, which stops processing and removes the bot from the room await task.cancel() ​ Event Callbacks DailyTransport provides a comprehensive callback system for handling various events. Register callbacks using the @transport.event_handler() decorator. ​ Connection Events ​ on_joined async callback Called when the bot successfully joins the room. Parameters: transport : The DailyTransport instance data : Dictionary containing room join information Copy Ask AI @transport.event_handler ( "on_joined" ) async def on_joined ( transport , data : Dict[ str , Any]): logger.info( f "Joined room with data: { data } " ) ​ on_left async callback Called when the bot leaves the room. Parameters: transport : The DailyTransport instance Copy Ask AI @transport.event_handler ( "on_left" ) async def on_left ( transport ): logger.info( "Left room" ) ​ Participant Events ​ on_first_participant_joined async callback Called when the first participant joins the room. Useful for initializing conversations. Parameters: transport : The DailyTransport instance participant : Dictionary containing participant information Copy Ask AI @transport.event_handler ( "on_first_participant_joined" ) async def on_first_participant_joined ( transport , participant : Dict[ str , Any]): await transport.capture_participant_transcription(participant[ "id" ]) await task.queue_frames([LLMMessagesFrame(initial_messages)]) ​ on_participant_joined async callback Called when any participant joins the room. Parameters: transport : The DailyTransport instance participant : Dictionary containing participant information Copy Ask AI @transport.event_handler ( "on_participant_joined" ) async def on_participant_joined ( transport , participant : Dict[ str , Any]): logger.info( f "Participant joined: { participant[ 'id' ] } " ) ​ on_participant_left async callback Called when a participant leaves the room. Parameters: transport : The DailyTransport instance participant : Dictionary containing participant information reason : String describing why the participant left, “leftCall” | “hidden” Copy Ask AI @transport.event_handler ( "on_participant_left" ) async def on_participant_left ( transport , participant : Dict[ str , Any], reason : str ): logger.info( f "Participant { participant[ 'id' ] } left: { reason } " ) # Cancel the pipeline task to stop processing and remove the bot from the Daily room await task.cancel() ​ on_participant_updated async callback Event emitted when a participant is updated. This can mean either the participant’s metadata was updated, or the tracks belonging to the participant changed. Parameters: transport : The DailyTransport instance participant : Dictionary containing updated participant information Copy Ask AI @transport.event_handler ( "on_participant_updated" ) async def on_participant_updated ( transport , participant : Dict[ str , Any]): logger.info( f "Participant updated: { participant } " ) ​ Communication Events ​ on_app_message async callback Event emitted when a custom app message is received from another participant or via the REST API. Parameters: transport : The DailyTransport instance message : The message content (any type) sender : String identifier of the message sender Copy Ask AI @transport.event_handler ( "on_app_message" ) async def on_app_message ( transport , message : Any, sender : str ): logger.info( f "Message from { sender } : { message } " ) ​ on_call_state_updated async callback Event emitted when the call state changes, normally as a consequence of joining or leaving the call. Parameters: transport : The DailyTransport instance state : String representing the new call state. Learn more about call states . Copy Ask AI @transport.event_handler ( "on_call_state_updated" ) async def on_call_state_updated ( transport , state : str ): logger.info( f "Call state updated: { state } " ) ​ Dial Events ​ Dial-in ​ on_dialin_ready async callback Event emitted when dial-in is ready. This happens after the room has connected to the SIP endpoint and the system is ready to receive dial-in calls. Parameters: transport : The DailyTransport instance sip_endpoint : String containing the SIP endpoint information Copy Ask AI @transport.event_handler ( "on_dialin_ready" ) async def on_dialin_ready ( transport , sip_endpoint : str ): logger.info( f "Dial-in ready at: { sip_endpoint } " ) ​ on_dialin_connected async callback Event emitted when the session with the dial-in remote end is established (i.e. SIP endpoint or PSTN are connectd to the Daily room). Note: connected does not mean media (audio or video) has started flowing between the room and PSTN, it means the room received the connection request and both endpoints are negotiating the media flow. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. See DialinConnectedEvent . Copy Ask AI @transport.event_handler ( "on_dialin_connected" ) async def on_dialin_connected ( transport , data : Any): logger.info( f "Dial-in connected: { data } " ) ​ on_dialin_stopped async callback Event emitted when the dial-in remote end disconnects the call. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. See DialinStoppedEvent . Copy Ask AI @transport.event_handler ( "on_dialin_stopped" ) async def on_dialin_stopped ( transport , data : Any): logger.info( f "Dial-in stopped: { data } " ) ​ on_dialin_error async callback Event emitted in the case of dial-in errors which are fatal and the service cannot proceed. For example, an error in SDP negotiation is fatal to the media/SIP pipeline and will result in dialin-error being triggered. Parameters: transport : The DailyTransport instance data : Dictionary containing error information. See DialinEvent . Copy Ask AI @transport.event_handler ( "on_dialin_error" ) async def on_dialin_error ( transport , data : Any): logger.error( f "Dial-in error: { data } " ) ​ on_dialin_warning async callback Event emitted there is a dial-in non-fatal error, such as the selected codec not being used and a fallback codec being utilized. Parameters: transport : The DailyTransport instance data : Dictionary containing warning information. See DialinEvent . Copy Ask AI @transport.event_handler ( "on_dialin_warning" ) async def on_dialin_warning ( transport , data : Any): logger.warning( f "Dial-in warning: { data } " ) ​ Dial-out ​ on_dialout_answered async callback Event emitted when the session with the dial-out remote end is answered. Parameters: transport : The DailyTransport instance data : Dictionary containing call information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_answered" ) async def on_dialout_answered ( transport , data : Any): logger.info( f "Dial-out answered: { data } " ) ​ on_dialout_connected async callback Event emitted when the session with the dial-out remote end is established. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_connected" ) async def on_dialout_connected ( transport , data : Any): logger.info( f "Dial-out connected: { data } " ) ​ on_dialout_stopped async callback Event emitted when the dial-out session is stopped. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_stopped" ) async def on_dialout_stopped ( transport , data : Any): logger.info( f "Dial-out stopped: { data } " ) ​ on_dialout_error async callback Event emitted in the case of dial-out errors which are fatal and the service cannot proceed. For example, an error in SDP negotiation is fatal to the media/SIP pipeline and will result in dialout-error being triggered. Parameters: transport : The DailyTransport instance data : Dictionary containing error information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_error" ) async def on_dialout_error ( transport , data : Any): logger.error( f "Dial-out error: { data } " ) ​ on_dialout_warning async callback Event emitted there is a dial-out non-fatal error, such as the selected codec not being used and a fallback codec being utilized. Parameters: transport : The DailyTransport instance data : Dictionary containing warning information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_warning" ) async def on_dialout_warning ( transport , data : Any): logger.warning( f "Dial-out warning: { data } " ) ​ Transcription Events ​ on_transcription_message async callback Called when a transcription message is received. This includes both interim and final transcriptions. Parameters: transport : The DailyTransport instance message : Dictionary containing transcription data including. Learn more . Copy Ask AI @transport.event_handler ( "on_transcription_message" ) async def on_transcription_message ( transport , message : Dict[ str , Any]): participant_id = message.get( "participantId" ) text = message.get( "text" ) is_final = message[ "rawResponse" ][ "is_final" ] logger.info( f "Transcription from { participant_id } : { text } (final: { is_final } )" ) ​ Recording Events ​ on_recording_started async callback Called when a room recording starts successfully. Parameters: transport : The DailyTransport instance status : Dictionary containing recording status information. Learn more . Copy Ask AI @transport.event_handler ( "on_recording_started" ) async def on_recording_started ( transport , status : Dict[ str , Any]): logger.info( f "Recording started with status: { status } " ) ​ on_recording_stopped async callback Called when a room recording stops. Parameters: transport : The DailyTransport instance stream_id : String identifier of the stopped recording stream Copy Ask AI @transport.event_handler ( "on_recording_stopped" ) async def on_recording_stopped ( transport , stream_id : str ): logger.info( f "Recording stopped for stream: { stream_id } " ) ​ on_recording_error async callback Called when an error occurs during recording. Parameters: transport : The DailyTransport instance stream_id : String identifier of the recording stream message : Error message describing what went wrong Copy Ask AI @transport.event_handler ( "on_recording_error" ) async def on_recording_error ( transport , stream_id : str , message : str ): logger.error( f "Recording error for stream { stream_id } : { message } " ) ​ Error Events ​ on_error async callback Called when a transport error occurs. Parameters: transport : The DailyTransport instance error : String containing error details Copy Ask AI @transport.event_handler ( "on_error" ) async def on_error ( transport , error : str ): logger.error( f "Transport error: { error } " ) ​ Notes on Callbacks All callbacks are asynchronous Callbacks are executed in the order they were registered Multiple handlers can be registered for the same event Exceptions in callbacks are logged but don’t stop the transport Callbacks should be lightweight to avoid blocking the event loop Heavy processing should be offloaded to separate tasks ​ Frame Types ​ Input Frames ​ InputAudioRawFrame Frame Raw audio data from participants ​ UserImageRawFrame Frame Video frames from participants ​ Output Frames ​ OutputAudioRawFrame Frame Audio data to be sent ​ OutputImageRawFrame Frame Video frames to be sent ​ Methods ​ Room Management ​ participants method Returns a dictionary of all participants currently in the room. Copy Ask AI def participants () -> Dict[ str , Any] See participants for more details. ​ participant_counts method Returns participant count statistics for the room. Copy Ask AI def participant_counts () -> Dict[ str , int ] See participant_counts for more details. ​ send_message async method Sends a message to participants in the Daily room. Messages can be directed to all participants or targeted to a specific participant. Copy Ask AI async def send_message ( frame : TransportMessageFrame | TransportMessageUrgentFrame) Parameters: frame : A transport message frame containing the message to send. Can be either: TransportMessageFrame : Standard message TransportMessageUrgentFrame : High-priority message Message Structure: Messages should conform to the expected format for the receiving system. For RTVI messages, use: Copy Ask AI { "label" : "rtvi-ai" , # Required for RTVI protocol "type" : "message-type" , # Defines how the client will process it "id" : "unique-id" , # Unique identifier for the message "data" : { # Custom payload # Your custom data fields } } Example - Standard Message: Copy Ask AI import uuid # Send a standard message to all participants message_data = { "label" : "rtvi-ai" , "type" : "custom-message" , "id" : str (uuid.uuid4()), "data" : { "message" : "Hello everyone!" , "timestamp" : datetime.datetime.now().isoformat() } } await transport.send_message(TransportMessageFrame( message = message_data)) Example - Urgent Message to Specific Participant: Copy Ask AI # Send urgent message to specific participant urgent_data = { "label" : "rtvi-ai" , "type" : "server-message" , "id" : str (uuid.uuid4()), "data" : { "message" : "Private message" , "importance" : "high" } } await transport.send_message( DailyTransportMessageUrgentFrame( message = urgent_data, participant_id = "participant-123" ) ) The message will be delivered to: All participants if no specific participant_id is set Only the specified participant if participant_id is set Through the Daily app messaging system Can be received by registering an on_app_message handler Use TransportMessageUrgentFrame for high-priority messages that need immediate delivery. ​ Media Control ​ send_image async method Sends an image frame to the room. Copy Ask AI async def send_image ( frame : OutputImageRawFrame | SpriteFrame) -> None Parameters: frame : Image frame to send, either raw image data or sprite animation ​ send_audio async method Sends an audio frame to the room. Copy Ask AI async def send_audio ( frame : OutputAudioRawFrame) -> None Parameters: frame : Audio frame to send ​ Video Management ​ capture_participant_video async method Starts capturing video from a specific participant. Copy Ask AI async def capture_participant_video ( participant_id : str , framerate : int = 30 , video_source : str = "camera" , color_format : str = "RGB" ) -> None Parameters: participant_id : ID of the participant to capture framerate : Target frame rate (default: 30) video_source : Video source type (default: “camera”) color_format : Color format of the video (default: “RGB”) To request an image from the camera stream, set the framerate to 0 and push a UserImageRequestFrame . ​ Transcription Control ​ capture_participant_transcription async method Starts capturing and transcribing audio from a specific participant. Copy Ask AI async def capture_participant_transcription ( participant_id : str ) -> None Parameters: participant_id : ID of the participant to transcribe ​ update_transcription async method Updates the transcription configuration for specific participants or instances. Copy Ask AI async def update_transcription ( participants : List[ str ] | None = None , instance_id : str | None = None ) -> None Parameters: participants : Optional list of participant IDs to transcribe. If None, transcribes all participants. instance_id : Optional specific transcription instance ID to update Example: Copy Ask AI # Update transcription for specific participants await transport.update_transcription( participants = [ "participant-123" , "participant-456" ]) # Update specific transcription instance await transport.update_transcription( instance_id = "transcription-789" ) ​ Recording Control ​ start_recording async method Starts recording the room session. Copy Ask AI async def start_recording ( streaming_settings : Dict = None , stream_id : str = None , force_new : bool = None ) -> None Parameters: streaming_settings : Recording configuration settings stream_id : Optional stream identifier force_new : Force start a new recording See start_recording for more details. ​ stop_recording async method Stops an active recording. Copy Ask AI async def stop_recording ( stream_id : str = None ) -> None Parameters: stream_id : Optional stream identifier to stop specific recording See stop_recording for more details. ​ Dial-out Control ​ start_dialout async method Initiates a dial-out call. Copy Ask AI async def start_dialout ( settings : Dict = None ) -> None Parameters: settings : Dial-out configuration settings See start_dialout for more details. ​ stop_dialout async method Stops an active dial-out call. Copy Ask AI async def stop_dialout ( participant_id : str ) -> None Parameters: participant_id : ID of the participant to disconnect See stop_dialout for more details. ​ send_dtmf async method Sends DTMF tones in an existing dial-out session. Copy Ask AI async def send_dtmf ( settings : Dict = None ) -> None Parameters: settings : DTMF settings See send_dtmf for more details. ​ Subscription Management ​ update_subscriptions async method Updates subscriptions and subscription profiles. This function allows you to update subscription profiles and at the same time assign specific subscription profiles to a participant and even change specific settings for some participants. Copy Ask AI async def update_subscriptions ( participant_settings : Dict = None , profile_settings : Dict = None ) -> None Parameters: participant_settings : Per-participant subscription settings profile_settings : Global subscription profile settings See update_subscriptions for more details. ​ Participant Admin When using the DailyTransport, if your bot is given a meeting token with owner or participant admin permissions, it can remote-control other participants’ capabilities and permissions. With this control, the bot can modify settings for how others send and receive media, how they are seen in a session, and whether they have administrative controls. ​ update_remote_participants async method Updates permissions and settings for specific participants in the room. Copy Ask AI async def update_remote_participants ( remote_participants : Mapping[ str , Any] = None ) -> None Parameters: remote_participants : Dictionary mapping participant IDs to their updated settings This method allows fine-grained control over participant capabilities and permissions in the Daily room. Examples: Copy Ask AI # Mute a participant's microphone await transport.update_remote_participants({ "participant-123" : { "inputsEnabled" : { "microphone" : False } } }) # Revoke send permissions (mutes the participant and prevents them from unmuting) await transport.update_remote_participants({ "participant-123" : { "permissions" : { "canSend" : [] # Empty list revokes all send permissions } } }) # Configure selective audio routing await transport.update_remote_participants({ "participant-123" : { # the customer "permissions" : { "canReceive" : { "base" : False , # Don't receive from anyone by default "byUserId" : { "hold-music" : True # Only receive from hold music player } } } } }) # Set up user-to-user permissions mapping (e.g., when transferring calls) # Assumes customer was previously unable to hear anyone (they were on hold) and agent was speaking to bot await transport.update_remote_participants({ "participant-123" : { # the customer "permissions" : { "base" : False , "canReceive" : { "byUserId" : { "agent" : True }} } }, "participant-456" : { # the agent "permissions" : { "base" : False , "canReceive" : { "byUserId" : { "customer" : True }} } } }) # Promote a participant to be a participant admin await transport.update_remote_participants({ "participant-123" : { "permissions" : { "canAdmin" : [ "participants" ] } } }) Permissions Settings: For comprehensive details on permissions settings, see the Daily API documentation: daily-python’s update_remote_participants() documentation daily-js’s updateParticipant() -> updatePermission and participants().permissions ​ Chat messages ​ send_prebuilt_chat_message async method When testing with Daily Prebuilt, you can send messages to the Prebuilt chat using this method. Parameters: message (str): The chat message to send user_name (str): The name of the user sending the message See send_prebuilt_chat_message for more details. ​ Properties ​ participant_id string Returns the transport’s participant ID in the room. ​ room_url string Returns the transport’s room URL for the room currently in use. ​ Notes Supports real-time audio/video communication Handles WebRTC connection management Provides voice activity detection Includes transcription capabilities Manages participant interactions Supports recording and streaming Thread-safe processing All callbacks are asynchronous Multiple handlers can be registered for the same event Exceptions in callbacks are logged but don’t stop the transport Supported Services SmallWebRTCTransport On this page Overview Installation Configuration Constructor Parameters DailyParams Configuration Audio Output Configuration Audio Input Configuration Video Output Configuration Voice Activity Detection (VAD) Feature Settings Basic Usage Event Callbacks Connection Events Participant Events Communication Events Dial Events Dial-in Dial-out Transcription Events Recording Events Error Events Notes on Callbacks Frame Types Input Frames Output Frames Methods Room Management Media Control Video Management Transcription Control Recording Control Dial-out Control Subscription Management Participant Admin Chat messages Properties Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/transport_daily_689492f4.txt b/transport_daily_689492f4.txt
new file mode 100644
index 0000000000000000000000000000000000000000..fa323689b6b7d259c8a9878654bc70100ddb8e22
--- /dev/null
+++ b/transport_daily_689492f4.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/transport/daily#error-events
+Title: Daily WebRTC - Pipecat
+==================================================
+
+Daily WebRTC - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Transport Daily WebRTC Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Daily WebRTC SmallWebRTCTransport FastAPI WebSocket WebSocket Server Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview DailyTransport provides real-time audio and video communication capabilities using Daily’s WebRTC platform. It supports bidirectional audio/video streams, transcription, voice activity detection (VAD), and participant management. ​ Installation To use DailyTransport , install the required dependencies: Copy Ask AI pip install "pipecat-ai[daily]" You’ll also need to set up your Daily API key as an environment variable: DAILY_API_KEY . You can obtain a Daily API key by signing up at Daily . ​ Configuration ​ Constructor Parameters ​ room_url str required Daily room URL ​ token str | None Daily room token ​ bot_name str required Name of the bot in the room ​ params DailyParams default: "DailyParams()" Transport configuration parameters ​ DailyParams Configuration ​ api_url str default: "https://api.daily.co/v1" Daily API endpoint URL ​ api_key str default: "" Daily API key for authentication ​ Audio Output Configuration ​ audio_out_enabled bool default: "False" Enable audio output capabilities ​ audio_out_is_live bool default: "False" Enable live audio streaming mode ​ audio_out_sample_rate int default: "None" Audio output sample rate in Hz ​ audio_out_channels int default: "1" Number of audio output channels ​ audio_out_bitrate int default: "96000" Audio output bitrate in bits per second ​ Audio Input Configuration ​ audio_in_enabled bool default: "False" Enable audio input capabilities ​ audio_in_passthrough bool default: "False" When enabled, incoming audio frames are pushed downstream ​ audio_in_sample_rate int default: "None" Audio input sample rate in Hz ​ audio_in_channels int default: "1" Number of audio input channels ​ audio_in_filter Optional[BaseAudioFilter] default: "None" Audio filter for input processing. Supported filters are: KrispFilter() and NoisereduceFilter() . See the KrispFilter() reference docs and NoisereduceFilter() reference docs ` for more information. ​ Video Output Configuration ​ video_out_enabled bool default: "False" Enable video output capabilities ​ video_out_is_live bool default: "False" Enable live video streaming mode ​ video_out_width int default: "1024" Video output width in pixels ​ video_out_height int default: "768" Video output height in pixels ​ video_out_bitrate int default: "800000" Video output bitrate in bits per second ​ video_out_framerate int default: "30" Video output frame rate ​ video_out_color_format str default: "RGB" Video color format (RGB, BGR, etc.) ​ Voice Activity Detection (VAD) ​ vad_analyzer VADAnalyzer | None default: "None" Voice Activity Detection analyzer. You can set this to either SileroVADAnalyzer() or WebRTCVADAnalyzer() . SileroVADAnalyzer is the recommended option. Learn more about the SileroVADAnalyzer . ​ Feature Settings ​ dialin_settings Optional[DailyDialinSettings] default: "None" Configuration for dial-in functionality ​ transcription_enabled bool default: "False" Enable real-time transcription ​ transcription_settings DailyTranscriptionSettings Configuration for transcription features Copy Ask AI class DailyTranscriptionSettings ( BaseModel ): language: str = "en" # Default language model: str = "nova-2-general" # Transcription model profanity_filter: bool = True # Filter profanity redact: bool = False # Redact sensitive information endpointing: bool = True # Enable speech endpointing punctuate: bool = True # Add punctuation includeRawResponse: bool = True # Include raw API response extra: Mapping[ str , Any] = { "interim_results" : True # Provide any Deepgram-specific settings } ​ Basic Usage Copy Ask AI from pipecat.transports.services.daily import DailyTransport, DailyParams from pipecat.audio.vad.silero import SileroVADAnalyzer # Configure transport transport = DailyTransport( room_url = "https://your-domain.daily.co/room-name" , token = "your-room-token" , bot_name = "AI Assistant" , params = DailyParams( audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer( params = VADParams( stop_secs = 0.5 )), ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), # Handle incoming audio/video stt, # Speech-to-text llm, # Language model tts, # Text-to-speech transport.output() # Handle outgoing audio/video ]) # Use transport event handlers to manage the call lifecycle # Event handler for when the user joins the room: @transport.event_handler ( "on_first_participant_joined" ) async def on_first_participant_joined ( transport , participant ): # Start transcription for the user await transport.capture_participant_transcription(participant[ "id" ]) # Kick off the conversation. await task.queue_frames([context_aggregator.user().get_context_frame()]) # Event handler for when the user leaves the room: @transport.event_handler ( "on_participant_left" ) async def on_participant_left ( transport , participant , reason ): # Cancel the pipeline, which stops processing and removes the bot from the room await task.cancel() ​ Event Callbacks DailyTransport provides a comprehensive callback system for handling various events. Register callbacks using the @transport.event_handler() decorator. ​ Connection Events ​ on_joined async callback Called when the bot successfully joins the room. Parameters: transport : The DailyTransport instance data : Dictionary containing room join information Copy Ask AI @transport.event_handler ( "on_joined" ) async def on_joined ( transport , data : Dict[ str , Any]): logger.info( f "Joined room with data: { data } " ) ​ on_left async callback Called when the bot leaves the room. Parameters: transport : The DailyTransport instance Copy Ask AI @transport.event_handler ( "on_left" ) async def on_left ( transport ): logger.info( "Left room" ) ​ Participant Events ​ on_first_participant_joined async callback Called when the first participant joins the room. Useful for initializing conversations. Parameters: transport : The DailyTransport instance participant : Dictionary containing participant information Copy Ask AI @transport.event_handler ( "on_first_participant_joined" ) async def on_first_participant_joined ( transport , participant : Dict[ str , Any]): await transport.capture_participant_transcription(participant[ "id" ]) await task.queue_frames([LLMMessagesFrame(initial_messages)]) ​ on_participant_joined async callback Called when any participant joins the room. Parameters: transport : The DailyTransport instance participant : Dictionary containing participant information Copy Ask AI @transport.event_handler ( "on_participant_joined" ) async def on_participant_joined ( transport , participant : Dict[ str , Any]): logger.info( f "Participant joined: { participant[ 'id' ] } " ) ​ on_participant_left async callback Called when a participant leaves the room. Parameters: transport : The DailyTransport instance participant : Dictionary containing participant information reason : String describing why the participant left, “leftCall” | “hidden” Copy Ask AI @transport.event_handler ( "on_participant_left" ) async def on_participant_left ( transport , participant : Dict[ str , Any], reason : str ): logger.info( f "Participant { participant[ 'id' ] } left: { reason } " ) # Cancel the pipeline task to stop processing and remove the bot from the Daily room await task.cancel() ​ on_participant_updated async callback Event emitted when a participant is updated. This can mean either the participant’s metadata was updated, or the tracks belonging to the participant changed. Parameters: transport : The DailyTransport instance participant : Dictionary containing updated participant information Copy Ask AI @transport.event_handler ( "on_participant_updated" ) async def on_participant_updated ( transport , participant : Dict[ str , Any]): logger.info( f "Participant updated: { participant } " ) ​ Communication Events ​ on_app_message async callback Event emitted when a custom app message is received from another participant or via the REST API. Parameters: transport : The DailyTransport instance message : The message content (any type) sender : String identifier of the message sender Copy Ask AI @transport.event_handler ( "on_app_message" ) async def on_app_message ( transport , message : Any, sender : str ): logger.info( f "Message from { sender } : { message } " ) ​ on_call_state_updated async callback Event emitted when the call state changes, normally as a consequence of joining or leaving the call. Parameters: transport : The DailyTransport instance state : String representing the new call state. Learn more about call states . Copy Ask AI @transport.event_handler ( "on_call_state_updated" ) async def on_call_state_updated ( transport , state : str ): logger.info( f "Call state updated: { state } " ) ​ Dial Events ​ Dial-in ​ on_dialin_ready async callback Event emitted when dial-in is ready. This happens after the room has connected to the SIP endpoint and the system is ready to receive dial-in calls. Parameters: transport : The DailyTransport instance sip_endpoint : String containing the SIP endpoint information Copy Ask AI @transport.event_handler ( "on_dialin_ready" ) async def on_dialin_ready ( transport , sip_endpoint : str ): logger.info( f "Dial-in ready at: { sip_endpoint } " ) ​ on_dialin_connected async callback Event emitted when the session with the dial-in remote end is established (i.e. SIP endpoint or PSTN are connectd to the Daily room). Note: connected does not mean media (audio or video) has started flowing between the room and PSTN, it means the room received the connection request and both endpoints are negotiating the media flow. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. See DialinConnectedEvent . Copy Ask AI @transport.event_handler ( "on_dialin_connected" ) async def on_dialin_connected ( transport , data : Any): logger.info( f "Dial-in connected: { data } " ) ​ on_dialin_stopped async callback Event emitted when the dial-in remote end disconnects the call. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. See DialinStoppedEvent . Copy Ask AI @transport.event_handler ( "on_dialin_stopped" ) async def on_dialin_stopped ( transport , data : Any): logger.info( f "Dial-in stopped: { data } " ) ​ on_dialin_error async callback Event emitted in the case of dial-in errors which are fatal and the service cannot proceed. For example, an error in SDP negotiation is fatal to the media/SIP pipeline and will result in dialin-error being triggered. Parameters: transport : The DailyTransport instance data : Dictionary containing error information. See DialinEvent . Copy Ask AI @transport.event_handler ( "on_dialin_error" ) async def on_dialin_error ( transport , data : Any): logger.error( f "Dial-in error: { data } " ) ​ on_dialin_warning async callback Event emitted there is a dial-in non-fatal error, such as the selected codec not being used and a fallback codec being utilized. Parameters: transport : The DailyTransport instance data : Dictionary containing warning information. See DialinEvent . Copy Ask AI @transport.event_handler ( "on_dialin_warning" ) async def on_dialin_warning ( transport , data : Any): logger.warning( f "Dial-in warning: { data } " ) ​ Dial-out ​ on_dialout_answered async callback Event emitted when the session with the dial-out remote end is answered. Parameters: transport : The DailyTransport instance data : Dictionary containing call information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_answered" ) async def on_dialout_answered ( transport , data : Any): logger.info( f "Dial-out answered: { data } " ) ​ on_dialout_connected async callback Event emitted when the session with the dial-out remote end is established. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_connected" ) async def on_dialout_connected ( transport , data : Any): logger.info( f "Dial-out connected: { data } " ) ​ on_dialout_stopped async callback Event emitted when the dial-out session is stopped. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_stopped" ) async def on_dialout_stopped ( transport , data : Any): logger.info( f "Dial-out stopped: { data } " ) ​ on_dialout_error async callback Event emitted in the case of dial-out errors which are fatal and the service cannot proceed. For example, an error in SDP negotiation is fatal to the media/SIP pipeline and will result in dialout-error being triggered. Parameters: transport : The DailyTransport instance data : Dictionary containing error information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_error" ) async def on_dialout_error ( transport , data : Any): logger.error( f "Dial-out error: { data } " ) ​ on_dialout_warning async callback Event emitted there is a dial-out non-fatal error, such as the selected codec not being used and a fallback codec being utilized. Parameters: transport : The DailyTransport instance data : Dictionary containing warning information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_warning" ) async def on_dialout_warning ( transport , data : Any): logger.warning( f "Dial-out warning: { data } " ) ​ Transcription Events ​ on_transcription_message async callback Called when a transcription message is received. This includes both interim and final transcriptions. Parameters: transport : The DailyTransport instance message : Dictionary containing transcription data including. Learn more . Copy Ask AI @transport.event_handler ( "on_transcription_message" ) async def on_transcription_message ( transport , message : Dict[ str , Any]): participant_id = message.get( "participantId" ) text = message.get( "text" ) is_final = message[ "rawResponse" ][ "is_final" ] logger.info( f "Transcription from { participant_id } : { text } (final: { is_final } )" ) ​ Recording Events ​ on_recording_started async callback Called when a room recording starts successfully. Parameters: transport : The DailyTransport instance status : Dictionary containing recording status information. Learn more . Copy Ask AI @transport.event_handler ( "on_recording_started" ) async def on_recording_started ( transport , status : Dict[ str , Any]): logger.info( f "Recording started with status: { status } " ) ​ on_recording_stopped async callback Called when a room recording stops. Parameters: transport : The DailyTransport instance stream_id : String identifier of the stopped recording stream Copy Ask AI @transport.event_handler ( "on_recording_stopped" ) async def on_recording_stopped ( transport , stream_id : str ): logger.info( f "Recording stopped for stream: { stream_id } " ) ​ on_recording_error async callback Called when an error occurs during recording. Parameters: transport : The DailyTransport instance stream_id : String identifier of the recording stream message : Error message describing what went wrong Copy Ask AI @transport.event_handler ( "on_recording_error" ) async def on_recording_error ( transport , stream_id : str , message : str ): logger.error( f "Recording error for stream { stream_id } : { message } " ) ​ Error Events ​ on_error async callback Called when a transport error occurs. Parameters: transport : The DailyTransport instance error : String containing error details Copy Ask AI @transport.event_handler ( "on_error" ) async def on_error ( transport , error : str ): logger.error( f "Transport error: { error } " ) ​ Notes on Callbacks All callbacks are asynchronous Callbacks are executed in the order they were registered Multiple handlers can be registered for the same event Exceptions in callbacks are logged but don’t stop the transport Callbacks should be lightweight to avoid blocking the event loop Heavy processing should be offloaded to separate tasks ​ Frame Types ​ Input Frames ​ InputAudioRawFrame Frame Raw audio data from participants ​ UserImageRawFrame Frame Video frames from participants ​ Output Frames ​ OutputAudioRawFrame Frame Audio data to be sent ​ OutputImageRawFrame Frame Video frames to be sent ​ Methods ​ Room Management ​ participants method Returns a dictionary of all participants currently in the room. Copy Ask AI def participants () -> Dict[ str , Any] See participants for more details. ​ participant_counts method Returns participant count statistics for the room. Copy Ask AI def participant_counts () -> Dict[ str , int ] See participant_counts for more details. ​ send_message async method Sends a message to participants in the Daily room. Messages can be directed to all participants or targeted to a specific participant. Copy Ask AI async def send_message ( frame : TransportMessageFrame | TransportMessageUrgentFrame) Parameters: frame : A transport message frame containing the message to send. Can be either: TransportMessageFrame : Standard message TransportMessageUrgentFrame : High-priority message Message Structure: Messages should conform to the expected format for the receiving system. For RTVI messages, use: Copy Ask AI { "label" : "rtvi-ai" , # Required for RTVI protocol "type" : "message-type" , # Defines how the client will process it "id" : "unique-id" , # Unique identifier for the message "data" : { # Custom payload # Your custom data fields } } Example - Standard Message: Copy Ask AI import uuid # Send a standard message to all participants message_data = { "label" : "rtvi-ai" , "type" : "custom-message" , "id" : str (uuid.uuid4()), "data" : { "message" : "Hello everyone!" , "timestamp" : datetime.datetime.now().isoformat() } } await transport.send_message(TransportMessageFrame( message = message_data)) Example - Urgent Message to Specific Participant: Copy Ask AI # Send urgent message to specific participant urgent_data = { "label" : "rtvi-ai" , "type" : "server-message" , "id" : str (uuid.uuid4()), "data" : { "message" : "Private message" , "importance" : "high" } } await transport.send_message( DailyTransportMessageUrgentFrame( message = urgent_data, participant_id = "participant-123" ) ) The message will be delivered to: All participants if no specific participant_id is set Only the specified participant if participant_id is set Through the Daily app messaging system Can be received by registering an on_app_message handler Use TransportMessageUrgentFrame for high-priority messages that need immediate delivery. ​ Media Control ​ send_image async method Sends an image frame to the room. Copy Ask AI async def send_image ( frame : OutputImageRawFrame | SpriteFrame) -> None Parameters: frame : Image frame to send, either raw image data or sprite animation ​ send_audio async method Sends an audio frame to the room. Copy Ask AI async def send_audio ( frame : OutputAudioRawFrame) -> None Parameters: frame : Audio frame to send ​ Video Management ​ capture_participant_video async method Starts capturing video from a specific participant. Copy Ask AI async def capture_participant_video ( participant_id : str , framerate : int = 30 , video_source : str = "camera" , color_format : str = "RGB" ) -> None Parameters: participant_id : ID of the participant to capture framerate : Target frame rate (default: 30) video_source : Video source type (default: “camera”) color_format : Color format of the video (default: “RGB”) To request an image from the camera stream, set the framerate to 0 and push a UserImageRequestFrame . ​ Transcription Control ​ capture_participant_transcription async method Starts capturing and transcribing audio from a specific participant. Copy Ask AI async def capture_participant_transcription ( participant_id : str ) -> None Parameters: participant_id : ID of the participant to transcribe ​ update_transcription async method Updates the transcription configuration for specific participants or instances. Copy Ask AI async def update_transcription ( participants : List[ str ] | None = None , instance_id : str | None = None ) -> None Parameters: participants : Optional list of participant IDs to transcribe. If None, transcribes all participants. instance_id : Optional specific transcription instance ID to update Example: Copy Ask AI # Update transcription for specific participants await transport.update_transcription( participants = [ "participant-123" , "participant-456" ]) # Update specific transcription instance await transport.update_transcription( instance_id = "transcription-789" ) ​ Recording Control ​ start_recording async method Starts recording the room session. Copy Ask AI async def start_recording ( streaming_settings : Dict = None , stream_id : str = None , force_new : bool = None ) -> None Parameters: streaming_settings : Recording configuration settings stream_id : Optional stream identifier force_new : Force start a new recording See start_recording for more details. ​ stop_recording async method Stops an active recording. Copy Ask AI async def stop_recording ( stream_id : str = None ) -> None Parameters: stream_id : Optional stream identifier to stop specific recording See stop_recording for more details. ​ Dial-out Control ​ start_dialout async method Initiates a dial-out call. Copy Ask AI async def start_dialout ( settings : Dict = None ) -> None Parameters: settings : Dial-out configuration settings See start_dialout for more details. ​ stop_dialout async method Stops an active dial-out call. Copy Ask AI async def stop_dialout ( participant_id : str ) -> None Parameters: participant_id : ID of the participant to disconnect See stop_dialout for more details. ​ send_dtmf async method Sends DTMF tones in an existing dial-out session. Copy Ask AI async def send_dtmf ( settings : Dict = None ) -> None Parameters: settings : DTMF settings See send_dtmf for more details. ​ Subscription Management ​ update_subscriptions async method Updates subscriptions and subscription profiles. This function allows you to update subscription profiles and at the same time assign specific subscription profiles to a participant and even change specific settings for some participants. Copy Ask AI async def update_subscriptions ( participant_settings : Dict = None , profile_settings : Dict = None ) -> None Parameters: participant_settings : Per-participant subscription settings profile_settings : Global subscription profile settings See update_subscriptions for more details. ​ Participant Admin When using the DailyTransport, if your bot is given a meeting token with owner or participant admin permissions, it can remote-control other participants’ capabilities and permissions. With this control, the bot can modify settings for how others send and receive media, how they are seen in a session, and whether they have administrative controls. ​ update_remote_participants async method Updates permissions and settings for specific participants in the room. Copy Ask AI async def update_remote_participants ( remote_participants : Mapping[ str , Any] = None ) -> None Parameters: remote_participants : Dictionary mapping participant IDs to their updated settings This method allows fine-grained control over participant capabilities and permissions in the Daily room. Examples: Copy Ask AI # Mute a participant's microphone await transport.update_remote_participants({ "participant-123" : { "inputsEnabled" : { "microphone" : False } } }) # Revoke send permissions (mutes the participant and prevents them from unmuting) await transport.update_remote_participants({ "participant-123" : { "permissions" : { "canSend" : [] # Empty list revokes all send permissions } } }) # Configure selective audio routing await transport.update_remote_participants({ "participant-123" : { # the customer "permissions" : { "canReceive" : { "base" : False , # Don't receive from anyone by default "byUserId" : { "hold-music" : True # Only receive from hold music player } } } } }) # Set up user-to-user permissions mapping (e.g., when transferring calls) # Assumes customer was previously unable to hear anyone (they were on hold) and agent was speaking to bot await transport.update_remote_participants({ "participant-123" : { # the customer "permissions" : { "base" : False , "canReceive" : { "byUserId" : { "agent" : True }} } }, "participant-456" : { # the agent "permissions" : { "base" : False , "canReceive" : { "byUserId" : { "customer" : True }} } } }) # Promote a participant to be a participant admin await transport.update_remote_participants({ "participant-123" : { "permissions" : { "canAdmin" : [ "participants" ] } } }) Permissions Settings: For comprehensive details on permissions settings, see the Daily API documentation: daily-python’s update_remote_participants() documentation daily-js’s updateParticipant() -> updatePermission and participants().permissions ​ Chat messages ​ send_prebuilt_chat_message async method When testing with Daily Prebuilt, you can send messages to the Prebuilt chat using this method. Parameters: message (str): The chat message to send user_name (str): The name of the user sending the message See send_prebuilt_chat_message for more details. ​ Properties ​ participant_id string Returns the transport’s participant ID in the room. ​ room_url string Returns the transport’s room URL for the room currently in use. ​ Notes Supports real-time audio/video communication Handles WebRTC connection management Provides voice activity detection Includes transcription capabilities Manages participant interactions Supports recording and streaming Thread-safe processing All callbacks are asynchronous Multiple handlers can be registered for the same event Exceptions in callbacks are logged but don’t stop the transport Supported Services SmallWebRTCTransport On this page Overview Installation Configuration Constructor Parameters DailyParams Configuration Audio Output Configuration Audio Input Configuration Video Output Configuration Voice Activity Detection (VAD) Feature Settings Basic Usage Event Callbacks Connection Events Participant Events Communication Events Dial Events Dial-in Dial-out Transcription Events Recording Events Error Events Notes on Callbacks Frame Types Input Frames Output Frames Methods Room Management Media Control Video Management Transcription Control Recording Control Dial-out Control Subscription Management Participant Admin Chat messages Properties Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/transport_daily_68f49f07.txt b/transport_daily_68f49f07.txt
new file mode 100644
index 0000000000000000000000000000000000000000..3b97a3dfdba44922e414505b86ae5d26f6481e3d
--- /dev/null
+++ b/transport_daily_68f49f07.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/transport/daily#param-on-recording-started
+Title: Daily WebRTC - Pipecat
+==================================================
+
+Daily WebRTC - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Transport Daily WebRTC Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Daily WebRTC SmallWebRTCTransport FastAPI WebSocket WebSocket Server Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview DailyTransport provides real-time audio and video communication capabilities using Daily’s WebRTC platform. It supports bidirectional audio/video streams, transcription, voice activity detection (VAD), and participant management. ​ Installation To use DailyTransport , install the required dependencies: Copy Ask AI pip install "pipecat-ai[daily]" You’ll also need to set up your Daily API key as an environment variable: DAILY_API_KEY . You can obtain a Daily API key by signing up at Daily . ​ Configuration ​ Constructor Parameters ​ room_url str required Daily room URL ​ token str | None Daily room token ​ bot_name str required Name of the bot in the room ​ params DailyParams default: "DailyParams()" Transport configuration parameters ​ DailyParams Configuration ​ api_url str default: "https://api.daily.co/v1" Daily API endpoint URL ​ api_key str default: "" Daily API key for authentication ​ Audio Output Configuration ​ audio_out_enabled bool default: "False" Enable audio output capabilities ​ audio_out_is_live bool default: "False" Enable live audio streaming mode ​ audio_out_sample_rate int default: "None" Audio output sample rate in Hz ​ audio_out_channels int default: "1" Number of audio output channels ​ audio_out_bitrate int default: "96000" Audio output bitrate in bits per second ​ Audio Input Configuration ​ audio_in_enabled bool default: "False" Enable audio input capabilities ​ audio_in_passthrough bool default: "False" When enabled, incoming audio frames are pushed downstream ​ audio_in_sample_rate int default: "None" Audio input sample rate in Hz ​ audio_in_channels int default: "1" Number of audio input channels ​ audio_in_filter Optional[BaseAudioFilter] default: "None" Audio filter for input processing. Supported filters are: KrispFilter() and NoisereduceFilter() . See the KrispFilter() reference docs and NoisereduceFilter() reference docs ` for more information. ​ Video Output Configuration ​ video_out_enabled bool default: "False" Enable video output capabilities ​ video_out_is_live bool default: "False" Enable live video streaming mode ​ video_out_width int default: "1024" Video output width in pixels ​ video_out_height int default: "768" Video output height in pixels ​ video_out_bitrate int default: "800000" Video output bitrate in bits per second ​ video_out_framerate int default: "30" Video output frame rate ​ video_out_color_format str default: "RGB" Video color format (RGB, BGR, etc.) ​ Voice Activity Detection (VAD) ​ vad_analyzer VADAnalyzer | None default: "None" Voice Activity Detection analyzer. You can set this to either SileroVADAnalyzer() or WebRTCVADAnalyzer() . SileroVADAnalyzer is the recommended option. Learn more about the SileroVADAnalyzer . ​ Feature Settings ​ dialin_settings Optional[DailyDialinSettings] default: "None" Configuration for dial-in functionality ​ transcription_enabled bool default: "False" Enable real-time transcription ​ transcription_settings DailyTranscriptionSettings Configuration for transcription features Copy Ask AI class DailyTranscriptionSettings ( BaseModel ): language: str = "en" # Default language model: str = "nova-2-general" # Transcription model profanity_filter: bool = True # Filter profanity redact: bool = False # Redact sensitive information endpointing: bool = True # Enable speech endpointing punctuate: bool = True # Add punctuation includeRawResponse: bool = True # Include raw API response extra: Mapping[ str , Any] = { "interim_results" : True # Provide any Deepgram-specific settings } ​ Basic Usage Copy Ask AI from pipecat.transports.services.daily import DailyTransport, DailyParams from pipecat.audio.vad.silero import SileroVADAnalyzer # Configure transport transport = DailyTransport( room_url = "https://your-domain.daily.co/room-name" , token = "your-room-token" , bot_name = "AI Assistant" , params = DailyParams( audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer( params = VADParams( stop_secs = 0.5 )), ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), # Handle incoming audio/video stt, # Speech-to-text llm, # Language model tts, # Text-to-speech transport.output() # Handle outgoing audio/video ]) # Use transport event handlers to manage the call lifecycle # Event handler for when the user joins the room: @transport.event_handler ( "on_first_participant_joined" ) async def on_first_participant_joined ( transport , participant ): # Start transcription for the user await transport.capture_participant_transcription(participant[ "id" ]) # Kick off the conversation. await task.queue_frames([context_aggregator.user().get_context_frame()]) # Event handler for when the user leaves the room: @transport.event_handler ( "on_participant_left" ) async def on_participant_left ( transport , participant , reason ): # Cancel the pipeline, which stops processing and removes the bot from the room await task.cancel() ​ Event Callbacks DailyTransport provides a comprehensive callback system for handling various events. Register callbacks using the @transport.event_handler() decorator. ​ Connection Events ​ on_joined async callback Called when the bot successfully joins the room. Parameters: transport : The DailyTransport instance data : Dictionary containing room join information Copy Ask AI @transport.event_handler ( "on_joined" ) async def on_joined ( transport , data : Dict[ str , Any]): logger.info( f "Joined room with data: { data } " ) ​ on_left async callback Called when the bot leaves the room. Parameters: transport : The DailyTransport instance Copy Ask AI @transport.event_handler ( "on_left" ) async def on_left ( transport ): logger.info( "Left room" ) ​ Participant Events ​ on_first_participant_joined async callback Called when the first participant joins the room. Useful for initializing conversations. Parameters: transport : The DailyTransport instance participant : Dictionary containing participant information Copy Ask AI @transport.event_handler ( "on_first_participant_joined" ) async def on_first_participant_joined ( transport , participant : Dict[ str , Any]): await transport.capture_participant_transcription(participant[ "id" ]) await task.queue_frames([LLMMessagesFrame(initial_messages)]) ​ on_participant_joined async callback Called when any participant joins the room. Parameters: transport : The DailyTransport instance participant : Dictionary containing participant information Copy Ask AI @transport.event_handler ( "on_participant_joined" ) async def on_participant_joined ( transport , participant : Dict[ str , Any]): logger.info( f "Participant joined: { participant[ 'id' ] } " ) ​ on_participant_left async callback Called when a participant leaves the room. Parameters: transport : The DailyTransport instance participant : Dictionary containing participant information reason : String describing why the participant left, “leftCall” | “hidden” Copy Ask AI @transport.event_handler ( "on_participant_left" ) async def on_participant_left ( transport , participant : Dict[ str , Any], reason : str ): logger.info( f "Participant { participant[ 'id' ] } left: { reason } " ) # Cancel the pipeline task to stop processing and remove the bot from the Daily room await task.cancel() ​ on_participant_updated async callback Event emitted when a participant is updated. This can mean either the participant’s metadata was updated, or the tracks belonging to the participant changed. Parameters: transport : The DailyTransport instance participant : Dictionary containing updated participant information Copy Ask AI @transport.event_handler ( "on_participant_updated" ) async def on_participant_updated ( transport , participant : Dict[ str , Any]): logger.info( f "Participant updated: { participant } " ) ​ Communication Events ​ on_app_message async callback Event emitted when a custom app message is received from another participant or via the REST API. Parameters: transport : The DailyTransport instance message : The message content (any type) sender : String identifier of the message sender Copy Ask AI @transport.event_handler ( "on_app_message" ) async def on_app_message ( transport , message : Any, sender : str ): logger.info( f "Message from { sender } : { message } " ) ​ on_call_state_updated async callback Event emitted when the call state changes, normally as a consequence of joining or leaving the call. Parameters: transport : The DailyTransport instance state : String representing the new call state. Learn more about call states . Copy Ask AI @transport.event_handler ( "on_call_state_updated" ) async def on_call_state_updated ( transport , state : str ): logger.info( f "Call state updated: { state } " ) ​ Dial Events ​ Dial-in ​ on_dialin_ready async callback Event emitted when dial-in is ready. This happens after the room has connected to the SIP endpoint and the system is ready to receive dial-in calls. Parameters: transport : The DailyTransport instance sip_endpoint : String containing the SIP endpoint information Copy Ask AI @transport.event_handler ( "on_dialin_ready" ) async def on_dialin_ready ( transport , sip_endpoint : str ): logger.info( f "Dial-in ready at: { sip_endpoint } " ) ​ on_dialin_connected async callback Event emitted when the session with the dial-in remote end is established (i.e. SIP endpoint or PSTN are connectd to the Daily room). Note: connected does not mean media (audio or video) has started flowing between the room and PSTN, it means the room received the connection request and both endpoints are negotiating the media flow. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. See DialinConnectedEvent . Copy Ask AI @transport.event_handler ( "on_dialin_connected" ) async def on_dialin_connected ( transport , data : Any): logger.info( f "Dial-in connected: { data } " ) ​ on_dialin_stopped async callback Event emitted when the dial-in remote end disconnects the call. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. See DialinStoppedEvent . Copy Ask AI @transport.event_handler ( "on_dialin_stopped" ) async def on_dialin_stopped ( transport , data : Any): logger.info( f "Dial-in stopped: { data } " ) ​ on_dialin_error async callback Event emitted in the case of dial-in errors which are fatal and the service cannot proceed. For example, an error in SDP negotiation is fatal to the media/SIP pipeline and will result in dialin-error being triggered. Parameters: transport : The DailyTransport instance data : Dictionary containing error information. See DialinEvent . Copy Ask AI @transport.event_handler ( "on_dialin_error" ) async def on_dialin_error ( transport , data : Any): logger.error( f "Dial-in error: { data } " ) ​ on_dialin_warning async callback Event emitted there is a dial-in non-fatal error, such as the selected codec not being used and a fallback codec being utilized. Parameters: transport : The DailyTransport instance data : Dictionary containing warning information. See DialinEvent . Copy Ask AI @transport.event_handler ( "on_dialin_warning" ) async def on_dialin_warning ( transport , data : Any): logger.warning( f "Dial-in warning: { data } " ) ​ Dial-out ​ on_dialout_answered async callback Event emitted when the session with the dial-out remote end is answered. Parameters: transport : The DailyTransport instance data : Dictionary containing call information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_answered" ) async def on_dialout_answered ( transport , data : Any): logger.info( f "Dial-out answered: { data } " ) ​ on_dialout_connected async callback Event emitted when the session with the dial-out remote end is established. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_connected" ) async def on_dialout_connected ( transport , data : Any): logger.info( f "Dial-out connected: { data } " ) ​ on_dialout_stopped async callback Event emitted when the dial-out session is stopped. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_stopped" ) async def on_dialout_stopped ( transport , data : Any): logger.info( f "Dial-out stopped: { data } " ) ​ on_dialout_error async callback Event emitted in the case of dial-out errors which are fatal and the service cannot proceed. For example, an error in SDP negotiation is fatal to the media/SIP pipeline and will result in dialout-error being triggered. Parameters: transport : The DailyTransport instance data : Dictionary containing error information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_error" ) async def on_dialout_error ( transport , data : Any): logger.error( f "Dial-out error: { data } " ) ​ on_dialout_warning async callback Event emitted there is a dial-out non-fatal error, such as the selected codec not being used and a fallback codec being utilized. Parameters: transport : The DailyTransport instance data : Dictionary containing warning information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_warning" ) async def on_dialout_warning ( transport , data : Any): logger.warning( f "Dial-out warning: { data } " ) ​ Transcription Events ​ on_transcription_message async callback Called when a transcription message is received. This includes both interim and final transcriptions. Parameters: transport : The DailyTransport instance message : Dictionary containing transcription data including. Learn more . Copy Ask AI @transport.event_handler ( "on_transcription_message" ) async def on_transcription_message ( transport , message : Dict[ str , Any]): participant_id = message.get( "participantId" ) text = message.get( "text" ) is_final = message[ "rawResponse" ][ "is_final" ] logger.info( f "Transcription from { participant_id } : { text } (final: { is_final } )" ) ​ Recording Events ​ on_recording_started async callback Called when a room recording starts successfully. Parameters: transport : The DailyTransport instance status : Dictionary containing recording status information. Learn more . Copy Ask AI @transport.event_handler ( "on_recording_started" ) async def on_recording_started ( transport , status : Dict[ str , Any]): logger.info( f "Recording started with status: { status } " ) ​ on_recording_stopped async callback Called when a room recording stops. Parameters: transport : The DailyTransport instance stream_id : String identifier of the stopped recording stream Copy Ask AI @transport.event_handler ( "on_recording_stopped" ) async def on_recording_stopped ( transport , stream_id : str ): logger.info( f "Recording stopped for stream: { stream_id } " ) ​ on_recording_error async callback Called when an error occurs during recording. Parameters: transport : The DailyTransport instance stream_id : String identifier of the recording stream message : Error message describing what went wrong Copy Ask AI @transport.event_handler ( "on_recording_error" ) async def on_recording_error ( transport , stream_id : str , message : str ): logger.error( f "Recording error for stream { stream_id } : { message } " ) ​ Error Events ​ on_error async callback Called when a transport error occurs. Parameters: transport : The DailyTransport instance error : String containing error details Copy Ask AI @transport.event_handler ( "on_error" ) async def on_error ( transport , error : str ): logger.error( f "Transport error: { error } " ) ​ Notes on Callbacks All callbacks are asynchronous Callbacks are executed in the order they were registered Multiple handlers can be registered for the same event Exceptions in callbacks are logged but don’t stop the transport Callbacks should be lightweight to avoid blocking the event loop Heavy processing should be offloaded to separate tasks ​ Frame Types ​ Input Frames ​ InputAudioRawFrame Frame Raw audio data from participants ​ UserImageRawFrame Frame Video frames from participants ​ Output Frames ​ OutputAudioRawFrame Frame Audio data to be sent ​ OutputImageRawFrame Frame Video frames to be sent ​ Methods ​ Room Management ​ participants method Returns a dictionary of all participants currently in the room. Copy Ask AI def participants () -> Dict[ str , Any] See participants for more details. ​ participant_counts method Returns participant count statistics for the room. Copy Ask AI def participant_counts () -> Dict[ str , int ] See participant_counts for more details. ​ send_message async method Sends a message to participants in the Daily room. Messages can be directed to all participants or targeted to a specific participant. Copy Ask AI async def send_message ( frame : TransportMessageFrame | TransportMessageUrgentFrame) Parameters: frame : A transport message frame containing the message to send. Can be either: TransportMessageFrame : Standard message TransportMessageUrgentFrame : High-priority message Message Structure: Messages should conform to the expected format for the receiving system. For RTVI messages, use: Copy Ask AI { "label" : "rtvi-ai" , # Required for RTVI protocol "type" : "message-type" , # Defines how the client will process it "id" : "unique-id" , # Unique identifier for the message "data" : { # Custom payload # Your custom data fields } } Example - Standard Message: Copy Ask AI import uuid # Send a standard message to all participants message_data = { "label" : "rtvi-ai" , "type" : "custom-message" , "id" : str (uuid.uuid4()), "data" : { "message" : "Hello everyone!" , "timestamp" : datetime.datetime.now().isoformat() } } await transport.send_message(TransportMessageFrame( message = message_data)) Example - Urgent Message to Specific Participant: Copy Ask AI # Send urgent message to specific participant urgent_data = { "label" : "rtvi-ai" , "type" : "server-message" , "id" : str (uuid.uuid4()), "data" : { "message" : "Private message" , "importance" : "high" } } await transport.send_message( DailyTransportMessageUrgentFrame( message = urgent_data, participant_id = "participant-123" ) ) The message will be delivered to: All participants if no specific participant_id is set Only the specified participant if participant_id is set Through the Daily app messaging system Can be received by registering an on_app_message handler Use TransportMessageUrgentFrame for high-priority messages that need immediate delivery. ​ Media Control ​ send_image async method Sends an image frame to the room. Copy Ask AI async def send_image ( frame : OutputImageRawFrame | SpriteFrame) -> None Parameters: frame : Image frame to send, either raw image data or sprite animation ​ send_audio async method Sends an audio frame to the room. Copy Ask AI async def send_audio ( frame : OutputAudioRawFrame) -> None Parameters: frame : Audio frame to send ​ Video Management ​ capture_participant_video async method Starts capturing video from a specific participant. Copy Ask AI async def capture_participant_video ( participant_id : str , framerate : int = 30 , video_source : str = "camera" , color_format : str = "RGB" ) -> None Parameters: participant_id : ID of the participant to capture framerate : Target frame rate (default: 30) video_source : Video source type (default: “camera”) color_format : Color format of the video (default: “RGB”) To request an image from the camera stream, set the framerate to 0 and push a UserImageRequestFrame . ​ Transcription Control ​ capture_participant_transcription async method Starts capturing and transcribing audio from a specific participant. Copy Ask AI async def capture_participant_transcription ( participant_id : str ) -> None Parameters: participant_id : ID of the participant to transcribe ​ update_transcription async method Updates the transcription configuration for specific participants or instances. Copy Ask AI async def update_transcription ( participants : List[ str ] | None = None , instance_id : str | None = None ) -> None Parameters: participants : Optional list of participant IDs to transcribe. If None, transcribes all participants. instance_id : Optional specific transcription instance ID to update Example: Copy Ask AI # Update transcription for specific participants await transport.update_transcription( participants = [ "participant-123" , "participant-456" ]) # Update specific transcription instance await transport.update_transcription( instance_id = "transcription-789" ) ​ Recording Control ​ start_recording async method Starts recording the room session. Copy Ask AI async def start_recording ( streaming_settings : Dict = None , stream_id : str = None , force_new : bool = None ) -> None Parameters: streaming_settings : Recording configuration settings stream_id : Optional stream identifier force_new : Force start a new recording See start_recording for more details. ​ stop_recording async method Stops an active recording. Copy Ask AI async def stop_recording ( stream_id : str = None ) -> None Parameters: stream_id : Optional stream identifier to stop specific recording See stop_recording for more details. ​ Dial-out Control ​ start_dialout async method Initiates a dial-out call. Copy Ask AI async def start_dialout ( settings : Dict = None ) -> None Parameters: settings : Dial-out configuration settings See start_dialout for more details. ​ stop_dialout async method Stops an active dial-out call. Copy Ask AI async def stop_dialout ( participant_id : str ) -> None Parameters: participant_id : ID of the participant to disconnect See stop_dialout for more details. ​ send_dtmf async method Sends DTMF tones in an existing dial-out session. Copy Ask AI async def send_dtmf ( settings : Dict = None ) -> None Parameters: settings : DTMF settings See send_dtmf for more details. ​ Subscription Management ​ update_subscriptions async method Updates subscriptions and subscription profiles. This function allows you to update subscription profiles and at the same time assign specific subscription profiles to a participant and even change specific settings for some participants. Copy Ask AI async def update_subscriptions ( participant_settings : Dict = None , profile_settings : Dict = None ) -> None Parameters: participant_settings : Per-participant subscription settings profile_settings : Global subscription profile settings See update_subscriptions for more details. ​ Participant Admin When using the DailyTransport, if your bot is given a meeting token with owner or participant admin permissions, it can remote-control other participants’ capabilities and permissions. With this control, the bot can modify settings for how others send and receive media, how they are seen in a session, and whether they have administrative controls. ​ update_remote_participants async method Updates permissions and settings for specific participants in the room. Copy Ask AI async def update_remote_participants ( remote_participants : Mapping[ str , Any] = None ) -> None Parameters: remote_participants : Dictionary mapping participant IDs to their updated settings This method allows fine-grained control over participant capabilities and permissions in the Daily room. Examples: Copy Ask AI # Mute a participant's microphone await transport.update_remote_participants({ "participant-123" : { "inputsEnabled" : { "microphone" : False } } }) # Revoke send permissions (mutes the participant and prevents them from unmuting) await transport.update_remote_participants({ "participant-123" : { "permissions" : { "canSend" : [] # Empty list revokes all send permissions } } }) # Configure selective audio routing await transport.update_remote_participants({ "participant-123" : { # the customer "permissions" : { "canReceive" : { "base" : False , # Don't receive from anyone by default "byUserId" : { "hold-music" : True # Only receive from hold music player } } } } }) # Set up user-to-user permissions mapping (e.g., when transferring calls) # Assumes customer was previously unable to hear anyone (they were on hold) and agent was speaking to bot await transport.update_remote_participants({ "participant-123" : { # the customer "permissions" : { "base" : False , "canReceive" : { "byUserId" : { "agent" : True }} } }, "participant-456" : { # the agent "permissions" : { "base" : False , "canReceive" : { "byUserId" : { "customer" : True }} } } }) # Promote a participant to be a participant admin await transport.update_remote_participants({ "participant-123" : { "permissions" : { "canAdmin" : [ "participants" ] } } }) Permissions Settings: For comprehensive details on permissions settings, see the Daily API documentation: daily-python’s update_remote_participants() documentation daily-js’s updateParticipant() -> updatePermission and participants().permissions ​ Chat messages ​ send_prebuilt_chat_message async method When testing with Daily Prebuilt, you can send messages to the Prebuilt chat using this method. Parameters: message (str): The chat message to send user_name (str): The name of the user sending the message See send_prebuilt_chat_message for more details. ​ Properties ​ participant_id string Returns the transport’s participant ID in the room. ​ room_url string Returns the transport’s room URL for the room currently in use. ​ Notes Supports real-time audio/video communication Handles WebRTC connection management Provides voice activity detection Includes transcription capabilities Manages participant interactions Supports recording and streaming Thread-safe processing All callbacks are asynchronous Multiple handlers can be registered for the same event Exceptions in callbacks are logged but don’t stop the transport Supported Services SmallWebRTCTransport On this page Overview Installation Configuration Constructor Parameters DailyParams Configuration Audio Output Configuration Audio Input Configuration Video Output Configuration Voice Activity Detection (VAD) Feature Settings Basic Usage Event Callbacks Connection Events Participant Events Communication Events Dial Events Dial-in Dial-out Transcription Events Recording Events Error Events Notes on Callbacks Frame Types Input Frames Output Frames Methods Room Management Media Control Video Management Transcription Control Recording Control Dial-out Control Subscription Management Participant Admin Chat messages Properties Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/transport_daily_70ddec2d.txt b/transport_daily_70ddec2d.txt
new file mode 100644
index 0000000000000000000000000000000000000000..10476a5f4889a2c6669d95d39d3b5ed6cc536bd6
--- /dev/null
+++ b/transport_daily_70ddec2d.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/transport/daily#param-video-out-bitrate
+Title: Daily WebRTC - Pipecat
+==================================================
+
+Daily WebRTC - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Transport Daily WebRTC Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Daily WebRTC SmallWebRTCTransport FastAPI WebSocket WebSocket Server Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview DailyTransport provides real-time audio and video communication capabilities using Daily’s WebRTC platform. It supports bidirectional audio/video streams, transcription, voice activity detection (VAD), and participant management. ​ Installation To use DailyTransport , install the required dependencies: Copy Ask AI pip install "pipecat-ai[daily]" You’ll also need to set up your Daily API key as an environment variable: DAILY_API_KEY . You can obtain a Daily API key by signing up at Daily . ​ Configuration ​ Constructor Parameters ​ room_url str required Daily room URL ​ token str | None Daily room token ​ bot_name str required Name of the bot in the room ​ params DailyParams default: "DailyParams()" Transport configuration parameters ​ DailyParams Configuration ​ api_url str default: "https://api.daily.co/v1" Daily API endpoint URL ​ api_key str default: "" Daily API key for authentication ​ Audio Output Configuration ​ audio_out_enabled bool default: "False" Enable audio output capabilities ​ audio_out_is_live bool default: "False" Enable live audio streaming mode ​ audio_out_sample_rate int default: "None" Audio output sample rate in Hz ​ audio_out_channels int default: "1" Number of audio output channels ​ audio_out_bitrate int default: "96000" Audio output bitrate in bits per second ​ Audio Input Configuration ​ audio_in_enabled bool default: "False" Enable audio input capabilities ​ audio_in_passthrough bool default: "False" When enabled, incoming audio frames are pushed downstream ​ audio_in_sample_rate int default: "None" Audio input sample rate in Hz ​ audio_in_channels int default: "1" Number of audio input channels ​ audio_in_filter Optional[BaseAudioFilter] default: "None" Audio filter for input processing. Supported filters are: KrispFilter() and NoisereduceFilter() . See the KrispFilter() reference docs and NoisereduceFilter() reference docs ` for more information. ​ Video Output Configuration ​ video_out_enabled bool default: "False" Enable video output capabilities ​ video_out_is_live bool default: "False" Enable live video streaming mode ​ video_out_width int default: "1024" Video output width in pixels ​ video_out_height int default: "768" Video output height in pixels ​ video_out_bitrate int default: "800000" Video output bitrate in bits per second ​ video_out_framerate int default: "30" Video output frame rate ​ video_out_color_format str default: "RGB" Video color format (RGB, BGR, etc.) ​ Voice Activity Detection (VAD) ​ vad_analyzer VADAnalyzer | None default: "None" Voice Activity Detection analyzer. You can set this to either SileroVADAnalyzer() or WebRTCVADAnalyzer() . SileroVADAnalyzer is the recommended option. Learn more about the SileroVADAnalyzer . ​ Feature Settings ​ dialin_settings Optional[DailyDialinSettings] default: "None" Configuration for dial-in functionality ​ transcription_enabled bool default: "False" Enable real-time transcription ​ transcription_settings DailyTranscriptionSettings Configuration for transcription features Copy Ask AI class DailyTranscriptionSettings ( BaseModel ): language: str = "en" # Default language model: str = "nova-2-general" # Transcription model profanity_filter: bool = True # Filter profanity redact: bool = False # Redact sensitive information endpointing: bool = True # Enable speech endpointing punctuate: bool = True # Add punctuation includeRawResponse: bool = True # Include raw API response extra: Mapping[ str , Any] = { "interim_results" : True # Provide any Deepgram-specific settings } ​ Basic Usage Copy Ask AI from pipecat.transports.services.daily import DailyTransport, DailyParams from pipecat.audio.vad.silero import SileroVADAnalyzer # Configure transport transport = DailyTransport( room_url = "https://your-domain.daily.co/room-name" , token = "your-room-token" , bot_name = "AI Assistant" , params = DailyParams( audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer( params = VADParams( stop_secs = 0.5 )), ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), # Handle incoming audio/video stt, # Speech-to-text llm, # Language model tts, # Text-to-speech transport.output() # Handle outgoing audio/video ]) # Use transport event handlers to manage the call lifecycle # Event handler for when the user joins the room: @transport.event_handler ( "on_first_participant_joined" ) async def on_first_participant_joined ( transport , participant ): # Start transcription for the user await transport.capture_participant_transcription(participant[ "id" ]) # Kick off the conversation. await task.queue_frames([context_aggregator.user().get_context_frame()]) # Event handler for when the user leaves the room: @transport.event_handler ( "on_participant_left" ) async def on_participant_left ( transport , participant , reason ): # Cancel the pipeline, which stops processing and removes the bot from the room await task.cancel() ​ Event Callbacks DailyTransport provides a comprehensive callback system for handling various events. Register callbacks using the @transport.event_handler() decorator. ​ Connection Events ​ on_joined async callback Called when the bot successfully joins the room. Parameters: transport : The DailyTransport instance data : Dictionary containing room join information Copy Ask AI @transport.event_handler ( "on_joined" ) async def on_joined ( transport , data : Dict[ str , Any]): logger.info( f "Joined room with data: { data } " ) ​ on_left async callback Called when the bot leaves the room. Parameters: transport : The DailyTransport instance Copy Ask AI @transport.event_handler ( "on_left" ) async def on_left ( transport ): logger.info( "Left room" ) ​ Participant Events ​ on_first_participant_joined async callback Called when the first participant joins the room. Useful for initializing conversations. Parameters: transport : The DailyTransport instance participant : Dictionary containing participant information Copy Ask AI @transport.event_handler ( "on_first_participant_joined" ) async def on_first_participant_joined ( transport , participant : Dict[ str , Any]): await transport.capture_participant_transcription(participant[ "id" ]) await task.queue_frames([LLMMessagesFrame(initial_messages)]) ​ on_participant_joined async callback Called when any participant joins the room. Parameters: transport : The DailyTransport instance participant : Dictionary containing participant information Copy Ask AI @transport.event_handler ( "on_participant_joined" ) async def on_participant_joined ( transport , participant : Dict[ str , Any]): logger.info( f "Participant joined: { participant[ 'id' ] } " ) ​ on_participant_left async callback Called when a participant leaves the room. Parameters: transport : The DailyTransport instance participant : Dictionary containing participant information reason : String describing why the participant left, “leftCall” | “hidden” Copy Ask AI @transport.event_handler ( "on_participant_left" ) async def on_participant_left ( transport , participant : Dict[ str , Any], reason : str ): logger.info( f "Participant { participant[ 'id' ] } left: { reason } " ) # Cancel the pipeline task to stop processing and remove the bot from the Daily room await task.cancel() ​ on_participant_updated async callback Event emitted when a participant is updated. This can mean either the participant’s metadata was updated, or the tracks belonging to the participant changed. Parameters: transport : The DailyTransport instance participant : Dictionary containing updated participant information Copy Ask AI @transport.event_handler ( "on_participant_updated" ) async def on_participant_updated ( transport , participant : Dict[ str , Any]): logger.info( f "Participant updated: { participant } " ) ​ Communication Events ​ on_app_message async callback Event emitted when a custom app message is received from another participant or via the REST API. Parameters: transport : The DailyTransport instance message : The message content (any type) sender : String identifier of the message sender Copy Ask AI @transport.event_handler ( "on_app_message" ) async def on_app_message ( transport , message : Any, sender : str ): logger.info( f "Message from { sender } : { message } " ) ​ on_call_state_updated async callback Event emitted when the call state changes, normally as a consequence of joining or leaving the call. Parameters: transport : The DailyTransport instance state : String representing the new call state. Learn more about call states . Copy Ask AI @transport.event_handler ( "on_call_state_updated" ) async def on_call_state_updated ( transport , state : str ): logger.info( f "Call state updated: { state } " ) ​ Dial Events ​ Dial-in ​ on_dialin_ready async callback Event emitted when dial-in is ready. This happens after the room has connected to the SIP endpoint and the system is ready to receive dial-in calls. Parameters: transport : The DailyTransport instance sip_endpoint : String containing the SIP endpoint information Copy Ask AI @transport.event_handler ( "on_dialin_ready" ) async def on_dialin_ready ( transport , sip_endpoint : str ): logger.info( f "Dial-in ready at: { sip_endpoint } " ) ​ on_dialin_connected async callback Event emitted when the session with the dial-in remote end is established (i.e. SIP endpoint or PSTN are connectd to the Daily room). Note: connected does not mean media (audio or video) has started flowing between the room and PSTN, it means the room received the connection request and both endpoints are negotiating the media flow. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. See DialinConnectedEvent . Copy Ask AI @transport.event_handler ( "on_dialin_connected" ) async def on_dialin_connected ( transport , data : Any): logger.info( f "Dial-in connected: { data } " ) ​ on_dialin_stopped async callback Event emitted when the dial-in remote end disconnects the call. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. See DialinStoppedEvent . Copy Ask AI @transport.event_handler ( "on_dialin_stopped" ) async def on_dialin_stopped ( transport , data : Any): logger.info( f "Dial-in stopped: { data } " ) ​ on_dialin_error async callback Event emitted in the case of dial-in errors which are fatal and the service cannot proceed. For example, an error in SDP negotiation is fatal to the media/SIP pipeline and will result in dialin-error being triggered. Parameters: transport : The DailyTransport instance data : Dictionary containing error information. See DialinEvent . Copy Ask AI @transport.event_handler ( "on_dialin_error" ) async def on_dialin_error ( transport , data : Any): logger.error( f "Dial-in error: { data } " ) ​ on_dialin_warning async callback Event emitted there is a dial-in non-fatal error, such as the selected codec not being used and a fallback codec being utilized. Parameters: transport : The DailyTransport instance data : Dictionary containing warning information. See DialinEvent . Copy Ask AI @transport.event_handler ( "on_dialin_warning" ) async def on_dialin_warning ( transport , data : Any): logger.warning( f "Dial-in warning: { data } " ) ​ Dial-out ​ on_dialout_answered async callback Event emitted when the session with the dial-out remote end is answered. Parameters: transport : The DailyTransport instance data : Dictionary containing call information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_answered" ) async def on_dialout_answered ( transport , data : Any): logger.info( f "Dial-out answered: { data } " ) ​ on_dialout_connected async callback Event emitted when the session with the dial-out remote end is established. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_connected" ) async def on_dialout_connected ( transport , data : Any): logger.info( f "Dial-out connected: { data } " ) ​ on_dialout_stopped async callback Event emitted when the dial-out session is stopped. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_stopped" ) async def on_dialout_stopped ( transport , data : Any): logger.info( f "Dial-out stopped: { data } " ) ​ on_dialout_error async callback Event emitted in the case of dial-out errors which are fatal and the service cannot proceed. For example, an error in SDP negotiation is fatal to the media/SIP pipeline and will result in dialout-error being triggered. Parameters: transport : The DailyTransport instance data : Dictionary containing error information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_error" ) async def on_dialout_error ( transport , data : Any): logger.error( f "Dial-out error: { data } " ) ​ on_dialout_warning async callback Event emitted there is a dial-out non-fatal error, such as the selected codec not being used and a fallback codec being utilized. Parameters: transport : The DailyTransport instance data : Dictionary containing warning information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_warning" ) async def on_dialout_warning ( transport , data : Any): logger.warning( f "Dial-out warning: { data } " ) ​ Transcription Events ​ on_transcription_message async callback Called when a transcription message is received. This includes both interim and final transcriptions. Parameters: transport : The DailyTransport instance message : Dictionary containing transcription data including. Learn more . Copy Ask AI @transport.event_handler ( "on_transcription_message" ) async def on_transcription_message ( transport , message : Dict[ str , Any]): participant_id = message.get( "participantId" ) text = message.get( "text" ) is_final = message[ "rawResponse" ][ "is_final" ] logger.info( f "Transcription from { participant_id } : { text } (final: { is_final } )" ) ​ Recording Events ​ on_recording_started async callback Called when a room recording starts successfully. Parameters: transport : The DailyTransport instance status : Dictionary containing recording status information. Learn more . Copy Ask AI @transport.event_handler ( "on_recording_started" ) async def on_recording_started ( transport , status : Dict[ str , Any]): logger.info( f "Recording started with status: { status } " ) ​ on_recording_stopped async callback Called when a room recording stops. Parameters: transport : The DailyTransport instance stream_id : String identifier of the stopped recording stream Copy Ask AI @transport.event_handler ( "on_recording_stopped" ) async def on_recording_stopped ( transport , stream_id : str ): logger.info( f "Recording stopped for stream: { stream_id } " ) ​ on_recording_error async callback Called when an error occurs during recording. Parameters: transport : The DailyTransport instance stream_id : String identifier of the recording stream message : Error message describing what went wrong Copy Ask AI @transport.event_handler ( "on_recording_error" ) async def on_recording_error ( transport , stream_id : str , message : str ): logger.error( f "Recording error for stream { stream_id } : { message } " ) ​ Error Events ​ on_error async callback Called when a transport error occurs. Parameters: transport : The DailyTransport instance error : String containing error details Copy Ask AI @transport.event_handler ( "on_error" ) async def on_error ( transport , error : str ): logger.error( f "Transport error: { error } " ) ​ Notes on Callbacks All callbacks are asynchronous Callbacks are executed in the order they were registered Multiple handlers can be registered for the same event Exceptions in callbacks are logged but don’t stop the transport Callbacks should be lightweight to avoid blocking the event loop Heavy processing should be offloaded to separate tasks ​ Frame Types ​ Input Frames ​ InputAudioRawFrame Frame Raw audio data from participants ​ UserImageRawFrame Frame Video frames from participants ​ Output Frames ​ OutputAudioRawFrame Frame Audio data to be sent ​ OutputImageRawFrame Frame Video frames to be sent ​ Methods ​ Room Management ​ participants method Returns a dictionary of all participants currently in the room. Copy Ask AI def participants () -> Dict[ str , Any] See participants for more details. ​ participant_counts method Returns participant count statistics for the room. Copy Ask AI def participant_counts () -> Dict[ str , int ] See participant_counts for more details. ​ send_message async method Sends a message to participants in the Daily room. Messages can be directed to all participants or targeted to a specific participant. Copy Ask AI async def send_message ( frame : TransportMessageFrame | TransportMessageUrgentFrame) Parameters: frame : A transport message frame containing the message to send. Can be either: TransportMessageFrame : Standard message TransportMessageUrgentFrame : High-priority message Message Structure: Messages should conform to the expected format for the receiving system. For RTVI messages, use: Copy Ask AI { "label" : "rtvi-ai" , # Required for RTVI protocol "type" : "message-type" , # Defines how the client will process it "id" : "unique-id" , # Unique identifier for the message "data" : { # Custom payload # Your custom data fields } } Example - Standard Message: Copy Ask AI import uuid # Send a standard message to all participants message_data = { "label" : "rtvi-ai" , "type" : "custom-message" , "id" : str (uuid.uuid4()), "data" : { "message" : "Hello everyone!" , "timestamp" : datetime.datetime.now().isoformat() } } await transport.send_message(TransportMessageFrame( message = message_data)) Example - Urgent Message to Specific Participant: Copy Ask AI # Send urgent message to specific participant urgent_data = { "label" : "rtvi-ai" , "type" : "server-message" , "id" : str (uuid.uuid4()), "data" : { "message" : "Private message" , "importance" : "high" } } await transport.send_message( DailyTransportMessageUrgentFrame( message = urgent_data, participant_id = "participant-123" ) ) The message will be delivered to: All participants if no specific participant_id is set Only the specified participant if participant_id is set Through the Daily app messaging system Can be received by registering an on_app_message handler Use TransportMessageUrgentFrame for high-priority messages that need immediate delivery. ​ Media Control ​ send_image async method Sends an image frame to the room. Copy Ask AI async def send_image ( frame : OutputImageRawFrame | SpriteFrame) -> None Parameters: frame : Image frame to send, either raw image data or sprite animation ​ send_audio async method Sends an audio frame to the room. Copy Ask AI async def send_audio ( frame : OutputAudioRawFrame) -> None Parameters: frame : Audio frame to send ​ Video Management ​ capture_participant_video async method Starts capturing video from a specific participant. Copy Ask AI async def capture_participant_video ( participant_id : str , framerate : int = 30 , video_source : str = "camera" , color_format : str = "RGB" ) -> None Parameters: participant_id : ID of the participant to capture framerate : Target frame rate (default: 30) video_source : Video source type (default: “camera”) color_format : Color format of the video (default: “RGB”) To request an image from the camera stream, set the framerate to 0 and push a UserImageRequestFrame . ​ Transcription Control ​ capture_participant_transcription async method Starts capturing and transcribing audio from a specific participant. Copy Ask AI async def capture_participant_transcription ( participant_id : str ) -> None Parameters: participant_id : ID of the participant to transcribe ​ update_transcription async method Updates the transcription configuration for specific participants or instances. Copy Ask AI async def update_transcription ( participants : List[ str ] | None = None , instance_id : str | None = None ) -> None Parameters: participants : Optional list of participant IDs to transcribe. If None, transcribes all participants. instance_id : Optional specific transcription instance ID to update Example: Copy Ask AI # Update transcription for specific participants await transport.update_transcription( participants = [ "participant-123" , "participant-456" ]) # Update specific transcription instance await transport.update_transcription( instance_id = "transcription-789" ) ​ Recording Control ​ start_recording async method Starts recording the room session. Copy Ask AI async def start_recording ( streaming_settings : Dict = None , stream_id : str = None , force_new : bool = None ) -> None Parameters: streaming_settings : Recording configuration settings stream_id : Optional stream identifier force_new : Force start a new recording See start_recording for more details. ​ stop_recording async method Stops an active recording. Copy Ask AI async def stop_recording ( stream_id : str = None ) -> None Parameters: stream_id : Optional stream identifier to stop specific recording See stop_recording for more details. ​ Dial-out Control ​ start_dialout async method Initiates a dial-out call. Copy Ask AI async def start_dialout ( settings : Dict = None ) -> None Parameters: settings : Dial-out configuration settings See start_dialout for more details. ​ stop_dialout async method Stops an active dial-out call. Copy Ask AI async def stop_dialout ( participant_id : str ) -> None Parameters: participant_id : ID of the participant to disconnect See stop_dialout for more details. ​ send_dtmf async method Sends DTMF tones in an existing dial-out session. Copy Ask AI async def send_dtmf ( settings : Dict = None ) -> None Parameters: settings : DTMF settings See send_dtmf for more details. ​ Subscription Management ​ update_subscriptions async method Updates subscriptions and subscription profiles. This function allows you to update subscription profiles and at the same time assign specific subscription profiles to a participant and even change specific settings for some participants. Copy Ask AI async def update_subscriptions ( participant_settings : Dict = None , profile_settings : Dict = None ) -> None Parameters: participant_settings : Per-participant subscription settings profile_settings : Global subscription profile settings See update_subscriptions for more details. ​ Participant Admin When using the DailyTransport, if your bot is given a meeting token with owner or participant admin permissions, it can remote-control other participants’ capabilities and permissions. With this control, the bot can modify settings for how others send and receive media, how they are seen in a session, and whether they have administrative controls. ​ update_remote_participants async method Updates permissions and settings for specific participants in the room. Copy Ask AI async def update_remote_participants ( remote_participants : Mapping[ str , Any] = None ) -> None Parameters: remote_participants : Dictionary mapping participant IDs to their updated settings This method allows fine-grained control over participant capabilities and permissions in the Daily room. Examples: Copy Ask AI # Mute a participant's microphone await transport.update_remote_participants({ "participant-123" : { "inputsEnabled" : { "microphone" : False } } }) # Revoke send permissions (mutes the participant and prevents them from unmuting) await transport.update_remote_participants({ "participant-123" : { "permissions" : { "canSend" : [] # Empty list revokes all send permissions } } }) # Configure selective audio routing await transport.update_remote_participants({ "participant-123" : { # the customer "permissions" : { "canReceive" : { "base" : False , # Don't receive from anyone by default "byUserId" : { "hold-music" : True # Only receive from hold music player } } } } }) # Set up user-to-user permissions mapping (e.g., when transferring calls) # Assumes customer was previously unable to hear anyone (they were on hold) and agent was speaking to bot await transport.update_remote_participants({ "participant-123" : { # the customer "permissions" : { "base" : False , "canReceive" : { "byUserId" : { "agent" : True }} } }, "participant-456" : { # the agent "permissions" : { "base" : False , "canReceive" : { "byUserId" : { "customer" : True }} } } }) # Promote a participant to be a participant admin await transport.update_remote_participants({ "participant-123" : { "permissions" : { "canAdmin" : [ "participants" ] } } }) Permissions Settings: For comprehensive details on permissions settings, see the Daily API documentation: daily-python’s update_remote_participants() documentation daily-js’s updateParticipant() -> updatePermission and participants().permissions ​ Chat messages ​ send_prebuilt_chat_message async method When testing with Daily Prebuilt, you can send messages to the Prebuilt chat using this method. Parameters: message (str): The chat message to send user_name (str): The name of the user sending the message See send_prebuilt_chat_message for more details. ​ Properties ​ participant_id string Returns the transport’s participant ID in the room. ​ room_url string Returns the transport’s room URL for the room currently in use. ​ Notes Supports real-time audio/video communication Handles WebRTC connection management Provides voice activity detection Includes transcription capabilities Manages participant interactions Supports recording and streaming Thread-safe processing All callbacks are asynchronous Multiple handlers can be registered for the same event Exceptions in callbacks are logged but don’t stop the transport Supported Services SmallWebRTCTransport On this page Overview Installation Configuration Constructor Parameters DailyParams Configuration Audio Output Configuration Audio Input Configuration Video Output Configuration Voice Activity Detection (VAD) Feature Settings Basic Usage Event Callbacks Connection Events Participant Events Communication Events Dial Events Dial-in Dial-out Transcription Events Recording Events Error Events Notes on Callbacks Frame Types Input Frames Output Frames Methods Room Management Media Control Video Management Transcription Control Recording Control Dial-out Control Subscription Management Participant Admin Chat messages Properties Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/transport_daily_775cb40c.txt b/transport_daily_775cb40c.txt
new file mode 100644
index 0000000000000000000000000000000000000000..2f08f04c3a56114b875d1efb8ecfa68a143623b8
--- /dev/null
+++ b/transport_daily_775cb40c.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/transport/daily#communication-events
+Title: Daily WebRTC - Pipecat
+==================================================
+
+Daily WebRTC - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Transport Daily WebRTC Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Daily WebRTC SmallWebRTCTransport FastAPI WebSocket WebSocket Server Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview DailyTransport provides real-time audio and video communication capabilities using Daily’s WebRTC platform. It supports bidirectional audio/video streams, transcription, voice activity detection (VAD), and participant management. ​ Installation To use DailyTransport , install the required dependencies: Copy Ask AI pip install "pipecat-ai[daily]" You’ll also need to set up your Daily API key as an environment variable: DAILY_API_KEY . You can obtain a Daily API key by signing up at Daily . ​ Configuration ​ Constructor Parameters ​ room_url str required Daily room URL ​ token str | None Daily room token ​ bot_name str required Name of the bot in the room ​ params DailyParams default: "DailyParams()" Transport configuration parameters ​ DailyParams Configuration ​ api_url str default: "https://api.daily.co/v1" Daily API endpoint URL ​ api_key str default: "" Daily API key for authentication ​ Audio Output Configuration ​ audio_out_enabled bool default: "False" Enable audio output capabilities ​ audio_out_is_live bool default: "False" Enable live audio streaming mode ​ audio_out_sample_rate int default: "None" Audio output sample rate in Hz ​ audio_out_channels int default: "1" Number of audio output channels ​ audio_out_bitrate int default: "96000" Audio output bitrate in bits per second ​ Audio Input Configuration ​ audio_in_enabled bool default: "False" Enable audio input capabilities ​ audio_in_passthrough bool default: "False" When enabled, incoming audio frames are pushed downstream ​ audio_in_sample_rate int default: "None" Audio input sample rate in Hz ​ audio_in_channels int default: "1" Number of audio input channels ​ audio_in_filter Optional[BaseAudioFilter] default: "None" Audio filter for input processing. Supported filters are: KrispFilter() and NoisereduceFilter() . See the KrispFilter() reference docs and NoisereduceFilter() reference docs ` for more information. ​ Video Output Configuration ​ video_out_enabled bool default: "False" Enable video output capabilities ​ video_out_is_live bool default: "False" Enable live video streaming mode ​ video_out_width int default: "1024" Video output width in pixels ​ video_out_height int default: "768" Video output height in pixels ​ video_out_bitrate int default: "800000" Video output bitrate in bits per second ​ video_out_framerate int default: "30" Video output frame rate ​ video_out_color_format str default: "RGB" Video color format (RGB, BGR, etc.) ​ Voice Activity Detection (VAD) ​ vad_analyzer VADAnalyzer | None default: "None" Voice Activity Detection analyzer. You can set this to either SileroVADAnalyzer() or WebRTCVADAnalyzer() . SileroVADAnalyzer is the recommended option. Learn more about the SileroVADAnalyzer . ​ Feature Settings ​ dialin_settings Optional[DailyDialinSettings] default: "None" Configuration for dial-in functionality ​ transcription_enabled bool default: "False" Enable real-time transcription ​ transcription_settings DailyTranscriptionSettings Configuration for transcription features Copy Ask AI class DailyTranscriptionSettings ( BaseModel ): language: str = "en" # Default language model: str = "nova-2-general" # Transcription model profanity_filter: bool = True # Filter profanity redact: bool = False # Redact sensitive information endpointing: bool = True # Enable speech endpointing punctuate: bool = True # Add punctuation includeRawResponse: bool = True # Include raw API response extra: Mapping[ str , Any] = { "interim_results" : True # Provide any Deepgram-specific settings } ​ Basic Usage Copy Ask AI from pipecat.transports.services.daily import DailyTransport, DailyParams from pipecat.audio.vad.silero import SileroVADAnalyzer # Configure transport transport = DailyTransport( room_url = "https://your-domain.daily.co/room-name" , token = "your-room-token" , bot_name = "AI Assistant" , params = DailyParams( audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer( params = VADParams( stop_secs = 0.5 )), ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), # Handle incoming audio/video stt, # Speech-to-text llm, # Language model tts, # Text-to-speech transport.output() # Handle outgoing audio/video ]) # Use transport event handlers to manage the call lifecycle # Event handler for when the user joins the room: @transport.event_handler ( "on_first_participant_joined" ) async def on_first_participant_joined ( transport , participant ): # Start transcription for the user await transport.capture_participant_transcription(participant[ "id" ]) # Kick off the conversation. await task.queue_frames([context_aggregator.user().get_context_frame()]) # Event handler for when the user leaves the room: @transport.event_handler ( "on_participant_left" ) async def on_participant_left ( transport , participant , reason ): # Cancel the pipeline, which stops processing and removes the bot from the room await task.cancel() ​ Event Callbacks DailyTransport provides a comprehensive callback system for handling various events. Register callbacks using the @transport.event_handler() decorator. ​ Connection Events ​ on_joined async callback Called when the bot successfully joins the room. Parameters: transport : The DailyTransport instance data : Dictionary containing room join information Copy Ask AI @transport.event_handler ( "on_joined" ) async def on_joined ( transport , data : Dict[ str , Any]): logger.info( f "Joined room with data: { data } " ) ​ on_left async callback Called when the bot leaves the room. Parameters: transport : The DailyTransport instance Copy Ask AI @transport.event_handler ( "on_left" ) async def on_left ( transport ): logger.info( "Left room" ) ​ Participant Events ​ on_first_participant_joined async callback Called when the first participant joins the room. Useful for initializing conversations. Parameters: transport : The DailyTransport instance participant : Dictionary containing participant information Copy Ask AI @transport.event_handler ( "on_first_participant_joined" ) async def on_first_participant_joined ( transport , participant : Dict[ str , Any]): await transport.capture_participant_transcription(participant[ "id" ]) await task.queue_frames([LLMMessagesFrame(initial_messages)]) ​ on_participant_joined async callback Called when any participant joins the room. Parameters: transport : The DailyTransport instance participant : Dictionary containing participant information Copy Ask AI @transport.event_handler ( "on_participant_joined" ) async def on_participant_joined ( transport , participant : Dict[ str , Any]): logger.info( f "Participant joined: { participant[ 'id' ] } " ) ​ on_participant_left async callback Called when a participant leaves the room. Parameters: transport : The DailyTransport instance participant : Dictionary containing participant information reason : String describing why the participant left, “leftCall” | “hidden” Copy Ask AI @transport.event_handler ( "on_participant_left" ) async def on_participant_left ( transport , participant : Dict[ str , Any], reason : str ): logger.info( f "Participant { participant[ 'id' ] } left: { reason } " ) # Cancel the pipeline task to stop processing and remove the bot from the Daily room await task.cancel() ​ on_participant_updated async callback Event emitted when a participant is updated. This can mean either the participant’s metadata was updated, or the tracks belonging to the participant changed. Parameters: transport : The DailyTransport instance participant : Dictionary containing updated participant information Copy Ask AI @transport.event_handler ( "on_participant_updated" ) async def on_participant_updated ( transport , participant : Dict[ str , Any]): logger.info( f "Participant updated: { participant } " ) ​ Communication Events ​ on_app_message async callback Event emitted when a custom app message is received from another participant or via the REST API. Parameters: transport : The DailyTransport instance message : The message content (any type) sender : String identifier of the message sender Copy Ask AI @transport.event_handler ( "on_app_message" ) async def on_app_message ( transport , message : Any, sender : str ): logger.info( f "Message from { sender } : { message } " ) ​ on_call_state_updated async callback Event emitted when the call state changes, normally as a consequence of joining or leaving the call. Parameters: transport : The DailyTransport instance state : String representing the new call state. Learn more about call states . Copy Ask AI @transport.event_handler ( "on_call_state_updated" ) async def on_call_state_updated ( transport , state : str ): logger.info( f "Call state updated: { state } " ) ​ Dial Events ​ Dial-in ​ on_dialin_ready async callback Event emitted when dial-in is ready. This happens after the room has connected to the SIP endpoint and the system is ready to receive dial-in calls. Parameters: transport : The DailyTransport instance sip_endpoint : String containing the SIP endpoint information Copy Ask AI @transport.event_handler ( "on_dialin_ready" ) async def on_dialin_ready ( transport , sip_endpoint : str ): logger.info( f "Dial-in ready at: { sip_endpoint } " ) ​ on_dialin_connected async callback Event emitted when the session with the dial-in remote end is established (i.e. SIP endpoint or PSTN are connectd to the Daily room). Note: connected does not mean media (audio or video) has started flowing between the room and PSTN, it means the room received the connection request and both endpoints are negotiating the media flow. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. See DialinConnectedEvent . Copy Ask AI @transport.event_handler ( "on_dialin_connected" ) async def on_dialin_connected ( transport , data : Any): logger.info( f "Dial-in connected: { data } " ) ​ on_dialin_stopped async callback Event emitted when the dial-in remote end disconnects the call. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. See DialinStoppedEvent . Copy Ask AI @transport.event_handler ( "on_dialin_stopped" ) async def on_dialin_stopped ( transport , data : Any): logger.info( f "Dial-in stopped: { data } " ) ​ on_dialin_error async callback Event emitted in the case of dial-in errors which are fatal and the service cannot proceed. For example, an error in SDP negotiation is fatal to the media/SIP pipeline and will result in dialin-error being triggered. Parameters: transport : The DailyTransport instance data : Dictionary containing error information. See DialinEvent . Copy Ask AI @transport.event_handler ( "on_dialin_error" ) async def on_dialin_error ( transport , data : Any): logger.error( f "Dial-in error: { data } " ) ​ on_dialin_warning async callback Event emitted there is a dial-in non-fatal error, such as the selected codec not being used and a fallback codec being utilized. Parameters: transport : The DailyTransport instance data : Dictionary containing warning information. See DialinEvent . Copy Ask AI @transport.event_handler ( "on_dialin_warning" ) async def on_dialin_warning ( transport , data : Any): logger.warning( f "Dial-in warning: { data } " ) ​ Dial-out ​ on_dialout_answered async callback Event emitted when the session with the dial-out remote end is answered. Parameters: transport : The DailyTransport instance data : Dictionary containing call information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_answered" ) async def on_dialout_answered ( transport , data : Any): logger.info( f "Dial-out answered: { data } " ) ​ on_dialout_connected async callback Event emitted when the session with the dial-out remote end is established. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_connected" ) async def on_dialout_connected ( transport , data : Any): logger.info( f "Dial-out connected: { data } " ) ​ on_dialout_stopped async callback Event emitted when the dial-out session is stopped. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_stopped" ) async def on_dialout_stopped ( transport , data : Any): logger.info( f "Dial-out stopped: { data } " ) ​ on_dialout_error async callback Event emitted in the case of dial-out errors which are fatal and the service cannot proceed. For example, an error in SDP negotiation is fatal to the media/SIP pipeline and will result in dialout-error being triggered. Parameters: transport : The DailyTransport instance data : Dictionary containing error information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_error" ) async def on_dialout_error ( transport , data : Any): logger.error( f "Dial-out error: { data } " ) ​ on_dialout_warning async callback Event emitted there is a dial-out non-fatal error, such as the selected codec not being used and a fallback codec being utilized. Parameters: transport : The DailyTransport instance data : Dictionary containing warning information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_warning" ) async def on_dialout_warning ( transport , data : Any): logger.warning( f "Dial-out warning: { data } " ) ​ Transcription Events ​ on_transcription_message async callback Called when a transcription message is received. This includes both interim and final transcriptions. Parameters: transport : The DailyTransport instance message : Dictionary containing transcription data including. Learn more . Copy Ask AI @transport.event_handler ( "on_transcription_message" ) async def on_transcription_message ( transport , message : Dict[ str , Any]): participant_id = message.get( "participantId" ) text = message.get( "text" ) is_final = message[ "rawResponse" ][ "is_final" ] logger.info( f "Transcription from { participant_id } : { text } (final: { is_final } )" ) ​ Recording Events ​ on_recording_started async callback Called when a room recording starts successfully. Parameters: transport : The DailyTransport instance status : Dictionary containing recording status information. Learn more . Copy Ask AI @transport.event_handler ( "on_recording_started" ) async def on_recording_started ( transport , status : Dict[ str , Any]): logger.info( f "Recording started with status: { status } " ) ​ on_recording_stopped async callback Called when a room recording stops. Parameters: transport : The DailyTransport instance stream_id : String identifier of the stopped recording stream Copy Ask AI @transport.event_handler ( "on_recording_stopped" ) async def on_recording_stopped ( transport , stream_id : str ): logger.info( f "Recording stopped for stream: { stream_id } " ) ​ on_recording_error async callback Called when an error occurs during recording. Parameters: transport : The DailyTransport instance stream_id : String identifier of the recording stream message : Error message describing what went wrong Copy Ask AI @transport.event_handler ( "on_recording_error" ) async def on_recording_error ( transport , stream_id : str , message : str ): logger.error( f "Recording error for stream { stream_id } : { message } " ) ​ Error Events ​ on_error async callback Called when a transport error occurs. Parameters: transport : The DailyTransport instance error : String containing error details Copy Ask AI @transport.event_handler ( "on_error" ) async def on_error ( transport , error : str ): logger.error( f "Transport error: { error } " ) ​ Notes on Callbacks All callbacks are asynchronous Callbacks are executed in the order they were registered Multiple handlers can be registered for the same event Exceptions in callbacks are logged but don’t stop the transport Callbacks should be lightweight to avoid blocking the event loop Heavy processing should be offloaded to separate tasks ​ Frame Types ​ Input Frames ​ InputAudioRawFrame Frame Raw audio data from participants ​ UserImageRawFrame Frame Video frames from participants ​ Output Frames ​ OutputAudioRawFrame Frame Audio data to be sent ​ OutputImageRawFrame Frame Video frames to be sent ​ Methods ​ Room Management ​ participants method Returns a dictionary of all participants currently in the room. Copy Ask AI def participants () -> Dict[ str , Any] See participants for more details. ​ participant_counts method Returns participant count statistics for the room. Copy Ask AI def participant_counts () -> Dict[ str , int ] See participant_counts for more details. ​ send_message async method Sends a message to participants in the Daily room. Messages can be directed to all participants or targeted to a specific participant. Copy Ask AI async def send_message ( frame : TransportMessageFrame | TransportMessageUrgentFrame) Parameters: frame : A transport message frame containing the message to send. Can be either: TransportMessageFrame : Standard message TransportMessageUrgentFrame : High-priority message Message Structure: Messages should conform to the expected format for the receiving system. For RTVI messages, use: Copy Ask AI { "label" : "rtvi-ai" , # Required for RTVI protocol "type" : "message-type" , # Defines how the client will process it "id" : "unique-id" , # Unique identifier for the message "data" : { # Custom payload # Your custom data fields } } Example - Standard Message: Copy Ask AI import uuid # Send a standard message to all participants message_data = { "label" : "rtvi-ai" , "type" : "custom-message" , "id" : str (uuid.uuid4()), "data" : { "message" : "Hello everyone!" , "timestamp" : datetime.datetime.now().isoformat() } } await transport.send_message(TransportMessageFrame( message = message_data)) Example - Urgent Message to Specific Participant: Copy Ask AI # Send urgent message to specific participant urgent_data = { "label" : "rtvi-ai" , "type" : "server-message" , "id" : str (uuid.uuid4()), "data" : { "message" : "Private message" , "importance" : "high" } } await transport.send_message( DailyTransportMessageUrgentFrame( message = urgent_data, participant_id = "participant-123" ) ) The message will be delivered to: All participants if no specific participant_id is set Only the specified participant if participant_id is set Through the Daily app messaging system Can be received by registering an on_app_message handler Use TransportMessageUrgentFrame for high-priority messages that need immediate delivery. ​ Media Control ​ send_image async method Sends an image frame to the room. Copy Ask AI async def send_image ( frame : OutputImageRawFrame | SpriteFrame) -> None Parameters: frame : Image frame to send, either raw image data or sprite animation ​ send_audio async method Sends an audio frame to the room. Copy Ask AI async def send_audio ( frame : OutputAudioRawFrame) -> None Parameters: frame : Audio frame to send ​ Video Management ​ capture_participant_video async method Starts capturing video from a specific participant. Copy Ask AI async def capture_participant_video ( participant_id : str , framerate : int = 30 , video_source : str = "camera" , color_format : str = "RGB" ) -> None Parameters: participant_id : ID of the participant to capture framerate : Target frame rate (default: 30) video_source : Video source type (default: “camera”) color_format : Color format of the video (default: “RGB”) To request an image from the camera stream, set the framerate to 0 and push a UserImageRequestFrame . ​ Transcription Control ​ capture_participant_transcription async method Starts capturing and transcribing audio from a specific participant. Copy Ask AI async def capture_participant_transcription ( participant_id : str ) -> None Parameters: participant_id : ID of the participant to transcribe ​ update_transcription async method Updates the transcription configuration for specific participants or instances. Copy Ask AI async def update_transcription ( participants : List[ str ] | None = None , instance_id : str | None = None ) -> None Parameters: participants : Optional list of participant IDs to transcribe. If None, transcribes all participants. instance_id : Optional specific transcription instance ID to update Example: Copy Ask AI # Update transcription for specific participants await transport.update_transcription( participants = [ "participant-123" , "participant-456" ]) # Update specific transcription instance await transport.update_transcription( instance_id = "transcription-789" ) ​ Recording Control ​ start_recording async method Starts recording the room session. Copy Ask AI async def start_recording ( streaming_settings : Dict = None , stream_id : str = None , force_new : bool = None ) -> None Parameters: streaming_settings : Recording configuration settings stream_id : Optional stream identifier force_new : Force start a new recording See start_recording for more details. ​ stop_recording async method Stops an active recording. Copy Ask AI async def stop_recording ( stream_id : str = None ) -> None Parameters: stream_id : Optional stream identifier to stop specific recording See stop_recording for more details. ​ Dial-out Control ​ start_dialout async method Initiates a dial-out call. Copy Ask AI async def start_dialout ( settings : Dict = None ) -> None Parameters: settings : Dial-out configuration settings See start_dialout for more details. ​ stop_dialout async method Stops an active dial-out call. Copy Ask AI async def stop_dialout ( participant_id : str ) -> None Parameters: participant_id : ID of the participant to disconnect See stop_dialout for more details. ​ send_dtmf async method Sends DTMF tones in an existing dial-out session. Copy Ask AI async def send_dtmf ( settings : Dict = None ) -> None Parameters: settings : DTMF settings See send_dtmf for more details. ​ Subscription Management ​ update_subscriptions async method Updates subscriptions and subscription profiles. This function allows you to update subscription profiles and at the same time assign specific subscription profiles to a participant and even change specific settings for some participants. Copy Ask AI async def update_subscriptions ( participant_settings : Dict = None , profile_settings : Dict = None ) -> None Parameters: participant_settings : Per-participant subscription settings profile_settings : Global subscription profile settings See update_subscriptions for more details. ​ Participant Admin When using the DailyTransport, if your bot is given a meeting token with owner or participant admin permissions, it can remote-control other participants’ capabilities and permissions. With this control, the bot can modify settings for how others send and receive media, how they are seen in a session, and whether they have administrative controls. ​ update_remote_participants async method Updates permissions and settings for specific participants in the room. Copy Ask AI async def update_remote_participants ( remote_participants : Mapping[ str , Any] = None ) -> None Parameters: remote_participants : Dictionary mapping participant IDs to their updated settings This method allows fine-grained control over participant capabilities and permissions in the Daily room. Examples: Copy Ask AI # Mute a participant's microphone await transport.update_remote_participants({ "participant-123" : { "inputsEnabled" : { "microphone" : False } } }) # Revoke send permissions (mutes the participant and prevents them from unmuting) await transport.update_remote_participants({ "participant-123" : { "permissions" : { "canSend" : [] # Empty list revokes all send permissions } } }) # Configure selective audio routing await transport.update_remote_participants({ "participant-123" : { # the customer "permissions" : { "canReceive" : { "base" : False , # Don't receive from anyone by default "byUserId" : { "hold-music" : True # Only receive from hold music player } } } } }) # Set up user-to-user permissions mapping (e.g., when transferring calls) # Assumes customer was previously unable to hear anyone (they were on hold) and agent was speaking to bot await transport.update_remote_participants({ "participant-123" : { # the customer "permissions" : { "base" : False , "canReceive" : { "byUserId" : { "agent" : True }} } }, "participant-456" : { # the agent "permissions" : { "base" : False , "canReceive" : { "byUserId" : { "customer" : True }} } } }) # Promote a participant to be a participant admin await transport.update_remote_participants({ "participant-123" : { "permissions" : { "canAdmin" : [ "participants" ] } } }) Permissions Settings: For comprehensive details on permissions settings, see the Daily API documentation: daily-python’s update_remote_participants() documentation daily-js’s updateParticipant() -> updatePermission and participants().permissions ​ Chat messages ​ send_prebuilt_chat_message async method When testing with Daily Prebuilt, you can send messages to the Prebuilt chat using this method. Parameters: message (str): The chat message to send user_name (str): The name of the user sending the message See send_prebuilt_chat_message for more details. ​ Properties ​ participant_id string Returns the transport’s participant ID in the room. ​ room_url string Returns the transport’s room URL for the room currently in use. ​ Notes Supports real-time audio/video communication Handles WebRTC connection management Provides voice activity detection Includes transcription capabilities Manages participant interactions Supports recording and streaming Thread-safe processing All callbacks are asynchronous Multiple handlers can be registered for the same event Exceptions in callbacks are logged but don’t stop the transport Supported Services SmallWebRTCTransport On this page Overview Installation Configuration Constructor Parameters DailyParams Configuration Audio Output Configuration Audio Input Configuration Video Output Configuration Voice Activity Detection (VAD) Feature Settings Basic Usage Event Callbacks Connection Events Participant Events Communication Events Dial Events Dial-in Dial-out Transcription Events Recording Events Error Events Notes on Callbacks Frame Types Input Frames Output Frames Methods Room Management Media Control Video Management Transcription Control Recording Control Dial-out Control Subscription Management Participant Admin Chat messages Properties Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/transport_daily_79cca1b7.txt b/transport_daily_79cca1b7.txt
new file mode 100644
index 0000000000000000000000000000000000000000..90b37562d18296a201b0aefca096da77cb660c71
--- /dev/null
+++ b/transport_daily_79cca1b7.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/transport/daily#param-audio-in-enabled
+Title: Daily WebRTC - Pipecat
+==================================================
+
+Daily WebRTC - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Transport Daily WebRTC Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Daily WebRTC SmallWebRTCTransport FastAPI WebSocket WebSocket Server Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview DailyTransport provides real-time audio and video communication capabilities using Daily’s WebRTC platform. It supports bidirectional audio/video streams, transcription, voice activity detection (VAD), and participant management. ​ Installation To use DailyTransport , install the required dependencies: Copy Ask AI pip install "pipecat-ai[daily]" You’ll also need to set up your Daily API key as an environment variable: DAILY_API_KEY . You can obtain a Daily API key by signing up at Daily . ​ Configuration ​ Constructor Parameters ​ room_url str required Daily room URL ​ token str | None Daily room token ​ bot_name str required Name of the bot in the room ​ params DailyParams default: "DailyParams()" Transport configuration parameters ​ DailyParams Configuration ​ api_url str default: "https://api.daily.co/v1" Daily API endpoint URL ​ api_key str default: "" Daily API key for authentication ​ Audio Output Configuration ​ audio_out_enabled bool default: "False" Enable audio output capabilities ​ audio_out_is_live bool default: "False" Enable live audio streaming mode ​ audio_out_sample_rate int default: "None" Audio output sample rate in Hz ​ audio_out_channels int default: "1" Number of audio output channels ​ audio_out_bitrate int default: "96000" Audio output bitrate in bits per second ​ Audio Input Configuration ​ audio_in_enabled bool default: "False" Enable audio input capabilities ​ audio_in_passthrough bool default: "False" When enabled, incoming audio frames are pushed downstream ​ audio_in_sample_rate int default: "None" Audio input sample rate in Hz ​ audio_in_channels int default: "1" Number of audio input channels ​ audio_in_filter Optional[BaseAudioFilter] default: "None" Audio filter for input processing. Supported filters are: KrispFilter() and NoisereduceFilter() . See the KrispFilter() reference docs and NoisereduceFilter() reference docs ` for more information. ​ Video Output Configuration ​ video_out_enabled bool default: "False" Enable video output capabilities ​ video_out_is_live bool default: "False" Enable live video streaming mode ​ video_out_width int default: "1024" Video output width in pixels ​ video_out_height int default: "768" Video output height in pixels ​ video_out_bitrate int default: "800000" Video output bitrate in bits per second ​ video_out_framerate int default: "30" Video output frame rate ​ video_out_color_format str default: "RGB" Video color format (RGB, BGR, etc.) ​ Voice Activity Detection (VAD) ​ vad_analyzer VADAnalyzer | None default: "None" Voice Activity Detection analyzer. You can set this to either SileroVADAnalyzer() or WebRTCVADAnalyzer() . SileroVADAnalyzer is the recommended option. Learn more about the SileroVADAnalyzer . ​ Feature Settings ​ dialin_settings Optional[DailyDialinSettings] default: "None" Configuration for dial-in functionality ​ transcription_enabled bool default: "False" Enable real-time transcription ​ transcription_settings DailyTranscriptionSettings Configuration for transcription features Copy Ask AI class DailyTranscriptionSettings ( BaseModel ): language: str = "en" # Default language model: str = "nova-2-general" # Transcription model profanity_filter: bool = True # Filter profanity redact: bool = False # Redact sensitive information endpointing: bool = True # Enable speech endpointing punctuate: bool = True # Add punctuation includeRawResponse: bool = True # Include raw API response extra: Mapping[ str , Any] = { "interim_results" : True # Provide any Deepgram-specific settings } ​ Basic Usage Copy Ask AI from pipecat.transports.services.daily import DailyTransport, DailyParams from pipecat.audio.vad.silero import SileroVADAnalyzer # Configure transport transport = DailyTransport( room_url = "https://your-domain.daily.co/room-name" , token = "your-room-token" , bot_name = "AI Assistant" , params = DailyParams( audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer( params = VADParams( stop_secs = 0.5 )), ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), # Handle incoming audio/video stt, # Speech-to-text llm, # Language model tts, # Text-to-speech transport.output() # Handle outgoing audio/video ]) # Use transport event handlers to manage the call lifecycle # Event handler for when the user joins the room: @transport.event_handler ( "on_first_participant_joined" ) async def on_first_participant_joined ( transport , participant ): # Start transcription for the user await transport.capture_participant_transcription(participant[ "id" ]) # Kick off the conversation. await task.queue_frames([context_aggregator.user().get_context_frame()]) # Event handler for when the user leaves the room: @transport.event_handler ( "on_participant_left" ) async def on_participant_left ( transport , participant , reason ): # Cancel the pipeline, which stops processing and removes the bot from the room await task.cancel() ​ Event Callbacks DailyTransport provides a comprehensive callback system for handling various events. Register callbacks using the @transport.event_handler() decorator. ​ Connection Events ​ on_joined async callback Called when the bot successfully joins the room. Parameters: transport : The DailyTransport instance data : Dictionary containing room join information Copy Ask AI @transport.event_handler ( "on_joined" ) async def on_joined ( transport , data : Dict[ str , Any]): logger.info( f "Joined room with data: { data } " ) ​ on_left async callback Called when the bot leaves the room. Parameters: transport : The DailyTransport instance Copy Ask AI @transport.event_handler ( "on_left" ) async def on_left ( transport ): logger.info( "Left room" ) ​ Participant Events ​ on_first_participant_joined async callback Called when the first participant joins the room. Useful for initializing conversations. Parameters: transport : The DailyTransport instance participant : Dictionary containing participant information Copy Ask AI @transport.event_handler ( "on_first_participant_joined" ) async def on_first_participant_joined ( transport , participant : Dict[ str , Any]): await transport.capture_participant_transcription(participant[ "id" ]) await task.queue_frames([LLMMessagesFrame(initial_messages)]) ​ on_participant_joined async callback Called when any participant joins the room. Parameters: transport : The DailyTransport instance participant : Dictionary containing participant information Copy Ask AI @transport.event_handler ( "on_participant_joined" ) async def on_participant_joined ( transport , participant : Dict[ str , Any]): logger.info( f "Participant joined: { participant[ 'id' ] } " ) ​ on_participant_left async callback Called when a participant leaves the room. Parameters: transport : The DailyTransport instance participant : Dictionary containing participant information reason : String describing why the participant left, “leftCall” | “hidden” Copy Ask AI @transport.event_handler ( "on_participant_left" ) async def on_participant_left ( transport , participant : Dict[ str , Any], reason : str ): logger.info( f "Participant { participant[ 'id' ] } left: { reason } " ) # Cancel the pipeline task to stop processing and remove the bot from the Daily room await task.cancel() ​ on_participant_updated async callback Event emitted when a participant is updated. This can mean either the participant’s metadata was updated, or the tracks belonging to the participant changed. Parameters: transport : The DailyTransport instance participant : Dictionary containing updated participant information Copy Ask AI @transport.event_handler ( "on_participant_updated" ) async def on_participant_updated ( transport , participant : Dict[ str , Any]): logger.info( f "Participant updated: { participant } " ) ​ Communication Events ​ on_app_message async callback Event emitted when a custom app message is received from another participant or via the REST API. Parameters: transport : The DailyTransport instance message : The message content (any type) sender : String identifier of the message sender Copy Ask AI @transport.event_handler ( "on_app_message" ) async def on_app_message ( transport , message : Any, sender : str ): logger.info( f "Message from { sender } : { message } " ) ​ on_call_state_updated async callback Event emitted when the call state changes, normally as a consequence of joining or leaving the call. Parameters: transport : The DailyTransport instance state : String representing the new call state. Learn more about call states . Copy Ask AI @transport.event_handler ( "on_call_state_updated" ) async def on_call_state_updated ( transport , state : str ): logger.info( f "Call state updated: { state } " ) ​ Dial Events ​ Dial-in ​ on_dialin_ready async callback Event emitted when dial-in is ready. This happens after the room has connected to the SIP endpoint and the system is ready to receive dial-in calls. Parameters: transport : The DailyTransport instance sip_endpoint : String containing the SIP endpoint information Copy Ask AI @transport.event_handler ( "on_dialin_ready" ) async def on_dialin_ready ( transport , sip_endpoint : str ): logger.info( f "Dial-in ready at: { sip_endpoint } " ) ​ on_dialin_connected async callback Event emitted when the session with the dial-in remote end is established (i.e. SIP endpoint or PSTN are connectd to the Daily room). Note: connected does not mean media (audio or video) has started flowing between the room and PSTN, it means the room received the connection request and both endpoints are negotiating the media flow. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. See DialinConnectedEvent . Copy Ask AI @transport.event_handler ( "on_dialin_connected" ) async def on_dialin_connected ( transport , data : Any): logger.info( f "Dial-in connected: { data } " ) ​ on_dialin_stopped async callback Event emitted when the dial-in remote end disconnects the call. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. See DialinStoppedEvent . Copy Ask AI @transport.event_handler ( "on_dialin_stopped" ) async def on_dialin_stopped ( transport , data : Any): logger.info( f "Dial-in stopped: { data } " ) ​ on_dialin_error async callback Event emitted in the case of dial-in errors which are fatal and the service cannot proceed. For example, an error in SDP negotiation is fatal to the media/SIP pipeline and will result in dialin-error being triggered. Parameters: transport : The DailyTransport instance data : Dictionary containing error information. See DialinEvent . Copy Ask AI @transport.event_handler ( "on_dialin_error" ) async def on_dialin_error ( transport , data : Any): logger.error( f "Dial-in error: { data } " ) ​ on_dialin_warning async callback Event emitted there is a dial-in non-fatal error, such as the selected codec not being used and a fallback codec being utilized. Parameters: transport : The DailyTransport instance data : Dictionary containing warning information. See DialinEvent . Copy Ask AI @transport.event_handler ( "on_dialin_warning" ) async def on_dialin_warning ( transport , data : Any): logger.warning( f "Dial-in warning: { data } " ) ​ Dial-out ​ on_dialout_answered async callback Event emitted when the session with the dial-out remote end is answered. Parameters: transport : The DailyTransport instance data : Dictionary containing call information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_answered" ) async def on_dialout_answered ( transport , data : Any): logger.info( f "Dial-out answered: { data } " ) ​ on_dialout_connected async callback Event emitted when the session with the dial-out remote end is established. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_connected" ) async def on_dialout_connected ( transport , data : Any): logger.info( f "Dial-out connected: { data } " ) ​ on_dialout_stopped async callback Event emitted when the dial-out session is stopped. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_stopped" ) async def on_dialout_stopped ( transport , data : Any): logger.info( f "Dial-out stopped: { data } " ) ​ on_dialout_error async callback Event emitted in the case of dial-out errors which are fatal and the service cannot proceed. For example, an error in SDP negotiation is fatal to the media/SIP pipeline and will result in dialout-error being triggered. Parameters: transport : The DailyTransport instance data : Dictionary containing error information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_error" ) async def on_dialout_error ( transport , data : Any): logger.error( f "Dial-out error: { data } " ) ​ on_dialout_warning async callback Event emitted there is a dial-out non-fatal error, such as the selected codec not being used and a fallback codec being utilized. Parameters: transport : The DailyTransport instance data : Dictionary containing warning information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_warning" ) async def on_dialout_warning ( transport , data : Any): logger.warning( f "Dial-out warning: { data } " ) ​ Transcription Events ​ on_transcription_message async callback Called when a transcription message is received. This includes both interim and final transcriptions. Parameters: transport : The DailyTransport instance message : Dictionary containing transcription data including. Learn more . Copy Ask AI @transport.event_handler ( "on_transcription_message" ) async def on_transcription_message ( transport , message : Dict[ str , Any]): participant_id = message.get( "participantId" ) text = message.get( "text" ) is_final = message[ "rawResponse" ][ "is_final" ] logger.info( f "Transcription from { participant_id } : { text } (final: { is_final } )" ) ​ Recording Events ​ on_recording_started async callback Called when a room recording starts successfully. Parameters: transport : The DailyTransport instance status : Dictionary containing recording status information. Learn more . Copy Ask AI @transport.event_handler ( "on_recording_started" ) async def on_recording_started ( transport , status : Dict[ str , Any]): logger.info( f "Recording started with status: { status } " ) ​ on_recording_stopped async callback Called when a room recording stops. Parameters: transport : The DailyTransport instance stream_id : String identifier of the stopped recording stream Copy Ask AI @transport.event_handler ( "on_recording_stopped" ) async def on_recording_stopped ( transport , stream_id : str ): logger.info( f "Recording stopped for stream: { stream_id } " ) ​ on_recording_error async callback Called when an error occurs during recording. Parameters: transport : The DailyTransport instance stream_id : String identifier of the recording stream message : Error message describing what went wrong Copy Ask AI @transport.event_handler ( "on_recording_error" ) async def on_recording_error ( transport , stream_id : str , message : str ): logger.error( f "Recording error for stream { stream_id } : { message } " ) ​ Error Events ​ on_error async callback Called when a transport error occurs. Parameters: transport : The DailyTransport instance error : String containing error details Copy Ask AI @transport.event_handler ( "on_error" ) async def on_error ( transport , error : str ): logger.error( f "Transport error: { error } " ) ​ Notes on Callbacks All callbacks are asynchronous Callbacks are executed in the order they were registered Multiple handlers can be registered for the same event Exceptions in callbacks are logged but don’t stop the transport Callbacks should be lightweight to avoid blocking the event loop Heavy processing should be offloaded to separate tasks ​ Frame Types ​ Input Frames ​ InputAudioRawFrame Frame Raw audio data from participants ​ UserImageRawFrame Frame Video frames from participants ​ Output Frames ​ OutputAudioRawFrame Frame Audio data to be sent ​ OutputImageRawFrame Frame Video frames to be sent ​ Methods ​ Room Management ​ participants method Returns a dictionary of all participants currently in the room. Copy Ask AI def participants () -> Dict[ str , Any] See participants for more details. ​ participant_counts method Returns participant count statistics for the room. Copy Ask AI def participant_counts () -> Dict[ str , int ] See participant_counts for more details. ​ send_message async method Sends a message to participants in the Daily room. Messages can be directed to all participants or targeted to a specific participant. Copy Ask AI async def send_message ( frame : TransportMessageFrame | TransportMessageUrgentFrame) Parameters: frame : A transport message frame containing the message to send. Can be either: TransportMessageFrame : Standard message TransportMessageUrgentFrame : High-priority message Message Structure: Messages should conform to the expected format for the receiving system. For RTVI messages, use: Copy Ask AI { "label" : "rtvi-ai" , # Required for RTVI protocol "type" : "message-type" , # Defines how the client will process it "id" : "unique-id" , # Unique identifier for the message "data" : { # Custom payload # Your custom data fields } } Example - Standard Message: Copy Ask AI import uuid # Send a standard message to all participants message_data = { "label" : "rtvi-ai" , "type" : "custom-message" , "id" : str (uuid.uuid4()), "data" : { "message" : "Hello everyone!" , "timestamp" : datetime.datetime.now().isoformat() } } await transport.send_message(TransportMessageFrame( message = message_data)) Example - Urgent Message to Specific Participant: Copy Ask AI # Send urgent message to specific participant urgent_data = { "label" : "rtvi-ai" , "type" : "server-message" , "id" : str (uuid.uuid4()), "data" : { "message" : "Private message" , "importance" : "high" } } await transport.send_message( DailyTransportMessageUrgentFrame( message = urgent_data, participant_id = "participant-123" ) ) The message will be delivered to: All participants if no specific participant_id is set Only the specified participant if participant_id is set Through the Daily app messaging system Can be received by registering an on_app_message handler Use TransportMessageUrgentFrame for high-priority messages that need immediate delivery. ​ Media Control ​ send_image async method Sends an image frame to the room. Copy Ask AI async def send_image ( frame : OutputImageRawFrame | SpriteFrame) -> None Parameters: frame : Image frame to send, either raw image data or sprite animation ​ send_audio async method Sends an audio frame to the room. Copy Ask AI async def send_audio ( frame : OutputAudioRawFrame) -> None Parameters: frame : Audio frame to send ​ Video Management ​ capture_participant_video async method Starts capturing video from a specific participant. Copy Ask AI async def capture_participant_video ( participant_id : str , framerate : int = 30 , video_source : str = "camera" , color_format : str = "RGB" ) -> None Parameters: participant_id : ID of the participant to capture framerate : Target frame rate (default: 30) video_source : Video source type (default: “camera”) color_format : Color format of the video (default: “RGB”) To request an image from the camera stream, set the framerate to 0 and push a UserImageRequestFrame . ​ Transcription Control ​ capture_participant_transcription async method Starts capturing and transcribing audio from a specific participant. Copy Ask AI async def capture_participant_transcription ( participant_id : str ) -> None Parameters: participant_id : ID of the participant to transcribe ​ update_transcription async method Updates the transcription configuration for specific participants or instances. Copy Ask AI async def update_transcription ( participants : List[ str ] | None = None , instance_id : str | None = None ) -> None Parameters: participants : Optional list of participant IDs to transcribe. If None, transcribes all participants. instance_id : Optional specific transcription instance ID to update Example: Copy Ask AI # Update transcription for specific participants await transport.update_transcription( participants = [ "participant-123" , "participant-456" ]) # Update specific transcription instance await transport.update_transcription( instance_id = "transcription-789" ) ​ Recording Control ​ start_recording async method Starts recording the room session. Copy Ask AI async def start_recording ( streaming_settings : Dict = None , stream_id : str = None , force_new : bool = None ) -> None Parameters: streaming_settings : Recording configuration settings stream_id : Optional stream identifier force_new : Force start a new recording See start_recording for more details. ​ stop_recording async method Stops an active recording. Copy Ask AI async def stop_recording ( stream_id : str = None ) -> None Parameters: stream_id : Optional stream identifier to stop specific recording See stop_recording for more details. ​ Dial-out Control ​ start_dialout async method Initiates a dial-out call. Copy Ask AI async def start_dialout ( settings : Dict = None ) -> None Parameters: settings : Dial-out configuration settings See start_dialout for more details. ​ stop_dialout async method Stops an active dial-out call. Copy Ask AI async def stop_dialout ( participant_id : str ) -> None Parameters: participant_id : ID of the participant to disconnect See stop_dialout for more details. ​ send_dtmf async method Sends DTMF tones in an existing dial-out session. Copy Ask AI async def send_dtmf ( settings : Dict = None ) -> None Parameters: settings : DTMF settings See send_dtmf for more details. ​ Subscription Management ​ update_subscriptions async method Updates subscriptions and subscription profiles. This function allows you to update subscription profiles and at the same time assign specific subscription profiles to a participant and even change specific settings for some participants. Copy Ask AI async def update_subscriptions ( participant_settings : Dict = None , profile_settings : Dict = None ) -> None Parameters: participant_settings : Per-participant subscription settings profile_settings : Global subscription profile settings See update_subscriptions for more details. ​ Participant Admin When using the DailyTransport, if your bot is given a meeting token with owner or participant admin permissions, it can remote-control other participants’ capabilities and permissions. With this control, the bot can modify settings for how others send and receive media, how they are seen in a session, and whether they have administrative controls. ​ update_remote_participants async method Updates permissions and settings for specific participants in the room. Copy Ask AI async def update_remote_participants ( remote_participants : Mapping[ str , Any] = None ) -> None Parameters: remote_participants : Dictionary mapping participant IDs to their updated settings This method allows fine-grained control over participant capabilities and permissions in the Daily room. Examples: Copy Ask AI # Mute a participant's microphone await transport.update_remote_participants({ "participant-123" : { "inputsEnabled" : { "microphone" : False } } }) # Revoke send permissions (mutes the participant and prevents them from unmuting) await transport.update_remote_participants({ "participant-123" : { "permissions" : { "canSend" : [] # Empty list revokes all send permissions } } }) # Configure selective audio routing await transport.update_remote_participants({ "participant-123" : { # the customer "permissions" : { "canReceive" : { "base" : False , # Don't receive from anyone by default "byUserId" : { "hold-music" : True # Only receive from hold music player } } } } }) # Set up user-to-user permissions mapping (e.g., when transferring calls) # Assumes customer was previously unable to hear anyone (they were on hold) and agent was speaking to bot await transport.update_remote_participants({ "participant-123" : { # the customer "permissions" : { "base" : False , "canReceive" : { "byUserId" : { "agent" : True }} } }, "participant-456" : { # the agent "permissions" : { "base" : False , "canReceive" : { "byUserId" : { "customer" : True }} } } }) # Promote a participant to be a participant admin await transport.update_remote_participants({ "participant-123" : { "permissions" : { "canAdmin" : [ "participants" ] } } }) Permissions Settings: For comprehensive details on permissions settings, see the Daily API documentation: daily-python’s update_remote_participants() documentation daily-js’s updateParticipant() -> updatePermission and participants().permissions ​ Chat messages ​ send_prebuilt_chat_message async method When testing with Daily Prebuilt, you can send messages to the Prebuilt chat using this method. Parameters: message (str): The chat message to send user_name (str): The name of the user sending the message See send_prebuilt_chat_message for more details. ​ Properties ​ participant_id string Returns the transport’s participant ID in the room. ​ room_url string Returns the transport’s room URL for the room currently in use. ​ Notes Supports real-time audio/video communication Handles WebRTC connection management Provides voice activity detection Includes transcription capabilities Manages participant interactions Supports recording and streaming Thread-safe processing All callbacks are asynchronous Multiple handlers can be registered for the same event Exceptions in callbacks are logged but don’t stop the transport Supported Services SmallWebRTCTransport On this page Overview Installation Configuration Constructor Parameters DailyParams Configuration Audio Output Configuration Audio Input Configuration Video Output Configuration Voice Activity Detection (VAD) Feature Settings Basic Usage Event Callbacks Connection Events Participant Events Communication Events Dial Events Dial-in Dial-out Transcription Events Recording Events Error Events Notes on Callbacks Frame Types Input Frames Output Frames Methods Room Management Media Control Video Management Transcription Control Recording Control Dial-out Control Subscription Management Participant Admin Chat messages Properties Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/transport_daily_a6cf182f.txt b/transport_daily_a6cf182f.txt
new file mode 100644
index 0000000000000000000000000000000000000000..9ec1d01eaa8f5ebb22fa226d27c027e0e348a185
--- /dev/null
+++ b/transport_daily_a6cf182f.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/transport/daily#room-management
+Title: Daily WebRTC - Pipecat
+==================================================
+
+Daily WebRTC - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Transport Daily WebRTC Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Daily WebRTC SmallWebRTCTransport FastAPI WebSocket WebSocket Server Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview DailyTransport provides real-time audio and video communication capabilities using Daily’s WebRTC platform. It supports bidirectional audio/video streams, transcription, voice activity detection (VAD), and participant management. ​ Installation To use DailyTransport , install the required dependencies: Copy Ask AI pip install "pipecat-ai[daily]" You’ll also need to set up your Daily API key as an environment variable: DAILY_API_KEY . You can obtain a Daily API key by signing up at Daily . ​ Configuration ​ Constructor Parameters ​ room_url str required Daily room URL ​ token str | None Daily room token ​ bot_name str required Name of the bot in the room ​ params DailyParams default: "DailyParams()" Transport configuration parameters ​ DailyParams Configuration ​ api_url str default: "https://api.daily.co/v1" Daily API endpoint URL ​ api_key str default: "" Daily API key for authentication ​ Audio Output Configuration ​ audio_out_enabled bool default: "False" Enable audio output capabilities ​ audio_out_is_live bool default: "False" Enable live audio streaming mode ​ audio_out_sample_rate int default: "None" Audio output sample rate in Hz ​ audio_out_channels int default: "1" Number of audio output channels ​ audio_out_bitrate int default: "96000" Audio output bitrate in bits per second ​ Audio Input Configuration ​ audio_in_enabled bool default: "False" Enable audio input capabilities ​ audio_in_passthrough bool default: "False" When enabled, incoming audio frames are pushed downstream ​ audio_in_sample_rate int default: "None" Audio input sample rate in Hz ​ audio_in_channels int default: "1" Number of audio input channels ​ audio_in_filter Optional[BaseAudioFilter] default: "None" Audio filter for input processing. Supported filters are: KrispFilter() and NoisereduceFilter() . See the KrispFilter() reference docs and NoisereduceFilter() reference docs ` for more information. ​ Video Output Configuration ​ video_out_enabled bool default: "False" Enable video output capabilities ​ video_out_is_live bool default: "False" Enable live video streaming mode ​ video_out_width int default: "1024" Video output width in pixels ​ video_out_height int default: "768" Video output height in pixels ​ video_out_bitrate int default: "800000" Video output bitrate in bits per second ​ video_out_framerate int default: "30" Video output frame rate ​ video_out_color_format str default: "RGB" Video color format (RGB, BGR, etc.) ​ Voice Activity Detection (VAD) ​ vad_analyzer VADAnalyzer | None default: "None" Voice Activity Detection analyzer. You can set this to either SileroVADAnalyzer() or WebRTCVADAnalyzer() . SileroVADAnalyzer is the recommended option. Learn more about the SileroVADAnalyzer . ​ Feature Settings ​ dialin_settings Optional[DailyDialinSettings] default: "None" Configuration for dial-in functionality ​ transcription_enabled bool default: "False" Enable real-time transcription ​ transcription_settings DailyTranscriptionSettings Configuration for transcription features Copy Ask AI class DailyTranscriptionSettings ( BaseModel ): language: str = "en" # Default language model: str = "nova-2-general" # Transcription model profanity_filter: bool = True # Filter profanity redact: bool = False # Redact sensitive information endpointing: bool = True # Enable speech endpointing punctuate: bool = True # Add punctuation includeRawResponse: bool = True # Include raw API response extra: Mapping[ str , Any] = { "interim_results" : True # Provide any Deepgram-specific settings } ​ Basic Usage Copy Ask AI from pipecat.transports.services.daily import DailyTransport, DailyParams from pipecat.audio.vad.silero import SileroVADAnalyzer # Configure transport transport = DailyTransport( room_url = "https://your-domain.daily.co/room-name" , token = "your-room-token" , bot_name = "AI Assistant" , params = DailyParams( audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer( params = VADParams( stop_secs = 0.5 )), ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), # Handle incoming audio/video stt, # Speech-to-text llm, # Language model tts, # Text-to-speech transport.output() # Handle outgoing audio/video ]) # Use transport event handlers to manage the call lifecycle # Event handler for when the user joins the room: @transport.event_handler ( "on_first_participant_joined" ) async def on_first_participant_joined ( transport , participant ): # Start transcription for the user await transport.capture_participant_transcription(participant[ "id" ]) # Kick off the conversation. await task.queue_frames([context_aggregator.user().get_context_frame()]) # Event handler for when the user leaves the room: @transport.event_handler ( "on_participant_left" ) async def on_participant_left ( transport , participant , reason ): # Cancel the pipeline, which stops processing and removes the bot from the room await task.cancel() ​ Event Callbacks DailyTransport provides a comprehensive callback system for handling various events. Register callbacks using the @transport.event_handler() decorator. ​ Connection Events ​ on_joined async callback Called when the bot successfully joins the room. Parameters: transport : The DailyTransport instance data : Dictionary containing room join information Copy Ask AI @transport.event_handler ( "on_joined" ) async def on_joined ( transport , data : Dict[ str , Any]): logger.info( f "Joined room with data: { data } " ) ​ on_left async callback Called when the bot leaves the room. Parameters: transport : The DailyTransport instance Copy Ask AI @transport.event_handler ( "on_left" ) async def on_left ( transport ): logger.info( "Left room" ) ​ Participant Events ​ on_first_participant_joined async callback Called when the first participant joins the room. Useful for initializing conversations. Parameters: transport : The DailyTransport instance participant : Dictionary containing participant information Copy Ask AI @transport.event_handler ( "on_first_participant_joined" ) async def on_first_participant_joined ( transport , participant : Dict[ str , Any]): await transport.capture_participant_transcription(participant[ "id" ]) await task.queue_frames([LLMMessagesFrame(initial_messages)]) ​ on_participant_joined async callback Called when any participant joins the room. Parameters: transport : The DailyTransport instance participant : Dictionary containing participant information Copy Ask AI @transport.event_handler ( "on_participant_joined" ) async def on_participant_joined ( transport , participant : Dict[ str , Any]): logger.info( f "Participant joined: { participant[ 'id' ] } " ) ​ on_participant_left async callback Called when a participant leaves the room. Parameters: transport : The DailyTransport instance participant : Dictionary containing participant information reason : String describing why the participant left, “leftCall” | “hidden” Copy Ask AI @transport.event_handler ( "on_participant_left" ) async def on_participant_left ( transport , participant : Dict[ str , Any], reason : str ): logger.info( f "Participant { participant[ 'id' ] } left: { reason } " ) # Cancel the pipeline task to stop processing and remove the bot from the Daily room await task.cancel() ​ on_participant_updated async callback Event emitted when a participant is updated. This can mean either the participant’s metadata was updated, or the tracks belonging to the participant changed. Parameters: transport : The DailyTransport instance participant : Dictionary containing updated participant information Copy Ask AI @transport.event_handler ( "on_participant_updated" ) async def on_participant_updated ( transport , participant : Dict[ str , Any]): logger.info( f "Participant updated: { participant } " ) ​ Communication Events ​ on_app_message async callback Event emitted when a custom app message is received from another participant or via the REST API. Parameters: transport : The DailyTransport instance message : The message content (any type) sender : String identifier of the message sender Copy Ask AI @transport.event_handler ( "on_app_message" ) async def on_app_message ( transport , message : Any, sender : str ): logger.info( f "Message from { sender } : { message } " ) ​ on_call_state_updated async callback Event emitted when the call state changes, normally as a consequence of joining or leaving the call. Parameters: transport : The DailyTransport instance state : String representing the new call state. Learn more about call states . Copy Ask AI @transport.event_handler ( "on_call_state_updated" ) async def on_call_state_updated ( transport , state : str ): logger.info( f "Call state updated: { state } " ) ​ Dial Events ​ Dial-in ​ on_dialin_ready async callback Event emitted when dial-in is ready. This happens after the room has connected to the SIP endpoint and the system is ready to receive dial-in calls. Parameters: transport : The DailyTransport instance sip_endpoint : String containing the SIP endpoint information Copy Ask AI @transport.event_handler ( "on_dialin_ready" ) async def on_dialin_ready ( transport , sip_endpoint : str ): logger.info( f "Dial-in ready at: { sip_endpoint } " ) ​ on_dialin_connected async callback Event emitted when the session with the dial-in remote end is established (i.e. SIP endpoint or PSTN are connectd to the Daily room). Note: connected does not mean media (audio or video) has started flowing between the room and PSTN, it means the room received the connection request and both endpoints are negotiating the media flow. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. See DialinConnectedEvent . Copy Ask AI @transport.event_handler ( "on_dialin_connected" ) async def on_dialin_connected ( transport , data : Any): logger.info( f "Dial-in connected: { data } " ) ​ on_dialin_stopped async callback Event emitted when the dial-in remote end disconnects the call. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. See DialinStoppedEvent . Copy Ask AI @transport.event_handler ( "on_dialin_stopped" ) async def on_dialin_stopped ( transport , data : Any): logger.info( f "Dial-in stopped: { data } " ) ​ on_dialin_error async callback Event emitted in the case of dial-in errors which are fatal and the service cannot proceed. For example, an error in SDP negotiation is fatal to the media/SIP pipeline and will result in dialin-error being triggered. Parameters: transport : The DailyTransport instance data : Dictionary containing error information. See DialinEvent . Copy Ask AI @transport.event_handler ( "on_dialin_error" ) async def on_dialin_error ( transport , data : Any): logger.error( f "Dial-in error: { data } " ) ​ on_dialin_warning async callback Event emitted there is a dial-in non-fatal error, such as the selected codec not being used and a fallback codec being utilized. Parameters: transport : The DailyTransport instance data : Dictionary containing warning information. See DialinEvent . Copy Ask AI @transport.event_handler ( "on_dialin_warning" ) async def on_dialin_warning ( transport , data : Any): logger.warning( f "Dial-in warning: { data } " ) ​ Dial-out ​ on_dialout_answered async callback Event emitted when the session with the dial-out remote end is answered. Parameters: transport : The DailyTransport instance data : Dictionary containing call information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_answered" ) async def on_dialout_answered ( transport , data : Any): logger.info( f "Dial-out answered: { data } " ) ​ on_dialout_connected async callback Event emitted when the session with the dial-out remote end is established. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_connected" ) async def on_dialout_connected ( transport , data : Any): logger.info( f "Dial-out connected: { data } " ) ​ on_dialout_stopped async callback Event emitted when the dial-out session is stopped. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_stopped" ) async def on_dialout_stopped ( transport , data : Any): logger.info( f "Dial-out stopped: { data } " ) ​ on_dialout_error async callback Event emitted in the case of dial-out errors which are fatal and the service cannot proceed. For example, an error in SDP negotiation is fatal to the media/SIP pipeline and will result in dialout-error being triggered. Parameters: transport : The DailyTransport instance data : Dictionary containing error information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_error" ) async def on_dialout_error ( transport , data : Any): logger.error( f "Dial-out error: { data } " ) ​ on_dialout_warning async callback Event emitted there is a dial-out non-fatal error, such as the selected codec not being used and a fallback codec being utilized. Parameters: transport : The DailyTransport instance data : Dictionary containing warning information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_warning" ) async def on_dialout_warning ( transport , data : Any): logger.warning( f "Dial-out warning: { data } " ) ​ Transcription Events ​ on_transcription_message async callback Called when a transcription message is received. This includes both interim and final transcriptions. Parameters: transport : The DailyTransport instance message : Dictionary containing transcription data including. Learn more . Copy Ask AI @transport.event_handler ( "on_transcription_message" ) async def on_transcription_message ( transport , message : Dict[ str , Any]): participant_id = message.get( "participantId" ) text = message.get( "text" ) is_final = message[ "rawResponse" ][ "is_final" ] logger.info( f "Transcription from { participant_id } : { text } (final: { is_final } )" ) ​ Recording Events ​ on_recording_started async callback Called when a room recording starts successfully. Parameters: transport : The DailyTransport instance status : Dictionary containing recording status information. Learn more . Copy Ask AI @transport.event_handler ( "on_recording_started" ) async def on_recording_started ( transport , status : Dict[ str , Any]): logger.info( f "Recording started with status: { status } " ) ​ on_recording_stopped async callback Called when a room recording stops. Parameters: transport : The DailyTransport instance stream_id : String identifier of the stopped recording stream Copy Ask AI @transport.event_handler ( "on_recording_stopped" ) async def on_recording_stopped ( transport , stream_id : str ): logger.info( f "Recording stopped for stream: { stream_id } " ) ​ on_recording_error async callback Called when an error occurs during recording. Parameters: transport : The DailyTransport instance stream_id : String identifier of the recording stream message : Error message describing what went wrong Copy Ask AI @transport.event_handler ( "on_recording_error" ) async def on_recording_error ( transport , stream_id : str , message : str ): logger.error( f "Recording error for stream { stream_id } : { message } " ) ​ Error Events ​ on_error async callback Called when a transport error occurs. Parameters: transport : The DailyTransport instance error : String containing error details Copy Ask AI @transport.event_handler ( "on_error" ) async def on_error ( transport , error : str ): logger.error( f "Transport error: { error } " ) ​ Notes on Callbacks All callbacks are asynchronous Callbacks are executed in the order they were registered Multiple handlers can be registered for the same event Exceptions in callbacks are logged but don’t stop the transport Callbacks should be lightweight to avoid blocking the event loop Heavy processing should be offloaded to separate tasks ​ Frame Types ​ Input Frames ​ InputAudioRawFrame Frame Raw audio data from participants ​ UserImageRawFrame Frame Video frames from participants ​ Output Frames ​ OutputAudioRawFrame Frame Audio data to be sent ​ OutputImageRawFrame Frame Video frames to be sent ​ Methods ​ Room Management ​ participants method Returns a dictionary of all participants currently in the room. Copy Ask AI def participants () -> Dict[ str , Any] See participants for more details. ​ participant_counts method Returns participant count statistics for the room. Copy Ask AI def participant_counts () -> Dict[ str , int ] See participant_counts for more details. ​ send_message async method Sends a message to participants in the Daily room. Messages can be directed to all participants or targeted to a specific participant. Copy Ask AI async def send_message ( frame : TransportMessageFrame | TransportMessageUrgentFrame) Parameters: frame : A transport message frame containing the message to send. Can be either: TransportMessageFrame : Standard message TransportMessageUrgentFrame : High-priority message Message Structure: Messages should conform to the expected format for the receiving system. For RTVI messages, use: Copy Ask AI { "label" : "rtvi-ai" , # Required for RTVI protocol "type" : "message-type" , # Defines how the client will process it "id" : "unique-id" , # Unique identifier for the message "data" : { # Custom payload # Your custom data fields } } Example - Standard Message: Copy Ask AI import uuid # Send a standard message to all participants message_data = { "label" : "rtvi-ai" , "type" : "custom-message" , "id" : str (uuid.uuid4()), "data" : { "message" : "Hello everyone!" , "timestamp" : datetime.datetime.now().isoformat() } } await transport.send_message(TransportMessageFrame( message = message_data)) Example - Urgent Message to Specific Participant: Copy Ask AI # Send urgent message to specific participant urgent_data = { "label" : "rtvi-ai" , "type" : "server-message" , "id" : str (uuid.uuid4()), "data" : { "message" : "Private message" , "importance" : "high" } } await transport.send_message( DailyTransportMessageUrgentFrame( message = urgent_data, participant_id = "participant-123" ) ) The message will be delivered to: All participants if no specific participant_id is set Only the specified participant if participant_id is set Through the Daily app messaging system Can be received by registering an on_app_message handler Use TransportMessageUrgentFrame for high-priority messages that need immediate delivery. ​ Media Control ​ send_image async method Sends an image frame to the room. Copy Ask AI async def send_image ( frame : OutputImageRawFrame | SpriteFrame) -> None Parameters: frame : Image frame to send, either raw image data or sprite animation ​ send_audio async method Sends an audio frame to the room. Copy Ask AI async def send_audio ( frame : OutputAudioRawFrame) -> None Parameters: frame : Audio frame to send ​ Video Management ​ capture_participant_video async method Starts capturing video from a specific participant. Copy Ask AI async def capture_participant_video ( participant_id : str , framerate : int = 30 , video_source : str = "camera" , color_format : str = "RGB" ) -> None Parameters: participant_id : ID of the participant to capture framerate : Target frame rate (default: 30) video_source : Video source type (default: “camera”) color_format : Color format of the video (default: “RGB”) To request an image from the camera stream, set the framerate to 0 and push a UserImageRequestFrame . ​ Transcription Control ​ capture_participant_transcription async method Starts capturing and transcribing audio from a specific participant. Copy Ask AI async def capture_participant_transcription ( participant_id : str ) -> None Parameters: participant_id : ID of the participant to transcribe ​ update_transcription async method Updates the transcription configuration for specific participants or instances. Copy Ask AI async def update_transcription ( participants : List[ str ] | None = None , instance_id : str | None = None ) -> None Parameters: participants : Optional list of participant IDs to transcribe. If None, transcribes all participants. instance_id : Optional specific transcription instance ID to update Example: Copy Ask AI # Update transcription for specific participants await transport.update_transcription( participants = [ "participant-123" , "participant-456" ]) # Update specific transcription instance await transport.update_transcription( instance_id = "transcription-789" ) ​ Recording Control ​ start_recording async method Starts recording the room session. Copy Ask AI async def start_recording ( streaming_settings : Dict = None , stream_id : str = None , force_new : bool = None ) -> None Parameters: streaming_settings : Recording configuration settings stream_id : Optional stream identifier force_new : Force start a new recording See start_recording for more details. ​ stop_recording async method Stops an active recording. Copy Ask AI async def stop_recording ( stream_id : str = None ) -> None Parameters: stream_id : Optional stream identifier to stop specific recording See stop_recording for more details. ​ Dial-out Control ​ start_dialout async method Initiates a dial-out call. Copy Ask AI async def start_dialout ( settings : Dict = None ) -> None Parameters: settings : Dial-out configuration settings See start_dialout for more details. ​ stop_dialout async method Stops an active dial-out call. Copy Ask AI async def stop_dialout ( participant_id : str ) -> None Parameters: participant_id : ID of the participant to disconnect See stop_dialout for more details. ​ send_dtmf async method Sends DTMF tones in an existing dial-out session. Copy Ask AI async def send_dtmf ( settings : Dict = None ) -> None Parameters: settings : DTMF settings See send_dtmf for more details. ​ Subscription Management ​ update_subscriptions async method Updates subscriptions and subscription profiles. This function allows you to update subscription profiles and at the same time assign specific subscription profiles to a participant and even change specific settings for some participants. Copy Ask AI async def update_subscriptions ( participant_settings : Dict = None , profile_settings : Dict = None ) -> None Parameters: participant_settings : Per-participant subscription settings profile_settings : Global subscription profile settings See update_subscriptions for more details. ​ Participant Admin When using the DailyTransport, if your bot is given a meeting token with owner or participant admin permissions, it can remote-control other participants’ capabilities and permissions. With this control, the bot can modify settings for how others send and receive media, how they are seen in a session, and whether they have administrative controls. ​ update_remote_participants async method Updates permissions and settings for specific participants in the room. Copy Ask AI async def update_remote_participants ( remote_participants : Mapping[ str , Any] = None ) -> None Parameters: remote_participants : Dictionary mapping participant IDs to their updated settings This method allows fine-grained control over participant capabilities and permissions in the Daily room. Examples: Copy Ask AI # Mute a participant's microphone await transport.update_remote_participants({ "participant-123" : { "inputsEnabled" : { "microphone" : False } } }) # Revoke send permissions (mutes the participant and prevents them from unmuting) await transport.update_remote_participants({ "participant-123" : { "permissions" : { "canSend" : [] # Empty list revokes all send permissions } } }) # Configure selective audio routing await transport.update_remote_participants({ "participant-123" : { # the customer "permissions" : { "canReceive" : { "base" : False , # Don't receive from anyone by default "byUserId" : { "hold-music" : True # Only receive from hold music player } } } } }) # Set up user-to-user permissions mapping (e.g., when transferring calls) # Assumes customer was previously unable to hear anyone (they were on hold) and agent was speaking to bot await transport.update_remote_participants({ "participant-123" : { # the customer "permissions" : { "base" : False , "canReceive" : { "byUserId" : { "agent" : True }} } }, "participant-456" : { # the agent "permissions" : { "base" : False , "canReceive" : { "byUserId" : { "customer" : True }} } } }) # Promote a participant to be a participant admin await transport.update_remote_participants({ "participant-123" : { "permissions" : { "canAdmin" : [ "participants" ] } } }) Permissions Settings: For comprehensive details on permissions settings, see the Daily API documentation: daily-python’s update_remote_participants() documentation daily-js’s updateParticipant() -> updatePermission and participants().permissions ​ Chat messages ​ send_prebuilt_chat_message async method When testing with Daily Prebuilt, you can send messages to the Prebuilt chat using this method. Parameters: message (str): The chat message to send user_name (str): The name of the user sending the message See send_prebuilt_chat_message for more details. ​ Properties ​ participant_id string Returns the transport’s participant ID in the room. ​ room_url string Returns the transport’s room URL for the room currently in use. ​ Notes Supports real-time audio/video communication Handles WebRTC connection management Provides voice activity detection Includes transcription capabilities Manages participant interactions Supports recording and streaming Thread-safe processing All callbacks are asynchronous Multiple handlers can be registered for the same event Exceptions in callbacks are logged but don’t stop the transport Supported Services SmallWebRTCTransport On this page Overview Installation Configuration Constructor Parameters DailyParams Configuration Audio Output Configuration Audio Input Configuration Video Output Configuration Voice Activity Detection (VAD) Feature Settings Basic Usage Event Callbacks Connection Events Participant Events Communication Events Dial Events Dial-in Dial-out Transcription Events Recording Events Error Events Notes on Callbacks Frame Types Input Frames Output Frames Methods Room Management Media Control Video Management Transcription Control Recording Control Dial-out Control Subscription Management Participant Admin Chat messages Properties Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/transport_daily_aae35385.txt b/transport_daily_aae35385.txt
new file mode 100644
index 0000000000000000000000000000000000000000..fca13cb1250e5dc0c6708756686410528b740f4c
--- /dev/null
+++ b/transport_daily_aae35385.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/transport/daily#recording-events
+Title: Daily WebRTC - Pipecat
+==================================================
+
+Daily WebRTC - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Transport Daily WebRTC Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Daily WebRTC SmallWebRTCTransport FastAPI WebSocket WebSocket Server Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview DailyTransport provides real-time audio and video communication capabilities using Daily’s WebRTC platform. It supports bidirectional audio/video streams, transcription, voice activity detection (VAD), and participant management. ​ Installation To use DailyTransport , install the required dependencies: Copy Ask AI pip install "pipecat-ai[daily]" You’ll also need to set up your Daily API key as an environment variable: DAILY_API_KEY . You can obtain a Daily API key by signing up at Daily . ​ Configuration ​ Constructor Parameters ​ room_url str required Daily room URL ​ token str | None Daily room token ​ bot_name str required Name of the bot in the room ​ params DailyParams default: "DailyParams()" Transport configuration parameters ​ DailyParams Configuration ​ api_url str default: "https://api.daily.co/v1" Daily API endpoint URL ​ api_key str default: "" Daily API key for authentication ​ Audio Output Configuration ​ audio_out_enabled bool default: "False" Enable audio output capabilities ​ audio_out_is_live bool default: "False" Enable live audio streaming mode ​ audio_out_sample_rate int default: "None" Audio output sample rate in Hz ​ audio_out_channels int default: "1" Number of audio output channels ​ audio_out_bitrate int default: "96000" Audio output bitrate in bits per second ​ Audio Input Configuration ​ audio_in_enabled bool default: "False" Enable audio input capabilities ​ audio_in_passthrough bool default: "False" When enabled, incoming audio frames are pushed downstream ​ audio_in_sample_rate int default: "None" Audio input sample rate in Hz ​ audio_in_channels int default: "1" Number of audio input channels ​ audio_in_filter Optional[BaseAudioFilter] default: "None" Audio filter for input processing. Supported filters are: KrispFilter() and NoisereduceFilter() . See the KrispFilter() reference docs and NoisereduceFilter() reference docs ` for more information. ​ Video Output Configuration ​ video_out_enabled bool default: "False" Enable video output capabilities ​ video_out_is_live bool default: "False" Enable live video streaming mode ​ video_out_width int default: "1024" Video output width in pixels ​ video_out_height int default: "768" Video output height in pixels ​ video_out_bitrate int default: "800000" Video output bitrate in bits per second ​ video_out_framerate int default: "30" Video output frame rate ​ video_out_color_format str default: "RGB" Video color format (RGB, BGR, etc.) ​ Voice Activity Detection (VAD) ​ vad_analyzer VADAnalyzer | None default: "None" Voice Activity Detection analyzer. You can set this to either SileroVADAnalyzer() or WebRTCVADAnalyzer() . SileroVADAnalyzer is the recommended option. Learn more about the SileroVADAnalyzer . ​ Feature Settings ​ dialin_settings Optional[DailyDialinSettings] default: "None" Configuration for dial-in functionality ​ transcription_enabled bool default: "False" Enable real-time transcription ​ transcription_settings DailyTranscriptionSettings Configuration for transcription features Copy Ask AI class DailyTranscriptionSettings ( BaseModel ): language: str = "en" # Default language model: str = "nova-2-general" # Transcription model profanity_filter: bool = True # Filter profanity redact: bool = False # Redact sensitive information endpointing: bool = True # Enable speech endpointing punctuate: bool = True # Add punctuation includeRawResponse: bool = True # Include raw API response extra: Mapping[ str , Any] = { "interim_results" : True # Provide any Deepgram-specific settings } ​ Basic Usage Copy Ask AI from pipecat.transports.services.daily import DailyTransport, DailyParams from pipecat.audio.vad.silero import SileroVADAnalyzer # Configure transport transport = DailyTransport( room_url = "https://your-domain.daily.co/room-name" , token = "your-room-token" , bot_name = "AI Assistant" , params = DailyParams( audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer( params = VADParams( stop_secs = 0.5 )), ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), # Handle incoming audio/video stt, # Speech-to-text llm, # Language model tts, # Text-to-speech transport.output() # Handle outgoing audio/video ]) # Use transport event handlers to manage the call lifecycle # Event handler for when the user joins the room: @transport.event_handler ( "on_first_participant_joined" ) async def on_first_participant_joined ( transport , participant ): # Start transcription for the user await transport.capture_participant_transcription(participant[ "id" ]) # Kick off the conversation. await task.queue_frames([context_aggregator.user().get_context_frame()]) # Event handler for when the user leaves the room: @transport.event_handler ( "on_participant_left" ) async def on_participant_left ( transport , participant , reason ): # Cancel the pipeline, which stops processing and removes the bot from the room await task.cancel() ​ Event Callbacks DailyTransport provides a comprehensive callback system for handling various events. Register callbacks using the @transport.event_handler() decorator. ​ Connection Events ​ on_joined async callback Called when the bot successfully joins the room. Parameters: transport : The DailyTransport instance data : Dictionary containing room join information Copy Ask AI @transport.event_handler ( "on_joined" ) async def on_joined ( transport , data : Dict[ str , Any]): logger.info( f "Joined room with data: { data } " ) ​ on_left async callback Called when the bot leaves the room. Parameters: transport : The DailyTransport instance Copy Ask AI @transport.event_handler ( "on_left" ) async def on_left ( transport ): logger.info( "Left room" ) ​ Participant Events ​ on_first_participant_joined async callback Called when the first participant joins the room. Useful for initializing conversations. Parameters: transport : The DailyTransport instance participant : Dictionary containing participant information Copy Ask AI @transport.event_handler ( "on_first_participant_joined" ) async def on_first_participant_joined ( transport , participant : Dict[ str , Any]): await transport.capture_participant_transcription(participant[ "id" ]) await task.queue_frames([LLMMessagesFrame(initial_messages)]) ​ on_participant_joined async callback Called when any participant joins the room. Parameters: transport : The DailyTransport instance participant : Dictionary containing participant information Copy Ask AI @transport.event_handler ( "on_participant_joined" ) async def on_participant_joined ( transport , participant : Dict[ str , Any]): logger.info( f "Participant joined: { participant[ 'id' ] } " ) ​ on_participant_left async callback Called when a participant leaves the room. Parameters: transport : The DailyTransport instance participant : Dictionary containing participant information reason : String describing why the participant left, “leftCall” | “hidden” Copy Ask AI @transport.event_handler ( "on_participant_left" ) async def on_participant_left ( transport , participant : Dict[ str , Any], reason : str ): logger.info( f "Participant { participant[ 'id' ] } left: { reason } " ) # Cancel the pipeline task to stop processing and remove the bot from the Daily room await task.cancel() ​ on_participant_updated async callback Event emitted when a participant is updated. This can mean either the participant’s metadata was updated, or the tracks belonging to the participant changed. Parameters: transport : The DailyTransport instance participant : Dictionary containing updated participant information Copy Ask AI @transport.event_handler ( "on_participant_updated" ) async def on_participant_updated ( transport , participant : Dict[ str , Any]): logger.info( f "Participant updated: { participant } " ) ​ Communication Events ​ on_app_message async callback Event emitted when a custom app message is received from another participant or via the REST API. Parameters: transport : The DailyTransport instance message : The message content (any type) sender : String identifier of the message sender Copy Ask AI @transport.event_handler ( "on_app_message" ) async def on_app_message ( transport , message : Any, sender : str ): logger.info( f "Message from { sender } : { message } " ) ​ on_call_state_updated async callback Event emitted when the call state changes, normally as a consequence of joining or leaving the call. Parameters: transport : The DailyTransport instance state : String representing the new call state. Learn more about call states . Copy Ask AI @transport.event_handler ( "on_call_state_updated" ) async def on_call_state_updated ( transport , state : str ): logger.info( f "Call state updated: { state } " ) ​ Dial Events ​ Dial-in ​ on_dialin_ready async callback Event emitted when dial-in is ready. This happens after the room has connected to the SIP endpoint and the system is ready to receive dial-in calls. Parameters: transport : The DailyTransport instance sip_endpoint : String containing the SIP endpoint information Copy Ask AI @transport.event_handler ( "on_dialin_ready" ) async def on_dialin_ready ( transport , sip_endpoint : str ): logger.info( f "Dial-in ready at: { sip_endpoint } " ) ​ on_dialin_connected async callback Event emitted when the session with the dial-in remote end is established (i.e. SIP endpoint or PSTN are connectd to the Daily room). Note: connected does not mean media (audio or video) has started flowing between the room and PSTN, it means the room received the connection request and both endpoints are negotiating the media flow. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. See DialinConnectedEvent . Copy Ask AI @transport.event_handler ( "on_dialin_connected" ) async def on_dialin_connected ( transport , data : Any): logger.info( f "Dial-in connected: { data } " ) ​ on_dialin_stopped async callback Event emitted when the dial-in remote end disconnects the call. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. See DialinStoppedEvent . Copy Ask AI @transport.event_handler ( "on_dialin_stopped" ) async def on_dialin_stopped ( transport , data : Any): logger.info( f "Dial-in stopped: { data } " ) ​ on_dialin_error async callback Event emitted in the case of dial-in errors which are fatal and the service cannot proceed. For example, an error in SDP negotiation is fatal to the media/SIP pipeline and will result in dialin-error being triggered. Parameters: transport : The DailyTransport instance data : Dictionary containing error information. See DialinEvent . Copy Ask AI @transport.event_handler ( "on_dialin_error" ) async def on_dialin_error ( transport , data : Any): logger.error( f "Dial-in error: { data } " ) ​ on_dialin_warning async callback Event emitted there is a dial-in non-fatal error, such as the selected codec not being used and a fallback codec being utilized. Parameters: transport : The DailyTransport instance data : Dictionary containing warning information. See DialinEvent . Copy Ask AI @transport.event_handler ( "on_dialin_warning" ) async def on_dialin_warning ( transport , data : Any): logger.warning( f "Dial-in warning: { data } " ) ​ Dial-out ​ on_dialout_answered async callback Event emitted when the session with the dial-out remote end is answered. Parameters: transport : The DailyTransport instance data : Dictionary containing call information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_answered" ) async def on_dialout_answered ( transport , data : Any): logger.info( f "Dial-out answered: { data } " ) ​ on_dialout_connected async callback Event emitted when the session with the dial-out remote end is established. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_connected" ) async def on_dialout_connected ( transport , data : Any): logger.info( f "Dial-out connected: { data } " ) ​ on_dialout_stopped async callback Event emitted when the dial-out session is stopped. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_stopped" ) async def on_dialout_stopped ( transport , data : Any): logger.info( f "Dial-out stopped: { data } " ) ​ on_dialout_error async callback Event emitted in the case of dial-out errors which are fatal and the service cannot proceed. For example, an error in SDP negotiation is fatal to the media/SIP pipeline and will result in dialout-error being triggered. Parameters: transport : The DailyTransport instance data : Dictionary containing error information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_error" ) async def on_dialout_error ( transport , data : Any): logger.error( f "Dial-out error: { data } " ) ​ on_dialout_warning async callback Event emitted there is a dial-out non-fatal error, such as the selected codec not being used and a fallback codec being utilized. Parameters: transport : The DailyTransport instance data : Dictionary containing warning information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_warning" ) async def on_dialout_warning ( transport , data : Any): logger.warning( f "Dial-out warning: { data } " ) ​ Transcription Events ​ on_transcription_message async callback Called when a transcription message is received. This includes both interim and final transcriptions. Parameters: transport : The DailyTransport instance message : Dictionary containing transcription data including. Learn more . Copy Ask AI @transport.event_handler ( "on_transcription_message" ) async def on_transcription_message ( transport , message : Dict[ str , Any]): participant_id = message.get( "participantId" ) text = message.get( "text" ) is_final = message[ "rawResponse" ][ "is_final" ] logger.info( f "Transcription from { participant_id } : { text } (final: { is_final } )" ) ​ Recording Events ​ on_recording_started async callback Called when a room recording starts successfully. Parameters: transport : The DailyTransport instance status : Dictionary containing recording status information. Learn more . Copy Ask AI @transport.event_handler ( "on_recording_started" ) async def on_recording_started ( transport , status : Dict[ str , Any]): logger.info( f "Recording started with status: { status } " ) ​ on_recording_stopped async callback Called when a room recording stops. Parameters: transport : The DailyTransport instance stream_id : String identifier of the stopped recording stream Copy Ask AI @transport.event_handler ( "on_recording_stopped" ) async def on_recording_stopped ( transport , stream_id : str ): logger.info( f "Recording stopped for stream: { stream_id } " ) ​ on_recording_error async callback Called when an error occurs during recording. Parameters: transport : The DailyTransport instance stream_id : String identifier of the recording stream message : Error message describing what went wrong Copy Ask AI @transport.event_handler ( "on_recording_error" ) async def on_recording_error ( transport , stream_id : str , message : str ): logger.error( f "Recording error for stream { stream_id } : { message } " ) ​ Error Events ​ on_error async callback Called when a transport error occurs. Parameters: transport : The DailyTransport instance error : String containing error details Copy Ask AI @transport.event_handler ( "on_error" ) async def on_error ( transport , error : str ): logger.error( f "Transport error: { error } " ) ​ Notes on Callbacks All callbacks are asynchronous Callbacks are executed in the order they were registered Multiple handlers can be registered for the same event Exceptions in callbacks are logged but don’t stop the transport Callbacks should be lightweight to avoid blocking the event loop Heavy processing should be offloaded to separate tasks ​ Frame Types ​ Input Frames ​ InputAudioRawFrame Frame Raw audio data from participants ​ UserImageRawFrame Frame Video frames from participants ​ Output Frames ​ OutputAudioRawFrame Frame Audio data to be sent ​ OutputImageRawFrame Frame Video frames to be sent ​ Methods ​ Room Management ​ participants method Returns a dictionary of all participants currently in the room. Copy Ask AI def participants () -> Dict[ str , Any] See participants for more details. ​ participant_counts method Returns participant count statistics for the room. Copy Ask AI def participant_counts () -> Dict[ str , int ] See participant_counts for more details. ​ send_message async method Sends a message to participants in the Daily room. Messages can be directed to all participants or targeted to a specific participant. Copy Ask AI async def send_message ( frame : TransportMessageFrame | TransportMessageUrgentFrame) Parameters: frame : A transport message frame containing the message to send. Can be either: TransportMessageFrame : Standard message TransportMessageUrgentFrame : High-priority message Message Structure: Messages should conform to the expected format for the receiving system. For RTVI messages, use: Copy Ask AI { "label" : "rtvi-ai" , # Required for RTVI protocol "type" : "message-type" , # Defines how the client will process it "id" : "unique-id" , # Unique identifier for the message "data" : { # Custom payload # Your custom data fields } } Example - Standard Message: Copy Ask AI import uuid # Send a standard message to all participants message_data = { "label" : "rtvi-ai" , "type" : "custom-message" , "id" : str (uuid.uuid4()), "data" : { "message" : "Hello everyone!" , "timestamp" : datetime.datetime.now().isoformat() } } await transport.send_message(TransportMessageFrame( message = message_data)) Example - Urgent Message to Specific Participant: Copy Ask AI # Send urgent message to specific participant urgent_data = { "label" : "rtvi-ai" , "type" : "server-message" , "id" : str (uuid.uuid4()), "data" : { "message" : "Private message" , "importance" : "high" } } await transport.send_message( DailyTransportMessageUrgentFrame( message = urgent_data, participant_id = "participant-123" ) ) The message will be delivered to: All participants if no specific participant_id is set Only the specified participant if participant_id is set Through the Daily app messaging system Can be received by registering an on_app_message handler Use TransportMessageUrgentFrame for high-priority messages that need immediate delivery. ​ Media Control ​ send_image async method Sends an image frame to the room. Copy Ask AI async def send_image ( frame : OutputImageRawFrame | SpriteFrame) -> None Parameters: frame : Image frame to send, either raw image data or sprite animation ​ send_audio async method Sends an audio frame to the room. Copy Ask AI async def send_audio ( frame : OutputAudioRawFrame) -> None Parameters: frame : Audio frame to send ​ Video Management ​ capture_participant_video async method Starts capturing video from a specific participant. Copy Ask AI async def capture_participant_video ( participant_id : str , framerate : int = 30 , video_source : str = "camera" , color_format : str = "RGB" ) -> None Parameters: participant_id : ID of the participant to capture framerate : Target frame rate (default: 30) video_source : Video source type (default: “camera”) color_format : Color format of the video (default: “RGB”) To request an image from the camera stream, set the framerate to 0 and push a UserImageRequestFrame . ​ Transcription Control ​ capture_participant_transcription async method Starts capturing and transcribing audio from a specific participant. Copy Ask AI async def capture_participant_transcription ( participant_id : str ) -> None Parameters: participant_id : ID of the participant to transcribe ​ update_transcription async method Updates the transcription configuration for specific participants or instances. Copy Ask AI async def update_transcription ( participants : List[ str ] | None = None , instance_id : str | None = None ) -> None Parameters: participants : Optional list of participant IDs to transcribe. If None, transcribes all participants. instance_id : Optional specific transcription instance ID to update Example: Copy Ask AI # Update transcription for specific participants await transport.update_transcription( participants = [ "participant-123" , "participant-456" ]) # Update specific transcription instance await transport.update_transcription( instance_id = "transcription-789" ) ​ Recording Control ​ start_recording async method Starts recording the room session. Copy Ask AI async def start_recording ( streaming_settings : Dict = None , stream_id : str = None , force_new : bool = None ) -> None Parameters: streaming_settings : Recording configuration settings stream_id : Optional stream identifier force_new : Force start a new recording See start_recording for more details. ​ stop_recording async method Stops an active recording. Copy Ask AI async def stop_recording ( stream_id : str = None ) -> None Parameters: stream_id : Optional stream identifier to stop specific recording See stop_recording for more details. ​ Dial-out Control ​ start_dialout async method Initiates a dial-out call. Copy Ask AI async def start_dialout ( settings : Dict = None ) -> None Parameters: settings : Dial-out configuration settings See start_dialout for more details. ​ stop_dialout async method Stops an active dial-out call. Copy Ask AI async def stop_dialout ( participant_id : str ) -> None Parameters: participant_id : ID of the participant to disconnect See stop_dialout for more details. ​ send_dtmf async method Sends DTMF tones in an existing dial-out session. Copy Ask AI async def send_dtmf ( settings : Dict = None ) -> None Parameters: settings : DTMF settings See send_dtmf for more details. ​ Subscription Management ​ update_subscriptions async method Updates subscriptions and subscription profiles. This function allows you to update subscription profiles and at the same time assign specific subscription profiles to a participant and even change specific settings for some participants. Copy Ask AI async def update_subscriptions ( participant_settings : Dict = None , profile_settings : Dict = None ) -> None Parameters: participant_settings : Per-participant subscription settings profile_settings : Global subscription profile settings See update_subscriptions for more details. ​ Participant Admin When using the DailyTransport, if your bot is given a meeting token with owner or participant admin permissions, it can remote-control other participants’ capabilities and permissions. With this control, the bot can modify settings for how others send and receive media, how they are seen in a session, and whether they have administrative controls. ​ update_remote_participants async method Updates permissions and settings for specific participants in the room. Copy Ask AI async def update_remote_participants ( remote_participants : Mapping[ str , Any] = None ) -> None Parameters: remote_participants : Dictionary mapping participant IDs to their updated settings This method allows fine-grained control over participant capabilities and permissions in the Daily room. Examples: Copy Ask AI # Mute a participant's microphone await transport.update_remote_participants({ "participant-123" : { "inputsEnabled" : { "microphone" : False } } }) # Revoke send permissions (mutes the participant and prevents them from unmuting) await transport.update_remote_participants({ "participant-123" : { "permissions" : { "canSend" : [] # Empty list revokes all send permissions } } }) # Configure selective audio routing await transport.update_remote_participants({ "participant-123" : { # the customer "permissions" : { "canReceive" : { "base" : False , # Don't receive from anyone by default "byUserId" : { "hold-music" : True # Only receive from hold music player } } } } }) # Set up user-to-user permissions mapping (e.g., when transferring calls) # Assumes customer was previously unable to hear anyone (they were on hold) and agent was speaking to bot await transport.update_remote_participants({ "participant-123" : { # the customer "permissions" : { "base" : False , "canReceive" : { "byUserId" : { "agent" : True }} } }, "participant-456" : { # the agent "permissions" : { "base" : False , "canReceive" : { "byUserId" : { "customer" : True }} } } }) # Promote a participant to be a participant admin await transport.update_remote_participants({ "participant-123" : { "permissions" : { "canAdmin" : [ "participants" ] } } }) Permissions Settings: For comprehensive details on permissions settings, see the Daily API documentation: daily-python’s update_remote_participants() documentation daily-js’s updateParticipant() -> updatePermission and participants().permissions ​ Chat messages ​ send_prebuilt_chat_message async method When testing with Daily Prebuilt, you can send messages to the Prebuilt chat using this method. Parameters: message (str): The chat message to send user_name (str): The name of the user sending the message See send_prebuilt_chat_message for more details. ​ Properties ​ participant_id string Returns the transport’s participant ID in the room. ​ room_url string Returns the transport’s room URL for the room currently in use. ​ Notes Supports real-time audio/video communication Handles WebRTC connection management Provides voice activity detection Includes transcription capabilities Manages participant interactions Supports recording and streaming Thread-safe processing All callbacks are asynchronous Multiple handlers can be registered for the same event Exceptions in callbacks are logged but don’t stop the transport Supported Services SmallWebRTCTransport On this page Overview Installation Configuration Constructor Parameters DailyParams Configuration Audio Output Configuration Audio Input Configuration Video Output Configuration Voice Activity Detection (VAD) Feature Settings Basic Usage Event Callbacks Connection Events Participant Events Communication Events Dial Events Dial-in Dial-out Transcription Events Recording Events Error Events Notes on Callbacks Frame Types Input Frames Output Frames Methods Room Management Media Control Video Management Transcription Control Recording Control Dial-out Control Subscription Management Participant Admin Chat messages Properties Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/transport_daily_b3b04aa1.txt b/transport_daily_b3b04aa1.txt
new file mode 100644
index 0000000000000000000000000000000000000000..bd2e2e5adcc403f4116ad33cc4bf39369eee8247
--- /dev/null
+++ b/transport_daily_b3b04aa1.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/transport/daily#param-update-subscriptions
+Title: Daily WebRTC - Pipecat
+==================================================
+
+Daily WebRTC - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Transport Daily WebRTC Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Daily WebRTC SmallWebRTCTransport FastAPI WebSocket WebSocket Server Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview DailyTransport provides real-time audio and video communication capabilities using Daily’s WebRTC platform. It supports bidirectional audio/video streams, transcription, voice activity detection (VAD), and participant management. ​ Installation To use DailyTransport , install the required dependencies: Copy Ask AI pip install "pipecat-ai[daily]" You’ll also need to set up your Daily API key as an environment variable: DAILY_API_KEY . You can obtain a Daily API key by signing up at Daily . ​ Configuration ​ Constructor Parameters ​ room_url str required Daily room URL ​ token str | None Daily room token ​ bot_name str required Name of the bot in the room ​ params DailyParams default: "DailyParams()" Transport configuration parameters ​ DailyParams Configuration ​ api_url str default: "https://api.daily.co/v1" Daily API endpoint URL ​ api_key str default: "" Daily API key for authentication ​ Audio Output Configuration ​ audio_out_enabled bool default: "False" Enable audio output capabilities ​ audio_out_is_live bool default: "False" Enable live audio streaming mode ​ audio_out_sample_rate int default: "None" Audio output sample rate in Hz ​ audio_out_channels int default: "1" Number of audio output channels ​ audio_out_bitrate int default: "96000" Audio output bitrate in bits per second ​ Audio Input Configuration ​ audio_in_enabled bool default: "False" Enable audio input capabilities ​ audio_in_passthrough bool default: "False" When enabled, incoming audio frames are pushed downstream ​ audio_in_sample_rate int default: "None" Audio input sample rate in Hz ​ audio_in_channels int default: "1" Number of audio input channels ​ audio_in_filter Optional[BaseAudioFilter] default: "None" Audio filter for input processing. Supported filters are: KrispFilter() and NoisereduceFilter() . See the KrispFilter() reference docs and NoisereduceFilter() reference docs ` for more information. ​ Video Output Configuration ​ video_out_enabled bool default: "False" Enable video output capabilities ​ video_out_is_live bool default: "False" Enable live video streaming mode ​ video_out_width int default: "1024" Video output width in pixels ​ video_out_height int default: "768" Video output height in pixels ​ video_out_bitrate int default: "800000" Video output bitrate in bits per second ​ video_out_framerate int default: "30" Video output frame rate ​ video_out_color_format str default: "RGB" Video color format (RGB, BGR, etc.) ​ Voice Activity Detection (VAD) ​ vad_analyzer VADAnalyzer | None default: "None" Voice Activity Detection analyzer. You can set this to either SileroVADAnalyzer() or WebRTCVADAnalyzer() . SileroVADAnalyzer is the recommended option. Learn more about the SileroVADAnalyzer . ​ Feature Settings ​ dialin_settings Optional[DailyDialinSettings] default: "None" Configuration for dial-in functionality ​ transcription_enabled bool default: "False" Enable real-time transcription ​ transcription_settings DailyTranscriptionSettings Configuration for transcription features Copy Ask AI class DailyTranscriptionSettings ( BaseModel ): language: str = "en" # Default language model: str = "nova-2-general" # Transcription model profanity_filter: bool = True # Filter profanity redact: bool = False # Redact sensitive information endpointing: bool = True # Enable speech endpointing punctuate: bool = True # Add punctuation includeRawResponse: bool = True # Include raw API response extra: Mapping[ str , Any] = { "interim_results" : True # Provide any Deepgram-specific settings } ​ Basic Usage Copy Ask AI from pipecat.transports.services.daily import DailyTransport, DailyParams from pipecat.audio.vad.silero import SileroVADAnalyzer # Configure transport transport = DailyTransport( room_url = "https://your-domain.daily.co/room-name" , token = "your-room-token" , bot_name = "AI Assistant" , params = DailyParams( audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer( params = VADParams( stop_secs = 0.5 )), ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), # Handle incoming audio/video stt, # Speech-to-text llm, # Language model tts, # Text-to-speech transport.output() # Handle outgoing audio/video ]) # Use transport event handlers to manage the call lifecycle # Event handler for when the user joins the room: @transport.event_handler ( "on_first_participant_joined" ) async def on_first_participant_joined ( transport , participant ): # Start transcription for the user await transport.capture_participant_transcription(participant[ "id" ]) # Kick off the conversation. await task.queue_frames([context_aggregator.user().get_context_frame()]) # Event handler for when the user leaves the room: @transport.event_handler ( "on_participant_left" ) async def on_participant_left ( transport , participant , reason ): # Cancel the pipeline, which stops processing and removes the bot from the room await task.cancel() ​ Event Callbacks DailyTransport provides a comprehensive callback system for handling various events. Register callbacks using the @transport.event_handler() decorator. ​ Connection Events ​ on_joined async callback Called when the bot successfully joins the room. Parameters: transport : The DailyTransport instance data : Dictionary containing room join information Copy Ask AI @transport.event_handler ( "on_joined" ) async def on_joined ( transport , data : Dict[ str , Any]): logger.info( f "Joined room with data: { data } " ) ​ on_left async callback Called when the bot leaves the room. Parameters: transport : The DailyTransport instance Copy Ask AI @transport.event_handler ( "on_left" ) async def on_left ( transport ): logger.info( "Left room" ) ​ Participant Events ​ on_first_participant_joined async callback Called when the first participant joins the room. Useful for initializing conversations. Parameters: transport : The DailyTransport instance participant : Dictionary containing participant information Copy Ask AI @transport.event_handler ( "on_first_participant_joined" ) async def on_first_participant_joined ( transport , participant : Dict[ str , Any]): await transport.capture_participant_transcription(participant[ "id" ]) await task.queue_frames([LLMMessagesFrame(initial_messages)]) ​ on_participant_joined async callback Called when any participant joins the room. Parameters: transport : The DailyTransport instance participant : Dictionary containing participant information Copy Ask AI @transport.event_handler ( "on_participant_joined" ) async def on_participant_joined ( transport , participant : Dict[ str , Any]): logger.info( f "Participant joined: { participant[ 'id' ] } " ) ​ on_participant_left async callback Called when a participant leaves the room. Parameters: transport : The DailyTransport instance participant : Dictionary containing participant information reason : String describing why the participant left, “leftCall” | “hidden” Copy Ask AI @transport.event_handler ( "on_participant_left" ) async def on_participant_left ( transport , participant : Dict[ str , Any], reason : str ): logger.info( f "Participant { participant[ 'id' ] } left: { reason } " ) # Cancel the pipeline task to stop processing and remove the bot from the Daily room await task.cancel() ​ on_participant_updated async callback Event emitted when a participant is updated. This can mean either the participant’s metadata was updated, or the tracks belonging to the participant changed. Parameters: transport : The DailyTransport instance participant : Dictionary containing updated participant information Copy Ask AI @transport.event_handler ( "on_participant_updated" ) async def on_participant_updated ( transport , participant : Dict[ str , Any]): logger.info( f "Participant updated: { participant } " ) ​ Communication Events ​ on_app_message async callback Event emitted when a custom app message is received from another participant or via the REST API. Parameters: transport : The DailyTransport instance message : The message content (any type) sender : String identifier of the message sender Copy Ask AI @transport.event_handler ( "on_app_message" ) async def on_app_message ( transport , message : Any, sender : str ): logger.info( f "Message from { sender } : { message } " ) ​ on_call_state_updated async callback Event emitted when the call state changes, normally as a consequence of joining or leaving the call. Parameters: transport : The DailyTransport instance state : String representing the new call state. Learn more about call states . Copy Ask AI @transport.event_handler ( "on_call_state_updated" ) async def on_call_state_updated ( transport , state : str ): logger.info( f "Call state updated: { state } " ) ​ Dial Events ​ Dial-in ​ on_dialin_ready async callback Event emitted when dial-in is ready. This happens after the room has connected to the SIP endpoint and the system is ready to receive dial-in calls. Parameters: transport : The DailyTransport instance sip_endpoint : String containing the SIP endpoint information Copy Ask AI @transport.event_handler ( "on_dialin_ready" ) async def on_dialin_ready ( transport , sip_endpoint : str ): logger.info( f "Dial-in ready at: { sip_endpoint } " ) ​ on_dialin_connected async callback Event emitted when the session with the dial-in remote end is established (i.e. SIP endpoint or PSTN are connectd to the Daily room). Note: connected does not mean media (audio or video) has started flowing between the room and PSTN, it means the room received the connection request and both endpoints are negotiating the media flow. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. See DialinConnectedEvent . Copy Ask AI @transport.event_handler ( "on_dialin_connected" ) async def on_dialin_connected ( transport , data : Any): logger.info( f "Dial-in connected: { data } " ) ​ on_dialin_stopped async callback Event emitted when the dial-in remote end disconnects the call. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. See DialinStoppedEvent . Copy Ask AI @transport.event_handler ( "on_dialin_stopped" ) async def on_dialin_stopped ( transport , data : Any): logger.info( f "Dial-in stopped: { data } " ) ​ on_dialin_error async callback Event emitted in the case of dial-in errors which are fatal and the service cannot proceed. For example, an error in SDP negotiation is fatal to the media/SIP pipeline and will result in dialin-error being triggered. Parameters: transport : The DailyTransport instance data : Dictionary containing error information. See DialinEvent . Copy Ask AI @transport.event_handler ( "on_dialin_error" ) async def on_dialin_error ( transport , data : Any): logger.error( f "Dial-in error: { data } " ) ​ on_dialin_warning async callback Event emitted there is a dial-in non-fatal error, such as the selected codec not being used and a fallback codec being utilized. Parameters: transport : The DailyTransport instance data : Dictionary containing warning information. See DialinEvent . Copy Ask AI @transport.event_handler ( "on_dialin_warning" ) async def on_dialin_warning ( transport , data : Any): logger.warning( f "Dial-in warning: { data } " ) ​ Dial-out ​ on_dialout_answered async callback Event emitted when the session with the dial-out remote end is answered. Parameters: transport : The DailyTransport instance data : Dictionary containing call information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_answered" ) async def on_dialout_answered ( transport , data : Any): logger.info( f "Dial-out answered: { data } " ) ​ on_dialout_connected async callback Event emitted when the session with the dial-out remote end is established. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_connected" ) async def on_dialout_connected ( transport , data : Any): logger.info( f "Dial-out connected: { data } " ) ​ on_dialout_stopped async callback Event emitted when the dial-out session is stopped. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_stopped" ) async def on_dialout_stopped ( transport , data : Any): logger.info( f "Dial-out stopped: { data } " ) ​ on_dialout_error async callback Event emitted in the case of dial-out errors which are fatal and the service cannot proceed. For example, an error in SDP negotiation is fatal to the media/SIP pipeline and will result in dialout-error being triggered. Parameters: transport : The DailyTransport instance data : Dictionary containing error information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_error" ) async def on_dialout_error ( transport , data : Any): logger.error( f "Dial-out error: { data } " ) ​ on_dialout_warning async callback Event emitted there is a dial-out non-fatal error, such as the selected codec not being used and a fallback codec being utilized. Parameters: transport : The DailyTransport instance data : Dictionary containing warning information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_warning" ) async def on_dialout_warning ( transport , data : Any): logger.warning( f "Dial-out warning: { data } " ) ​ Transcription Events ​ on_transcription_message async callback Called when a transcription message is received. This includes both interim and final transcriptions. Parameters: transport : The DailyTransport instance message : Dictionary containing transcription data including. Learn more . Copy Ask AI @transport.event_handler ( "on_transcription_message" ) async def on_transcription_message ( transport , message : Dict[ str , Any]): participant_id = message.get( "participantId" ) text = message.get( "text" ) is_final = message[ "rawResponse" ][ "is_final" ] logger.info( f "Transcription from { participant_id } : { text } (final: { is_final } )" ) ​ Recording Events ​ on_recording_started async callback Called when a room recording starts successfully. Parameters: transport : The DailyTransport instance status : Dictionary containing recording status information. Learn more . Copy Ask AI @transport.event_handler ( "on_recording_started" ) async def on_recording_started ( transport , status : Dict[ str , Any]): logger.info( f "Recording started with status: { status } " ) ​ on_recording_stopped async callback Called when a room recording stops. Parameters: transport : The DailyTransport instance stream_id : String identifier of the stopped recording stream Copy Ask AI @transport.event_handler ( "on_recording_stopped" ) async def on_recording_stopped ( transport , stream_id : str ): logger.info( f "Recording stopped for stream: { stream_id } " ) ​ on_recording_error async callback Called when an error occurs during recording. Parameters: transport : The DailyTransport instance stream_id : String identifier of the recording stream message : Error message describing what went wrong Copy Ask AI @transport.event_handler ( "on_recording_error" ) async def on_recording_error ( transport , stream_id : str , message : str ): logger.error( f "Recording error for stream { stream_id } : { message } " ) ​ Error Events ​ on_error async callback Called when a transport error occurs. Parameters: transport : The DailyTransport instance error : String containing error details Copy Ask AI @transport.event_handler ( "on_error" ) async def on_error ( transport , error : str ): logger.error( f "Transport error: { error } " ) ​ Notes on Callbacks All callbacks are asynchronous Callbacks are executed in the order they were registered Multiple handlers can be registered for the same event Exceptions in callbacks are logged but don’t stop the transport Callbacks should be lightweight to avoid blocking the event loop Heavy processing should be offloaded to separate tasks ​ Frame Types ​ Input Frames ​ InputAudioRawFrame Frame Raw audio data from participants ​ UserImageRawFrame Frame Video frames from participants ​ Output Frames ​ OutputAudioRawFrame Frame Audio data to be sent ​ OutputImageRawFrame Frame Video frames to be sent ​ Methods ​ Room Management ​ participants method Returns a dictionary of all participants currently in the room. Copy Ask AI def participants () -> Dict[ str , Any] See participants for more details. ​ participant_counts method Returns participant count statistics for the room. Copy Ask AI def participant_counts () -> Dict[ str , int ] See participant_counts for more details. ​ send_message async method Sends a message to participants in the Daily room. Messages can be directed to all participants or targeted to a specific participant. Copy Ask AI async def send_message ( frame : TransportMessageFrame | TransportMessageUrgentFrame) Parameters: frame : A transport message frame containing the message to send. Can be either: TransportMessageFrame : Standard message TransportMessageUrgentFrame : High-priority message Message Structure: Messages should conform to the expected format for the receiving system. For RTVI messages, use: Copy Ask AI { "label" : "rtvi-ai" , # Required for RTVI protocol "type" : "message-type" , # Defines how the client will process it "id" : "unique-id" , # Unique identifier for the message "data" : { # Custom payload # Your custom data fields } } Example - Standard Message: Copy Ask AI import uuid # Send a standard message to all participants message_data = { "label" : "rtvi-ai" , "type" : "custom-message" , "id" : str (uuid.uuid4()), "data" : { "message" : "Hello everyone!" , "timestamp" : datetime.datetime.now().isoformat() } } await transport.send_message(TransportMessageFrame( message = message_data)) Example - Urgent Message to Specific Participant: Copy Ask AI # Send urgent message to specific participant urgent_data = { "label" : "rtvi-ai" , "type" : "server-message" , "id" : str (uuid.uuid4()), "data" : { "message" : "Private message" , "importance" : "high" } } await transport.send_message( DailyTransportMessageUrgentFrame( message = urgent_data, participant_id = "participant-123" ) ) The message will be delivered to: All participants if no specific participant_id is set Only the specified participant if participant_id is set Through the Daily app messaging system Can be received by registering an on_app_message handler Use TransportMessageUrgentFrame for high-priority messages that need immediate delivery. ​ Media Control ​ send_image async method Sends an image frame to the room. Copy Ask AI async def send_image ( frame : OutputImageRawFrame | SpriteFrame) -> None Parameters: frame : Image frame to send, either raw image data or sprite animation ​ send_audio async method Sends an audio frame to the room. Copy Ask AI async def send_audio ( frame : OutputAudioRawFrame) -> None Parameters: frame : Audio frame to send ​ Video Management ​ capture_participant_video async method Starts capturing video from a specific participant. Copy Ask AI async def capture_participant_video ( participant_id : str , framerate : int = 30 , video_source : str = "camera" , color_format : str = "RGB" ) -> None Parameters: participant_id : ID of the participant to capture framerate : Target frame rate (default: 30) video_source : Video source type (default: “camera”) color_format : Color format of the video (default: “RGB”) To request an image from the camera stream, set the framerate to 0 and push a UserImageRequestFrame . ​ Transcription Control ​ capture_participant_transcription async method Starts capturing and transcribing audio from a specific participant. Copy Ask AI async def capture_participant_transcription ( participant_id : str ) -> None Parameters: participant_id : ID of the participant to transcribe ​ update_transcription async method Updates the transcription configuration for specific participants or instances. Copy Ask AI async def update_transcription ( participants : List[ str ] | None = None , instance_id : str | None = None ) -> None Parameters: participants : Optional list of participant IDs to transcribe. If None, transcribes all participants. instance_id : Optional specific transcription instance ID to update Example: Copy Ask AI # Update transcription for specific participants await transport.update_transcription( participants = [ "participant-123" , "participant-456" ]) # Update specific transcription instance await transport.update_transcription( instance_id = "transcription-789" ) ​ Recording Control ​ start_recording async method Starts recording the room session. Copy Ask AI async def start_recording ( streaming_settings : Dict = None , stream_id : str = None , force_new : bool = None ) -> None Parameters: streaming_settings : Recording configuration settings stream_id : Optional stream identifier force_new : Force start a new recording See start_recording for more details. ​ stop_recording async method Stops an active recording. Copy Ask AI async def stop_recording ( stream_id : str = None ) -> None Parameters: stream_id : Optional stream identifier to stop specific recording See stop_recording for more details. ​ Dial-out Control ​ start_dialout async method Initiates a dial-out call. Copy Ask AI async def start_dialout ( settings : Dict = None ) -> None Parameters: settings : Dial-out configuration settings See start_dialout for more details. ​ stop_dialout async method Stops an active dial-out call. Copy Ask AI async def stop_dialout ( participant_id : str ) -> None Parameters: participant_id : ID of the participant to disconnect See stop_dialout for more details. ​ send_dtmf async method Sends DTMF tones in an existing dial-out session. Copy Ask AI async def send_dtmf ( settings : Dict = None ) -> None Parameters: settings : DTMF settings See send_dtmf for more details. ​ Subscription Management ​ update_subscriptions async method Updates subscriptions and subscription profiles. This function allows you to update subscription profiles and at the same time assign specific subscription profiles to a participant and even change specific settings for some participants. Copy Ask AI async def update_subscriptions ( participant_settings : Dict = None , profile_settings : Dict = None ) -> None Parameters: participant_settings : Per-participant subscription settings profile_settings : Global subscription profile settings See update_subscriptions for more details. ​ Participant Admin When using the DailyTransport, if your bot is given a meeting token with owner or participant admin permissions, it can remote-control other participants’ capabilities and permissions. With this control, the bot can modify settings for how others send and receive media, how they are seen in a session, and whether they have administrative controls. ​ update_remote_participants async method Updates permissions and settings for specific participants in the room. Copy Ask AI async def update_remote_participants ( remote_participants : Mapping[ str , Any] = None ) -> None Parameters: remote_participants : Dictionary mapping participant IDs to their updated settings This method allows fine-grained control over participant capabilities and permissions in the Daily room. Examples: Copy Ask AI # Mute a participant's microphone await transport.update_remote_participants({ "participant-123" : { "inputsEnabled" : { "microphone" : False } } }) # Revoke send permissions (mutes the participant and prevents them from unmuting) await transport.update_remote_participants({ "participant-123" : { "permissions" : { "canSend" : [] # Empty list revokes all send permissions } } }) # Configure selective audio routing await transport.update_remote_participants({ "participant-123" : { # the customer "permissions" : { "canReceive" : { "base" : False , # Don't receive from anyone by default "byUserId" : { "hold-music" : True # Only receive from hold music player } } } } }) # Set up user-to-user permissions mapping (e.g., when transferring calls) # Assumes customer was previously unable to hear anyone (they were on hold) and agent was speaking to bot await transport.update_remote_participants({ "participant-123" : { # the customer "permissions" : { "base" : False , "canReceive" : { "byUserId" : { "agent" : True }} } }, "participant-456" : { # the agent "permissions" : { "base" : False , "canReceive" : { "byUserId" : { "customer" : True }} } } }) # Promote a participant to be a participant admin await transport.update_remote_participants({ "participant-123" : { "permissions" : { "canAdmin" : [ "participants" ] } } }) Permissions Settings: For comprehensive details on permissions settings, see the Daily API documentation: daily-python’s update_remote_participants() documentation daily-js’s updateParticipant() -> updatePermission and participants().permissions ​ Chat messages ​ send_prebuilt_chat_message async method When testing with Daily Prebuilt, you can send messages to the Prebuilt chat using this method. Parameters: message (str): The chat message to send user_name (str): The name of the user sending the message See send_prebuilt_chat_message for more details. ​ Properties ​ participant_id string Returns the transport’s participant ID in the room. ​ room_url string Returns the transport’s room URL for the room currently in use. ​ Notes Supports real-time audio/video communication Handles WebRTC connection management Provides voice activity detection Includes transcription capabilities Manages participant interactions Supports recording and streaming Thread-safe processing All callbacks are asynchronous Multiple handlers can be registered for the same event Exceptions in callbacks are logged but don’t stop the transport Supported Services SmallWebRTCTransport On this page Overview Installation Configuration Constructor Parameters DailyParams Configuration Audio Output Configuration Audio Input Configuration Video Output Configuration Voice Activity Detection (VAD) Feature Settings Basic Usage Event Callbacks Connection Events Participant Events Communication Events Dial Events Dial-in Dial-out Transcription Events Recording Events Error Events Notes on Callbacks Frame Types Input Frames Output Frames Methods Room Management Media Control Video Management Transcription Control Recording Control Dial-out Control Subscription Management Participant Admin Chat messages Properties Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/transport_daily_b514c0b0.txt b/transport_daily_b514c0b0.txt
new file mode 100644
index 0000000000000000000000000000000000000000..9fd1c824bfe5b424b82503ec70fbf2b86282780d
--- /dev/null
+++ b/transport_daily_b514c0b0.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/transport/daily#param-audio-out-bitrate
+Title: Daily WebRTC - Pipecat
+==================================================
+
+Daily WebRTC - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Transport Daily WebRTC Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Daily WebRTC SmallWebRTCTransport FastAPI WebSocket WebSocket Server Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview DailyTransport provides real-time audio and video communication capabilities using Daily’s WebRTC platform. It supports bidirectional audio/video streams, transcription, voice activity detection (VAD), and participant management. ​ Installation To use DailyTransport , install the required dependencies: Copy Ask AI pip install "pipecat-ai[daily]" You’ll also need to set up your Daily API key as an environment variable: DAILY_API_KEY . You can obtain a Daily API key by signing up at Daily . ​ Configuration ​ Constructor Parameters ​ room_url str required Daily room URL ​ token str | None Daily room token ​ bot_name str required Name of the bot in the room ​ params DailyParams default: "DailyParams()" Transport configuration parameters ​ DailyParams Configuration ​ api_url str default: "https://api.daily.co/v1" Daily API endpoint URL ​ api_key str default: "" Daily API key for authentication ​ Audio Output Configuration ​ audio_out_enabled bool default: "False" Enable audio output capabilities ​ audio_out_is_live bool default: "False" Enable live audio streaming mode ​ audio_out_sample_rate int default: "None" Audio output sample rate in Hz ​ audio_out_channels int default: "1" Number of audio output channels ​ audio_out_bitrate int default: "96000" Audio output bitrate in bits per second ​ Audio Input Configuration ​ audio_in_enabled bool default: "False" Enable audio input capabilities ​ audio_in_passthrough bool default: "False" When enabled, incoming audio frames are pushed downstream ​ audio_in_sample_rate int default: "None" Audio input sample rate in Hz ​ audio_in_channels int default: "1" Number of audio input channels ​ audio_in_filter Optional[BaseAudioFilter] default: "None" Audio filter for input processing. Supported filters are: KrispFilter() and NoisereduceFilter() . See the KrispFilter() reference docs and NoisereduceFilter() reference docs ` for more information. ​ Video Output Configuration ​ video_out_enabled bool default: "False" Enable video output capabilities ​ video_out_is_live bool default: "False" Enable live video streaming mode ​ video_out_width int default: "1024" Video output width in pixels ​ video_out_height int default: "768" Video output height in pixels ​ video_out_bitrate int default: "800000" Video output bitrate in bits per second ​ video_out_framerate int default: "30" Video output frame rate ​ video_out_color_format str default: "RGB" Video color format (RGB, BGR, etc.) ​ Voice Activity Detection (VAD) ​ vad_analyzer VADAnalyzer | None default: "None" Voice Activity Detection analyzer. You can set this to either SileroVADAnalyzer() or WebRTCVADAnalyzer() . SileroVADAnalyzer is the recommended option. Learn more about the SileroVADAnalyzer . ​ Feature Settings ​ dialin_settings Optional[DailyDialinSettings] default: "None" Configuration for dial-in functionality ​ transcription_enabled bool default: "False" Enable real-time transcription ​ transcription_settings DailyTranscriptionSettings Configuration for transcription features Copy Ask AI class DailyTranscriptionSettings ( BaseModel ): language: str = "en" # Default language model: str = "nova-2-general" # Transcription model profanity_filter: bool = True # Filter profanity redact: bool = False # Redact sensitive information endpointing: bool = True # Enable speech endpointing punctuate: bool = True # Add punctuation includeRawResponse: bool = True # Include raw API response extra: Mapping[ str , Any] = { "interim_results" : True # Provide any Deepgram-specific settings } ​ Basic Usage Copy Ask AI from pipecat.transports.services.daily import DailyTransport, DailyParams from pipecat.audio.vad.silero import SileroVADAnalyzer # Configure transport transport = DailyTransport( room_url = "https://your-domain.daily.co/room-name" , token = "your-room-token" , bot_name = "AI Assistant" , params = DailyParams( audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer( params = VADParams( stop_secs = 0.5 )), ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), # Handle incoming audio/video stt, # Speech-to-text llm, # Language model tts, # Text-to-speech transport.output() # Handle outgoing audio/video ]) # Use transport event handlers to manage the call lifecycle # Event handler for when the user joins the room: @transport.event_handler ( "on_first_participant_joined" ) async def on_first_participant_joined ( transport , participant ): # Start transcription for the user await transport.capture_participant_transcription(participant[ "id" ]) # Kick off the conversation. await task.queue_frames([context_aggregator.user().get_context_frame()]) # Event handler for when the user leaves the room: @transport.event_handler ( "on_participant_left" ) async def on_participant_left ( transport , participant , reason ): # Cancel the pipeline, which stops processing and removes the bot from the room await task.cancel() ​ Event Callbacks DailyTransport provides a comprehensive callback system for handling various events. Register callbacks using the @transport.event_handler() decorator. ​ Connection Events ​ on_joined async callback Called when the bot successfully joins the room. Parameters: transport : The DailyTransport instance data : Dictionary containing room join information Copy Ask AI @transport.event_handler ( "on_joined" ) async def on_joined ( transport , data : Dict[ str , Any]): logger.info( f "Joined room with data: { data } " ) ​ on_left async callback Called when the bot leaves the room. Parameters: transport : The DailyTransport instance Copy Ask AI @transport.event_handler ( "on_left" ) async def on_left ( transport ): logger.info( "Left room" ) ​ Participant Events ​ on_first_participant_joined async callback Called when the first participant joins the room. Useful for initializing conversations. Parameters: transport : The DailyTransport instance participant : Dictionary containing participant information Copy Ask AI @transport.event_handler ( "on_first_participant_joined" ) async def on_first_participant_joined ( transport , participant : Dict[ str , Any]): await transport.capture_participant_transcription(participant[ "id" ]) await task.queue_frames([LLMMessagesFrame(initial_messages)]) ​ on_participant_joined async callback Called when any participant joins the room. Parameters: transport : The DailyTransport instance participant : Dictionary containing participant information Copy Ask AI @transport.event_handler ( "on_participant_joined" ) async def on_participant_joined ( transport , participant : Dict[ str , Any]): logger.info( f "Participant joined: { participant[ 'id' ] } " ) ​ on_participant_left async callback Called when a participant leaves the room. Parameters: transport : The DailyTransport instance participant : Dictionary containing participant information reason : String describing why the participant left, “leftCall” | “hidden” Copy Ask AI @transport.event_handler ( "on_participant_left" ) async def on_participant_left ( transport , participant : Dict[ str , Any], reason : str ): logger.info( f "Participant { participant[ 'id' ] } left: { reason } " ) # Cancel the pipeline task to stop processing and remove the bot from the Daily room await task.cancel() ​ on_participant_updated async callback Event emitted when a participant is updated. This can mean either the participant’s metadata was updated, or the tracks belonging to the participant changed. Parameters: transport : The DailyTransport instance participant : Dictionary containing updated participant information Copy Ask AI @transport.event_handler ( "on_participant_updated" ) async def on_participant_updated ( transport , participant : Dict[ str , Any]): logger.info( f "Participant updated: { participant } " ) ​ Communication Events ​ on_app_message async callback Event emitted when a custom app message is received from another participant or via the REST API. Parameters: transport : The DailyTransport instance message : The message content (any type) sender : String identifier of the message sender Copy Ask AI @transport.event_handler ( "on_app_message" ) async def on_app_message ( transport , message : Any, sender : str ): logger.info( f "Message from { sender } : { message } " ) ​ on_call_state_updated async callback Event emitted when the call state changes, normally as a consequence of joining or leaving the call. Parameters: transport : The DailyTransport instance state : String representing the new call state. Learn more about call states . Copy Ask AI @transport.event_handler ( "on_call_state_updated" ) async def on_call_state_updated ( transport , state : str ): logger.info( f "Call state updated: { state } " ) ​ Dial Events ​ Dial-in ​ on_dialin_ready async callback Event emitted when dial-in is ready. This happens after the room has connected to the SIP endpoint and the system is ready to receive dial-in calls. Parameters: transport : The DailyTransport instance sip_endpoint : String containing the SIP endpoint information Copy Ask AI @transport.event_handler ( "on_dialin_ready" ) async def on_dialin_ready ( transport , sip_endpoint : str ): logger.info( f "Dial-in ready at: { sip_endpoint } " ) ​ on_dialin_connected async callback Event emitted when the session with the dial-in remote end is established (i.e. SIP endpoint or PSTN are connectd to the Daily room). Note: connected does not mean media (audio or video) has started flowing between the room and PSTN, it means the room received the connection request and both endpoints are negotiating the media flow. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. See DialinConnectedEvent . Copy Ask AI @transport.event_handler ( "on_dialin_connected" ) async def on_dialin_connected ( transport , data : Any): logger.info( f "Dial-in connected: { data } " ) ​ on_dialin_stopped async callback Event emitted when the dial-in remote end disconnects the call. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. See DialinStoppedEvent . Copy Ask AI @transport.event_handler ( "on_dialin_stopped" ) async def on_dialin_stopped ( transport , data : Any): logger.info( f "Dial-in stopped: { data } " ) ​ on_dialin_error async callback Event emitted in the case of dial-in errors which are fatal and the service cannot proceed. For example, an error in SDP negotiation is fatal to the media/SIP pipeline and will result in dialin-error being triggered. Parameters: transport : The DailyTransport instance data : Dictionary containing error information. See DialinEvent . Copy Ask AI @transport.event_handler ( "on_dialin_error" ) async def on_dialin_error ( transport , data : Any): logger.error( f "Dial-in error: { data } " ) ​ on_dialin_warning async callback Event emitted there is a dial-in non-fatal error, such as the selected codec not being used and a fallback codec being utilized. Parameters: transport : The DailyTransport instance data : Dictionary containing warning information. See DialinEvent . Copy Ask AI @transport.event_handler ( "on_dialin_warning" ) async def on_dialin_warning ( transport , data : Any): logger.warning( f "Dial-in warning: { data } " ) ​ Dial-out ​ on_dialout_answered async callback Event emitted when the session with the dial-out remote end is answered. Parameters: transport : The DailyTransport instance data : Dictionary containing call information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_answered" ) async def on_dialout_answered ( transport , data : Any): logger.info( f "Dial-out answered: { data } " ) ​ on_dialout_connected async callback Event emitted when the session with the dial-out remote end is established. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_connected" ) async def on_dialout_connected ( transport , data : Any): logger.info( f "Dial-out connected: { data } " ) ​ on_dialout_stopped async callback Event emitted when the dial-out session is stopped. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_stopped" ) async def on_dialout_stopped ( transport , data : Any): logger.info( f "Dial-out stopped: { data } " ) ​ on_dialout_error async callback Event emitted in the case of dial-out errors which are fatal and the service cannot proceed. For example, an error in SDP negotiation is fatal to the media/SIP pipeline and will result in dialout-error being triggered. Parameters: transport : The DailyTransport instance data : Dictionary containing error information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_error" ) async def on_dialout_error ( transport , data : Any): logger.error( f "Dial-out error: { data } " ) ​ on_dialout_warning async callback Event emitted there is a dial-out non-fatal error, such as the selected codec not being used and a fallback codec being utilized. Parameters: transport : The DailyTransport instance data : Dictionary containing warning information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_warning" ) async def on_dialout_warning ( transport , data : Any): logger.warning( f "Dial-out warning: { data } " ) ​ Transcription Events ​ on_transcription_message async callback Called when a transcription message is received. This includes both interim and final transcriptions. Parameters: transport : The DailyTransport instance message : Dictionary containing transcription data including. Learn more . Copy Ask AI @transport.event_handler ( "on_transcription_message" ) async def on_transcription_message ( transport , message : Dict[ str , Any]): participant_id = message.get( "participantId" ) text = message.get( "text" ) is_final = message[ "rawResponse" ][ "is_final" ] logger.info( f "Transcription from { participant_id } : { text } (final: { is_final } )" ) ​ Recording Events ​ on_recording_started async callback Called when a room recording starts successfully. Parameters: transport : The DailyTransport instance status : Dictionary containing recording status information. Learn more . Copy Ask AI @transport.event_handler ( "on_recording_started" ) async def on_recording_started ( transport , status : Dict[ str , Any]): logger.info( f "Recording started with status: { status } " ) ​ on_recording_stopped async callback Called when a room recording stops. Parameters: transport : The DailyTransport instance stream_id : String identifier of the stopped recording stream Copy Ask AI @transport.event_handler ( "on_recording_stopped" ) async def on_recording_stopped ( transport , stream_id : str ): logger.info( f "Recording stopped for stream: { stream_id } " ) ​ on_recording_error async callback Called when an error occurs during recording. Parameters: transport : The DailyTransport instance stream_id : String identifier of the recording stream message : Error message describing what went wrong Copy Ask AI @transport.event_handler ( "on_recording_error" ) async def on_recording_error ( transport , stream_id : str , message : str ): logger.error( f "Recording error for stream { stream_id } : { message } " ) ​ Error Events ​ on_error async callback Called when a transport error occurs. Parameters: transport : The DailyTransport instance error : String containing error details Copy Ask AI @transport.event_handler ( "on_error" ) async def on_error ( transport , error : str ): logger.error( f "Transport error: { error } " ) ​ Notes on Callbacks All callbacks are asynchronous Callbacks are executed in the order they were registered Multiple handlers can be registered for the same event Exceptions in callbacks are logged but don’t stop the transport Callbacks should be lightweight to avoid blocking the event loop Heavy processing should be offloaded to separate tasks ​ Frame Types ​ Input Frames ​ InputAudioRawFrame Frame Raw audio data from participants ​ UserImageRawFrame Frame Video frames from participants ​ Output Frames ​ OutputAudioRawFrame Frame Audio data to be sent ​ OutputImageRawFrame Frame Video frames to be sent ​ Methods ​ Room Management ​ participants method Returns a dictionary of all participants currently in the room. Copy Ask AI def participants () -> Dict[ str , Any] See participants for more details. ​ participant_counts method Returns participant count statistics for the room. Copy Ask AI def participant_counts () -> Dict[ str , int ] See participant_counts for more details. ​ send_message async method Sends a message to participants in the Daily room. Messages can be directed to all participants or targeted to a specific participant. Copy Ask AI async def send_message ( frame : TransportMessageFrame | TransportMessageUrgentFrame) Parameters: frame : A transport message frame containing the message to send. Can be either: TransportMessageFrame : Standard message TransportMessageUrgentFrame : High-priority message Message Structure: Messages should conform to the expected format for the receiving system. For RTVI messages, use: Copy Ask AI { "label" : "rtvi-ai" , # Required for RTVI protocol "type" : "message-type" , # Defines how the client will process it "id" : "unique-id" , # Unique identifier for the message "data" : { # Custom payload # Your custom data fields } } Example - Standard Message: Copy Ask AI import uuid # Send a standard message to all participants message_data = { "label" : "rtvi-ai" , "type" : "custom-message" , "id" : str (uuid.uuid4()), "data" : { "message" : "Hello everyone!" , "timestamp" : datetime.datetime.now().isoformat() } } await transport.send_message(TransportMessageFrame( message = message_data)) Example - Urgent Message to Specific Participant: Copy Ask AI # Send urgent message to specific participant urgent_data = { "label" : "rtvi-ai" , "type" : "server-message" , "id" : str (uuid.uuid4()), "data" : { "message" : "Private message" , "importance" : "high" } } await transport.send_message( DailyTransportMessageUrgentFrame( message = urgent_data, participant_id = "participant-123" ) ) The message will be delivered to: All participants if no specific participant_id is set Only the specified participant if participant_id is set Through the Daily app messaging system Can be received by registering an on_app_message handler Use TransportMessageUrgentFrame for high-priority messages that need immediate delivery. ​ Media Control ​ send_image async method Sends an image frame to the room. Copy Ask AI async def send_image ( frame : OutputImageRawFrame | SpriteFrame) -> None Parameters: frame : Image frame to send, either raw image data or sprite animation ​ send_audio async method Sends an audio frame to the room. Copy Ask AI async def send_audio ( frame : OutputAudioRawFrame) -> None Parameters: frame : Audio frame to send ​ Video Management ​ capture_participant_video async method Starts capturing video from a specific participant. Copy Ask AI async def capture_participant_video ( participant_id : str , framerate : int = 30 , video_source : str = "camera" , color_format : str = "RGB" ) -> None Parameters: participant_id : ID of the participant to capture framerate : Target frame rate (default: 30) video_source : Video source type (default: “camera”) color_format : Color format of the video (default: “RGB”) To request an image from the camera stream, set the framerate to 0 and push a UserImageRequestFrame . ​ Transcription Control ​ capture_participant_transcription async method Starts capturing and transcribing audio from a specific participant. Copy Ask AI async def capture_participant_transcription ( participant_id : str ) -> None Parameters: participant_id : ID of the participant to transcribe ​ update_transcription async method Updates the transcription configuration for specific participants or instances. Copy Ask AI async def update_transcription ( participants : List[ str ] | None = None , instance_id : str | None = None ) -> None Parameters: participants : Optional list of participant IDs to transcribe. If None, transcribes all participants. instance_id : Optional specific transcription instance ID to update Example: Copy Ask AI # Update transcription for specific participants await transport.update_transcription( participants = [ "participant-123" , "participant-456" ]) # Update specific transcription instance await transport.update_transcription( instance_id = "transcription-789" ) ​ Recording Control ​ start_recording async method Starts recording the room session. Copy Ask AI async def start_recording ( streaming_settings : Dict = None , stream_id : str = None , force_new : bool = None ) -> None Parameters: streaming_settings : Recording configuration settings stream_id : Optional stream identifier force_new : Force start a new recording See start_recording for more details. ​ stop_recording async method Stops an active recording. Copy Ask AI async def stop_recording ( stream_id : str = None ) -> None Parameters: stream_id : Optional stream identifier to stop specific recording See stop_recording for more details. ​ Dial-out Control ​ start_dialout async method Initiates a dial-out call. Copy Ask AI async def start_dialout ( settings : Dict = None ) -> None Parameters: settings : Dial-out configuration settings See start_dialout for more details. ​ stop_dialout async method Stops an active dial-out call. Copy Ask AI async def stop_dialout ( participant_id : str ) -> None Parameters: participant_id : ID of the participant to disconnect See stop_dialout for more details. ​ send_dtmf async method Sends DTMF tones in an existing dial-out session. Copy Ask AI async def send_dtmf ( settings : Dict = None ) -> None Parameters: settings : DTMF settings See send_dtmf for more details. ​ Subscription Management ​ update_subscriptions async method Updates subscriptions and subscription profiles. This function allows you to update subscription profiles and at the same time assign specific subscription profiles to a participant and even change specific settings for some participants. Copy Ask AI async def update_subscriptions ( participant_settings : Dict = None , profile_settings : Dict = None ) -> None Parameters: participant_settings : Per-participant subscription settings profile_settings : Global subscription profile settings See update_subscriptions for more details. ​ Participant Admin When using the DailyTransport, if your bot is given a meeting token with owner or participant admin permissions, it can remote-control other participants’ capabilities and permissions. With this control, the bot can modify settings for how others send and receive media, how they are seen in a session, and whether they have administrative controls. ​ update_remote_participants async method Updates permissions and settings for specific participants in the room. Copy Ask AI async def update_remote_participants ( remote_participants : Mapping[ str , Any] = None ) -> None Parameters: remote_participants : Dictionary mapping participant IDs to their updated settings This method allows fine-grained control over participant capabilities and permissions in the Daily room. Examples: Copy Ask AI # Mute a participant's microphone await transport.update_remote_participants({ "participant-123" : { "inputsEnabled" : { "microphone" : False } } }) # Revoke send permissions (mutes the participant and prevents them from unmuting) await transport.update_remote_participants({ "participant-123" : { "permissions" : { "canSend" : [] # Empty list revokes all send permissions } } }) # Configure selective audio routing await transport.update_remote_participants({ "participant-123" : { # the customer "permissions" : { "canReceive" : { "base" : False , # Don't receive from anyone by default "byUserId" : { "hold-music" : True # Only receive from hold music player } } } } }) # Set up user-to-user permissions mapping (e.g., when transferring calls) # Assumes customer was previously unable to hear anyone (they were on hold) and agent was speaking to bot await transport.update_remote_participants({ "participant-123" : { # the customer "permissions" : { "base" : False , "canReceive" : { "byUserId" : { "agent" : True }} } }, "participant-456" : { # the agent "permissions" : { "base" : False , "canReceive" : { "byUserId" : { "customer" : True }} } } }) # Promote a participant to be a participant admin await transport.update_remote_participants({ "participant-123" : { "permissions" : { "canAdmin" : [ "participants" ] } } }) Permissions Settings: For comprehensive details on permissions settings, see the Daily API documentation: daily-python’s update_remote_participants() documentation daily-js’s updateParticipant() -> updatePermission and participants().permissions ​ Chat messages ​ send_prebuilt_chat_message async method When testing with Daily Prebuilt, you can send messages to the Prebuilt chat using this method. Parameters: message (str): The chat message to send user_name (str): The name of the user sending the message See send_prebuilt_chat_message for more details. ​ Properties ​ participant_id string Returns the transport’s participant ID in the room. ​ room_url string Returns the transport’s room URL for the room currently in use. ​ Notes Supports real-time audio/video communication Handles WebRTC connection management Provides voice activity detection Includes transcription capabilities Manages participant interactions Supports recording and streaming Thread-safe processing All callbacks are asynchronous Multiple handlers can be registered for the same event Exceptions in callbacks are logged but don’t stop the transport Supported Services SmallWebRTCTransport On this page Overview Installation Configuration Constructor Parameters DailyParams Configuration Audio Output Configuration Audio Input Configuration Video Output Configuration Voice Activity Detection (VAD) Feature Settings Basic Usage Event Callbacks Connection Events Participant Events Communication Events Dial Events Dial-in Dial-out Transcription Events Recording Events Error Events Notes on Callbacks Frame Types Input Frames Output Frames Methods Room Management Media Control Video Management Transcription Control Recording Control Dial-out Control Subscription Management Participant Admin Chat messages Properties Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/transport_daily_b9d063f6.txt b/transport_daily_b9d063f6.txt
new file mode 100644
index 0000000000000000000000000000000000000000..a61864426fc91a37f5565d6e8d336223df660457
--- /dev/null
+++ b/transport_daily_b9d063f6.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/transport/daily#param-on-dialout-stopped
+Title: Daily WebRTC - Pipecat
+==================================================
+
+Daily WebRTC - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Transport Daily WebRTC Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Daily WebRTC SmallWebRTCTransport FastAPI WebSocket WebSocket Server Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview DailyTransport provides real-time audio and video communication capabilities using Daily’s WebRTC platform. It supports bidirectional audio/video streams, transcription, voice activity detection (VAD), and participant management. ​ Installation To use DailyTransport , install the required dependencies: Copy Ask AI pip install "pipecat-ai[daily]" You’ll also need to set up your Daily API key as an environment variable: DAILY_API_KEY . You can obtain a Daily API key by signing up at Daily . ​ Configuration ​ Constructor Parameters ​ room_url str required Daily room URL ​ token str | None Daily room token ​ bot_name str required Name of the bot in the room ​ params DailyParams default: "DailyParams()" Transport configuration parameters ​ DailyParams Configuration ​ api_url str default: "https://api.daily.co/v1" Daily API endpoint URL ​ api_key str default: "" Daily API key for authentication ​ Audio Output Configuration ​ audio_out_enabled bool default: "False" Enable audio output capabilities ​ audio_out_is_live bool default: "False" Enable live audio streaming mode ​ audio_out_sample_rate int default: "None" Audio output sample rate in Hz ​ audio_out_channels int default: "1" Number of audio output channels ​ audio_out_bitrate int default: "96000" Audio output bitrate in bits per second ​ Audio Input Configuration ​ audio_in_enabled bool default: "False" Enable audio input capabilities ​ audio_in_passthrough bool default: "False" When enabled, incoming audio frames are pushed downstream ​ audio_in_sample_rate int default: "None" Audio input sample rate in Hz ​ audio_in_channels int default: "1" Number of audio input channels ​ audio_in_filter Optional[BaseAudioFilter] default: "None" Audio filter for input processing. Supported filters are: KrispFilter() and NoisereduceFilter() . See the KrispFilter() reference docs and NoisereduceFilter() reference docs ` for more information. ​ Video Output Configuration ​ video_out_enabled bool default: "False" Enable video output capabilities ​ video_out_is_live bool default: "False" Enable live video streaming mode ​ video_out_width int default: "1024" Video output width in pixels ​ video_out_height int default: "768" Video output height in pixels ​ video_out_bitrate int default: "800000" Video output bitrate in bits per second ​ video_out_framerate int default: "30" Video output frame rate ​ video_out_color_format str default: "RGB" Video color format (RGB, BGR, etc.) ​ Voice Activity Detection (VAD) ​ vad_analyzer VADAnalyzer | None default: "None" Voice Activity Detection analyzer. You can set this to either SileroVADAnalyzer() or WebRTCVADAnalyzer() . SileroVADAnalyzer is the recommended option. Learn more about the SileroVADAnalyzer . ​ Feature Settings ​ dialin_settings Optional[DailyDialinSettings] default: "None" Configuration for dial-in functionality ​ transcription_enabled bool default: "False" Enable real-time transcription ​ transcription_settings DailyTranscriptionSettings Configuration for transcription features Copy Ask AI class DailyTranscriptionSettings ( BaseModel ): language: str = "en" # Default language model: str = "nova-2-general" # Transcription model profanity_filter: bool = True # Filter profanity redact: bool = False # Redact sensitive information endpointing: bool = True # Enable speech endpointing punctuate: bool = True # Add punctuation includeRawResponse: bool = True # Include raw API response extra: Mapping[ str , Any] = { "interim_results" : True # Provide any Deepgram-specific settings } ​ Basic Usage Copy Ask AI from pipecat.transports.services.daily import DailyTransport, DailyParams from pipecat.audio.vad.silero import SileroVADAnalyzer # Configure transport transport = DailyTransport( room_url = "https://your-domain.daily.co/room-name" , token = "your-room-token" , bot_name = "AI Assistant" , params = DailyParams( audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer( params = VADParams( stop_secs = 0.5 )), ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), # Handle incoming audio/video stt, # Speech-to-text llm, # Language model tts, # Text-to-speech transport.output() # Handle outgoing audio/video ]) # Use transport event handlers to manage the call lifecycle # Event handler for when the user joins the room: @transport.event_handler ( "on_first_participant_joined" ) async def on_first_participant_joined ( transport , participant ): # Start transcription for the user await transport.capture_participant_transcription(participant[ "id" ]) # Kick off the conversation. await task.queue_frames([context_aggregator.user().get_context_frame()]) # Event handler for when the user leaves the room: @transport.event_handler ( "on_participant_left" ) async def on_participant_left ( transport , participant , reason ): # Cancel the pipeline, which stops processing and removes the bot from the room await task.cancel() ​ Event Callbacks DailyTransport provides a comprehensive callback system for handling various events. Register callbacks using the @transport.event_handler() decorator. ​ Connection Events ​ on_joined async callback Called when the bot successfully joins the room. Parameters: transport : The DailyTransport instance data : Dictionary containing room join information Copy Ask AI @transport.event_handler ( "on_joined" ) async def on_joined ( transport , data : Dict[ str , Any]): logger.info( f "Joined room with data: { data } " ) ​ on_left async callback Called when the bot leaves the room. Parameters: transport : The DailyTransport instance Copy Ask AI @transport.event_handler ( "on_left" ) async def on_left ( transport ): logger.info( "Left room" ) ​ Participant Events ​ on_first_participant_joined async callback Called when the first participant joins the room. Useful for initializing conversations. Parameters: transport : The DailyTransport instance participant : Dictionary containing participant information Copy Ask AI @transport.event_handler ( "on_first_participant_joined" ) async def on_first_participant_joined ( transport , participant : Dict[ str , Any]): await transport.capture_participant_transcription(participant[ "id" ]) await task.queue_frames([LLMMessagesFrame(initial_messages)]) ​ on_participant_joined async callback Called when any participant joins the room. Parameters: transport : The DailyTransport instance participant : Dictionary containing participant information Copy Ask AI @transport.event_handler ( "on_participant_joined" ) async def on_participant_joined ( transport , participant : Dict[ str , Any]): logger.info( f "Participant joined: { participant[ 'id' ] } " ) ​ on_participant_left async callback Called when a participant leaves the room. Parameters: transport : The DailyTransport instance participant : Dictionary containing participant information reason : String describing why the participant left, “leftCall” | “hidden” Copy Ask AI @transport.event_handler ( "on_participant_left" ) async def on_participant_left ( transport , participant : Dict[ str , Any], reason : str ): logger.info( f "Participant { participant[ 'id' ] } left: { reason } " ) # Cancel the pipeline task to stop processing and remove the bot from the Daily room await task.cancel() ​ on_participant_updated async callback Event emitted when a participant is updated. This can mean either the participant’s metadata was updated, or the tracks belonging to the participant changed. Parameters: transport : The DailyTransport instance participant : Dictionary containing updated participant information Copy Ask AI @transport.event_handler ( "on_participant_updated" ) async def on_participant_updated ( transport , participant : Dict[ str , Any]): logger.info( f "Participant updated: { participant } " ) ​ Communication Events ​ on_app_message async callback Event emitted when a custom app message is received from another participant or via the REST API. Parameters: transport : The DailyTransport instance message : The message content (any type) sender : String identifier of the message sender Copy Ask AI @transport.event_handler ( "on_app_message" ) async def on_app_message ( transport , message : Any, sender : str ): logger.info( f "Message from { sender } : { message } " ) ​ on_call_state_updated async callback Event emitted when the call state changes, normally as a consequence of joining or leaving the call. Parameters: transport : The DailyTransport instance state : String representing the new call state. Learn more about call states . Copy Ask AI @transport.event_handler ( "on_call_state_updated" ) async def on_call_state_updated ( transport , state : str ): logger.info( f "Call state updated: { state } " ) ​ Dial Events ​ Dial-in ​ on_dialin_ready async callback Event emitted when dial-in is ready. This happens after the room has connected to the SIP endpoint and the system is ready to receive dial-in calls. Parameters: transport : The DailyTransport instance sip_endpoint : String containing the SIP endpoint information Copy Ask AI @transport.event_handler ( "on_dialin_ready" ) async def on_dialin_ready ( transport , sip_endpoint : str ): logger.info( f "Dial-in ready at: { sip_endpoint } " ) ​ on_dialin_connected async callback Event emitted when the session with the dial-in remote end is established (i.e. SIP endpoint or PSTN are connectd to the Daily room). Note: connected does not mean media (audio or video) has started flowing between the room and PSTN, it means the room received the connection request and both endpoints are negotiating the media flow. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. See DialinConnectedEvent . Copy Ask AI @transport.event_handler ( "on_dialin_connected" ) async def on_dialin_connected ( transport , data : Any): logger.info( f "Dial-in connected: { data } " ) ​ on_dialin_stopped async callback Event emitted when the dial-in remote end disconnects the call. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. See DialinStoppedEvent . Copy Ask AI @transport.event_handler ( "on_dialin_stopped" ) async def on_dialin_stopped ( transport , data : Any): logger.info( f "Dial-in stopped: { data } " ) ​ on_dialin_error async callback Event emitted in the case of dial-in errors which are fatal and the service cannot proceed. For example, an error in SDP negotiation is fatal to the media/SIP pipeline and will result in dialin-error being triggered. Parameters: transport : The DailyTransport instance data : Dictionary containing error information. See DialinEvent . Copy Ask AI @transport.event_handler ( "on_dialin_error" ) async def on_dialin_error ( transport , data : Any): logger.error( f "Dial-in error: { data } " ) ​ on_dialin_warning async callback Event emitted there is a dial-in non-fatal error, such as the selected codec not being used and a fallback codec being utilized. Parameters: transport : The DailyTransport instance data : Dictionary containing warning information. See DialinEvent . Copy Ask AI @transport.event_handler ( "on_dialin_warning" ) async def on_dialin_warning ( transport , data : Any): logger.warning( f "Dial-in warning: { data } " ) ​ Dial-out ​ on_dialout_answered async callback Event emitted when the session with the dial-out remote end is answered. Parameters: transport : The DailyTransport instance data : Dictionary containing call information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_answered" ) async def on_dialout_answered ( transport , data : Any): logger.info( f "Dial-out answered: { data } " ) ​ on_dialout_connected async callback Event emitted when the session with the dial-out remote end is established. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_connected" ) async def on_dialout_connected ( transport , data : Any): logger.info( f "Dial-out connected: { data } " ) ​ on_dialout_stopped async callback Event emitted when the dial-out session is stopped. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_stopped" ) async def on_dialout_stopped ( transport , data : Any): logger.info( f "Dial-out stopped: { data } " ) ​ on_dialout_error async callback Event emitted in the case of dial-out errors which are fatal and the service cannot proceed. For example, an error in SDP negotiation is fatal to the media/SIP pipeline and will result in dialout-error being triggered. Parameters: transport : The DailyTransport instance data : Dictionary containing error information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_error" ) async def on_dialout_error ( transport , data : Any): logger.error( f "Dial-out error: { data } " ) ​ on_dialout_warning async callback Event emitted there is a dial-out non-fatal error, such as the selected codec not being used and a fallback codec being utilized. Parameters: transport : The DailyTransport instance data : Dictionary containing warning information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_warning" ) async def on_dialout_warning ( transport , data : Any): logger.warning( f "Dial-out warning: { data } " ) ​ Transcription Events ​ on_transcription_message async callback Called when a transcription message is received. This includes both interim and final transcriptions. Parameters: transport : The DailyTransport instance message : Dictionary containing transcription data including. Learn more . Copy Ask AI @transport.event_handler ( "on_transcription_message" ) async def on_transcription_message ( transport , message : Dict[ str , Any]): participant_id = message.get( "participantId" ) text = message.get( "text" ) is_final = message[ "rawResponse" ][ "is_final" ] logger.info( f "Transcription from { participant_id } : { text } (final: { is_final } )" ) ​ Recording Events ​ on_recording_started async callback Called when a room recording starts successfully. Parameters: transport : The DailyTransport instance status : Dictionary containing recording status information. Learn more . Copy Ask AI @transport.event_handler ( "on_recording_started" ) async def on_recording_started ( transport , status : Dict[ str , Any]): logger.info( f "Recording started with status: { status } " ) ​ on_recording_stopped async callback Called when a room recording stops. Parameters: transport : The DailyTransport instance stream_id : String identifier of the stopped recording stream Copy Ask AI @transport.event_handler ( "on_recording_stopped" ) async def on_recording_stopped ( transport , stream_id : str ): logger.info( f "Recording stopped for stream: { stream_id } " ) ​ on_recording_error async callback Called when an error occurs during recording. Parameters: transport : The DailyTransport instance stream_id : String identifier of the recording stream message : Error message describing what went wrong Copy Ask AI @transport.event_handler ( "on_recording_error" ) async def on_recording_error ( transport , stream_id : str , message : str ): logger.error( f "Recording error for stream { stream_id } : { message } " ) ​ Error Events ​ on_error async callback Called when a transport error occurs. Parameters: transport : The DailyTransport instance error : String containing error details Copy Ask AI @transport.event_handler ( "on_error" ) async def on_error ( transport , error : str ): logger.error( f "Transport error: { error } " ) ​ Notes on Callbacks All callbacks are asynchronous Callbacks are executed in the order they were registered Multiple handlers can be registered for the same event Exceptions in callbacks are logged but don’t stop the transport Callbacks should be lightweight to avoid blocking the event loop Heavy processing should be offloaded to separate tasks ​ Frame Types ​ Input Frames ​ InputAudioRawFrame Frame Raw audio data from participants ​ UserImageRawFrame Frame Video frames from participants ​ Output Frames ​ OutputAudioRawFrame Frame Audio data to be sent ​ OutputImageRawFrame Frame Video frames to be sent ​ Methods ​ Room Management ​ participants method Returns a dictionary of all participants currently in the room. Copy Ask AI def participants () -> Dict[ str , Any] See participants for more details. ​ participant_counts method Returns participant count statistics for the room. Copy Ask AI def participant_counts () -> Dict[ str , int ] See participant_counts for more details. ​ send_message async method Sends a message to participants in the Daily room. Messages can be directed to all participants or targeted to a specific participant. Copy Ask AI async def send_message ( frame : TransportMessageFrame | TransportMessageUrgentFrame) Parameters: frame : A transport message frame containing the message to send. Can be either: TransportMessageFrame : Standard message TransportMessageUrgentFrame : High-priority message Message Structure: Messages should conform to the expected format for the receiving system. For RTVI messages, use: Copy Ask AI { "label" : "rtvi-ai" , # Required for RTVI protocol "type" : "message-type" , # Defines how the client will process it "id" : "unique-id" , # Unique identifier for the message "data" : { # Custom payload # Your custom data fields } } Example - Standard Message: Copy Ask AI import uuid # Send a standard message to all participants message_data = { "label" : "rtvi-ai" , "type" : "custom-message" , "id" : str (uuid.uuid4()), "data" : { "message" : "Hello everyone!" , "timestamp" : datetime.datetime.now().isoformat() } } await transport.send_message(TransportMessageFrame( message = message_data)) Example - Urgent Message to Specific Participant: Copy Ask AI # Send urgent message to specific participant urgent_data = { "label" : "rtvi-ai" , "type" : "server-message" , "id" : str (uuid.uuid4()), "data" : { "message" : "Private message" , "importance" : "high" } } await transport.send_message( DailyTransportMessageUrgentFrame( message = urgent_data, participant_id = "participant-123" ) ) The message will be delivered to: All participants if no specific participant_id is set Only the specified participant if participant_id is set Through the Daily app messaging system Can be received by registering an on_app_message handler Use TransportMessageUrgentFrame for high-priority messages that need immediate delivery. ​ Media Control ​ send_image async method Sends an image frame to the room. Copy Ask AI async def send_image ( frame : OutputImageRawFrame | SpriteFrame) -> None Parameters: frame : Image frame to send, either raw image data or sprite animation ​ send_audio async method Sends an audio frame to the room. Copy Ask AI async def send_audio ( frame : OutputAudioRawFrame) -> None Parameters: frame : Audio frame to send ​ Video Management ​ capture_participant_video async method Starts capturing video from a specific participant. Copy Ask AI async def capture_participant_video ( participant_id : str , framerate : int = 30 , video_source : str = "camera" , color_format : str = "RGB" ) -> None Parameters: participant_id : ID of the participant to capture framerate : Target frame rate (default: 30) video_source : Video source type (default: “camera”) color_format : Color format of the video (default: “RGB”) To request an image from the camera stream, set the framerate to 0 and push a UserImageRequestFrame . ​ Transcription Control ​ capture_participant_transcription async method Starts capturing and transcribing audio from a specific participant. Copy Ask AI async def capture_participant_transcription ( participant_id : str ) -> None Parameters: participant_id : ID of the participant to transcribe ​ update_transcription async method Updates the transcription configuration for specific participants or instances. Copy Ask AI async def update_transcription ( participants : List[ str ] | None = None , instance_id : str | None = None ) -> None Parameters: participants : Optional list of participant IDs to transcribe. If None, transcribes all participants. instance_id : Optional specific transcription instance ID to update Example: Copy Ask AI # Update transcription for specific participants await transport.update_transcription( participants = [ "participant-123" , "participant-456" ]) # Update specific transcription instance await transport.update_transcription( instance_id = "transcription-789" ) ​ Recording Control ​ start_recording async method Starts recording the room session. Copy Ask AI async def start_recording ( streaming_settings : Dict = None , stream_id : str = None , force_new : bool = None ) -> None Parameters: streaming_settings : Recording configuration settings stream_id : Optional stream identifier force_new : Force start a new recording See start_recording for more details. ​ stop_recording async method Stops an active recording. Copy Ask AI async def stop_recording ( stream_id : str = None ) -> None Parameters: stream_id : Optional stream identifier to stop specific recording See stop_recording for more details. ​ Dial-out Control ​ start_dialout async method Initiates a dial-out call. Copy Ask AI async def start_dialout ( settings : Dict = None ) -> None Parameters: settings : Dial-out configuration settings See start_dialout for more details. ​ stop_dialout async method Stops an active dial-out call. Copy Ask AI async def stop_dialout ( participant_id : str ) -> None Parameters: participant_id : ID of the participant to disconnect See stop_dialout for more details. ​ send_dtmf async method Sends DTMF tones in an existing dial-out session. Copy Ask AI async def send_dtmf ( settings : Dict = None ) -> None Parameters: settings : DTMF settings See send_dtmf for more details. ​ Subscription Management ​ update_subscriptions async method Updates subscriptions and subscription profiles. This function allows you to update subscription profiles and at the same time assign specific subscription profiles to a participant and even change specific settings for some participants. Copy Ask AI async def update_subscriptions ( participant_settings : Dict = None , profile_settings : Dict = None ) -> None Parameters: participant_settings : Per-participant subscription settings profile_settings : Global subscription profile settings See update_subscriptions for more details. ​ Participant Admin When using the DailyTransport, if your bot is given a meeting token with owner or participant admin permissions, it can remote-control other participants’ capabilities and permissions. With this control, the bot can modify settings for how others send and receive media, how they are seen in a session, and whether they have administrative controls. ​ update_remote_participants async method Updates permissions and settings for specific participants in the room. Copy Ask AI async def update_remote_participants ( remote_participants : Mapping[ str , Any] = None ) -> None Parameters: remote_participants : Dictionary mapping participant IDs to their updated settings This method allows fine-grained control over participant capabilities and permissions in the Daily room. Examples: Copy Ask AI # Mute a participant's microphone await transport.update_remote_participants({ "participant-123" : { "inputsEnabled" : { "microphone" : False } } }) # Revoke send permissions (mutes the participant and prevents them from unmuting) await transport.update_remote_participants({ "participant-123" : { "permissions" : { "canSend" : [] # Empty list revokes all send permissions } } }) # Configure selective audio routing await transport.update_remote_participants({ "participant-123" : { # the customer "permissions" : { "canReceive" : { "base" : False , # Don't receive from anyone by default "byUserId" : { "hold-music" : True # Only receive from hold music player } } } } }) # Set up user-to-user permissions mapping (e.g., when transferring calls) # Assumes customer was previously unable to hear anyone (they were on hold) and agent was speaking to bot await transport.update_remote_participants({ "participant-123" : { # the customer "permissions" : { "base" : False , "canReceive" : { "byUserId" : { "agent" : True }} } }, "participant-456" : { # the agent "permissions" : { "base" : False , "canReceive" : { "byUserId" : { "customer" : True }} } } }) # Promote a participant to be a participant admin await transport.update_remote_participants({ "participant-123" : { "permissions" : { "canAdmin" : [ "participants" ] } } }) Permissions Settings: For comprehensive details on permissions settings, see the Daily API documentation: daily-python’s update_remote_participants() documentation daily-js’s updateParticipant() -> updatePermission and participants().permissions ​ Chat messages ​ send_prebuilt_chat_message async method When testing with Daily Prebuilt, you can send messages to the Prebuilt chat using this method. Parameters: message (str): The chat message to send user_name (str): The name of the user sending the message See send_prebuilt_chat_message for more details. ​ Properties ​ participant_id string Returns the transport’s participant ID in the room. ​ room_url string Returns the transport’s room URL for the room currently in use. ​ Notes Supports real-time audio/video communication Handles WebRTC connection management Provides voice activity detection Includes transcription capabilities Manages participant interactions Supports recording and streaming Thread-safe processing All callbacks are asynchronous Multiple handlers can be registered for the same event Exceptions in callbacks are logged but don’t stop the transport Supported Services SmallWebRTCTransport On this page Overview Installation Configuration Constructor Parameters DailyParams Configuration Audio Output Configuration Audio Input Configuration Video Output Configuration Voice Activity Detection (VAD) Feature Settings Basic Usage Event Callbacks Connection Events Participant Events Communication Events Dial Events Dial-in Dial-out Transcription Events Recording Events Error Events Notes on Callbacks Frame Types Input Frames Output Frames Methods Room Management Media Control Video Management Transcription Control Recording Control Dial-out Control Subscription Management Participant Admin Chat messages Properties Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/transport_daily_c53487fb.txt b/transport_daily_c53487fb.txt
new file mode 100644
index 0000000000000000000000000000000000000000..f173711c8c7d1cee12b47c4acd3e4972a235cc6a
--- /dev/null
+++ b/transport_daily_c53487fb.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/transport/daily#param-transcription-settings
+Title: Daily WebRTC - Pipecat
+==================================================
+
+Daily WebRTC - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Transport Daily WebRTC Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Daily WebRTC SmallWebRTCTransport FastAPI WebSocket WebSocket Server Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview DailyTransport provides real-time audio and video communication capabilities using Daily’s WebRTC platform. It supports bidirectional audio/video streams, transcription, voice activity detection (VAD), and participant management. ​ Installation To use DailyTransport , install the required dependencies: Copy Ask AI pip install "pipecat-ai[daily]" You’ll also need to set up your Daily API key as an environment variable: DAILY_API_KEY . You can obtain a Daily API key by signing up at Daily . ​ Configuration ​ Constructor Parameters ​ room_url str required Daily room URL ​ token str | None Daily room token ​ bot_name str required Name of the bot in the room ​ params DailyParams default: "DailyParams()" Transport configuration parameters ​ DailyParams Configuration ​ api_url str default: "https://api.daily.co/v1" Daily API endpoint URL ​ api_key str default: "" Daily API key for authentication ​ Audio Output Configuration ​ audio_out_enabled bool default: "False" Enable audio output capabilities ​ audio_out_is_live bool default: "False" Enable live audio streaming mode ​ audio_out_sample_rate int default: "None" Audio output sample rate in Hz ​ audio_out_channels int default: "1" Number of audio output channels ​ audio_out_bitrate int default: "96000" Audio output bitrate in bits per second ​ Audio Input Configuration ​ audio_in_enabled bool default: "False" Enable audio input capabilities ​ audio_in_passthrough bool default: "False" When enabled, incoming audio frames are pushed downstream ​ audio_in_sample_rate int default: "None" Audio input sample rate in Hz ​ audio_in_channels int default: "1" Number of audio input channels ​ audio_in_filter Optional[BaseAudioFilter] default: "None" Audio filter for input processing. Supported filters are: KrispFilter() and NoisereduceFilter() . See the KrispFilter() reference docs and NoisereduceFilter() reference docs ` for more information. ​ Video Output Configuration ​ video_out_enabled bool default: "False" Enable video output capabilities ​ video_out_is_live bool default: "False" Enable live video streaming mode ​ video_out_width int default: "1024" Video output width in pixels ​ video_out_height int default: "768" Video output height in pixels ​ video_out_bitrate int default: "800000" Video output bitrate in bits per second ​ video_out_framerate int default: "30" Video output frame rate ​ video_out_color_format str default: "RGB" Video color format (RGB, BGR, etc.) ​ Voice Activity Detection (VAD) ​ vad_analyzer VADAnalyzer | None default: "None" Voice Activity Detection analyzer. You can set this to either SileroVADAnalyzer() or WebRTCVADAnalyzer() . SileroVADAnalyzer is the recommended option. Learn more about the SileroVADAnalyzer . ​ Feature Settings ​ dialin_settings Optional[DailyDialinSettings] default: "None" Configuration for dial-in functionality ​ transcription_enabled bool default: "False" Enable real-time transcription ​ transcription_settings DailyTranscriptionSettings Configuration for transcription features Copy Ask AI class DailyTranscriptionSettings ( BaseModel ): language: str = "en" # Default language model: str = "nova-2-general" # Transcription model profanity_filter: bool = True # Filter profanity redact: bool = False # Redact sensitive information endpointing: bool = True # Enable speech endpointing punctuate: bool = True # Add punctuation includeRawResponse: bool = True # Include raw API response extra: Mapping[ str , Any] = { "interim_results" : True # Provide any Deepgram-specific settings } ​ Basic Usage Copy Ask AI from pipecat.transports.services.daily import DailyTransport, DailyParams from pipecat.audio.vad.silero import SileroVADAnalyzer # Configure transport transport = DailyTransport( room_url = "https://your-domain.daily.co/room-name" , token = "your-room-token" , bot_name = "AI Assistant" , params = DailyParams( audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer( params = VADParams( stop_secs = 0.5 )), ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), # Handle incoming audio/video stt, # Speech-to-text llm, # Language model tts, # Text-to-speech transport.output() # Handle outgoing audio/video ]) # Use transport event handlers to manage the call lifecycle # Event handler for when the user joins the room: @transport.event_handler ( "on_first_participant_joined" ) async def on_first_participant_joined ( transport , participant ): # Start transcription for the user await transport.capture_participant_transcription(participant[ "id" ]) # Kick off the conversation. await task.queue_frames([context_aggregator.user().get_context_frame()]) # Event handler for when the user leaves the room: @transport.event_handler ( "on_participant_left" ) async def on_participant_left ( transport , participant , reason ): # Cancel the pipeline, which stops processing and removes the bot from the room await task.cancel() ​ Event Callbacks DailyTransport provides a comprehensive callback system for handling various events. Register callbacks using the @transport.event_handler() decorator. ​ Connection Events ​ on_joined async callback Called when the bot successfully joins the room. Parameters: transport : The DailyTransport instance data : Dictionary containing room join information Copy Ask AI @transport.event_handler ( "on_joined" ) async def on_joined ( transport , data : Dict[ str , Any]): logger.info( f "Joined room with data: { data } " ) ​ on_left async callback Called when the bot leaves the room. Parameters: transport : The DailyTransport instance Copy Ask AI @transport.event_handler ( "on_left" ) async def on_left ( transport ): logger.info( "Left room" ) ​ Participant Events ​ on_first_participant_joined async callback Called when the first participant joins the room. Useful for initializing conversations. Parameters: transport : The DailyTransport instance participant : Dictionary containing participant information Copy Ask AI @transport.event_handler ( "on_first_participant_joined" ) async def on_first_participant_joined ( transport , participant : Dict[ str , Any]): await transport.capture_participant_transcription(participant[ "id" ]) await task.queue_frames([LLMMessagesFrame(initial_messages)]) ​ on_participant_joined async callback Called when any participant joins the room. Parameters: transport : The DailyTransport instance participant : Dictionary containing participant information Copy Ask AI @transport.event_handler ( "on_participant_joined" ) async def on_participant_joined ( transport , participant : Dict[ str , Any]): logger.info( f "Participant joined: { participant[ 'id' ] } " ) ​ on_participant_left async callback Called when a participant leaves the room. Parameters: transport : The DailyTransport instance participant : Dictionary containing participant information reason : String describing why the participant left, “leftCall” | “hidden” Copy Ask AI @transport.event_handler ( "on_participant_left" ) async def on_participant_left ( transport , participant : Dict[ str , Any], reason : str ): logger.info( f "Participant { participant[ 'id' ] } left: { reason } " ) # Cancel the pipeline task to stop processing and remove the bot from the Daily room await task.cancel() ​ on_participant_updated async callback Event emitted when a participant is updated. This can mean either the participant’s metadata was updated, or the tracks belonging to the participant changed. Parameters: transport : The DailyTransport instance participant : Dictionary containing updated participant information Copy Ask AI @transport.event_handler ( "on_participant_updated" ) async def on_participant_updated ( transport , participant : Dict[ str , Any]): logger.info( f "Participant updated: { participant } " ) ​ Communication Events ​ on_app_message async callback Event emitted when a custom app message is received from another participant or via the REST API. Parameters: transport : The DailyTransport instance message : The message content (any type) sender : String identifier of the message sender Copy Ask AI @transport.event_handler ( "on_app_message" ) async def on_app_message ( transport , message : Any, sender : str ): logger.info( f "Message from { sender } : { message } " ) ​ on_call_state_updated async callback Event emitted when the call state changes, normally as a consequence of joining or leaving the call. Parameters: transport : The DailyTransport instance state : String representing the new call state. Learn more about call states . Copy Ask AI @transport.event_handler ( "on_call_state_updated" ) async def on_call_state_updated ( transport , state : str ): logger.info( f "Call state updated: { state } " ) ​ Dial Events ​ Dial-in ​ on_dialin_ready async callback Event emitted when dial-in is ready. This happens after the room has connected to the SIP endpoint and the system is ready to receive dial-in calls. Parameters: transport : The DailyTransport instance sip_endpoint : String containing the SIP endpoint information Copy Ask AI @transport.event_handler ( "on_dialin_ready" ) async def on_dialin_ready ( transport , sip_endpoint : str ): logger.info( f "Dial-in ready at: { sip_endpoint } " ) ​ on_dialin_connected async callback Event emitted when the session with the dial-in remote end is established (i.e. SIP endpoint or PSTN are connectd to the Daily room). Note: connected does not mean media (audio or video) has started flowing between the room and PSTN, it means the room received the connection request and both endpoints are negotiating the media flow. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. See DialinConnectedEvent . Copy Ask AI @transport.event_handler ( "on_dialin_connected" ) async def on_dialin_connected ( transport , data : Any): logger.info( f "Dial-in connected: { data } " ) ​ on_dialin_stopped async callback Event emitted when the dial-in remote end disconnects the call. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. See DialinStoppedEvent . Copy Ask AI @transport.event_handler ( "on_dialin_stopped" ) async def on_dialin_stopped ( transport , data : Any): logger.info( f "Dial-in stopped: { data } " ) ​ on_dialin_error async callback Event emitted in the case of dial-in errors which are fatal and the service cannot proceed. For example, an error in SDP negotiation is fatal to the media/SIP pipeline and will result in dialin-error being triggered. Parameters: transport : The DailyTransport instance data : Dictionary containing error information. See DialinEvent . Copy Ask AI @transport.event_handler ( "on_dialin_error" ) async def on_dialin_error ( transport , data : Any): logger.error( f "Dial-in error: { data } " ) ​ on_dialin_warning async callback Event emitted there is a dial-in non-fatal error, such as the selected codec not being used and a fallback codec being utilized. Parameters: transport : The DailyTransport instance data : Dictionary containing warning information. See DialinEvent . Copy Ask AI @transport.event_handler ( "on_dialin_warning" ) async def on_dialin_warning ( transport , data : Any): logger.warning( f "Dial-in warning: { data } " ) ​ Dial-out ​ on_dialout_answered async callback Event emitted when the session with the dial-out remote end is answered. Parameters: transport : The DailyTransport instance data : Dictionary containing call information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_answered" ) async def on_dialout_answered ( transport , data : Any): logger.info( f "Dial-out answered: { data } " ) ​ on_dialout_connected async callback Event emitted when the session with the dial-out remote end is established. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_connected" ) async def on_dialout_connected ( transport , data : Any): logger.info( f "Dial-out connected: { data } " ) ​ on_dialout_stopped async callback Event emitted when the dial-out session is stopped. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_stopped" ) async def on_dialout_stopped ( transport , data : Any): logger.info( f "Dial-out stopped: { data } " ) ​ on_dialout_error async callback Event emitted in the case of dial-out errors which are fatal and the service cannot proceed. For example, an error in SDP negotiation is fatal to the media/SIP pipeline and will result in dialout-error being triggered. Parameters: transport : The DailyTransport instance data : Dictionary containing error information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_error" ) async def on_dialout_error ( transport , data : Any): logger.error( f "Dial-out error: { data } " ) ​ on_dialout_warning async callback Event emitted there is a dial-out non-fatal error, such as the selected codec not being used and a fallback codec being utilized. Parameters: transport : The DailyTransport instance data : Dictionary containing warning information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_warning" ) async def on_dialout_warning ( transport , data : Any): logger.warning( f "Dial-out warning: { data } " ) ​ Transcription Events ​ on_transcription_message async callback Called when a transcription message is received. This includes both interim and final transcriptions. Parameters: transport : The DailyTransport instance message : Dictionary containing transcription data including. Learn more . Copy Ask AI @transport.event_handler ( "on_transcription_message" ) async def on_transcription_message ( transport , message : Dict[ str , Any]): participant_id = message.get( "participantId" ) text = message.get( "text" ) is_final = message[ "rawResponse" ][ "is_final" ] logger.info( f "Transcription from { participant_id } : { text } (final: { is_final } )" ) ​ Recording Events ​ on_recording_started async callback Called when a room recording starts successfully. Parameters: transport : The DailyTransport instance status : Dictionary containing recording status information. Learn more . Copy Ask AI @transport.event_handler ( "on_recording_started" ) async def on_recording_started ( transport , status : Dict[ str , Any]): logger.info( f "Recording started with status: { status } " ) ​ on_recording_stopped async callback Called when a room recording stops. Parameters: transport : The DailyTransport instance stream_id : String identifier of the stopped recording stream Copy Ask AI @transport.event_handler ( "on_recording_stopped" ) async def on_recording_stopped ( transport , stream_id : str ): logger.info( f "Recording stopped for stream: { stream_id } " ) ​ on_recording_error async callback Called when an error occurs during recording. Parameters: transport : The DailyTransport instance stream_id : String identifier of the recording stream message : Error message describing what went wrong Copy Ask AI @transport.event_handler ( "on_recording_error" ) async def on_recording_error ( transport , stream_id : str , message : str ): logger.error( f "Recording error for stream { stream_id } : { message } " ) ​ Error Events ​ on_error async callback Called when a transport error occurs. Parameters: transport : The DailyTransport instance error : String containing error details Copy Ask AI @transport.event_handler ( "on_error" ) async def on_error ( transport , error : str ): logger.error( f "Transport error: { error } " ) ​ Notes on Callbacks All callbacks are asynchronous Callbacks are executed in the order they were registered Multiple handlers can be registered for the same event Exceptions in callbacks are logged but don’t stop the transport Callbacks should be lightweight to avoid blocking the event loop Heavy processing should be offloaded to separate tasks ​ Frame Types ​ Input Frames ​ InputAudioRawFrame Frame Raw audio data from participants ​ UserImageRawFrame Frame Video frames from participants ​ Output Frames ​ OutputAudioRawFrame Frame Audio data to be sent ​ OutputImageRawFrame Frame Video frames to be sent ​ Methods ​ Room Management ​ participants method Returns a dictionary of all participants currently in the room. Copy Ask AI def participants () -> Dict[ str , Any] See participants for more details. ​ participant_counts method Returns participant count statistics for the room. Copy Ask AI def participant_counts () -> Dict[ str , int ] See participant_counts for more details. ​ send_message async method Sends a message to participants in the Daily room. Messages can be directed to all participants or targeted to a specific participant. Copy Ask AI async def send_message ( frame : TransportMessageFrame | TransportMessageUrgentFrame) Parameters: frame : A transport message frame containing the message to send. Can be either: TransportMessageFrame : Standard message TransportMessageUrgentFrame : High-priority message Message Structure: Messages should conform to the expected format for the receiving system. For RTVI messages, use: Copy Ask AI { "label" : "rtvi-ai" , # Required for RTVI protocol "type" : "message-type" , # Defines how the client will process it "id" : "unique-id" , # Unique identifier for the message "data" : { # Custom payload # Your custom data fields } } Example - Standard Message: Copy Ask AI import uuid # Send a standard message to all participants message_data = { "label" : "rtvi-ai" , "type" : "custom-message" , "id" : str (uuid.uuid4()), "data" : { "message" : "Hello everyone!" , "timestamp" : datetime.datetime.now().isoformat() } } await transport.send_message(TransportMessageFrame( message = message_data)) Example - Urgent Message to Specific Participant: Copy Ask AI # Send urgent message to specific participant urgent_data = { "label" : "rtvi-ai" , "type" : "server-message" , "id" : str (uuid.uuid4()), "data" : { "message" : "Private message" , "importance" : "high" } } await transport.send_message( DailyTransportMessageUrgentFrame( message = urgent_data, participant_id = "participant-123" ) ) The message will be delivered to: All participants if no specific participant_id is set Only the specified participant if participant_id is set Through the Daily app messaging system Can be received by registering an on_app_message handler Use TransportMessageUrgentFrame for high-priority messages that need immediate delivery. ​ Media Control ​ send_image async method Sends an image frame to the room. Copy Ask AI async def send_image ( frame : OutputImageRawFrame | SpriteFrame) -> None Parameters: frame : Image frame to send, either raw image data or sprite animation ​ send_audio async method Sends an audio frame to the room. Copy Ask AI async def send_audio ( frame : OutputAudioRawFrame) -> None Parameters: frame : Audio frame to send ​ Video Management ​ capture_participant_video async method Starts capturing video from a specific participant. Copy Ask AI async def capture_participant_video ( participant_id : str , framerate : int = 30 , video_source : str = "camera" , color_format : str = "RGB" ) -> None Parameters: participant_id : ID of the participant to capture framerate : Target frame rate (default: 30) video_source : Video source type (default: “camera”) color_format : Color format of the video (default: “RGB”) To request an image from the camera stream, set the framerate to 0 and push a UserImageRequestFrame . ​ Transcription Control ​ capture_participant_transcription async method Starts capturing and transcribing audio from a specific participant. Copy Ask AI async def capture_participant_transcription ( participant_id : str ) -> None Parameters: participant_id : ID of the participant to transcribe ​ update_transcription async method Updates the transcription configuration for specific participants or instances. Copy Ask AI async def update_transcription ( participants : List[ str ] | None = None , instance_id : str | None = None ) -> None Parameters: participants : Optional list of participant IDs to transcribe. If None, transcribes all participants. instance_id : Optional specific transcription instance ID to update Example: Copy Ask AI # Update transcription for specific participants await transport.update_transcription( participants = [ "participant-123" , "participant-456" ]) # Update specific transcription instance await transport.update_transcription( instance_id = "transcription-789" ) ​ Recording Control ​ start_recording async method Starts recording the room session. Copy Ask AI async def start_recording ( streaming_settings : Dict = None , stream_id : str = None , force_new : bool = None ) -> None Parameters: streaming_settings : Recording configuration settings stream_id : Optional stream identifier force_new : Force start a new recording See start_recording for more details. ​ stop_recording async method Stops an active recording. Copy Ask AI async def stop_recording ( stream_id : str = None ) -> None Parameters: stream_id : Optional stream identifier to stop specific recording See stop_recording for more details. ​ Dial-out Control ​ start_dialout async method Initiates a dial-out call. Copy Ask AI async def start_dialout ( settings : Dict = None ) -> None Parameters: settings : Dial-out configuration settings See start_dialout for more details. ​ stop_dialout async method Stops an active dial-out call. Copy Ask AI async def stop_dialout ( participant_id : str ) -> None Parameters: participant_id : ID of the participant to disconnect See stop_dialout for more details. ​ send_dtmf async method Sends DTMF tones in an existing dial-out session. Copy Ask AI async def send_dtmf ( settings : Dict = None ) -> None Parameters: settings : DTMF settings See send_dtmf for more details. ​ Subscription Management ​ update_subscriptions async method Updates subscriptions and subscription profiles. This function allows you to update subscription profiles and at the same time assign specific subscription profiles to a participant and even change specific settings for some participants. Copy Ask AI async def update_subscriptions ( participant_settings : Dict = None , profile_settings : Dict = None ) -> None Parameters: participant_settings : Per-participant subscription settings profile_settings : Global subscription profile settings See update_subscriptions for more details. ​ Participant Admin When using the DailyTransport, if your bot is given a meeting token with owner or participant admin permissions, it can remote-control other participants’ capabilities and permissions. With this control, the bot can modify settings for how others send and receive media, how they are seen in a session, and whether they have administrative controls. ​ update_remote_participants async method Updates permissions and settings for specific participants in the room. Copy Ask AI async def update_remote_participants ( remote_participants : Mapping[ str , Any] = None ) -> None Parameters: remote_participants : Dictionary mapping participant IDs to their updated settings This method allows fine-grained control over participant capabilities and permissions in the Daily room. Examples: Copy Ask AI # Mute a participant's microphone await transport.update_remote_participants({ "participant-123" : { "inputsEnabled" : { "microphone" : False } } }) # Revoke send permissions (mutes the participant and prevents them from unmuting) await transport.update_remote_participants({ "participant-123" : { "permissions" : { "canSend" : [] # Empty list revokes all send permissions } } }) # Configure selective audio routing await transport.update_remote_participants({ "participant-123" : { # the customer "permissions" : { "canReceive" : { "base" : False , # Don't receive from anyone by default "byUserId" : { "hold-music" : True # Only receive from hold music player } } } } }) # Set up user-to-user permissions mapping (e.g., when transferring calls) # Assumes customer was previously unable to hear anyone (they were on hold) and agent was speaking to bot await transport.update_remote_participants({ "participant-123" : { # the customer "permissions" : { "base" : False , "canReceive" : { "byUserId" : { "agent" : True }} } }, "participant-456" : { # the agent "permissions" : { "base" : False , "canReceive" : { "byUserId" : { "customer" : True }} } } }) # Promote a participant to be a participant admin await transport.update_remote_participants({ "participant-123" : { "permissions" : { "canAdmin" : [ "participants" ] } } }) Permissions Settings: For comprehensive details on permissions settings, see the Daily API documentation: daily-python’s update_remote_participants() documentation daily-js’s updateParticipant() -> updatePermission and participants().permissions ​ Chat messages ​ send_prebuilt_chat_message async method When testing with Daily Prebuilt, you can send messages to the Prebuilt chat using this method. Parameters: message (str): The chat message to send user_name (str): The name of the user sending the message See send_prebuilt_chat_message for more details. ​ Properties ​ participant_id string Returns the transport’s participant ID in the room. ​ room_url string Returns the transport’s room URL for the room currently in use. ​ Notes Supports real-time audio/video communication Handles WebRTC connection management Provides voice activity detection Includes transcription capabilities Manages participant interactions Supports recording and streaming Thread-safe processing All callbacks are asynchronous Multiple handlers can be registered for the same event Exceptions in callbacks are logged but don’t stop the transport Supported Services SmallWebRTCTransport On this page Overview Installation Configuration Constructor Parameters DailyParams Configuration Audio Output Configuration Audio Input Configuration Video Output Configuration Voice Activity Detection (VAD) Feature Settings Basic Usage Event Callbacks Connection Events Participant Events Communication Events Dial Events Dial-in Dial-out Transcription Events Recording Events Error Events Notes on Callbacks Frame Types Input Frames Output Frames Methods Room Management Media Control Video Management Transcription Control Recording Control Dial-out Control Subscription Management Participant Admin Chat messages Properties Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/transport_daily_cf7c4369.txt b/transport_daily_cf7c4369.txt
new file mode 100644
index 0000000000000000000000000000000000000000..23c678e8c48c8a4e554796cd07f48a69484b9b77
--- /dev/null
+++ b/transport_daily_cf7c4369.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/transport/daily#param-on-recording-error
+Title: Daily WebRTC - Pipecat
+==================================================
+
+Daily WebRTC - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Transport Daily WebRTC Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Daily WebRTC SmallWebRTCTransport FastAPI WebSocket WebSocket Server Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview DailyTransport provides real-time audio and video communication capabilities using Daily’s WebRTC platform. It supports bidirectional audio/video streams, transcription, voice activity detection (VAD), and participant management. ​ Installation To use DailyTransport , install the required dependencies: Copy Ask AI pip install "pipecat-ai[daily]" You’ll also need to set up your Daily API key as an environment variable: DAILY_API_KEY . You can obtain a Daily API key by signing up at Daily . ​ Configuration ​ Constructor Parameters ​ room_url str required Daily room URL ​ token str | None Daily room token ​ bot_name str required Name of the bot in the room ​ params DailyParams default: "DailyParams()" Transport configuration parameters ​ DailyParams Configuration ​ api_url str default: "https://api.daily.co/v1" Daily API endpoint URL ​ api_key str default: "" Daily API key for authentication ​ Audio Output Configuration ​ audio_out_enabled bool default: "False" Enable audio output capabilities ​ audio_out_is_live bool default: "False" Enable live audio streaming mode ​ audio_out_sample_rate int default: "None" Audio output sample rate in Hz ​ audio_out_channels int default: "1" Number of audio output channels ​ audio_out_bitrate int default: "96000" Audio output bitrate in bits per second ​ Audio Input Configuration ​ audio_in_enabled bool default: "False" Enable audio input capabilities ​ audio_in_passthrough bool default: "False" When enabled, incoming audio frames are pushed downstream ​ audio_in_sample_rate int default: "None" Audio input sample rate in Hz ​ audio_in_channels int default: "1" Number of audio input channels ​ audio_in_filter Optional[BaseAudioFilter] default: "None" Audio filter for input processing. Supported filters are: KrispFilter() and NoisereduceFilter() . See the KrispFilter() reference docs and NoisereduceFilter() reference docs ` for more information. ​ Video Output Configuration ​ video_out_enabled bool default: "False" Enable video output capabilities ​ video_out_is_live bool default: "False" Enable live video streaming mode ​ video_out_width int default: "1024" Video output width in pixels ​ video_out_height int default: "768" Video output height in pixels ​ video_out_bitrate int default: "800000" Video output bitrate in bits per second ​ video_out_framerate int default: "30" Video output frame rate ​ video_out_color_format str default: "RGB" Video color format (RGB, BGR, etc.) ​ Voice Activity Detection (VAD) ​ vad_analyzer VADAnalyzer | None default: "None" Voice Activity Detection analyzer. You can set this to either SileroVADAnalyzer() or WebRTCVADAnalyzer() . SileroVADAnalyzer is the recommended option. Learn more about the SileroVADAnalyzer . ​ Feature Settings ​ dialin_settings Optional[DailyDialinSettings] default: "None" Configuration for dial-in functionality ​ transcription_enabled bool default: "False" Enable real-time transcription ​ transcription_settings DailyTranscriptionSettings Configuration for transcription features Copy Ask AI class DailyTranscriptionSettings ( BaseModel ): language: str = "en" # Default language model: str = "nova-2-general" # Transcription model profanity_filter: bool = True # Filter profanity redact: bool = False # Redact sensitive information endpointing: bool = True # Enable speech endpointing punctuate: bool = True # Add punctuation includeRawResponse: bool = True # Include raw API response extra: Mapping[ str , Any] = { "interim_results" : True # Provide any Deepgram-specific settings } ​ Basic Usage Copy Ask AI from pipecat.transports.services.daily import DailyTransport, DailyParams from pipecat.audio.vad.silero import SileroVADAnalyzer # Configure transport transport = DailyTransport( room_url = "https://your-domain.daily.co/room-name" , token = "your-room-token" , bot_name = "AI Assistant" , params = DailyParams( audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer( params = VADParams( stop_secs = 0.5 )), ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), # Handle incoming audio/video stt, # Speech-to-text llm, # Language model tts, # Text-to-speech transport.output() # Handle outgoing audio/video ]) # Use transport event handlers to manage the call lifecycle # Event handler for when the user joins the room: @transport.event_handler ( "on_first_participant_joined" ) async def on_first_participant_joined ( transport , participant ): # Start transcription for the user await transport.capture_participant_transcription(participant[ "id" ]) # Kick off the conversation. await task.queue_frames([context_aggregator.user().get_context_frame()]) # Event handler for when the user leaves the room: @transport.event_handler ( "on_participant_left" ) async def on_participant_left ( transport , participant , reason ): # Cancel the pipeline, which stops processing and removes the bot from the room await task.cancel() ​ Event Callbacks DailyTransport provides a comprehensive callback system for handling various events. Register callbacks using the @transport.event_handler() decorator. ​ Connection Events ​ on_joined async callback Called when the bot successfully joins the room. Parameters: transport : The DailyTransport instance data : Dictionary containing room join information Copy Ask AI @transport.event_handler ( "on_joined" ) async def on_joined ( transport , data : Dict[ str , Any]): logger.info( f "Joined room with data: { data } " ) ​ on_left async callback Called when the bot leaves the room. Parameters: transport : The DailyTransport instance Copy Ask AI @transport.event_handler ( "on_left" ) async def on_left ( transport ): logger.info( "Left room" ) ​ Participant Events ​ on_first_participant_joined async callback Called when the first participant joins the room. Useful for initializing conversations. Parameters: transport : The DailyTransport instance participant : Dictionary containing participant information Copy Ask AI @transport.event_handler ( "on_first_participant_joined" ) async def on_first_participant_joined ( transport , participant : Dict[ str , Any]): await transport.capture_participant_transcription(participant[ "id" ]) await task.queue_frames([LLMMessagesFrame(initial_messages)]) ​ on_participant_joined async callback Called when any participant joins the room. Parameters: transport : The DailyTransport instance participant : Dictionary containing participant information Copy Ask AI @transport.event_handler ( "on_participant_joined" ) async def on_participant_joined ( transport , participant : Dict[ str , Any]): logger.info( f "Participant joined: { participant[ 'id' ] } " ) ​ on_participant_left async callback Called when a participant leaves the room. Parameters: transport : The DailyTransport instance participant : Dictionary containing participant information reason : String describing why the participant left, “leftCall” | “hidden” Copy Ask AI @transport.event_handler ( "on_participant_left" ) async def on_participant_left ( transport , participant : Dict[ str , Any], reason : str ): logger.info( f "Participant { participant[ 'id' ] } left: { reason } " ) # Cancel the pipeline task to stop processing and remove the bot from the Daily room await task.cancel() ​ on_participant_updated async callback Event emitted when a participant is updated. This can mean either the participant’s metadata was updated, or the tracks belonging to the participant changed. Parameters: transport : The DailyTransport instance participant : Dictionary containing updated participant information Copy Ask AI @transport.event_handler ( "on_participant_updated" ) async def on_participant_updated ( transport , participant : Dict[ str , Any]): logger.info( f "Participant updated: { participant } " ) ​ Communication Events ​ on_app_message async callback Event emitted when a custom app message is received from another participant or via the REST API. Parameters: transport : The DailyTransport instance message : The message content (any type) sender : String identifier of the message sender Copy Ask AI @transport.event_handler ( "on_app_message" ) async def on_app_message ( transport , message : Any, sender : str ): logger.info( f "Message from { sender } : { message } " ) ​ on_call_state_updated async callback Event emitted when the call state changes, normally as a consequence of joining or leaving the call. Parameters: transport : The DailyTransport instance state : String representing the new call state. Learn more about call states . Copy Ask AI @transport.event_handler ( "on_call_state_updated" ) async def on_call_state_updated ( transport , state : str ): logger.info( f "Call state updated: { state } " ) ​ Dial Events ​ Dial-in ​ on_dialin_ready async callback Event emitted when dial-in is ready. This happens after the room has connected to the SIP endpoint and the system is ready to receive dial-in calls. Parameters: transport : The DailyTransport instance sip_endpoint : String containing the SIP endpoint information Copy Ask AI @transport.event_handler ( "on_dialin_ready" ) async def on_dialin_ready ( transport , sip_endpoint : str ): logger.info( f "Dial-in ready at: { sip_endpoint } " ) ​ on_dialin_connected async callback Event emitted when the session with the dial-in remote end is established (i.e. SIP endpoint or PSTN are connectd to the Daily room). Note: connected does not mean media (audio or video) has started flowing between the room and PSTN, it means the room received the connection request and both endpoints are negotiating the media flow. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. See DialinConnectedEvent . Copy Ask AI @transport.event_handler ( "on_dialin_connected" ) async def on_dialin_connected ( transport , data : Any): logger.info( f "Dial-in connected: { data } " ) ​ on_dialin_stopped async callback Event emitted when the dial-in remote end disconnects the call. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. See DialinStoppedEvent . Copy Ask AI @transport.event_handler ( "on_dialin_stopped" ) async def on_dialin_stopped ( transport , data : Any): logger.info( f "Dial-in stopped: { data } " ) ​ on_dialin_error async callback Event emitted in the case of dial-in errors which are fatal and the service cannot proceed. For example, an error in SDP negotiation is fatal to the media/SIP pipeline and will result in dialin-error being triggered. Parameters: transport : The DailyTransport instance data : Dictionary containing error information. See DialinEvent . Copy Ask AI @transport.event_handler ( "on_dialin_error" ) async def on_dialin_error ( transport , data : Any): logger.error( f "Dial-in error: { data } " ) ​ on_dialin_warning async callback Event emitted there is a dial-in non-fatal error, such as the selected codec not being used and a fallback codec being utilized. Parameters: transport : The DailyTransport instance data : Dictionary containing warning information. See DialinEvent . Copy Ask AI @transport.event_handler ( "on_dialin_warning" ) async def on_dialin_warning ( transport , data : Any): logger.warning( f "Dial-in warning: { data } " ) ​ Dial-out ​ on_dialout_answered async callback Event emitted when the session with the dial-out remote end is answered. Parameters: transport : The DailyTransport instance data : Dictionary containing call information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_answered" ) async def on_dialout_answered ( transport , data : Any): logger.info( f "Dial-out answered: { data } " ) ​ on_dialout_connected async callback Event emitted when the session with the dial-out remote end is established. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_connected" ) async def on_dialout_connected ( transport , data : Any): logger.info( f "Dial-out connected: { data } " ) ​ on_dialout_stopped async callback Event emitted when the dial-out session is stopped. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_stopped" ) async def on_dialout_stopped ( transport , data : Any): logger.info( f "Dial-out stopped: { data } " ) ​ on_dialout_error async callback Event emitted in the case of dial-out errors which are fatal and the service cannot proceed. For example, an error in SDP negotiation is fatal to the media/SIP pipeline and will result in dialout-error being triggered. Parameters: transport : The DailyTransport instance data : Dictionary containing error information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_error" ) async def on_dialout_error ( transport , data : Any): logger.error( f "Dial-out error: { data } " ) ​ on_dialout_warning async callback Event emitted there is a dial-out non-fatal error, such as the selected codec not being used and a fallback codec being utilized. Parameters: transport : The DailyTransport instance data : Dictionary containing warning information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_warning" ) async def on_dialout_warning ( transport , data : Any): logger.warning( f "Dial-out warning: { data } " ) ​ Transcription Events ​ on_transcription_message async callback Called when a transcription message is received. This includes both interim and final transcriptions. Parameters: transport : The DailyTransport instance message : Dictionary containing transcription data including. Learn more . Copy Ask AI @transport.event_handler ( "on_transcription_message" ) async def on_transcription_message ( transport , message : Dict[ str , Any]): participant_id = message.get( "participantId" ) text = message.get( "text" ) is_final = message[ "rawResponse" ][ "is_final" ] logger.info( f "Transcription from { participant_id } : { text } (final: { is_final } )" ) ​ Recording Events ​ on_recording_started async callback Called when a room recording starts successfully. Parameters: transport : The DailyTransport instance status : Dictionary containing recording status information. Learn more . Copy Ask AI @transport.event_handler ( "on_recording_started" ) async def on_recording_started ( transport , status : Dict[ str , Any]): logger.info( f "Recording started with status: { status } " ) ​ on_recording_stopped async callback Called when a room recording stops. Parameters: transport : The DailyTransport instance stream_id : String identifier of the stopped recording stream Copy Ask AI @transport.event_handler ( "on_recording_stopped" ) async def on_recording_stopped ( transport , stream_id : str ): logger.info( f "Recording stopped for stream: { stream_id } " ) ​ on_recording_error async callback Called when an error occurs during recording. Parameters: transport : The DailyTransport instance stream_id : String identifier of the recording stream message : Error message describing what went wrong Copy Ask AI @transport.event_handler ( "on_recording_error" ) async def on_recording_error ( transport , stream_id : str , message : str ): logger.error( f "Recording error for stream { stream_id } : { message } " ) ​ Error Events ​ on_error async callback Called when a transport error occurs. Parameters: transport : The DailyTransport instance error : String containing error details Copy Ask AI @transport.event_handler ( "on_error" ) async def on_error ( transport , error : str ): logger.error( f "Transport error: { error } " ) ​ Notes on Callbacks All callbacks are asynchronous Callbacks are executed in the order they were registered Multiple handlers can be registered for the same event Exceptions in callbacks are logged but don’t stop the transport Callbacks should be lightweight to avoid blocking the event loop Heavy processing should be offloaded to separate tasks ​ Frame Types ​ Input Frames ​ InputAudioRawFrame Frame Raw audio data from participants ​ UserImageRawFrame Frame Video frames from participants ​ Output Frames ​ OutputAudioRawFrame Frame Audio data to be sent ​ OutputImageRawFrame Frame Video frames to be sent ​ Methods ​ Room Management ​ participants method Returns a dictionary of all participants currently in the room. Copy Ask AI def participants () -> Dict[ str , Any] See participants for more details. ​ participant_counts method Returns participant count statistics for the room. Copy Ask AI def participant_counts () -> Dict[ str , int ] See participant_counts for more details. ​ send_message async method Sends a message to participants in the Daily room. Messages can be directed to all participants or targeted to a specific participant. Copy Ask AI async def send_message ( frame : TransportMessageFrame | TransportMessageUrgentFrame) Parameters: frame : A transport message frame containing the message to send. Can be either: TransportMessageFrame : Standard message TransportMessageUrgentFrame : High-priority message Message Structure: Messages should conform to the expected format for the receiving system. For RTVI messages, use: Copy Ask AI { "label" : "rtvi-ai" , # Required for RTVI protocol "type" : "message-type" , # Defines how the client will process it "id" : "unique-id" , # Unique identifier for the message "data" : { # Custom payload # Your custom data fields } } Example - Standard Message: Copy Ask AI import uuid # Send a standard message to all participants message_data = { "label" : "rtvi-ai" , "type" : "custom-message" , "id" : str (uuid.uuid4()), "data" : { "message" : "Hello everyone!" , "timestamp" : datetime.datetime.now().isoformat() } } await transport.send_message(TransportMessageFrame( message = message_data)) Example - Urgent Message to Specific Participant: Copy Ask AI # Send urgent message to specific participant urgent_data = { "label" : "rtvi-ai" , "type" : "server-message" , "id" : str (uuid.uuid4()), "data" : { "message" : "Private message" , "importance" : "high" } } await transport.send_message( DailyTransportMessageUrgentFrame( message = urgent_data, participant_id = "participant-123" ) ) The message will be delivered to: All participants if no specific participant_id is set Only the specified participant if participant_id is set Through the Daily app messaging system Can be received by registering an on_app_message handler Use TransportMessageUrgentFrame for high-priority messages that need immediate delivery. ​ Media Control ​ send_image async method Sends an image frame to the room. Copy Ask AI async def send_image ( frame : OutputImageRawFrame | SpriteFrame) -> None Parameters: frame : Image frame to send, either raw image data or sprite animation ​ send_audio async method Sends an audio frame to the room. Copy Ask AI async def send_audio ( frame : OutputAudioRawFrame) -> None Parameters: frame : Audio frame to send ​ Video Management ​ capture_participant_video async method Starts capturing video from a specific participant. Copy Ask AI async def capture_participant_video ( participant_id : str , framerate : int = 30 , video_source : str = "camera" , color_format : str = "RGB" ) -> None Parameters: participant_id : ID of the participant to capture framerate : Target frame rate (default: 30) video_source : Video source type (default: “camera”) color_format : Color format of the video (default: “RGB”) To request an image from the camera stream, set the framerate to 0 and push a UserImageRequestFrame . ​ Transcription Control ​ capture_participant_transcription async method Starts capturing and transcribing audio from a specific participant. Copy Ask AI async def capture_participant_transcription ( participant_id : str ) -> None Parameters: participant_id : ID of the participant to transcribe ​ update_transcription async method Updates the transcription configuration for specific participants or instances. Copy Ask AI async def update_transcription ( participants : List[ str ] | None = None , instance_id : str | None = None ) -> None Parameters: participants : Optional list of participant IDs to transcribe. If None, transcribes all participants. instance_id : Optional specific transcription instance ID to update Example: Copy Ask AI # Update transcription for specific participants await transport.update_transcription( participants = [ "participant-123" , "participant-456" ]) # Update specific transcription instance await transport.update_transcription( instance_id = "transcription-789" ) ​ Recording Control ​ start_recording async method Starts recording the room session. Copy Ask AI async def start_recording ( streaming_settings : Dict = None , stream_id : str = None , force_new : bool = None ) -> None Parameters: streaming_settings : Recording configuration settings stream_id : Optional stream identifier force_new : Force start a new recording See start_recording for more details. ​ stop_recording async method Stops an active recording. Copy Ask AI async def stop_recording ( stream_id : str = None ) -> None Parameters: stream_id : Optional stream identifier to stop specific recording See stop_recording for more details. ​ Dial-out Control ​ start_dialout async method Initiates a dial-out call. Copy Ask AI async def start_dialout ( settings : Dict = None ) -> None Parameters: settings : Dial-out configuration settings See start_dialout for more details. ​ stop_dialout async method Stops an active dial-out call. Copy Ask AI async def stop_dialout ( participant_id : str ) -> None Parameters: participant_id : ID of the participant to disconnect See stop_dialout for more details. ​ send_dtmf async method Sends DTMF tones in an existing dial-out session. Copy Ask AI async def send_dtmf ( settings : Dict = None ) -> None Parameters: settings : DTMF settings See send_dtmf for more details. ​ Subscription Management ​ update_subscriptions async method Updates subscriptions and subscription profiles. This function allows you to update subscription profiles and at the same time assign specific subscription profiles to a participant and even change specific settings for some participants. Copy Ask AI async def update_subscriptions ( participant_settings : Dict = None , profile_settings : Dict = None ) -> None Parameters: participant_settings : Per-participant subscription settings profile_settings : Global subscription profile settings See update_subscriptions for more details. ​ Participant Admin When using the DailyTransport, if your bot is given a meeting token with owner or participant admin permissions, it can remote-control other participants’ capabilities and permissions. With this control, the bot can modify settings for how others send and receive media, how they are seen in a session, and whether they have administrative controls. ​ update_remote_participants async method Updates permissions and settings for specific participants in the room. Copy Ask AI async def update_remote_participants ( remote_participants : Mapping[ str , Any] = None ) -> None Parameters: remote_participants : Dictionary mapping participant IDs to their updated settings This method allows fine-grained control over participant capabilities and permissions in the Daily room. Examples: Copy Ask AI # Mute a participant's microphone await transport.update_remote_participants({ "participant-123" : { "inputsEnabled" : { "microphone" : False } } }) # Revoke send permissions (mutes the participant and prevents them from unmuting) await transport.update_remote_participants({ "participant-123" : { "permissions" : { "canSend" : [] # Empty list revokes all send permissions } } }) # Configure selective audio routing await transport.update_remote_participants({ "participant-123" : { # the customer "permissions" : { "canReceive" : { "base" : False , # Don't receive from anyone by default "byUserId" : { "hold-music" : True # Only receive from hold music player } } } } }) # Set up user-to-user permissions mapping (e.g., when transferring calls) # Assumes customer was previously unable to hear anyone (they were on hold) and agent was speaking to bot await transport.update_remote_participants({ "participant-123" : { # the customer "permissions" : { "base" : False , "canReceive" : { "byUserId" : { "agent" : True }} } }, "participant-456" : { # the agent "permissions" : { "base" : False , "canReceive" : { "byUserId" : { "customer" : True }} } } }) # Promote a participant to be a participant admin await transport.update_remote_participants({ "participant-123" : { "permissions" : { "canAdmin" : [ "participants" ] } } }) Permissions Settings: For comprehensive details on permissions settings, see the Daily API documentation: daily-python’s update_remote_participants() documentation daily-js’s updateParticipant() -> updatePermission and participants().permissions ​ Chat messages ​ send_prebuilt_chat_message async method When testing with Daily Prebuilt, you can send messages to the Prebuilt chat using this method. Parameters: message (str): The chat message to send user_name (str): The name of the user sending the message See send_prebuilt_chat_message for more details. ​ Properties ​ participant_id string Returns the transport’s participant ID in the room. ​ room_url string Returns the transport’s room URL for the room currently in use. ​ Notes Supports real-time audio/video communication Handles WebRTC connection management Provides voice activity detection Includes transcription capabilities Manages participant interactions Supports recording and streaming Thread-safe processing All callbacks are asynchronous Multiple handlers can be registered for the same event Exceptions in callbacks are logged but don’t stop the transport Supported Services SmallWebRTCTransport On this page Overview Installation Configuration Constructor Parameters DailyParams Configuration Audio Output Configuration Audio Input Configuration Video Output Configuration Voice Activity Detection (VAD) Feature Settings Basic Usage Event Callbacks Connection Events Participant Events Communication Events Dial Events Dial-in Dial-out Transcription Events Recording Events Error Events Notes on Callbacks Frame Types Input Frames Output Frames Methods Room Management Media Control Video Management Transcription Control Recording Control Dial-out Control Subscription Management Participant Admin Chat messages Properties Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/transport_daily_d1cfd037.txt b/transport_daily_d1cfd037.txt
new file mode 100644
index 0000000000000000000000000000000000000000..432e2f805951cda87ed5b719ac0c3a2add6cd13f
--- /dev/null
+++ b/transport_daily_d1cfd037.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/transport/daily#output-frames
+Title: Daily WebRTC - Pipecat
+==================================================
+
+Daily WebRTC - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Transport Daily WebRTC Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Daily WebRTC SmallWebRTCTransport FastAPI WebSocket WebSocket Server Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview DailyTransport provides real-time audio and video communication capabilities using Daily’s WebRTC platform. It supports bidirectional audio/video streams, transcription, voice activity detection (VAD), and participant management. ​ Installation To use DailyTransport , install the required dependencies: Copy Ask AI pip install "pipecat-ai[daily]" You’ll also need to set up your Daily API key as an environment variable: DAILY_API_KEY . You can obtain a Daily API key by signing up at Daily . ​ Configuration ​ Constructor Parameters ​ room_url str required Daily room URL ​ token str | None Daily room token ​ bot_name str required Name of the bot in the room ​ params DailyParams default: "DailyParams()" Transport configuration parameters ​ DailyParams Configuration ​ api_url str default: "https://api.daily.co/v1" Daily API endpoint URL ​ api_key str default: "" Daily API key for authentication ​ Audio Output Configuration ​ audio_out_enabled bool default: "False" Enable audio output capabilities ​ audio_out_is_live bool default: "False" Enable live audio streaming mode ​ audio_out_sample_rate int default: "None" Audio output sample rate in Hz ​ audio_out_channels int default: "1" Number of audio output channels ​ audio_out_bitrate int default: "96000" Audio output bitrate in bits per second ​ Audio Input Configuration ​ audio_in_enabled bool default: "False" Enable audio input capabilities ​ audio_in_passthrough bool default: "False" When enabled, incoming audio frames are pushed downstream ​ audio_in_sample_rate int default: "None" Audio input sample rate in Hz ​ audio_in_channels int default: "1" Number of audio input channels ​ audio_in_filter Optional[BaseAudioFilter] default: "None" Audio filter for input processing. Supported filters are: KrispFilter() and NoisereduceFilter() . See the KrispFilter() reference docs and NoisereduceFilter() reference docs ` for more information. ​ Video Output Configuration ​ video_out_enabled bool default: "False" Enable video output capabilities ​ video_out_is_live bool default: "False" Enable live video streaming mode ​ video_out_width int default: "1024" Video output width in pixels ​ video_out_height int default: "768" Video output height in pixels ​ video_out_bitrate int default: "800000" Video output bitrate in bits per second ​ video_out_framerate int default: "30" Video output frame rate ​ video_out_color_format str default: "RGB" Video color format (RGB, BGR, etc.) ​ Voice Activity Detection (VAD) ​ vad_analyzer VADAnalyzer | None default: "None" Voice Activity Detection analyzer. You can set this to either SileroVADAnalyzer() or WebRTCVADAnalyzer() . SileroVADAnalyzer is the recommended option. Learn more about the SileroVADAnalyzer . ​ Feature Settings ​ dialin_settings Optional[DailyDialinSettings] default: "None" Configuration for dial-in functionality ​ transcription_enabled bool default: "False" Enable real-time transcription ​ transcription_settings DailyTranscriptionSettings Configuration for transcription features Copy Ask AI class DailyTranscriptionSettings ( BaseModel ): language: str = "en" # Default language model: str = "nova-2-general" # Transcription model profanity_filter: bool = True # Filter profanity redact: bool = False # Redact sensitive information endpointing: bool = True # Enable speech endpointing punctuate: bool = True # Add punctuation includeRawResponse: bool = True # Include raw API response extra: Mapping[ str , Any] = { "interim_results" : True # Provide any Deepgram-specific settings } ​ Basic Usage Copy Ask AI from pipecat.transports.services.daily import DailyTransport, DailyParams from pipecat.audio.vad.silero import SileroVADAnalyzer # Configure transport transport = DailyTransport( room_url = "https://your-domain.daily.co/room-name" , token = "your-room-token" , bot_name = "AI Assistant" , params = DailyParams( audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer( params = VADParams( stop_secs = 0.5 )), ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), # Handle incoming audio/video stt, # Speech-to-text llm, # Language model tts, # Text-to-speech transport.output() # Handle outgoing audio/video ]) # Use transport event handlers to manage the call lifecycle # Event handler for when the user joins the room: @transport.event_handler ( "on_first_participant_joined" ) async def on_first_participant_joined ( transport , participant ): # Start transcription for the user await transport.capture_participant_transcription(participant[ "id" ]) # Kick off the conversation. await task.queue_frames([context_aggregator.user().get_context_frame()]) # Event handler for when the user leaves the room: @transport.event_handler ( "on_participant_left" ) async def on_participant_left ( transport , participant , reason ): # Cancel the pipeline, which stops processing and removes the bot from the room await task.cancel() ​ Event Callbacks DailyTransport provides a comprehensive callback system for handling various events. Register callbacks using the @transport.event_handler() decorator. ​ Connection Events ​ on_joined async callback Called when the bot successfully joins the room. Parameters: transport : The DailyTransport instance data : Dictionary containing room join information Copy Ask AI @transport.event_handler ( "on_joined" ) async def on_joined ( transport , data : Dict[ str , Any]): logger.info( f "Joined room with data: { data } " ) ​ on_left async callback Called when the bot leaves the room. Parameters: transport : The DailyTransport instance Copy Ask AI @transport.event_handler ( "on_left" ) async def on_left ( transport ): logger.info( "Left room" ) ​ Participant Events ​ on_first_participant_joined async callback Called when the first participant joins the room. Useful for initializing conversations. Parameters: transport : The DailyTransport instance participant : Dictionary containing participant information Copy Ask AI @transport.event_handler ( "on_first_participant_joined" ) async def on_first_participant_joined ( transport , participant : Dict[ str , Any]): await transport.capture_participant_transcription(participant[ "id" ]) await task.queue_frames([LLMMessagesFrame(initial_messages)]) ​ on_participant_joined async callback Called when any participant joins the room. Parameters: transport : The DailyTransport instance participant : Dictionary containing participant information Copy Ask AI @transport.event_handler ( "on_participant_joined" ) async def on_participant_joined ( transport , participant : Dict[ str , Any]): logger.info( f "Participant joined: { participant[ 'id' ] } " ) ​ on_participant_left async callback Called when a participant leaves the room. Parameters: transport : The DailyTransport instance participant : Dictionary containing participant information reason : String describing why the participant left, “leftCall” | “hidden” Copy Ask AI @transport.event_handler ( "on_participant_left" ) async def on_participant_left ( transport , participant : Dict[ str , Any], reason : str ): logger.info( f "Participant { participant[ 'id' ] } left: { reason } " ) # Cancel the pipeline task to stop processing and remove the bot from the Daily room await task.cancel() ​ on_participant_updated async callback Event emitted when a participant is updated. This can mean either the participant’s metadata was updated, or the tracks belonging to the participant changed. Parameters: transport : The DailyTransport instance participant : Dictionary containing updated participant information Copy Ask AI @transport.event_handler ( "on_participant_updated" ) async def on_participant_updated ( transport , participant : Dict[ str , Any]): logger.info( f "Participant updated: { participant } " ) ​ Communication Events ​ on_app_message async callback Event emitted when a custom app message is received from another participant or via the REST API. Parameters: transport : The DailyTransport instance message : The message content (any type) sender : String identifier of the message sender Copy Ask AI @transport.event_handler ( "on_app_message" ) async def on_app_message ( transport , message : Any, sender : str ): logger.info( f "Message from { sender } : { message } " ) ​ on_call_state_updated async callback Event emitted when the call state changes, normally as a consequence of joining or leaving the call. Parameters: transport : The DailyTransport instance state : String representing the new call state. Learn more about call states . Copy Ask AI @transport.event_handler ( "on_call_state_updated" ) async def on_call_state_updated ( transport , state : str ): logger.info( f "Call state updated: { state } " ) ​ Dial Events ​ Dial-in ​ on_dialin_ready async callback Event emitted when dial-in is ready. This happens after the room has connected to the SIP endpoint and the system is ready to receive dial-in calls. Parameters: transport : The DailyTransport instance sip_endpoint : String containing the SIP endpoint information Copy Ask AI @transport.event_handler ( "on_dialin_ready" ) async def on_dialin_ready ( transport , sip_endpoint : str ): logger.info( f "Dial-in ready at: { sip_endpoint } " ) ​ on_dialin_connected async callback Event emitted when the session with the dial-in remote end is established (i.e. SIP endpoint or PSTN are connectd to the Daily room). Note: connected does not mean media (audio or video) has started flowing between the room and PSTN, it means the room received the connection request and both endpoints are negotiating the media flow. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. See DialinConnectedEvent . Copy Ask AI @transport.event_handler ( "on_dialin_connected" ) async def on_dialin_connected ( transport , data : Any): logger.info( f "Dial-in connected: { data } " ) ​ on_dialin_stopped async callback Event emitted when the dial-in remote end disconnects the call. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. See DialinStoppedEvent . Copy Ask AI @transport.event_handler ( "on_dialin_stopped" ) async def on_dialin_stopped ( transport , data : Any): logger.info( f "Dial-in stopped: { data } " ) ​ on_dialin_error async callback Event emitted in the case of dial-in errors which are fatal and the service cannot proceed. For example, an error in SDP negotiation is fatal to the media/SIP pipeline and will result in dialin-error being triggered. Parameters: transport : The DailyTransport instance data : Dictionary containing error information. See DialinEvent . Copy Ask AI @transport.event_handler ( "on_dialin_error" ) async def on_dialin_error ( transport , data : Any): logger.error( f "Dial-in error: { data } " ) ​ on_dialin_warning async callback Event emitted there is a dial-in non-fatal error, such as the selected codec not being used and a fallback codec being utilized. Parameters: transport : The DailyTransport instance data : Dictionary containing warning information. See DialinEvent . Copy Ask AI @transport.event_handler ( "on_dialin_warning" ) async def on_dialin_warning ( transport , data : Any): logger.warning( f "Dial-in warning: { data } " ) ​ Dial-out ​ on_dialout_answered async callback Event emitted when the session with the dial-out remote end is answered. Parameters: transport : The DailyTransport instance data : Dictionary containing call information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_answered" ) async def on_dialout_answered ( transport , data : Any): logger.info( f "Dial-out answered: { data } " ) ​ on_dialout_connected async callback Event emitted when the session with the dial-out remote end is established. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_connected" ) async def on_dialout_connected ( transport , data : Any): logger.info( f "Dial-out connected: { data } " ) ​ on_dialout_stopped async callback Event emitted when the dial-out session is stopped. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_stopped" ) async def on_dialout_stopped ( transport , data : Any): logger.info( f "Dial-out stopped: { data } " ) ​ on_dialout_error async callback Event emitted in the case of dial-out errors which are fatal and the service cannot proceed. For example, an error in SDP negotiation is fatal to the media/SIP pipeline and will result in dialout-error being triggered. Parameters: transport : The DailyTransport instance data : Dictionary containing error information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_error" ) async def on_dialout_error ( transport , data : Any): logger.error( f "Dial-out error: { data } " ) ​ on_dialout_warning async callback Event emitted there is a dial-out non-fatal error, such as the selected codec not being used and a fallback codec being utilized. Parameters: transport : The DailyTransport instance data : Dictionary containing warning information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_warning" ) async def on_dialout_warning ( transport , data : Any): logger.warning( f "Dial-out warning: { data } " ) ​ Transcription Events ​ on_transcription_message async callback Called when a transcription message is received. This includes both interim and final transcriptions. Parameters: transport : The DailyTransport instance message : Dictionary containing transcription data including. Learn more . Copy Ask AI @transport.event_handler ( "on_transcription_message" ) async def on_transcription_message ( transport , message : Dict[ str , Any]): participant_id = message.get( "participantId" ) text = message.get( "text" ) is_final = message[ "rawResponse" ][ "is_final" ] logger.info( f "Transcription from { participant_id } : { text } (final: { is_final } )" ) ​ Recording Events ​ on_recording_started async callback Called when a room recording starts successfully. Parameters: transport : The DailyTransport instance status : Dictionary containing recording status information. Learn more . Copy Ask AI @transport.event_handler ( "on_recording_started" ) async def on_recording_started ( transport , status : Dict[ str , Any]): logger.info( f "Recording started with status: { status } " ) ​ on_recording_stopped async callback Called when a room recording stops. Parameters: transport : The DailyTransport instance stream_id : String identifier of the stopped recording stream Copy Ask AI @transport.event_handler ( "on_recording_stopped" ) async def on_recording_stopped ( transport , stream_id : str ): logger.info( f "Recording stopped for stream: { stream_id } " ) ​ on_recording_error async callback Called when an error occurs during recording. Parameters: transport : The DailyTransport instance stream_id : String identifier of the recording stream message : Error message describing what went wrong Copy Ask AI @transport.event_handler ( "on_recording_error" ) async def on_recording_error ( transport , stream_id : str , message : str ): logger.error( f "Recording error for stream { stream_id } : { message } " ) ​ Error Events ​ on_error async callback Called when a transport error occurs. Parameters: transport : The DailyTransport instance error : String containing error details Copy Ask AI @transport.event_handler ( "on_error" ) async def on_error ( transport , error : str ): logger.error( f "Transport error: { error } " ) ​ Notes on Callbacks All callbacks are asynchronous Callbacks are executed in the order they were registered Multiple handlers can be registered for the same event Exceptions in callbacks are logged but don’t stop the transport Callbacks should be lightweight to avoid blocking the event loop Heavy processing should be offloaded to separate tasks ​ Frame Types ​ Input Frames ​ InputAudioRawFrame Frame Raw audio data from participants ​ UserImageRawFrame Frame Video frames from participants ​ Output Frames ​ OutputAudioRawFrame Frame Audio data to be sent ​ OutputImageRawFrame Frame Video frames to be sent ​ Methods ​ Room Management ​ participants method Returns a dictionary of all participants currently in the room. Copy Ask AI def participants () -> Dict[ str , Any] See participants for more details. ​ participant_counts method Returns participant count statistics for the room. Copy Ask AI def participant_counts () -> Dict[ str , int ] See participant_counts for more details. ​ send_message async method Sends a message to participants in the Daily room. Messages can be directed to all participants or targeted to a specific participant. Copy Ask AI async def send_message ( frame : TransportMessageFrame | TransportMessageUrgentFrame) Parameters: frame : A transport message frame containing the message to send. Can be either: TransportMessageFrame : Standard message TransportMessageUrgentFrame : High-priority message Message Structure: Messages should conform to the expected format for the receiving system. For RTVI messages, use: Copy Ask AI { "label" : "rtvi-ai" , # Required for RTVI protocol "type" : "message-type" , # Defines how the client will process it "id" : "unique-id" , # Unique identifier for the message "data" : { # Custom payload # Your custom data fields } } Example - Standard Message: Copy Ask AI import uuid # Send a standard message to all participants message_data = { "label" : "rtvi-ai" , "type" : "custom-message" , "id" : str (uuid.uuid4()), "data" : { "message" : "Hello everyone!" , "timestamp" : datetime.datetime.now().isoformat() } } await transport.send_message(TransportMessageFrame( message = message_data)) Example - Urgent Message to Specific Participant: Copy Ask AI # Send urgent message to specific participant urgent_data = { "label" : "rtvi-ai" , "type" : "server-message" , "id" : str (uuid.uuid4()), "data" : { "message" : "Private message" , "importance" : "high" } } await transport.send_message( DailyTransportMessageUrgentFrame( message = urgent_data, participant_id = "participant-123" ) ) The message will be delivered to: All participants if no specific participant_id is set Only the specified participant if participant_id is set Through the Daily app messaging system Can be received by registering an on_app_message handler Use TransportMessageUrgentFrame for high-priority messages that need immediate delivery. ​ Media Control ​ send_image async method Sends an image frame to the room. Copy Ask AI async def send_image ( frame : OutputImageRawFrame | SpriteFrame) -> None Parameters: frame : Image frame to send, either raw image data or sprite animation ​ send_audio async method Sends an audio frame to the room. Copy Ask AI async def send_audio ( frame : OutputAudioRawFrame) -> None Parameters: frame : Audio frame to send ​ Video Management ​ capture_participant_video async method Starts capturing video from a specific participant. Copy Ask AI async def capture_participant_video ( participant_id : str , framerate : int = 30 , video_source : str = "camera" , color_format : str = "RGB" ) -> None Parameters: participant_id : ID of the participant to capture framerate : Target frame rate (default: 30) video_source : Video source type (default: “camera”) color_format : Color format of the video (default: “RGB”) To request an image from the camera stream, set the framerate to 0 and push a UserImageRequestFrame . ​ Transcription Control ​ capture_participant_transcription async method Starts capturing and transcribing audio from a specific participant. Copy Ask AI async def capture_participant_transcription ( participant_id : str ) -> None Parameters: participant_id : ID of the participant to transcribe ​ update_transcription async method Updates the transcription configuration for specific participants or instances. Copy Ask AI async def update_transcription ( participants : List[ str ] | None = None , instance_id : str | None = None ) -> None Parameters: participants : Optional list of participant IDs to transcribe. If None, transcribes all participants. instance_id : Optional specific transcription instance ID to update Example: Copy Ask AI # Update transcription for specific participants await transport.update_transcription( participants = [ "participant-123" , "participant-456" ]) # Update specific transcription instance await transport.update_transcription( instance_id = "transcription-789" ) ​ Recording Control ​ start_recording async method Starts recording the room session. Copy Ask AI async def start_recording ( streaming_settings : Dict = None , stream_id : str = None , force_new : bool = None ) -> None Parameters: streaming_settings : Recording configuration settings stream_id : Optional stream identifier force_new : Force start a new recording See start_recording for more details. ​ stop_recording async method Stops an active recording. Copy Ask AI async def stop_recording ( stream_id : str = None ) -> None Parameters: stream_id : Optional stream identifier to stop specific recording See stop_recording for more details. ​ Dial-out Control ​ start_dialout async method Initiates a dial-out call. Copy Ask AI async def start_dialout ( settings : Dict = None ) -> None Parameters: settings : Dial-out configuration settings See start_dialout for more details. ​ stop_dialout async method Stops an active dial-out call. Copy Ask AI async def stop_dialout ( participant_id : str ) -> None Parameters: participant_id : ID of the participant to disconnect See stop_dialout for more details. ​ send_dtmf async method Sends DTMF tones in an existing dial-out session. Copy Ask AI async def send_dtmf ( settings : Dict = None ) -> None Parameters: settings : DTMF settings See send_dtmf for more details. ​ Subscription Management ​ update_subscriptions async method Updates subscriptions and subscription profiles. This function allows you to update subscription profiles and at the same time assign specific subscription profiles to a participant and even change specific settings for some participants. Copy Ask AI async def update_subscriptions ( participant_settings : Dict = None , profile_settings : Dict = None ) -> None Parameters: participant_settings : Per-participant subscription settings profile_settings : Global subscription profile settings See update_subscriptions for more details. ​ Participant Admin When using the DailyTransport, if your bot is given a meeting token with owner or participant admin permissions, it can remote-control other participants’ capabilities and permissions. With this control, the bot can modify settings for how others send and receive media, how they are seen in a session, and whether they have administrative controls. ​ update_remote_participants async method Updates permissions and settings for specific participants in the room. Copy Ask AI async def update_remote_participants ( remote_participants : Mapping[ str , Any] = None ) -> None Parameters: remote_participants : Dictionary mapping participant IDs to their updated settings This method allows fine-grained control over participant capabilities and permissions in the Daily room. Examples: Copy Ask AI # Mute a participant's microphone await transport.update_remote_participants({ "participant-123" : { "inputsEnabled" : { "microphone" : False } } }) # Revoke send permissions (mutes the participant and prevents them from unmuting) await transport.update_remote_participants({ "participant-123" : { "permissions" : { "canSend" : [] # Empty list revokes all send permissions } } }) # Configure selective audio routing await transport.update_remote_participants({ "participant-123" : { # the customer "permissions" : { "canReceive" : { "base" : False , # Don't receive from anyone by default "byUserId" : { "hold-music" : True # Only receive from hold music player } } } } }) # Set up user-to-user permissions mapping (e.g., when transferring calls) # Assumes customer was previously unable to hear anyone (they were on hold) and agent was speaking to bot await transport.update_remote_participants({ "participant-123" : { # the customer "permissions" : { "base" : False , "canReceive" : { "byUserId" : { "agent" : True }} } }, "participant-456" : { # the agent "permissions" : { "base" : False , "canReceive" : { "byUserId" : { "customer" : True }} } } }) # Promote a participant to be a participant admin await transport.update_remote_participants({ "participant-123" : { "permissions" : { "canAdmin" : [ "participants" ] } } }) Permissions Settings: For comprehensive details on permissions settings, see the Daily API documentation: daily-python’s update_remote_participants() documentation daily-js’s updateParticipant() -> updatePermission and participants().permissions ​ Chat messages ​ send_prebuilt_chat_message async method When testing with Daily Prebuilt, you can send messages to the Prebuilt chat using this method. Parameters: message (str): The chat message to send user_name (str): The name of the user sending the message See send_prebuilt_chat_message for more details. ​ Properties ​ participant_id string Returns the transport’s participant ID in the room. ​ room_url string Returns the transport’s room URL for the room currently in use. ​ Notes Supports real-time audio/video communication Handles WebRTC connection management Provides voice activity detection Includes transcription capabilities Manages participant interactions Supports recording and streaming Thread-safe processing All callbacks are asynchronous Multiple handlers can be registered for the same event Exceptions in callbacks are logged but don’t stop the transport Supported Services SmallWebRTCTransport On this page Overview Installation Configuration Constructor Parameters DailyParams Configuration Audio Output Configuration Audio Input Configuration Video Output Configuration Voice Activity Detection (VAD) Feature Settings Basic Usage Event Callbacks Connection Events Participant Events Communication Events Dial Events Dial-in Dial-out Transcription Events Recording Events Error Events Notes on Callbacks Frame Types Input Frames Output Frames Methods Room Management Media Control Video Management Transcription Control Recording Control Dial-out Control Subscription Management Participant Admin Chat messages Properties Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/transport_daily_d4b62035.txt b/transport_daily_d4b62035.txt
new file mode 100644
index 0000000000000000000000000000000000000000..8bca7ae012618c0ae5b0488c2f6dfeaee5b9ca56
--- /dev/null
+++ b/transport_daily_d4b62035.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/transport/daily#param-update-transcription
+Title: Daily WebRTC - Pipecat
+==================================================
+
+Daily WebRTC - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Transport Daily WebRTC Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Daily WebRTC SmallWebRTCTransport FastAPI WebSocket WebSocket Server Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview DailyTransport provides real-time audio and video communication capabilities using Daily’s WebRTC platform. It supports bidirectional audio/video streams, transcription, voice activity detection (VAD), and participant management. ​ Installation To use DailyTransport , install the required dependencies: Copy Ask AI pip install "pipecat-ai[daily]" You’ll also need to set up your Daily API key as an environment variable: DAILY_API_KEY . You can obtain a Daily API key by signing up at Daily . ​ Configuration ​ Constructor Parameters ​ room_url str required Daily room URL ​ token str | None Daily room token ​ bot_name str required Name of the bot in the room ​ params DailyParams default: "DailyParams()" Transport configuration parameters ​ DailyParams Configuration ​ api_url str default: "https://api.daily.co/v1" Daily API endpoint URL ​ api_key str default: "" Daily API key for authentication ​ Audio Output Configuration ​ audio_out_enabled bool default: "False" Enable audio output capabilities ​ audio_out_is_live bool default: "False" Enable live audio streaming mode ​ audio_out_sample_rate int default: "None" Audio output sample rate in Hz ​ audio_out_channels int default: "1" Number of audio output channels ​ audio_out_bitrate int default: "96000" Audio output bitrate in bits per second ​ Audio Input Configuration ​ audio_in_enabled bool default: "False" Enable audio input capabilities ​ audio_in_passthrough bool default: "False" When enabled, incoming audio frames are pushed downstream ​ audio_in_sample_rate int default: "None" Audio input sample rate in Hz ​ audio_in_channels int default: "1" Number of audio input channels ​ audio_in_filter Optional[BaseAudioFilter] default: "None" Audio filter for input processing. Supported filters are: KrispFilter() and NoisereduceFilter() . See the KrispFilter() reference docs and NoisereduceFilter() reference docs ` for more information. ​ Video Output Configuration ​ video_out_enabled bool default: "False" Enable video output capabilities ​ video_out_is_live bool default: "False" Enable live video streaming mode ​ video_out_width int default: "1024" Video output width in pixels ​ video_out_height int default: "768" Video output height in pixels ​ video_out_bitrate int default: "800000" Video output bitrate in bits per second ​ video_out_framerate int default: "30" Video output frame rate ​ video_out_color_format str default: "RGB" Video color format (RGB, BGR, etc.) ​ Voice Activity Detection (VAD) ​ vad_analyzer VADAnalyzer | None default: "None" Voice Activity Detection analyzer. You can set this to either SileroVADAnalyzer() or WebRTCVADAnalyzer() . SileroVADAnalyzer is the recommended option. Learn more about the SileroVADAnalyzer . ​ Feature Settings ​ dialin_settings Optional[DailyDialinSettings] default: "None" Configuration for dial-in functionality ​ transcription_enabled bool default: "False" Enable real-time transcription ​ transcription_settings DailyTranscriptionSettings Configuration for transcription features Copy Ask AI class DailyTranscriptionSettings ( BaseModel ): language: str = "en" # Default language model: str = "nova-2-general" # Transcription model profanity_filter: bool = True # Filter profanity redact: bool = False # Redact sensitive information endpointing: bool = True # Enable speech endpointing punctuate: bool = True # Add punctuation includeRawResponse: bool = True # Include raw API response extra: Mapping[ str , Any] = { "interim_results" : True # Provide any Deepgram-specific settings } ​ Basic Usage Copy Ask AI from pipecat.transports.services.daily import DailyTransport, DailyParams from pipecat.audio.vad.silero import SileroVADAnalyzer # Configure transport transport = DailyTransport( room_url = "https://your-domain.daily.co/room-name" , token = "your-room-token" , bot_name = "AI Assistant" , params = DailyParams( audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer( params = VADParams( stop_secs = 0.5 )), ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), # Handle incoming audio/video stt, # Speech-to-text llm, # Language model tts, # Text-to-speech transport.output() # Handle outgoing audio/video ]) # Use transport event handlers to manage the call lifecycle # Event handler for when the user joins the room: @transport.event_handler ( "on_first_participant_joined" ) async def on_first_participant_joined ( transport , participant ): # Start transcription for the user await transport.capture_participant_transcription(participant[ "id" ]) # Kick off the conversation. await task.queue_frames([context_aggregator.user().get_context_frame()]) # Event handler for when the user leaves the room: @transport.event_handler ( "on_participant_left" ) async def on_participant_left ( transport , participant , reason ): # Cancel the pipeline, which stops processing and removes the bot from the room await task.cancel() ​ Event Callbacks DailyTransport provides a comprehensive callback system for handling various events. Register callbacks using the @transport.event_handler() decorator. ​ Connection Events ​ on_joined async callback Called when the bot successfully joins the room. Parameters: transport : The DailyTransport instance data : Dictionary containing room join information Copy Ask AI @transport.event_handler ( "on_joined" ) async def on_joined ( transport , data : Dict[ str , Any]): logger.info( f "Joined room with data: { data } " ) ​ on_left async callback Called when the bot leaves the room. Parameters: transport : The DailyTransport instance Copy Ask AI @transport.event_handler ( "on_left" ) async def on_left ( transport ): logger.info( "Left room" ) ​ Participant Events ​ on_first_participant_joined async callback Called when the first participant joins the room. Useful for initializing conversations. Parameters: transport : The DailyTransport instance participant : Dictionary containing participant information Copy Ask AI @transport.event_handler ( "on_first_participant_joined" ) async def on_first_participant_joined ( transport , participant : Dict[ str , Any]): await transport.capture_participant_transcription(participant[ "id" ]) await task.queue_frames([LLMMessagesFrame(initial_messages)]) ​ on_participant_joined async callback Called when any participant joins the room. Parameters: transport : The DailyTransport instance participant : Dictionary containing participant information Copy Ask AI @transport.event_handler ( "on_participant_joined" ) async def on_participant_joined ( transport , participant : Dict[ str , Any]): logger.info( f "Participant joined: { participant[ 'id' ] } " ) ​ on_participant_left async callback Called when a participant leaves the room. Parameters: transport : The DailyTransport instance participant : Dictionary containing participant information reason : String describing why the participant left, “leftCall” | “hidden” Copy Ask AI @transport.event_handler ( "on_participant_left" ) async def on_participant_left ( transport , participant : Dict[ str , Any], reason : str ): logger.info( f "Participant { participant[ 'id' ] } left: { reason } " ) # Cancel the pipeline task to stop processing and remove the bot from the Daily room await task.cancel() ​ on_participant_updated async callback Event emitted when a participant is updated. This can mean either the participant’s metadata was updated, or the tracks belonging to the participant changed. Parameters: transport : The DailyTransport instance participant : Dictionary containing updated participant information Copy Ask AI @transport.event_handler ( "on_participant_updated" ) async def on_participant_updated ( transport , participant : Dict[ str , Any]): logger.info( f "Participant updated: { participant } " ) ​ Communication Events ​ on_app_message async callback Event emitted when a custom app message is received from another participant or via the REST API. Parameters: transport : The DailyTransport instance message : The message content (any type) sender : String identifier of the message sender Copy Ask AI @transport.event_handler ( "on_app_message" ) async def on_app_message ( transport , message : Any, sender : str ): logger.info( f "Message from { sender } : { message } " ) ​ on_call_state_updated async callback Event emitted when the call state changes, normally as a consequence of joining or leaving the call. Parameters: transport : The DailyTransport instance state : String representing the new call state. Learn more about call states . Copy Ask AI @transport.event_handler ( "on_call_state_updated" ) async def on_call_state_updated ( transport , state : str ): logger.info( f "Call state updated: { state } " ) ​ Dial Events ​ Dial-in ​ on_dialin_ready async callback Event emitted when dial-in is ready. This happens after the room has connected to the SIP endpoint and the system is ready to receive dial-in calls. Parameters: transport : The DailyTransport instance sip_endpoint : String containing the SIP endpoint information Copy Ask AI @transport.event_handler ( "on_dialin_ready" ) async def on_dialin_ready ( transport , sip_endpoint : str ): logger.info( f "Dial-in ready at: { sip_endpoint } " ) ​ on_dialin_connected async callback Event emitted when the session with the dial-in remote end is established (i.e. SIP endpoint or PSTN are connectd to the Daily room). Note: connected does not mean media (audio or video) has started flowing between the room and PSTN, it means the room received the connection request and both endpoints are negotiating the media flow. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. See DialinConnectedEvent . Copy Ask AI @transport.event_handler ( "on_dialin_connected" ) async def on_dialin_connected ( transport , data : Any): logger.info( f "Dial-in connected: { data } " ) ​ on_dialin_stopped async callback Event emitted when the dial-in remote end disconnects the call. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. See DialinStoppedEvent . Copy Ask AI @transport.event_handler ( "on_dialin_stopped" ) async def on_dialin_stopped ( transport , data : Any): logger.info( f "Dial-in stopped: { data } " ) ​ on_dialin_error async callback Event emitted in the case of dial-in errors which are fatal and the service cannot proceed. For example, an error in SDP negotiation is fatal to the media/SIP pipeline and will result in dialin-error being triggered. Parameters: transport : The DailyTransport instance data : Dictionary containing error information. See DialinEvent . Copy Ask AI @transport.event_handler ( "on_dialin_error" ) async def on_dialin_error ( transport , data : Any): logger.error( f "Dial-in error: { data } " ) ​ on_dialin_warning async callback Event emitted there is a dial-in non-fatal error, such as the selected codec not being used and a fallback codec being utilized. Parameters: transport : The DailyTransport instance data : Dictionary containing warning information. See DialinEvent . Copy Ask AI @transport.event_handler ( "on_dialin_warning" ) async def on_dialin_warning ( transport , data : Any): logger.warning( f "Dial-in warning: { data } " ) ​ Dial-out ​ on_dialout_answered async callback Event emitted when the session with the dial-out remote end is answered. Parameters: transport : The DailyTransport instance data : Dictionary containing call information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_answered" ) async def on_dialout_answered ( transport , data : Any): logger.info( f "Dial-out answered: { data } " ) ​ on_dialout_connected async callback Event emitted when the session with the dial-out remote end is established. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_connected" ) async def on_dialout_connected ( transport , data : Any): logger.info( f "Dial-out connected: { data } " ) ​ on_dialout_stopped async callback Event emitted when the dial-out session is stopped. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_stopped" ) async def on_dialout_stopped ( transport , data : Any): logger.info( f "Dial-out stopped: { data } " ) ​ on_dialout_error async callback Event emitted in the case of dial-out errors which are fatal and the service cannot proceed. For example, an error in SDP negotiation is fatal to the media/SIP pipeline and will result in dialout-error being triggered. Parameters: transport : The DailyTransport instance data : Dictionary containing error information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_error" ) async def on_dialout_error ( transport , data : Any): logger.error( f "Dial-out error: { data } " ) ​ on_dialout_warning async callback Event emitted there is a dial-out non-fatal error, such as the selected codec not being used and a fallback codec being utilized. Parameters: transport : The DailyTransport instance data : Dictionary containing warning information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_warning" ) async def on_dialout_warning ( transport , data : Any): logger.warning( f "Dial-out warning: { data } " ) ​ Transcription Events ​ on_transcription_message async callback Called when a transcription message is received. This includes both interim and final transcriptions. Parameters: transport : The DailyTransport instance message : Dictionary containing transcription data including. Learn more . Copy Ask AI @transport.event_handler ( "on_transcription_message" ) async def on_transcription_message ( transport , message : Dict[ str , Any]): participant_id = message.get( "participantId" ) text = message.get( "text" ) is_final = message[ "rawResponse" ][ "is_final" ] logger.info( f "Transcription from { participant_id } : { text } (final: { is_final } )" ) ​ Recording Events ​ on_recording_started async callback Called when a room recording starts successfully. Parameters: transport : The DailyTransport instance status : Dictionary containing recording status information. Learn more . Copy Ask AI @transport.event_handler ( "on_recording_started" ) async def on_recording_started ( transport , status : Dict[ str , Any]): logger.info( f "Recording started with status: { status } " ) ​ on_recording_stopped async callback Called when a room recording stops. Parameters: transport : The DailyTransport instance stream_id : String identifier of the stopped recording stream Copy Ask AI @transport.event_handler ( "on_recording_stopped" ) async def on_recording_stopped ( transport , stream_id : str ): logger.info( f "Recording stopped for stream: { stream_id } " ) ​ on_recording_error async callback Called when an error occurs during recording. Parameters: transport : The DailyTransport instance stream_id : String identifier of the recording stream message : Error message describing what went wrong Copy Ask AI @transport.event_handler ( "on_recording_error" ) async def on_recording_error ( transport , stream_id : str , message : str ): logger.error( f "Recording error for stream { stream_id } : { message } " ) ​ Error Events ​ on_error async callback Called when a transport error occurs. Parameters: transport : The DailyTransport instance error : String containing error details Copy Ask AI @transport.event_handler ( "on_error" ) async def on_error ( transport , error : str ): logger.error( f "Transport error: { error } " ) ​ Notes on Callbacks All callbacks are asynchronous Callbacks are executed in the order they were registered Multiple handlers can be registered for the same event Exceptions in callbacks are logged but don’t stop the transport Callbacks should be lightweight to avoid blocking the event loop Heavy processing should be offloaded to separate tasks ​ Frame Types ​ Input Frames ​ InputAudioRawFrame Frame Raw audio data from participants ​ UserImageRawFrame Frame Video frames from participants ​ Output Frames ​ OutputAudioRawFrame Frame Audio data to be sent ​ OutputImageRawFrame Frame Video frames to be sent ​ Methods ​ Room Management ​ participants method Returns a dictionary of all participants currently in the room. Copy Ask AI def participants () -> Dict[ str , Any] See participants for more details. ​ participant_counts method Returns participant count statistics for the room. Copy Ask AI def participant_counts () -> Dict[ str , int ] See participant_counts for more details. ​ send_message async method Sends a message to participants in the Daily room. Messages can be directed to all participants or targeted to a specific participant. Copy Ask AI async def send_message ( frame : TransportMessageFrame | TransportMessageUrgentFrame) Parameters: frame : A transport message frame containing the message to send. Can be either: TransportMessageFrame : Standard message TransportMessageUrgentFrame : High-priority message Message Structure: Messages should conform to the expected format for the receiving system. For RTVI messages, use: Copy Ask AI { "label" : "rtvi-ai" , # Required for RTVI protocol "type" : "message-type" , # Defines how the client will process it "id" : "unique-id" , # Unique identifier for the message "data" : { # Custom payload # Your custom data fields } } Example - Standard Message: Copy Ask AI import uuid # Send a standard message to all participants message_data = { "label" : "rtvi-ai" , "type" : "custom-message" , "id" : str (uuid.uuid4()), "data" : { "message" : "Hello everyone!" , "timestamp" : datetime.datetime.now().isoformat() } } await transport.send_message(TransportMessageFrame( message = message_data)) Example - Urgent Message to Specific Participant: Copy Ask AI # Send urgent message to specific participant urgent_data = { "label" : "rtvi-ai" , "type" : "server-message" , "id" : str (uuid.uuid4()), "data" : { "message" : "Private message" , "importance" : "high" } } await transport.send_message( DailyTransportMessageUrgentFrame( message = urgent_data, participant_id = "participant-123" ) ) The message will be delivered to: All participants if no specific participant_id is set Only the specified participant if participant_id is set Through the Daily app messaging system Can be received by registering an on_app_message handler Use TransportMessageUrgentFrame for high-priority messages that need immediate delivery. ​ Media Control ​ send_image async method Sends an image frame to the room. Copy Ask AI async def send_image ( frame : OutputImageRawFrame | SpriteFrame) -> None Parameters: frame : Image frame to send, either raw image data or sprite animation ​ send_audio async method Sends an audio frame to the room. Copy Ask AI async def send_audio ( frame : OutputAudioRawFrame) -> None Parameters: frame : Audio frame to send ​ Video Management ​ capture_participant_video async method Starts capturing video from a specific participant. Copy Ask AI async def capture_participant_video ( participant_id : str , framerate : int = 30 , video_source : str = "camera" , color_format : str = "RGB" ) -> None Parameters: participant_id : ID of the participant to capture framerate : Target frame rate (default: 30) video_source : Video source type (default: “camera”) color_format : Color format of the video (default: “RGB”) To request an image from the camera stream, set the framerate to 0 and push a UserImageRequestFrame . ​ Transcription Control ​ capture_participant_transcription async method Starts capturing and transcribing audio from a specific participant. Copy Ask AI async def capture_participant_transcription ( participant_id : str ) -> None Parameters: participant_id : ID of the participant to transcribe ​ update_transcription async method Updates the transcription configuration for specific participants or instances. Copy Ask AI async def update_transcription ( participants : List[ str ] | None = None , instance_id : str | None = None ) -> None Parameters: participants : Optional list of participant IDs to transcribe. If None, transcribes all participants. instance_id : Optional specific transcription instance ID to update Example: Copy Ask AI # Update transcription for specific participants await transport.update_transcription( participants = [ "participant-123" , "participant-456" ]) # Update specific transcription instance await transport.update_transcription( instance_id = "transcription-789" ) ​ Recording Control ​ start_recording async method Starts recording the room session. Copy Ask AI async def start_recording ( streaming_settings : Dict = None , stream_id : str = None , force_new : bool = None ) -> None Parameters: streaming_settings : Recording configuration settings stream_id : Optional stream identifier force_new : Force start a new recording See start_recording for more details. ​ stop_recording async method Stops an active recording. Copy Ask AI async def stop_recording ( stream_id : str = None ) -> None Parameters: stream_id : Optional stream identifier to stop specific recording See stop_recording for more details. ​ Dial-out Control ​ start_dialout async method Initiates a dial-out call. Copy Ask AI async def start_dialout ( settings : Dict = None ) -> None Parameters: settings : Dial-out configuration settings See start_dialout for more details. ​ stop_dialout async method Stops an active dial-out call. Copy Ask AI async def stop_dialout ( participant_id : str ) -> None Parameters: participant_id : ID of the participant to disconnect See stop_dialout for more details. ​ send_dtmf async method Sends DTMF tones in an existing dial-out session. Copy Ask AI async def send_dtmf ( settings : Dict = None ) -> None Parameters: settings : DTMF settings See send_dtmf for more details. ​ Subscription Management ​ update_subscriptions async method Updates subscriptions and subscription profiles. This function allows you to update subscription profiles and at the same time assign specific subscription profiles to a participant and even change specific settings for some participants. Copy Ask AI async def update_subscriptions ( participant_settings : Dict = None , profile_settings : Dict = None ) -> None Parameters: participant_settings : Per-participant subscription settings profile_settings : Global subscription profile settings See update_subscriptions for more details. ​ Participant Admin When using the DailyTransport, if your bot is given a meeting token with owner or participant admin permissions, it can remote-control other participants’ capabilities and permissions. With this control, the bot can modify settings for how others send and receive media, how they are seen in a session, and whether they have administrative controls. ​ update_remote_participants async method Updates permissions and settings for specific participants in the room. Copy Ask AI async def update_remote_participants ( remote_participants : Mapping[ str , Any] = None ) -> None Parameters: remote_participants : Dictionary mapping participant IDs to their updated settings This method allows fine-grained control over participant capabilities and permissions in the Daily room. Examples: Copy Ask AI # Mute a participant's microphone await transport.update_remote_participants({ "participant-123" : { "inputsEnabled" : { "microphone" : False } } }) # Revoke send permissions (mutes the participant and prevents them from unmuting) await transport.update_remote_participants({ "participant-123" : { "permissions" : { "canSend" : [] # Empty list revokes all send permissions } } }) # Configure selective audio routing await transport.update_remote_participants({ "participant-123" : { # the customer "permissions" : { "canReceive" : { "base" : False , # Don't receive from anyone by default "byUserId" : { "hold-music" : True # Only receive from hold music player } } } } }) # Set up user-to-user permissions mapping (e.g., when transferring calls) # Assumes customer was previously unable to hear anyone (they were on hold) and agent was speaking to bot await transport.update_remote_participants({ "participant-123" : { # the customer "permissions" : { "base" : False , "canReceive" : { "byUserId" : { "agent" : True }} } }, "participant-456" : { # the agent "permissions" : { "base" : False , "canReceive" : { "byUserId" : { "customer" : True }} } } }) # Promote a participant to be a participant admin await transport.update_remote_participants({ "participant-123" : { "permissions" : { "canAdmin" : [ "participants" ] } } }) Permissions Settings: For comprehensive details on permissions settings, see the Daily API documentation: daily-python’s update_remote_participants() documentation daily-js’s updateParticipant() -> updatePermission and participants().permissions ​ Chat messages ​ send_prebuilt_chat_message async method When testing with Daily Prebuilt, you can send messages to the Prebuilt chat using this method. Parameters: message (str): The chat message to send user_name (str): The name of the user sending the message See send_prebuilt_chat_message for more details. ​ Properties ​ participant_id string Returns the transport’s participant ID in the room. ​ room_url string Returns the transport’s room URL for the room currently in use. ​ Notes Supports real-time audio/video communication Handles WebRTC connection management Provides voice activity detection Includes transcription capabilities Manages participant interactions Supports recording and streaming Thread-safe processing All callbacks are asynchronous Multiple handlers can be registered for the same event Exceptions in callbacks are logged but don’t stop the transport Supported Services SmallWebRTCTransport On this page Overview Installation Configuration Constructor Parameters DailyParams Configuration Audio Output Configuration Audio Input Configuration Video Output Configuration Voice Activity Detection (VAD) Feature Settings Basic Usage Event Callbacks Connection Events Participant Events Communication Events Dial Events Dial-in Dial-out Transcription Events Recording Events Error Events Notes on Callbacks Frame Types Input Frames Output Frames Methods Room Management Media Control Video Management Transcription Control Recording Control Dial-out Control Subscription Management Participant Admin Chat messages Properties Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/transport_daily_da33d840.txt b/transport_daily_da33d840.txt
new file mode 100644
index 0000000000000000000000000000000000000000..764e4dd8e9f77f3b7629fbaa51f9a04866cf1507
--- /dev/null
+++ b/transport_daily_da33d840.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/transport/daily#notes-on-callbacks
+Title: Daily WebRTC - Pipecat
+==================================================
+
+Daily WebRTC - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Transport Daily WebRTC Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Daily WebRTC SmallWebRTCTransport FastAPI WebSocket WebSocket Server Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview DailyTransport provides real-time audio and video communication capabilities using Daily’s WebRTC platform. It supports bidirectional audio/video streams, transcription, voice activity detection (VAD), and participant management. ​ Installation To use DailyTransport , install the required dependencies: Copy Ask AI pip install "pipecat-ai[daily]" You’ll also need to set up your Daily API key as an environment variable: DAILY_API_KEY . You can obtain a Daily API key by signing up at Daily . ​ Configuration ​ Constructor Parameters ​ room_url str required Daily room URL ​ token str | None Daily room token ​ bot_name str required Name of the bot in the room ​ params DailyParams default: "DailyParams()" Transport configuration parameters ​ DailyParams Configuration ​ api_url str default: "https://api.daily.co/v1" Daily API endpoint URL ​ api_key str default: "" Daily API key for authentication ​ Audio Output Configuration ​ audio_out_enabled bool default: "False" Enable audio output capabilities ​ audio_out_is_live bool default: "False" Enable live audio streaming mode ​ audio_out_sample_rate int default: "None" Audio output sample rate in Hz ​ audio_out_channels int default: "1" Number of audio output channels ​ audio_out_bitrate int default: "96000" Audio output bitrate in bits per second ​ Audio Input Configuration ​ audio_in_enabled bool default: "False" Enable audio input capabilities ​ audio_in_passthrough bool default: "False" When enabled, incoming audio frames are pushed downstream ​ audio_in_sample_rate int default: "None" Audio input sample rate in Hz ​ audio_in_channels int default: "1" Number of audio input channels ​ audio_in_filter Optional[BaseAudioFilter] default: "None" Audio filter for input processing. Supported filters are: KrispFilter() and NoisereduceFilter() . See the KrispFilter() reference docs and NoisereduceFilter() reference docs ` for more information. ​ Video Output Configuration ​ video_out_enabled bool default: "False" Enable video output capabilities ​ video_out_is_live bool default: "False" Enable live video streaming mode ​ video_out_width int default: "1024" Video output width in pixels ​ video_out_height int default: "768" Video output height in pixels ​ video_out_bitrate int default: "800000" Video output bitrate in bits per second ​ video_out_framerate int default: "30" Video output frame rate ​ video_out_color_format str default: "RGB" Video color format (RGB, BGR, etc.) ​ Voice Activity Detection (VAD) ​ vad_analyzer VADAnalyzer | None default: "None" Voice Activity Detection analyzer. You can set this to either SileroVADAnalyzer() or WebRTCVADAnalyzer() . SileroVADAnalyzer is the recommended option. Learn more about the SileroVADAnalyzer . ​ Feature Settings ​ dialin_settings Optional[DailyDialinSettings] default: "None" Configuration for dial-in functionality ​ transcription_enabled bool default: "False" Enable real-time transcription ​ transcription_settings DailyTranscriptionSettings Configuration for transcription features Copy Ask AI class DailyTranscriptionSettings ( BaseModel ): language: str = "en" # Default language model: str = "nova-2-general" # Transcription model profanity_filter: bool = True # Filter profanity redact: bool = False # Redact sensitive information endpointing: bool = True # Enable speech endpointing punctuate: bool = True # Add punctuation includeRawResponse: bool = True # Include raw API response extra: Mapping[ str , Any] = { "interim_results" : True # Provide any Deepgram-specific settings } ​ Basic Usage Copy Ask AI from pipecat.transports.services.daily import DailyTransport, DailyParams from pipecat.audio.vad.silero import SileroVADAnalyzer # Configure transport transport = DailyTransport( room_url = "https://your-domain.daily.co/room-name" , token = "your-room-token" , bot_name = "AI Assistant" , params = DailyParams( audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer( params = VADParams( stop_secs = 0.5 )), ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), # Handle incoming audio/video stt, # Speech-to-text llm, # Language model tts, # Text-to-speech transport.output() # Handle outgoing audio/video ]) # Use transport event handlers to manage the call lifecycle # Event handler for when the user joins the room: @transport.event_handler ( "on_first_participant_joined" ) async def on_first_participant_joined ( transport , participant ): # Start transcription for the user await transport.capture_participant_transcription(participant[ "id" ]) # Kick off the conversation. await task.queue_frames([context_aggregator.user().get_context_frame()]) # Event handler for when the user leaves the room: @transport.event_handler ( "on_participant_left" ) async def on_participant_left ( transport , participant , reason ): # Cancel the pipeline, which stops processing and removes the bot from the room await task.cancel() ​ Event Callbacks DailyTransport provides a comprehensive callback system for handling various events. Register callbacks using the @transport.event_handler() decorator. ​ Connection Events ​ on_joined async callback Called when the bot successfully joins the room. Parameters: transport : The DailyTransport instance data : Dictionary containing room join information Copy Ask AI @transport.event_handler ( "on_joined" ) async def on_joined ( transport , data : Dict[ str , Any]): logger.info( f "Joined room with data: { data } " ) ​ on_left async callback Called when the bot leaves the room. Parameters: transport : The DailyTransport instance Copy Ask AI @transport.event_handler ( "on_left" ) async def on_left ( transport ): logger.info( "Left room" ) ​ Participant Events ​ on_first_participant_joined async callback Called when the first participant joins the room. Useful for initializing conversations. Parameters: transport : The DailyTransport instance participant : Dictionary containing participant information Copy Ask AI @transport.event_handler ( "on_first_participant_joined" ) async def on_first_participant_joined ( transport , participant : Dict[ str , Any]): await transport.capture_participant_transcription(participant[ "id" ]) await task.queue_frames([LLMMessagesFrame(initial_messages)]) ​ on_participant_joined async callback Called when any participant joins the room. Parameters: transport : The DailyTransport instance participant : Dictionary containing participant information Copy Ask AI @transport.event_handler ( "on_participant_joined" ) async def on_participant_joined ( transport , participant : Dict[ str , Any]): logger.info( f "Participant joined: { participant[ 'id' ] } " ) ​ on_participant_left async callback Called when a participant leaves the room. Parameters: transport : The DailyTransport instance participant : Dictionary containing participant information reason : String describing why the participant left, “leftCall” | “hidden” Copy Ask AI @transport.event_handler ( "on_participant_left" ) async def on_participant_left ( transport , participant : Dict[ str , Any], reason : str ): logger.info( f "Participant { participant[ 'id' ] } left: { reason } " ) # Cancel the pipeline task to stop processing and remove the bot from the Daily room await task.cancel() ​ on_participant_updated async callback Event emitted when a participant is updated. This can mean either the participant’s metadata was updated, or the tracks belonging to the participant changed. Parameters: transport : The DailyTransport instance participant : Dictionary containing updated participant information Copy Ask AI @transport.event_handler ( "on_participant_updated" ) async def on_participant_updated ( transport , participant : Dict[ str , Any]): logger.info( f "Participant updated: { participant } " ) ​ Communication Events ​ on_app_message async callback Event emitted when a custom app message is received from another participant or via the REST API. Parameters: transport : The DailyTransport instance message : The message content (any type) sender : String identifier of the message sender Copy Ask AI @transport.event_handler ( "on_app_message" ) async def on_app_message ( transport , message : Any, sender : str ): logger.info( f "Message from { sender } : { message } " ) ​ on_call_state_updated async callback Event emitted when the call state changes, normally as a consequence of joining or leaving the call. Parameters: transport : The DailyTransport instance state : String representing the new call state. Learn more about call states . Copy Ask AI @transport.event_handler ( "on_call_state_updated" ) async def on_call_state_updated ( transport , state : str ): logger.info( f "Call state updated: { state } " ) ​ Dial Events ​ Dial-in ​ on_dialin_ready async callback Event emitted when dial-in is ready. This happens after the room has connected to the SIP endpoint and the system is ready to receive dial-in calls. Parameters: transport : The DailyTransport instance sip_endpoint : String containing the SIP endpoint information Copy Ask AI @transport.event_handler ( "on_dialin_ready" ) async def on_dialin_ready ( transport , sip_endpoint : str ): logger.info( f "Dial-in ready at: { sip_endpoint } " ) ​ on_dialin_connected async callback Event emitted when the session with the dial-in remote end is established (i.e. SIP endpoint or PSTN are connectd to the Daily room). Note: connected does not mean media (audio or video) has started flowing between the room and PSTN, it means the room received the connection request and both endpoints are negotiating the media flow. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. See DialinConnectedEvent . Copy Ask AI @transport.event_handler ( "on_dialin_connected" ) async def on_dialin_connected ( transport , data : Any): logger.info( f "Dial-in connected: { data } " ) ​ on_dialin_stopped async callback Event emitted when the dial-in remote end disconnects the call. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. See DialinStoppedEvent . Copy Ask AI @transport.event_handler ( "on_dialin_stopped" ) async def on_dialin_stopped ( transport , data : Any): logger.info( f "Dial-in stopped: { data } " ) ​ on_dialin_error async callback Event emitted in the case of dial-in errors which are fatal and the service cannot proceed. For example, an error in SDP negotiation is fatal to the media/SIP pipeline and will result in dialin-error being triggered. Parameters: transport : The DailyTransport instance data : Dictionary containing error information. See DialinEvent . Copy Ask AI @transport.event_handler ( "on_dialin_error" ) async def on_dialin_error ( transport , data : Any): logger.error( f "Dial-in error: { data } " ) ​ on_dialin_warning async callback Event emitted there is a dial-in non-fatal error, such as the selected codec not being used and a fallback codec being utilized. Parameters: transport : The DailyTransport instance data : Dictionary containing warning information. See DialinEvent . Copy Ask AI @transport.event_handler ( "on_dialin_warning" ) async def on_dialin_warning ( transport , data : Any): logger.warning( f "Dial-in warning: { data } " ) ​ Dial-out ​ on_dialout_answered async callback Event emitted when the session with the dial-out remote end is answered. Parameters: transport : The DailyTransport instance data : Dictionary containing call information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_answered" ) async def on_dialout_answered ( transport , data : Any): logger.info( f "Dial-out answered: { data } " ) ​ on_dialout_connected async callback Event emitted when the session with the dial-out remote end is established. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_connected" ) async def on_dialout_connected ( transport , data : Any): logger.info( f "Dial-out connected: { data } " ) ​ on_dialout_stopped async callback Event emitted when the dial-out session is stopped. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_stopped" ) async def on_dialout_stopped ( transport , data : Any): logger.info( f "Dial-out stopped: { data } " ) ​ on_dialout_error async callback Event emitted in the case of dial-out errors which are fatal and the service cannot proceed. For example, an error in SDP negotiation is fatal to the media/SIP pipeline and will result in dialout-error being triggered. Parameters: transport : The DailyTransport instance data : Dictionary containing error information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_error" ) async def on_dialout_error ( transport , data : Any): logger.error( f "Dial-out error: { data } " ) ​ on_dialout_warning async callback Event emitted there is a dial-out non-fatal error, such as the selected codec not being used and a fallback codec being utilized. Parameters: transport : The DailyTransport instance data : Dictionary containing warning information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_warning" ) async def on_dialout_warning ( transport , data : Any): logger.warning( f "Dial-out warning: { data } " ) ​ Transcription Events ​ on_transcription_message async callback Called when a transcription message is received. This includes both interim and final transcriptions. Parameters: transport : The DailyTransport instance message : Dictionary containing transcription data including. Learn more . Copy Ask AI @transport.event_handler ( "on_transcription_message" ) async def on_transcription_message ( transport , message : Dict[ str , Any]): participant_id = message.get( "participantId" ) text = message.get( "text" ) is_final = message[ "rawResponse" ][ "is_final" ] logger.info( f "Transcription from { participant_id } : { text } (final: { is_final } )" ) ​ Recording Events ​ on_recording_started async callback Called when a room recording starts successfully. Parameters: transport : The DailyTransport instance status : Dictionary containing recording status information. Learn more . Copy Ask AI @transport.event_handler ( "on_recording_started" ) async def on_recording_started ( transport , status : Dict[ str , Any]): logger.info( f "Recording started with status: { status } " ) ​ on_recording_stopped async callback Called when a room recording stops. Parameters: transport : The DailyTransport instance stream_id : String identifier of the stopped recording stream Copy Ask AI @transport.event_handler ( "on_recording_stopped" ) async def on_recording_stopped ( transport , stream_id : str ): logger.info( f "Recording stopped for stream: { stream_id } " ) ​ on_recording_error async callback Called when an error occurs during recording. Parameters: transport : The DailyTransport instance stream_id : String identifier of the recording stream message : Error message describing what went wrong Copy Ask AI @transport.event_handler ( "on_recording_error" ) async def on_recording_error ( transport , stream_id : str , message : str ): logger.error( f "Recording error for stream { stream_id } : { message } " ) ​ Error Events ​ on_error async callback Called when a transport error occurs. Parameters: transport : The DailyTransport instance error : String containing error details Copy Ask AI @transport.event_handler ( "on_error" ) async def on_error ( transport , error : str ): logger.error( f "Transport error: { error } " ) ​ Notes on Callbacks All callbacks are asynchronous Callbacks are executed in the order they were registered Multiple handlers can be registered for the same event Exceptions in callbacks are logged but don’t stop the transport Callbacks should be lightweight to avoid blocking the event loop Heavy processing should be offloaded to separate tasks ​ Frame Types ​ Input Frames ​ InputAudioRawFrame Frame Raw audio data from participants ​ UserImageRawFrame Frame Video frames from participants ​ Output Frames ​ OutputAudioRawFrame Frame Audio data to be sent ​ OutputImageRawFrame Frame Video frames to be sent ​ Methods ​ Room Management ​ participants method Returns a dictionary of all participants currently in the room. Copy Ask AI def participants () -> Dict[ str , Any] See participants for more details. ​ participant_counts method Returns participant count statistics for the room. Copy Ask AI def participant_counts () -> Dict[ str , int ] See participant_counts for more details. ​ send_message async method Sends a message to participants in the Daily room. Messages can be directed to all participants or targeted to a specific participant. Copy Ask AI async def send_message ( frame : TransportMessageFrame | TransportMessageUrgentFrame) Parameters: frame : A transport message frame containing the message to send. Can be either: TransportMessageFrame : Standard message TransportMessageUrgentFrame : High-priority message Message Structure: Messages should conform to the expected format for the receiving system. For RTVI messages, use: Copy Ask AI { "label" : "rtvi-ai" , # Required for RTVI protocol "type" : "message-type" , # Defines how the client will process it "id" : "unique-id" , # Unique identifier for the message "data" : { # Custom payload # Your custom data fields } } Example - Standard Message: Copy Ask AI import uuid # Send a standard message to all participants message_data = { "label" : "rtvi-ai" , "type" : "custom-message" , "id" : str (uuid.uuid4()), "data" : { "message" : "Hello everyone!" , "timestamp" : datetime.datetime.now().isoformat() } } await transport.send_message(TransportMessageFrame( message = message_data)) Example - Urgent Message to Specific Participant: Copy Ask AI # Send urgent message to specific participant urgent_data = { "label" : "rtvi-ai" , "type" : "server-message" , "id" : str (uuid.uuid4()), "data" : { "message" : "Private message" , "importance" : "high" } } await transport.send_message( DailyTransportMessageUrgentFrame( message = urgent_data, participant_id = "participant-123" ) ) The message will be delivered to: All participants if no specific participant_id is set Only the specified participant if participant_id is set Through the Daily app messaging system Can be received by registering an on_app_message handler Use TransportMessageUrgentFrame for high-priority messages that need immediate delivery. ​ Media Control ​ send_image async method Sends an image frame to the room. Copy Ask AI async def send_image ( frame : OutputImageRawFrame | SpriteFrame) -> None Parameters: frame : Image frame to send, either raw image data or sprite animation ​ send_audio async method Sends an audio frame to the room. Copy Ask AI async def send_audio ( frame : OutputAudioRawFrame) -> None Parameters: frame : Audio frame to send ​ Video Management ​ capture_participant_video async method Starts capturing video from a specific participant. Copy Ask AI async def capture_participant_video ( participant_id : str , framerate : int = 30 , video_source : str = "camera" , color_format : str = "RGB" ) -> None Parameters: participant_id : ID of the participant to capture framerate : Target frame rate (default: 30) video_source : Video source type (default: “camera”) color_format : Color format of the video (default: “RGB”) To request an image from the camera stream, set the framerate to 0 and push a UserImageRequestFrame . ​ Transcription Control ​ capture_participant_transcription async method Starts capturing and transcribing audio from a specific participant. Copy Ask AI async def capture_participant_transcription ( participant_id : str ) -> None Parameters: participant_id : ID of the participant to transcribe ​ update_transcription async method Updates the transcription configuration for specific participants or instances. Copy Ask AI async def update_transcription ( participants : List[ str ] | None = None , instance_id : str | None = None ) -> None Parameters: participants : Optional list of participant IDs to transcribe. If None, transcribes all participants. instance_id : Optional specific transcription instance ID to update Example: Copy Ask AI # Update transcription for specific participants await transport.update_transcription( participants = [ "participant-123" , "participant-456" ]) # Update specific transcription instance await transport.update_transcription( instance_id = "transcription-789" ) ​ Recording Control ​ start_recording async method Starts recording the room session. Copy Ask AI async def start_recording ( streaming_settings : Dict = None , stream_id : str = None , force_new : bool = None ) -> None Parameters: streaming_settings : Recording configuration settings stream_id : Optional stream identifier force_new : Force start a new recording See start_recording for more details. ​ stop_recording async method Stops an active recording. Copy Ask AI async def stop_recording ( stream_id : str = None ) -> None Parameters: stream_id : Optional stream identifier to stop specific recording See stop_recording for more details. ​ Dial-out Control ​ start_dialout async method Initiates a dial-out call. Copy Ask AI async def start_dialout ( settings : Dict = None ) -> None Parameters: settings : Dial-out configuration settings See start_dialout for more details. ​ stop_dialout async method Stops an active dial-out call. Copy Ask AI async def stop_dialout ( participant_id : str ) -> None Parameters: participant_id : ID of the participant to disconnect See stop_dialout for more details. ​ send_dtmf async method Sends DTMF tones in an existing dial-out session. Copy Ask AI async def send_dtmf ( settings : Dict = None ) -> None Parameters: settings : DTMF settings See send_dtmf for more details. ​ Subscription Management ​ update_subscriptions async method Updates subscriptions and subscription profiles. This function allows you to update subscription profiles and at the same time assign specific subscription profiles to a participant and even change specific settings for some participants. Copy Ask AI async def update_subscriptions ( participant_settings : Dict = None , profile_settings : Dict = None ) -> None Parameters: participant_settings : Per-participant subscription settings profile_settings : Global subscription profile settings See update_subscriptions for more details. ​ Participant Admin When using the DailyTransport, if your bot is given a meeting token with owner or participant admin permissions, it can remote-control other participants’ capabilities and permissions. With this control, the bot can modify settings for how others send and receive media, how they are seen in a session, and whether they have administrative controls. ​ update_remote_participants async method Updates permissions and settings for specific participants in the room. Copy Ask AI async def update_remote_participants ( remote_participants : Mapping[ str , Any] = None ) -> None Parameters: remote_participants : Dictionary mapping participant IDs to their updated settings This method allows fine-grained control over participant capabilities and permissions in the Daily room. Examples: Copy Ask AI # Mute a participant's microphone await transport.update_remote_participants({ "participant-123" : { "inputsEnabled" : { "microphone" : False } } }) # Revoke send permissions (mutes the participant and prevents them from unmuting) await transport.update_remote_participants({ "participant-123" : { "permissions" : { "canSend" : [] # Empty list revokes all send permissions } } }) # Configure selective audio routing await transport.update_remote_participants({ "participant-123" : { # the customer "permissions" : { "canReceive" : { "base" : False , # Don't receive from anyone by default "byUserId" : { "hold-music" : True # Only receive from hold music player } } } } }) # Set up user-to-user permissions mapping (e.g., when transferring calls) # Assumes customer was previously unable to hear anyone (they were on hold) and agent was speaking to bot await transport.update_remote_participants({ "participant-123" : { # the customer "permissions" : { "base" : False , "canReceive" : { "byUserId" : { "agent" : True }} } }, "participant-456" : { # the agent "permissions" : { "base" : False , "canReceive" : { "byUserId" : { "customer" : True }} } } }) # Promote a participant to be a participant admin await transport.update_remote_participants({ "participant-123" : { "permissions" : { "canAdmin" : [ "participants" ] } } }) Permissions Settings: For comprehensive details on permissions settings, see the Daily API documentation: daily-python’s update_remote_participants() documentation daily-js’s updateParticipant() -> updatePermission and participants().permissions ​ Chat messages ​ send_prebuilt_chat_message async method When testing with Daily Prebuilt, you can send messages to the Prebuilt chat using this method. Parameters: message (str): The chat message to send user_name (str): The name of the user sending the message See send_prebuilt_chat_message for more details. ​ Properties ​ participant_id string Returns the transport’s participant ID in the room. ​ room_url string Returns the transport’s room URL for the room currently in use. ​ Notes Supports real-time audio/video communication Handles WebRTC connection management Provides voice activity detection Includes transcription capabilities Manages participant interactions Supports recording and streaming Thread-safe processing All callbacks are asynchronous Multiple handlers can be registered for the same event Exceptions in callbacks are logged but don’t stop the transport Supported Services SmallWebRTCTransport On this page Overview Installation Configuration Constructor Parameters DailyParams Configuration Audio Output Configuration Audio Input Configuration Video Output Configuration Voice Activity Detection (VAD) Feature Settings Basic Usage Event Callbacks Connection Events Participant Events Communication Events Dial Events Dial-in Dial-out Transcription Events Recording Events Error Events Notes on Callbacks Frame Types Input Frames Output Frames Methods Room Management Media Control Video Management Transcription Control Recording Control Dial-out Control Subscription Management Participant Admin Chat messages Properties Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/transport_daily_dc2b47a4.txt b/transport_daily_dc2b47a4.txt
new file mode 100644
index 0000000000000000000000000000000000000000..be705981e481dc7525882abbd3eaaaa9ea1307aa
--- /dev/null
+++ b/transport_daily_dc2b47a4.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/transport/daily#param-send-image
+Title: Daily WebRTC - Pipecat
+==================================================
+
+Daily WebRTC - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Transport Daily WebRTC Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Daily WebRTC SmallWebRTCTransport FastAPI WebSocket WebSocket Server Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview DailyTransport provides real-time audio and video communication capabilities using Daily’s WebRTC platform. It supports bidirectional audio/video streams, transcription, voice activity detection (VAD), and participant management. ​ Installation To use DailyTransport , install the required dependencies: Copy Ask AI pip install "pipecat-ai[daily]" You’ll also need to set up your Daily API key as an environment variable: DAILY_API_KEY . You can obtain a Daily API key by signing up at Daily . ​ Configuration ​ Constructor Parameters ​ room_url str required Daily room URL ​ token str | None Daily room token ​ bot_name str required Name of the bot in the room ​ params DailyParams default: "DailyParams()" Transport configuration parameters ​ DailyParams Configuration ​ api_url str default: "https://api.daily.co/v1" Daily API endpoint URL ​ api_key str default: "" Daily API key for authentication ​ Audio Output Configuration ​ audio_out_enabled bool default: "False" Enable audio output capabilities ​ audio_out_is_live bool default: "False" Enable live audio streaming mode ​ audio_out_sample_rate int default: "None" Audio output sample rate in Hz ​ audio_out_channels int default: "1" Number of audio output channels ​ audio_out_bitrate int default: "96000" Audio output bitrate in bits per second ​ Audio Input Configuration ​ audio_in_enabled bool default: "False" Enable audio input capabilities ​ audio_in_passthrough bool default: "False" When enabled, incoming audio frames are pushed downstream ​ audio_in_sample_rate int default: "None" Audio input sample rate in Hz ​ audio_in_channels int default: "1" Number of audio input channels ​ audio_in_filter Optional[BaseAudioFilter] default: "None" Audio filter for input processing. Supported filters are: KrispFilter() and NoisereduceFilter() . See the KrispFilter() reference docs and NoisereduceFilter() reference docs ` for more information. ​ Video Output Configuration ​ video_out_enabled bool default: "False" Enable video output capabilities ​ video_out_is_live bool default: "False" Enable live video streaming mode ​ video_out_width int default: "1024" Video output width in pixels ​ video_out_height int default: "768" Video output height in pixels ​ video_out_bitrate int default: "800000" Video output bitrate in bits per second ​ video_out_framerate int default: "30" Video output frame rate ​ video_out_color_format str default: "RGB" Video color format (RGB, BGR, etc.) ​ Voice Activity Detection (VAD) ​ vad_analyzer VADAnalyzer | None default: "None" Voice Activity Detection analyzer. You can set this to either SileroVADAnalyzer() or WebRTCVADAnalyzer() . SileroVADAnalyzer is the recommended option. Learn more about the SileroVADAnalyzer . ​ Feature Settings ​ dialin_settings Optional[DailyDialinSettings] default: "None" Configuration for dial-in functionality ​ transcription_enabled bool default: "False" Enable real-time transcription ​ transcription_settings DailyTranscriptionSettings Configuration for transcription features Copy Ask AI class DailyTranscriptionSettings ( BaseModel ): language: str = "en" # Default language model: str = "nova-2-general" # Transcription model profanity_filter: bool = True # Filter profanity redact: bool = False # Redact sensitive information endpointing: bool = True # Enable speech endpointing punctuate: bool = True # Add punctuation includeRawResponse: bool = True # Include raw API response extra: Mapping[ str , Any] = { "interim_results" : True # Provide any Deepgram-specific settings } ​ Basic Usage Copy Ask AI from pipecat.transports.services.daily import DailyTransport, DailyParams from pipecat.audio.vad.silero import SileroVADAnalyzer # Configure transport transport = DailyTransport( room_url = "https://your-domain.daily.co/room-name" , token = "your-room-token" , bot_name = "AI Assistant" , params = DailyParams( audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer( params = VADParams( stop_secs = 0.5 )), ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), # Handle incoming audio/video stt, # Speech-to-text llm, # Language model tts, # Text-to-speech transport.output() # Handle outgoing audio/video ]) # Use transport event handlers to manage the call lifecycle # Event handler for when the user joins the room: @transport.event_handler ( "on_first_participant_joined" ) async def on_first_participant_joined ( transport , participant ): # Start transcription for the user await transport.capture_participant_transcription(participant[ "id" ]) # Kick off the conversation. await task.queue_frames([context_aggregator.user().get_context_frame()]) # Event handler for when the user leaves the room: @transport.event_handler ( "on_participant_left" ) async def on_participant_left ( transport , participant , reason ): # Cancel the pipeline, which stops processing and removes the bot from the room await task.cancel() ​ Event Callbacks DailyTransport provides a comprehensive callback system for handling various events. Register callbacks using the @transport.event_handler() decorator. ​ Connection Events ​ on_joined async callback Called when the bot successfully joins the room. Parameters: transport : The DailyTransport instance data : Dictionary containing room join information Copy Ask AI @transport.event_handler ( "on_joined" ) async def on_joined ( transport , data : Dict[ str , Any]): logger.info( f "Joined room with data: { data } " ) ​ on_left async callback Called when the bot leaves the room. Parameters: transport : The DailyTransport instance Copy Ask AI @transport.event_handler ( "on_left" ) async def on_left ( transport ): logger.info( "Left room" ) ​ Participant Events ​ on_first_participant_joined async callback Called when the first participant joins the room. Useful for initializing conversations. Parameters: transport : The DailyTransport instance participant : Dictionary containing participant information Copy Ask AI @transport.event_handler ( "on_first_participant_joined" ) async def on_first_participant_joined ( transport , participant : Dict[ str , Any]): await transport.capture_participant_transcription(participant[ "id" ]) await task.queue_frames([LLMMessagesFrame(initial_messages)]) ​ on_participant_joined async callback Called when any participant joins the room. Parameters: transport : The DailyTransport instance participant : Dictionary containing participant information Copy Ask AI @transport.event_handler ( "on_participant_joined" ) async def on_participant_joined ( transport , participant : Dict[ str , Any]): logger.info( f "Participant joined: { participant[ 'id' ] } " ) ​ on_participant_left async callback Called when a participant leaves the room. Parameters: transport : The DailyTransport instance participant : Dictionary containing participant information reason : String describing why the participant left, “leftCall” | “hidden” Copy Ask AI @transport.event_handler ( "on_participant_left" ) async def on_participant_left ( transport , participant : Dict[ str , Any], reason : str ): logger.info( f "Participant { participant[ 'id' ] } left: { reason } " ) # Cancel the pipeline task to stop processing and remove the bot from the Daily room await task.cancel() ​ on_participant_updated async callback Event emitted when a participant is updated. This can mean either the participant’s metadata was updated, or the tracks belonging to the participant changed. Parameters: transport : The DailyTransport instance participant : Dictionary containing updated participant information Copy Ask AI @transport.event_handler ( "on_participant_updated" ) async def on_participant_updated ( transport , participant : Dict[ str , Any]): logger.info( f "Participant updated: { participant } " ) ​ Communication Events ​ on_app_message async callback Event emitted when a custom app message is received from another participant or via the REST API. Parameters: transport : The DailyTransport instance message : The message content (any type) sender : String identifier of the message sender Copy Ask AI @transport.event_handler ( "on_app_message" ) async def on_app_message ( transport , message : Any, sender : str ): logger.info( f "Message from { sender } : { message } " ) ​ on_call_state_updated async callback Event emitted when the call state changes, normally as a consequence of joining or leaving the call. Parameters: transport : The DailyTransport instance state : String representing the new call state. Learn more about call states . Copy Ask AI @transport.event_handler ( "on_call_state_updated" ) async def on_call_state_updated ( transport , state : str ): logger.info( f "Call state updated: { state } " ) ​ Dial Events ​ Dial-in ​ on_dialin_ready async callback Event emitted when dial-in is ready. This happens after the room has connected to the SIP endpoint and the system is ready to receive dial-in calls. Parameters: transport : The DailyTransport instance sip_endpoint : String containing the SIP endpoint information Copy Ask AI @transport.event_handler ( "on_dialin_ready" ) async def on_dialin_ready ( transport , sip_endpoint : str ): logger.info( f "Dial-in ready at: { sip_endpoint } " ) ​ on_dialin_connected async callback Event emitted when the session with the dial-in remote end is established (i.e. SIP endpoint or PSTN are connectd to the Daily room). Note: connected does not mean media (audio or video) has started flowing between the room and PSTN, it means the room received the connection request and both endpoints are negotiating the media flow. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. See DialinConnectedEvent . Copy Ask AI @transport.event_handler ( "on_dialin_connected" ) async def on_dialin_connected ( transport , data : Any): logger.info( f "Dial-in connected: { data } " ) ​ on_dialin_stopped async callback Event emitted when the dial-in remote end disconnects the call. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. See DialinStoppedEvent . Copy Ask AI @transport.event_handler ( "on_dialin_stopped" ) async def on_dialin_stopped ( transport , data : Any): logger.info( f "Dial-in stopped: { data } " ) ​ on_dialin_error async callback Event emitted in the case of dial-in errors which are fatal and the service cannot proceed. For example, an error in SDP negotiation is fatal to the media/SIP pipeline and will result in dialin-error being triggered. Parameters: transport : The DailyTransport instance data : Dictionary containing error information. See DialinEvent . Copy Ask AI @transport.event_handler ( "on_dialin_error" ) async def on_dialin_error ( transport , data : Any): logger.error( f "Dial-in error: { data } " ) ​ on_dialin_warning async callback Event emitted there is a dial-in non-fatal error, such as the selected codec not being used and a fallback codec being utilized. Parameters: transport : The DailyTransport instance data : Dictionary containing warning information. See DialinEvent . Copy Ask AI @transport.event_handler ( "on_dialin_warning" ) async def on_dialin_warning ( transport , data : Any): logger.warning( f "Dial-in warning: { data } " ) ​ Dial-out ​ on_dialout_answered async callback Event emitted when the session with the dial-out remote end is answered. Parameters: transport : The DailyTransport instance data : Dictionary containing call information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_answered" ) async def on_dialout_answered ( transport , data : Any): logger.info( f "Dial-out answered: { data } " ) ​ on_dialout_connected async callback Event emitted when the session with the dial-out remote end is established. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_connected" ) async def on_dialout_connected ( transport , data : Any): logger.info( f "Dial-out connected: { data } " ) ​ on_dialout_stopped async callback Event emitted when the dial-out session is stopped. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_stopped" ) async def on_dialout_stopped ( transport , data : Any): logger.info( f "Dial-out stopped: { data } " ) ​ on_dialout_error async callback Event emitted in the case of dial-out errors which are fatal and the service cannot proceed. For example, an error in SDP negotiation is fatal to the media/SIP pipeline and will result in dialout-error being triggered. Parameters: transport : The DailyTransport instance data : Dictionary containing error information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_error" ) async def on_dialout_error ( transport , data : Any): logger.error( f "Dial-out error: { data } " ) ​ on_dialout_warning async callback Event emitted there is a dial-out non-fatal error, such as the selected codec not being used and a fallback codec being utilized. Parameters: transport : The DailyTransport instance data : Dictionary containing warning information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_warning" ) async def on_dialout_warning ( transport , data : Any): logger.warning( f "Dial-out warning: { data } " ) ​ Transcription Events ​ on_transcription_message async callback Called when a transcription message is received. This includes both interim and final transcriptions. Parameters: transport : The DailyTransport instance message : Dictionary containing transcription data including. Learn more . Copy Ask AI @transport.event_handler ( "on_transcription_message" ) async def on_transcription_message ( transport , message : Dict[ str , Any]): participant_id = message.get( "participantId" ) text = message.get( "text" ) is_final = message[ "rawResponse" ][ "is_final" ] logger.info( f "Transcription from { participant_id } : { text } (final: { is_final } )" ) ​ Recording Events ​ on_recording_started async callback Called when a room recording starts successfully. Parameters: transport : The DailyTransport instance status : Dictionary containing recording status information. Learn more . Copy Ask AI @transport.event_handler ( "on_recording_started" ) async def on_recording_started ( transport , status : Dict[ str , Any]): logger.info( f "Recording started with status: { status } " ) ​ on_recording_stopped async callback Called when a room recording stops. Parameters: transport : The DailyTransport instance stream_id : String identifier of the stopped recording stream Copy Ask AI @transport.event_handler ( "on_recording_stopped" ) async def on_recording_stopped ( transport , stream_id : str ): logger.info( f "Recording stopped for stream: { stream_id } " ) ​ on_recording_error async callback Called when an error occurs during recording. Parameters: transport : The DailyTransport instance stream_id : String identifier of the recording stream message : Error message describing what went wrong Copy Ask AI @transport.event_handler ( "on_recording_error" ) async def on_recording_error ( transport , stream_id : str , message : str ): logger.error( f "Recording error for stream { stream_id } : { message } " ) ​ Error Events ​ on_error async callback Called when a transport error occurs. Parameters: transport : The DailyTransport instance error : String containing error details Copy Ask AI @transport.event_handler ( "on_error" ) async def on_error ( transport , error : str ): logger.error( f "Transport error: { error } " ) ​ Notes on Callbacks All callbacks are asynchronous Callbacks are executed in the order they were registered Multiple handlers can be registered for the same event Exceptions in callbacks are logged but don’t stop the transport Callbacks should be lightweight to avoid blocking the event loop Heavy processing should be offloaded to separate tasks ​ Frame Types ​ Input Frames ​ InputAudioRawFrame Frame Raw audio data from participants ​ UserImageRawFrame Frame Video frames from participants ​ Output Frames ​ OutputAudioRawFrame Frame Audio data to be sent ​ OutputImageRawFrame Frame Video frames to be sent ​ Methods ​ Room Management ​ participants method Returns a dictionary of all participants currently in the room. Copy Ask AI def participants () -> Dict[ str , Any] See participants for more details. ​ participant_counts method Returns participant count statistics for the room. Copy Ask AI def participant_counts () -> Dict[ str , int ] See participant_counts for more details. ​ send_message async method Sends a message to participants in the Daily room. Messages can be directed to all participants or targeted to a specific participant. Copy Ask AI async def send_message ( frame : TransportMessageFrame | TransportMessageUrgentFrame) Parameters: frame : A transport message frame containing the message to send. Can be either: TransportMessageFrame : Standard message TransportMessageUrgentFrame : High-priority message Message Structure: Messages should conform to the expected format for the receiving system. For RTVI messages, use: Copy Ask AI { "label" : "rtvi-ai" , # Required for RTVI protocol "type" : "message-type" , # Defines how the client will process it "id" : "unique-id" , # Unique identifier for the message "data" : { # Custom payload # Your custom data fields } } Example - Standard Message: Copy Ask AI import uuid # Send a standard message to all participants message_data = { "label" : "rtvi-ai" , "type" : "custom-message" , "id" : str (uuid.uuid4()), "data" : { "message" : "Hello everyone!" , "timestamp" : datetime.datetime.now().isoformat() } } await transport.send_message(TransportMessageFrame( message = message_data)) Example - Urgent Message to Specific Participant: Copy Ask AI # Send urgent message to specific participant urgent_data = { "label" : "rtvi-ai" , "type" : "server-message" , "id" : str (uuid.uuid4()), "data" : { "message" : "Private message" , "importance" : "high" } } await transport.send_message( DailyTransportMessageUrgentFrame( message = urgent_data, participant_id = "participant-123" ) ) The message will be delivered to: All participants if no specific participant_id is set Only the specified participant if participant_id is set Through the Daily app messaging system Can be received by registering an on_app_message handler Use TransportMessageUrgentFrame for high-priority messages that need immediate delivery. ​ Media Control ​ send_image async method Sends an image frame to the room. Copy Ask AI async def send_image ( frame : OutputImageRawFrame | SpriteFrame) -> None Parameters: frame : Image frame to send, either raw image data or sprite animation ​ send_audio async method Sends an audio frame to the room. Copy Ask AI async def send_audio ( frame : OutputAudioRawFrame) -> None Parameters: frame : Audio frame to send ​ Video Management ​ capture_participant_video async method Starts capturing video from a specific participant. Copy Ask AI async def capture_participant_video ( participant_id : str , framerate : int = 30 , video_source : str = "camera" , color_format : str = "RGB" ) -> None Parameters: participant_id : ID of the participant to capture framerate : Target frame rate (default: 30) video_source : Video source type (default: “camera”) color_format : Color format of the video (default: “RGB”) To request an image from the camera stream, set the framerate to 0 and push a UserImageRequestFrame . ​ Transcription Control ​ capture_participant_transcription async method Starts capturing and transcribing audio from a specific participant. Copy Ask AI async def capture_participant_transcription ( participant_id : str ) -> None Parameters: participant_id : ID of the participant to transcribe ​ update_transcription async method Updates the transcription configuration for specific participants or instances. Copy Ask AI async def update_transcription ( participants : List[ str ] | None = None , instance_id : str | None = None ) -> None Parameters: participants : Optional list of participant IDs to transcribe. If None, transcribes all participants. instance_id : Optional specific transcription instance ID to update Example: Copy Ask AI # Update transcription for specific participants await transport.update_transcription( participants = [ "participant-123" , "participant-456" ]) # Update specific transcription instance await transport.update_transcription( instance_id = "transcription-789" ) ​ Recording Control ​ start_recording async method Starts recording the room session. Copy Ask AI async def start_recording ( streaming_settings : Dict = None , stream_id : str = None , force_new : bool = None ) -> None Parameters: streaming_settings : Recording configuration settings stream_id : Optional stream identifier force_new : Force start a new recording See start_recording for more details. ​ stop_recording async method Stops an active recording. Copy Ask AI async def stop_recording ( stream_id : str = None ) -> None Parameters: stream_id : Optional stream identifier to stop specific recording See stop_recording for more details. ​ Dial-out Control ​ start_dialout async method Initiates a dial-out call. Copy Ask AI async def start_dialout ( settings : Dict = None ) -> None Parameters: settings : Dial-out configuration settings See start_dialout for more details. ​ stop_dialout async method Stops an active dial-out call. Copy Ask AI async def stop_dialout ( participant_id : str ) -> None Parameters: participant_id : ID of the participant to disconnect See stop_dialout for more details. ​ send_dtmf async method Sends DTMF tones in an existing dial-out session. Copy Ask AI async def send_dtmf ( settings : Dict = None ) -> None Parameters: settings : DTMF settings See send_dtmf for more details. ​ Subscription Management ​ update_subscriptions async method Updates subscriptions and subscription profiles. This function allows you to update subscription profiles and at the same time assign specific subscription profiles to a participant and even change specific settings for some participants. Copy Ask AI async def update_subscriptions ( participant_settings : Dict = None , profile_settings : Dict = None ) -> None Parameters: participant_settings : Per-participant subscription settings profile_settings : Global subscription profile settings See update_subscriptions for more details. ​ Participant Admin When using the DailyTransport, if your bot is given a meeting token with owner or participant admin permissions, it can remote-control other participants’ capabilities and permissions. With this control, the bot can modify settings for how others send and receive media, how they are seen in a session, and whether they have administrative controls. ​ update_remote_participants async method Updates permissions and settings for specific participants in the room. Copy Ask AI async def update_remote_participants ( remote_participants : Mapping[ str , Any] = None ) -> None Parameters: remote_participants : Dictionary mapping participant IDs to their updated settings This method allows fine-grained control over participant capabilities and permissions in the Daily room. Examples: Copy Ask AI # Mute a participant's microphone await transport.update_remote_participants({ "participant-123" : { "inputsEnabled" : { "microphone" : False } } }) # Revoke send permissions (mutes the participant and prevents them from unmuting) await transport.update_remote_participants({ "participant-123" : { "permissions" : { "canSend" : [] # Empty list revokes all send permissions } } }) # Configure selective audio routing await transport.update_remote_participants({ "participant-123" : { # the customer "permissions" : { "canReceive" : { "base" : False , # Don't receive from anyone by default "byUserId" : { "hold-music" : True # Only receive from hold music player } } } } }) # Set up user-to-user permissions mapping (e.g., when transferring calls) # Assumes customer was previously unable to hear anyone (they were on hold) and agent was speaking to bot await transport.update_remote_participants({ "participant-123" : { # the customer "permissions" : { "base" : False , "canReceive" : { "byUserId" : { "agent" : True }} } }, "participant-456" : { # the agent "permissions" : { "base" : False , "canReceive" : { "byUserId" : { "customer" : True }} } } }) # Promote a participant to be a participant admin await transport.update_remote_participants({ "participant-123" : { "permissions" : { "canAdmin" : [ "participants" ] } } }) Permissions Settings: For comprehensive details on permissions settings, see the Daily API documentation: daily-python’s update_remote_participants() documentation daily-js’s updateParticipant() -> updatePermission and participants().permissions ​ Chat messages ​ send_prebuilt_chat_message async method When testing with Daily Prebuilt, you can send messages to the Prebuilt chat using this method. Parameters: message (str): The chat message to send user_name (str): The name of the user sending the message See send_prebuilt_chat_message for more details. ​ Properties ​ participant_id string Returns the transport’s participant ID in the room. ​ room_url string Returns the transport’s room URL for the room currently in use. ​ Notes Supports real-time audio/video communication Handles WebRTC connection management Provides voice activity detection Includes transcription capabilities Manages participant interactions Supports recording and streaming Thread-safe processing All callbacks are asynchronous Multiple handlers can be registered for the same event Exceptions in callbacks are logged but don’t stop the transport Supported Services SmallWebRTCTransport On this page Overview Installation Configuration Constructor Parameters DailyParams Configuration Audio Output Configuration Audio Input Configuration Video Output Configuration Voice Activity Detection (VAD) Feature Settings Basic Usage Event Callbacks Connection Events Participant Events Communication Events Dial Events Dial-in Dial-out Transcription Events Recording Events Error Events Notes on Callbacks Frame Types Input Frames Output Frames Methods Room Management Media Control Video Management Transcription Control Recording Control Dial-out Control Subscription Management Participant Admin Chat messages Properties Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/transport_daily_def06aa2.txt b/transport_daily_def06aa2.txt
new file mode 100644
index 0000000000000000000000000000000000000000..4711de23faec629a8731a809e8ed6bd0a635d907
--- /dev/null
+++ b/transport_daily_def06aa2.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/transport/daily#configuration
+Title: Daily WebRTC - Pipecat
+==================================================
+
+Daily WebRTC - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Transport Daily WebRTC Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Daily WebRTC SmallWebRTCTransport FastAPI WebSocket WebSocket Server Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview DailyTransport provides real-time audio and video communication capabilities using Daily’s WebRTC platform. It supports bidirectional audio/video streams, transcription, voice activity detection (VAD), and participant management. ​ Installation To use DailyTransport , install the required dependencies: Copy Ask AI pip install "pipecat-ai[daily]" You’ll also need to set up your Daily API key as an environment variable: DAILY_API_KEY . You can obtain a Daily API key by signing up at Daily . ​ Configuration ​ Constructor Parameters ​ room_url str required Daily room URL ​ token str | None Daily room token ​ bot_name str required Name of the bot in the room ​ params DailyParams default: "DailyParams()" Transport configuration parameters ​ DailyParams Configuration ​ api_url str default: "https://api.daily.co/v1" Daily API endpoint URL ​ api_key str default: "" Daily API key for authentication ​ Audio Output Configuration ​ audio_out_enabled bool default: "False" Enable audio output capabilities ​ audio_out_is_live bool default: "False" Enable live audio streaming mode ​ audio_out_sample_rate int default: "None" Audio output sample rate in Hz ​ audio_out_channels int default: "1" Number of audio output channels ​ audio_out_bitrate int default: "96000" Audio output bitrate in bits per second ​ Audio Input Configuration ​ audio_in_enabled bool default: "False" Enable audio input capabilities ​ audio_in_passthrough bool default: "False" When enabled, incoming audio frames are pushed downstream ​ audio_in_sample_rate int default: "None" Audio input sample rate in Hz ​ audio_in_channels int default: "1" Number of audio input channels ​ audio_in_filter Optional[BaseAudioFilter] default: "None" Audio filter for input processing. Supported filters are: KrispFilter() and NoisereduceFilter() . See the KrispFilter() reference docs and NoisereduceFilter() reference docs ` for more information. ​ Video Output Configuration ​ video_out_enabled bool default: "False" Enable video output capabilities ​ video_out_is_live bool default: "False" Enable live video streaming mode ​ video_out_width int default: "1024" Video output width in pixels ​ video_out_height int default: "768" Video output height in pixels ​ video_out_bitrate int default: "800000" Video output bitrate in bits per second ​ video_out_framerate int default: "30" Video output frame rate ​ video_out_color_format str default: "RGB" Video color format (RGB, BGR, etc.) ​ Voice Activity Detection (VAD) ​ vad_analyzer VADAnalyzer | None default: "None" Voice Activity Detection analyzer. You can set this to either SileroVADAnalyzer() or WebRTCVADAnalyzer() . SileroVADAnalyzer is the recommended option. Learn more about the SileroVADAnalyzer . ​ Feature Settings ​ dialin_settings Optional[DailyDialinSettings] default: "None" Configuration for dial-in functionality ​ transcription_enabled bool default: "False" Enable real-time transcription ​ transcription_settings DailyTranscriptionSettings Configuration for transcription features Copy Ask AI class DailyTranscriptionSettings ( BaseModel ): language: str = "en" # Default language model: str = "nova-2-general" # Transcription model profanity_filter: bool = True # Filter profanity redact: bool = False # Redact sensitive information endpointing: bool = True # Enable speech endpointing punctuate: bool = True # Add punctuation includeRawResponse: bool = True # Include raw API response extra: Mapping[ str , Any] = { "interim_results" : True # Provide any Deepgram-specific settings } ​ Basic Usage Copy Ask AI from pipecat.transports.services.daily import DailyTransport, DailyParams from pipecat.audio.vad.silero import SileroVADAnalyzer # Configure transport transport = DailyTransport( room_url = "https://your-domain.daily.co/room-name" , token = "your-room-token" , bot_name = "AI Assistant" , params = DailyParams( audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer( params = VADParams( stop_secs = 0.5 )), ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), # Handle incoming audio/video stt, # Speech-to-text llm, # Language model tts, # Text-to-speech transport.output() # Handle outgoing audio/video ]) # Use transport event handlers to manage the call lifecycle # Event handler for when the user joins the room: @transport.event_handler ( "on_first_participant_joined" ) async def on_first_participant_joined ( transport , participant ): # Start transcription for the user await transport.capture_participant_transcription(participant[ "id" ]) # Kick off the conversation. await task.queue_frames([context_aggregator.user().get_context_frame()]) # Event handler for when the user leaves the room: @transport.event_handler ( "on_participant_left" ) async def on_participant_left ( transport , participant , reason ): # Cancel the pipeline, which stops processing and removes the bot from the room await task.cancel() ​ Event Callbacks DailyTransport provides a comprehensive callback system for handling various events. Register callbacks using the @transport.event_handler() decorator. ​ Connection Events ​ on_joined async callback Called when the bot successfully joins the room. Parameters: transport : The DailyTransport instance data : Dictionary containing room join information Copy Ask AI @transport.event_handler ( "on_joined" ) async def on_joined ( transport , data : Dict[ str , Any]): logger.info( f "Joined room with data: { data } " ) ​ on_left async callback Called when the bot leaves the room. Parameters: transport : The DailyTransport instance Copy Ask AI @transport.event_handler ( "on_left" ) async def on_left ( transport ): logger.info( "Left room" ) ​ Participant Events ​ on_first_participant_joined async callback Called when the first participant joins the room. Useful for initializing conversations. Parameters: transport : The DailyTransport instance participant : Dictionary containing participant information Copy Ask AI @transport.event_handler ( "on_first_participant_joined" ) async def on_first_participant_joined ( transport , participant : Dict[ str , Any]): await transport.capture_participant_transcription(participant[ "id" ]) await task.queue_frames([LLMMessagesFrame(initial_messages)]) ​ on_participant_joined async callback Called when any participant joins the room. Parameters: transport : The DailyTransport instance participant : Dictionary containing participant information Copy Ask AI @transport.event_handler ( "on_participant_joined" ) async def on_participant_joined ( transport , participant : Dict[ str , Any]): logger.info( f "Participant joined: { participant[ 'id' ] } " ) ​ on_participant_left async callback Called when a participant leaves the room. Parameters: transport : The DailyTransport instance participant : Dictionary containing participant information reason : String describing why the participant left, “leftCall” | “hidden” Copy Ask AI @transport.event_handler ( "on_participant_left" ) async def on_participant_left ( transport , participant : Dict[ str , Any], reason : str ): logger.info( f "Participant { participant[ 'id' ] } left: { reason } " ) # Cancel the pipeline task to stop processing and remove the bot from the Daily room await task.cancel() ​ on_participant_updated async callback Event emitted when a participant is updated. This can mean either the participant’s metadata was updated, or the tracks belonging to the participant changed. Parameters: transport : The DailyTransport instance participant : Dictionary containing updated participant information Copy Ask AI @transport.event_handler ( "on_participant_updated" ) async def on_participant_updated ( transport , participant : Dict[ str , Any]): logger.info( f "Participant updated: { participant } " ) ​ Communication Events ​ on_app_message async callback Event emitted when a custom app message is received from another participant or via the REST API. Parameters: transport : The DailyTransport instance message : The message content (any type) sender : String identifier of the message sender Copy Ask AI @transport.event_handler ( "on_app_message" ) async def on_app_message ( transport , message : Any, sender : str ): logger.info( f "Message from { sender } : { message } " ) ​ on_call_state_updated async callback Event emitted when the call state changes, normally as a consequence of joining or leaving the call. Parameters: transport : The DailyTransport instance state : String representing the new call state. Learn more about call states . Copy Ask AI @transport.event_handler ( "on_call_state_updated" ) async def on_call_state_updated ( transport , state : str ): logger.info( f "Call state updated: { state } " ) ​ Dial Events ​ Dial-in ​ on_dialin_ready async callback Event emitted when dial-in is ready. This happens after the room has connected to the SIP endpoint and the system is ready to receive dial-in calls. Parameters: transport : The DailyTransport instance sip_endpoint : String containing the SIP endpoint information Copy Ask AI @transport.event_handler ( "on_dialin_ready" ) async def on_dialin_ready ( transport , sip_endpoint : str ): logger.info( f "Dial-in ready at: { sip_endpoint } " ) ​ on_dialin_connected async callback Event emitted when the session with the dial-in remote end is established (i.e. SIP endpoint or PSTN are connectd to the Daily room). Note: connected does not mean media (audio or video) has started flowing between the room and PSTN, it means the room received the connection request and both endpoints are negotiating the media flow. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. See DialinConnectedEvent . Copy Ask AI @transport.event_handler ( "on_dialin_connected" ) async def on_dialin_connected ( transport , data : Any): logger.info( f "Dial-in connected: { data } " ) ​ on_dialin_stopped async callback Event emitted when the dial-in remote end disconnects the call. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. See DialinStoppedEvent . Copy Ask AI @transport.event_handler ( "on_dialin_stopped" ) async def on_dialin_stopped ( transport , data : Any): logger.info( f "Dial-in stopped: { data } " ) ​ on_dialin_error async callback Event emitted in the case of dial-in errors which are fatal and the service cannot proceed. For example, an error in SDP negotiation is fatal to the media/SIP pipeline and will result in dialin-error being triggered. Parameters: transport : The DailyTransport instance data : Dictionary containing error information. See DialinEvent . Copy Ask AI @transport.event_handler ( "on_dialin_error" ) async def on_dialin_error ( transport , data : Any): logger.error( f "Dial-in error: { data } " ) ​ on_dialin_warning async callback Event emitted there is a dial-in non-fatal error, such as the selected codec not being used and a fallback codec being utilized. Parameters: transport : The DailyTransport instance data : Dictionary containing warning information. See DialinEvent . Copy Ask AI @transport.event_handler ( "on_dialin_warning" ) async def on_dialin_warning ( transport , data : Any): logger.warning( f "Dial-in warning: { data } " ) ​ Dial-out ​ on_dialout_answered async callback Event emitted when the session with the dial-out remote end is answered. Parameters: transport : The DailyTransport instance data : Dictionary containing call information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_answered" ) async def on_dialout_answered ( transport , data : Any): logger.info( f "Dial-out answered: { data } " ) ​ on_dialout_connected async callback Event emitted when the session with the dial-out remote end is established. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_connected" ) async def on_dialout_connected ( transport , data : Any): logger.info( f "Dial-out connected: { data } " ) ​ on_dialout_stopped async callback Event emitted when the dial-out session is stopped. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_stopped" ) async def on_dialout_stopped ( transport , data : Any): logger.info( f "Dial-out stopped: { data } " ) ​ on_dialout_error async callback Event emitted in the case of dial-out errors which are fatal and the service cannot proceed. For example, an error in SDP negotiation is fatal to the media/SIP pipeline and will result in dialout-error being triggered. Parameters: transport : The DailyTransport instance data : Dictionary containing error information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_error" ) async def on_dialout_error ( transport , data : Any): logger.error( f "Dial-out error: { data } " ) ​ on_dialout_warning async callback Event emitted there is a dial-out non-fatal error, such as the selected codec not being used and a fallback codec being utilized. Parameters: transport : The DailyTransport instance data : Dictionary containing warning information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_warning" ) async def on_dialout_warning ( transport , data : Any): logger.warning( f "Dial-out warning: { data } " ) ​ Transcription Events ​ on_transcription_message async callback Called when a transcription message is received. This includes both interim and final transcriptions. Parameters: transport : The DailyTransport instance message : Dictionary containing transcription data including. Learn more . Copy Ask AI @transport.event_handler ( "on_transcription_message" ) async def on_transcription_message ( transport , message : Dict[ str , Any]): participant_id = message.get( "participantId" ) text = message.get( "text" ) is_final = message[ "rawResponse" ][ "is_final" ] logger.info( f "Transcription from { participant_id } : { text } (final: { is_final } )" ) ​ Recording Events ​ on_recording_started async callback Called when a room recording starts successfully. Parameters: transport : The DailyTransport instance status : Dictionary containing recording status information. Learn more . Copy Ask AI @transport.event_handler ( "on_recording_started" ) async def on_recording_started ( transport , status : Dict[ str , Any]): logger.info( f "Recording started with status: { status } " ) ​ on_recording_stopped async callback Called when a room recording stops. Parameters: transport : The DailyTransport instance stream_id : String identifier of the stopped recording stream Copy Ask AI @transport.event_handler ( "on_recording_stopped" ) async def on_recording_stopped ( transport , stream_id : str ): logger.info( f "Recording stopped for stream: { stream_id } " ) ​ on_recording_error async callback Called when an error occurs during recording. Parameters: transport : The DailyTransport instance stream_id : String identifier of the recording stream message : Error message describing what went wrong Copy Ask AI @transport.event_handler ( "on_recording_error" ) async def on_recording_error ( transport , stream_id : str , message : str ): logger.error( f "Recording error for stream { stream_id } : { message } " ) ​ Error Events ​ on_error async callback Called when a transport error occurs. Parameters: transport : The DailyTransport instance error : String containing error details Copy Ask AI @transport.event_handler ( "on_error" ) async def on_error ( transport , error : str ): logger.error( f "Transport error: { error } " ) ​ Notes on Callbacks All callbacks are asynchronous Callbacks are executed in the order they were registered Multiple handlers can be registered for the same event Exceptions in callbacks are logged but don’t stop the transport Callbacks should be lightweight to avoid blocking the event loop Heavy processing should be offloaded to separate tasks ​ Frame Types ​ Input Frames ​ InputAudioRawFrame Frame Raw audio data from participants ​ UserImageRawFrame Frame Video frames from participants ​ Output Frames ​ OutputAudioRawFrame Frame Audio data to be sent ​ OutputImageRawFrame Frame Video frames to be sent ​ Methods ​ Room Management ​ participants method Returns a dictionary of all participants currently in the room. Copy Ask AI def participants () -> Dict[ str , Any] See participants for more details. ​ participant_counts method Returns participant count statistics for the room. Copy Ask AI def participant_counts () -> Dict[ str , int ] See participant_counts for more details. ​ send_message async method Sends a message to participants in the Daily room. Messages can be directed to all participants or targeted to a specific participant. Copy Ask AI async def send_message ( frame : TransportMessageFrame | TransportMessageUrgentFrame) Parameters: frame : A transport message frame containing the message to send. Can be either: TransportMessageFrame : Standard message TransportMessageUrgentFrame : High-priority message Message Structure: Messages should conform to the expected format for the receiving system. For RTVI messages, use: Copy Ask AI { "label" : "rtvi-ai" , # Required for RTVI protocol "type" : "message-type" , # Defines how the client will process it "id" : "unique-id" , # Unique identifier for the message "data" : { # Custom payload # Your custom data fields } } Example - Standard Message: Copy Ask AI import uuid # Send a standard message to all participants message_data = { "label" : "rtvi-ai" , "type" : "custom-message" , "id" : str (uuid.uuid4()), "data" : { "message" : "Hello everyone!" , "timestamp" : datetime.datetime.now().isoformat() } } await transport.send_message(TransportMessageFrame( message = message_data)) Example - Urgent Message to Specific Participant: Copy Ask AI # Send urgent message to specific participant urgent_data = { "label" : "rtvi-ai" , "type" : "server-message" , "id" : str (uuid.uuid4()), "data" : { "message" : "Private message" , "importance" : "high" } } await transport.send_message( DailyTransportMessageUrgentFrame( message = urgent_data, participant_id = "participant-123" ) ) The message will be delivered to: All participants if no specific participant_id is set Only the specified participant if participant_id is set Through the Daily app messaging system Can be received by registering an on_app_message handler Use TransportMessageUrgentFrame for high-priority messages that need immediate delivery. ​ Media Control ​ send_image async method Sends an image frame to the room. Copy Ask AI async def send_image ( frame : OutputImageRawFrame | SpriteFrame) -> None Parameters: frame : Image frame to send, either raw image data or sprite animation ​ send_audio async method Sends an audio frame to the room. Copy Ask AI async def send_audio ( frame : OutputAudioRawFrame) -> None Parameters: frame : Audio frame to send ​ Video Management ​ capture_participant_video async method Starts capturing video from a specific participant. Copy Ask AI async def capture_participant_video ( participant_id : str , framerate : int = 30 , video_source : str = "camera" , color_format : str = "RGB" ) -> None Parameters: participant_id : ID of the participant to capture framerate : Target frame rate (default: 30) video_source : Video source type (default: “camera”) color_format : Color format of the video (default: “RGB”) To request an image from the camera stream, set the framerate to 0 and push a UserImageRequestFrame . ​ Transcription Control ​ capture_participant_transcription async method Starts capturing and transcribing audio from a specific participant. Copy Ask AI async def capture_participant_transcription ( participant_id : str ) -> None Parameters: participant_id : ID of the participant to transcribe ​ update_transcription async method Updates the transcription configuration for specific participants or instances. Copy Ask AI async def update_transcription ( participants : List[ str ] | None = None , instance_id : str | None = None ) -> None Parameters: participants : Optional list of participant IDs to transcribe. If None, transcribes all participants. instance_id : Optional specific transcription instance ID to update Example: Copy Ask AI # Update transcription for specific participants await transport.update_transcription( participants = [ "participant-123" , "participant-456" ]) # Update specific transcription instance await transport.update_transcription( instance_id = "transcription-789" ) ​ Recording Control ​ start_recording async method Starts recording the room session. Copy Ask AI async def start_recording ( streaming_settings : Dict = None , stream_id : str = None , force_new : bool = None ) -> None Parameters: streaming_settings : Recording configuration settings stream_id : Optional stream identifier force_new : Force start a new recording See start_recording for more details. ​ stop_recording async method Stops an active recording. Copy Ask AI async def stop_recording ( stream_id : str = None ) -> None Parameters: stream_id : Optional stream identifier to stop specific recording See stop_recording for more details. ​ Dial-out Control ​ start_dialout async method Initiates a dial-out call. Copy Ask AI async def start_dialout ( settings : Dict = None ) -> None Parameters: settings : Dial-out configuration settings See start_dialout for more details. ​ stop_dialout async method Stops an active dial-out call. Copy Ask AI async def stop_dialout ( participant_id : str ) -> None Parameters: participant_id : ID of the participant to disconnect See stop_dialout for more details. ​ send_dtmf async method Sends DTMF tones in an existing dial-out session. Copy Ask AI async def send_dtmf ( settings : Dict = None ) -> None Parameters: settings : DTMF settings See send_dtmf for more details. ​ Subscription Management ​ update_subscriptions async method Updates subscriptions and subscription profiles. This function allows you to update subscription profiles and at the same time assign specific subscription profiles to a participant and even change specific settings for some participants. Copy Ask AI async def update_subscriptions ( participant_settings : Dict = None , profile_settings : Dict = None ) -> None Parameters: participant_settings : Per-participant subscription settings profile_settings : Global subscription profile settings See update_subscriptions for more details. ​ Participant Admin When using the DailyTransport, if your bot is given a meeting token with owner or participant admin permissions, it can remote-control other participants’ capabilities and permissions. With this control, the bot can modify settings for how others send and receive media, how they are seen in a session, and whether they have administrative controls. ​ update_remote_participants async method Updates permissions and settings for specific participants in the room. Copy Ask AI async def update_remote_participants ( remote_participants : Mapping[ str , Any] = None ) -> None Parameters: remote_participants : Dictionary mapping participant IDs to their updated settings This method allows fine-grained control over participant capabilities and permissions in the Daily room. Examples: Copy Ask AI # Mute a participant's microphone await transport.update_remote_participants({ "participant-123" : { "inputsEnabled" : { "microphone" : False } } }) # Revoke send permissions (mutes the participant and prevents them from unmuting) await transport.update_remote_participants({ "participant-123" : { "permissions" : { "canSend" : [] # Empty list revokes all send permissions } } }) # Configure selective audio routing await transport.update_remote_participants({ "participant-123" : { # the customer "permissions" : { "canReceive" : { "base" : False , # Don't receive from anyone by default "byUserId" : { "hold-music" : True # Only receive from hold music player } } } } }) # Set up user-to-user permissions mapping (e.g., when transferring calls) # Assumes customer was previously unable to hear anyone (they were on hold) and agent was speaking to bot await transport.update_remote_participants({ "participant-123" : { # the customer "permissions" : { "base" : False , "canReceive" : { "byUserId" : { "agent" : True }} } }, "participant-456" : { # the agent "permissions" : { "base" : False , "canReceive" : { "byUserId" : { "customer" : True }} } } }) # Promote a participant to be a participant admin await transport.update_remote_participants({ "participant-123" : { "permissions" : { "canAdmin" : [ "participants" ] } } }) Permissions Settings: For comprehensive details on permissions settings, see the Daily API documentation: daily-python’s update_remote_participants() documentation daily-js’s updateParticipant() -> updatePermission and participants().permissions ​ Chat messages ​ send_prebuilt_chat_message async method When testing with Daily Prebuilt, you can send messages to the Prebuilt chat using this method. Parameters: message (str): The chat message to send user_name (str): The name of the user sending the message See send_prebuilt_chat_message for more details. ​ Properties ​ participant_id string Returns the transport’s participant ID in the room. ​ room_url string Returns the transport’s room URL for the room currently in use. ​ Notes Supports real-time audio/video communication Handles WebRTC connection management Provides voice activity detection Includes transcription capabilities Manages participant interactions Supports recording and streaming Thread-safe processing All callbacks are asynchronous Multiple handlers can be registered for the same event Exceptions in callbacks are logged but don’t stop the transport Supported Services SmallWebRTCTransport On this page Overview Installation Configuration Constructor Parameters DailyParams Configuration Audio Output Configuration Audio Input Configuration Video Output Configuration Voice Activity Detection (VAD) Feature Settings Basic Usage Event Callbacks Connection Events Participant Events Communication Events Dial Events Dial-in Dial-out Transcription Events Recording Events Error Events Notes on Callbacks Frame Types Input Frames Output Frames Methods Room Management Media Control Video Management Transcription Control Recording Control Dial-out Control Subscription Management Participant Admin Chat messages Properties Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/transport_daily_e512242c.txt b/transport_daily_e512242c.txt
new file mode 100644
index 0000000000000000000000000000000000000000..aef01827b67c7cc5c1af765f04783498f82e503a
--- /dev/null
+++ b/transport_daily_e512242c.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/transport/daily#feature-settings
+Title: Daily WebRTC - Pipecat
+==================================================
+
+Daily WebRTC - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Transport Daily WebRTC Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Daily WebRTC SmallWebRTCTransport FastAPI WebSocket WebSocket Server Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview DailyTransport provides real-time audio and video communication capabilities using Daily’s WebRTC platform. It supports bidirectional audio/video streams, transcription, voice activity detection (VAD), and participant management. ​ Installation To use DailyTransport , install the required dependencies: Copy Ask AI pip install "pipecat-ai[daily]" You’ll also need to set up your Daily API key as an environment variable: DAILY_API_KEY . You can obtain a Daily API key by signing up at Daily . ​ Configuration ​ Constructor Parameters ​ room_url str required Daily room URL ​ token str | None Daily room token ​ bot_name str required Name of the bot in the room ​ params DailyParams default: "DailyParams()" Transport configuration parameters ​ DailyParams Configuration ​ api_url str default: "https://api.daily.co/v1" Daily API endpoint URL ​ api_key str default: "" Daily API key for authentication ​ Audio Output Configuration ​ audio_out_enabled bool default: "False" Enable audio output capabilities ​ audio_out_is_live bool default: "False" Enable live audio streaming mode ​ audio_out_sample_rate int default: "None" Audio output sample rate in Hz ​ audio_out_channels int default: "1" Number of audio output channels ​ audio_out_bitrate int default: "96000" Audio output bitrate in bits per second ​ Audio Input Configuration ​ audio_in_enabled bool default: "False" Enable audio input capabilities ​ audio_in_passthrough bool default: "False" When enabled, incoming audio frames are pushed downstream ​ audio_in_sample_rate int default: "None" Audio input sample rate in Hz ​ audio_in_channels int default: "1" Number of audio input channels ​ audio_in_filter Optional[BaseAudioFilter] default: "None" Audio filter for input processing. Supported filters are: KrispFilter() and NoisereduceFilter() . See the KrispFilter() reference docs and NoisereduceFilter() reference docs ` for more information. ​ Video Output Configuration ​ video_out_enabled bool default: "False" Enable video output capabilities ​ video_out_is_live bool default: "False" Enable live video streaming mode ​ video_out_width int default: "1024" Video output width in pixels ​ video_out_height int default: "768" Video output height in pixels ​ video_out_bitrate int default: "800000" Video output bitrate in bits per second ​ video_out_framerate int default: "30" Video output frame rate ​ video_out_color_format str default: "RGB" Video color format (RGB, BGR, etc.) ​ Voice Activity Detection (VAD) ​ vad_analyzer VADAnalyzer | None default: "None" Voice Activity Detection analyzer. You can set this to either SileroVADAnalyzer() or WebRTCVADAnalyzer() . SileroVADAnalyzer is the recommended option. Learn more about the SileroVADAnalyzer . ​ Feature Settings ​ dialin_settings Optional[DailyDialinSettings] default: "None" Configuration for dial-in functionality ​ transcription_enabled bool default: "False" Enable real-time transcription ​ transcription_settings DailyTranscriptionSettings Configuration for transcription features Copy Ask AI class DailyTranscriptionSettings ( BaseModel ): language: str = "en" # Default language model: str = "nova-2-general" # Transcription model profanity_filter: bool = True # Filter profanity redact: bool = False # Redact sensitive information endpointing: bool = True # Enable speech endpointing punctuate: bool = True # Add punctuation includeRawResponse: bool = True # Include raw API response extra: Mapping[ str , Any] = { "interim_results" : True # Provide any Deepgram-specific settings } ​ Basic Usage Copy Ask AI from pipecat.transports.services.daily import DailyTransport, DailyParams from pipecat.audio.vad.silero import SileroVADAnalyzer # Configure transport transport = DailyTransport( room_url = "https://your-domain.daily.co/room-name" , token = "your-room-token" , bot_name = "AI Assistant" , params = DailyParams( audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer( params = VADParams( stop_secs = 0.5 )), ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), # Handle incoming audio/video stt, # Speech-to-text llm, # Language model tts, # Text-to-speech transport.output() # Handle outgoing audio/video ]) # Use transport event handlers to manage the call lifecycle # Event handler for when the user joins the room: @transport.event_handler ( "on_first_participant_joined" ) async def on_first_participant_joined ( transport , participant ): # Start transcription for the user await transport.capture_participant_transcription(participant[ "id" ]) # Kick off the conversation. await task.queue_frames([context_aggregator.user().get_context_frame()]) # Event handler for when the user leaves the room: @transport.event_handler ( "on_participant_left" ) async def on_participant_left ( transport , participant , reason ): # Cancel the pipeline, which stops processing and removes the bot from the room await task.cancel() ​ Event Callbacks DailyTransport provides a comprehensive callback system for handling various events. Register callbacks using the @transport.event_handler() decorator. ​ Connection Events ​ on_joined async callback Called when the bot successfully joins the room. Parameters: transport : The DailyTransport instance data : Dictionary containing room join information Copy Ask AI @transport.event_handler ( "on_joined" ) async def on_joined ( transport , data : Dict[ str , Any]): logger.info( f "Joined room with data: { data } " ) ​ on_left async callback Called when the bot leaves the room. Parameters: transport : The DailyTransport instance Copy Ask AI @transport.event_handler ( "on_left" ) async def on_left ( transport ): logger.info( "Left room" ) ​ Participant Events ​ on_first_participant_joined async callback Called when the first participant joins the room. Useful for initializing conversations. Parameters: transport : The DailyTransport instance participant : Dictionary containing participant information Copy Ask AI @transport.event_handler ( "on_first_participant_joined" ) async def on_first_participant_joined ( transport , participant : Dict[ str , Any]): await transport.capture_participant_transcription(participant[ "id" ]) await task.queue_frames([LLMMessagesFrame(initial_messages)]) ​ on_participant_joined async callback Called when any participant joins the room. Parameters: transport : The DailyTransport instance participant : Dictionary containing participant information Copy Ask AI @transport.event_handler ( "on_participant_joined" ) async def on_participant_joined ( transport , participant : Dict[ str , Any]): logger.info( f "Participant joined: { participant[ 'id' ] } " ) ​ on_participant_left async callback Called when a participant leaves the room. Parameters: transport : The DailyTransport instance participant : Dictionary containing participant information reason : String describing why the participant left, “leftCall” | “hidden” Copy Ask AI @transport.event_handler ( "on_participant_left" ) async def on_participant_left ( transport , participant : Dict[ str , Any], reason : str ): logger.info( f "Participant { participant[ 'id' ] } left: { reason } " ) # Cancel the pipeline task to stop processing and remove the bot from the Daily room await task.cancel() ​ on_participant_updated async callback Event emitted when a participant is updated. This can mean either the participant’s metadata was updated, or the tracks belonging to the participant changed. Parameters: transport : The DailyTransport instance participant : Dictionary containing updated participant information Copy Ask AI @transport.event_handler ( "on_participant_updated" ) async def on_participant_updated ( transport , participant : Dict[ str , Any]): logger.info( f "Participant updated: { participant } " ) ​ Communication Events ​ on_app_message async callback Event emitted when a custom app message is received from another participant or via the REST API. Parameters: transport : The DailyTransport instance message : The message content (any type) sender : String identifier of the message sender Copy Ask AI @transport.event_handler ( "on_app_message" ) async def on_app_message ( transport , message : Any, sender : str ): logger.info( f "Message from { sender } : { message } " ) ​ on_call_state_updated async callback Event emitted when the call state changes, normally as a consequence of joining or leaving the call. Parameters: transport : The DailyTransport instance state : String representing the new call state. Learn more about call states . Copy Ask AI @transport.event_handler ( "on_call_state_updated" ) async def on_call_state_updated ( transport , state : str ): logger.info( f "Call state updated: { state } " ) ​ Dial Events ​ Dial-in ​ on_dialin_ready async callback Event emitted when dial-in is ready. This happens after the room has connected to the SIP endpoint and the system is ready to receive dial-in calls. Parameters: transport : The DailyTransport instance sip_endpoint : String containing the SIP endpoint information Copy Ask AI @transport.event_handler ( "on_dialin_ready" ) async def on_dialin_ready ( transport , sip_endpoint : str ): logger.info( f "Dial-in ready at: { sip_endpoint } " ) ​ on_dialin_connected async callback Event emitted when the session with the dial-in remote end is established (i.e. SIP endpoint or PSTN are connectd to the Daily room). Note: connected does not mean media (audio or video) has started flowing between the room and PSTN, it means the room received the connection request and both endpoints are negotiating the media flow. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. See DialinConnectedEvent . Copy Ask AI @transport.event_handler ( "on_dialin_connected" ) async def on_dialin_connected ( transport , data : Any): logger.info( f "Dial-in connected: { data } " ) ​ on_dialin_stopped async callback Event emitted when the dial-in remote end disconnects the call. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. See DialinStoppedEvent . Copy Ask AI @transport.event_handler ( "on_dialin_stopped" ) async def on_dialin_stopped ( transport , data : Any): logger.info( f "Dial-in stopped: { data } " ) ​ on_dialin_error async callback Event emitted in the case of dial-in errors which are fatal and the service cannot proceed. For example, an error in SDP negotiation is fatal to the media/SIP pipeline and will result in dialin-error being triggered. Parameters: transport : The DailyTransport instance data : Dictionary containing error information. See DialinEvent . Copy Ask AI @transport.event_handler ( "on_dialin_error" ) async def on_dialin_error ( transport , data : Any): logger.error( f "Dial-in error: { data } " ) ​ on_dialin_warning async callback Event emitted there is a dial-in non-fatal error, such as the selected codec not being used and a fallback codec being utilized. Parameters: transport : The DailyTransport instance data : Dictionary containing warning information. See DialinEvent . Copy Ask AI @transport.event_handler ( "on_dialin_warning" ) async def on_dialin_warning ( transport , data : Any): logger.warning( f "Dial-in warning: { data } " ) ​ Dial-out ​ on_dialout_answered async callback Event emitted when the session with the dial-out remote end is answered. Parameters: transport : The DailyTransport instance data : Dictionary containing call information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_answered" ) async def on_dialout_answered ( transport , data : Any): logger.info( f "Dial-out answered: { data } " ) ​ on_dialout_connected async callback Event emitted when the session with the dial-out remote end is established. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_connected" ) async def on_dialout_connected ( transport , data : Any): logger.info( f "Dial-out connected: { data } " ) ​ on_dialout_stopped async callback Event emitted when the dial-out session is stopped. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_stopped" ) async def on_dialout_stopped ( transport , data : Any): logger.info( f "Dial-out stopped: { data } " ) ​ on_dialout_error async callback Event emitted in the case of dial-out errors which are fatal and the service cannot proceed. For example, an error in SDP negotiation is fatal to the media/SIP pipeline and will result in dialout-error being triggered. Parameters: transport : The DailyTransport instance data : Dictionary containing error information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_error" ) async def on_dialout_error ( transport , data : Any): logger.error( f "Dial-out error: { data } " ) ​ on_dialout_warning async callback Event emitted there is a dial-out non-fatal error, such as the selected codec not being used and a fallback codec being utilized. Parameters: transport : The DailyTransport instance data : Dictionary containing warning information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_warning" ) async def on_dialout_warning ( transport , data : Any): logger.warning( f "Dial-out warning: { data } " ) ​ Transcription Events ​ on_transcription_message async callback Called when a transcription message is received. This includes both interim and final transcriptions. Parameters: transport : The DailyTransport instance message : Dictionary containing transcription data including. Learn more . Copy Ask AI @transport.event_handler ( "on_transcription_message" ) async def on_transcription_message ( transport , message : Dict[ str , Any]): participant_id = message.get( "participantId" ) text = message.get( "text" ) is_final = message[ "rawResponse" ][ "is_final" ] logger.info( f "Transcription from { participant_id } : { text } (final: { is_final } )" ) ​ Recording Events ​ on_recording_started async callback Called when a room recording starts successfully. Parameters: transport : The DailyTransport instance status : Dictionary containing recording status information. Learn more . Copy Ask AI @transport.event_handler ( "on_recording_started" ) async def on_recording_started ( transport , status : Dict[ str , Any]): logger.info( f "Recording started with status: { status } " ) ​ on_recording_stopped async callback Called when a room recording stops. Parameters: transport : The DailyTransport instance stream_id : String identifier of the stopped recording stream Copy Ask AI @transport.event_handler ( "on_recording_stopped" ) async def on_recording_stopped ( transport , stream_id : str ): logger.info( f "Recording stopped for stream: { stream_id } " ) ​ on_recording_error async callback Called when an error occurs during recording. Parameters: transport : The DailyTransport instance stream_id : String identifier of the recording stream message : Error message describing what went wrong Copy Ask AI @transport.event_handler ( "on_recording_error" ) async def on_recording_error ( transport , stream_id : str , message : str ): logger.error( f "Recording error for stream { stream_id } : { message } " ) ​ Error Events ​ on_error async callback Called when a transport error occurs. Parameters: transport : The DailyTransport instance error : String containing error details Copy Ask AI @transport.event_handler ( "on_error" ) async def on_error ( transport , error : str ): logger.error( f "Transport error: { error } " ) ​ Notes on Callbacks All callbacks are asynchronous Callbacks are executed in the order they were registered Multiple handlers can be registered for the same event Exceptions in callbacks are logged but don’t stop the transport Callbacks should be lightweight to avoid blocking the event loop Heavy processing should be offloaded to separate tasks ​ Frame Types ​ Input Frames ​ InputAudioRawFrame Frame Raw audio data from participants ​ UserImageRawFrame Frame Video frames from participants ​ Output Frames ​ OutputAudioRawFrame Frame Audio data to be sent ​ OutputImageRawFrame Frame Video frames to be sent ​ Methods ​ Room Management ​ participants method Returns a dictionary of all participants currently in the room. Copy Ask AI def participants () -> Dict[ str , Any] See participants for more details. ​ participant_counts method Returns participant count statistics for the room. Copy Ask AI def participant_counts () -> Dict[ str , int ] See participant_counts for more details. ​ send_message async method Sends a message to participants in the Daily room. Messages can be directed to all participants or targeted to a specific participant. Copy Ask AI async def send_message ( frame : TransportMessageFrame | TransportMessageUrgentFrame) Parameters: frame : A transport message frame containing the message to send. Can be either: TransportMessageFrame : Standard message TransportMessageUrgentFrame : High-priority message Message Structure: Messages should conform to the expected format for the receiving system. For RTVI messages, use: Copy Ask AI { "label" : "rtvi-ai" , # Required for RTVI protocol "type" : "message-type" , # Defines how the client will process it "id" : "unique-id" , # Unique identifier for the message "data" : { # Custom payload # Your custom data fields } } Example - Standard Message: Copy Ask AI import uuid # Send a standard message to all participants message_data = { "label" : "rtvi-ai" , "type" : "custom-message" , "id" : str (uuid.uuid4()), "data" : { "message" : "Hello everyone!" , "timestamp" : datetime.datetime.now().isoformat() } } await transport.send_message(TransportMessageFrame( message = message_data)) Example - Urgent Message to Specific Participant: Copy Ask AI # Send urgent message to specific participant urgent_data = { "label" : "rtvi-ai" , "type" : "server-message" , "id" : str (uuid.uuid4()), "data" : { "message" : "Private message" , "importance" : "high" } } await transport.send_message( DailyTransportMessageUrgentFrame( message = urgent_data, participant_id = "participant-123" ) ) The message will be delivered to: All participants if no specific participant_id is set Only the specified participant if participant_id is set Through the Daily app messaging system Can be received by registering an on_app_message handler Use TransportMessageUrgentFrame for high-priority messages that need immediate delivery. ​ Media Control ​ send_image async method Sends an image frame to the room. Copy Ask AI async def send_image ( frame : OutputImageRawFrame | SpriteFrame) -> None Parameters: frame : Image frame to send, either raw image data or sprite animation ​ send_audio async method Sends an audio frame to the room. Copy Ask AI async def send_audio ( frame : OutputAudioRawFrame) -> None Parameters: frame : Audio frame to send ​ Video Management ​ capture_participant_video async method Starts capturing video from a specific participant. Copy Ask AI async def capture_participant_video ( participant_id : str , framerate : int = 30 , video_source : str = "camera" , color_format : str = "RGB" ) -> None Parameters: participant_id : ID of the participant to capture framerate : Target frame rate (default: 30) video_source : Video source type (default: “camera”) color_format : Color format of the video (default: “RGB”) To request an image from the camera stream, set the framerate to 0 and push a UserImageRequestFrame . ​ Transcription Control ​ capture_participant_transcription async method Starts capturing and transcribing audio from a specific participant. Copy Ask AI async def capture_participant_transcription ( participant_id : str ) -> None Parameters: participant_id : ID of the participant to transcribe ​ update_transcription async method Updates the transcription configuration for specific participants or instances. Copy Ask AI async def update_transcription ( participants : List[ str ] | None = None , instance_id : str | None = None ) -> None Parameters: participants : Optional list of participant IDs to transcribe. If None, transcribes all participants. instance_id : Optional specific transcription instance ID to update Example: Copy Ask AI # Update transcription for specific participants await transport.update_transcription( participants = [ "participant-123" , "participant-456" ]) # Update specific transcription instance await transport.update_transcription( instance_id = "transcription-789" ) ​ Recording Control ​ start_recording async method Starts recording the room session. Copy Ask AI async def start_recording ( streaming_settings : Dict = None , stream_id : str = None , force_new : bool = None ) -> None Parameters: streaming_settings : Recording configuration settings stream_id : Optional stream identifier force_new : Force start a new recording See start_recording for more details. ​ stop_recording async method Stops an active recording. Copy Ask AI async def stop_recording ( stream_id : str = None ) -> None Parameters: stream_id : Optional stream identifier to stop specific recording See stop_recording for more details. ​ Dial-out Control ​ start_dialout async method Initiates a dial-out call. Copy Ask AI async def start_dialout ( settings : Dict = None ) -> None Parameters: settings : Dial-out configuration settings See start_dialout for more details. ​ stop_dialout async method Stops an active dial-out call. Copy Ask AI async def stop_dialout ( participant_id : str ) -> None Parameters: participant_id : ID of the participant to disconnect See stop_dialout for more details. ​ send_dtmf async method Sends DTMF tones in an existing dial-out session. Copy Ask AI async def send_dtmf ( settings : Dict = None ) -> None Parameters: settings : DTMF settings See send_dtmf for more details. ​ Subscription Management ​ update_subscriptions async method Updates subscriptions and subscription profiles. This function allows you to update subscription profiles and at the same time assign specific subscription profiles to a participant and even change specific settings for some participants. Copy Ask AI async def update_subscriptions ( participant_settings : Dict = None , profile_settings : Dict = None ) -> None Parameters: participant_settings : Per-participant subscription settings profile_settings : Global subscription profile settings See update_subscriptions for more details. ​ Participant Admin When using the DailyTransport, if your bot is given a meeting token with owner or participant admin permissions, it can remote-control other participants’ capabilities and permissions. With this control, the bot can modify settings for how others send and receive media, how they are seen in a session, and whether they have administrative controls. ​ update_remote_participants async method Updates permissions and settings for specific participants in the room. Copy Ask AI async def update_remote_participants ( remote_participants : Mapping[ str , Any] = None ) -> None Parameters: remote_participants : Dictionary mapping participant IDs to their updated settings This method allows fine-grained control over participant capabilities and permissions in the Daily room. Examples: Copy Ask AI # Mute a participant's microphone await transport.update_remote_participants({ "participant-123" : { "inputsEnabled" : { "microphone" : False } } }) # Revoke send permissions (mutes the participant and prevents them from unmuting) await transport.update_remote_participants({ "participant-123" : { "permissions" : { "canSend" : [] # Empty list revokes all send permissions } } }) # Configure selective audio routing await transport.update_remote_participants({ "participant-123" : { # the customer "permissions" : { "canReceive" : { "base" : False , # Don't receive from anyone by default "byUserId" : { "hold-music" : True # Only receive from hold music player } } } } }) # Set up user-to-user permissions mapping (e.g., when transferring calls) # Assumes customer was previously unable to hear anyone (they were on hold) and agent was speaking to bot await transport.update_remote_participants({ "participant-123" : { # the customer "permissions" : { "base" : False , "canReceive" : { "byUserId" : { "agent" : True }} } }, "participant-456" : { # the agent "permissions" : { "base" : False , "canReceive" : { "byUserId" : { "customer" : True }} } } }) # Promote a participant to be a participant admin await transport.update_remote_participants({ "participant-123" : { "permissions" : { "canAdmin" : [ "participants" ] } } }) Permissions Settings: For comprehensive details on permissions settings, see the Daily API documentation: daily-python’s update_remote_participants() documentation daily-js’s updateParticipant() -> updatePermission and participants().permissions ​ Chat messages ​ send_prebuilt_chat_message async method When testing with Daily Prebuilt, you can send messages to the Prebuilt chat using this method. Parameters: message (str): The chat message to send user_name (str): The name of the user sending the message See send_prebuilt_chat_message for more details. ​ Properties ​ participant_id string Returns the transport’s participant ID in the room. ​ room_url string Returns the transport’s room URL for the room currently in use. ​ Notes Supports real-time audio/video communication Handles WebRTC connection management Provides voice activity detection Includes transcription capabilities Manages participant interactions Supports recording and streaming Thread-safe processing All callbacks are asynchronous Multiple handlers can be registered for the same event Exceptions in callbacks are logged but don’t stop the transport Supported Services SmallWebRTCTransport On this page Overview Installation Configuration Constructor Parameters DailyParams Configuration Audio Output Configuration Audio Input Configuration Video Output Configuration Voice Activity Detection (VAD) Feature Settings Basic Usage Event Callbacks Connection Events Participant Events Communication Events Dial Events Dial-in Dial-out Transcription Events Recording Events Error Events Notes on Callbacks Frame Types Input Frames Output Frames Methods Room Management Media Control Video Management Transcription Control Recording Control Dial-out Control Subscription Management Participant Admin Chat messages Properties Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/transport_daily_e6c3197b.txt b/transport_daily_e6c3197b.txt
new file mode 100644
index 0000000000000000000000000000000000000000..345405470f51e03396ee813eb14da7c384338ec6
--- /dev/null
+++ b/transport_daily_e6c3197b.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/transport/daily#transcription-events
+Title: Daily WebRTC - Pipecat
+==================================================
+
+Daily WebRTC - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Transport Daily WebRTC Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Daily WebRTC SmallWebRTCTransport FastAPI WebSocket WebSocket Server Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview DailyTransport provides real-time audio and video communication capabilities using Daily’s WebRTC platform. It supports bidirectional audio/video streams, transcription, voice activity detection (VAD), and participant management. ​ Installation To use DailyTransport , install the required dependencies: Copy Ask AI pip install "pipecat-ai[daily]" You’ll also need to set up your Daily API key as an environment variable: DAILY_API_KEY . You can obtain a Daily API key by signing up at Daily . ​ Configuration ​ Constructor Parameters ​ room_url str required Daily room URL ​ token str | None Daily room token ​ bot_name str required Name of the bot in the room ​ params DailyParams default: "DailyParams()" Transport configuration parameters ​ DailyParams Configuration ​ api_url str default: "https://api.daily.co/v1" Daily API endpoint URL ​ api_key str default: "" Daily API key for authentication ​ Audio Output Configuration ​ audio_out_enabled bool default: "False" Enable audio output capabilities ​ audio_out_is_live bool default: "False" Enable live audio streaming mode ​ audio_out_sample_rate int default: "None" Audio output sample rate in Hz ​ audio_out_channels int default: "1" Number of audio output channels ​ audio_out_bitrate int default: "96000" Audio output bitrate in bits per second ​ Audio Input Configuration ​ audio_in_enabled bool default: "False" Enable audio input capabilities ​ audio_in_passthrough bool default: "False" When enabled, incoming audio frames are pushed downstream ​ audio_in_sample_rate int default: "None" Audio input sample rate in Hz ​ audio_in_channels int default: "1" Number of audio input channels ​ audio_in_filter Optional[BaseAudioFilter] default: "None" Audio filter for input processing. Supported filters are: KrispFilter() and NoisereduceFilter() . See the KrispFilter() reference docs and NoisereduceFilter() reference docs ` for more information. ​ Video Output Configuration ​ video_out_enabled bool default: "False" Enable video output capabilities ​ video_out_is_live bool default: "False" Enable live video streaming mode ​ video_out_width int default: "1024" Video output width in pixels ​ video_out_height int default: "768" Video output height in pixels ​ video_out_bitrate int default: "800000" Video output bitrate in bits per second ​ video_out_framerate int default: "30" Video output frame rate ​ video_out_color_format str default: "RGB" Video color format (RGB, BGR, etc.) ​ Voice Activity Detection (VAD) ​ vad_analyzer VADAnalyzer | None default: "None" Voice Activity Detection analyzer. You can set this to either SileroVADAnalyzer() or WebRTCVADAnalyzer() . SileroVADAnalyzer is the recommended option. Learn more about the SileroVADAnalyzer . ​ Feature Settings ​ dialin_settings Optional[DailyDialinSettings] default: "None" Configuration for dial-in functionality ​ transcription_enabled bool default: "False" Enable real-time transcription ​ transcription_settings DailyTranscriptionSettings Configuration for transcription features Copy Ask AI class DailyTranscriptionSettings ( BaseModel ): language: str = "en" # Default language model: str = "nova-2-general" # Transcription model profanity_filter: bool = True # Filter profanity redact: bool = False # Redact sensitive information endpointing: bool = True # Enable speech endpointing punctuate: bool = True # Add punctuation includeRawResponse: bool = True # Include raw API response extra: Mapping[ str , Any] = { "interim_results" : True # Provide any Deepgram-specific settings } ​ Basic Usage Copy Ask AI from pipecat.transports.services.daily import DailyTransport, DailyParams from pipecat.audio.vad.silero import SileroVADAnalyzer # Configure transport transport = DailyTransport( room_url = "https://your-domain.daily.co/room-name" , token = "your-room-token" , bot_name = "AI Assistant" , params = DailyParams( audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer( params = VADParams( stop_secs = 0.5 )), ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), # Handle incoming audio/video stt, # Speech-to-text llm, # Language model tts, # Text-to-speech transport.output() # Handle outgoing audio/video ]) # Use transport event handlers to manage the call lifecycle # Event handler for when the user joins the room: @transport.event_handler ( "on_first_participant_joined" ) async def on_first_participant_joined ( transport , participant ): # Start transcription for the user await transport.capture_participant_transcription(participant[ "id" ]) # Kick off the conversation. await task.queue_frames([context_aggregator.user().get_context_frame()]) # Event handler for when the user leaves the room: @transport.event_handler ( "on_participant_left" ) async def on_participant_left ( transport , participant , reason ): # Cancel the pipeline, which stops processing and removes the bot from the room await task.cancel() ​ Event Callbacks DailyTransport provides a comprehensive callback system for handling various events. Register callbacks using the @transport.event_handler() decorator. ​ Connection Events ​ on_joined async callback Called when the bot successfully joins the room. Parameters: transport : The DailyTransport instance data : Dictionary containing room join information Copy Ask AI @transport.event_handler ( "on_joined" ) async def on_joined ( transport , data : Dict[ str , Any]): logger.info( f "Joined room with data: { data } " ) ​ on_left async callback Called when the bot leaves the room. Parameters: transport : The DailyTransport instance Copy Ask AI @transport.event_handler ( "on_left" ) async def on_left ( transport ): logger.info( "Left room" ) ​ Participant Events ​ on_first_participant_joined async callback Called when the first participant joins the room. Useful for initializing conversations. Parameters: transport : The DailyTransport instance participant : Dictionary containing participant information Copy Ask AI @transport.event_handler ( "on_first_participant_joined" ) async def on_first_participant_joined ( transport , participant : Dict[ str , Any]): await transport.capture_participant_transcription(participant[ "id" ]) await task.queue_frames([LLMMessagesFrame(initial_messages)]) ​ on_participant_joined async callback Called when any participant joins the room. Parameters: transport : The DailyTransport instance participant : Dictionary containing participant information Copy Ask AI @transport.event_handler ( "on_participant_joined" ) async def on_participant_joined ( transport , participant : Dict[ str , Any]): logger.info( f "Participant joined: { participant[ 'id' ] } " ) ​ on_participant_left async callback Called when a participant leaves the room. Parameters: transport : The DailyTransport instance participant : Dictionary containing participant information reason : String describing why the participant left, “leftCall” | “hidden” Copy Ask AI @transport.event_handler ( "on_participant_left" ) async def on_participant_left ( transport , participant : Dict[ str , Any], reason : str ): logger.info( f "Participant { participant[ 'id' ] } left: { reason } " ) # Cancel the pipeline task to stop processing and remove the bot from the Daily room await task.cancel() ​ on_participant_updated async callback Event emitted when a participant is updated. This can mean either the participant’s metadata was updated, or the tracks belonging to the participant changed. Parameters: transport : The DailyTransport instance participant : Dictionary containing updated participant information Copy Ask AI @transport.event_handler ( "on_participant_updated" ) async def on_participant_updated ( transport , participant : Dict[ str , Any]): logger.info( f "Participant updated: { participant } " ) ​ Communication Events ​ on_app_message async callback Event emitted when a custom app message is received from another participant or via the REST API. Parameters: transport : The DailyTransport instance message : The message content (any type) sender : String identifier of the message sender Copy Ask AI @transport.event_handler ( "on_app_message" ) async def on_app_message ( transport , message : Any, sender : str ): logger.info( f "Message from { sender } : { message } " ) ​ on_call_state_updated async callback Event emitted when the call state changes, normally as a consequence of joining or leaving the call. Parameters: transport : The DailyTransport instance state : String representing the new call state. Learn more about call states . Copy Ask AI @transport.event_handler ( "on_call_state_updated" ) async def on_call_state_updated ( transport , state : str ): logger.info( f "Call state updated: { state } " ) ​ Dial Events ​ Dial-in ​ on_dialin_ready async callback Event emitted when dial-in is ready. This happens after the room has connected to the SIP endpoint and the system is ready to receive dial-in calls. Parameters: transport : The DailyTransport instance sip_endpoint : String containing the SIP endpoint information Copy Ask AI @transport.event_handler ( "on_dialin_ready" ) async def on_dialin_ready ( transport , sip_endpoint : str ): logger.info( f "Dial-in ready at: { sip_endpoint } " ) ​ on_dialin_connected async callback Event emitted when the session with the dial-in remote end is established (i.e. SIP endpoint or PSTN are connectd to the Daily room). Note: connected does not mean media (audio or video) has started flowing between the room and PSTN, it means the room received the connection request and both endpoints are negotiating the media flow. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. See DialinConnectedEvent . Copy Ask AI @transport.event_handler ( "on_dialin_connected" ) async def on_dialin_connected ( transport , data : Any): logger.info( f "Dial-in connected: { data } " ) ​ on_dialin_stopped async callback Event emitted when the dial-in remote end disconnects the call. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. See DialinStoppedEvent . Copy Ask AI @transport.event_handler ( "on_dialin_stopped" ) async def on_dialin_stopped ( transport , data : Any): logger.info( f "Dial-in stopped: { data } " ) ​ on_dialin_error async callback Event emitted in the case of dial-in errors which are fatal and the service cannot proceed. For example, an error in SDP negotiation is fatal to the media/SIP pipeline and will result in dialin-error being triggered. Parameters: transport : The DailyTransport instance data : Dictionary containing error information. See DialinEvent . Copy Ask AI @transport.event_handler ( "on_dialin_error" ) async def on_dialin_error ( transport , data : Any): logger.error( f "Dial-in error: { data } " ) ​ on_dialin_warning async callback Event emitted there is a dial-in non-fatal error, such as the selected codec not being used and a fallback codec being utilized. Parameters: transport : The DailyTransport instance data : Dictionary containing warning information. See DialinEvent . Copy Ask AI @transport.event_handler ( "on_dialin_warning" ) async def on_dialin_warning ( transport , data : Any): logger.warning( f "Dial-in warning: { data } " ) ​ Dial-out ​ on_dialout_answered async callback Event emitted when the session with the dial-out remote end is answered. Parameters: transport : The DailyTransport instance data : Dictionary containing call information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_answered" ) async def on_dialout_answered ( transport , data : Any): logger.info( f "Dial-out answered: { data } " ) ​ on_dialout_connected async callback Event emitted when the session with the dial-out remote end is established. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_connected" ) async def on_dialout_connected ( transport , data : Any): logger.info( f "Dial-out connected: { data } " ) ​ on_dialout_stopped async callback Event emitted when the dial-out session is stopped. Parameters: transport : The DailyTransport instance data : Dictionary containing connection information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_stopped" ) async def on_dialout_stopped ( transport , data : Any): logger.info( f "Dial-out stopped: { data } " ) ​ on_dialout_error async callback Event emitted in the case of dial-out errors which are fatal and the service cannot proceed. For example, an error in SDP negotiation is fatal to the media/SIP pipeline and will result in dialout-error being triggered. Parameters: transport : The DailyTransport instance data : Dictionary containing error information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_error" ) async def on_dialout_error ( transport , data : Any): logger.error( f "Dial-out error: { data } " ) ​ on_dialout_warning async callback Event emitted there is a dial-out non-fatal error, such as the selected codec not being used and a fallback codec being utilized. Parameters: transport : The DailyTransport instance data : Dictionary containing warning information. Learn more . Copy Ask AI @transport.event_handler ( "on_dialout_warning" ) async def on_dialout_warning ( transport , data : Any): logger.warning( f "Dial-out warning: { data } " ) ​ Transcription Events ​ on_transcription_message async callback Called when a transcription message is received. This includes both interim and final transcriptions. Parameters: transport : The DailyTransport instance message : Dictionary containing transcription data including. Learn more . Copy Ask AI @transport.event_handler ( "on_transcription_message" ) async def on_transcription_message ( transport , message : Dict[ str , Any]): participant_id = message.get( "participantId" ) text = message.get( "text" ) is_final = message[ "rawResponse" ][ "is_final" ] logger.info( f "Transcription from { participant_id } : { text } (final: { is_final } )" ) ​ Recording Events ​ on_recording_started async callback Called when a room recording starts successfully. Parameters: transport : The DailyTransport instance status : Dictionary containing recording status information. Learn more . Copy Ask AI @transport.event_handler ( "on_recording_started" ) async def on_recording_started ( transport , status : Dict[ str , Any]): logger.info( f "Recording started with status: { status } " ) ​ on_recording_stopped async callback Called when a room recording stops. Parameters: transport : The DailyTransport instance stream_id : String identifier of the stopped recording stream Copy Ask AI @transport.event_handler ( "on_recording_stopped" ) async def on_recording_stopped ( transport , stream_id : str ): logger.info( f "Recording stopped for stream: { stream_id } " ) ​ on_recording_error async callback Called when an error occurs during recording. Parameters: transport : The DailyTransport instance stream_id : String identifier of the recording stream message : Error message describing what went wrong Copy Ask AI @transport.event_handler ( "on_recording_error" ) async def on_recording_error ( transport , stream_id : str , message : str ): logger.error( f "Recording error for stream { stream_id } : { message } " ) ​ Error Events ​ on_error async callback Called when a transport error occurs. Parameters: transport : The DailyTransport instance error : String containing error details Copy Ask AI @transport.event_handler ( "on_error" ) async def on_error ( transport , error : str ): logger.error( f "Transport error: { error } " ) ​ Notes on Callbacks All callbacks are asynchronous Callbacks are executed in the order they were registered Multiple handlers can be registered for the same event Exceptions in callbacks are logged but don’t stop the transport Callbacks should be lightweight to avoid blocking the event loop Heavy processing should be offloaded to separate tasks ​ Frame Types ​ Input Frames ​ InputAudioRawFrame Frame Raw audio data from participants ​ UserImageRawFrame Frame Video frames from participants ​ Output Frames ​ OutputAudioRawFrame Frame Audio data to be sent ​ OutputImageRawFrame Frame Video frames to be sent ​ Methods ​ Room Management ​ participants method Returns a dictionary of all participants currently in the room. Copy Ask AI def participants () -> Dict[ str , Any] See participants for more details. ​ participant_counts method Returns participant count statistics for the room. Copy Ask AI def participant_counts () -> Dict[ str , int ] See participant_counts for more details. ​ send_message async method Sends a message to participants in the Daily room. Messages can be directed to all participants or targeted to a specific participant. Copy Ask AI async def send_message ( frame : TransportMessageFrame | TransportMessageUrgentFrame) Parameters: frame : A transport message frame containing the message to send. Can be either: TransportMessageFrame : Standard message TransportMessageUrgentFrame : High-priority message Message Structure: Messages should conform to the expected format for the receiving system. For RTVI messages, use: Copy Ask AI { "label" : "rtvi-ai" , # Required for RTVI protocol "type" : "message-type" , # Defines how the client will process it "id" : "unique-id" , # Unique identifier for the message "data" : { # Custom payload # Your custom data fields } } Example - Standard Message: Copy Ask AI import uuid # Send a standard message to all participants message_data = { "label" : "rtvi-ai" , "type" : "custom-message" , "id" : str (uuid.uuid4()), "data" : { "message" : "Hello everyone!" , "timestamp" : datetime.datetime.now().isoformat() } } await transport.send_message(TransportMessageFrame( message = message_data)) Example - Urgent Message to Specific Participant: Copy Ask AI # Send urgent message to specific participant urgent_data = { "label" : "rtvi-ai" , "type" : "server-message" , "id" : str (uuid.uuid4()), "data" : { "message" : "Private message" , "importance" : "high" } } await transport.send_message( DailyTransportMessageUrgentFrame( message = urgent_data, participant_id = "participant-123" ) ) The message will be delivered to: All participants if no specific participant_id is set Only the specified participant if participant_id is set Through the Daily app messaging system Can be received by registering an on_app_message handler Use TransportMessageUrgentFrame for high-priority messages that need immediate delivery. ​ Media Control ​ send_image async method Sends an image frame to the room. Copy Ask AI async def send_image ( frame : OutputImageRawFrame | SpriteFrame) -> None Parameters: frame : Image frame to send, either raw image data or sprite animation ​ send_audio async method Sends an audio frame to the room. Copy Ask AI async def send_audio ( frame : OutputAudioRawFrame) -> None Parameters: frame : Audio frame to send ​ Video Management ​ capture_participant_video async method Starts capturing video from a specific participant. Copy Ask AI async def capture_participant_video ( participant_id : str , framerate : int = 30 , video_source : str = "camera" , color_format : str = "RGB" ) -> None Parameters: participant_id : ID of the participant to capture framerate : Target frame rate (default: 30) video_source : Video source type (default: “camera”) color_format : Color format of the video (default: “RGB”) To request an image from the camera stream, set the framerate to 0 and push a UserImageRequestFrame . ​ Transcription Control ​ capture_participant_transcription async method Starts capturing and transcribing audio from a specific participant. Copy Ask AI async def capture_participant_transcription ( participant_id : str ) -> None Parameters: participant_id : ID of the participant to transcribe ​ update_transcription async method Updates the transcription configuration for specific participants or instances. Copy Ask AI async def update_transcription ( participants : List[ str ] | None = None , instance_id : str | None = None ) -> None Parameters: participants : Optional list of participant IDs to transcribe. If None, transcribes all participants. instance_id : Optional specific transcription instance ID to update Example: Copy Ask AI # Update transcription for specific participants await transport.update_transcription( participants = [ "participant-123" , "participant-456" ]) # Update specific transcription instance await transport.update_transcription( instance_id = "transcription-789" ) ​ Recording Control ​ start_recording async method Starts recording the room session. Copy Ask AI async def start_recording ( streaming_settings : Dict = None , stream_id : str = None , force_new : bool = None ) -> None Parameters: streaming_settings : Recording configuration settings stream_id : Optional stream identifier force_new : Force start a new recording See start_recording for more details. ​ stop_recording async method Stops an active recording. Copy Ask AI async def stop_recording ( stream_id : str = None ) -> None Parameters: stream_id : Optional stream identifier to stop specific recording See stop_recording for more details. ​ Dial-out Control ​ start_dialout async method Initiates a dial-out call. Copy Ask AI async def start_dialout ( settings : Dict = None ) -> None Parameters: settings : Dial-out configuration settings See start_dialout for more details. ​ stop_dialout async method Stops an active dial-out call. Copy Ask AI async def stop_dialout ( participant_id : str ) -> None Parameters: participant_id : ID of the participant to disconnect See stop_dialout for more details. ​ send_dtmf async method Sends DTMF tones in an existing dial-out session. Copy Ask AI async def send_dtmf ( settings : Dict = None ) -> None Parameters: settings : DTMF settings See send_dtmf for more details. ​ Subscription Management ​ update_subscriptions async method Updates subscriptions and subscription profiles. This function allows you to update subscription profiles and at the same time assign specific subscription profiles to a participant and even change specific settings for some participants. Copy Ask AI async def update_subscriptions ( participant_settings : Dict = None , profile_settings : Dict = None ) -> None Parameters: participant_settings : Per-participant subscription settings profile_settings : Global subscription profile settings See update_subscriptions for more details. ​ Participant Admin When using the DailyTransport, if your bot is given a meeting token with owner or participant admin permissions, it can remote-control other participants’ capabilities and permissions. With this control, the bot can modify settings for how others send and receive media, how they are seen in a session, and whether they have administrative controls. ​ update_remote_participants async method Updates permissions and settings for specific participants in the room. Copy Ask AI async def update_remote_participants ( remote_participants : Mapping[ str , Any] = None ) -> None Parameters: remote_participants : Dictionary mapping participant IDs to their updated settings This method allows fine-grained control over participant capabilities and permissions in the Daily room. Examples: Copy Ask AI # Mute a participant's microphone await transport.update_remote_participants({ "participant-123" : { "inputsEnabled" : { "microphone" : False } } }) # Revoke send permissions (mutes the participant and prevents them from unmuting) await transport.update_remote_participants({ "participant-123" : { "permissions" : { "canSend" : [] # Empty list revokes all send permissions } } }) # Configure selective audio routing await transport.update_remote_participants({ "participant-123" : { # the customer "permissions" : { "canReceive" : { "base" : False , # Don't receive from anyone by default "byUserId" : { "hold-music" : True # Only receive from hold music player } } } } }) # Set up user-to-user permissions mapping (e.g., when transferring calls) # Assumes customer was previously unable to hear anyone (they were on hold) and agent was speaking to bot await transport.update_remote_participants({ "participant-123" : { # the customer "permissions" : { "base" : False , "canReceive" : { "byUserId" : { "agent" : True }} } }, "participant-456" : { # the agent "permissions" : { "base" : False , "canReceive" : { "byUserId" : { "customer" : True }} } } }) # Promote a participant to be a participant admin await transport.update_remote_participants({ "participant-123" : { "permissions" : { "canAdmin" : [ "participants" ] } } }) Permissions Settings: For comprehensive details on permissions settings, see the Daily API documentation: daily-python’s update_remote_participants() documentation daily-js’s updateParticipant() -> updatePermission and participants().permissions ​ Chat messages ​ send_prebuilt_chat_message async method When testing with Daily Prebuilt, you can send messages to the Prebuilt chat using this method. Parameters: message (str): The chat message to send user_name (str): The name of the user sending the message See send_prebuilt_chat_message for more details. ​ Properties ​ participant_id string Returns the transport’s participant ID in the room. ​ room_url string Returns the transport’s room URL for the room currently in use. ​ Notes Supports real-time audio/video communication Handles WebRTC connection management Provides voice activity detection Includes transcription capabilities Manages participant interactions Supports recording and streaming Thread-safe processing All callbacks are asynchronous Multiple handlers can be registered for the same event Exceptions in callbacks are logged but don’t stop the transport Supported Services SmallWebRTCTransport On this page Overview Installation Configuration Constructor Parameters DailyParams Configuration Audio Output Configuration Audio Input Configuration Video Output Configuration Voice Activity Detection (VAD) Feature Settings Basic Usage Event Callbacks Connection Events Participant Events Communication Events Dial Events Dial-in Dial-out Transcription Events Recording Events Error Events Notes on Callbacks Frame Types Input Frames Output Frames Methods Room Management Media Control Video Management Transcription Control Recording Control Dial-out Control Subscription Management Participant Admin Chat messages Properties Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/transport_fastapi-websocket_0cd31054.txt b/transport_fastapi-websocket_0cd31054.txt
new file mode 100644
index 0000000000000000000000000000000000000000..70edca42138785cbbb16cdc897b7d924846aadd9
--- /dev/null
+++ b/transport_fastapi-websocket_0cd31054.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/transport/fastapi-websocket#event-callbacks
+Title: FastAPI WebSocket - Pipecat
+==================================================
+
+FastAPI WebSocket - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Transport FastAPI WebSocket Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Daily WebRTC SmallWebRTCTransport FastAPI WebSocket WebSocket Server Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview FastAPIWebsocketTransport provides WebSocket support for FastAPI web applications, enabling real-time audio communication. It supports bidirectional audio streams and voice activity detection (VAD). FastAPIWebsocketTransport is best suited for server-side applications and prototyping client/server apps. For client/server production applications, we strongly recommend using a WebRTC-based transport for robust network and media handling. ​ Installation To use FastAPIWebsocketTransport , install the required dependencies: Copy Ask AI pip install "pipecat-ai[websocket]" ​ Configuration ​ Constructor Parameters ​ websocket WebSocket required FastAPI WebSocket connection instance ​ params FastAPIWebsocketParams required Transport configuration parameters ​ FastAPIWebsocketParams Configuration ​ add_wav_header bool default: "False" Add WAV header to audio frames ​ serializer FrameSerializer required Frame serializer for WebSocket messages. Common options include: ExotelFrameSerializer - For Exotel Websocket streaming integration PlivoFrameSerializer - For Plivo Websocket streaming integration TelnyxFrameSerializer - For Telnyx WebSocket streaming integration TwilioFrameSerializer - For Twilio Media Streams integration See the Frame Serializers documentation for more details. ​ session_timeout int | None default: "None" Session timeout in seconds. If set, triggers timeout event when no activity is detected ​ Audio Configuration ​ audio_in_enabled bool default: false Enable audio input from the WebRTC client ​ audio_in_passthrough bool default: "False" When enabled, incoming audio frames are pushed downstream ​ audio_out_enabled bool default: "False" Enable audio output capabilities ​ audio_out_sample_rate int default: "None" Audio output sample rate in Hz ​ audio_out_channels int default: "1" Number of audio output channels ​ Voice Activity Detection (VAD) ​ vad_analyzer VADAnalyzer | None default: "None" Voice Activity Detection analyzer. You can set this to either SileroVADAnalyzer() or WebRTCVADAnalyzer() . SileroVADAnalyzer is the recommended option. Learn more about the SileroVADAnalyzer . ​ Basic Usage Copy Ask AI from fastapi import FastAPI, WebSocket from pipecat.transports.network.fastapi_websocket import ( FastAPIWebsocketTransport, FastAPIWebsocketParams ) from pipecat.audio.vad.silero import SileroVADAnalyzer from pipecat.pipeline.pipeline import Pipeline from pipecat.serializers.twilio import TwilioFrameSerializer app = FastAPI() @app.websocket ( "/ws" ) async def websocket_endpoint ( websocket : WebSocket): await websocket.accept() # Configure transport transport = FastAPIWebsocketTransport( websocket = websocket, params = FastAPIWebsocketParams( audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer(), serializer = TwilioFrameSerializer(), ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), # Handle incoming audio stt, # Speech-to-text llm, # Language model tts, # Text-to-speech transport.output() # Handle outgoing audio ]) # Run pipeline task = PipelineTask(pipeline) await PipelineRunner().run(task) Check out the Twilio Chatbot example to see how to use the FastAPI transport in a phone application. ​ Event Callbacks FastAPIWebsocketTransport provides callbacks for handling client connection events. Register callbacks using the @transport.event_handler() decorator. ​ Connection Events ​ on_client_connected async callback Called when a client connects to the WebSocket endpoint. Parameters: transport : The FastAPIWebsocketTransport instance client : FastAPI WebSocket connection object Copy Ask AI @transport.event_handler ( "on_client_connected" ) async def on_client_connected ( transport , client ): logger.info( "Client connected" ) # Initialize conversation await task.queue_frames([LLMMessagesFrame(initial_messages)]) ​ on_client_disconnected async callback Called when a client disconnects from the WebSocket endpoint. Parameters: transport : The FastAPIWebsocketTransport instance client : FastAPI WebSocket connection object Copy Ask AI @transport.event_handler ( "on_client_disconnected" ) async def on_client_disconnected ( transport , client ): logger.info( "Client disconnected" ) await task.queue_frames([EndFrame()]) ​ on_session_timeout async callback Called when a session times out (if session_timeout is configured). Parameters: transport : The FastAPIWebsocketTransport instance client : FastAPI WebSocket connection object Copy Ask AI @transport.event_handler ( "on_session_timeout" ) async def on_session_timeout ( transport , client ): logger.info( "Session timeout" ) # Handle timeout (e.g., send message, close connection) ​ Frame Types ​ Input Frames ​ InputAudioRawFrame Frame Raw audio data from the WebSocket client ​ Output Frames ​ OutputAudioRawFrame Frame Audio data to be sent to the WebSocket client ​ Notes Integrates with FastAPI web applications Supports real-time audio communication Handles WebSocket connection management Provides voice activity detection Supports session timeouts All callbacks are asynchronous Compatible with various frame serializers SmallWebRTCTransport WebSocket Server On this page Overview Installation Configuration Constructor Parameters FastAPIWebsocketParams Configuration Audio Configuration Voice Activity Detection (VAD) Basic Usage Event Callbacks Connection Events Frame Types Input Frames Output Frames Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/transport_fastapi-websocket_3a63432a.txt b/transport_fastapi-websocket_3a63432a.txt
new file mode 100644
index 0000000000000000000000000000000000000000..c5a67bc9585f8a86aad3280e74a2a97338ba7650
--- /dev/null
+++ b/transport_fastapi-websocket_3a63432a.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/transport/fastapi-websocket#param-add-wav-header
+Title: FastAPI WebSocket - Pipecat
+==================================================
+
+FastAPI WebSocket - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Transport FastAPI WebSocket Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Daily WebRTC SmallWebRTCTransport FastAPI WebSocket WebSocket Server Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview FastAPIWebsocketTransport provides WebSocket support for FastAPI web applications, enabling real-time audio communication. It supports bidirectional audio streams and voice activity detection (VAD). FastAPIWebsocketTransport is best suited for server-side applications and prototyping client/server apps. For client/server production applications, we strongly recommend using a WebRTC-based transport for robust network and media handling. ​ Installation To use FastAPIWebsocketTransport , install the required dependencies: Copy Ask AI pip install "pipecat-ai[websocket]" ​ Configuration ​ Constructor Parameters ​ websocket WebSocket required FastAPI WebSocket connection instance ​ params FastAPIWebsocketParams required Transport configuration parameters ​ FastAPIWebsocketParams Configuration ​ add_wav_header bool default: "False" Add WAV header to audio frames ​ serializer FrameSerializer required Frame serializer for WebSocket messages. Common options include: ExotelFrameSerializer - For Exotel Websocket streaming integration PlivoFrameSerializer - For Plivo Websocket streaming integration TelnyxFrameSerializer - For Telnyx WebSocket streaming integration TwilioFrameSerializer - For Twilio Media Streams integration See the Frame Serializers documentation for more details. ​ session_timeout int | None default: "None" Session timeout in seconds. If set, triggers timeout event when no activity is detected ​ Audio Configuration ​ audio_in_enabled bool default: false Enable audio input from the WebRTC client ​ audio_in_passthrough bool default: "False" When enabled, incoming audio frames are pushed downstream ​ audio_out_enabled bool default: "False" Enable audio output capabilities ​ audio_out_sample_rate int default: "None" Audio output sample rate in Hz ​ audio_out_channels int default: "1" Number of audio output channels ​ Voice Activity Detection (VAD) ​ vad_analyzer VADAnalyzer | None default: "None" Voice Activity Detection analyzer. You can set this to either SileroVADAnalyzer() or WebRTCVADAnalyzer() . SileroVADAnalyzer is the recommended option. Learn more about the SileroVADAnalyzer . ​ Basic Usage Copy Ask AI from fastapi import FastAPI, WebSocket from pipecat.transports.network.fastapi_websocket import ( FastAPIWebsocketTransport, FastAPIWebsocketParams ) from pipecat.audio.vad.silero import SileroVADAnalyzer from pipecat.pipeline.pipeline import Pipeline from pipecat.serializers.twilio import TwilioFrameSerializer app = FastAPI() @app.websocket ( "/ws" ) async def websocket_endpoint ( websocket : WebSocket): await websocket.accept() # Configure transport transport = FastAPIWebsocketTransport( websocket = websocket, params = FastAPIWebsocketParams( audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer(), serializer = TwilioFrameSerializer(), ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), # Handle incoming audio stt, # Speech-to-text llm, # Language model tts, # Text-to-speech transport.output() # Handle outgoing audio ]) # Run pipeline task = PipelineTask(pipeline) await PipelineRunner().run(task) Check out the Twilio Chatbot example to see how to use the FastAPI transport in a phone application. ​ Event Callbacks FastAPIWebsocketTransport provides callbacks for handling client connection events. Register callbacks using the @transport.event_handler() decorator. ​ Connection Events ​ on_client_connected async callback Called when a client connects to the WebSocket endpoint. Parameters: transport : The FastAPIWebsocketTransport instance client : FastAPI WebSocket connection object Copy Ask AI @transport.event_handler ( "on_client_connected" ) async def on_client_connected ( transport , client ): logger.info( "Client connected" ) # Initialize conversation await task.queue_frames([LLMMessagesFrame(initial_messages)]) ​ on_client_disconnected async callback Called when a client disconnects from the WebSocket endpoint. Parameters: transport : The FastAPIWebsocketTransport instance client : FastAPI WebSocket connection object Copy Ask AI @transport.event_handler ( "on_client_disconnected" ) async def on_client_disconnected ( transport , client ): logger.info( "Client disconnected" ) await task.queue_frames([EndFrame()]) ​ on_session_timeout async callback Called when a session times out (if session_timeout is configured). Parameters: transport : The FastAPIWebsocketTransport instance client : FastAPI WebSocket connection object Copy Ask AI @transport.event_handler ( "on_session_timeout" ) async def on_session_timeout ( transport , client ): logger.info( "Session timeout" ) # Handle timeout (e.g., send message, close connection) ​ Frame Types ​ Input Frames ​ InputAudioRawFrame Frame Raw audio data from the WebSocket client ​ Output Frames ​ OutputAudioRawFrame Frame Audio data to be sent to the WebSocket client ​ Notes Integrates with FastAPI web applications Supports real-time audio communication Handles WebSocket connection management Provides voice activity detection Supports session timeouts All callbacks are asynchronous Compatible with various frame serializers SmallWebRTCTransport WebSocket Server On this page Overview Installation Configuration Constructor Parameters FastAPIWebsocketParams Configuration Audio Configuration Voice Activity Detection (VAD) Basic Usage Event Callbacks Connection Events Frame Types Input Frames Output Frames Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/transport_fastapi-websocket_3ca9826c.txt b/transport_fastapi-websocket_3ca9826c.txt
new file mode 100644
index 0000000000000000000000000000000000000000..7f257f8b457d615446f1fc249b7a97dd4f1deb9f
--- /dev/null
+++ b/transport_fastapi-websocket_3ca9826c.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/transport/fastapi-websocket#param-input-audio-raw-frame
+Title: FastAPI WebSocket - Pipecat
+==================================================
+
+FastAPI WebSocket - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Transport FastAPI WebSocket Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Daily WebRTC SmallWebRTCTransport FastAPI WebSocket WebSocket Server Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview FastAPIWebsocketTransport provides WebSocket support for FastAPI web applications, enabling real-time audio communication. It supports bidirectional audio streams and voice activity detection (VAD). FastAPIWebsocketTransport is best suited for server-side applications and prototyping client/server apps. For client/server production applications, we strongly recommend using a WebRTC-based transport for robust network and media handling. ​ Installation To use FastAPIWebsocketTransport , install the required dependencies: Copy Ask AI pip install "pipecat-ai[websocket]" ​ Configuration ​ Constructor Parameters ​ websocket WebSocket required FastAPI WebSocket connection instance ​ params FastAPIWebsocketParams required Transport configuration parameters ​ FastAPIWebsocketParams Configuration ​ add_wav_header bool default: "False" Add WAV header to audio frames ​ serializer FrameSerializer required Frame serializer for WebSocket messages. Common options include: ExotelFrameSerializer - For Exotel Websocket streaming integration PlivoFrameSerializer - For Plivo Websocket streaming integration TelnyxFrameSerializer - For Telnyx WebSocket streaming integration TwilioFrameSerializer - For Twilio Media Streams integration See the Frame Serializers documentation for more details. ​ session_timeout int | None default: "None" Session timeout in seconds. If set, triggers timeout event when no activity is detected ​ Audio Configuration ​ audio_in_enabled bool default: false Enable audio input from the WebRTC client ​ audio_in_passthrough bool default: "False" When enabled, incoming audio frames are pushed downstream ​ audio_out_enabled bool default: "False" Enable audio output capabilities ​ audio_out_sample_rate int default: "None" Audio output sample rate in Hz ​ audio_out_channels int default: "1" Number of audio output channels ​ Voice Activity Detection (VAD) ​ vad_analyzer VADAnalyzer | None default: "None" Voice Activity Detection analyzer. You can set this to either SileroVADAnalyzer() or WebRTCVADAnalyzer() . SileroVADAnalyzer is the recommended option. Learn more about the SileroVADAnalyzer . ​ Basic Usage Copy Ask AI from fastapi import FastAPI, WebSocket from pipecat.transports.network.fastapi_websocket import ( FastAPIWebsocketTransport, FastAPIWebsocketParams ) from pipecat.audio.vad.silero import SileroVADAnalyzer from pipecat.pipeline.pipeline import Pipeline from pipecat.serializers.twilio import TwilioFrameSerializer app = FastAPI() @app.websocket ( "/ws" ) async def websocket_endpoint ( websocket : WebSocket): await websocket.accept() # Configure transport transport = FastAPIWebsocketTransport( websocket = websocket, params = FastAPIWebsocketParams( audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer(), serializer = TwilioFrameSerializer(), ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), # Handle incoming audio stt, # Speech-to-text llm, # Language model tts, # Text-to-speech transport.output() # Handle outgoing audio ]) # Run pipeline task = PipelineTask(pipeline) await PipelineRunner().run(task) Check out the Twilio Chatbot example to see how to use the FastAPI transport in a phone application. ​ Event Callbacks FastAPIWebsocketTransport provides callbacks for handling client connection events. Register callbacks using the @transport.event_handler() decorator. ​ Connection Events ​ on_client_connected async callback Called when a client connects to the WebSocket endpoint. Parameters: transport : The FastAPIWebsocketTransport instance client : FastAPI WebSocket connection object Copy Ask AI @transport.event_handler ( "on_client_connected" ) async def on_client_connected ( transport , client ): logger.info( "Client connected" ) # Initialize conversation await task.queue_frames([LLMMessagesFrame(initial_messages)]) ​ on_client_disconnected async callback Called when a client disconnects from the WebSocket endpoint. Parameters: transport : The FastAPIWebsocketTransport instance client : FastAPI WebSocket connection object Copy Ask AI @transport.event_handler ( "on_client_disconnected" ) async def on_client_disconnected ( transport , client ): logger.info( "Client disconnected" ) await task.queue_frames([EndFrame()]) ​ on_session_timeout async callback Called when a session times out (if session_timeout is configured). Parameters: transport : The FastAPIWebsocketTransport instance client : FastAPI WebSocket connection object Copy Ask AI @transport.event_handler ( "on_session_timeout" ) async def on_session_timeout ( transport , client ): logger.info( "Session timeout" ) # Handle timeout (e.g., send message, close connection) ​ Frame Types ​ Input Frames ​ InputAudioRawFrame Frame Raw audio data from the WebSocket client ​ Output Frames ​ OutputAudioRawFrame Frame Audio data to be sent to the WebSocket client ​ Notes Integrates with FastAPI web applications Supports real-time audio communication Handles WebSocket connection management Provides voice activity detection Supports session timeouts All callbacks are asynchronous Compatible with various frame serializers SmallWebRTCTransport WebSocket Server On this page Overview Installation Configuration Constructor Parameters FastAPIWebsocketParams Configuration Audio Configuration Voice Activity Detection (VAD) Basic Usage Event Callbacks Connection Events Frame Types Input Frames Output Frames Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/transport_fastapi-websocket_7405cea4.txt b/transport_fastapi-websocket_7405cea4.txt
new file mode 100644
index 0000000000000000000000000000000000000000..3d0856f9597e6d669ec649d30a5ddb101ab6d886
--- /dev/null
+++ b/transport_fastapi-websocket_7405cea4.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/transport/fastapi-websocket#constructor-parameters
+Title: FastAPI WebSocket - Pipecat
+==================================================
+
+FastAPI WebSocket - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Transport FastAPI WebSocket Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Daily WebRTC SmallWebRTCTransport FastAPI WebSocket WebSocket Server Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview FastAPIWebsocketTransport provides WebSocket support for FastAPI web applications, enabling real-time audio communication. It supports bidirectional audio streams and voice activity detection (VAD). FastAPIWebsocketTransport is best suited for server-side applications and prototyping client/server apps. For client/server production applications, we strongly recommend using a WebRTC-based transport for robust network and media handling. ​ Installation To use FastAPIWebsocketTransport , install the required dependencies: Copy Ask AI pip install "pipecat-ai[websocket]" ​ Configuration ​ Constructor Parameters ​ websocket WebSocket required FastAPI WebSocket connection instance ​ params FastAPIWebsocketParams required Transport configuration parameters ​ FastAPIWebsocketParams Configuration ​ add_wav_header bool default: "False" Add WAV header to audio frames ​ serializer FrameSerializer required Frame serializer for WebSocket messages. Common options include: ExotelFrameSerializer - For Exotel Websocket streaming integration PlivoFrameSerializer - For Plivo Websocket streaming integration TelnyxFrameSerializer - For Telnyx WebSocket streaming integration TwilioFrameSerializer - For Twilio Media Streams integration See the Frame Serializers documentation for more details. ​ session_timeout int | None default: "None" Session timeout in seconds. If set, triggers timeout event when no activity is detected ​ Audio Configuration ​ audio_in_enabled bool default: false Enable audio input from the WebRTC client ​ audio_in_passthrough bool default: "False" When enabled, incoming audio frames are pushed downstream ​ audio_out_enabled bool default: "False" Enable audio output capabilities ​ audio_out_sample_rate int default: "None" Audio output sample rate in Hz ​ audio_out_channels int default: "1" Number of audio output channels ​ Voice Activity Detection (VAD) ​ vad_analyzer VADAnalyzer | None default: "None" Voice Activity Detection analyzer. You can set this to either SileroVADAnalyzer() or WebRTCVADAnalyzer() . SileroVADAnalyzer is the recommended option. Learn more about the SileroVADAnalyzer . ​ Basic Usage Copy Ask AI from fastapi import FastAPI, WebSocket from pipecat.transports.network.fastapi_websocket import ( FastAPIWebsocketTransport, FastAPIWebsocketParams ) from pipecat.audio.vad.silero import SileroVADAnalyzer from pipecat.pipeline.pipeline import Pipeline from pipecat.serializers.twilio import TwilioFrameSerializer app = FastAPI() @app.websocket ( "/ws" ) async def websocket_endpoint ( websocket : WebSocket): await websocket.accept() # Configure transport transport = FastAPIWebsocketTransport( websocket = websocket, params = FastAPIWebsocketParams( audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer(), serializer = TwilioFrameSerializer(), ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), # Handle incoming audio stt, # Speech-to-text llm, # Language model tts, # Text-to-speech transport.output() # Handle outgoing audio ]) # Run pipeline task = PipelineTask(pipeline) await PipelineRunner().run(task) Check out the Twilio Chatbot example to see how to use the FastAPI transport in a phone application. ​ Event Callbacks FastAPIWebsocketTransport provides callbacks for handling client connection events. Register callbacks using the @transport.event_handler() decorator. ​ Connection Events ​ on_client_connected async callback Called when a client connects to the WebSocket endpoint. Parameters: transport : The FastAPIWebsocketTransport instance client : FastAPI WebSocket connection object Copy Ask AI @transport.event_handler ( "on_client_connected" ) async def on_client_connected ( transport , client ): logger.info( "Client connected" ) # Initialize conversation await task.queue_frames([LLMMessagesFrame(initial_messages)]) ​ on_client_disconnected async callback Called when a client disconnects from the WebSocket endpoint. Parameters: transport : The FastAPIWebsocketTransport instance client : FastAPI WebSocket connection object Copy Ask AI @transport.event_handler ( "on_client_disconnected" ) async def on_client_disconnected ( transport , client ): logger.info( "Client disconnected" ) await task.queue_frames([EndFrame()]) ​ on_session_timeout async callback Called when a session times out (if session_timeout is configured). Parameters: transport : The FastAPIWebsocketTransport instance client : FastAPI WebSocket connection object Copy Ask AI @transport.event_handler ( "on_session_timeout" ) async def on_session_timeout ( transport , client ): logger.info( "Session timeout" ) # Handle timeout (e.g., send message, close connection) ​ Frame Types ​ Input Frames ​ InputAudioRawFrame Frame Raw audio data from the WebSocket client ​ Output Frames ​ OutputAudioRawFrame Frame Audio data to be sent to the WebSocket client ​ Notes Integrates with FastAPI web applications Supports real-time audio communication Handles WebSocket connection management Provides voice activity detection Supports session timeouts All callbacks are asynchronous Compatible with various frame serializers SmallWebRTCTransport WebSocket Server On this page Overview Installation Configuration Constructor Parameters FastAPIWebsocketParams Configuration Audio Configuration Voice Activity Detection (VAD) Basic Usage Event Callbacks Connection Events Frame Types Input Frames Output Frames Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/transport_fastapi-websocket_88a6a3a1.txt b/transport_fastapi-websocket_88a6a3a1.txt
new file mode 100644
index 0000000000000000000000000000000000000000..5288a1a2bf9f8d1092ae275c24598b259e967190
--- /dev/null
+++ b/transport_fastapi-websocket_88a6a3a1.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/transport/fastapi-websocket#overview
+Title: FastAPI WebSocket - Pipecat
+==================================================
+
+FastAPI WebSocket - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Transport FastAPI WebSocket Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Daily WebRTC SmallWebRTCTransport FastAPI WebSocket WebSocket Server Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview FastAPIWebsocketTransport provides WebSocket support for FastAPI web applications, enabling real-time audio communication. It supports bidirectional audio streams and voice activity detection (VAD). FastAPIWebsocketTransport is best suited for server-side applications and prototyping client/server apps. For client/server production applications, we strongly recommend using a WebRTC-based transport for robust network and media handling. ​ Installation To use FastAPIWebsocketTransport , install the required dependencies: Copy Ask AI pip install "pipecat-ai[websocket]" ​ Configuration ​ Constructor Parameters ​ websocket WebSocket required FastAPI WebSocket connection instance ​ params FastAPIWebsocketParams required Transport configuration parameters ​ FastAPIWebsocketParams Configuration ​ add_wav_header bool default: "False" Add WAV header to audio frames ​ serializer FrameSerializer required Frame serializer for WebSocket messages. Common options include: ExotelFrameSerializer - For Exotel Websocket streaming integration PlivoFrameSerializer - For Plivo Websocket streaming integration TelnyxFrameSerializer - For Telnyx WebSocket streaming integration TwilioFrameSerializer - For Twilio Media Streams integration See the Frame Serializers documentation for more details. ​ session_timeout int | None default: "None" Session timeout in seconds. If set, triggers timeout event when no activity is detected ​ Audio Configuration ​ audio_in_enabled bool default: false Enable audio input from the WebRTC client ​ audio_in_passthrough bool default: "False" When enabled, incoming audio frames are pushed downstream ​ audio_out_enabled bool default: "False" Enable audio output capabilities ​ audio_out_sample_rate int default: "None" Audio output sample rate in Hz ​ audio_out_channels int default: "1" Number of audio output channels ​ Voice Activity Detection (VAD) ​ vad_analyzer VADAnalyzer | None default: "None" Voice Activity Detection analyzer. You can set this to either SileroVADAnalyzer() or WebRTCVADAnalyzer() . SileroVADAnalyzer is the recommended option. Learn more about the SileroVADAnalyzer . ​ Basic Usage Copy Ask AI from fastapi import FastAPI, WebSocket from pipecat.transports.network.fastapi_websocket import ( FastAPIWebsocketTransport, FastAPIWebsocketParams ) from pipecat.audio.vad.silero import SileroVADAnalyzer from pipecat.pipeline.pipeline import Pipeline from pipecat.serializers.twilio import TwilioFrameSerializer app = FastAPI() @app.websocket ( "/ws" ) async def websocket_endpoint ( websocket : WebSocket): await websocket.accept() # Configure transport transport = FastAPIWebsocketTransport( websocket = websocket, params = FastAPIWebsocketParams( audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer(), serializer = TwilioFrameSerializer(), ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), # Handle incoming audio stt, # Speech-to-text llm, # Language model tts, # Text-to-speech transport.output() # Handle outgoing audio ]) # Run pipeline task = PipelineTask(pipeline) await PipelineRunner().run(task) Check out the Twilio Chatbot example to see how to use the FastAPI transport in a phone application. ​ Event Callbacks FastAPIWebsocketTransport provides callbacks for handling client connection events. Register callbacks using the @transport.event_handler() decorator. ​ Connection Events ​ on_client_connected async callback Called when a client connects to the WebSocket endpoint. Parameters: transport : The FastAPIWebsocketTransport instance client : FastAPI WebSocket connection object Copy Ask AI @transport.event_handler ( "on_client_connected" ) async def on_client_connected ( transport , client ): logger.info( "Client connected" ) # Initialize conversation await task.queue_frames([LLMMessagesFrame(initial_messages)]) ​ on_client_disconnected async callback Called when a client disconnects from the WebSocket endpoint. Parameters: transport : The FastAPIWebsocketTransport instance client : FastAPI WebSocket connection object Copy Ask AI @transport.event_handler ( "on_client_disconnected" ) async def on_client_disconnected ( transport , client ): logger.info( "Client disconnected" ) await task.queue_frames([EndFrame()]) ​ on_session_timeout async callback Called when a session times out (if session_timeout is configured). Parameters: transport : The FastAPIWebsocketTransport instance client : FastAPI WebSocket connection object Copy Ask AI @transport.event_handler ( "on_session_timeout" ) async def on_session_timeout ( transport , client ): logger.info( "Session timeout" ) # Handle timeout (e.g., send message, close connection) ​ Frame Types ​ Input Frames ​ InputAudioRawFrame Frame Raw audio data from the WebSocket client ​ Output Frames ​ OutputAudioRawFrame Frame Audio data to be sent to the WebSocket client ​ Notes Integrates with FastAPI web applications Supports real-time audio communication Handles WebSocket connection management Provides voice activity detection Supports session timeouts All callbacks are asynchronous Compatible with various frame serializers SmallWebRTCTransport WebSocket Server On this page Overview Installation Configuration Constructor Parameters FastAPIWebsocketParams Configuration Audio Configuration Voice Activity Detection (VAD) Basic Usage Event Callbacks Connection Events Frame Types Input Frames Output Frames Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/transport_fastapi-websocket_a31fe7e8.txt b/transport_fastapi-websocket_a31fe7e8.txt
new file mode 100644
index 0000000000000000000000000000000000000000..b9b048a9c27fbde8a4474914c43cdf7b802c6c41
--- /dev/null
+++ b/transport_fastapi-websocket_a31fe7e8.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/transport/fastapi-websocket#input-frames
+Title: FastAPI WebSocket - Pipecat
+==================================================
+
+FastAPI WebSocket - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Transport FastAPI WebSocket Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Daily WebRTC SmallWebRTCTransport FastAPI WebSocket WebSocket Server Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview FastAPIWebsocketTransport provides WebSocket support for FastAPI web applications, enabling real-time audio communication. It supports bidirectional audio streams and voice activity detection (VAD). FastAPIWebsocketTransport is best suited for server-side applications and prototyping client/server apps. For client/server production applications, we strongly recommend using a WebRTC-based transport for robust network and media handling. ​ Installation To use FastAPIWebsocketTransport , install the required dependencies: Copy Ask AI pip install "pipecat-ai[websocket]" ​ Configuration ​ Constructor Parameters ​ websocket WebSocket required FastAPI WebSocket connection instance ​ params FastAPIWebsocketParams required Transport configuration parameters ​ FastAPIWebsocketParams Configuration ​ add_wav_header bool default: "False" Add WAV header to audio frames ​ serializer FrameSerializer required Frame serializer for WebSocket messages. Common options include: ExotelFrameSerializer - For Exotel Websocket streaming integration PlivoFrameSerializer - For Plivo Websocket streaming integration TelnyxFrameSerializer - For Telnyx WebSocket streaming integration TwilioFrameSerializer - For Twilio Media Streams integration See the Frame Serializers documentation for more details. ​ session_timeout int | None default: "None" Session timeout in seconds. If set, triggers timeout event when no activity is detected ​ Audio Configuration ​ audio_in_enabled bool default: false Enable audio input from the WebRTC client ​ audio_in_passthrough bool default: "False" When enabled, incoming audio frames are pushed downstream ​ audio_out_enabled bool default: "False" Enable audio output capabilities ​ audio_out_sample_rate int default: "None" Audio output sample rate in Hz ​ audio_out_channels int default: "1" Number of audio output channels ​ Voice Activity Detection (VAD) ​ vad_analyzer VADAnalyzer | None default: "None" Voice Activity Detection analyzer. You can set this to either SileroVADAnalyzer() or WebRTCVADAnalyzer() . SileroVADAnalyzer is the recommended option. Learn more about the SileroVADAnalyzer . ​ Basic Usage Copy Ask AI from fastapi import FastAPI, WebSocket from pipecat.transports.network.fastapi_websocket import ( FastAPIWebsocketTransport, FastAPIWebsocketParams ) from pipecat.audio.vad.silero import SileroVADAnalyzer from pipecat.pipeline.pipeline import Pipeline from pipecat.serializers.twilio import TwilioFrameSerializer app = FastAPI() @app.websocket ( "/ws" ) async def websocket_endpoint ( websocket : WebSocket): await websocket.accept() # Configure transport transport = FastAPIWebsocketTransport( websocket = websocket, params = FastAPIWebsocketParams( audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer(), serializer = TwilioFrameSerializer(), ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), # Handle incoming audio stt, # Speech-to-text llm, # Language model tts, # Text-to-speech transport.output() # Handle outgoing audio ]) # Run pipeline task = PipelineTask(pipeline) await PipelineRunner().run(task) Check out the Twilio Chatbot example to see how to use the FastAPI transport in a phone application. ​ Event Callbacks FastAPIWebsocketTransport provides callbacks for handling client connection events. Register callbacks using the @transport.event_handler() decorator. ​ Connection Events ​ on_client_connected async callback Called when a client connects to the WebSocket endpoint. Parameters: transport : The FastAPIWebsocketTransport instance client : FastAPI WebSocket connection object Copy Ask AI @transport.event_handler ( "on_client_connected" ) async def on_client_connected ( transport , client ): logger.info( "Client connected" ) # Initialize conversation await task.queue_frames([LLMMessagesFrame(initial_messages)]) ​ on_client_disconnected async callback Called when a client disconnects from the WebSocket endpoint. Parameters: transport : The FastAPIWebsocketTransport instance client : FastAPI WebSocket connection object Copy Ask AI @transport.event_handler ( "on_client_disconnected" ) async def on_client_disconnected ( transport , client ): logger.info( "Client disconnected" ) await task.queue_frames([EndFrame()]) ​ on_session_timeout async callback Called when a session times out (if session_timeout is configured). Parameters: transport : The FastAPIWebsocketTransport instance client : FastAPI WebSocket connection object Copy Ask AI @transport.event_handler ( "on_session_timeout" ) async def on_session_timeout ( transport , client ): logger.info( "Session timeout" ) # Handle timeout (e.g., send message, close connection) ​ Frame Types ​ Input Frames ​ InputAudioRawFrame Frame Raw audio data from the WebSocket client ​ Output Frames ​ OutputAudioRawFrame Frame Audio data to be sent to the WebSocket client ​ Notes Integrates with FastAPI web applications Supports real-time audio communication Handles WebSocket connection management Provides voice activity detection Supports session timeouts All callbacks are asynchronous Compatible with various frame serializers SmallWebRTCTransport WebSocket Server On this page Overview Installation Configuration Constructor Parameters FastAPIWebsocketParams Configuration Audio Configuration Voice Activity Detection (VAD) Basic Usage Event Callbacks Connection Events Frame Types Input Frames Output Frames Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/transport_fastapi-websocket_aa72842a.txt b/transport_fastapi-websocket_aa72842a.txt
new file mode 100644
index 0000000000000000000000000000000000000000..93dc1902f4f65580e2813c868087562c7896da24
--- /dev/null
+++ b/transport_fastapi-websocket_aa72842a.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/transport/fastapi-websocket#param-session-timeout
+Title: FastAPI WebSocket - Pipecat
+==================================================
+
+FastAPI WebSocket - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Transport FastAPI WebSocket Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Daily WebRTC SmallWebRTCTransport FastAPI WebSocket WebSocket Server Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview FastAPIWebsocketTransport provides WebSocket support for FastAPI web applications, enabling real-time audio communication. It supports bidirectional audio streams and voice activity detection (VAD). FastAPIWebsocketTransport is best suited for server-side applications and prototyping client/server apps. For client/server production applications, we strongly recommend using a WebRTC-based transport for robust network and media handling. ​ Installation To use FastAPIWebsocketTransport , install the required dependencies: Copy Ask AI pip install "pipecat-ai[websocket]" ​ Configuration ​ Constructor Parameters ​ websocket WebSocket required FastAPI WebSocket connection instance ​ params FastAPIWebsocketParams required Transport configuration parameters ​ FastAPIWebsocketParams Configuration ​ add_wav_header bool default: "False" Add WAV header to audio frames ​ serializer FrameSerializer required Frame serializer for WebSocket messages. Common options include: ExotelFrameSerializer - For Exotel Websocket streaming integration PlivoFrameSerializer - For Plivo Websocket streaming integration TelnyxFrameSerializer - For Telnyx WebSocket streaming integration TwilioFrameSerializer - For Twilio Media Streams integration See the Frame Serializers documentation for more details. ​ session_timeout int | None default: "None" Session timeout in seconds. If set, triggers timeout event when no activity is detected ​ Audio Configuration ​ audio_in_enabled bool default: false Enable audio input from the WebRTC client ​ audio_in_passthrough bool default: "False" When enabled, incoming audio frames are pushed downstream ​ audio_out_enabled bool default: "False" Enable audio output capabilities ​ audio_out_sample_rate int default: "None" Audio output sample rate in Hz ​ audio_out_channels int default: "1" Number of audio output channels ​ Voice Activity Detection (VAD) ​ vad_analyzer VADAnalyzer | None default: "None" Voice Activity Detection analyzer. You can set this to either SileroVADAnalyzer() or WebRTCVADAnalyzer() . SileroVADAnalyzer is the recommended option. Learn more about the SileroVADAnalyzer . ​ Basic Usage Copy Ask AI from fastapi import FastAPI, WebSocket from pipecat.transports.network.fastapi_websocket import ( FastAPIWebsocketTransport, FastAPIWebsocketParams ) from pipecat.audio.vad.silero import SileroVADAnalyzer from pipecat.pipeline.pipeline import Pipeline from pipecat.serializers.twilio import TwilioFrameSerializer app = FastAPI() @app.websocket ( "/ws" ) async def websocket_endpoint ( websocket : WebSocket): await websocket.accept() # Configure transport transport = FastAPIWebsocketTransport( websocket = websocket, params = FastAPIWebsocketParams( audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer(), serializer = TwilioFrameSerializer(), ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), # Handle incoming audio stt, # Speech-to-text llm, # Language model tts, # Text-to-speech transport.output() # Handle outgoing audio ]) # Run pipeline task = PipelineTask(pipeline) await PipelineRunner().run(task) Check out the Twilio Chatbot example to see how to use the FastAPI transport in a phone application. ​ Event Callbacks FastAPIWebsocketTransport provides callbacks for handling client connection events. Register callbacks using the @transport.event_handler() decorator. ​ Connection Events ​ on_client_connected async callback Called when a client connects to the WebSocket endpoint. Parameters: transport : The FastAPIWebsocketTransport instance client : FastAPI WebSocket connection object Copy Ask AI @transport.event_handler ( "on_client_connected" ) async def on_client_connected ( transport , client ): logger.info( "Client connected" ) # Initialize conversation await task.queue_frames([LLMMessagesFrame(initial_messages)]) ​ on_client_disconnected async callback Called when a client disconnects from the WebSocket endpoint. Parameters: transport : The FastAPIWebsocketTransport instance client : FastAPI WebSocket connection object Copy Ask AI @transport.event_handler ( "on_client_disconnected" ) async def on_client_disconnected ( transport , client ): logger.info( "Client disconnected" ) await task.queue_frames([EndFrame()]) ​ on_session_timeout async callback Called when a session times out (if session_timeout is configured). Parameters: transport : The FastAPIWebsocketTransport instance client : FastAPI WebSocket connection object Copy Ask AI @transport.event_handler ( "on_session_timeout" ) async def on_session_timeout ( transport , client ): logger.info( "Session timeout" ) # Handle timeout (e.g., send message, close connection) ​ Frame Types ​ Input Frames ​ InputAudioRawFrame Frame Raw audio data from the WebSocket client ​ Output Frames ​ OutputAudioRawFrame Frame Audio data to be sent to the WebSocket client ​ Notes Integrates with FastAPI web applications Supports real-time audio communication Handles WebSocket connection management Provides voice activity detection Supports session timeouts All callbacks are asynchronous Compatible with various frame serializers SmallWebRTCTransport WebSocket Server On this page Overview Installation Configuration Constructor Parameters FastAPIWebsocketParams Configuration Audio Configuration Voice Activity Detection (VAD) Basic Usage Event Callbacks Connection Events Frame Types Input Frames Output Frames Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/transport_fastapi-websocket_baed7cf4.txt b/transport_fastapi-websocket_baed7cf4.txt
new file mode 100644
index 0000000000000000000000000000000000000000..fd1b8233246cbbee22cc215c7b696920648d69e8
--- /dev/null
+++ b/transport_fastapi-websocket_baed7cf4.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/transport/fastapi-websocket#voice-activity-detection-vad
+Title: FastAPI WebSocket - Pipecat
+==================================================
+
+FastAPI WebSocket - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Transport FastAPI WebSocket Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Daily WebRTC SmallWebRTCTransport FastAPI WebSocket WebSocket Server Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview FastAPIWebsocketTransport provides WebSocket support for FastAPI web applications, enabling real-time audio communication. It supports bidirectional audio streams and voice activity detection (VAD). FastAPIWebsocketTransport is best suited for server-side applications and prototyping client/server apps. For client/server production applications, we strongly recommend using a WebRTC-based transport for robust network and media handling. ​ Installation To use FastAPIWebsocketTransport , install the required dependencies: Copy Ask AI pip install "pipecat-ai[websocket]" ​ Configuration ​ Constructor Parameters ​ websocket WebSocket required FastAPI WebSocket connection instance ​ params FastAPIWebsocketParams required Transport configuration parameters ​ FastAPIWebsocketParams Configuration ​ add_wav_header bool default: "False" Add WAV header to audio frames ​ serializer FrameSerializer required Frame serializer for WebSocket messages. Common options include: ExotelFrameSerializer - For Exotel Websocket streaming integration PlivoFrameSerializer - For Plivo Websocket streaming integration TelnyxFrameSerializer - For Telnyx WebSocket streaming integration TwilioFrameSerializer - For Twilio Media Streams integration See the Frame Serializers documentation for more details. ​ session_timeout int | None default: "None" Session timeout in seconds. If set, triggers timeout event when no activity is detected ​ Audio Configuration ​ audio_in_enabled bool default: false Enable audio input from the WebRTC client ​ audio_in_passthrough bool default: "False" When enabled, incoming audio frames are pushed downstream ​ audio_out_enabled bool default: "False" Enable audio output capabilities ​ audio_out_sample_rate int default: "None" Audio output sample rate in Hz ​ audio_out_channels int default: "1" Number of audio output channels ​ Voice Activity Detection (VAD) ​ vad_analyzer VADAnalyzer | None default: "None" Voice Activity Detection analyzer. You can set this to either SileroVADAnalyzer() or WebRTCVADAnalyzer() . SileroVADAnalyzer is the recommended option. Learn more about the SileroVADAnalyzer . ​ Basic Usage Copy Ask AI from fastapi import FastAPI, WebSocket from pipecat.transports.network.fastapi_websocket import ( FastAPIWebsocketTransport, FastAPIWebsocketParams ) from pipecat.audio.vad.silero import SileroVADAnalyzer from pipecat.pipeline.pipeline import Pipeline from pipecat.serializers.twilio import TwilioFrameSerializer app = FastAPI() @app.websocket ( "/ws" ) async def websocket_endpoint ( websocket : WebSocket): await websocket.accept() # Configure transport transport = FastAPIWebsocketTransport( websocket = websocket, params = FastAPIWebsocketParams( audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer(), serializer = TwilioFrameSerializer(), ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), # Handle incoming audio stt, # Speech-to-text llm, # Language model tts, # Text-to-speech transport.output() # Handle outgoing audio ]) # Run pipeline task = PipelineTask(pipeline) await PipelineRunner().run(task) Check out the Twilio Chatbot example to see how to use the FastAPI transport in a phone application. ​ Event Callbacks FastAPIWebsocketTransport provides callbacks for handling client connection events. Register callbacks using the @transport.event_handler() decorator. ​ Connection Events ​ on_client_connected async callback Called when a client connects to the WebSocket endpoint. Parameters: transport : The FastAPIWebsocketTransport instance client : FastAPI WebSocket connection object Copy Ask AI @transport.event_handler ( "on_client_connected" ) async def on_client_connected ( transport , client ): logger.info( "Client connected" ) # Initialize conversation await task.queue_frames([LLMMessagesFrame(initial_messages)]) ​ on_client_disconnected async callback Called when a client disconnects from the WebSocket endpoint. Parameters: transport : The FastAPIWebsocketTransport instance client : FastAPI WebSocket connection object Copy Ask AI @transport.event_handler ( "on_client_disconnected" ) async def on_client_disconnected ( transport , client ): logger.info( "Client disconnected" ) await task.queue_frames([EndFrame()]) ​ on_session_timeout async callback Called when a session times out (if session_timeout is configured). Parameters: transport : The FastAPIWebsocketTransport instance client : FastAPI WebSocket connection object Copy Ask AI @transport.event_handler ( "on_session_timeout" ) async def on_session_timeout ( transport , client ): logger.info( "Session timeout" ) # Handle timeout (e.g., send message, close connection) ​ Frame Types ​ Input Frames ​ InputAudioRawFrame Frame Raw audio data from the WebSocket client ​ Output Frames ​ OutputAudioRawFrame Frame Audio data to be sent to the WebSocket client ​ Notes Integrates with FastAPI web applications Supports real-time audio communication Handles WebSocket connection management Provides voice activity detection Supports session timeouts All callbacks are asynchronous Compatible with various frame serializers SmallWebRTCTransport WebSocket Server On this page Overview Installation Configuration Constructor Parameters FastAPIWebsocketParams Configuration Audio Configuration Voice Activity Detection (VAD) Basic Usage Event Callbacks Connection Events Frame Types Input Frames Output Frames Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/transport_fastapi-websocket_d3a249fb.txt b/transport_fastapi-websocket_d3a249fb.txt
new file mode 100644
index 0000000000000000000000000000000000000000..774eaf1d5e0ae70f58531092ccaa5a6f948e501b
--- /dev/null
+++ b/transport_fastapi-websocket_d3a249fb.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/transport/fastapi-websocket
+Title: FastAPI WebSocket - Pipecat
+==================================================
+
+FastAPI WebSocket - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Transport FastAPI WebSocket Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Daily WebRTC SmallWebRTCTransport FastAPI WebSocket WebSocket Server Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview FastAPIWebsocketTransport provides WebSocket support for FastAPI web applications, enabling real-time audio communication. It supports bidirectional audio streams and voice activity detection (VAD). FastAPIWebsocketTransport is best suited for server-side applications and prototyping client/server apps. For client/server production applications, we strongly recommend using a WebRTC-based transport for robust network and media handling. ​ Installation To use FastAPIWebsocketTransport , install the required dependencies: Copy Ask AI pip install "pipecat-ai[websocket]" ​ Configuration ​ Constructor Parameters ​ websocket WebSocket required FastAPI WebSocket connection instance ​ params FastAPIWebsocketParams required Transport configuration parameters ​ FastAPIWebsocketParams Configuration ​ add_wav_header bool default: "False" Add WAV header to audio frames ​ serializer FrameSerializer required Frame serializer for WebSocket messages. Common options include: ExotelFrameSerializer - For Exotel Websocket streaming integration PlivoFrameSerializer - For Plivo Websocket streaming integration TelnyxFrameSerializer - For Telnyx WebSocket streaming integration TwilioFrameSerializer - For Twilio Media Streams integration See the Frame Serializers documentation for more details. ​ session_timeout int | None default: "None" Session timeout in seconds. If set, triggers timeout event when no activity is detected ​ Audio Configuration ​ audio_in_enabled bool default: false Enable audio input from the WebRTC client ​ audio_in_passthrough bool default: "False" When enabled, incoming audio frames are pushed downstream ​ audio_out_enabled bool default: "False" Enable audio output capabilities ​ audio_out_sample_rate int default: "None" Audio output sample rate in Hz ​ audio_out_channels int default: "1" Number of audio output channels ​ Voice Activity Detection (VAD) ​ vad_analyzer VADAnalyzer | None default: "None" Voice Activity Detection analyzer. You can set this to either SileroVADAnalyzer() or WebRTCVADAnalyzer() . SileroVADAnalyzer is the recommended option. Learn more about the SileroVADAnalyzer . ​ Basic Usage Copy Ask AI from fastapi import FastAPI, WebSocket from pipecat.transports.network.fastapi_websocket import ( FastAPIWebsocketTransport, FastAPIWebsocketParams ) from pipecat.audio.vad.silero import SileroVADAnalyzer from pipecat.pipeline.pipeline import Pipeline from pipecat.serializers.twilio import TwilioFrameSerializer app = FastAPI() @app.websocket ( "/ws" ) async def websocket_endpoint ( websocket : WebSocket): await websocket.accept() # Configure transport transport = FastAPIWebsocketTransport( websocket = websocket, params = FastAPIWebsocketParams( audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer(), serializer = TwilioFrameSerializer(), ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), # Handle incoming audio stt, # Speech-to-text llm, # Language model tts, # Text-to-speech transport.output() # Handle outgoing audio ]) # Run pipeline task = PipelineTask(pipeline) await PipelineRunner().run(task) Check out the Twilio Chatbot example to see how to use the FastAPI transport in a phone application. ​ Event Callbacks FastAPIWebsocketTransport provides callbacks for handling client connection events. Register callbacks using the @transport.event_handler() decorator. ​ Connection Events ​ on_client_connected async callback Called when a client connects to the WebSocket endpoint. Parameters: transport : The FastAPIWebsocketTransport instance client : FastAPI WebSocket connection object Copy Ask AI @transport.event_handler ( "on_client_connected" ) async def on_client_connected ( transport , client ): logger.info( "Client connected" ) # Initialize conversation await task.queue_frames([LLMMessagesFrame(initial_messages)]) ​ on_client_disconnected async callback Called when a client disconnects from the WebSocket endpoint. Parameters: transport : The FastAPIWebsocketTransport instance client : FastAPI WebSocket connection object Copy Ask AI @transport.event_handler ( "on_client_disconnected" ) async def on_client_disconnected ( transport , client ): logger.info( "Client disconnected" ) await task.queue_frames([EndFrame()]) ​ on_session_timeout async callback Called when a session times out (if session_timeout is configured). Parameters: transport : The FastAPIWebsocketTransport instance client : FastAPI WebSocket connection object Copy Ask AI @transport.event_handler ( "on_session_timeout" ) async def on_session_timeout ( transport , client ): logger.info( "Session timeout" ) # Handle timeout (e.g., send message, close connection) ​ Frame Types ​ Input Frames ​ InputAudioRawFrame Frame Raw audio data from the WebSocket client ​ Output Frames ​ OutputAudioRawFrame Frame Audio data to be sent to the WebSocket client ​ Notes Integrates with FastAPI web applications Supports real-time audio communication Handles WebSocket connection management Provides voice activity detection Supports session timeouts All callbacks are asynchronous Compatible with various frame serializers SmallWebRTCTransport WebSocket Server On this page Overview Installation Configuration Constructor Parameters FastAPIWebsocketParams Configuration Audio Configuration Voice Activity Detection (VAD) Basic Usage Event Callbacks Connection Events Frame Types Input Frames Output Frames Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/transport_fastapi-websocket_dbc900fa.txt b/transport_fastapi-websocket_dbc900fa.txt
new file mode 100644
index 0000000000000000000000000000000000000000..c9f572d8093e4ad07cede733b546f68e3bc235ad
--- /dev/null
+++ b/transport_fastapi-websocket_dbc900fa.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/transport/fastapi-websocket#notes
+Title: FastAPI WebSocket - Pipecat
+==================================================
+
+FastAPI WebSocket - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Transport FastAPI WebSocket Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Daily WebRTC SmallWebRTCTransport FastAPI WebSocket WebSocket Server Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview FastAPIWebsocketTransport provides WebSocket support for FastAPI web applications, enabling real-time audio communication. It supports bidirectional audio streams and voice activity detection (VAD). FastAPIWebsocketTransport is best suited for server-side applications and prototyping client/server apps. For client/server production applications, we strongly recommend using a WebRTC-based transport for robust network and media handling. ​ Installation To use FastAPIWebsocketTransport , install the required dependencies: Copy Ask AI pip install "pipecat-ai[websocket]" ​ Configuration ​ Constructor Parameters ​ websocket WebSocket required FastAPI WebSocket connection instance ​ params FastAPIWebsocketParams required Transport configuration parameters ​ FastAPIWebsocketParams Configuration ​ add_wav_header bool default: "False" Add WAV header to audio frames ​ serializer FrameSerializer required Frame serializer for WebSocket messages. Common options include: ExotelFrameSerializer - For Exotel Websocket streaming integration PlivoFrameSerializer - For Plivo Websocket streaming integration TelnyxFrameSerializer - For Telnyx WebSocket streaming integration TwilioFrameSerializer - For Twilio Media Streams integration See the Frame Serializers documentation for more details. ​ session_timeout int | None default: "None" Session timeout in seconds. If set, triggers timeout event when no activity is detected ​ Audio Configuration ​ audio_in_enabled bool default: false Enable audio input from the WebRTC client ​ audio_in_passthrough bool default: "False" When enabled, incoming audio frames are pushed downstream ​ audio_out_enabled bool default: "False" Enable audio output capabilities ​ audio_out_sample_rate int default: "None" Audio output sample rate in Hz ​ audio_out_channels int default: "1" Number of audio output channels ​ Voice Activity Detection (VAD) ​ vad_analyzer VADAnalyzer | None default: "None" Voice Activity Detection analyzer. You can set this to either SileroVADAnalyzer() or WebRTCVADAnalyzer() . SileroVADAnalyzer is the recommended option. Learn more about the SileroVADAnalyzer . ​ Basic Usage Copy Ask AI from fastapi import FastAPI, WebSocket from pipecat.transports.network.fastapi_websocket import ( FastAPIWebsocketTransport, FastAPIWebsocketParams ) from pipecat.audio.vad.silero import SileroVADAnalyzer from pipecat.pipeline.pipeline import Pipeline from pipecat.serializers.twilio import TwilioFrameSerializer app = FastAPI() @app.websocket ( "/ws" ) async def websocket_endpoint ( websocket : WebSocket): await websocket.accept() # Configure transport transport = FastAPIWebsocketTransport( websocket = websocket, params = FastAPIWebsocketParams( audio_in_enabled = True , audio_out_enabled = True , vad_analyzer = SileroVADAnalyzer(), serializer = TwilioFrameSerializer(), ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), # Handle incoming audio stt, # Speech-to-text llm, # Language model tts, # Text-to-speech transport.output() # Handle outgoing audio ]) # Run pipeline task = PipelineTask(pipeline) await PipelineRunner().run(task) Check out the Twilio Chatbot example to see how to use the FastAPI transport in a phone application. ​ Event Callbacks FastAPIWebsocketTransport provides callbacks for handling client connection events. Register callbacks using the @transport.event_handler() decorator. ​ Connection Events ​ on_client_connected async callback Called when a client connects to the WebSocket endpoint. Parameters: transport : The FastAPIWebsocketTransport instance client : FastAPI WebSocket connection object Copy Ask AI @transport.event_handler ( "on_client_connected" ) async def on_client_connected ( transport , client ): logger.info( "Client connected" ) # Initialize conversation await task.queue_frames([LLMMessagesFrame(initial_messages)]) ​ on_client_disconnected async callback Called when a client disconnects from the WebSocket endpoint. Parameters: transport : The FastAPIWebsocketTransport instance client : FastAPI WebSocket connection object Copy Ask AI @transport.event_handler ( "on_client_disconnected" ) async def on_client_disconnected ( transport , client ): logger.info( "Client disconnected" ) await task.queue_frames([EndFrame()]) ​ on_session_timeout async callback Called when a session times out (if session_timeout is configured). Parameters: transport : The FastAPIWebsocketTransport instance client : FastAPI WebSocket connection object Copy Ask AI @transport.event_handler ( "on_session_timeout" ) async def on_session_timeout ( transport , client ): logger.info( "Session timeout" ) # Handle timeout (e.g., send message, close connection) ​ Frame Types ​ Input Frames ​ InputAudioRawFrame Frame Raw audio data from the WebSocket client ​ Output Frames ​ OutputAudioRawFrame Frame Audio data to be sent to the WebSocket client ​ Notes Integrates with FastAPI web applications Supports real-time audio communication Handles WebSocket connection management Provides voice activity detection Supports session timeouts All callbacks are asynchronous Compatible with various frame serializers SmallWebRTCTransport WebSocket Server On this page Overview Installation Configuration Constructor Parameters FastAPIWebsocketParams Configuration Audio Configuration Voice Activity Detection (VAD) Basic Usage Event Callbacks Connection Events Frame Types Input Frames Output Frames Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/transport_small-webrtc_00b99fc7.txt b/transport_small-webrtc_00b99fc7.txt
new file mode 100644
index 0000000000000000000000000000000000000000..81738d37caf3b3e4034a9b20f869cf1d943c0950
--- /dev/null
+++ b/transport_small-webrtc_00b99fc7.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/transport/small-webrtc
+Title: SmallWebRTCTransport - Pipecat
+==================================================
+
+SmallWebRTCTransport - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Transport SmallWebRTCTransport Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Daily WebRTC SmallWebRTCTransport FastAPI WebSocket WebSocket Server Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview SmallWebRTCTransport enables peer-to-peer WebRTC connections between clients and your Pipecat application. It implements bidirectional audio and video streaming using WebRTC for real-time communication. This transport is intended for lightweight implementations, particularly for local development and testing. It expects your clients to include a corresponding SmallWebRTCTransport implementation. See here for the JavaScript implementation. SmallWebRTCTransport is best used for testing and development. For production deployments with scale, consider using the DailyTransport , as it has global, low-latency infrastructure. ​ Installation To use SmallWebRTCTransport , install the required dependencies: Copy Ask AI pip install pipecat-ai[webrtc] ​ Class Reference ​ SmallWebRTCConnection SmallWebRTCConnection manages the WebRTC connection details, peer connection state, and ICE candidates. It handles the signaling process and media tracks. Copy Ask AI SmallWebRTCConnection( ice_servers = None ) ​ ice_servers Union[List[str], List[IceServer]] List of STUN/TURN server URLs for ICE connection establishment. Can be provided as strings or as IceServer objects. ​ Methods ​ initialize async method Initialize the connection with a client’s SDP offer. Parameters: sdp : String containing the Session Description Protocol data from client’s offer type : String representing the SDP message type (typically “offer”) Copy Ask AI await webrtc_connection.initialize( sdp = client_sdp, type = "offer" ) ​ connect async method Establish the WebRTC peer connection after initialization. Copy Ask AI await webrtc_connection.connect() ​ close async method Close the WebRTC peer connection. Copy Ask AI await webrtc_connection.close() ​ disconnect async method Disconnect the WebRTC peer connection and send a peer left message to the client. Copy Ask AI await webrtc_connection.disconnect() ​ renegotiate async method Handle connection renegotiation requests. Parameters: sdp : String containing the Session Description Protocol data for renegotiation type : String representing the SDP message type restart_pc : Boolean indicating whether to completely restart the peer connection (default: False) Copy Ask AI await webrtc_connection.renegotiate( sdp = new_sdp, type = "offer" , restart_pc = False ) ​ get_answer method Retrieve the SDP answer to send back to the client. Returns a dictionary with sdp , type , and pc_id fields. Copy Ask AI answer = webrtc_connection.get_answer() # Returns: {"sdp": "...", "type": "answer", "pc_id": "..."} ​ send_app_message method Send an application message to the client. Parameters: message : The message to send (will be JSON serialized) Copy Ask AI webrtc_connection.send_app_message({ "action" : "greeting" , "text" : "Hello!" }) ​ is_connected method Check if the connection is active. Copy Ask AI if webrtc_connection.is_connected(): print ( "Connection is active" ) ​ audio_input_track method Get the audio input track from the client. Copy Ask AI audio_track = webrtc_connection.audio_input_track() ​ video_input_track method Get the video input track from the client. Copy Ask AI video_track = webrtc_connection.video_input_track() ​ replace_audio_track method Replace the current audio track with a new one. Parameters: track : The new audio track to use Copy Ask AI webrtc_connection.replace_audio_track(new_audio_track) ​ replace_video_track method Replace the current video track with a new one. Parameters: track : The new video track to use Copy Ask AI webrtc_connection.replace_video_track(new_video_track) ​ ask_to_renegotiate method Request the client to initiate connection renegotiation. Copy Ask AI webrtc_connection.ask_to_renegotiate() ​ event_handler decorator Register an event handler for connection events. Events: "app-message" : Called when a message is received from the client "track-started" : Called when a new track is started "track-ended" : Called when a track ends "connecting" : Called when connection is being established "connected" : Called when connection is established "disconnected" : Called when connection is lost "closed" : Called when connection is closed "failed" : Called when connection fails "new" : Called when a new connection is created Copy Ask AI @webrtc_connection.event_handler ( "connected" ) async def on_connected ( connection ): print ( f "WebRTC connection established" ) ​ SmallWebRTCTransport SmallWebRTCTransport is the main transport class that manages both input and output transports for WebRTC communication. Copy Ask AI SmallWebRTCTransport( webrtc_connection: SmallWebRTCConnection, params: TransportParams, input_name: Optional[ str ] = None , output_name: Optional[ str ] = None ) ​ webrtc_connection SmallWebRTCConnection required An instance of SmallWebRTCConnection that manages the WebRTC connection ​ params TransportParams required Configuration parameters for the transport Show TransportParams properties ​ audio_in_enabled bool default: false Enable audio input from the WebRTC client ​ audio_in_passthrough bool default: "False" When enabled, incoming audio frames are pushed downstream ​ audio_out_enabled bool default: false Enable audio output to the WebRTC client ​ audio_in_sample_rate int Sample rate for incoming audio (Hz) ​ audio_out_sample_rate int Sample rate for outgoing audio (Hz) ​ audio_in_channels int default: 1 Number of audio input channels (1 for mono, 2 for stereo) ​ audio_out_channels int default: 1 Number of audio output channels (1 for mono, 2 for stereo) ​ video_in_enabled bool default: false Enable video input from the WebRTC client ​ video_out_enabled bool default: false Enable video output to the WebRTC client ​ video_out_width int default: 640 Width of outgoing video ​ video_out_height int default: 480 Height of outgoing video ​ video_out_framerate int default: 30 Framerate of outgoing video ​ vad_analyzer VADAnalyzer Custom VAD analyzer implementation ​ input_name str Optional name for the input transport ​ output_name str Optional name for the output transport ​ Methods ​ input method Returns the input transport instance. Copy Ask AI input_transport = webrtc_transport.input() ​ output method Returns the output transport instance. Copy Ask AI output_transport = webrtc_transport.output() ​ send_image async method Send an image frame to the client. Parameters: frame : The image frame to send (OutputImageRawFrame or SpriteFrame) Copy Ask AI await webrtc_transport.send_image(image_frame) ​ send_audio async method Send an audio frame to the client. Parameters: frame : The audio frame to send (OutputAudioRawFrame) Copy Ask AI await webrtc_transport.send_audio(audio_frame) ​ Event Handlers ​ on_app_message async callback Called when receiving application messages from the client. Parameters: message : The received message Copy Ask AI @webrtc_transport.event_handler ( "on_app_message" ) async def on_app_message ( message ): print ( f "Received message: { message } " ) ​ on_client_connected async callback Called when a client successfully connects. Parameters: transport : The SmallWebRTCTransport instance webrtc_connection : The connection that was established Copy Ask AI @webrtc_transport.event_handler ( "on_client_connected" ) async def on_client_connected ( transport , webrtc_connection ): print ( f "Client connected" ) ​ on_client_disconnected async callback Called when a client disconnects. Parameters: transport : The SmallWebRTCTransport instance webrtc_connection : The connection that was disconnected Copy Ask AI @webrtc_transport.event_handler ( "on_client_disconnected" ) async def on_client_disconnected ( transport , webrtc_connection ): print ( f "Client disconnected" ) ​ on_client_closed async callback Called when a client connection is closed. Parameters: transport : The SmallWebRTCTransport instance webrtc_connection : The connection that was closed Copy Ask AI @webrtc_transport.event_handler ( "on_client_closed" ) async def on_client_closed ( transport , webrtc_connection ): print ( f "Client connection closed" ) ​ Basic Usage This basic usage example shows the transport specific parts of a bot.py file required to configure your bot: Copy Ask AI from pipecat.audio.vad.silero import SileroVADAnalyzer from pipecat.pipeline.pipeline import Pipeline from pipecat.transports.base_transport import TransportParams from pipecat.transports.network.small_webrtc import SmallWebRTCTransport from pipecat.transports.network.webrtc_connection import SmallWebRTCConnection async def run_bot ( webrtc_connection ): # Create the WebRTC transport with the provided connection transport = SmallWebRTCTransport( webrtc_connection = webrtc_connection, params = TransportParams( audio_in_enabled = True , # Accept audio from the client audio_out_enabled = True , # Send audio to the client vad_analyzer = SileroVADAnalyzer(), ), ) # Set up your services and context # Create the pipeline pipeline = Pipeline([ transport.input(), # Receive audio from client stt, # Convert speech to text context_aggregator.user(), # Add user messages to context llm, # Process text with LLM tts, # Convert text to speech transport.output(), # Send audio responses to client context_aggregator.assistant(), # Add assistant responses to context ]) # Register event handlers @transport.event_handler ( "on_client_connected" ) async def on_client_connected ( transport , client ): logger.info( "Client connected" ) # Start the conversation when client connects await task.queue_frames([context_aggregator.user().get_context_frame()]) @transport.event_handler ( "on_client_disconnected" ) async def on_client_disconnected ( transport , client ): logger.info( "Client disconnected" ) @transport.event_handler ( "on_client_closed" ) async def on_client_closed ( transport , client ): logger.info( "Client closed" ) await task.cancel() ​ How to connect with SmallWebRTCTransport For a client/server connection, you have two options for how to connect the client to the server: Use a Pipecat client SDK with the SmallWebRTCTransport . See the Client SDK docs to get started. Using the WebRTC API directly. This is only recommended for advanced use cases where the Pipecat client SDKs don’t have an available transport. ​ Examples To see a complete implementation, check out the following examples: Video Transform Demonstrates real-time video processing using WebRTC transport Voice Agent Implements a voice assistant using WebRTC for audio communication ​ Media Handling ​ Audio Audio is processed in 20ms chunks by default. The transport handles audio format conversion and resampling as needed: Input audio is processed at 16kHz (mono) to be compatible with speech recognition services Output audio can be configured to match your application’s requirements, but it must be mono, 16-bit PCM audio ​ Video Video is streamed using RGB format by default. The transport provides: Frame conversion between different color formats (RGB, YUV, etc.) Configurable resolution and framerate ​ WebRTC ICE Servers Configuration When implementing WebRTC in your project, STUN (Session Traversal Utilities for NAT) and TURN (Traversal Using Relays around NAT) servers are usually needed in cases where users are behind routers or firewalls. In local networks (e.g., testing within the same home or office network), you usually don’t need to configure STUN or TURN servers. In such cases, WebRTC can often directly establish peer-to-peer connections without needing to traverse NAT or firewalls. ​ What are STUN and TURN Servers? STUN Server : Helps clients discover their public IP address and port when they’re behind a NAT (Network Address Translation) device (like a router). This allows WebRTC to attempt direct peer-to-peer communication by providing the public-facing IP and port. TURN Server : Used as a fallback when direct peer-to-peer communication isn’t possible due to strict NATs or firewalls blocking connections. The TURN server relays media traffic between peers. ​ Why are ICE Servers Important? ICE (Interactive Connectivity Establishment) is a framework used by WebRTC to handle network traversal and NAT issues. The iceServers configuration provides a list of STUN and TURN servers that WebRTC uses to find the best way to connect two peers. ​ Advanced Configuration ​ ICE Servers For better connectivity, especially when testing across different networks, you can provide STUN servers: Copy Ask AI webrtc_connection = SmallWebRTCConnection( ice_servers = [ "stun:stun.l.google.com:19302" , "stun:stun1.l.google.com:19302" ] ) You can also use IceServer objects for more advanced configuration: Copy Ask AI from pipecat.transports.network.webrtc_connection import IceServer webrtc_connection = SmallWebRTCConnection( ice_servers = [ IceServer( urls = "stun:stun.l.google.com:19302" ), IceServer( urls = "turn:turn.example.com:3478" , username = "username" , credential = "password" ) ] ) ​ Troubleshooting If clients have trouble connecting or streaming: Check browser console for WebRTC errors Ensure you’re using HTTPS in production (required for WebRTC) For testing across networks, consider using Daily which provides TURN servers Verify browser permissions for camera and microphone Daily WebRTC FastAPI WebSocket On this page Overview Installation Class Reference SmallWebRTCConnection Methods SmallWebRTCTransport Methods Event Handlers Basic Usage How to connect with SmallWebRTCTransport Examples Media Handling Audio Video WebRTC ICE Servers Configuration What are STUN and TURN Servers? Why are ICE Servers Important? Advanced Configuration ICE Servers Troubleshooting Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/transport_small-webrtc_12452732.txt b/transport_small-webrtc_12452732.txt
new file mode 100644
index 0000000000000000000000000000000000000000..f0555e5d62c0a831d400219ebfdbba2cc04782e2
--- /dev/null
+++ b/transport_small-webrtc_12452732.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/transport/small-webrtc#param-video-out-width
+Title: SmallWebRTCTransport - Pipecat
+==================================================
+
+SmallWebRTCTransport - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Transport SmallWebRTCTransport Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Daily WebRTC SmallWebRTCTransport FastAPI WebSocket WebSocket Server Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview SmallWebRTCTransport enables peer-to-peer WebRTC connections between clients and your Pipecat application. It implements bidirectional audio and video streaming using WebRTC for real-time communication. This transport is intended for lightweight implementations, particularly for local development and testing. It expects your clients to include a corresponding SmallWebRTCTransport implementation. See here for the JavaScript implementation. SmallWebRTCTransport is best used for testing and development. For production deployments with scale, consider using the DailyTransport , as it has global, low-latency infrastructure. ​ Installation To use SmallWebRTCTransport , install the required dependencies: Copy Ask AI pip install pipecat-ai[webrtc] ​ Class Reference ​ SmallWebRTCConnection SmallWebRTCConnection manages the WebRTC connection details, peer connection state, and ICE candidates. It handles the signaling process and media tracks. Copy Ask AI SmallWebRTCConnection( ice_servers = None ) ​ ice_servers Union[List[str], List[IceServer]] List of STUN/TURN server URLs for ICE connection establishment. Can be provided as strings or as IceServer objects. ​ Methods ​ initialize async method Initialize the connection with a client’s SDP offer. Parameters: sdp : String containing the Session Description Protocol data from client’s offer type : String representing the SDP message type (typically “offer”) Copy Ask AI await webrtc_connection.initialize( sdp = client_sdp, type = "offer" ) ​ connect async method Establish the WebRTC peer connection after initialization. Copy Ask AI await webrtc_connection.connect() ​ close async method Close the WebRTC peer connection. Copy Ask AI await webrtc_connection.close() ​ disconnect async method Disconnect the WebRTC peer connection and send a peer left message to the client. Copy Ask AI await webrtc_connection.disconnect() ​ renegotiate async method Handle connection renegotiation requests. Parameters: sdp : String containing the Session Description Protocol data for renegotiation type : String representing the SDP message type restart_pc : Boolean indicating whether to completely restart the peer connection (default: False) Copy Ask AI await webrtc_connection.renegotiate( sdp = new_sdp, type = "offer" , restart_pc = False ) ​ get_answer method Retrieve the SDP answer to send back to the client. Returns a dictionary with sdp , type , and pc_id fields. Copy Ask AI answer = webrtc_connection.get_answer() # Returns: {"sdp": "...", "type": "answer", "pc_id": "..."} ​ send_app_message method Send an application message to the client. Parameters: message : The message to send (will be JSON serialized) Copy Ask AI webrtc_connection.send_app_message({ "action" : "greeting" , "text" : "Hello!" }) ​ is_connected method Check if the connection is active. Copy Ask AI if webrtc_connection.is_connected(): print ( "Connection is active" ) ​ audio_input_track method Get the audio input track from the client. Copy Ask AI audio_track = webrtc_connection.audio_input_track() ​ video_input_track method Get the video input track from the client. Copy Ask AI video_track = webrtc_connection.video_input_track() ​ replace_audio_track method Replace the current audio track with a new one. Parameters: track : The new audio track to use Copy Ask AI webrtc_connection.replace_audio_track(new_audio_track) ​ replace_video_track method Replace the current video track with a new one. Parameters: track : The new video track to use Copy Ask AI webrtc_connection.replace_video_track(new_video_track) ​ ask_to_renegotiate method Request the client to initiate connection renegotiation. Copy Ask AI webrtc_connection.ask_to_renegotiate() ​ event_handler decorator Register an event handler for connection events. Events: "app-message" : Called when a message is received from the client "track-started" : Called when a new track is started "track-ended" : Called when a track ends "connecting" : Called when connection is being established "connected" : Called when connection is established "disconnected" : Called when connection is lost "closed" : Called when connection is closed "failed" : Called when connection fails "new" : Called when a new connection is created Copy Ask AI @webrtc_connection.event_handler ( "connected" ) async def on_connected ( connection ): print ( f "WebRTC connection established" ) ​ SmallWebRTCTransport SmallWebRTCTransport is the main transport class that manages both input and output transports for WebRTC communication. Copy Ask AI SmallWebRTCTransport( webrtc_connection: SmallWebRTCConnection, params: TransportParams, input_name: Optional[ str ] = None , output_name: Optional[ str ] = None ) ​ webrtc_connection SmallWebRTCConnection required An instance of SmallWebRTCConnection that manages the WebRTC connection ​ params TransportParams required Configuration parameters for the transport Show TransportParams properties ​ audio_in_enabled bool default: false Enable audio input from the WebRTC client ​ audio_in_passthrough bool default: "False" When enabled, incoming audio frames are pushed downstream ​ audio_out_enabled bool default: false Enable audio output to the WebRTC client ​ audio_in_sample_rate int Sample rate for incoming audio (Hz) ​ audio_out_sample_rate int Sample rate for outgoing audio (Hz) ​ audio_in_channels int default: 1 Number of audio input channels (1 for mono, 2 for stereo) ​ audio_out_channels int default: 1 Number of audio output channels (1 for mono, 2 for stereo) ​ video_in_enabled bool default: false Enable video input from the WebRTC client ​ video_out_enabled bool default: false Enable video output to the WebRTC client ​ video_out_width int default: 640 Width of outgoing video ​ video_out_height int default: 480 Height of outgoing video ​ video_out_framerate int default: 30 Framerate of outgoing video ​ vad_analyzer VADAnalyzer Custom VAD analyzer implementation ​ input_name str Optional name for the input transport ​ output_name str Optional name for the output transport ​ Methods ​ input method Returns the input transport instance. Copy Ask AI input_transport = webrtc_transport.input() ​ output method Returns the output transport instance. Copy Ask AI output_transport = webrtc_transport.output() ​ send_image async method Send an image frame to the client. Parameters: frame : The image frame to send (OutputImageRawFrame or SpriteFrame) Copy Ask AI await webrtc_transport.send_image(image_frame) ​ send_audio async method Send an audio frame to the client. Parameters: frame : The audio frame to send (OutputAudioRawFrame) Copy Ask AI await webrtc_transport.send_audio(audio_frame) ​ Event Handlers ​ on_app_message async callback Called when receiving application messages from the client. Parameters: message : The received message Copy Ask AI @webrtc_transport.event_handler ( "on_app_message" ) async def on_app_message ( message ): print ( f "Received message: { message } " ) ​ on_client_connected async callback Called when a client successfully connects. Parameters: transport : The SmallWebRTCTransport instance webrtc_connection : The connection that was established Copy Ask AI @webrtc_transport.event_handler ( "on_client_connected" ) async def on_client_connected ( transport , webrtc_connection ): print ( f "Client connected" ) ​ on_client_disconnected async callback Called when a client disconnects. Parameters: transport : The SmallWebRTCTransport instance webrtc_connection : The connection that was disconnected Copy Ask AI @webrtc_transport.event_handler ( "on_client_disconnected" ) async def on_client_disconnected ( transport , webrtc_connection ): print ( f "Client disconnected" ) ​ on_client_closed async callback Called when a client connection is closed. Parameters: transport : The SmallWebRTCTransport instance webrtc_connection : The connection that was closed Copy Ask AI @webrtc_transport.event_handler ( "on_client_closed" ) async def on_client_closed ( transport , webrtc_connection ): print ( f "Client connection closed" ) ​ Basic Usage This basic usage example shows the transport specific parts of a bot.py file required to configure your bot: Copy Ask AI from pipecat.audio.vad.silero import SileroVADAnalyzer from pipecat.pipeline.pipeline import Pipeline from pipecat.transports.base_transport import TransportParams from pipecat.transports.network.small_webrtc import SmallWebRTCTransport from pipecat.transports.network.webrtc_connection import SmallWebRTCConnection async def run_bot ( webrtc_connection ): # Create the WebRTC transport with the provided connection transport = SmallWebRTCTransport( webrtc_connection = webrtc_connection, params = TransportParams( audio_in_enabled = True , # Accept audio from the client audio_out_enabled = True , # Send audio to the client vad_analyzer = SileroVADAnalyzer(), ), ) # Set up your services and context # Create the pipeline pipeline = Pipeline([ transport.input(), # Receive audio from client stt, # Convert speech to text context_aggregator.user(), # Add user messages to context llm, # Process text with LLM tts, # Convert text to speech transport.output(), # Send audio responses to client context_aggregator.assistant(), # Add assistant responses to context ]) # Register event handlers @transport.event_handler ( "on_client_connected" ) async def on_client_connected ( transport , client ): logger.info( "Client connected" ) # Start the conversation when client connects await task.queue_frames([context_aggregator.user().get_context_frame()]) @transport.event_handler ( "on_client_disconnected" ) async def on_client_disconnected ( transport , client ): logger.info( "Client disconnected" ) @transport.event_handler ( "on_client_closed" ) async def on_client_closed ( transport , client ): logger.info( "Client closed" ) await task.cancel() ​ How to connect with SmallWebRTCTransport For a client/server connection, you have two options for how to connect the client to the server: Use a Pipecat client SDK with the SmallWebRTCTransport . See the Client SDK docs to get started. Using the WebRTC API directly. This is only recommended for advanced use cases where the Pipecat client SDKs don’t have an available transport. ​ Examples To see a complete implementation, check out the following examples: Video Transform Demonstrates real-time video processing using WebRTC transport Voice Agent Implements a voice assistant using WebRTC for audio communication ​ Media Handling ​ Audio Audio is processed in 20ms chunks by default. The transport handles audio format conversion and resampling as needed: Input audio is processed at 16kHz (mono) to be compatible with speech recognition services Output audio can be configured to match your application’s requirements, but it must be mono, 16-bit PCM audio ​ Video Video is streamed using RGB format by default. The transport provides: Frame conversion between different color formats (RGB, YUV, etc.) Configurable resolution and framerate ​ WebRTC ICE Servers Configuration When implementing WebRTC in your project, STUN (Session Traversal Utilities for NAT) and TURN (Traversal Using Relays around NAT) servers are usually needed in cases where users are behind routers or firewalls. In local networks (e.g., testing within the same home or office network), you usually don’t need to configure STUN or TURN servers. In such cases, WebRTC can often directly establish peer-to-peer connections without needing to traverse NAT or firewalls. ​ What are STUN and TURN Servers? STUN Server : Helps clients discover their public IP address and port when they’re behind a NAT (Network Address Translation) device (like a router). This allows WebRTC to attempt direct peer-to-peer communication by providing the public-facing IP and port. TURN Server : Used as a fallback when direct peer-to-peer communication isn’t possible due to strict NATs or firewalls blocking connections. The TURN server relays media traffic between peers. ​ Why are ICE Servers Important? ICE (Interactive Connectivity Establishment) is a framework used by WebRTC to handle network traversal and NAT issues. The iceServers configuration provides a list of STUN and TURN servers that WebRTC uses to find the best way to connect two peers. ​ Advanced Configuration ​ ICE Servers For better connectivity, especially when testing across different networks, you can provide STUN servers: Copy Ask AI webrtc_connection = SmallWebRTCConnection( ice_servers = [ "stun:stun.l.google.com:19302" , "stun:stun1.l.google.com:19302" ] ) You can also use IceServer objects for more advanced configuration: Copy Ask AI from pipecat.transports.network.webrtc_connection import IceServer webrtc_connection = SmallWebRTCConnection( ice_servers = [ IceServer( urls = "stun:stun.l.google.com:19302" ), IceServer( urls = "turn:turn.example.com:3478" , username = "username" , credential = "password" ) ] ) ​ Troubleshooting If clients have trouble connecting or streaming: Check browser console for WebRTC errors Ensure you’re using HTTPS in production (required for WebRTC) For testing across networks, consider using Daily which provides TURN servers Verify browser permissions for camera and microphone Daily WebRTC FastAPI WebSocket On this page Overview Installation Class Reference SmallWebRTCConnection Methods SmallWebRTCTransport Methods Event Handlers Basic Usage How to connect with SmallWebRTCTransport Examples Media Handling Audio Video WebRTC ICE Servers Configuration What are STUN and TURN Servers? Why are ICE Servers Important? Advanced Configuration ICE Servers Troubleshooting Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/transport_small-webrtc_249d1a2f.txt b/transport_small-webrtc_249d1a2f.txt
new file mode 100644
index 0000000000000000000000000000000000000000..70abe728e7b60a666627b1b88569791864964d84
--- /dev/null
+++ b/transport_small-webrtc_249d1a2f.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/transport/small-webrtc#param-get-answer
+Title: SmallWebRTCTransport - Pipecat
+==================================================
+
+SmallWebRTCTransport - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Transport SmallWebRTCTransport Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Daily WebRTC SmallWebRTCTransport FastAPI WebSocket WebSocket Server Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview SmallWebRTCTransport enables peer-to-peer WebRTC connections between clients and your Pipecat application. It implements bidirectional audio and video streaming using WebRTC for real-time communication. This transport is intended for lightweight implementations, particularly for local development and testing. It expects your clients to include a corresponding SmallWebRTCTransport implementation. See here for the JavaScript implementation. SmallWebRTCTransport is best used for testing and development. For production deployments with scale, consider using the DailyTransport , as it has global, low-latency infrastructure. ​ Installation To use SmallWebRTCTransport , install the required dependencies: Copy Ask AI pip install pipecat-ai[webrtc] ​ Class Reference ​ SmallWebRTCConnection SmallWebRTCConnection manages the WebRTC connection details, peer connection state, and ICE candidates. It handles the signaling process and media tracks. Copy Ask AI SmallWebRTCConnection( ice_servers = None ) ​ ice_servers Union[List[str], List[IceServer]] List of STUN/TURN server URLs for ICE connection establishment. Can be provided as strings or as IceServer objects. ​ Methods ​ initialize async method Initialize the connection with a client’s SDP offer. Parameters: sdp : String containing the Session Description Protocol data from client’s offer type : String representing the SDP message type (typically “offer”) Copy Ask AI await webrtc_connection.initialize( sdp = client_sdp, type = "offer" ) ​ connect async method Establish the WebRTC peer connection after initialization. Copy Ask AI await webrtc_connection.connect() ​ close async method Close the WebRTC peer connection. Copy Ask AI await webrtc_connection.close() ​ disconnect async method Disconnect the WebRTC peer connection and send a peer left message to the client. Copy Ask AI await webrtc_connection.disconnect() ​ renegotiate async method Handle connection renegotiation requests. Parameters: sdp : String containing the Session Description Protocol data for renegotiation type : String representing the SDP message type restart_pc : Boolean indicating whether to completely restart the peer connection (default: False) Copy Ask AI await webrtc_connection.renegotiate( sdp = new_sdp, type = "offer" , restart_pc = False ) ​ get_answer method Retrieve the SDP answer to send back to the client. Returns a dictionary with sdp , type , and pc_id fields. Copy Ask AI answer = webrtc_connection.get_answer() # Returns: {"sdp": "...", "type": "answer", "pc_id": "..."} ​ send_app_message method Send an application message to the client. Parameters: message : The message to send (will be JSON serialized) Copy Ask AI webrtc_connection.send_app_message({ "action" : "greeting" , "text" : "Hello!" }) ​ is_connected method Check if the connection is active. Copy Ask AI if webrtc_connection.is_connected(): print ( "Connection is active" ) ​ audio_input_track method Get the audio input track from the client. Copy Ask AI audio_track = webrtc_connection.audio_input_track() ​ video_input_track method Get the video input track from the client. Copy Ask AI video_track = webrtc_connection.video_input_track() ​ replace_audio_track method Replace the current audio track with a new one. Parameters: track : The new audio track to use Copy Ask AI webrtc_connection.replace_audio_track(new_audio_track) ​ replace_video_track method Replace the current video track with a new one. Parameters: track : The new video track to use Copy Ask AI webrtc_connection.replace_video_track(new_video_track) ​ ask_to_renegotiate method Request the client to initiate connection renegotiation. Copy Ask AI webrtc_connection.ask_to_renegotiate() ​ event_handler decorator Register an event handler for connection events. Events: "app-message" : Called when a message is received from the client "track-started" : Called when a new track is started "track-ended" : Called when a track ends "connecting" : Called when connection is being established "connected" : Called when connection is established "disconnected" : Called when connection is lost "closed" : Called when connection is closed "failed" : Called when connection fails "new" : Called when a new connection is created Copy Ask AI @webrtc_connection.event_handler ( "connected" ) async def on_connected ( connection ): print ( f "WebRTC connection established" ) ​ SmallWebRTCTransport SmallWebRTCTransport is the main transport class that manages both input and output transports for WebRTC communication. Copy Ask AI SmallWebRTCTransport( webrtc_connection: SmallWebRTCConnection, params: TransportParams, input_name: Optional[ str ] = None , output_name: Optional[ str ] = None ) ​ webrtc_connection SmallWebRTCConnection required An instance of SmallWebRTCConnection that manages the WebRTC connection ​ params TransportParams required Configuration parameters for the transport Show TransportParams properties ​ audio_in_enabled bool default: false Enable audio input from the WebRTC client ​ audio_in_passthrough bool default: "False" When enabled, incoming audio frames are pushed downstream ​ audio_out_enabled bool default: false Enable audio output to the WebRTC client ​ audio_in_sample_rate int Sample rate for incoming audio (Hz) ​ audio_out_sample_rate int Sample rate for outgoing audio (Hz) ​ audio_in_channels int default: 1 Number of audio input channels (1 for mono, 2 for stereo) ​ audio_out_channels int default: 1 Number of audio output channels (1 for mono, 2 for stereo) ​ video_in_enabled bool default: false Enable video input from the WebRTC client ​ video_out_enabled bool default: false Enable video output to the WebRTC client ​ video_out_width int default: 640 Width of outgoing video ​ video_out_height int default: 480 Height of outgoing video ​ video_out_framerate int default: 30 Framerate of outgoing video ​ vad_analyzer VADAnalyzer Custom VAD analyzer implementation ​ input_name str Optional name for the input transport ​ output_name str Optional name for the output transport ​ Methods ​ input method Returns the input transport instance. Copy Ask AI input_transport = webrtc_transport.input() ​ output method Returns the output transport instance. Copy Ask AI output_transport = webrtc_transport.output() ​ send_image async method Send an image frame to the client. Parameters: frame : The image frame to send (OutputImageRawFrame or SpriteFrame) Copy Ask AI await webrtc_transport.send_image(image_frame) ​ send_audio async method Send an audio frame to the client. Parameters: frame : The audio frame to send (OutputAudioRawFrame) Copy Ask AI await webrtc_transport.send_audio(audio_frame) ​ Event Handlers ​ on_app_message async callback Called when receiving application messages from the client. Parameters: message : The received message Copy Ask AI @webrtc_transport.event_handler ( "on_app_message" ) async def on_app_message ( message ): print ( f "Received message: { message } " ) ​ on_client_connected async callback Called when a client successfully connects. Parameters: transport : The SmallWebRTCTransport instance webrtc_connection : The connection that was established Copy Ask AI @webrtc_transport.event_handler ( "on_client_connected" ) async def on_client_connected ( transport , webrtc_connection ): print ( f "Client connected" ) ​ on_client_disconnected async callback Called when a client disconnects. Parameters: transport : The SmallWebRTCTransport instance webrtc_connection : The connection that was disconnected Copy Ask AI @webrtc_transport.event_handler ( "on_client_disconnected" ) async def on_client_disconnected ( transport , webrtc_connection ): print ( f "Client disconnected" ) ​ on_client_closed async callback Called when a client connection is closed. Parameters: transport : The SmallWebRTCTransport instance webrtc_connection : The connection that was closed Copy Ask AI @webrtc_transport.event_handler ( "on_client_closed" ) async def on_client_closed ( transport , webrtc_connection ): print ( f "Client connection closed" ) ​ Basic Usage This basic usage example shows the transport specific parts of a bot.py file required to configure your bot: Copy Ask AI from pipecat.audio.vad.silero import SileroVADAnalyzer from pipecat.pipeline.pipeline import Pipeline from pipecat.transports.base_transport import TransportParams from pipecat.transports.network.small_webrtc import SmallWebRTCTransport from pipecat.transports.network.webrtc_connection import SmallWebRTCConnection async def run_bot ( webrtc_connection ): # Create the WebRTC transport with the provided connection transport = SmallWebRTCTransport( webrtc_connection = webrtc_connection, params = TransportParams( audio_in_enabled = True , # Accept audio from the client audio_out_enabled = True , # Send audio to the client vad_analyzer = SileroVADAnalyzer(), ), ) # Set up your services and context # Create the pipeline pipeline = Pipeline([ transport.input(), # Receive audio from client stt, # Convert speech to text context_aggregator.user(), # Add user messages to context llm, # Process text with LLM tts, # Convert text to speech transport.output(), # Send audio responses to client context_aggregator.assistant(), # Add assistant responses to context ]) # Register event handlers @transport.event_handler ( "on_client_connected" ) async def on_client_connected ( transport , client ): logger.info( "Client connected" ) # Start the conversation when client connects await task.queue_frames([context_aggregator.user().get_context_frame()]) @transport.event_handler ( "on_client_disconnected" ) async def on_client_disconnected ( transport , client ): logger.info( "Client disconnected" ) @transport.event_handler ( "on_client_closed" ) async def on_client_closed ( transport , client ): logger.info( "Client closed" ) await task.cancel() ​ How to connect with SmallWebRTCTransport For a client/server connection, you have two options for how to connect the client to the server: Use a Pipecat client SDK with the SmallWebRTCTransport . See the Client SDK docs to get started. Using the WebRTC API directly. This is only recommended for advanced use cases where the Pipecat client SDKs don’t have an available transport. ​ Examples To see a complete implementation, check out the following examples: Video Transform Demonstrates real-time video processing using WebRTC transport Voice Agent Implements a voice assistant using WebRTC for audio communication ​ Media Handling ​ Audio Audio is processed in 20ms chunks by default. The transport handles audio format conversion and resampling as needed: Input audio is processed at 16kHz (mono) to be compatible with speech recognition services Output audio can be configured to match your application’s requirements, but it must be mono, 16-bit PCM audio ​ Video Video is streamed using RGB format by default. The transport provides: Frame conversion between different color formats (RGB, YUV, etc.) Configurable resolution and framerate ​ WebRTC ICE Servers Configuration When implementing WebRTC in your project, STUN (Session Traversal Utilities for NAT) and TURN (Traversal Using Relays around NAT) servers are usually needed in cases where users are behind routers or firewalls. In local networks (e.g., testing within the same home or office network), you usually don’t need to configure STUN or TURN servers. In such cases, WebRTC can often directly establish peer-to-peer connections without needing to traverse NAT or firewalls. ​ What are STUN and TURN Servers? STUN Server : Helps clients discover their public IP address and port when they’re behind a NAT (Network Address Translation) device (like a router). This allows WebRTC to attempt direct peer-to-peer communication by providing the public-facing IP and port. TURN Server : Used as a fallback when direct peer-to-peer communication isn’t possible due to strict NATs or firewalls blocking connections. The TURN server relays media traffic between peers. ​ Why are ICE Servers Important? ICE (Interactive Connectivity Establishment) is a framework used by WebRTC to handle network traversal and NAT issues. The iceServers configuration provides a list of STUN and TURN servers that WebRTC uses to find the best way to connect two peers. ​ Advanced Configuration ​ ICE Servers For better connectivity, especially when testing across different networks, you can provide STUN servers: Copy Ask AI webrtc_connection = SmallWebRTCConnection( ice_servers = [ "stun:stun.l.google.com:19302" , "stun:stun1.l.google.com:19302" ] ) You can also use IceServer objects for more advanced configuration: Copy Ask AI from pipecat.transports.network.webrtc_connection import IceServer webrtc_connection = SmallWebRTCConnection( ice_servers = [ IceServer( urls = "stun:stun.l.google.com:19302" ), IceServer( urls = "turn:turn.example.com:3478" , username = "username" , credential = "password" ) ] ) ​ Troubleshooting If clients have trouble connecting or streaming: Check browser console for WebRTC errors Ensure you’re using HTTPS in production (required for WebRTC) For testing across networks, consider using Daily which provides TURN servers Verify browser permissions for camera and microphone Daily WebRTC FastAPI WebSocket On this page Overview Installation Class Reference SmallWebRTCConnection Methods SmallWebRTCTransport Methods Event Handlers Basic Usage How to connect with SmallWebRTCTransport Examples Media Handling Audio Video WebRTC ICE Servers Configuration What are STUN and TURN Servers? Why are ICE Servers Important? Advanced Configuration ICE Servers Troubleshooting Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/transport_small-webrtc_4210e98a.txt b/transport_small-webrtc_4210e98a.txt
new file mode 100644
index 0000000000000000000000000000000000000000..65e56160a51a4baa823b4fd8621677d72e2f067e
--- /dev/null
+++ b/transport_small-webrtc_4210e98a.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/transport/small-webrtc#param-initialize
+Title: SmallWebRTCTransport - Pipecat
+==================================================
+
+SmallWebRTCTransport - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Transport SmallWebRTCTransport Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Daily WebRTC SmallWebRTCTransport FastAPI WebSocket WebSocket Server Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview SmallWebRTCTransport enables peer-to-peer WebRTC connections between clients and your Pipecat application. It implements bidirectional audio and video streaming using WebRTC for real-time communication. This transport is intended for lightweight implementations, particularly for local development and testing. It expects your clients to include a corresponding SmallWebRTCTransport implementation. See here for the JavaScript implementation. SmallWebRTCTransport is best used for testing and development. For production deployments with scale, consider using the DailyTransport , as it has global, low-latency infrastructure. ​ Installation To use SmallWebRTCTransport , install the required dependencies: Copy Ask AI pip install pipecat-ai[webrtc] ​ Class Reference ​ SmallWebRTCConnection SmallWebRTCConnection manages the WebRTC connection details, peer connection state, and ICE candidates. It handles the signaling process and media tracks. Copy Ask AI SmallWebRTCConnection( ice_servers = None ) ​ ice_servers Union[List[str], List[IceServer]] List of STUN/TURN server URLs for ICE connection establishment. Can be provided as strings or as IceServer objects. ​ Methods ​ initialize async method Initialize the connection with a client’s SDP offer. Parameters: sdp : String containing the Session Description Protocol data from client’s offer type : String representing the SDP message type (typically “offer”) Copy Ask AI await webrtc_connection.initialize( sdp = client_sdp, type = "offer" ) ​ connect async method Establish the WebRTC peer connection after initialization. Copy Ask AI await webrtc_connection.connect() ​ close async method Close the WebRTC peer connection. Copy Ask AI await webrtc_connection.close() ​ disconnect async method Disconnect the WebRTC peer connection and send a peer left message to the client. Copy Ask AI await webrtc_connection.disconnect() ​ renegotiate async method Handle connection renegotiation requests. Parameters: sdp : String containing the Session Description Protocol data for renegotiation type : String representing the SDP message type restart_pc : Boolean indicating whether to completely restart the peer connection (default: False) Copy Ask AI await webrtc_connection.renegotiate( sdp = new_sdp, type = "offer" , restart_pc = False ) ​ get_answer method Retrieve the SDP answer to send back to the client. Returns a dictionary with sdp , type , and pc_id fields. Copy Ask AI answer = webrtc_connection.get_answer() # Returns: {"sdp": "...", "type": "answer", "pc_id": "..."} ​ send_app_message method Send an application message to the client. Parameters: message : The message to send (will be JSON serialized) Copy Ask AI webrtc_connection.send_app_message({ "action" : "greeting" , "text" : "Hello!" }) ​ is_connected method Check if the connection is active. Copy Ask AI if webrtc_connection.is_connected(): print ( "Connection is active" ) ​ audio_input_track method Get the audio input track from the client. Copy Ask AI audio_track = webrtc_connection.audio_input_track() ​ video_input_track method Get the video input track from the client. Copy Ask AI video_track = webrtc_connection.video_input_track() ​ replace_audio_track method Replace the current audio track with a new one. Parameters: track : The new audio track to use Copy Ask AI webrtc_connection.replace_audio_track(new_audio_track) ​ replace_video_track method Replace the current video track with a new one. Parameters: track : The new video track to use Copy Ask AI webrtc_connection.replace_video_track(new_video_track) ​ ask_to_renegotiate method Request the client to initiate connection renegotiation. Copy Ask AI webrtc_connection.ask_to_renegotiate() ​ event_handler decorator Register an event handler for connection events. Events: "app-message" : Called when a message is received from the client "track-started" : Called when a new track is started "track-ended" : Called when a track ends "connecting" : Called when connection is being established "connected" : Called when connection is established "disconnected" : Called when connection is lost "closed" : Called when connection is closed "failed" : Called when connection fails "new" : Called when a new connection is created Copy Ask AI @webrtc_connection.event_handler ( "connected" ) async def on_connected ( connection ): print ( f "WebRTC connection established" ) ​ SmallWebRTCTransport SmallWebRTCTransport is the main transport class that manages both input and output transports for WebRTC communication. Copy Ask AI SmallWebRTCTransport( webrtc_connection: SmallWebRTCConnection, params: TransportParams, input_name: Optional[ str ] = None , output_name: Optional[ str ] = None ) ​ webrtc_connection SmallWebRTCConnection required An instance of SmallWebRTCConnection that manages the WebRTC connection ​ params TransportParams required Configuration parameters for the transport Show TransportParams properties ​ audio_in_enabled bool default: false Enable audio input from the WebRTC client ​ audio_in_passthrough bool default: "False" When enabled, incoming audio frames are pushed downstream ​ audio_out_enabled bool default: false Enable audio output to the WebRTC client ​ audio_in_sample_rate int Sample rate for incoming audio (Hz) ​ audio_out_sample_rate int Sample rate for outgoing audio (Hz) ​ audio_in_channels int default: 1 Number of audio input channels (1 for mono, 2 for stereo) ​ audio_out_channels int default: 1 Number of audio output channels (1 for mono, 2 for stereo) ​ video_in_enabled bool default: false Enable video input from the WebRTC client ​ video_out_enabled bool default: false Enable video output to the WebRTC client ​ video_out_width int default: 640 Width of outgoing video ​ video_out_height int default: 480 Height of outgoing video ​ video_out_framerate int default: 30 Framerate of outgoing video ​ vad_analyzer VADAnalyzer Custom VAD analyzer implementation ​ input_name str Optional name for the input transport ​ output_name str Optional name for the output transport ​ Methods ​ input method Returns the input transport instance. Copy Ask AI input_transport = webrtc_transport.input() ​ output method Returns the output transport instance. Copy Ask AI output_transport = webrtc_transport.output() ​ send_image async method Send an image frame to the client. Parameters: frame : The image frame to send (OutputImageRawFrame or SpriteFrame) Copy Ask AI await webrtc_transport.send_image(image_frame) ​ send_audio async method Send an audio frame to the client. Parameters: frame : The audio frame to send (OutputAudioRawFrame) Copy Ask AI await webrtc_transport.send_audio(audio_frame) ​ Event Handlers ​ on_app_message async callback Called when receiving application messages from the client. Parameters: message : The received message Copy Ask AI @webrtc_transport.event_handler ( "on_app_message" ) async def on_app_message ( message ): print ( f "Received message: { message } " ) ​ on_client_connected async callback Called when a client successfully connects. Parameters: transport : The SmallWebRTCTransport instance webrtc_connection : The connection that was established Copy Ask AI @webrtc_transport.event_handler ( "on_client_connected" ) async def on_client_connected ( transport , webrtc_connection ): print ( f "Client connected" ) ​ on_client_disconnected async callback Called when a client disconnects. Parameters: transport : The SmallWebRTCTransport instance webrtc_connection : The connection that was disconnected Copy Ask AI @webrtc_transport.event_handler ( "on_client_disconnected" ) async def on_client_disconnected ( transport , webrtc_connection ): print ( f "Client disconnected" ) ​ on_client_closed async callback Called when a client connection is closed. Parameters: transport : The SmallWebRTCTransport instance webrtc_connection : The connection that was closed Copy Ask AI @webrtc_transport.event_handler ( "on_client_closed" ) async def on_client_closed ( transport , webrtc_connection ): print ( f "Client connection closed" ) ​ Basic Usage This basic usage example shows the transport specific parts of a bot.py file required to configure your bot: Copy Ask AI from pipecat.audio.vad.silero import SileroVADAnalyzer from pipecat.pipeline.pipeline import Pipeline from pipecat.transports.base_transport import TransportParams from pipecat.transports.network.small_webrtc import SmallWebRTCTransport from pipecat.transports.network.webrtc_connection import SmallWebRTCConnection async def run_bot ( webrtc_connection ): # Create the WebRTC transport with the provided connection transport = SmallWebRTCTransport( webrtc_connection = webrtc_connection, params = TransportParams( audio_in_enabled = True , # Accept audio from the client audio_out_enabled = True , # Send audio to the client vad_analyzer = SileroVADAnalyzer(), ), ) # Set up your services and context # Create the pipeline pipeline = Pipeline([ transport.input(), # Receive audio from client stt, # Convert speech to text context_aggregator.user(), # Add user messages to context llm, # Process text with LLM tts, # Convert text to speech transport.output(), # Send audio responses to client context_aggregator.assistant(), # Add assistant responses to context ]) # Register event handlers @transport.event_handler ( "on_client_connected" ) async def on_client_connected ( transport , client ): logger.info( "Client connected" ) # Start the conversation when client connects await task.queue_frames([context_aggregator.user().get_context_frame()]) @transport.event_handler ( "on_client_disconnected" ) async def on_client_disconnected ( transport , client ): logger.info( "Client disconnected" ) @transport.event_handler ( "on_client_closed" ) async def on_client_closed ( transport , client ): logger.info( "Client closed" ) await task.cancel() ​ How to connect with SmallWebRTCTransport For a client/server connection, you have two options for how to connect the client to the server: Use a Pipecat client SDK with the SmallWebRTCTransport . See the Client SDK docs to get started. Using the WebRTC API directly. This is only recommended for advanced use cases where the Pipecat client SDKs don’t have an available transport. ​ Examples To see a complete implementation, check out the following examples: Video Transform Demonstrates real-time video processing using WebRTC transport Voice Agent Implements a voice assistant using WebRTC for audio communication ​ Media Handling ​ Audio Audio is processed in 20ms chunks by default. The transport handles audio format conversion and resampling as needed: Input audio is processed at 16kHz (mono) to be compatible with speech recognition services Output audio can be configured to match your application’s requirements, but it must be mono, 16-bit PCM audio ​ Video Video is streamed using RGB format by default. The transport provides: Frame conversion between different color formats (RGB, YUV, etc.) Configurable resolution and framerate ​ WebRTC ICE Servers Configuration When implementing WebRTC in your project, STUN (Session Traversal Utilities for NAT) and TURN (Traversal Using Relays around NAT) servers are usually needed in cases where users are behind routers or firewalls. In local networks (e.g., testing within the same home or office network), you usually don’t need to configure STUN or TURN servers. In such cases, WebRTC can often directly establish peer-to-peer connections without needing to traverse NAT or firewalls. ​ What are STUN and TURN Servers? STUN Server : Helps clients discover their public IP address and port when they’re behind a NAT (Network Address Translation) device (like a router). This allows WebRTC to attempt direct peer-to-peer communication by providing the public-facing IP and port. TURN Server : Used as a fallback when direct peer-to-peer communication isn’t possible due to strict NATs or firewalls blocking connections. The TURN server relays media traffic between peers. ​ Why are ICE Servers Important? ICE (Interactive Connectivity Establishment) is a framework used by WebRTC to handle network traversal and NAT issues. The iceServers configuration provides a list of STUN and TURN servers that WebRTC uses to find the best way to connect two peers. ​ Advanced Configuration ​ ICE Servers For better connectivity, especially when testing across different networks, you can provide STUN servers: Copy Ask AI webrtc_connection = SmallWebRTCConnection( ice_servers = [ "stun:stun.l.google.com:19302" , "stun:stun1.l.google.com:19302" ] ) You can also use IceServer objects for more advanced configuration: Copy Ask AI from pipecat.transports.network.webrtc_connection import IceServer webrtc_connection = SmallWebRTCConnection( ice_servers = [ IceServer( urls = "stun:stun.l.google.com:19302" ), IceServer( urls = "turn:turn.example.com:3478" , username = "username" , credential = "password" ) ] ) ​ Troubleshooting If clients have trouble connecting or streaming: Check browser console for WebRTC errors Ensure you’re using HTTPS in production (required for WebRTC) For testing across networks, consider using Daily which provides TURN servers Verify browser permissions for camera and microphone Daily WebRTC FastAPI WebSocket On this page Overview Installation Class Reference SmallWebRTCConnection Methods SmallWebRTCTransport Methods Event Handlers Basic Usage How to connect with SmallWebRTCTransport Examples Media Handling Audio Video WebRTC ICE Servers Configuration What are STUN and TURN Servers? Why are ICE Servers Important? Advanced Configuration ICE Servers Troubleshooting Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/transport_small-webrtc_430fc4e0.txt b/transport_small-webrtc_430fc4e0.txt
new file mode 100644
index 0000000000000000000000000000000000000000..b44a1440fe481850c208b506fd1b2862eca63403
--- /dev/null
+++ b/transport_small-webrtc_430fc4e0.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/transport/small-webrtc#param-on-client-closed
+Title: SmallWebRTCTransport - Pipecat
+==================================================
+
+SmallWebRTCTransport - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Transport SmallWebRTCTransport Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Daily WebRTC SmallWebRTCTransport FastAPI WebSocket WebSocket Server Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview SmallWebRTCTransport enables peer-to-peer WebRTC connections between clients and your Pipecat application. It implements bidirectional audio and video streaming using WebRTC for real-time communication. This transport is intended for lightweight implementations, particularly for local development and testing. It expects your clients to include a corresponding SmallWebRTCTransport implementation. See here for the JavaScript implementation. SmallWebRTCTransport is best used for testing and development. For production deployments with scale, consider using the DailyTransport , as it has global, low-latency infrastructure. ​ Installation To use SmallWebRTCTransport , install the required dependencies: Copy Ask AI pip install pipecat-ai[webrtc] ​ Class Reference ​ SmallWebRTCConnection SmallWebRTCConnection manages the WebRTC connection details, peer connection state, and ICE candidates. It handles the signaling process and media tracks. Copy Ask AI SmallWebRTCConnection( ice_servers = None ) ​ ice_servers Union[List[str], List[IceServer]] List of STUN/TURN server URLs for ICE connection establishment. Can be provided as strings or as IceServer objects. ​ Methods ​ initialize async method Initialize the connection with a client’s SDP offer. Parameters: sdp : String containing the Session Description Protocol data from client’s offer type : String representing the SDP message type (typically “offer”) Copy Ask AI await webrtc_connection.initialize( sdp = client_sdp, type = "offer" ) ​ connect async method Establish the WebRTC peer connection after initialization. Copy Ask AI await webrtc_connection.connect() ​ close async method Close the WebRTC peer connection. Copy Ask AI await webrtc_connection.close() ​ disconnect async method Disconnect the WebRTC peer connection and send a peer left message to the client. Copy Ask AI await webrtc_connection.disconnect() ​ renegotiate async method Handle connection renegotiation requests. Parameters: sdp : String containing the Session Description Protocol data for renegotiation type : String representing the SDP message type restart_pc : Boolean indicating whether to completely restart the peer connection (default: False) Copy Ask AI await webrtc_connection.renegotiate( sdp = new_sdp, type = "offer" , restart_pc = False ) ​ get_answer method Retrieve the SDP answer to send back to the client. Returns a dictionary with sdp , type , and pc_id fields. Copy Ask AI answer = webrtc_connection.get_answer() # Returns: {"sdp": "...", "type": "answer", "pc_id": "..."} ​ send_app_message method Send an application message to the client. Parameters: message : The message to send (will be JSON serialized) Copy Ask AI webrtc_connection.send_app_message({ "action" : "greeting" , "text" : "Hello!" }) ​ is_connected method Check if the connection is active. Copy Ask AI if webrtc_connection.is_connected(): print ( "Connection is active" ) ​ audio_input_track method Get the audio input track from the client. Copy Ask AI audio_track = webrtc_connection.audio_input_track() ​ video_input_track method Get the video input track from the client. Copy Ask AI video_track = webrtc_connection.video_input_track() ​ replace_audio_track method Replace the current audio track with a new one. Parameters: track : The new audio track to use Copy Ask AI webrtc_connection.replace_audio_track(new_audio_track) ​ replace_video_track method Replace the current video track with a new one. Parameters: track : The new video track to use Copy Ask AI webrtc_connection.replace_video_track(new_video_track) ​ ask_to_renegotiate method Request the client to initiate connection renegotiation. Copy Ask AI webrtc_connection.ask_to_renegotiate() ​ event_handler decorator Register an event handler for connection events. Events: "app-message" : Called when a message is received from the client "track-started" : Called when a new track is started "track-ended" : Called when a track ends "connecting" : Called when connection is being established "connected" : Called when connection is established "disconnected" : Called when connection is lost "closed" : Called when connection is closed "failed" : Called when connection fails "new" : Called when a new connection is created Copy Ask AI @webrtc_connection.event_handler ( "connected" ) async def on_connected ( connection ): print ( f "WebRTC connection established" ) ​ SmallWebRTCTransport SmallWebRTCTransport is the main transport class that manages both input and output transports for WebRTC communication. Copy Ask AI SmallWebRTCTransport( webrtc_connection: SmallWebRTCConnection, params: TransportParams, input_name: Optional[ str ] = None , output_name: Optional[ str ] = None ) ​ webrtc_connection SmallWebRTCConnection required An instance of SmallWebRTCConnection that manages the WebRTC connection ​ params TransportParams required Configuration parameters for the transport Show TransportParams properties ​ audio_in_enabled bool default: false Enable audio input from the WebRTC client ​ audio_in_passthrough bool default: "False" When enabled, incoming audio frames are pushed downstream ​ audio_out_enabled bool default: false Enable audio output to the WebRTC client ​ audio_in_sample_rate int Sample rate for incoming audio (Hz) ​ audio_out_sample_rate int Sample rate for outgoing audio (Hz) ​ audio_in_channels int default: 1 Number of audio input channels (1 for mono, 2 for stereo) ​ audio_out_channels int default: 1 Number of audio output channels (1 for mono, 2 for stereo) ​ video_in_enabled bool default: false Enable video input from the WebRTC client ​ video_out_enabled bool default: false Enable video output to the WebRTC client ​ video_out_width int default: 640 Width of outgoing video ​ video_out_height int default: 480 Height of outgoing video ​ video_out_framerate int default: 30 Framerate of outgoing video ​ vad_analyzer VADAnalyzer Custom VAD analyzer implementation ​ input_name str Optional name for the input transport ​ output_name str Optional name for the output transport ​ Methods ​ input method Returns the input transport instance. Copy Ask AI input_transport = webrtc_transport.input() ​ output method Returns the output transport instance. Copy Ask AI output_transport = webrtc_transport.output() ​ send_image async method Send an image frame to the client. Parameters: frame : The image frame to send (OutputImageRawFrame or SpriteFrame) Copy Ask AI await webrtc_transport.send_image(image_frame) ​ send_audio async method Send an audio frame to the client. Parameters: frame : The audio frame to send (OutputAudioRawFrame) Copy Ask AI await webrtc_transport.send_audio(audio_frame) ​ Event Handlers ​ on_app_message async callback Called when receiving application messages from the client. Parameters: message : The received message Copy Ask AI @webrtc_transport.event_handler ( "on_app_message" ) async def on_app_message ( message ): print ( f "Received message: { message } " ) ​ on_client_connected async callback Called when a client successfully connects. Parameters: transport : The SmallWebRTCTransport instance webrtc_connection : The connection that was established Copy Ask AI @webrtc_transport.event_handler ( "on_client_connected" ) async def on_client_connected ( transport , webrtc_connection ): print ( f "Client connected" ) ​ on_client_disconnected async callback Called when a client disconnects. Parameters: transport : The SmallWebRTCTransport instance webrtc_connection : The connection that was disconnected Copy Ask AI @webrtc_transport.event_handler ( "on_client_disconnected" ) async def on_client_disconnected ( transport , webrtc_connection ): print ( f "Client disconnected" ) ​ on_client_closed async callback Called when a client connection is closed. Parameters: transport : The SmallWebRTCTransport instance webrtc_connection : The connection that was closed Copy Ask AI @webrtc_transport.event_handler ( "on_client_closed" ) async def on_client_closed ( transport , webrtc_connection ): print ( f "Client connection closed" ) ​ Basic Usage This basic usage example shows the transport specific parts of a bot.py file required to configure your bot: Copy Ask AI from pipecat.audio.vad.silero import SileroVADAnalyzer from pipecat.pipeline.pipeline import Pipeline from pipecat.transports.base_transport import TransportParams from pipecat.transports.network.small_webrtc import SmallWebRTCTransport from pipecat.transports.network.webrtc_connection import SmallWebRTCConnection async def run_bot ( webrtc_connection ): # Create the WebRTC transport with the provided connection transport = SmallWebRTCTransport( webrtc_connection = webrtc_connection, params = TransportParams( audio_in_enabled = True , # Accept audio from the client audio_out_enabled = True , # Send audio to the client vad_analyzer = SileroVADAnalyzer(), ), ) # Set up your services and context # Create the pipeline pipeline = Pipeline([ transport.input(), # Receive audio from client stt, # Convert speech to text context_aggregator.user(), # Add user messages to context llm, # Process text with LLM tts, # Convert text to speech transport.output(), # Send audio responses to client context_aggregator.assistant(), # Add assistant responses to context ]) # Register event handlers @transport.event_handler ( "on_client_connected" ) async def on_client_connected ( transport , client ): logger.info( "Client connected" ) # Start the conversation when client connects await task.queue_frames([context_aggregator.user().get_context_frame()]) @transport.event_handler ( "on_client_disconnected" ) async def on_client_disconnected ( transport , client ): logger.info( "Client disconnected" ) @transport.event_handler ( "on_client_closed" ) async def on_client_closed ( transport , client ): logger.info( "Client closed" ) await task.cancel() ​ How to connect with SmallWebRTCTransport For a client/server connection, you have two options for how to connect the client to the server: Use a Pipecat client SDK with the SmallWebRTCTransport . See the Client SDK docs to get started. Using the WebRTC API directly. This is only recommended for advanced use cases where the Pipecat client SDKs don’t have an available transport. ​ Examples To see a complete implementation, check out the following examples: Video Transform Demonstrates real-time video processing using WebRTC transport Voice Agent Implements a voice assistant using WebRTC for audio communication ​ Media Handling ​ Audio Audio is processed in 20ms chunks by default. The transport handles audio format conversion and resampling as needed: Input audio is processed at 16kHz (mono) to be compatible with speech recognition services Output audio can be configured to match your application’s requirements, but it must be mono, 16-bit PCM audio ​ Video Video is streamed using RGB format by default. The transport provides: Frame conversion between different color formats (RGB, YUV, etc.) Configurable resolution and framerate ​ WebRTC ICE Servers Configuration When implementing WebRTC in your project, STUN (Session Traversal Utilities for NAT) and TURN (Traversal Using Relays around NAT) servers are usually needed in cases where users are behind routers or firewalls. In local networks (e.g., testing within the same home or office network), you usually don’t need to configure STUN or TURN servers. In such cases, WebRTC can often directly establish peer-to-peer connections without needing to traverse NAT or firewalls. ​ What are STUN and TURN Servers? STUN Server : Helps clients discover their public IP address and port when they’re behind a NAT (Network Address Translation) device (like a router). This allows WebRTC to attempt direct peer-to-peer communication by providing the public-facing IP and port. TURN Server : Used as a fallback when direct peer-to-peer communication isn’t possible due to strict NATs or firewalls blocking connections. The TURN server relays media traffic between peers. ​ Why are ICE Servers Important? ICE (Interactive Connectivity Establishment) is a framework used by WebRTC to handle network traversal and NAT issues. The iceServers configuration provides a list of STUN and TURN servers that WebRTC uses to find the best way to connect two peers. ​ Advanced Configuration ​ ICE Servers For better connectivity, especially when testing across different networks, you can provide STUN servers: Copy Ask AI webrtc_connection = SmallWebRTCConnection( ice_servers = [ "stun:stun.l.google.com:19302" , "stun:stun1.l.google.com:19302" ] ) You can also use IceServer objects for more advanced configuration: Copy Ask AI from pipecat.transports.network.webrtc_connection import IceServer webrtc_connection = SmallWebRTCConnection( ice_servers = [ IceServer( urls = "stun:stun.l.google.com:19302" ), IceServer( urls = "turn:turn.example.com:3478" , username = "username" , credential = "password" ) ] ) ​ Troubleshooting If clients have trouble connecting or streaming: Check browser console for WebRTC errors Ensure you’re using HTTPS in production (required for WebRTC) For testing across networks, consider using Daily which provides TURN servers Verify browser permissions for camera and microphone Daily WebRTC FastAPI WebSocket On this page Overview Installation Class Reference SmallWebRTCConnection Methods SmallWebRTCTransport Methods Event Handlers Basic Usage How to connect with SmallWebRTCTransport Examples Media Handling Audio Video WebRTC ICE Servers Configuration What are STUN and TURN Servers? Why are ICE Servers Important? Advanced Configuration ICE Servers Troubleshooting Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/transport_small-webrtc_536618af.txt b/transport_small-webrtc_536618af.txt
new file mode 100644
index 0000000000000000000000000000000000000000..de21fdb8d5d83eb446e0ccbe1a3500631855ccf6
--- /dev/null
+++ b/transport_small-webrtc_536618af.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/transport/small-webrtc#param-params
+Title: SmallWebRTCTransport - Pipecat
+==================================================
+
+SmallWebRTCTransport - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Transport SmallWebRTCTransport Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Daily WebRTC SmallWebRTCTransport FastAPI WebSocket WebSocket Server Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview SmallWebRTCTransport enables peer-to-peer WebRTC connections between clients and your Pipecat application. It implements bidirectional audio and video streaming using WebRTC for real-time communication. This transport is intended for lightweight implementations, particularly for local development and testing. It expects your clients to include a corresponding SmallWebRTCTransport implementation. See here for the JavaScript implementation. SmallWebRTCTransport is best used for testing and development. For production deployments with scale, consider using the DailyTransport , as it has global, low-latency infrastructure. ​ Installation To use SmallWebRTCTransport , install the required dependencies: Copy Ask AI pip install pipecat-ai[webrtc] ​ Class Reference ​ SmallWebRTCConnection SmallWebRTCConnection manages the WebRTC connection details, peer connection state, and ICE candidates. It handles the signaling process and media tracks. Copy Ask AI SmallWebRTCConnection( ice_servers = None ) ​ ice_servers Union[List[str], List[IceServer]] List of STUN/TURN server URLs for ICE connection establishment. Can be provided as strings or as IceServer objects. ​ Methods ​ initialize async method Initialize the connection with a client’s SDP offer. Parameters: sdp : String containing the Session Description Protocol data from client’s offer type : String representing the SDP message type (typically “offer”) Copy Ask AI await webrtc_connection.initialize( sdp = client_sdp, type = "offer" ) ​ connect async method Establish the WebRTC peer connection after initialization. Copy Ask AI await webrtc_connection.connect() ​ close async method Close the WebRTC peer connection. Copy Ask AI await webrtc_connection.close() ​ disconnect async method Disconnect the WebRTC peer connection and send a peer left message to the client. Copy Ask AI await webrtc_connection.disconnect() ​ renegotiate async method Handle connection renegotiation requests. Parameters: sdp : String containing the Session Description Protocol data for renegotiation type : String representing the SDP message type restart_pc : Boolean indicating whether to completely restart the peer connection (default: False) Copy Ask AI await webrtc_connection.renegotiate( sdp = new_sdp, type = "offer" , restart_pc = False ) ​ get_answer method Retrieve the SDP answer to send back to the client. Returns a dictionary with sdp , type , and pc_id fields. Copy Ask AI answer = webrtc_connection.get_answer() # Returns: {"sdp": "...", "type": "answer", "pc_id": "..."} ​ send_app_message method Send an application message to the client. Parameters: message : The message to send (will be JSON serialized) Copy Ask AI webrtc_connection.send_app_message({ "action" : "greeting" , "text" : "Hello!" }) ​ is_connected method Check if the connection is active. Copy Ask AI if webrtc_connection.is_connected(): print ( "Connection is active" ) ​ audio_input_track method Get the audio input track from the client. Copy Ask AI audio_track = webrtc_connection.audio_input_track() ​ video_input_track method Get the video input track from the client. Copy Ask AI video_track = webrtc_connection.video_input_track() ​ replace_audio_track method Replace the current audio track with a new one. Parameters: track : The new audio track to use Copy Ask AI webrtc_connection.replace_audio_track(new_audio_track) ​ replace_video_track method Replace the current video track with a new one. Parameters: track : The new video track to use Copy Ask AI webrtc_connection.replace_video_track(new_video_track) ​ ask_to_renegotiate method Request the client to initiate connection renegotiation. Copy Ask AI webrtc_connection.ask_to_renegotiate() ​ event_handler decorator Register an event handler for connection events. Events: "app-message" : Called when a message is received from the client "track-started" : Called when a new track is started "track-ended" : Called when a track ends "connecting" : Called when connection is being established "connected" : Called when connection is established "disconnected" : Called when connection is lost "closed" : Called when connection is closed "failed" : Called when connection fails "new" : Called when a new connection is created Copy Ask AI @webrtc_connection.event_handler ( "connected" ) async def on_connected ( connection ): print ( f "WebRTC connection established" ) ​ SmallWebRTCTransport SmallWebRTCTransport is the main transport class that manages both input and output transports for WebRTC communication. Copy Ask AI SmallWebRTCTransport( webrtc_connection: SmallWebRTCConnection, params: TransportParams, input_name: Optional[ str ] = None , output_name: Optional[ str ] = None ) ​ webrtc_connection SmallWebRTCConnection required An instance of SmallWebRTCConnection that manages the WebRTC connection ​ params TransportParams required Configuration parameters for the transport Show TransportParams properties ​ audio_in_enabled bool default: false Enable audio input from the WebRTC client ​ audio_in_passthrough bool default: "False" When enabled, incoming audio frames are pushed downstream ​ audio_out_enabled bool default: false Enable audio output to the WebRTC client ​ audio_in_sample_rate int Sample rate for incoming audio (Hz) ​ audio_out_sample_rate int Sample rate for outgoing audio (Hz) ​ audio_in_channels int default: 1 Number of audio input channels (1 for mono, 2 for stereo) ​ audio_out_channels int default: 1 Number of audio output channels (1 for mono, 2 for stereo) ​ video_in_enabled bool default: false Enable video input from the WebRTC client ​ video_out_enabled bool default: false Enable video output to the WebRTC client ​ video_out_width int default: 640 Width of outgoing video ​ video_out_height int default: 480 Height of outgoing video ​ video_out_framerate int default: 30 Framerate of outgoing video ​ vad_analyzer VADAnalyzer Custom VAD analyzer implementation ​ input_name str Optional name for the input transport ​ output_name str Optional name for the output transport ​ Methods ​ input method Returns the input transport instance. Copy Ask AI input_transport = webrtc_transport.input() ​ output method Returns the output transport instance. Copy Ask AI output_transport = webrtc_transport.output() ​ send_image async method Send an image frame to the client. Parameters: frame : The image frame to send (OutputImageRawFrame or SpriteFrame) Copy Ask AI await webrtc_transport.send_image(image_frame) ​ send_audio async method Send an audio frame to the client. Parameters: frame : The audio frame to send (OutputAudioRawFrame) Copy Ask AI await webrtc_transport.send_audio(audio_frame) ​ Event Handlers ​ on_app_message async callback Called when receiving application messages from the client. Parameters: message : The received message Copy Ask AI @webrtc_transport.event_handler ( "on_app_message" ) async def on_app_message ( message ): print ( f "Received message: { message } " ) ​ on_client_connected async callback Called when a client successfully connects. Parameters: transport : The SmallWebRTCTransport instance webrtc_connection : The connection that was established Copy Ask AI @webrtc_transport.event_handler ( "on_client_connected" ) async def on_client_connected ( transport , webrtc_connection ): print ( f "Client connected" ) ​ on_client_disconnected async callback Called when a client disconnects. Parameters: transport : The SmallWebRTCTransport instance webrtc_connection : The connection that was disconnected Copy Ask AI @webrtc_transport.event_handler ( "on_client_disconnected" ) async def on_client_disconnected ( transport , webrtc_connection ): print ( f "Client disconnected" ) ​ on_client_closed async callback Called when a client connection is closed. Parameters: transport : The SmallWebRTCTransport instance webrtc_connection : The connection that was closed Copy Ask AI @webrtc_transport.event_handler ( "on_client_closed" ) async def on_client_closed ( transport , webrtc_connection ): print ( f "Client connection closed" ) ​ Basic Usage This basic usage example shows the transport specific parts of a bot.py file required to configure your bot: Copy Ask AI from pipecat.audio.vad.silero import SileroVADAnalyzer from pipecat.pipeline.pipeline import Pipeline from pipecat.transports.base_transport import TransportParams from pipecat.transports.network.small_webrtc import SmallWebRTCTransport from pipecat.transports.network.webrtc_connection import SmallWebRTCConnection async def run_bot ( webrtc_connection ): # Create the WebRTC transport with the provided connection transport = SmallWebRTCTransport( webrtc_connection = webrtc_connection, params = TransportParams( audio_in_enabled = True , # Accept audio from the client audio_out_enabled = True , # Send audio to the client vad_analyzer = SileroVADAnalyzer(), ), ) # Set up your services and context # Create the pipeline pipeline = Pipeline([ transport.input(), # Receive audio from client stt, # Convert speech to text context_aggregator.user(), # Add user messages to context llm, # Process text with LLM tts, # Convert text to speech transport.output(), # Send audio responses to client context_aggregator.assistant(), # Add assistant responses to context ]) # Register event handlers @transport.event_handler ( "on_client_connected" ) async def on_client_connected ( transport , client ): logger.info( "Client connected" ) # Start the conversation when client connects await task.queue_frames([context_aggregator.user().get_context_frame()]) @transport.event_handler ( "on_client_disconnected" ) async def on_client_disconnected ( transport , client ): logger.info( "Client disconnected" ) @transport.event_handler ( "on_client_closed" ) async def on_client_closed ( transport , client ): logger.info( "Client closed" ) await task.cancel() ​ How to connect with SmallWebRTCTransport For a client/server connection, you have two options for how to connect the client to the server: Use a Pipecat client SDK with the SmallWebRTCTransport . See the Client SDK docs to get started. Using the WebRTC API directly. This is only recommended for advanced use cases where the Pipecat client SDKs don’t have an available transport. ​ Examples To see a complete implementation, check out the following examples: Video Transform Demonstrates real-time video processing using WebRTC transport Voice Agent Implements a voice assistant using WebRTC for audio communication ​ Media Handling ​ Audio Audio is processed in 20ms chunks by default. The transport handles audio format conversion and resampling as needed: Input audio is processed at 16kHz (mono) to be compatible with speech recognition services Output audio can be configured to match your application’s requirements, but it must be mono, 16-bit PCM audio ​ Video Video is streamed using RGB format by default. The transport provides: Frame conversion between different color formats (RGB, YUV, etc.) Configurable resolution and framerate ​ WebRTC ICE Servers Configuration When implementing WebRTC in your project, STUN (Session Traversal Utilities for NAT) and TURN (Traversal Using Relays around NAT) servers are usually needed in cases where users are behind routers or firewalls. In local networks (e.g., testing within the same home or office network), you usually don’t need to configure STUN or TURN servers. In such cases, WebRTC can often directly establish peer-to-peer connections without needing to traverse NAT or firewalls. ​ What are STUN and TURN Servers? STUN Server : Helps clients discover their public IP address and port when they’re behind a NAT (Network Address Translation) device (like a router). This allows WebRTC to attempt direct peer-to-peer communication by providing the public-facing IP and port. TURN Server : Used as a fallback when direct peer-to-peer communication isn’t possible due to strict NATs or firewalls blocking connections. The TURN server relays media traffic between peers. ​ Why are ICE Servers Important? ICE (Interactive Connectivity Establishment) is a framework used by WebRTC to handle network traversal and NAT issues. The iceServers configuration provides a list of STUN and TURN servers that WebRTC uses to find the best way to connect two peers. ​ Advanced Configuration ​ ICE Servers For better connectivity, especially when testing across different networks, you can provide STUN servers: Copy Ask AI webrtc_connection = SmallWebRTCConnection( ice_servers = [ "stun:stun.l.google.com:19302" , "stun:stun1.l.google.com:19302" ] ) You can also use IceServer objects for more advanced configuration: Copy Ask AI from pipecat.transports.network.webrtc_connection import IceServer webrtc_connection = SmallWebRTCConnection( ice_servers = [ IceServer( urls = "stun:stun.l.google.com:19302" ), IceServer( urls = "turn:turn.example.com:3478" , username = "username" , credential = "password" ) ] ) ​ Troubleshooting If clients have trouble connecting or streaming: Check browser console for WebRTC errors Ensure you’re using HTTPS in production (required for WebRTC) For testing across networks, consider using Daily which provides TURN servers Verify browser permissions for camera and microphone Daily WebRTC FastAPI WebSocket On this page Overview Installation Class Reference SmallWebRTCConnection Methods SmallWebRTCTransport Methods Event Handlers Basic Usage How to connect with SmallWebRTCTransport Examples Media Handling Audio Video WebRTC ICE Servers Configuration What are STUN and TURN Servers? Why are ICE Servers Important? Advanced Configuration ICE Servers Troubleshooting Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/transport_small-webrtc_5a1f60ab.txt b/transport_small-webrtc_5a1f60ab.txt
new file mode 100644
index 0000000000000000000000000000000000000000..aa4cd78d939e151c282948e31fb3000c37e0e6d4
--- /dev/null
+++ b/transport_small-webrtc_5a1f60ab.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/transport/small-webrtc#param-on-client-disconnected
+Title: SmallWebRTCTransport - Pipecat
+==================================================
+
+SmallWebRTCTransport - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Transport SmallWebRTCTransport Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Daily WebRTC SmallWebRTCTransport FastAPI WebSocket WebSocket Server Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview SmallWebRTCTransport enables peer-to-peer WebRTC connections between clients and your Pipecat application. It implements bidirectional audio and video streaming using WebRTC for real-time communication. This transport is intended for lightweight implementations, particularly for local development and testing. It expects your clients to include a corresponding SmallWebRTCTransport implementation. See here for the JavaScript implementation. SmallWebRTCTransport is best used for testing and development. For production deployments with scale, consider using the DailyTransport , as it has global, low-latency infrastructure. ​ Installation To use SmallWebRTCTransport , install the required dependencies: Copy Ask AI pip install pipecat-ai[webrtc] ​ Class Reference ​ SmallWebRTCConnection SmallWebRTCConnection manages the WebRTC connection details, peer connection state, and ICE candidates. It handles the signaling process and media tracks. Copy Ask AI SmallWebRTCConnection( ice_servers = None ) ​ ice_servers Union[List[str], List[IceServer]] List of STUN/TURN server URLs for ICE connection establishment. Can be provided as strings or as IceServer objects. ​ Methods ​ initialize async method Initialize the connection with a client’s SDP offer. Parameters: sdp : String containing the Session Description Protocol data from client’s offer type : String representing the SDP message type (typically “offer”) Copy Ask AI await webrtc_connection.initialize( sdp = client_sdp, type = "offer" ) ​ connect async method Establish the WebRTC peer connection after initialization. Copy Ask AI await webrtc_connection.connect() ​ close async method Close the WebRTC peer connection. Copy Ask AI await webrtc_connection.close() ​ disconnect async method Disconnect the WebRTC peer connection and send a peer left message to the client. Copy Ask AI await webrtc_connection.disconnect() ​ renegotiate async method Handle connection renegotiation requests. Parameters: sdp : String containing the Session Description Protocol data for renegotiation type : String representing the SDP message type restart_pc : Boolean indicating whether to completely restart the peer connection (default: False) Copy Ask AI await webrtc_connection.renegotiate( sdp = new_sdp, type = "offer" , restart_pc = False ) ​ get_answer method Retrieve the SDP answer to send back to the client. Returns a dictionary with sdp , type , and pc_id fields. Copy Ask AI answer = webrtc_connection.get_answer() # Returns: {"sdp": "...", "type": "answer", "pc_id": "..."} ​ send_app_message method Send an application message to the client. Parameters: message : The message to send (will be JSON serialized) Copy Ask AI webrtc_connection.send_app_message({ "action" : "greeting" , "text" : "Hello!" }) ​ is_connected method Check if the connection is active. Copy Ask AI if webrtc_connection.is_connected(): print ( "Connection is active" ) ​ audio_input_track method Get the audio input track from the client. Copy Ask AI audio_track = webrtc_connection.audio_input_track() ​ video_input_track method Get the video input track from the client. Copy Ask AI video_track = webrtc_connection.video_input_track() ​ replace_audio_track method Replace the current audio track with a new one. Parameters: track : The new audio track to use Copy Ask AI webrtc_connection.replace_audio_track(new_audio_track) ​ replace_video_track method Replace the current video track with a new one. Parameters: track : The new video track to use Copy Ask AI webrtc_connection.replace_video_track(new_video_track) ​ ask_to_renegotiate method Request the client to initiate connection renegotiation. Copy Ask AI webrtc_connection.ask_to_renegotiate() ​ event_handler decorator Register an event handler for connection events. Events: "app-message" : Called when a message is received from the client "track-started" : Called when a new track is started "track-ended" : Called when a track ends "connecting" : Called when connection is being established "connected" : Called when connection is established "disconnected" : Called when connection is lost "closed" : Called when connection is closed "failed" : Called when connection fails "new" : Called when a new connection is created Copy Ask AI @webrtc_connection.event_handler ( "connected" ) async def on_connected ( connection ): print ( f "WebRTC connection established" ) ​ SmallWebRTCTransport SmallWebRTCTransport is the main transport class that manages both input and output transports for WebRTC communication. Copy Ask AI SmallWebRTCTransport( webrtc_connection: SmallWebRTCConnection, params: TransportParams, input_name: Optional[ str ] = None , output_name: Optional[ str ] = None ) ​ webrtc_connection SmallWebRTCConnection required An instance of SmallWebRTCConnection that manages the WebRTC connection ​ params TransportParams required Configuration parameters for the transport Show TransportParams properties ​ audio_in_enabled bool default: false Enable audio input from the WebRTC client ​ audio_in_passthrough bool default: "False" When enabled, incoming audio frames are pushed downstream ​ audio_out_enabled bool default: false Enable audio output to the WebRTC client ​ audio_in_sample_rate int Sample rate for incoming audio (Hz) ​ audio_out_sample_rate int Sample rate for outgoing audio (Hz) ​ audio_in_channels int default: 1 Number of audio input channels (1 for mono, 2 for stereo) ​ audio_out_channels int default: 1 Number of audio output channels (1 for mono, 2 for stereo) ​ video_in_enabled bool default: false Enable video input from the WebRTC client ​ video_out_enabled bool default: false Enable video output to the WebRTC client ​ video_out_width int default: 640 Width of outgoing video ​ video_out_height int default: 480 Height of outgoing video ​ video_out_framerate int default: 30 Framerate of outgoing video ​ vad_analyzer VADAnalyzer Custom VAD analyzer implementation ​ input_name str Optional name for the input transport ​ output_name str Optional name for the output transport ​ Methods ​ input method Returns the input transport instance. Copy Ask AI input_transport = webrtc_transport.input() ​ output method Returns the output transport instance. Copy Ask AI output_transport = webrtc_transport.output() ​ send_image async method Send an image frame to the client. Parameters: frame : The image frame to send (OutputImageRawFrame or SpriteFrame) Copy Ask AI await webrtc_transport.send_image(image_frame) ​ send_audio async method Send an audio frame to the client. Parameters: frame : The audio frame to send (OutputAudioRawFrame) Copy Ask AI await webrtc_transport.send_audio(audio_frame) ​ Event Handlers ​ on_app_message async callback Called when receiving application messages from the client. Parameters: message : The received message Copy Ask AI @webrtc_transport.event_handler ( "on_app_message" ) async def on_app_message ( message ): print ( f "Received message: { message } " ) ​ on_client_connected async callback Called when a client successfully connects. Parameters: transport : The SmallWebRTCTransport instance webrtc_connection : The connection that was established Copy Ask AI @webrtc_transport.event_handler ( "on_client_connected" ) async def on_client_connected ( transport , webrtc_connection ): print ( f "Client connected" ) ​ on_client_disconnected async callback Called when a client disconnects. Parameters: transport : The SmallWebRTCTransport instance webrtc_connection : The connection that was disconnected Copy Ask AI @webrtc_transport.event_handler ( "on_client_disconnected" ) async def on_client_disconnected ( transport , webrtc_connection ): print ( f "Client disconnected" ) ​ on_client_closed async callback Called when a client connection is closed. Parameters: transport : The SmallWebRTCTransport instance webrtc_connection : The connection that was closed Copy Ask AI @webrtc_transport.event_handler ( "on_client_closed" ) async def on_client_closed ( transport , webrtc_connection ): print ( f "Client connection closed" ) ​ Basic Usage This basic usage example shows the transport specific parts of a bot.py file required to configure your bot: Copy Ask AI from pipecat.audio.vad.silero import SileroVADAnalyzer from pipecat.pipeline.pipeline import Pipeline from pipecat.transports.base_transport import TransportParams from pipecat.transports.network.small_webrtc import SmallWebRTCTransport from pipecat.transports.network.webrtc_connection import SmallWebRTCConnection async def run_bot ( webrtc_connection ): # Create the WebRTC transport with the provided connection transport = SmallWebRTCTransport( webrtc_connection = webrtc_connection, params = TransportParams( audio_in_enabled = True , # Accept audio from the client audio_out_enabled = True , # Send audio to the client vad_analyzer = SileroVADAnalyzer(), ), ) # Set up your services and context # Create the pipeline pipeline = Pipeline([ transport.input(), # Receive audio from client stt, # Convert speech to text context_aggregator.user(), # Add user messages to context llm, # Process text with LLM tts, # Convert text to speech transport.output(), # Send audio responses to client context_aggregator.assistant(), # Add assistant responses to context ]) # Register event handlers @transport.event_handler ( "on_client_connected" ) async def on_client_connected ( transport , client ): logger.info( "Client connected" ) # Start the conversation when client connects await task.queue_frames([context_aggregator.user().get_context_frame()]) @transport.event_handler ( "on_client_disconnected" ) async def on_client_disconnected ( transport , client ): logger.info( "Client disconnected" ) @transport.event_handler ( "on_client_closed" ) async def on_client_closed ( transport , client ): logger.info( "Client closed" ) await task.cancel() ​ How to connect with SmallWebRTCTransport For a client/server connection, you have two options for how to connect the client to the server: Use a Pipecat client SDK with the SmallWebRTCTransport . See the Client SDK docs to get started. Using the WebRTC API directly. This is only recommended for advanced use cases where the Pipecat client SDKs don’t have an available transport. ​ Examples To see a complete implementation, check out the following examples: Video Transform Demonstrates real-time video processing using WebRTC transport Voice Agent Implements a voice assistant using WebRTC for audio communication ​ Media Handling ​ Audio Audio is processed in 20ms chunks by default. The transport handles audio format conversion and resampling as needed: Input audio is processed at 16kHz (mono) to be compatible with speech recognition services Output audio can be configured to match your application’s requirements, but it must be mono, 16-bit PCM audio ​ Video Video is streamed using RGB format by default. The transport provides: Frame conversion between different color formats (RGB, YUV, etc.) Configurable resolution and framerate ​ WebRTC ICE Servers Configuration When implementing WebRTC in your project, STUN (Session Traversal Utilities for NAT) and TURN (Traversal Using Relays around NAT) servers are usually needed in cases where users are behind routers or firewalls. In local networks (e.g., testing within the same home or office network), you usually don’t need to configure STUN or TURN servers. In such cases, WebRTC can often directly establish peer-to-peer connections without needing to traverse NAT or firewalls. ​ What are STUN and TURN Servers? STUN Server : Helps clients discover their public IP address and port when they’re behind a NAT (Network Address Translation) device (like a router). This allows WebRTC to attempt direct peer-to-peer communication by providing the public-facing IP and port. TURN Server : Used as a fallback when direct peer-to-peer communication isn’t possible due to strict NATs or firewalls blocking connections. The TURN server relays media traffic between peers. ​ Why are ICE Servers Important? ICE (Interactive Connectivity Establishment) is a framework used by WebRTC to handle network traversal and NAT issues. The iceServers configuration provides a list of STUN and TURN servers that WebRTC uses to find the best way to connect two peers. ​ Advanced Configuration ​ ICE Servers For better connectivity, especially when testing across different networks, you can provide STUN servers: Copy Ask AI webrtc_connection = SmallWebRTCConnection( ice_servers = [ "stun:stun.l.google.com:19302" , "stun:stun1.l.google.com:19302" ] ) You can also use IceServer objects for more advanced configuration: Copy Ask AI from pipecat.transports.network.webrtc_connection import IceServer webrtc_connection = SmallWebRTCConnection( ice_servers = [ IceServer( urls = "stun:stun.l.google.com:19302" ), IceServer( urls = "turn:turn.example.com:3478" , username = "username" , credential = "password" ) ] ) ​ Troubleshooting If clients have trouble connecting or streaming: Check browser console for WebRTC errors Ensure you’re using HTTPS in production (required for WebRTC) For testing across networks, consider using Daily which provides TURN servers Verify browser permissions for camera and microphone Daily WebRTC FastAPI WebSocket On this page Overview Installation Class Reference SmallWebRTCConnection Methods SmallWebRTCTransport Methods Event Handlers Basic Usage How to connect with SmallWebRTCTransport Examples Media Handling Audio Video WebRTC ICE Servers Configuration What are STUN and TURN Servers? Why are ICE Servers Important? Advanced Configuration ICE Servers Troubleshooting Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/transport_small-webrtc_6d90f6ed.txt b/transport_small-webrtc_6d90f6ed.txt
new file mode 100644
index 0000000000000000000000000000000000000000..fe9895873740c13d72d276abee7bd03b811428b4
--- /dev/null
+++ b/transport_small-webrtc_6d90f6ed.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/transport/small-webrtc#param-audio-input-track
+Title: SmallWebRTCTransport - Pipecat
+==================================================
+
+SmallWebRTCTransport - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Transport SmallWebRTCTransport Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Daily WebRTC SmallWebRTCTransport FastAPI WebSocket WebSocket Server Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview SmallWebRTCTransport enables peer-to-peer WebRTC connections between clients and your Pipecat application. It implements bidirectional audio and video streaming using WebRTC for real-time communication. This transport is intended for lightweight implementations, particularly for local development and testing. It expects your clients to include a corresponding SmallWebRTCTransport implementation. See here for the JavaScript implementation. SmallWebRTCTransport is best used for testing and development. For production deployments with scale, consider using the DailyTransport , as it has global, low-latency infrastructure. ​ Installation To use SmallWebRTCTransport , install the required dependencies: Copy Ask AI pip install pipecat-ai[webrtc] ​ Class Reference ​ SmallWebRTCConnection SmallWebRTCConnection manages the WebRTC connection details, peer connection state, and ICE candidates. It handles the signaling process and media tracks. Copy Ask AI SmallWebRTCConnection( ice_servers = None ) ​ ice_servers Union[List[str], List[IceServer]] List of STUN/TURN server URLs for ICE connection establishment. Can be provided as strings or as IceServer objects. ​ Methods ​ initialize async method Initialize the connection with a client’s SDP offer. Parameters: sdp : String containing the Session Description Protocol data from client’s offer type : String representing the SDP message type (typically “offer”) Copy Ask AI await webrtc_connection.initialize( sdp = client_sdp, type = "offer" ) ​ connect async method Establish the WebRTC peer connection after initialization. Copy Ask AI await webrtc_connection.connect() ​ close async method Close the WebRTC peer connection. Copy Ask AI await webrtc_connection.close() ​ disconnect async method Disconnect the WebRTC peer connection and send a peer left message to the client. Copy Ask AI await webrtc_connection.disconnect() ​ renegotiate async method Handle connection renegotiation requests. Parameters: sdp : String containing the Session Description Protocol data for renegotiation type : String representing the SDP message type restart_pc : Boolean indicating whether to completely restart the peer connection (default: False) Copy Ask AI await webrtc_connection.renegotiate( sdp = new_sdp, type = "offer" , restart_pc = False ) ​ get_answer method Retrieve the SDP answer to send back to the client. Returns a dictionary with sdp , type , and pc_id fields. Copy Ask AI answer = webrtc_connection.get_answer() # Returns: {"sdp": "...", "type": "answer", "pc_id": "..."} ​ send_app_message method Send an application message to the client. Parameters: message : The message to send (will be JSON serialized) Copy Ask AI webrtc_connection.send_app_message({ "action" : "greeting" , "text" : "Hello!" }) ​ is_connected method Check if the connection is active. Copy Ask AI if webrtc_connection.is_connected(): print ( "Connection is active" ) ​ audio_input_track method Get the audio input track from the client. Copy Ask AI audio_track = webrtc_connection.audio_input_track() ​ video_input_track method Get the video input track from the client. Copy Ask AI video_track = webrtc_connection.video_input_track() ​ replace_audio_track method Replace the current audio track with a new one. Parameters: track : The new audio track to use Copy Ask AI webrtc_connection.replace_audio_track(new_audio_track) ​ replace_video_track method Replace the current video track with a new one. Parameters: track : The new video track to use Copy Ask AI webrtc_connection.replace_video_track(new_video_track) ​ ask_to_renegotiate method Request the client to initiate connection renegotiation. Copy Ask AI webrtc_connection.ask_to_renegotiate() ​ event_handler decorator Register an event handler for connection events. Events: "app-message" : Called when a message is received from the client "track-started" : Called when a new track is started "track-ended" : Called when a track ends "connecting" : Called when connection is being established "connected" : Called when connection is established "disconnected" : Called when connection is lost "closed" : Called when connection is closed "failed" : Called when connection fails "new" : Called when a new connection is created Copy Ask AI @webrtc_connection.event_handler ( "connected" ) async def on_connected ( connection ): print ( f "WebRTC connection established" ) ​ SmallWebRTCTransport SmallWebRTCTransport is the main transport class that manages both input and output transports for WebRTC communication. Copy Ask AI SmallWebRTCTransport( webrtc_connection: SmallWebRTCConnection, params: TransportParams, input_name: Optional[ str ] = None , output_name: Optional[ str ] = None ) ​ webrtc_connection SmallWebRTCConnection required An instance of SmallWebRTCConnection that manages the WebRTC connection ​ params TransportParams required Configuration parameters for the transport Show TransportParams properties ​ audio_in_enabled bool default: false Enable audio input from the WebRTC client ​ audio_in_passthrough bool default: "False" When enabled, incoming audio frames are pushed downstream ​ audio_out_enabled bool default: false Enable audio output to the WebRTC client ​ audio_in_sample_rate int Sample rate for incoming audio (Hz) ​ audio_out_sample_rate int Sample rate for outgoing audio (Hz) ​ audio_in_channels int default: 1 Number of audio input channels (1 for mono, 2 for stereo) ​ audio_out_channels int default: 1 Number of audio output channels (1 for mono, 2 for stereo) ​ video_in_enabled bool default: false Enable video input from the WebRTC client ​ video_out_enabled bool default: false Enable video output to the WebRTC client ​ video_out_width int default: 640 Width of outgoing video ​ video_out_height int default: 480 Height of outgoing video ​ video_out_framerate int default: 30 Framerate of outgoing video ​ vad_analyzer VADAnalyzer Custom VAD analyzer implementation ​ input_name str Optional name for the input transport ​ output_name str Optional name for the output transport ​ Methods ​ input method Returns the input transport instance. Copy Ask AI input_transport = webrtc_transport.input() ​ output method Returns the output transport instance. Copy Ask AI output_transport = webrtc_transport.output() ​ send_image async method Send an image frame to the client. Parameters: frame : The image frame to send (OutputImageRawFrame or SpriteFrame) Copy Ask AI await webrtc_transport.send_image(image_frame) ​ send_audio async method Send an audio frame to the client. Parameters: frame : The audio frame to send (OutputAudioRawFrame) Copy Ask AI await webrtc_transport.send_audio(audio_frame) ​ Event Handlers ​ on_app_message async callback Called when receiving application messages from the client. Parameters: message : The received message Copy Ask AI @webrtc_transport.event_handler ( "on_app_message" ) async def on_app_message ( message ): print ( f "Received message: { message } " ) ​ on_client_connected async callback Called when a client successfully connects. Parameters: transport : The SmallWebRTCTransport instance webrtc_connection : The connection that was established Copy Ask AI @webrtc_transport.event_handler ( "on_client_connected" ) async def on_client_connected ( transport , webrtc_connection ): print ( f "Client connected" ) ​ on_client_disconnected async callback Called when a client disconnects. Parameters: transport : The SmallWebRTCTransport instance webrtc_connection : The connection that was disconnected Copy Ask AI @webrtc_transport.event_handler ( "on_client_disconnected" ) async def on_client_disconnected ( transport , webrtc_connection ): print ( f "Client disconnected" ) ​ on_client_closed async callback Called when a client connection is closed. Parameters: transport : The SmallWebRTCTransport instance webrtc_connection : The connection that was closed Copy Ask AI @webrtc_transport.event_handler ( "on_client_closed" ) async def on_client_closed ( transport , webrtc_connection ): print ( f "Client connection closed" ) ​ Basic Usage This basic usage example shows the transport specific parts of a bot.py file required to configure your bot: Copy Ask AI from pipecat.audio.vad.silero import SileroVADAnalyzer from pipecat.pipeline.pipeline import Pipeline from pipecat.transports.base_transport import TransportParams from pipecat.transports.network.small_webrtc import SmallWebRTCTransport from pipecat.transports.network.webrtc_connection import SmallWebRTCConnection async def run_bot ( webrtc_connection ): # Create the WebRTC transport with the provided connection transport = SmallWebRTCTransport( webrtc_connection = webrtc_connection, params = TransportParams( audio_in_enabled = True , # Accept audio from the client audio_out_enabled = True , # Send audio to the client vad_analyzer = SileroVADAnalyzer(), ), ) # Set up your services and context # Create the pipeline pipeline = Pipeline([ transport.input(), # Receive audio from client stt, # Convert speech to text context_aggregator.user(), # Add user messages to context llm, # Process text with LLM tts, # Convert text to speech transport.output(), # Send audio responses to client context_aggregator.assistant(), # Add assistant responses to context ]) # Register event handlers @transport.event_handler ( "on_client_connected" ) async def on_client_connected ( transport , client ): logger.info( "Client connected" ) # Start the conversation when client connects await task.queue_frames([context_aggregator.user().get_context_frame()]) @transport.event_handler ( "on_client_disconnected" ) async def on_client_disconnected ( transport , client ): logger.info( "Client disconnected" ) @transport.event_handler ( "on_client_closed" ) async def on_client_closed ( transport , client ): logger.info( "Client closed" ) await task.cancel() ​ How to connect with SmallWebRTCTransport For a client/server connection, you have two options for how to connect the client to the server: Use a Pipecat client SDK with the SmallWebRTCTransport . See the Client SDK docs to get started. Using the WebRTC API directly. This is only recommended for advanced use cases where the Pipecat client SDKs don’t have an available transport. ​ Examples To see a complete implementation, check out the following examples: Video Transform Demonstrates real-time video processing using WebRTC transport Voice Agent Implements a voice assistant using WebRTC for audio communication ​ Media Handling ​ Audio Audio is processed in 20ms chunks by default. The transport handles audio format conversion and resampling as needed: Input audio is processed at 16kHz (mono) to be compatible with speech recognition services Output audio can be configured to match your application’s requirements, but it must be mono, 16-bit PCM audio ​ Video Video is streamed using RGB format by default. The transport provides: Frame conversion between different color formats (RGB, YUV, etc.) Configurable resolution and framerate ​ WebRTC ICE Servers Configuration When implementing WebRTC in your project, STUN (Session Traversal Utilities for NAT) and TURN (Traversal Using Relays around NAT) servers are usually needed in cases where users are behind routers or firewalls. In local networks (e.g., testing within the same home or office network), you usually don’t need to configure STUN or TURN servers. In such cases, WebRTC can often directly establish peer-to-peer connections without needing to traverse NAT or firewalls. ​ What are STUN and TURN Servers? STUN Server : Helps clients discover their public IP address and port when they’re behind a NAT (Network Address Translation) device (like a router). This allows WebRTC to attempt direct peer-to-peer communication by providing the public-facing IP and port. TURN Server : Used as a fallback when direct peer-to-peer communication isn’t possible due to strict NATs or firewalls blocking connections. The TURN server relays media traffic between peers. ​ Why are ICE Servers Important? ICE (Interactive Connectivity Establishment) is a framework used by WebRTC to handle network traversal and NAT issues. The iceServers configuration provides a list of STUN and TURN servers that WebRTC uses to find the best way to connect two peers. ​ Advanced Configuration ​ ICE Servers For better connectivity, especially when testing across different networks, you can provide STUN servers: Copy Ask AI webrtc_connection = SmallWebRTCConnection( ice_servers = [ "stun:stun.l.google.com:19302" , "stun:stun1.l.google.com:19302" ] ) You can also use IceServer objects for more advanced configuration: Copy Ask AI from pipecat.transports.network.webrtc_connection import IceServer webrtc_connection = SmallWebRTCConnection( ice_servers = [ IceServer( urls = "stun:stun.l.google.com:19302" ), IceServer( urls = "turn:turn.example.com:3478" , username = "username" , credential = "password" ) ] ) ​ Troubleshooting If clients have trouble connecting or streaming: Check browser console for WebRTC errors Ensure you’re using HTTPS in production (required for WebRTC) For testing across networks, consider using Daily which provides TURN servers Verify browser permissions for camera and microphone Daily WebRTC FastAPI WebSocket On this page Overview Installation Class Reference SmallWebRTCConnection Methods SmallWebRTCTransport Methods Event Handlers Basic Usage How to connect with SmallWebRTCTransport Examples Media Handling Audio Video WebRTC ICE Servers Configuration What are STUN and TURN Servers? Why are ICE Servers Important? Advanced Configuration ICE Servers Troubleshooting Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/transport_small-webrtc_87f84e67.txt b/transport_small-webrtc_87f84e67.txt
new file mode 100644
index 0000000000000000000000000000000000000000..301266f5037351b0a06a1fa8d4261640d3293d49
--- /dev/null
+++ b/transport_small-webrtc_87f84e67.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/transport/small-webrtc#ice-servers
+Title: SmallWebRTCTransport - Pipecat
+==================================================
+
+SmallWebRTCTransport - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Transport SmallWebRTCTransport Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Daily WebRTC SmallWebRTCTransport FastAPI WebSocket WebSocket Server Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview SmallWebRTCTransport enables peer-to-peer WebRTC connections between clients and your Pipecat application. It implements bidirectional audio and video streaming using WebRTC for real-time communication. This transport is intended for lightweight implementations, particularly for local development and testing. It expects your clients to include a corresponding SmallWebRTCTransport implementation. See here for the JavaScript implementation. SmallWebRTCTransport is best used for testing and development. For production deployments with scale, consider using the DailyTransport , as it has global, low-latency infrastructure. ​ Installation To use SmallWebRTCTransport , install the required dependencies: Copy Ask AI pip install pipecat-ai[webrtc] ​ Class Reference ​ SmallWebRTCConnection SmallWebRTCConnection manages the WebRTC connection details, peer connection state, and ICE candidates. It handles the signaling process and media tracks. Copy Ask AI SmallWebRTCConnection( ice_servers = None ) ​ ice_servers Union[List[str], List[IceServer]] List of STUN/TURN server URLs for ICE connection establishment. Can be provided as strings or as IceServer objects. ​ Methods ​ initialize async method Initialize the connection with a client’s SDP offer. Parameters: sdp : String containing the Session Description Protocol data from client’s offer type : String representing the SDP message type (typically “offer”) Copy Ask AI await webrtc_connection.initialize( sdp = client_sdp, type = "offer" ) ​ connect async method Establish the WebRTC peer connection after initialization. Copy Ask AI await webrtc_connection.connect() ​ close async method Close the WebRTC peer connection. Copy Ask AI await webrtc_connection.close() ​ disconnect async method Disconnect the WebRTC peer connection and send a peer left message to the client. Copy Ask AI await webrtc_connection.disconnect() ​ renegotiate async method Handle connection renegotiation requests. Parameters: sdp : String containing the Session Description Protocol data for renegotiation type : String representing the SDP message type restart_pc : Boolean indicating whether to completely restart the peer connection (default: False) Copy Ask AI await webrtc_connection.renegotiate( sdp = new_sdp, type = "offer" , restart_pc = False ) ​ get_answer method Retrieve the SDP answer to send back to the client. Returns a dictionary with sdp , type , and pc_id fields. Copy Ask AI answer = webrtc_connection.get_answer() # Returns: {"sdp": "...", "type": "answer", "pc_id": "..."} ​ send_app_message method Send an application message to the client. Parameters: message : The message to send (will be JSON serialized) Copy Ask AI webrtc_connection.send_app_message({ "action" : "greeting" , "text" : "Hello!" }) ​ is_connected method Check if the connection is active. Copy Ask AI if webrtc_connection.is_connected(): print ( "Connection is active" ) ​ audio_input_track method Get the audio input track from the client. Copy Ask AI audio_track = webrtc_connection.audio_input_track() ​ video_input_track method Get the video input track from the client. Copy Ask AI video_track = webrtc_connection.video_input_track() ​ replace_audio_track method Replace the current audio track with a new one. Parameters: track : The new audio track to use Copy Ask AI webrtc_connection.replace_audio_track(new_audio_track) ​ replace_video_track method Replace the current video track with a new one. Parameters: track : The new video track to use Copy Ask AI webrtc_connection.replace_video_track(new_video_track) ​ ask_to_renegotiate method Request the client to initiate connection renegotiation. Copy Ask AI webrtc_connection.ask_to_renegotiate() ​ event_handler decorator Register an event handler for connection events. Events: "app-message" : Called when a message is received from the client "track-started" : Called when a new track is started "track-ended" : Called when a track ends "connecting" : Called when connection is being established "connected" : Called when connection is established "disconnected" : Called when connection is lost "closed" : Called when connection is closed "failed" : Called when connection fails "new" : Called when a new connection is created Copy Ask AI @webrtc_connection.event_handler ( "connected" ) async def on_connected ( connection ): print ( f "WebRTC connection established" ) ​ SmallWebRTCTransport SmallWebRTCTransport is the main transport class that manages both input and output transports for WebRTC communication. Copy Ask AI SmallWebRTCTransport( webrtc_connection: SmallWebRTCConnection, params: TransportParams, input_name: Optional[ str ] = None , output_name: Optional[ str ] = None ) ​ webrtc_connection SmallWebRTCConnection required An instance of SmallWebRTCConnection that manages the WebRTC connection ​ params TransportParams required Configuration parameters for the transport Show TransportParams properties ​ audio_in_enabled bool default: false Enable audio input from the WebRTC client ​ audio_in_passthrough bool default: "False" When enabled, incoming audio frames are pushed downstream ​ audio_out_enabled bool default: false Enable audio output to the WebRTC client ​ audio_in_sample_rate int Sample rate for incoming audio (Hz) ​ audio_out_sample_rate int Sample rate for outgoing audio (Hz) ​ audio_in_channels int default: 1 Number of audio input channels (1 for mono, 2 for stereo) ​ audio_out_channels int default: 1 Number of audio output channels (1 for mono, 2 for stereo) ​ video_in_enabled bool default: false Enable video input from the WebRTC client ​ video_out_enabled bool default: false Enable video output to the WebRTC client ​ video_out_width int default: 640 Width of outgoing video ​ video_out_height int default: 480 Height of outgoing video ​ video_out_framerate int default: 30 Framerate of outgoing video ​ vad_analyzer VADAnalyzer Custom VAD analyzer implementation ​ input_name str Optional name for the input transport ​ output_name str Optional name for the output transport ​ Methods ​ input method Returns the input transport instance. Copy Ask AI input_transport = webrtc_transport.input() ​ output method Returns the output transport instance. Copy Ask AI output_transport = webrtc_transport.output() ​ send_image async method Send an image frame to the client. Parameters: frame : The image frame to send (OutputImageRawFrame or SpriteFrame) Copy Ask AI await webrtc_transport.send_image(image_frame) ​ send_audio async method Send an audio frame to the client. Parameters: frame : The audio frame to send (OutputAudioRawFrame) Copy Ask AI await webrtc_transport.send_audio(audio_frame) ​ Event Handlers ​ on_app_message async callback Called when receiving application messages from the client. Parameters: message : The received message Copy Ask AI @webrtc_transport.event_handler ( "on_app_message" ) async def on_app_message ( message ): print ( f "Received message: { message } " ) ​ on_client_connected async callback Called when a client successfully connects. Parameters: transport : The SmallWebRTCTransport instance webrtc_connection : The connection that was established Copy Ask AI @webrtc_transport.event_handler ( "on_client_connected" ) async def on_client_connected ( transport , webrtc_connection ): print ( f "Client connected" ) ​ on_client_disconnected async callback Called when a client disconnects. Parameters: transport : The SmallWebRTCTransport instance webrtc_connection : The connection that was disconnected Copy Ask AI @webrtc_transport.event_handler ( "on_client_disconnected" ) async def on_client_disconnected ( transport , webrtc_connection ): print ( f "Client disconnected" ) ​ on_client_closed async callback Called when a client connection is closed. Parameters: transport : The SmallWebRTCTransport instance webrtc_connection : The connection that was closed Copy Ask AI @webrtc_transport.event_handler ( "on_client_closed" ) async def on_client_closed ( transport , webrtc_connection ): print ( f "Client connection closed" ) ​ Basic Usage This basic usage example shows the transport specific parts of a bot.py file required to configure your bot: Copy Ask AI from pipecat.audio.vad.silero import SileroVADAnalyzer from pipecat.pipeline.pipeline import Pipeline from pipecat.transports.base_transport import TransportParams from pipecat.transports.network.small_webrtc import SmallWebRTCTransport from pipecat.transports.network.webrtc_connection import SmallWebRTCConnection async def run_bot ( webrtc_connection ): # Create the WebRTC transport with the provided connection transport = SmallWebRTCTransport( webrtc_connection = webrtc_connection, params = TransportParams( audio_in_enabled = True , # Accept audio from the client audio_out_enabled = True , # Send audio to the client vad_analyzer = SileroVADAnalyzer(), ), ) # Set up your services and context # Create the pipeline pipeline = Pipeline([ transport.input(), # Receive audio from client stt, # Convert speech to text context_aggregator.user(), # Add user messages to context llm, # Process text with LLM tts, # Convert text to speech transport.output(), # Send audio responses to client context_aggregator.assistant(), # Add assistant responses to context ]) # Register event handlers @transport.event_handler ( "on_client_connected" ) async def on_client_connected ( transport , client ): logger.info( "Client connected" ) # Start the conversation when client connects await task.queue_frames([context_aggregator.user().get_context_frame()]) @transport.event_handler ( "on_client_disconnected" ) async def on_client_disconnected ( transport , client ): logger.info( "Client disconnected" ) @transport.event_handler ( "on_client_closed" ) async def on_client_closed ( transport , client ): logger.info( "Client closed" ) await task.cancel() ​ How to connect with SmallWebRTCTransport For a client/server connection, you have two options for how to connect the client to the server: Use a Pipecat client SDK with the SmallWebRTCTransport . See the Client SDK docs to get started. Using the WebRTC API directly. This is only recommended for advanced use cases where the Pipecat client SDKs don’t have an available transport. ​ Examples To see a complete implementation, check out the following examples: Video Transform Demonstrates real-time video processing using WebRTC transport Voice Agent Implements a voice assistant using WebRTC for audio communication ​ Media Handling ​ Audio Audio is processed in 20ms chunks by default. The transport handles audio format conversion and resampling as needed: Input audio is processed at 16kHz (mono) to be compatible with speech recognition services Output audio can be configured to match your application’s requirements, but it must be mono, 16-bit PCM audio ​ Video Video is streamed using RGB format by default. The transport provides: Frame conversion between different color formats (RGB, YUV, etc.) Configurable resolution and framerate ​ WebRTC ICE Servers Configuration When implementing WebRTC in your project, STUN (Session Traversal Utilities for NAT) and TURN (Traversal Using Relays around NAT) servers are usually needed in cases where users are behind routers or firewalls. In local networks (e.g., testing within the same home or office network), you usually don’t need to configure STUN or TURN servers. In such cases, WebRTC can often directly establish peer-to-peer connections without needing to traverse NAT or firewalls. ​ What are STUN and TURN Servers? STUN Server : Helps clients discover their public IP address and port when they’re behind a NAT (Network Address Translation) device (like a router). This allows WebRTC to attempt direct peer-to-peer communication by providing the public-facing IP and port. TURN Server : Used as a fallback when direct peer-to-peer communication isn’t possible due to strict NATs or firewalls blocking connections. The TURN server relays media traffic between peers. ​ Why are ICE Servers Important? ICE (Interactive Connectivity Establishment) is a framework used by WebRTC to handle network traversal and NAT issues. The iceServers configuration provides a list of STUN and TURN servers that WebRTC uses to find the best way to connect two peers. ​ Advanced Configuration ​ ICE Servers For better connectivity, especially when testing across different networks, you can provide STUN servers: Copy Ask AI webrtc_connection = SmallWebRTCConnection( ice_servers = [ "stun:stun.l.google.com:19302" , "stun:stun1.l.google.com:19302" ] ) You can also use IceServer objects for more advanced configuration: Copy Ask AI from pipecat.transports.network.webrtc_connection import IceServer webrtc_connection = SmallWebRTCConnection( ice_servers = [ IceServer( urls = "stun:stun.l.google.com:19302" ), IceServer( urls = "turn:turn.example.com:3478" , username = "username" , credential = "password" ) ] ) ​ Troubleshooting If clients have trouble connecting or streaming: Check browser console for WebRTC errors Ensure you’re using HTTPS in production (required for WebRTC) For testing across networks, consider using Daily which provides TURN servers Verify browser permissions for camera and microphone Daily WebRTC FastAPI WebSocket On this page Overview Installation Class Reference SmallWebRTCConnection Methods SmallWebRTCTransport Methods Event Handlers Basic Usage How to connect with SmallWebRTCTransport Examples Media Handling Audio Video WebRTC ICE Servers Configuration What are STUN and TURN Servers? Why are ICE Servers Important? Advanced Configuration ICE Servers Troubleshooting Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/transport_small-webrtc_8fa754ac.txt b/transport_small-webrtc_8fa754ac.txt
new file mode 100644
index 0000000000000000000000000000000000000000..14320d593ec35b481ee66136fd1c32d995b22ccd
--- /dev/null
+++ b/transport_small-webrtc_8fa754ac.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/transport/small-webrtc#installation
+Title: SmallWebRTCTransport - Pipecat
+==================================================
+
+SmallWebRTCTransport - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Transport SmallWebRTCTransport Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Daily WebRTC SmallWebRTCTransport FastAPI WebSocket WebSocket Server Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview SmallWebRTCTransport enables peer-to-peer WebRTC connections between clients and your Pipecat application. It implements bidirectional audio and video streaming using WebRTC for real-time communication. This transport is intended for lightweight implementations, particularly for local development and testing. It expects your clients to include a corresponding SmallWebRTCTransport implementation. See here for the JavaScript implementation. SmallWebRTCTransport is best used for testing and development. For production deployments with scale, consider using the DailyTransport , as it has global, low-latency infrastructure. ​ Installation To use SmallWebRTCTransport , install the required dependencies: Copy Ask AI pip install pipecat-ai[webrtc] ​ Class Reference ​ SmallWebRTCConnection SmallWebRTCConnection manages the WebRTC connection details, peer connection state, and ICE candidates. It handles the signaling process and media tracks. Copy Ask AI SmallWebRTCConnection( ice_servers = None ) ​ ice_servers Union[List[str], List[IceServer]] List of STUN/TURN server URLs for ICE connection establishment. Can be provided as strings or as IceServer objects. ​ Methods ​ initialize async method Initialize the connection with a client’s SDP offer. Parameters: sdp : String containing the Session Description Protocol data from client’s offer type : String representing the SDP message type (typically “offer”) Copy Ask AI await webrtc_connection.initialize( sdp = client_sdp, type = "offer" ) ​ connect async method Establish the WebRTC peer connection after initialization. Copy Ask AI await webrtc_connection.connect() ​ close async method Close the WebRTC peer connection. Copy Ask AI await webrtc_connection.close() ​ disconnect async method Disconnect the WebRTC peer connection and send a peer left message to the client. Copy Ask AI await webrtc_connection.disconnect() ​ renegotiate async method Handle connection renegotiation requests. Parameters: sdp : String containing the Session Description Protocol data for renegotiation type : String representing the SDP message type restart_pc : Boolean indicating whether to completely restart the peer connection (default: False) Copy Ask AI await webrtc_connection.renegotiate( sdp = new_sdp, type = "offer" , restart_pc = False ) ​ get_answer method Retrieve the SDP answer to send back to the client. Returns a dictionary with sdp , type , and pc_id fields. Copy Ask AI answer = webrtc_connection.get_answer() # Returns: {"sdp": "...", "type": "answer", "pc_id": "..."} ​ send_app_message method Send an application message to the client. Parameters: message : The message to send (will be JSON serialized) Copy Ask AI webrtc_connection.send_app_message({ "action" : "greeting" , "text" : "Hello!" }) ​ is_connected method Check if the connection is active. Copy Ask AI if webrtc_connection.is_connected(): print ( "Connection is active" ) ​ audio_input_track method Get the audio input track from the client. Copy Ask AI audio_track = webrtc_connection.audio_input_track() ​ video_input_track method Get the video input track from the client. Copy Ask AI video_track = webrtc_connection.video_input_track() ​ replace_audio_track method Replace the current audio track with a new one. Parameters: track : The new audio track to use Copy Ask AI webrtc_connection.replace_audio_track(new_audio_track) ​ replace_video_track method Replace the current video track with a new one. Parameters: track : The new video track to use Copy Ask AI webrtc_connection.replace_video_track(new_video_track) ​ ask_to_renegotiate method Request the client to initiate connection renegotiation. Copy Ask AI webrtc_connection.ask_to_renegotiate() ​ event_handler decorator Register an event handler for connection events. Events: "app-message" : Called when a message is received from the client "track-started" : Called when a new track is started "track-ended" : Called when a track ends "connecting" : Called when connection is being established "connected" : Called when connection is established "disconnected" : Called when connection is lost "closed" : Called when connection is closed "failed" : Called when connection fails "new" : Called when a new connection is created Copy Ask AI @webrtc_connection.event_handler ( "connected" ) async def on_connected ( connection ): print ( f "WebRTC connection established" ) ​ SmallWebRTCTransport SmallWebRTCTransport is the main transport class that manages both input and output transports for WebRTC communication. Copy Ask AI SmallWebRTCTransport( webrtc_connection: SmallWebRTCConnection, params: TransportParams, input_name: Optional[ str ] = None , output_name: Optional[ str ] = None ) ​ webrtc_connection SmallWebRTCConnection required An instance of SmallWebRTCConnection that manages the WebRTC connection ​ params TransportParams required Configuration parameters for the transport Show TransportParams properties ​ audio_in_enabled bool default: false Enable audio input from the WebRTC client ​ audio_in_passthrough bool default: "False" When enabled, incoming audio frames are pushed downstream ​ audio_out_enabled bool default: false Enable audio output to the WebRTC client ​ audio_in_sample_rate int Sample rate for incoming audio (Hz) ​ audio_out_sample_rate int Sample rate for outgoing audio (Hz) ​ audio_in_channels int default: 1 Number of audio input channels (1 for mono, 2 for stereo) ​ audio_out_channels int default: 1 Number of audio output channels (1 for mono, 2 for stereo) ​ video_in_enabled bool default: false Enable video input from the WebRTC client ​ video_out_enabled bool default: false Enable video output to the WebRTC client ​ video_out_width int default: 640 Width of outgoing video ​ video_out_height int default: 480 Height of outgoing video ​ video_out_framerate int default: 30 Framerate of outgoing video ​ vad_analyzer VADAnalyzer Custom VAD analyzer implementation ​ input_name str Optional name for the input transport ​ output_name str Optional name for the output transport ​ Methods ​ input method Returns the input transport instance. Copy Ask AI input_transport = webrtc_transport.input() ​ output method Returns the output transport instance. Copy Ask AI output_transport = webrtc_transport.output() ​ send_image async method Send an image frame to the client. Parameters: frame : The image frame to send (OutputImageRawFrame or SpriteFrame) Copy Ask AI await webrtc_transport.send_image(image_frame) ​ send_audio async method Send an audio frame to the client. Parameters: frame : The audio frame to send (OutputAudioRawFrame) Copy Ask AI await webrtc_transport.send_audio(audio_frame) ​ Event Handlers ​ on_app_message async callback Called when receiving application messages from the client. Parameters: message : The received message Copy Ask AI @webrtc_transport.event_handler ( "on_app_message" ) async def on_app_message ( message ): print ( f "Received message: { message } " ) ​ on_client_connected async callback Called when a client successfully connects. Parameters: transport : The SmallWebRTCTransport instance webrtc_connection : The connection that was established Copy Ask AI @webrtc_transport.event_handler ( "on_client_connected" ) async def on_client_connected ( transport , webrtc_connection ): print ( f "Client connected" ) ​ on_client_disconnected async callback Called when a client disconnects. Parameters: transport : The SmallWebRTCTransport instance webrtc_connection : The connection that was disconnected Copy Ask AI @webrtc_transport.event_handler ( "on_client_disconnected" ) async def on_client_disconnected ( transport , webrtc_connection ): print ( f "Client disconnected" ) ​ on_client_closed async callback Called when a client connection is closed. Parameters: transport : The SmallWebRTCTransport instance webrtc_connection : The connection that was closed Copy Ask AI @webrtc_transport.event_handler ( "on_client_closed" ) async def on_client_closed ( transport , webrtc_connection ): print ( f "Client connection closed" ) ​ Basic Usage This basic usage example shows the transport specific parts of a bot.py file required to configure your bot: Copy Ask AI from pipecat.audio.vad.silero import SileroVADAnalyzer from pipecat.pipeline.pipeline import Pipeline from pipecat.transports.base_transport import TransportParams from pipecat.transports.network.small_webrtc import SmallWebRTCTransport from pipecat.transports.network.webrtc_connection import SmallWebRTCConnection async def run_bot ( webrtc_connection ): # Create the WebRTC transport with the provided connection transport = SmallWebRTCTransport( webrtc_connection = webrtc_connection, params = TransportParams( audio_in_enabled = True , # Accept audio from the client audio_out_enabled = True , # Send audio to the client vad_analyzer = SileroVADAnalyzer(), ), ) # Set up your services and context # Create the pipeline pipeline = Pipeline([ transport.input(), # Receive audio from client stt, # Convert speech to text context_aggregator.user(), # Add user messages to context llm, # Process text with LLM tts, # Convert text to speech transport.output(), # Send audio responses to client context_aggregator.assistant(), # Add assistant responses to context ]) # Register event handlers @transport.event_handler ( "on_client_connected" ) async def on_client_connected ( transport , client ): logger.info( "Client connected" ) # Start the conversation when client connects await task.queue_frames([context_aggregator.user().get_context_frame()]) @transport.event_handler ( "on_client_disconnected" ) async def on_client_disconnected ( transport , client ): logger.info( "Client disconnected" ) @transport.event_handler ( "on_client_closed" ) async def on_client_closed ( transport , client ): logger.info( "Client closed" ) await task.cancel() ​ How to connect with SmallWebRTCTransport For a client/server connection, you have two options for how to connect the client to the server: Use a Pipecat client SDK with the SmallWebRTCTransport . See the Client SDK docs to get started. Using the WebRTC API directly. This is only recommended for advanced use cases where the Pipecat client SDKs don’t have an available transport. ​ Examples To see a complete implementation, check out the following examples: Video Transform Demonstrates real-time video processing using WebRTC transport Voice Agent Implements a voice assistant using WebRTC for audio communication ​ Media Handling ​ Audio Audio is processed in 20ms chunks by default. The transport handles audio format conversion and resampling as needed: Input audio is processed at 16kHz (mono) to be compatible with speech recognition services Output audio can be configured to match your application’s requirements, but it must be mono, 16-bit PCM audio ​ Video Video is streamed using RGB format by default. The transport provides: Frame conversion between different color formats (RGB, YUV, etc.) Configurable resolution and framerate ​ WebRTC ICE Servers Configuration When implementing WebRTC in your project, STUN (Session Traversal Utilities for NAT) and TURN (Traversal Using Relays around NAT) servers are usually needed in cases where users are behind routers or firewalls. In local networks (e.g., testing within the same home or office network), you usually don’t need to configure STUN or TURN servers. In such cases, WebRTC can often directly establish peer-to-peer connections without needing to traverse NAT or firewalls. ​ What are STUN and TURN Servers? STUN Server : Helps clients discover their public IP address and port when they’re behind a NAT (Network Address Translation) device (like a router). This allows WebRTC to attempt direct peer-to-peer communication by providing the public-facing IP and port. TURN Server : Used as a fallback when direct peer-to-peer communication isn’t possible due to strict NATs or firewalls blocking connections. The TURN server relays media traffic between peers. ​ Why are ICE Servers Important? ICE (Interactive Connectivity Establishment) is a framework used by WebRTC to handle network traversal and NAT issues. The iceServers configuration provides a list of STUN and TURN servers that WebRTC uses to find the best way to connect two peers. ​ Advanced Configuration ​ ICE Servers For better connectivity, especially when testing across different networks, you can provide STUN servers: Copy Ask AI webrtc_connection = SmallWebRTCConnection( ice_servers = [ "stun:stun.l.google.com:19302" , "stun:stun1.l.google.com:19302" ] ) You can also use IceServer objects for more advanced configuration: Copy Ask AI from pipecat.transports.network.webrtc_connection import IceServer webrtc_connection = SmallWebRTCConnection( ice_servers = [ IceServer( urls = "stun:stun.l.google.com:19302" ), IceServer( urls = "turn:turn.example.com:3478" , username = "username" , credential = "password" ) ] ) ​ Troubleshooting If clients have trouble connecting or streaming: Check browser console for WebRTC errors Ensure you’re using HTTPS in production (required for WebRTC) For testing across networks, consider using Daily which provides TURN servers Verify browser permissions for camera and microphone Daily WebRTC FastAPI WebSocket On this page Overview Installation Class Reference SmallWebRTCConnection Methods SmallWebRTCTransport Methods Event Handlers Basic Usage How to connect with SmallWebRTCTransport Examples Media Handling Audio Video WebRTC ICE Servers Configuration What are STUN and TURN Servers? Why are ICE Servers Important? Advanced Configuration ICE Servers Troubleshooting Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/transport_small-webrtc_90264ee8.txt b/transport_small-webrtc_90264ee8.txt
new file mode 100644
index 0000000000000000000000000000000000000000..f45f3bcb81f23c746b4d6f2f45d8f445b2d963c9
--- /dev/null
+++ b/transport_small-webrtc_90264ee8.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/transport/small-webrtc#methods-2
+Title: SmallWebRTCTransport - Pipecat
+==================================================
+
+SmallWebRTCTransport - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Transport SmallWebRTCTransport Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Daily WebRTC SmallWebRTCTransport FastAPI WebSocket WebSocket Server Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview SmallWebRTCTransport enables peer-to-peer WebRTC connections between clients and your Pipecat application. It implements bidirectional audio and video streaming using WebRTC for real-time communication. This transport is intended for lightweight implementations, particularly for local development and testing. It expects your clients to include a corresponding SmallWebRTCTransport implementation. See here for the JavaScript implementation. SmallWebRTCTransport is best used for testing and development. For production deployments with scale, consider using the DailyTransport , as it has global, low-latency infrastructure. ​ Installation To use SmallWebRTCTransport , install the required dependencies: Copy Ask AI pip install pipecat-ai[webrtc] ​ Class Reference ​ SmallWebRTCConnection SmallWebRTCConnection manages the WebRTC connection details, peer connection state, and ICE candidates. It handles the signaling process and media tracks. Copy Ask AI SmallWebRTCConnection( ice_servers = None ) ​ ice_servers Union[List[str], List[IceServer]] List of STUN/TURN server URLs for ICE connection establishment. Can be provided as strings or as IceServer objects. ​ Methods ​ initialize async method Initialize the connection with a client’s SDP offer. Parameters: sdp : String containing the Session Description Protocol data from client’s offer type : String representing the SDP message type (typically “offer”) Copy Ask AI await webrtc_connection.initialize( sdp = client_sdp, type = "offer" ) ​ connect async method Establish the WebRTC peer connection after initialization. Copy Ask AI await webrtc_connection.connect() ​ close async method Close the WebRTC peer connection. Copy Ask AI await webrtc_connection.close() ​ disconnect async method Disconnect the WebRTC peer connection and send a peer left message to the client. Copy Ask AI await webrtc_connection.disconnect() ​ renegotiate async method Handle connection renegotiation requests. Parameters: sdp : String containing the Session Description Protocol data for renegotiation type : String representing the SDP message type restart_pc : Boolean indicating whether to completely restart the peer connection (default: False) Copy Ask AI await webrtc_connection.renegotiate( sdp = new_sdp, type = "offer" , restart_pc = False ) ​ get_answer method Retrieve the SDP answer to send back to the client. Returns a dictionary with sdp , type , and pc_id fields. Copy Ask AI answer = webrtc_connection.get_answer() # Returns: {"sdp": "...", "type": "answer", "pc_id": "..."} ​ send_app_message method Send an application message to the client. Parameters: message : The message to send (will be JSON serialized) Copy Ask AI webrtc_connection.send_app_message({ "action" : "greeting" , "text" : "Hello!" }) ​ is_connected method Check if the connection is active. Copy Ask AI if webrtc_connection.is_connected(): print ( "Connection is active" ) ​ audio_input_track method Get the audio input track from the client. Copy Ask AI audio_track = webrtc_connection.audio_input_track() ​ video_input_track method Get the video input track from the client. Copy Ask AI video_track = webrtc_connection.video_input_track() ​ replace_audio_track method Replace the current audio track with a new one. Parameters: track : The new audio track to use Copy Ask AI webrtc_connection.replace_audio_track(new_audio_track) ​ replace_video_track method Replace the current video track with a new one. Parameters: track : The new video track to use Copy Ask AI webrtc_connection.replace_video_track(new_video_track) ​ ask_to_renegotiate method Request the client to initiate connection renegotiation. Copy Ask AI webrtc_connection.ask_to_renegotiate() ​ event_handler decorator Register an event handler for connection events. Events: "app-message" : Called when a message is received from the client "track-started" : Called when a new track is started "track-ended" : Called when a track ends "connecting" : Called when connection is being established "connected" : Called when connection is established "disconnected" : Called when connection is lost "closed" : Called when connection is closed "failed" : Called when connection fails "new" : Called when a new connection is created Copy Ask AI @webrtc_connection.event_handler ( "connected" ) async def on_connected ( connection ): print ( f "WebRTC connection established" ) ​ SmallWebRTCTransport SmallWebRTCTransport is the main transport class that manages both input and output transports for WebRTC communication. Copy Ask AI SmallWebRTCTransport( webrtc_connection: SmallWebRTCConnection, params: TransportParams, input_name: Optional[ str ] = None , output_name: Optional[ str ] = None ) ​ webrtc_connection SmallWebRTCConnection required An instance of SmallWebRTCConnection that manages the WebRTC connection ​ params TransportParams required Configuration parameters for the transport Show TransportParams properties ​ audio_in_enabled bool default: false Enable audio input from the WebRTC client ​ audio_in_passthrough bool default: "False" When enabled, incoming audio frames are pushed downstream ​ audio_out_enabled bool default: false Enable audio output to the WebRTC client ​ audio_in_sample_rate int Sample rate for incoming audio (Hz) ​ audio_out_sample_rate int Sample rate for outgoing audio (Hz) ​ audio_in_channels int default: 1 Number of audio input channels (1 for mono, 2 for stereo) ​ audio_out_channels int default: 1 Number of audio output channels (1 for mono, 2 for stereo) ​ video_in_enabled bool default: false Enable video input from the WebRTC client ​ video_out_enabled bool default: false Enable video output to the WebRTC client ​ video_out_width int default: 640 Width of outgoing video ​ video_out_height int default: 480 Height of outgoing video ​ video_out_framerate int default: 30 Framerate of outgoing video ​ vad_analyzer VADAnalyzer Custom VAD analyzer implementation ​ input_name str Optional name for the input transport ​ output_name str Optional name for the output transport ​ Methods ​ input method Returns the input transport instance. Copy Ask AI input_transport = webrtc_transport.input() ​ output method Returns the output transport instance. Copy Ask AI output_transport = webrtc_transport.output() ​ send_image async method Send an image frame to the client. Parameters: frame : The image frame to send (OutputImageRawFrame or SpriteFrame) Copy Ask AI await webrtc_transport.send_image(image_frame) ​ send_audio async method Send an audio frame to the client. Parameters: frame : The audio frame to send (OutputAudioRawFrame) Copy Ask AI await webrtc_transport.send_audio(audio_frame) ​ Event Handlers ​ on_app_message async callback Called when receiving application messages from the client. Parameters: message : The received message Copy Ask AI @webrtc_transport.event_handler ( "on_app_message" ) async def on_app_message ( message ): print ( f "Received message: { message } " ) ​ on_client_connected async callback Called when a client successfully connects. Parameters: transport : The SmallWebRTCTransport instance webrtc_connection : The connection that was established Copy Ask AI @webrtc_transport.event_handler ( "on_client_connected" ) async def on_client_connected ( transport , webrtc_connection ): print ( f "Client connected" ) ​ on_client_disconnected async callback Called when a client disconnects. Parameters: transport : The SmallWebRTCTransport instance webrtc_connection : The connection that was disconnected Copy Ask AI @webrtc_transport.event_handler ( "on_client_disconnected" ) async def on_client_disconnected ( transport , webrtc_connection ): print ( f "Client disconnected" ) ​ on_client_closed async callback Called when a client connection is closed. Parameters: transport : The SmallWebRTCTransport instance webrtc_connection : The connection that was closed Copy Ask AI @webrtc_transport.event_handler ( "on_client_closed" ) async def on_client_closed ( transport , webrtc_connection ): print ( f "Client connection closed" ) ​ Basic Usage This basic usage example shows the transport specific parts of a bot.py file required to configure your bot: Copy Ask AI from pipecat.audio.vad.silero import SileroVADAnalyzer from pipecat.pipeline.pipeline import Pipeline from pipecat.transports.base_transport import TransportParams from pipecat.transports.network.small_webrtc import SmallWebRTCTransport from pipecat.transports.network.webrtc_connection import SmallWebRTCConnection async def run_bot ( webrtc_connection ): # Create the WebRTC transport with the provided connection transport = SmallWebRTCTransport( webrtc_connection = webrtc_connection, params = TransportParams( audio_in_enabled = True , # Accept audio from the client audio_out_enabled = True , # Send audio to the client vad_analyzer = SileroVADAnalyzer(), ), ) # Set up your services and context # Create the pipeline pipeline = Pipeline([ transport.input(), # Receive audio from client stt, # Convert speech to text context_aggregator.user(), # Add user messages to context llm, # Process text with LLM tts, # Convert text to speech transport.output(), # Send audio responses to client context_aggregator.assistant(), # Add assistant responses to context ]) # Register event handlers @transport.event_handler ( "on_client_connected" ) async def on_client_connected ( transport , client ): logger.info( "Client connected" ) # Start the conversation when client connects await task.queue_frames([context_aggregator.user().get_context_frame()]) @transport.event_handler ( "on_client_disconnected" ) async def on_client_disconnected ( transport , client ): logger.info( "Client disconnected" ) @transport.event_handler ( "on_client_closed" ) async def on_client_closed ( transport , client ): logger.info( "Client closed" ) await task.cancel() ​ How to connect with SmallWebRTCTransport For a client/server connection, you have two options for how to connect the client to the server: Use a Pipecat client SDK with the SmallWebRTCTransport . See the Client SDK docs to get started. Using the WebRTC API directly. This is only recommended for advanced use cases where the Pipecat client SDKs don’t have an available transport. ​ Examples To see a complete implementation, check out the following examples: Video Transform Demonstrates real-time video processing using WebRTC transport Voice Agent Implements a voice assistant using WebRTC for audio communication ​ Media Handling ​ Audio Audio is processed in 20ms chunks by default. The transport handles audio format conversion and resampling as needed: Input audio is processed at 16kHz (mono) to be compatible with speech recognition services Output audio can be configured to match your application’s requirements, but it must be mono, 16-bit PCM audio ​ Video Video is streamed using RGB format by default. The transport provides: Frame conversion between different color formats (RGB, YUV, etc.) Configurable resolution and framerate ​ WebRTC ICE Servers Configuration When implementing WebRTC in your project, STUN (Session Traversal Utilities for NAT) and TURN (Traversal Using Relays around NAT) servers are usually needed in cases where users are behind routers or firewalls. In local networks (e.g., testing within the same home or office network), you usually don’t need to configure STUN or TURN servers. In such cases, WebRTC can often directly establish peer-to-peer connections without needing to traverse NAT or firewalls. ​ What are STUN and TURN Servers? STUN Server : Helps clients discover their public IP address and port when they’re behind a NAT (Network Address Translation) device (like a router). This allows WebRTC to attempt direct peer-to-peer communication by providing the public-facing IP and port. TURN Server : Used as a fallback when direct peer-to-peer communication isn’t possible due to strict NATs or firewalls blocking connections. The TURN server relays media traffic between peers. ​ Why are ICE Servers Important? ICE (Interactive Connectivity Establishment) is a framework used by WebRTC to handle network traversal and NAT issues. The iceServers configuration provides a list of STUN and TURN servers that WebRTC uses to find the best way to connect two peers. ​ Advanced Configuration ​ ICE Servers For better connectivity, especially when testing across different networks, you can provide STUN servers: Copy Ask AI webrtc_connection = SmallWebRTCConnection( ice_servers = [ "stun:stun.l.google.com:19302" , "stun:stun1.l.google.com:19302" ] ) You can also use IceServer objects for more advanced configuration: Copy Ask AI from pipecat.transports.network.webrtc_connection import IceServer webrtc_connection = SmallWebRTCConnection( ice_servers = [ IceServer( urls = "stun:stun.l.google.com:19302" ), IceServer( urls = "turn:turn.example.com:3478" , username = "username" , credential = "password" ) ] ) ​ Troubleshooting If clients have trouble connecting or streaming: Check browser console for WebRTC errors Ensure you’re using HTTPS in production (required for WebRTC) For testing across networks, consider using Daily which provides TURN servers Verify browser permissions for camera and microphone Daily WebRTC FastAPI WebSocket On this page Overview Installation Class Reference SmallWebRTCConnection Methods SmallWebRTCTransport Methods Event Handlers Basic Usage How to connect with SmallWebRTCTransport Examples Media Handling Audio Video WebRTC ICE Servers Configuration What are STUN and TURN Servers? Why are ICE Servers Important? Advanced Configuration ICE Servers Troubleshooting Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/transport_small-webrtc_a27bdf76.txt b/transport_small-webrtc_a27bdf76.txt
new file mode 100644
index 0000000000000000000000000000000000000000..a5a32f49fadc797ba52698dff93b9a1a909d94db
--- /dev/null
+++ b/transport_small-webrtc_a27bdf76.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/transport/small-webrtc#param-is-connected
+Title: SmallWebRTCTransport - Pipecat
+==================================================
+
+SmallWebRTCTransport - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Transport SmallWebRTCTransport Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Daily WebRTC SmallWebRTCTransport FastAPI WebSocket WebSocket Server Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview SmallWebRTCTransport enables peer-to-peer WebRTC connections between clients and your Pipecat application. It implements bidirectional audio and video streaming using WebRTC for real-time communication. This transport is intended for lightweight implementations, particularly for local development and testing. It expects your clients to include a corresponding SmallWebRTCTransport implementation. See here for the JavaScript implementation. SmallWebRTCTransport is best used for testing and development. For production deployments with scale, consider using the DailyTransport , as it has global, low-latency infrastructure. ​ Installation To use SmallWebRTCTransport , install the required dependencies: Copy Ask AI pip install pipecat-ai[webrtc] ​ Class Reference ​ SmallWebRTCConnection SmallWebRTCConnection manages the WebRTC connection details, peer connection state, and ICE candidates. It handles the signaling process and media tracks. Copy Ask AI SmallWebRTCConnection( ice_servers = None ) ​ ice_servers Union[List[str], List[IceServer]] List of STUN/TURN server URLs for ICE connection establishment. Can be provided as strings or as IceServer objects. ​ Methods ​ initialize async method Initialize the connection with a client’s SDP offer. Parameters: sdp : String containing the Session Description Protocol data from client’s offer type : String representing the SDP message type (typically “offer”) Copy Ask AI await webrtc_connection.initialize( sdp = client_sdp, type = "offer" ) ​ connect async method Establish the WebRTC peer connection after initialization. Copy Ask AI await webrtc_connection.connect() ​ close async method Close the WebRTC peer connection. Copy Ask AI await webrtc_connection.close() ​ disconnect async method Disconnect the WebRTC peer connection and send a peer left message to the client. Copy Ask AI await webrtc_connection.disconnect() ​ renegotiate async method Handle connection renegotiation requests. Parameters: sdp : String containing the Session Description Protocol data for renegotiation type : String representing the SDP message type restart_pc : Boolean indicating whether to completely restart the peer connection (default: False) Copy Ask AI await webrtc_connection.renegotiate( sdp = new_sdp, type = "offer" , restart_pc = False ) ​ get_answer method Retrieve the SDP answer to send back to the client. Returns a dictionary with sdp , type , and pc_id fields. Copy Ask AI answer = webrtc_connection.get_answer() # Returns: {"sdp": "...", "type": "answer", "pc_id": "..."} ​ send_app_message method Send an application message to the client. Parameters: message : The message to send (will be JSON serialized) Copy Ask AI webrtc_connection.send_app_message({ "action" : "greeting" , "text" : "Hello!" }) ​ is_connected method Check if the connection is active. Copy Ask AI if webrtc_connection.is_connected(): print ( "Connection is active" ) ​ audio_input_track method Get the audio input track from the client. Copy Ask AI audio_track = webrtc_connection.audio_input_track() ​ video_input_track method Get the video input track from the client. Copy Ask AI video_track = webrtc_connection.video_input_track() ​ replace_audio_track method Replace the current audio track with a new one. Parameters: track : The new audio track to use Copy Ask AI webrtc_connection.replace_audio_track(new_audio_track) ​ replace_video_track method Replace the current video track with a new one. Parameters: track : The new video track to use Copy Ask AI webrtc_connection.replace_video_track(new_video_track) ​ ask_to_renegotiate method Request the client to initiate connection renegotiation. Copy Ask AI webrtc_connection.ask_to_renegotiate() ​ event_handler decorator Register an event handler for connection events. Events: "app-message" : Called when a message is received from the client "track-started" : Called when a new track is started "track-ended" : Called when a track ends "connecting" : Called when connection is being established "connected" : Called when connection is established "disconnected" : Called when connection is lost "closed" : Called when connection is closed "failed" : Called when connection fails "new" : Called when a new connection is created Copy Ask AI @webrtc_connection.event_handler ( "connected" ) async def on_connected ( connection ): print ( f "WebRTC connection established" ) ​ SmallWebRTCTransport SmallWebRTCTransport is the main transport class that manages both input and output transports for WebRTC communication. Copy Ask AI SmallWebRTCTransport( webrtc_connection: SmallWebRTCConnection, params: TransportParams, input_name: Optional[ str ] = None , output_name: Optional[ str ] = None ) ​ webrtc_connection SmallWebRTCConnection required An instance of SmallWebRTCConnection that manages the WebRTC connection ​ params TransportParams required Configuration parameters for the transport Show TransportParams properties ​ audio_in_enabled bool default: false Enable audio input from the WebRTC client ​ audio_in_passthrough bool default: "False" When enabled, incoming audio frames are pushed downstream ​ audio_out_enabled bool default: false Enable audio output to the WebRTC client ​ audio_in_sample_rate int Sample rate for incoming audio (Hz) ​ audio_out_sample_rate int Sample rate for outgoing audio (Hz) ​ audio_in_channels int default: 1 Number of audio input channels (1 for mono, 2 for stereo) ​ audio_out_channels int default: 1 Number of audio output channels (1 for mono, 2 for stereo) ​ video_in_enabled bool default: false Enable video input from the WebRTC client ​ video_out_enabled bool default: false Enable video output to the WebRTC client ​ video_out_width int default: 640 Width of outgoing video ​ video_out_height int default: 480 Height of outgoing video ​ video_out_framerate int default: 30 Framerate of outgoing video ​ vad_analyzer VADAnalyzer Custom VAD analyzer implementation ​ input_name str Optional name for the input transport ​ output_name str Optional name for the output transport ​ Methods ​ input method Returns the input transport instance. Copy Ask AI input_transport = webrtc_transport.input() ​ output method Returns the output transport instance. Copy Ask AI output_transport = webrtc_transport.output() ​ send_image async method Send an image frame to the client. Parameters: frame : The image frame to send (OutputImageRawFrame or SpriteFrame) Copy Ask AI await webrtc_transport.send_image(image_frame) ​ send_audio async method Send an audio frame to the client. Parameters: frame : The audio frame to send (OutputAudioRawFrame) Copy Ask AI await webrtc_transport.send_audio(audio_frame) ​ Event Handlers ​ on_app_message async callback Called when receiving application messages from the client. Parameters: message : The received message Copy Ask AI @webrtc_transport.event_handler ( "on_app_message" ) async def on_app_message ( message ): print ( f "Received message: { message } " ) ​ on_client_connected async callback Called when a client successfully connects. Parameters: transport : The SmallWebRTCTransport instance webrtc_connection : The connection that was established Copy Ask AI @webrtc_transport.event_handler ( "on_client_connected" ) async def on_client_connected ( transport , webrtc_connection ): print ( f "Client connected" ) ​ on_client_disconnected async callback Called when a client disconnects. Parameters: transport : The SmallWebRTCTransport instance webrtc_connection : The connection that was disconnected Copy Ask AI @webrtc_transport.event_handler ( "on_client_disconnected" ) async def on_client_disconnected ( transport , webrtc_connection ): print ( f "Client disconnected" ) ​ on_client_closed async callback Called when a client connection is closed. Parameters: transport : The SmallWebRTCTransport instance webrtc_connection : The connection that was closed Copy Ask AI @webrtc_transport.event_handler ( "on_client_closed" ) async def on_client_closed ( transport , webrtc_connection ): print ( f "Client connection closed" ) ​ Basic Usage This basic usage example shows the transport specific parts of a bot.py file required to configure your bot: Copy Ask AI from pipecat.audio.vad.silero import SileroVADAnalyzer from pipecat.pipeline.pipeline import Pipeline from pipecat.transports.base_transport import TransportParams from pipecat.transports.network.small_webrtc import SmallWebRTCTransport from pipecat.transports.network.webrtc_connection import SmallWebRTCConnection async def run_bot ( webrtc_connection ): # Create the WebRTC transport with the provided connection transport = SmallWebRTCTransport( webrtc_connection = webrtc_connection, params = TransportParams( audio_in_enabled = True , # Accept audio from the client audio_out_enabled = True , # Send audio to the client vad_analyzer = SileroVADAnalyzer(), ), ) # Set up your services and context # Create the pipeline pipeline = Pipeline([ transport.input(), # Receive audio from client stt, # Convert speech to text context_aggregator.user(), # Add user messages to context llm, # Process text with LLM tts, # Convert text to speech transport.output(), # Send audio responses to client context_aggregator.assistant(), # Add assistant responses to context ]) # Register event handlers @transport.event_handler ( "on_client_connected" ) async def on_client_connected ( transport , client ): logger.info( "Client connected" ) # Start the conversation when client connects await task.queue_frames([context_aggregator.user().get_context_frame()]) @transport.event_handler ( "on_client_disconnected" ) async def on_client_disconnected ( transport , client ): logger.info( "Client disconnected" ) @transport.event_handler ( "on_client_closed" ) async def on_client_closed ( transport , client ): logger.info( "Client closed" ) await task.cancel() ​ How to connect with SmallWebRTCTransport For a client/server connection, you have two options for how to connect the client to the server: Use a Pipecat client SDK with the SmallWebRTCTransport . See the Client SDK docs to get started. Using the WebRTC API directly. This is only recommended for advanced use cases where the Pipecat client SDKs don’t have an available transport. ​ Examples To see a complete implementation, check out the following examples: Video Transform Demonstrates real-time video processing using WebRTC transport Voice Agent Implements a voice assistant using WebRTC for audio communication ​ Media Handling ​ Audio Audio is processed in 20ms chunks by default. The transport handles audio format conversion and resampling as needed: Input audio is processed at 16kHz (mono) to be compatible with speech recognition services Output audio can be configured to match your application’s requirements, but it must be mono, 16-bit PCM audio ​ Video Video is streamed using RGB format by default. The transport provides: Frame conversion between different color formats (RGB, YUV, etc.) Configurable resolution and framerate ​ WebRTC ICE Servers Configuration When implementing WebRTC in your project, STUN (Session Traversal Utilities for NAT) and TURN (Traversal Using Relays around NAT) servers are usually needed in cases where users are behind routers or firewalls. In local networks (e.g., testing within the same home or office network), you usually don’t need to configure STUN or TURN servers. In such cases, WebRTC can often directly establish peer-to-peer connections without needing to traverse NAT or firewalls. ​ What are STUN and TURN Servers? STUN Server : Helps clients discover their public IP address and port when they’re behind a NAT (Network Address Translation) device (like a router). This allows WebRTC to attempt direct peer-to-peer communication by providing the public-facing IP and port. TURN Server : Used as a fallback when direct peer-to-peer communication isn’t possible due to strict NATs or firewalls blocking connections. The TURN server relays media traffic between peers. ​ Why are ICE Servers Important? ICE (Interactive Connectivity Establishment) is a framework used by WebRTC to handle network traversal and NAT issues. The iceServers configuration provides a list of STUN and TURN servers that WebRTC uses to find the best way to connect two peers. ​ Advanced Configuration ​ ICE Servers For better connectivity, especially when testing across different networks, you can provide STUN servers: Copy Ask AI webrtc_connection = SmallWebRTCConnection( ice_servers = [ "stun:stun.l.google.com:19302" , "stun:stun1.l.google.com:19302" ] ) You can also use IceServer objects for more advanced configuration: Copy Ask AI from pipecat.transports.network.webrtc_connection import IceServer webrtc_connection = SmallWebRTCConnection( ice_servers = [ IceServer( urls = "stun:stun.l.google.com:19302" ), IceServer( urls = "turn:turn.example.com:3478" , username = "username" , credential = "password" ) ] ) ​ Troubleshooting If clients have trouble connecting or streaming: Check browser console for WebRTC errors Ensure you’re using HTTPS in production (required for WebRTC) For testing across networks, consider using Daily which provides TURN servers Verify browser permissions for camera and microphone Daily WebRTC FastAPI WebSocket On this page Overview Installation Class Reference SmallWebRTCConnection Methods SmallWebRTCTransport Methods Event Handlers Basic Usage How to connect with SmallWebRTCTransport Examples Media Handling Audio Video WebRTC ICE Servers Configuration What are STUN and TURN Servers? Why are ICE Servers Important? Advanced Configuration ICE Servers Troubleshooting Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/transport_small-webrtc_a830fea2.txt b/transport_small-webrtc_a830fea2.txt
new file mode 100644
index 0000000000000000000000000000000000000000..4e1402d060bbc4738772624c73f757b452d2537f
--- /dev/null
+++ b/transport_small-webrtc_a830fea2.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/transport/small-webrtc#param-ask-to-renegotiate
+Title: SmallWebRTCTransport - Pipecat
+==================================================
+
+SmallWebRTCTransport - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Transport SmallWebRTCTransport Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Daily WebRTC SmallWebRTCTransport FastAPI WebSocket WebSocket Server Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview SmallWebRTCTransport enables peer-to-peer WebRTC connections between clients and your Pipecat application. It implements bidirectional audio and video streaming using WebRTC for real-time communication. This transport is intended for lightweight implementations, particularly for local development and testing. It expects your clients to include a corresponding SmallWebRTCTransport implementation. See here for the JavaScript implementation. SmallWebRTCTransport is best used for testing and development. For production deployments with scale, consider using the DailyTransport , as it has global, low-latency infrastructure. ​ Installation To use SmallWebRTCTransport , install the required dependencies: Copy Ask AI pip install pipecat-ai[webrtc] ​ Class Reference ​ SmallWebRTCConnection SmallWebRTCConnection manages the WebRTC connection details, peer connection state, and ICE candidates. It handles the signaling process and media tracks. Copy Ask AI SmallWebRTCConnection( ice_servers = None ) ​ ice_servers Union[List[str], List[IceServer]] List of STUN/TURN server URLs for ICE connection establishment. Can be provided as strings or as IceServer objects. ​ Methods ​ initialize async method Initialize the connection with a client’s SDP offer. Parameters: sdp : String containing the Session Description Protocol data from client’s offer type : String representing the SDP message type (typically “offer”) Copy Ask AI await webrtc_connection.initialize( sdp = client_sdp, type = "offer" ) ​ connect async method Establish the WebRTC peer connection after initialization. Copy Ask AI await webrtc_connection.connect() ​ close async method Close the WebRTC peer connection. Copy Ask AI await webrtc_connection.close() ​ disconnect async method Disconnect the WebRTC peer connection and send a peer left message to the client. Copy Ask AI await webrtc_connection.disconnect() ​ renegotiate async method Handle connection renegotiation requests. Parameters: sdp : String containing the Session Description Protocol data for renegotiation type : String representing the SDP message type restart_pc : Boolean indicating whether to completely restart the peer connection (default: False) Copy Ask AI await webrtc_connection.renegotiate( sdp = new_sdp, type = "offer" , restart_pc = False ) ​ get_answer method Retrieve the SDP answer to send back to the client. Returns a dictionary with sdp , type , and pc_id fields. Copy Ask AI answer = webrtc_connection.get_answer() # Returns: {"sdp": "...", "type": "answer", "pc_id": "..."} ​ send_app_message method Send an application message to the client. Parameters: message : The message to send (will be JSON serialized) Copy Ask AI webrtc_connection.send_app_message({ "action" : "greeting" , "text" : "Hello!" }) ​ is_connected method Check if the connection is active. Copy Ask AI if webrtc_connection.is_connected(): print ( "Connection is active" ) ​ audio_input_track method Get the audio input track from the client. Copy Ask AI audio_track = webrtc_connection.audio_input_track() ​ video_input_track method Get the video input track from the client. Copy Ask AI video_track = webrtc_connection.video_input_track() ​ replace_audio_track method Replace the current audio track with a new one. Parameters: track : The new audio track to use Copy Ask AI webrtc_connection.replace_audio_track(new_audio_track) ​ replace_video_track method Replace the current video track with a new one. Parameters: track : The new video track to use Copy Ask AI webrtc_connection.replace_video_track(new_video_track) ​ ask_to_renegotiate method Request the client to initiate connection renegotiation. Copy Ask AI webrtc_connection.ask_to_renegotiate() ​ event_handler decorator Register an event handler for connection events. Events: "app-message" : Called when a message is received from the client "track-started" : Called when a new track is started "track-ended" : Called when a track ends "connecting" : Called when connection is being established "connected" : Called when connection is established "disconnected" : Called when connection is lost "closed" : Called when connection is closed "failed" : Called when connection fails "new" : Called when a new connection is created Copy Ask AI @webrtc_connection.event_handler ( "connected" ) async def on_connected ( connection ): print ( f "WebRTC connection established" ) ​ SmallWebRTCTransport SmallWebRTCTransport is the main transport class that manages both input and output transports for WebRTC communication. Copy Ask AI SmallWebRTCTransport( webrtc_connection: SmallWebRTCConnection, params: TransportParams, input_name: Optional[ str ] = None , output_name: Optional[ str ] = None ) ​ webrtc_connection SmallWebRTCConnection required An instance of SmallWebRTCConnection that manages the WebRTC connection ​ params TransportParams required Configuration parameters for the transport Show TransportParams properties ​ audio_in_enabled bool default: false Enable audio input from the WebRTC client ​ audio_in_passthrough bool default: "False" When enabled, incoming audio frames are pushed downstream ​ audio_out_enabled bool default: false Enable audio output to the WebRTC client ​ audio_in_sample_rate int Sample rate for incoming audio (Hz) ​ audio_out_sample_rate int Sample rate for outgoing audio (Hz) ​ audio_in_channels int default: 1 Number of audio input channels (1 for mono, 2 for stereo) ​ audio_out_channels int default: 1 Number of audio output channels (1 for mono, 2 for stereo) ​ video_in_enabled bool default: false Enable video input from the WebRTC client ​ video_out_enabled bool default: false Enable video output to the WebRTC client ​ video_out_width int default: 640 Width of outgoing video ​ video_out_height int default: 480 Height of outgoing video ​ video_out_framerate int default: 30 Framerate of outgoing video ​ vad_analyzer VADAnalyzer Custom VAD analyzer implementation ​ input_name str Optional name for the input transport ​ output_name str Optional name for the output transport ​ Methods ​ input method Returns the input transport instance. Copy Ask AI input_transport = webrtc_transport.input() ​ output method Returns the output transport instance. Copy Ask AI output_transport = webrtc_transport.output() ​ send_image async method Send an image frame to the client. Parameters: frame : The image frame to send (OutputImageRawFrame or SpriteFrame) Copy Ask AI await webrtc_transport.send_image(image_frame) ​ send_audio async method Send an audio frame to the client. Parameters: frame : The audio frame to send (OutputAudioRawFrame) Copy Ask AI await webrtc_transport.send_audio(audio_frame) ​ Event Handlers ​ on_app_message async callback Called when receiving application messages from the client. Parameters: message : The received message Copy Ask AI @webrtc_transport.event_handler ( "on_app_message" ) async def on_app_message ( message ): print ( f "Received message: { message } " ) ​ on_client_connected async callback Called when a client successfully connects. Parameters: transport : The SmallWebRTCTransport instance webrtc_connection : The connection that was established Copy Ask AI @webrtc_transport.event_handler ( "on_client_connected" ) async def on_client_connected ( transport , webrtc_connection ): print ( f "Client connected" ) ​ on_client_disconnected async callback Called when a client disconnects. Parameters: transport : The SmallWebRTCTransport instance webrtc_connection : The connection that was disconnected Copy Ask AI @webrtc_transport.event_handler ( "on_client_disconnected" ) async def on_client_disconnected ( transport , webrtc_connection ): print ( f "Client disconnected" ) ​ on_client_closed async callback Called when a client connection is closed. Parameters: transport : The SmallWebRTCTransport instance webrtc_connection : The connection that was closed Copy Ask AI @webrtc_transport.event_handler ( "on_client_closed" ) async def on_client_closed ( transport , webrtc_connection ): print ( f "Client connection closed" ) ​ Basic Usage This basic usage example shows the transport specific parts of a bot.py file required to configure your bot: Copy Ask AI from pipecat.audio.vad.silero import SileroVADAnalyzer from pipecat.pipeline.pipeline import Pipeline from pipecat.transports.base_transport import TransportParams from pipecat.transports.network.small_webrtc import SmallWebRTCTransport from pipecat.transports.network.webrtc_connection import SmallWebRTCConnection async def run_bot ( webrtc_connection ): # Create the WebRTC transport with the provided connection transport = SmallWebRTCTransport( webrtc_connection = webrtc_connection, params = TransportParams( audio_in_enabled = True , # Accept audio from the client audio_out_enabled = True , # Send audio to the client vad_analyzer = SileroVADAnalyzer(), ), ) # Set up your services and context # Create the pipeline pipeline = Pipeline([ transport.input(), # Receive audio from client stt, # Convert speech to text context_aggregator.user(), # Add user messages to context llm, # Process text with LLM tts, # Convert text to speech transport.output(), # Send audio responses to client context_aggregator.assistant(), # Add assistant responses to context ]) # Register event handlers @transport.event_handler ( "on_client_connected" ) async def on_client_connected ( transport , client ): logger.info( "Client connected" ) # Start the conversation when client connects await task.queue_frames([context_aggregator.user().get_context_frame()]) @transport.event_handler ( "on_client_disconnected" ) async def on_client_disconnected ( transport , client ): logger.info( "Client disconnected" ) @transport.event_handler ( "on_client_closed" ) async def on_client_closed ( transport , client ): logger.info( "Client closed" ) await task.cancel() ​ How to connect with SmallWebRTCTransport For a client/server connection, you have two options for how to connect the client to the server: Use a Pipecat client SDK with the SmallWebRTCTransport . See the Client SDK docs to get started. Using the WebRTC API directly. This is only recommended for advanced use cases where the Pipecat client SDKs don’t have an available transport. ​ Examples To see a complete implementation, check out the following examples: Video Transform Demonstrates real-time video processing using WebRTC transport Voice Agent Implements a voice assistant using WebRTC for audio communication ​ Media Handling ​ Audio Audio is processed in 20ms chunks by default. The transport handles audio format conversion and resampling as needed: Input audio is processed at 16kHz (mono) to be compatible with speech recognition services Output audio can be configured to match your application’s requirements, but it must be mono, 16-bit PCM audio ​ Video Video is streamed using RGB format by default. The transport provides: Frame conversion between different color formats (RGB, YUV, etc.) Configurable resolution and framerate ​ WebRTC ICE Servers Configuration When implementing WebRTC in your project, STUN (Session Traversal Utilities for NAT) and TURN (Traversal Using Relays around NAT) servers are usually needed in cases where users are behind routers or firewalls. In local networks (e.g., testing within the same home or office network), you usually don’t need to configure STUN or TURN servers. In such cases, WebRTC can often directly establish peer-to-peer connections without needing to traverse NAT or firewalls. ​ What are STUN and TURN Servers? STUN Server : Helps clients discover their public IP address and port when they’re behind a NAT (Network Address Translation) device (like a router). This allows WebRTC to attempt direct peer-to-peer communication by providing the public-facing IP and port. TURN Server : Used as a fallback when direct peer-to-peer communication isn’t possible due to strict NATs or firewalls blocking connections. The TURN server relays media traffic between peers. ​ Why are ICE Servers Important? ICE (Interactive Connectivity Establishment) is a framework used by WebRTC to handle network traversal and NAT issues. The iceServers configuration provides a list of STUN and TURN servers that WebRTC uses to find the best way to connect two peers. ​ Advanced Configuration ​ ICE Servers For better connectivity, especially when testing across different networks, you can provide STUN servers: Copy Ask AI webrtc_connection = SmallWebRTCConnection( ice_servers = [ "stun:stun.l.google.com:19302" , "stun:stun1.l.google.com:19302" ] ) You can also use IceServer objects for more advanced configuration: Copy Ask AI from pipecat.transports.network.webrtc_connection import IceServer webrtc_connection = SmallWebRTCConnection( ice_servers = [ IceServer( urls = "stun:stun.l.google.com:19302" ), IceServer( urls = "turn:turn.example.com:3478" , username = "username" , credential = "password" ) ] ) ​ Troubleshooting If clients have trouble connecting or streaming: Check browser console for WebRTC errors Ensure you’re using HTTPS in production (required for WebRTC) For testing across networks, consider using Daily which provides TURN servers Verify browser permissions for camera and microphone Daily WebRTC FastAPI WebSocket On this page Overview Installation Class Reference SmallWebRTCConnection Methods SmallWebRTCTransport Methods Event Handlers Basic Usage How to connect with SmallWebRTCTransport Examples Media Handling Audio Video WebRTC ICE Servers Configuration What are STUN and TURN Servers? Why are ICE Servers Important? Advanced Configuration ICE Servers Troubleshooting Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/transport_small-webrtc_b4239ec3.txt b/transport_small-webrtc_b4239ec3.txt
new file mode 100644
index 0000000000000000000000000000000000000000..7d528ab61eb2d7c25c7f8fffad74c295a62347e9
--- /dev/null
+++ b/transport_small-webrtc_b4239ec3.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/transport/small-webrtc#webrtc-ice-servers-configuration
+Title: SmallWebRTCTransport - Pipecat
+==================================================
+
+SmallWebRTCTransport - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Transport SmallWebRTCTransport Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Daily WebRTC SmallWebRTCTransport FastAPI WebSocket WebSocket Server Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview SmallWebRTCTransport enables peer-to-peer WebRTC connections between clients and your Pipecat application. It implements bidirectional audio and video streaming using WebRTC for real-time communication. This transport is intended for lightweight implementations, particularly for local development and testing. It expects your clients to include a corresponding SmallWebRTCTransport implementation. See here for the JavaScript implementation. SmallWebRTCTransport is best used for testing and development. For production deployments with scale, consider using the DailyTransport , as it has global, low-latency infrastructure. ​ Installation To use SmallWebRTCTransport , install the required dependencies: Copy Ask AI pip install pipecat-ai[webrtc] ​ Class Reference ​ SmallWebRTCConnection SmallWebRTCConnection manages the WebRTC connection details, peer connection state, and ICE candidates. It handles the signaling process and media tracks. Copy Ask AI SmallWebRTCConnection( ice_servers = None ) ​ ice_servers Union[List[str], List[IceServer]] List of STUN/TURN server URLs for ICE connection establishment. Can be provided as strings or as IceServer objects. ​ Methods ​ initialize async method Initialize the connection with a client’s SDP offer. Parameters: sdp : String containing the Session Description Protocol data from client’s offer type : String representing the SDP message type (typically “offer”) Copy Ask AI await webrtc_connection.initialize( sdp = client_sdp, type = "offer" ) ​ connect async method Establish the WebRTC peer connection after initialization. Copy Ask AI await webrtc_connection.connect() ​ close async method Close the WebRTC peer connection. Copy Ask AI await webrtc_connection.close() ​ disconnect async method Disconnect the WebRTC peer connection and send a peer left message to the client. Copy Ask AI await webrtc_connection.disconnect() ​ renegotiate async method Handle connection renegotiation requests. Parameters: sdp : String containing the Session Description Protocol data for renegotiation type : String representing the SDP message type restart_pc : Boolean indicating whether to completely restart the peer connection (default: False) Copy Ask AI await webrtc_connection.renegotiate( sdp = new_sdp, type = "offer" , restart_pc = False ) ​ get_answer method Retrieve the SDP answer to send back to the client. Returns a dictionary with sdp , type , and pc_id fields. Copy Ask AI answer = webrtc_connection.get_answer() # Returns: {"sdp": "...", "type": "answer", "pc_id": "..."} ​ send_app_message method Send an application message to the client. Parameters: message : The message to send (will be JSON serialized) Copy Ask AI webrtc_connection.send_app_message({ "action" : "greeting" , "text" : "Hello!" }) ​ is_connected method Check if the connection is active. Copy Ask AI if webrtc_connection.is_connected(): print ( "Connection is active" ) ​ audio_input_track method Get the audio input track from the client. Copy Ask AI audio_track = webrtc_connection.audio_input_track() ​ video_input_track method Get the video input track from the client. Copy Ask AI video_track = webrtc_connection.video_input_track() ​ replace_audio_track method Replace the current audio track with a new one. Parameters: track : The new audio track to use Copy Ask AI webrtc_connection.replace_audio_track(new_audio_track) ​ replace_video_track method Replace the current video track with a new one. Parameters: track : The new video track to use Copy Ask AI webrtc_connection.replace_video_track(new_video_track) ​ ask_to_renegotiate method Request the client to initiate connection renegotiation. Copy Ask AI webrtc_connection.ask_to_renegotiate() ​ event_handler decorator Register an event handler for connection events. Events: "app-message" : Called when a message is received from the client "track-started" : Called when a new track is started "track-ended" : Called when a track ends "connecting" : Called when connection is being established "connected" : Called when connection is established "disconnected" : Called when connection is lost "closed" : Called when connection is closed "failed" : Called when connection fails "new" : Called when a new connection is created Copy Ask AI @webrtc_connection.event_handler ( "connected" ) async def on_connected ( connection ): print ( f "WebRTC connection established" ) ​ SmallWebRTCTransport SmallWebRTCTransport is the main transport class that manages both input and output transports for WebRTC communication. Copy Ask AI SmallWebRTCTransport( webrtc_connection: SmallWebRTCConnection, params: TransportParams, input_name: Optional[ str ] = None , output_name: Optional[ str ] = None ) ​ webrtc_connection SmallWebRTCConnection required An instance of SmallWebRTCConnection that manages the WebRTC connection ​ params TransportParams required Configuration parameters for the transport Show TransportParams properties ​ audio_in_enabled bool default: false Enable audio input from the WebRTC client ​ audio_in_passthrough bool default: "False" When enabled, incoming audio frames are pushed downstream ​ audio_out_enabled bool default: false Enable audio output to the WebRTC client ​ audio_in_sample_rate int Sample rate for incoming audio (Hz) ​ audio_out_sample_rate int Sample rate for outgoing audio (Hz) ​ audio_in_channels int default: 1 Number of audio input channels (1 for mono, 2 for stereo) ​ audio_out_channels int default: 1 Number of audio output channels (1 for mono, 2 for stereo) ​ video_in_enabled bool default: false Enable video input from the WebRTC client ​ video_out_enabled bool default: false Enable video output to the WebRTC client ​ video_out_width int default: 640 Width of outgoing video ​ video_out_height int default: 480 Height of outgoing video ​ video_out_framerate int default: 30 Framerate of outgoing video ​ vad_analyzer VADAnalyzer Custom VAD analyzer implementation ​ input_name str Optional name for the input transport ​ output_name str Optional name for the output transport ​ Methods ​ input method Returns the input transport instance. Copy Ask AI input_transport = webrtc_transport.input() ​ output method Returns the output transport instance. Copy Ask AI output_transport = webrtc_transport.output() ​ send_image async method Send an image frame to the client. Parameters: frame : The image frame to send (OutputImageRawFrame or SpriteFrame) Copy Ask AI await webrtc_transport.send_image(image_frame) ​ send_audio async method Send an audio frame to the client. Parameters: frame : The audio frame to send (OutputAudioRawFrame) Copy Ask AI await webrtc_transport.send_audio(audio_frame) ​ Event Handlers ​ on_app_message async callback Called when receiving application messages from the client. Parameters: message : The received message Copy Ask AI @webrtc_transport.event_handler ( "on_app_message" ) async def on_app_message ( message ): print ( f "Received message: { message } " ) ​ on_client_connected async callback Called when a client successfully connects. Parameters: transport : The SmallWebRTCTransport instance webrtc_connection : The connection that was established Copy Ask AI @webrtc_transport.event_handler ( "on_client_connected" ) async def on_client_connected ( transport , webrtc_connection ): print ( f "Client connected" ) ​ on_client_disconnected async callback Called when a client disconnects. Parameters: transport : The SmallWebRTCTransport instance webrtc_connection : The connection that was disconnected Copy Ask AI @webrtc_transport.event_handler ( "on_client_disconnected" ) async def on_client_disconnected ( transport , webrtc_connection ): print ( f "Client disconnected" ) ​ on_client_closed async callback Called when a client connection is closed. Parameters: transport : The SmallWebRTCTransport instance webrtc_connection : The connection that was closed Copy Ask AI @webrtc_transport.event_handler ( "on_client_closed" ) async def on_client_closed ( transport , webrtc_connection ): print ( f "Client connection closed" ) ​ Basic Usage This basic usage example shows the transport specific parts of a bot.py file required to configure your bot: Copy Ask AI from pipecat.audio.vad.silero import SileroVADAnalyzer from pipecat.pipeline.pipeline import Pipeline from pipecat.transports.base_transport import TransportParams from pipecat.transports.network.small_webrtc import SmallWebRTCTransport from pipecat.transports.network.webrtc_connection import SmallWebRTCConnection async def run_bot ( webrtc_connection ): # Create the WebRTC transport with the provided connection transport = SmallWebRTCTransport( webrtc_connection = webrtc_connection, params = TransportParams( audio_in_enabled = True , # Accept audio from the client audio_out_enabled = True , # Send audio to the client vad_analyzer = SileroVADAnalyzer(), ), ) # Set up your services and context # Create the pipeline pipeline = Pipeline([ transport.input(), # Receive audio from client stt, # Convert speech to text context_aggregator.user(), # Add user messages to context llm, # Process text with LLM tts, # Convert text to speech transport.output(), # Send audio responses to client context_aggregator.assistant(), # Add assistant responses to context ]) # Register event handlers @transport.event_handler ( "on_client_connected" ) async def on_client_connected ( transport , client ): logger.info( "Client connected" ) # Start the conversation when client connects await task.queue_frames([context_aggregator.user().get_context_frame()]) @transport.event_handler ( "on_client_disconnected" ) async def on_client_disconnected ( transport , client ): logger.info( "Client disconnected" ) @transport.event_handler ( "on_client_closed" ) async def on_client_closed ( transport , client ): logger.info( "Client closed" ) await task.cancel() ​ How to connect with SmallWebRTCTransport For a client/server connection, you have two options for how to connect the client to the server: Use a Pipecat client SDK with the SmallWebRTCTransport . See the Client SDK docs to get started. Using the WebRTC API directly. This is only recommended for advanced use cases where the Pipecat client SDKs don’t have an available transport. ​ Examples To see a complete implementation, check out the following examples: Video Transform Demonstrates real-time video processing using WebRTC transport Voice Agent Implements a voice assistant using WebRTC for audio communication ​ Media Handling ​ Audio Audio is processed in 20ms chunks by default. The transport handles audio format conversion and resampling as needed: Input audio is processed at 16kHz (mono) to be compatible with speech recognition services Output audio can be configured to match your application’s requirements, but it must be mono, 16-bit PCM audio ​ Video Video is streamed using RGB format by default. The transport provides: Frame conversion between different color formats (RGB, YUV, etc.) Configurable resolution and framerate ​ WebRTC ICE Servers Configuration When implementing WebRTC in your project, STUN (Session Traversal Utilities for NAT) and TURN (Traversal Using Relays around NAT) servers are usually needed in cases where users are behind routers or firewalls. In local networks (e.g., testing within the same home or office network), you usually don’t need to configure STUN or TURN servers. In such cases, WebRTC can often directly establish peer-to-peer connections without needing to traverse NAT or firewalls. ​ What are STUN and TURN Servers? STUN Server : Helps clients discover their public IP address and port when they’re behind a NAT (Network Address Translation) device (like a router). This allows WebRTC to attempt direct peer-to-peer communication by providing the public-facing IP and port. TURN Server : Used as a fallback when direct peer-to-peer communication isn’t possible due to strict NATs or firewalls blocking connections. The TURN server relays media traffic between peers. ​ Why are ICE Servers Important? ICE (Interactive Connectivity Establishment) is a framework used by WebRTC to handle network traversal and NAT issues. The iceServers configuration provides a list of STUN and TURN servers that WebRTC uses to find the best way to connect two peers. ​ Advanced Configuration ​ ICE Servers For better connectivity, especially when testing across different networks, you can provide STUN servers: Copy Ask AI webrtc_connection = SmallWebRTCConnection( ice_servers = [ "stun:stun.l.google.com:19302" , "stun:stun1.l.google.com:19302" ] ) You can also use IceServer objects for more advanced configuration: Copy Ask AI from pipecat.transports.network.webrtc_connection import IceServer webrtc_connection = SmallWebRTCConnection( ice_servers = [ IceServer( urls = "stun:stun.l.google.com:19302" ), IceServer( urls = "turn:turn.example.com:3478" , username = "username" , credential = "password" ) ] ) ​ Troubleshooting If clients have trouble connecting or streaming: Check browser console for WebRTC errors Ensure you’re using HTTPS in production (required for WebRTC) For testing across networks, consider using Daily which provides TURN servers Verify browser permissions for camera and microphone Daily WebRTC FastAPI WebSocket On this page Overview Installation Class Reference SmallWebRTCConnection Methods SmallWebRTCTransport Methods Event Handlers Basic Usage How to connect with SmallWebRTCTransport Examples Media Handling Audio Video WebRTC ICE Servers Configuration What are STUN and TURN Servers? Why are ICE Servers Important? Advanced Configuration ICE Servers Troubleshooting Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/transport_small-webrtc_b8ab44d1.txt b/transport_small-webrtc_b8ab44d1.txt
new file mode 100644
index 0000000000000000000000000000000000000000..eb982ba1a1118a3b92ea7d88ba68f310cbd0522c
--- /dev/null
+++ b/transport_small-webrtc_b8ab44d1.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/transport/small-webrtc#class-reference
+Title: SmallWebRTCTransport - Pipecat
+==================================================
+
+SmallWebRTCTransport - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Transport SmallWebRTCTransport Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Daily WebRTC SmallWebRTCTransport FastAPI WebSocket WebSocket Server Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview SmallWebRTCTransport enables peer-to-peer WebRTC connections between clients and your Pipecat application. It implements bidirectional audio and video streaming using WebRTC for real-time communication. This transport is intended for lightweight implementations, particularly for local development and testing. It expects your clients to include a corresponding SmallWebRTCTransport implementation. See here for the JavaScript implementation. SmallWebRTCTransport is best used for testing and development. For production deployments with scale, consider using the DailyTransport , as it has global, low-latency infrastructure. ​ Installation To use SmallWebRTCTransport , install the required dependencies: Copy Ask AI pip install pipecat-ai[webrtc] ​ Class Reference ​ SmallWebRTCConnection SmallWebRTCConnection manages the WebRTC connection details, peer connection state, and ICE candidates. It handles the signaling process and media tracks. Copy Ask AI SmallWebRTCConnection( ice_servers = None ) ​ ice_servers Union[List[str], List[IceServer]] List of STUN/TURN server URLs for ICE connection establishment. Can be provided as strings or as IceServer objects. ​ Methods ​ initialize async method Initialize the connection with a client’s SDP offer. Parameters: sdp : String containing the Session Description Protocol data from client’s offer type : String representing the SDP message type (typically “offer”) Copy Ask AI await webrtc_connection.initialize( sdp = client_sdp, type = "offer" ) ​ connect async method Establish the WebRTC peer connection after initialization. Copy Ask AI await webrtc_connection.connect() ​ close async method Close the WebRTC peer connection. Copy Ask AI await webrtc_connection.close() ​ disconnect async method Disconnect the WebRTC peer connection and send a peer left message to the client. Copy Ask AI await webrtc_connection.disconnect() ​ renegotiate async method Handle connection renegotiation requests. Parameters: sdp : String containing the Session Description Protocol data for renegotiation type : String representing the SDP message type restart_pc : Boolean indicating whether to completely restart the peer connection (default: False) Copy Ask AI await webrtc_connection.renegotiate( sdp = new_sdp, type = "offer" , restart_pc = False ) ​ get_answer method Retrieve the SDP answer to send back to the client. Returns a dictionary with sdp , type , and pc_id fields. Copy Ask AI answer = webrtc_connection.get_answer() # Returns: {"sdp": "...", "type": "answer", "pc_id": "..."} ​ send_app_message method Send an application message to the client. Parameters: message : The message to send (will be JSON serialized) Copy Ask AI webrtc_connection.send_app_message({ "action" : "greeting" , "text" : "Hello!" }) ​ is_connected method Check if the connection is active. Copy Ask AI if webrtc_connection.is_connected(): print ( "Connection is active" ) ​ audio_input_track method Get the audio input track from the client. Copy Ask AI audio_track = webrtc_connection.audio_input_track() ​ video_input_track method Get the video input track from the client. Copy Ask AI video_track = webrtc_connection.video_input_track() ​ replace_audio_track method Replace the current audio track with a new one. Parameters: track : The new audio track to use Copy Ask AI webrtc_connection.replace_audio_track(new_audio_track) ​ replace_video_track method Replace the current video track with a new one. Parameters: track : The new video track to use Copy Ask AI webrtc_connection.replace_video_track(new_video_track) ​ ask_to_renegotiate method Request the client to initiate connection renegotiation. Copy Ask AI webrtc_connection.ask_to_renegotiate() ​ event_handler decorator Register an event handler for connection events. Events: "app-message" : Called when a message is received from the client "track-started" : Called when a new track is started "track-ended" : Called when a track ends "connecting" : Called when connection is being established "connected" : Called when connection is established "disconnected" : Called when connection is lost "closed" : Called when connection is closed "failed" : Called when connection fails "new" : Called when a new connection is created Copy Ask AI @webrtc_connection.event_handler ( "connected" ) async def on_connected ( connection ): print ( f "WebRTC connection established" ) ​ SmallWebRTCTransport SmallWebRTCTransport is the main transport class that manages both input and output transports for WebRTC communication. Copy Ask AI SmallWebRTCTransport( webrtc_connection: SmallWebRTCConnection, params: TransportParams, input_name: Optional[ str ] = None , output_name: Optional[ str ] = None ) ​ webrtc_connection SmallWebRTCConnection required An instance of SmallWebRTCConnection that manages the WebRTC connection ​ params TransportParams required Configuration parameters for the transport Show TransportParams properties ​ audio_in_enabled bool default: false Enable audio input from the WebRTC client ​ audio_in_passthrough bool default: "False" When enabled, incoming audio frames are pushed downstream ​ audio_out_enabled bool default: false Enable audio output to the WebRTC client ​ audio_in_sample_rate int Sample rate for incoming audio (Hz) ​ audio_out_sample_rate int Sample rate for outgoing audio (Hz) ​ audio_in_channels int default: 1 Number of audio input channels (1 for mono, 2 for stereo) ​ audio_out_channels int default: 1 Number of audio output channels (1 for mono, 2 for stereo) ​ video_in_enabled bool default: false Enable video input from the WebRTC client ​ video_out_enabled bool default: false Enable video output to the WebRTC client ​ video_out_width int default: 640 Width of outgoing video ​ video_out_height int default: 480 Height of outgoing video ​ video_out_framerate int default: 30 Framerate of outgoing video ​ vad_analyzer VADAnalyzer Custom VAD analyzer implementation ​ input_name str Optional name for the input transport ​ output_name str Optional name for the output transport ​ Methods ​ input method Returns the input transport instance. Copy Ask AI input_transport = webrtc_transport.input() ​ output method Returns the output transport instance. Copy Ask AI output_transport = webrtc_transport.output() ​ send_image async method Send an image frame to the client. Parameters: frame : The image frame to send (OutputImageRawFrame or SpriteFrame) Copy Ask AI await webrtc_transport.send_image(image_frame) ​ send_audio async method Send an audio frame to the client. Parameters: frame : The audio frame to send (OutputAudioRawFrame) Copy Ask AI await webrtc_transport.send_audio(audio_frame) ​ Event Handlers ​ on_app_message async callback Called when receiving application messages from the client. Parameters: message : The received message Copy Ask AI @webrtc_transport.event_handler ( "on_app_message" ) async def on_app_message ( message ): print ( f "Received message: { message } " ) ​ on_client_connected async callback Called when a client successfully connects. Parameters: transport : The SmallWebRTCTransport instance webrtc_connection : The connection that was established Copy Ask AI @webrtc_transport.event_handler ( "on_client_connected" ) async def on_client_connected ( transport , webrtc_connection ): print ( f "Client connected" ) ​ on_client_disconnected async callback Called when a client disconnects. Parameters: transport : The SmallWebRTCTransport instance webrtc_connection : The connection that was disconnected Copy Ask AI @webrtc_transport.event_handler ( "on_client_disconnected" ) async def on_client_disconnected ( transport , webrtc_connection ): print ( f "Client disconnected" ) ​ on_client_closed async callback Called when a client connection is closed. Parameters: transport : The SmallWebRTCTransport instance webrtc_connection : The connection that was closed Copy Ask AI @webrtc_transport.event_handler ( "on_client_closed" ) async def on_client_closed ( transport , webrtc_connection ): print ( f "Client connection closed" ) ​ Basic Usage This basic usage example shows the transport specific parts of a bot.py file required to configure your bot: Copy Ask AI from pipecat.audio.vad.silero import SileroVADAnalyzer from pipecat.pipeline.pipeline import Pipeline from pipecat.transports.base_transport import TransportParams from pipecat.transports.network.small_webrtc import SmallWebRTCTransport from pipecat.transports.network.webrtc_connection import SmallWebRTCConnection async def run_bot ( webrtc_connection ): # Create the WebRTC transport with the provided connection transport = SmallWebRTCTransport( webrtc_connection = webrtc_connection, params = TransportParams( audio_in_enabled = True , # Accept audio from the client audio_out_enabled = True , # Send audio to the client vad_analyzer = SileroVADAnalyzer(), ), ) # Set up your services and context # Create the pipeline pipeline = Pipeline([ transport.input(), # Receive audio from client stt, # Convert speech to text context_aggregator.user(), # Add user messages to context llm, # Process text with LLM tts, # Convert text to speech transport.output(), # Send audio responses to client context_aggregator.assistant(), # Add assistant responses to context ]) # Register event handlers @transport.event_handler ( "on_client_connected" ) async def on_client_connected ( transport , client ): logger.info( "Client connected" ) # Start the conversation when client connects await task.queue_frames([context_aggregator.user().get_context_frame()]) @transport.event_handler ( "on_client_disconnected" ) async def on_client_disconnected ( transport , client ): logger.info( "Client disconnected" ) @transport.event_handler ( "on_client_closed" ) async def on_client_closed ( transport , client ): logger.info( "Client closed" ) await task.cancel() ​ How to connect with SmallWebRTCTransport For a client/server connection, you have two options for how to connect the client to the server: Use a Pipecat client SDK with the SmallWebRTCTransport . See the Client SDK docs to get started. Using the WebRTC API directly. This is only recommended for advanced use cases where the Pipecat client SDKs don’t have an available transport. ​ Examples To see a complete implementation, check out the following examples: Video Transform Demonstrates real-time video processing using WebRTC transport Voice Agent Implements a voice assistant using WebRTC for audio communication ​ Media Handling ​ Audio Audio is processed in 20ms chunks by default. The transport handles audio format conversion and resampling as needed: Input audio is processed at 16kHz (mono) to be compatible with speech recognition services Output audio can be configured to match your application’s requirements, but it must be mono, 16-bit PCM audio ​ Video Video is streamed using RGB format by default. The transport provides: Frame conversion between different color formats (RGB, YUV, etc.) Configurable resolution and framerate ​ WebRTC ICE Servers Configuration When implementing WebRTC in your project, STUN (Session Traversal Utilities for NAT) and TURN (Traversal Using Relays around NAT) servers are usually needed in cases where users are behind routers or firewalls. In local networks (e.g., testing within the same home or office network), you usually don’t need to configure STUN or TURN servers. In such cases, WebRTC can often directly establish peer-to-peer connections without needing to traverse NAT or firewalls. ​ What are STUN and TURN Servers? STUN Server : Helps clients discover their public IP address and port when they’re behind a NAT (Network Address Translation) device (like a router). This allows WebRTC to attempt direct peer-to-peer communication by providing the public-facing IP and port. TURN Server : Used as a fallback when direct peer-to-peer communication isn’t possible due to strict NATs or firewalls blocking connections. The TURN server relays media traffic between peers. ​ Why are ICE Servers Important? ICE (Interactive Connectivity Establishment) is a framework used by WebRTC to handle network traversal and NAT issues. The iceServers configuration provides a list of STUN and TURN servers that WebRTC uses to find the best way to connect two peers. ​ Advanced Configuration ​ ICE Servers For better connectivity, especially when testing across different networks, you can provide STUN servers: Copy Ask AI webrtc_connection = SmallWebRTCConnection( ice_servers = [ "stun:stun.l.google.com:19302" , "stun:stun1.l.google.com:19302" ] ) You can also use IceServer objects for more advanced configuration: Copy Ask AI from pipecat.transports.network.webrtc_connection import IceServer webrtc_connection = SmallWebRTCConnection( ice_servers = [ IceServer( urls = "stun:stun.l.google.com:19302" ), IceServer( urls = "turn:turn.example.com:3478" , username = "username" , credential = "password" ) ] ) ​ Troubleshooting If clients have trouble connecting or streaming: Check browser console for WebRTC errors Ensure you’re using HTTPS in production (required for WebRTC) For testing across networks, consider using Daily which provides TURN servers Verify browser permissions for camera and microphone Daily WebRTC FastAPI WebSocket On this page Overview Installation Class Reference SmallWebRTCConnection Methods SmallWebRTCTransport Methods Event Handlers Basic Usage How to connect with SmallWebRTCTransport Examples Media Handling Audio Video WebRTC ICE Servers Configuration What are STUN and TURN Servers? Why are ICE Servers Important? Advanced Configuration ICE Servers Troubleshooting Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/transport_small-webrtc_cdf9a1da.txt b/transport_small-webrtc_cdf9a1da.txt
new file mode 100644
index 0000000000000000000000000000000000000000..8258c75989313817ec5820305a3a0bed5f8bc7ca
--- /dev/null
+++ b/transport_small-webrtc_cdf9a1da.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/transport/small-webrtc#param-video-input-track
+Title: SmallWebRTCTransport - Pipecat
+==================================================
+
+SmallWebRTCTransport - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Transport SmallWebRTCTransport Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Daily WebRTC SmallWebRTCTransport FastAPI WebSocket WebSocket Server Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview SmallWebRTCTransport enables peer-to-peer WebRTC connections between clients and your Pipecat application. It implements bidirectional audio and video streaming using WebRTC for real-time communication. This transport is intended for lightweight implementations, particularly for local development and testing. It expects your clients to include a corresponding SmallWebRTCTransport implementation. See here for the JavaScript implementation. SmallWebRTCTransport is best used for testing and development. For production deployments with scale, consider using the DailyTransport , as it has global, low-latency infrastructure. ​ Installation To use SmallWebRTCTransport , install the required dependencies: Copy Ask AI pip install pipecat-ai[webrtc] ​ Class Reference ​ SmallWebRTCConnection SmallWebRTCConnection manages the WebRTC connection details, peer connection state, and ICE candidates. It handles the signaling process and media tracks. Copy Ask AI SmallWebRTCConnection( ice_servers = None ) ​ ice_servers Union[List[str], List[IceServer]] List of STUN/TURN server URLs for ICE connection establishment. Can be provided as strings or as IceServer objects. ​ Methods ​ initialize async method Initialize the connection with a client’s SDP offer. Parameters: sdp : String containing the Session Description Protocol data from client’s offer type : String representing the SDP message type (typically “offer”) Copy Ask AI await webrtc_connection.initialize( sdp = client_sdp, type = "offer" ) ​ connect async method Establish the WebRTC peer connection after initialization. Copy Ask AI await webrtc_connection.connect() ​ close async method Close the WebRTC peer connection. Copy Ask AI await webrtc_connection.close() ​ disconnect async method Disconnect the WebRTC peer connection and send a peer left message to the client. Copy Ask AI await webrtc_connection.disconnect() ​ renegotiate async method Handle connection renegotiation requests. Parameters: sdp : String containing the Session Description Protocol data for renegotiation type : String representing the SDP message type restart_pc : Boolean indicating whether to completely restart the peer connection (default: False) Copy Ask AI await webrtc_connection.renegotiate( sdp = new_sdp, type = "offer" , restart_pc = False ) ​ get_answer method Retrieve the SDP answer to send back to the client. Returns a dictionary with sdp , type , and pc_id fields. Copy Ask AI answer = webrtc_connection.get_answer() # Returns: {"sdp": "...", "type": "answer", "pc_id": "..."} ​ send_app_message method Send an application message to the client. Parameters: message : The message to send (will be JSON serialized) Copy Ask AI webrtc_connection.send_app_message({ "action" : "greeting" , "text" : "Hello!" }) ​ is_connected method Check if the connection is active. Copy Ask AI if webrtc_connection.is_connected(): print ( "Connection is active" ) ​ audio_input_track method Get the audio input track from the client. Copy Ask AI audio_track = webrtc_connection.audio_input_track() ​ video_input_track method Get the video input track from the client. Copy Ask AI video_track = webrtc_connection.video_input_track() ​ replace_audio_track method Replace the current audio track with a new one. Parameters: track : The new audio track to use Copy Ask AI webrtc_connection.replace_audio_track(new_audio_track) ​ replace_video_track method Replace the current video track with a new one. Parameters: track : The new video track to use Copy Ask AI webrtc_connection.replace_video_track(new_video_track) ​ ask_to_renegotiate method Request the client to initiate connection renegotiation. Copy Ask AI webrtc_connection.ask_to_renegotiate() ​ event_handler decorator Register an event handler for connection events. Events: "app-message" : Called when a message is received from the client "track-started" : Called when a new track is started "track-ended" : Called when a track ends "connecting" : Called when connection is being established "connected" : Called when connection is established "disconnected" : Called when connection is lost "closed" : Called when connection is closed "failed" : Called when connection fails "new" : Called when a new connection is created Copy Ask AI @webrtc_connection.event_handler ( "connected" ) async def on_connected ( connection ): print ( f "WebRTC connection established" ) ​ SmallWebRTCTransport SmallWebRTCTransport is the main transport class that manages both input and output transports for WebRTC communication. Copy Ask AI SmallWebRTCTransport( webrtc_connection: SmallWebRTCConnection, params: TransportParams, input_name: Optional[ str ] = None , output_name: Optional[ str ] = None ) ​ webrtc_connection SmallWebRTCConnection required An instance of SmallWebRTCConnection that manages the WebRTC connection ​ params TransportParams required Configuration parameters for the transport Show TransportParams properties ​ audio_in_enabled bool default: false Enable audio input from the WebRTC client ​ audio_in_passthrough bool default: "False" When enabled, incoming audio frames are pushed downstream ​ audio_out_enabled bool default: false Enable audio output to the WebRTC client ​ audio_in_sample_rate int Sample rate for incoming audio (Hz) ​ audio_out_sample_rate int Sample rate for outgoing audio (Hz) ​ audio_in_channels int default: 1 Number of audio input channels (1 for mono, 2 for stereo) ​ audio_out_channels int default: 1 Number of audio output channels (1 for mono, 2 for stereo) ​ video_in_enabled bool default: false Enable video input from the WebRTC client ​ video_out_enabled bool default: false Enable video output to the WebRTC client ​ video_out_width int default: 640 Width of outgoing video ​ video_out_height int default: 480 Height of outgoing video ​ video_out_framerate int default: 30 Framerate of outgoing video ​ vad_analyzer VADAnalyzer Custom VAD analyzer implementation ​ input_name str Optional name for the input transport ​ output_name str Optional name for the output transport ​ Methods ​ input method Returns the input transport instance. Copy Ask AI input_transport = webrtc_transport.input() ​ output method Returns the output transport instance. Copy Ask AI output_transport = webrtc_transport.output() ​ send_image async method Send an image frame to the client. Parameters: frame : The image frame to send (OutputImageRawFrame or SpriteFrame) Copy Ask AI await webrtc_transport.send_image(image_frame) ​ send_audio async method Send an audio frame to the client. Parameters: frame : The audio frame to send (OutputAudioRawFrame) Copy Ask AI await webrtc_transport.send_audio(audio_frame) ​ Event Handlers ​ on_app_message async callback Called when receiving application messages from the client. Parameters: message : The received message Copy Ask AI @webrtc_transport.event_handler ( "on_app_message" ) async def on_app_message ( message ): print ( f "Received message: { message } " ) ​ on_client_connected async callback Called when a client successfully connects. Parameters: transport : The SmallWebRTCTransport instance webrtc_connection : The connection that was established Copy Ask AI @webrtc_transport.event_handler ( "on_client_connected" ) async def on_client_connected ( transport , webrtc_connection ): print ( f "Client connected" ) ​ on_client_disconnected async callback Called when a client disconnects. Parameters: transport : The SmallWebRTCTransport instance webrtc_connection : The connection that was disconnected Copy Ask AI @webrtc_transport.event_handler ( "on_client_disconnected" ) async def on_client_disconnected ( transport , webrtc_connection ): print ( f "Client disconnected" ) ​ on_client_closed async callback Called when a client connection is closed. Parameters: transport : The SmallWebRTCTransport instance webrtc_connection : The connection that was closed Copy Ask AI @webrtc_transport.event_handler ( "on_client_closed" ) async def on_client_closed ( transport , webrtc_connection ): print ( f "Client connection closed" ) ​ Basic Usage This basic usage example shows the transport specific parts of a bot.py file required to configure your bot: Copy Ask AI from pipecat.audio.vad.silero import SileroVADAnalyzer from pipecat.pipeline.pipeline import Pipeline from pipecat.transports.base_transport import TransportParams from pipecat.transports.network.small_webrtc import SmallWebRTCTransport from pipecat.transports.network.webrtc_connection import SmallWebRTCConnection async def run_bot ( webrtc_connection ): # Create the WebRTC transport with the provided connection transport = SmallWebRTCTransport( webrtc_connection = webrtc_connection, params = TransportParams( audio_in_enabled = True , # Accept audio from the client audio_out_enabled = True , # Send audio to the client vad_analyzer = SileroVADAnalyzer(), ), ) # Set up your services and context # Create the pipeline pipeline = Pipeline([ transport.input(), # Receive audio from client stt, # Convert speech to text context_aggregator.user(), # Add user messages to context llm, # Process text with LLM tts, # Convert text to speech transport.output(), # Send audio responses to client context_aggregator.assistant(), # Add assistant responses to context ]) # Register event handlers @transport.event_handler ( "on_client_connected" ) async def on_client_connected ( transport , client ): logger.info( "Client connected" ) # Start the conversation when client connects await task.queue_frames([context_aggregator.user().get_context_frame()]) @transport.event_handler ( "on_client_disconnected" ) async def on_client_disconnected ( transport , client ): logger.info( "Client disconnected" ) @transport.event_handler ( "on_client_closed" ) async def on_client_closed ( transport , client ): logger.info( "Client closed" ) await task.cancel() ​ How to connect with SmallWebRTCTransport For a client/server connection, you have two options for how to connect the client to the server: Use a Pipecat client SDK with the SmallWebRTCTransport . See the Client SDK docs to get started. Using the WebRTC API directly. This is only recommended for advanced use cases where the Pipecat client SDKs don’t have an available transport. ​ Examples To see a complete implementation, check out the following examples: Video Transform Demonstrates real-time video processing using WebRTC transport Voice Agent Implements a voice assistant using WebRTC for audio communication ​ Media Handling ​ Audio Audio is processed in 20ms chunks by default. The transport handles audio format conversion and resampling as needed: Input audio is processed at 16kHz (mono) to be compatible with speech recognition services Output audio can be configured to match your application’s requirements, but it must be mono, 16-bit PCM audio ​ Video Video is streamed using RGB format by default. The transport provides: Frame conversion between different color formats (RGB, YUV, etc.) Configurable resolution and framerate ​ WebRTC ICE Servers Configuration When implementing WebRTC in your project, STUN (Session Traversal Utilities for NAT) and TURN (Traversal Using Relays around NAT) servers are usually needed in cases where users are behind routers or firewalls. In local networks (e.g., testing within the same home or office network), you usually don’t need to configure STUN or TURN servers. In such cases, WebRTC can often directly establish peer-to-peer connections without needing to traverse NAT or firewalls. ​ What are STUN and TURN Servers? STUN Server : Helps clients discover their public IP address and port when they’re behind a NAT (Network Address Translation) device (like a router). This allows WebRTC to attempt direct peer-to-peer communication by providing the public-facing IP and port. TURN Server : Used as a fallback when direct peer-to-peer communication isn’t possible due to strict NATs or firewalls blocking connections. The TURN server relays media traffic between peers. ​ Why are ICE Servers Important? ICE (Interactive Connectivity Establishment) is a framework used by WebRTC to handle network traversal and NAT issues. The iceServers configuration provides a list of STUN and TURN servers that WebRTC uses to find the best way to connect two peers. ​ Advanced Configuration ​ ICE Servers For better connectivity, especially when testing across different networks, you can provide STUN servers: Copy Ask AI webrtc_connection = SmallWebRTCConnection( ice_servers = [ "stun:stun.l.google.com:19302" , "stun:stun1.l.google.com:19302" ] ) You can also use IceServer objects for more advanced configuration: Copy Ask AI from pipecat.transports.network.webrtc_connection import IceServer webrtc_connection = SmallWebRTCConnection( ice_servers = [ IceServer( urls = "stun:stun.l.google.com:19302" ), IceServer( urls = "turn:turn.example.com:3478" , username = "username" , credential = "password" ) ] ) ​ Troubleshooting If clients have trouble connecting or streaming: Check browser console for WebRTC errors Ensure you’re using HTTPS in production (required for WebRTC) For testing across networks, consider using Daily which provides TURN servers Verify browser permissions for camera and microphone Daily WebRTC FastAPI WebSocket On this page Overview Installation Class Reference SmallWebRTCConnection Methods SmallWebRTCTransport Methods Event Handlers Basic Usage How to connect with SmallWebRTCTransport Examples Media Handling Audio Video WebRTC ICE Servers Configuration What are STUN and TURN Servers? Why are ICE Servers Important? Advanced Configuration ICE Servers Troubleshooting Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/transport_small-webrtc_d11d29bb.txt b/transport_small-webrtc_d11d29bb.txt
new file mode 100644
index 0000000000000000000000000000000000000000..67e2167d968b704d358ac389ac94977f45f830be
--- /dev/null
+++ b/transport_small-webrtc_d11d29bb.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/transport/small-webrtc#event-handlers
+Title: SmallWebRTCTransport - Pipecat
+==================================================
+
+SmallWebRTCTransport - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Transport SmallWebRTCTransport Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Daily WebRTC SmallWebRTCTransport FastAPI WebSocket WebSocket Server Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview SmallWebRTCTransport enables peer-to-peer WebRTC connections between clients and your Pipecat application. It implements bidirectional audio and video streaming using WebRTC for real-time communication. This transport is intended for lightweight implementations, particularly for local development and testing. It expects your clients to include a corresponding SmallWebRTCTransport implementation. See here for the JavaScript implementation. SmallWebRTCTransport is best used for testing and development. For production deployments with scale, consider using the DailyTransport , as it has global, low-latency infrastructure. ​ Installation To use SmallWebRTCTransport , install the required dependencies: Copy Ask AI pip install pipecat-ai[webrtc] ​ Class Reference ​ SmallWebRTCConnection SmallWebRTCConnection manages the WebRTC connection details, peer connection state, and ICE candidates. It handles the signaling process and media tracks. Copy Ask AI SmallWebRTCConnection( ice_servers = None ) ​ ice_servers Union[List[str], List[IceServer]] List of STUN/TURN server URLs for ICE connection establishment. Can be provided as strings or as IceServer objects. ​ Methods ​ initialize async method Initialize the connection with a client’s SDP offer. Parameters: sdp : String containing the Session Description Protocol data from client’s offer type : String representing the SDP message type (typically “offer”) Copy Ask AI await webrtc_connection.initialize( sdp = client_sdp, type = "offer" ) ​ connect async method Establish the WebRTC peer connection after initialization. Copy Ask AI await webrtc_connection.connect() ​ close async method Close the WebRTC peer connection. Copy Ask AI await webrtc_connection.close() ​ disconnect async method Disconnect the WebRTC peer connection and send a peer left message to the client. Copy Ask AI await webrtc_connection.disconnect() ​ renegotiate async method Handle connection renegotiation requests. Parameters: sdp : String containing the Session Description Protocol data for renegotiation type : String representing the SDP message type restart_pc : Boolean indicating whether to completely restart the peer connection (default: False) Copy Ask AI await webrtc_connection.renegotiate( sdp = new_sdp, type = "offer" , restart_pc = False ) ​ get_answer method Retrieve the SDP answer to send back to the client. Returns a dictionary with sdp , type , and pc_id fields. Copy Ask AI answer = webrtc_connection.get_answer() # Returns: {"sdp": "...", "type": "answer", "pc_id": "..."} ​ send_app_message method Send an application message to the client. Parameters: message : The message to send (will be JSON serialized) Copy Ask AI webrtc_connection.send_app_message({ "action" : "greeting" , "text" : "Hello!" }) ​ is_connected method Check if the connection is active. Copy Ask AI if webrtc_connection.is_connected(): print ( "Connection is active" ) ​ audio_input_track method Get the audio input track from the client. Copy Ask AI audio_track = webrtc_connection.audio_input_track() ​ video_input_track method Get the video input track from the client. Copy Ask AI video_track = webrtc_connection.video_input_track() ​ replace_audio_track method Replace the current audio track with a new one. Parameters: track : The new audio track to use Copy Ask AI webrtc_connection.replace_audio_track(new_audio_track) ​ replace_video_track method Replace the current video track with a new one. Parameters: track : The new video track to use Copy Ask AI webrtc_connection.replace_video_track(new_video_track) ​ ask_to_renegotiate method Request the client to initiate connection renegotiation. Copy Ask AI webrtc_connection.ask_to_renegotiate() ​ event_handler decorator Register an event handler for connection events. Events: "app-message" : Called when a message is received from the client "track-started" : Called when a new track is started "track-ended" : Called when a track ends "connecting" : Called when connection is being established "connected" : Called when connection is established "disconnected" : Called when connection is lost "closed" : Called when connection is closed "failed" : Called when connection fails "new" : Called when a new connection is created Copy Ask AI @webrtc_connection.event_handler ( "connected" ) async def on_connected ( connection ): print ( f "WebRTC connection established" ) ​ SmallWebRTCTransport SmallWebRTCTransport is the main transport class that manages both input and output transports for WebRTC communication. Copy Ask AI SmallWebRTCTransport( webrtc_connection: SmallWebRTCConnection, params: TransportParams, input_name: Optional[ str ] = None , output_name: Optional[ str ] = None ) ​ webrtc_connection SmallWebRTCConnection required An instance of SmallWebRTCConnection that manages the WebRTC connection ​ params TransportParams required Configuration parameters for the transport Show TransportParams properties ​ audio_in_enabled bool default: false Enable audio input from the WebRTC client ​ audio_in_passthrough bool default: "False" When enabled, incoming audio frames are pushed downstream ​ audio_out_enabled bool default: false Enable audio output to the WebRTC client ​ audio_in_sample_rate int Sample rate for incoming audio (Hz) ​ audio_out_sample_rate int Sample rate for outgoing audio (Hz) ​ audio_in_channels int default: 1 Number of audio input channels (1 for mono, 2 for stereo) ​ audio_out_channels int default: 1 Number of audio output channels (1 for mono, 2 for stereo) ​ video_in_enabled bool default: false Enable video input from the WebRTC client ​ video_out_enabled bool default: false Enable video output to the WebRTC client ​ video_out_width int default: 640 Width of outgoing video ​ video_out_height int default: 480 Height of outgoing video ​ video_out_framerate int default: 30 Framerate of outgoing video ​ vad_analyzer VADAnalyzer Custom VAD analyzer implementation ​ input_name str Optional name for the input transport ​ output_name str Optional name for the output transport ​ Methods ​ input method Returns the input transport instance. Copy Ask AI input_transport = webrtc_transport.input() ​ output method Returns the output transport instance. Copy Ask AI output_transport = webrtc_transport.output() ​ send_image async method Send an image frame to the client. Parameters: frame : The image frame to send (OutputImageRawFrame or SpriteFrame) Copy Ask AI await webrtc_transport.send_image(image_frame) ​ send_audio async method Send an audio frame to the client. Parameters: frame : The audio frame to send (OutputAudioRawFrame) Copy Ask AI await webrtc_transport.send_audio(audio_frame) ​ Event Handlers ​ on_app_message async callback Called when receiving application messages from the client. Parameters: message : The received message Copy Ask AI @webrtc_transport.event_handler ( "on_app_message" ) async def on_app_message ( message ): print ( f "Received message: { message } " ) ​ on_client_connected async callback Called when a client successfully connects. Parameters: transport : The SmallWebRTCTransport instance webrtc_connection : The connection that was established Copy Ask AI @webrtc_transport.event_handler ( "on_client_connected" ) async def on_client_connected ( transport , webrtc_connection ): print ( f "Client connected" ) ​ on_client_disconnected async callback Called when a client disconnects. Parameters: transport : The SmallWebRTCTransport instance webrtc_connection : The connection that was disconnected Copy Ask AI @webrtc_transport.event_handler ( "on_client_disconnected" ) async def on_client_disconnected ( transport , webrtc_connection ): print ( f "Client disconnected" ) ​ on_client_closed async callback Called when a client connection is closed. Parameters: transport : The SmallWebRTCTransport instance webrtc_connection : The connection that was closed Copy Ask AI @webrtc_transport.event_handler ( "on_client_closed" ) async def on_client_closed ( transport , webrtc_connection ): print ( f "Client connection closed" ) ​ Basic Usage This basic usage example shows the transport specific parts of a bot.py file required to configure your bot: Copy Ask AI from pipecat.audio.vad.silero import SileroVADAnalyzer from pipecat.pipeline.pipeline import Pipeline from pipecat.transports.base_transport import TransportParams from pipecat.transports.network.small_webrtc import SmallWebRTCTransport from pipecat.transports.network.webrtc_connection import SmallWebRTCConnection async def run_bot ( webrtc_connection ): # Create the WebRTC transport with the provided connection transport = SmallWebRTCTransport( webrtc_connection = webrtc_connection, params = TransportParams( audio_in_enabled = True , # Accept audio from the client audio_out_enabled = True , # Send audio to the client vad_analyzer = SileroVADAnalyzer(), ), ) # Set up your services and context # Create the pipeline pipeline = Pipeline([ transport.input(), # Receive audio from client stt, # Convert speech to text context_aggregator.user(), # Add user messages to context llm, # Process text with LLM tts, # Convert text to speech transport.output(), # Send audio responses to client context_aggregator.assistant(), # Add assistant responses to context ]) # Register event handlers @transport.event_handler ( "on_client_connected" ) async def on_client_connected ( transport , client ): logger.info( "Client connected" ) # Start the conversation when client connects await task.queue_frames([context_aggregator.user().get_context_frame()]) @transport.event_handler ( "on_client_disconnected" ) async def on_client_disconnected ( transport , client ): logger.info( "Client disconnected" ) @transport.event_handler ( "on_client_closed" ) async def on_client_closed ( transport , client ): logger.info( "Client closed" ) await task.cancel() ​ How to connect with SmallWebRTCTransport For a client/server connection, you have two options for how to connect the client to the server: Use a Pipecat client SDK with the SmallWebRTCTransport . See the Client SDK docs to get started. Using the WebRTC API directly. This is only recommended for advanced use cases where the Pipecat client SDKs don’t have an available transport. ​ Examples To see a complete implementation, check out the following examples: Video Transform Demonstrates real-time video processing using WebRTC transport Voice Agent Implements a voice assistant using WebRTC for audio communication ​ Media Handling ​ Audio Audio is processed in 20ms chunks by default. The transport handles audio format conversion and resampling as needed: Input audio is processed at 16kHz (mono) to be compatible with speech recognition services Output audio can be configured to match your application’s requirements, but it must be mono, 16-bit PCM audio ​ Video Video is streamed using RGB format by default. The transport provides: Frame conversion between different color formats (RGB, YUV, etc.) Configurable resolution and framerate ​ WebRTC ICE Servers Configuration When implementing WebRTC in your project, STUN (Session Traversal Utilities for NAT) and TURN (Traversal Using Relays around NAT) servers are usually needed in cases where users are behind routers or firewalls. In local networks (e.g., testing within the same home or office network), you usually don’t need to configure STUN or TURN servers. In such cases, WebRTC can often directly establish peer-to-peer connections without needing to traverse NAT or firewalls. ​ What are STUN and TURN Servers? STUN Server : Helps clients discover their public IP address and port when they’re behind a NAT (Network Address Translation) device (like a router). This allows WebRTC to attempt direct peer-to-peer communication by providing the public-facing IP and port. TURN Server : Used as a fallback when direct peer-to-peer communication isn’t possible due to strict NATs or firewalls blocking connections. The TURN server relays media traffic between peers. ​ Why are ICE Servers Important? ICE (Interactive Connectivity Establishment) is a framework used by WebRTC to handle network traversal and NAT issues. The iceServers configuration provides a list of STUN and TURN servers that WebRTC uses to find the best way to connect two peers. ​ Advanced Configuration ​ ICE Servers For better connectivity, especially when testing across different networks, you can provide STUN servers: Copy Ask AI webrtc_connection = SmallWebRTCConnection( ice_servers = [ "stun:stun.l.google.com:19302" , "stun:stun1.l.google.com:19302" ] ) You can also use IceServer objects for more advanced configuration: Copy Ask AI from pipecat.transports.network.webrtc_connection import IceServer webrtc_connection = SmallWebRTCConnection( ice_servers = [ IceServer( urls = "stun:stun.l.google.com:19302" ), IceServer( urls = "turn:turn.example.com:3478" , username = "username" , credential = "password" ) ] ) ​ Troubleshooting If clients have trouble connecting or streaming: Check browser console for WebRTC errors Ensure you’re using HTTPS in production (required for WebRTC) For testing across networks, consider using Daily which provides TURN servers Verify browser permissions for camera and microphone Daily WebRTC FastAPI WebSocket On this page Overview Installation Class Reference SmallWebRTCConnection Methods SmallWebRTCTransport Methods Event Handlers Basic Usage How to connect with SmallWebRTCTransport Examples Media Handling Audio Video WebRTC ICE Servers Configuration What are STUN and TURN Servers? Why are ICE Servers Important? Advanced Configuration ICE Servers Troubleshooting Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/transport_small-webrtc_fe12e32e.txt b/transport_small-webrtc_fe12e32e.txt
new file mode 100644
index 0000000000000000000000000000000000000000..c675fed087a22bf4430d006a8cd3c34ab92740f3
--- /dev/null
+++ b/transport_small-webrtc_fe12e32e.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/transport/small-webrtc#audio
+Title: SmallWebRTCTransport - Pipecat
+==================================================
+
+SmallWebRTCTransport - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Transport SmallWebRTCTransport Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Daily WebRTC SmallWebRTCTransport FastAPI WebSocket WebSocket Server Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview SmallWebRTCTransport enables peer-to-peer WebRTC connections between clients and your Pipecat application. It implements bidirectional audio and video streaming using WebRTC for real-time communication. This transport is intended for lightweight implementations, particularly for local development and testing. It expects your clients to include a corresponding SmallWebRTCTransport implementation. See here for the JavaScript implementation. SmallWebRTCTransport is best used for testing and development. For production deployments with scale, consider using the DailyTransport , as it has global, low-latency infrastructure. ​ Installation To use SmallWebRTCTransport , install the required dependencies: Copy Ask AI pip install pipecat-ai[webrtc] ​ Class Reference ​ SmallWebRTCConnection SmallWebRTCConnection manages the WebRTC connection details, peer connection state, and ICE candidates. It handles the signaling process and media tracks. Copy Ask AI SmallWebRTCConnection( ice_servers = None ) ​ ice_servers Union[List[str], List[IceServer]] List of STUN/TURN server URLs for ICE connection establishment. Can be provided as strings or as IceServer objects. ​ Methods ​ initialize async method Initialize the connection with a client’s SDP offer. Parameters: sdp : String containing the Session Description Protocol data from client’s offer type : String representing the SDP message type (typically “offer”) Copy Ask AI await webrtc_connection.initialize( sdp = client_sdp, type = "offer" ) ​ connect async method Establish the WebRTC peer connection after initialization. Copy Ask AI await webrtc_connection.connect() ​ close async method Close the WebRTC peer connection. Copy Ask AI await webrtc_connection.close() ​ disconnect async method Disconnect the WebRTC peer connection and send a peer left message to the client. Copy Ask AI await webrtc_connection.disconnect() ​ renegotiate async method Handle connection renegotiation requests. Parameters: sdp : String containing the Session Description Protocol data for renegotiation type : String representing the SDP message type restart_pc : Boolean indicating whether to completely restart the peer connection (default: False) Copy Ask AI await webrtc_connection.renegotiate( sdp = new_sdp, type = "offer" , restart_pc = False ) ​ get_answer method Retrieve the SDP answer to send back to the client. Returns a dictionary with sdp , type , and pc_id fields. Copy Ask AI answer = webrtc_connection.get_answer() # Returns: {"sdp": "...", "type": "answer", "pc_id": "..."} ​ send_app_message method Send an application message to the client. Parameters: message : The message to send (will be JSON serialized) Copy Ask AI webrtc_connection.send_app_message({ "action" : "greeting" , "text" : "Hello!" }) ​ is_connected method Check if the connection is active. Copy Ask AI if webrtc_connection.is_connected(): print ( "Connection is active" ) ​ audio_input_track method Get the audio input track from the client. Copy Ask AI audio_track = webrtc_connection.audio_input_track() ​ video_input_track method Get the video input track from the client. Copy Ask AI video_track = webrtc_connection.video_input_track() ​ replace_audio_track method Replace the current audio track with a new one. Parameters: track : The new audio track to use Copy Ask AI webrtc_connection.replace_audio_track(new_audio_track) ​ replace_video_track method Replace the current video track with a new one. Parameters: track : The new video track to use Copy Ask AI webrtc_connection.replace_video_track(new_video_track) ​ ask_to_renegotiate method Request the client to initiate connection renegotiation. Copy Ask AI webrtc_connection.ask_to_renegotiate() ​ event_handler decorator Register an event handler for connection events. Events: "app-message" : Called when a message is received from the client "track-started" : Called when a new track is started "track-ended" : Called when a track ends "connecting" : Called when connection is being established "connected" : Called when connection is established "disconnected" : Called when connection is lost "closed" : Called when connection is closed "failed" : Called when connection fails "new" : Called when a new connection is created Copy Ask AI @webrtc_connection.event_handler ( "connected" ) async def on_connected ( connection ): print ( f "WebRTC connection established" ) ​ SmallWebRTCTransport SmallWebRTCTransport is the main transport class that manages both input and output transports for WebRTC communication. Copy Ask AI SmallWebRTCTransport( webrtc_connection: SmallWebRTCConnection, params: TransportParams, input_name: Optional[ str ] = None , output_name: Optional[ str ] = None ) ​ webrtc_connection SmallWebRTCConnection required An instance of SmallWebRTCConnection that manages the WebRTC connection ​ params TransportParams required Configuration parameters for the transport Show TransportParams properties ​ audio_in_enabled bool default: false Enable audio input from the WebRTC client ​ audio_in_passthrough bool default: "False" When enabled, incoming audio frames are pushed downstream ​ audio_out_enabled bool default: false Enable audio output to the WebRTC client ​ audio_in_sample_rate int Sample rate for incoming audio (Hz) ​ audio_out_sample_rate int Sample rate for outgoing audio (Hz) ​ audio_in_channels int default: 1 Number of audio input channels (1 for mono, 2 for stereo) ​ audio_out_channels int default: 1 Number of audio output channels (1 for mono, 2 for stereo) ​ video_in_enabled bool default: false Enable video input from the WebRTC client ​ video_out_enabled bool default: false Enable video output to the WebRTC client ​ video_out_width int default: 640 Width of outgoing video ​ video_out_height int default: 480 Height of outgoing video ​ video_out_framerate int default: 30 Framerate of outgoing video ​ vad_analyzer VADAnalyzer Custom VAD analyzer implementation ​ input_name str Optional name for the input transport ​ output_name str Optional name for the output transport ​ Methods ​ input method Returns the input transport instance. Copy Ask AI input_transport = webrtc_transport.input() ​ output method Returns the output transport instance. Copy Ask AI output_transport = webrtc_transport.output() ​ send_image async method Send an image frame to the client. Parameters: frame : The image frame to send (OutputImageRawFrame or SpriteFrame) Copy Ask AI await webrtc_transport.send_image(image_frame) ​ send_audio async method Send an audio frame to the client. Parameters: frame : The audio frame to send (OutputAudioRawFrame) Copy Ask AI await webrtc_transport.send_audio(audio_frame) ​ Event Handlers ​ on_app_message async callback Called when receiving application messages from the client. Parameters: message : The received message Copy Ask AI @webrtc_transport.event_handler ( "on_app_message" ) async def on_app_message ( message ): print ( f "Received message: { message } " ) ​ on_client_connected async callback Called when a client successfully connects. Parameters: transport : The SmallWebRTCTransport instance webrtc_connection : The connection that was established Copy Ask AI @webrtc_transport.event_handler ( "on_client_connected" ) async def on_client_connected ( transport , webrtc_connection ): print ( f "Client connected" ) ​ on_client_disconnected async callback Called when a client disconnects. Parameters: transport : The SmallWebRTCTransport instance webrtc_connection : The connection that was disconnected Copy Ask AI @webrtc_transport.event_handler ( "on_client_disconnected" ) async def on_client_disconnected ( transport , webrtc_connection ): print ( f "Client disconnected" ) ​ on_client_closed async callback Called when a client connection is closed. Parameters: transport : The SmallWebRTCTransport instance webrtc_connection : The connection that was closed Copy Ask AI @webrtc_transport.event_handler ( "on_client_closed" ) async def on_client_closed ( transport , webrtc_connection ): print ( f "Client connection closed" ) ​ Basic Usage This basic usage example shows the transport specific parts of a bot.py file required to configure your bot: Copy Ask AI from pipecat.audio.vad.silero import SileroVADAnalyzer from pipecat.pipeline.pipeline import Pipeline from pipecat.transports.base_transport import TransportParams from pipecat.transports.network.small_webrtc import SmallWebRTCTransport from pipecat.transports.network.webrtc_connection import SmallWebRTCConnection async def run_bot ( webrtc_connection ): # Create the WebRTC transport with the provided connection transport = SmallWebRTCTransport( webrtc_connection = webrtc_connection, params = TransportParams( audio_in_enabled = True , # Accept audio from the client audio_out_enabled = True , # Send audio to the client vad_analyzer = SileroVADAnalyzer(), ), ) # Set up your services and context # Create the pipeline pipeline = Pipeline([ transport.input(), # Receive audio from client stt, # Convert speech to text context_aggregator.user(), # Add user messages to context llm, # Process text with LLM tts, # Convert text to speech transport.output(), # Send audio responses to client context_aggregator.assistant(), # Add assistant responses to context ]) # Register event handlers @transport.event_handler ( "on_client_connected" ) async def on_client_connected ( transport , client ): logger.info( "Client connected" ) # Start the conversation when client connects await task.queue_frames([context_aggregator.user().get_context_frame()]) @transport.event_handler ( "on_client_disconnected" ) async def on_client_disconnected ( transport , client ): logger.info( "Client disconnected" ) @transport.event_handler ( "on_client_closed" ) async def on_client_closed ( transport , client ): logger.info( "Client closed" ) await task.cancel() ​ How to connect with SmallWebRTCTransport For a client/server connection, you have two options for how to connect the client to the server: Use a Pipecat client SDK with the SmallWebRTCTransport . See the Client SDK docs to get started. Using the WebRTC API directly. This is only recommended for advanced use cases where the Pipecat client SDKs don’t have an available transport. ​ Examples To see a complete implementation, check out the following examples: Video Transform Demonstrates real-time video processing using WebRTC transport Voice Agent Implements a voice assistant using WebRTC for audio communication ​ Media Handling ​ Audio Audio is processed in 20ms chunks by default. The transport handles audio format conversion and resampling as needed: Input audio is processed at 16kHz (mono) to be compatible with speech recognition services Output audio can be configured to match your application’s requirements, but it must be mono, 16-bit PCM audio ​ Video Video is streamed using RGB format by default. The transport provides: Frame conversion between different color formats (RGB, YUV, etc.) Configurable resolution and framerate ​ WebRTC ICE Servers Configuration When implementing WebRTC in your project, STUN (Session Traversal Utilities for NAT) and TURN (Traversal Using Relays around NAT) servers are usually needed in cases where users are behind routers or firewalls. In local networks (e.g., testing within the same home or office network), you usually don’t need to configure STUN or TURN servers. In such cases, WebRTC can often directly establish peer-to-peer connections without needing to traverse NAT or firewalls. ​ What are STUN and TURN Servers? STUN Server : Helps clients discover their public IP address and port when they’re behind a NAT (Network Address Translation) device (like a router). This allows WebRTC to attempt direct peer-to-peer communication by providing the public-facing IP and port. TURN Server : Used as a fallback when direct peer-to-peer communication isn’t possible due to strict NATs or firewalls blocking connections. The TURN server relays media traffic between peers. ​ Why are ICE Servers Important? ICE (Interactive Connectivity Establishment) is a framework used by WebRTC to handle network traversal and NAT issues. The iceServers configuration provides a list of STUN and TURN servers that WebRTC uses to find the best way to connect two peers. ​ Advanced Configuration ​ ICE Servers For better connectivity, especially when testing across different networks, you can provide STUN servers: Copy Ask AI webrtc_connection = SmallWebRTCConnection( ice_servers = [ "stun:stun.l.google.com:19302" , "stun:stun1.l.google.com:19302" ] ) You can also use IceServer objects for more advanced configuration: Copy Ask AI from pipecat.transports.network.webrtc_connection import IceServer webrtc_connection = SmallWebRTCConnection( ice_servers = [ IceServer( urls = "stun:stun.l.google.com:19302" ), IceServer( urls = "turn:turn.example.com:3478" , username = "username" , credential = "password" ) ] ) ​ Troubleshooting If clients have trouble connecting or streaming: Check browser console for WebRTC errors Ensure you’re using HTTPS in production (required for WebRTC) For testing across networks, consider using Daily which provides TURN servers Verify browser permissions for camera and microphone Daily WebRTC FastAPI WebSocket On this page Overview Installation Class Reference SmallWebRTCConnection Methods SmallWebRTCTransport Methods Event Handlers Basic Usage How to connect with SmallWebRTCTransport Examples Media Handling Audio Video WebRTC ICE Servers Configuration What are STUN and TURN Servers? Why are ICE Servers Important? Advanced Configuration ICE Servers Troubleshooting Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/transport_websocket-server_153d1388.txt b/transport_websocket-server_153d1388.txt
new file mode 100644
index 0000000000000000000000000000000000000000..a3f1e1e1a7fd18a614ea3401a6c39ac989cd6bec
--- /dev/null
+++ b/transport_websocket-server_153d1388.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/transport/websocket-server#param-on-session-timeout
+Title: WebSocket Server - Pipecat
+==================================================
+
+WebSocket Server - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Transport WebSocket Server Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Daily WebRTC SmallWebRTCTransport FastAPI WebSocket WebSocket Server Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview WebsocketServerTransport provides a WebSocket server implementation for real-time audio communication. It supports bidirectional audio streams and voice activity detection (VAD). WebsocketServerTransport is best suited for server-side applications and prototyping client/server apps. For client/server production applications, we strongly recommend using a WebRTC-based transport for robust network and media handling. ​ Installation To use WebsocketServerTransport , install the required dependencies: Copy Ask AI pip install "pipecat-ai[websocket]" ​ Configuration ​ Constructor Parameters ​ host str default: "localhost" Host address to bind the WebSocket server ​ port int default: "8765" Port number for the WebSocket server ​ params WebsocketServerParams default: "WebsocketServerParams()" Transport configuration parameters ​ WebsocketServerParams Configuration ​ add_wav_header bool default: "False" Add WAV header to audio frames ​ serializer FrameSerializer default: "ProtobufFrameSerializer()" Frame serializer for WebSocket messages ​ session_timeout int | None default: "None" Session timeout in seconds. If set, triggers timeout event when no activity is detected ​ Audio Configuration ​ audio_in_enabled bool default: false Enable audio input from the WebRTC client ​ audio_in_passthrough bool default: "False" When enabled, incoming audio frames are pushed downstream ​ audio_out_enabled bool default: "False" Enable audio output capabilities ​ audio_out_sample_rate int default: "None" Audio output sample rate in Hz ​ audio_out_channels int default: "1" Number of audio output channels ​ Voice Activity Detection (VAD) ​ vad_analyzer VADAnalyzer | None default: "None" Voice Activity Detection analyzer. You can set this to either SileroVADAnalyzer() or WebRTCVADAnalyzer() . SileroVADAnalyzer is the recommended option. Learn more about the SileroVADAnalyzer . ​ Basic Usage Copy Ask AI from pipecat.transports.network.websocket_server import ( WebsocketServerTransport, WebsocketServerParams ) from pipecat.audio.vad.silero import SileroVADAnalyzer from pipecat.pipeline.pipeline import Pipeline # Configure transport transport = WebsocketServerTransport( host = "localhost" , port = 8765 , params = WebsocketServerParams( audio_in_enabled = True , audio_out_enabled = True , add_wav_header = True , vad_analyzer = SileroVADAnalyzer(), session_timeout = 180 # 3 minutes ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), # Handle incoming audio stt, # Speech-to-text llm, # Language model tts, # Text-to-speech transport.output() # Handle outgoing audio ]) Check out the Websocket Server example to see how to use this transport in a pipeline. ​ Event Callbacks WebsocketServerTransport provides callbacks for handling client connection events. Register callbacks using the @transport.event_handler() decorator. ​ Connection Events ​ on_client_connected async callback Called when a client connects to the WebSocket server. Parameters: transport : The WebsocketServerTransport instance client : WebSocket client connection object Copy Ask AI @transport.event_handler ( "on_client_connected" ) async def on_client_connected ( transport , client ): logger.info( f "Client connected: { client.remote_address } " ) # Initialize conversation await task.queue_frames([LLMMessagesFrame(initial_messages)]) ​ on_client_disconnected async callback Called when a client disconnects from the WebSocket server. Parameters: transport : The WebsocketServerTransport instance client : WebSocket client connection object Copy Ask AI @transport.event_handler ( "on_client_disconnected" ) async def on_client_disconnected ( transport , client ): logger.info( f "Client disconnected: { client.remote_address } " ) ​ on_session_timeout async callback Called when a session times out (if session_timeout is configured). Parameters: transport : The WebsocketServerTransport instance client : WebSocket client connection object Copy Ask AI @transport.event_handler ( "on_session_timeout" ) async def on_session_timeout ( transport , client ): logger.info( f "Session timeout for client: { client.remote_address } " ) # Handle timeout (e.g., send message, close connection) ​ Frame Types ​ Input Frames ​ InputAudioRawFrame Frame Raw audio data from the WebSocket client ​ Output Frames ​ OutputAudioRawFrame Frame Audio data to be sent to the WebSocket client ​ Notes Supports real-time audio communication Best suited for server-side applications Handles WebSocket connection management Provides voice activity detection Supports session timeouts Single client per server (new connections replace existing ones) All callbacks are asynchronous FastAPI WebSocket Frame Serializer Overview On this page Overview Installation Configuration Constructor Parameters WebsocketServerParams Configuration Audio Configuration Voice Activity Detection (VAD) Basic Usage Event Callbacks Connection Events Frame Types Input Frames Output Frames Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/transport_websocket-server_7bfd44ee.txt b/transport_websocket-server_7bfd44ee.txt
new file mode 100644
index 0000000000000000000000000000000000000000..afa918bfb45b0c0ff3d67dfb481e60771f1cb773
--- /dev/null
+++ b/transport_websocket-server_7bfd44ee.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/transport/websocket-server#configuration
+Title: WebSocket Server - Pipecat
+==================================================
+
+WebSocket Server - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Transport WebSocket Server Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Daily WebRTC SmallWebRTCTransport FastAPI WebSocket WebSocket Server Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview WebsocketServerTransport provides a WebSocket server implementation for real-time audio communication. It supports bidirectional audio streams and voice activity detection (VAD). WebsocketServerTransport is best suited for server-side applications and prototyping client/server apps. For client/server production applications, we strongly recommend using a WebRTC-based transport for robust network and media handling. ​ Installation To use WebsocketServerTransport , install the required dependencies: Copy Ask AI pip install "pipecat-ai[websocket]" ​ Configuration ​ Constructor Parameters ​ host str default: "localhost" Host address to bind the WebSocket server ​ port int default: "8765" Port number for the WebSocket server ​ params WebsocketServerParams default: "WebsocketServerParams()" Transport configuration parameters ​ WebsocketServerParams Configuration ​ add_wav_header bool default: "False" Add WAV header to audio frames ​ serializer FrameSerializer default: "ProtobufFrameSerializer()" Frame serializer for WebSocket messages ​ session_timeout int | None default: "None" Session timeout in seconds. If set, triggers timeout event when no activity is detected ​ Audio Configuration ​ audio_in_enabled bool default: false Enable audio input from the WebRTC client ​ audio_in_passthrough bool default: "False" When enabled, incoming audio frames are pushed downstream ​ audio_out_enabled bool default: "False" Enable audio output capabilities ​ audio_out_sample_rate int default: "None" Audio output sample rate in Hz ​ audio_out_channels int default: "1" Number of audio output channels ​ Voice Activity Detection (VAD) ​ vad_analyzer VADAnalyzer | None default: "None" Voice Activity Detection analyzer. You can set this to either SileroVADAnalyzer() or WebRTCVADAnalyzer() . SileroVADAnalyzer is the recommended option. Learn more about the SileroVADAnalyzer . ​ Basic Usage Copy Ask AI from pipecat.transports.network.websocket_server import ( WebsocketServerTransport, WebsocketServerParams ) from pipecat.audio.vad.silero import SileroVADAnalyzer from pipecat.pipeline.pipeline import Pipeline # Configure transport transport = WebsocketServerTransport( host = "localhost" , port = 8765 , params = WebsocketServerParams( audio_in_enabled = True , audio_out_enabled = True , add_wav_header = True , vad_analyzer = SileroVADAnalyzer(), session_timeout = 180 # 3 minutes ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), # Handle incoming audio stt, # Speech-to-text llm, # Language model tts, # Text-to-speech transport.output() # Handle outgoing audio ]) Check out the Websocket Server example to see how to use this transport in a pipeline. ​ Event Callbacks WebsocketServerTransport provides callbacks for handling client connection events. Register callbacks using the @transport.event_handler() decorator. ​ Connection Events ​ on_client_connected async callback Called when a client connects to the WebSocket server. Parameters: transport : The WebsocketServerTransport instance client : WebSocket client connection object Copy Ask AI @transport.event_handler ( "on_client_connected" ) async def on_client_connected ( transport , client ): logger.info( f "Client connected: { client.remote_address } " ) # Initialize conversation await task.queue_frames([LLMMessagesFrame(initial_messages)]) ​ on_client_disconnected async callback Called when a client disconnects from the WebSocket server. Parameters: transport : The WebsocketServerTransport instance client : WebSocket client connection object Copy Ask AI @transport.event_handler ( "on_client_disconnected" ) async def on_client_disconnected ( transport , client ): logger.info( f "Client disconnected: { client.remote_address } " ) ​ on_session_timeout async callback Called when a session times out (if session_timeout is configured). Parameters: transport : The WebsocketServerTransport instance client : WebSocket client connection object Copy Ask AI @transport.event_handler ( "on_session_timeout" ) async def on_session_timeout ( transport , client ): logger.info( f "Session timeout for client: { client.remote_address } " ) # Handle timeout (e.g., send message, close connection) ​ Frame Types ​ Input Frames ​ InputAudioRawFrame Frame Raw audio data from the WebSocket client ​ Output Frames ​ OutputAudioRawFrame Frame Audio data to be sent to the WebSocket client ​ Notes Supports real-time audio communication Best suited for server-side applications Handles WebSocket connection management Provides voice activity detection Supports session timeouts Single client per server (new connections replace existing ones) All callbacks are asynchronous FastAPI WebSocket Frame Serializer Overview On this page Overview Installation Configuration Constructor Parameters WebsocketServerParams Configuration Audio Configuration Voice Activity Detection (VAD) Basic Usage Event Callbacks Connection Events Frame Types Input Frames Output Frames Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/transport_websocket-server_938c49cf.txt b/transport_websocket-server_938c49cf.txt
new file mode 100644
index 0000000000000000000000000000000000000000..6266bcd6949f9f6ef10eb00421a13bf0badc7e99
--- /dev/null
+++ b/transport_websocket-server_938c49cf.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/transport/websocket-server
+Title: WebSocket Server - Pipecat
+==================================================
+
+WebSocket Server - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Transport WebSocket Server Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Daily WebRTC SmallWebRTCTransport FastAPI WebSocket WebSocket Server Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview WebsocketServerTransport provides a WebSocket server implementation for real-time audio communication. It supports bidirectional audio streams and voice activity detection (VAD). WebsocketServerTransport is best suited for server-side applications and prototyping client/server apps. For client/server production applications, we strongly recommend using a WebRTC-based transport for robust network and media handling. ​ Installation To use WebsocketServerTransport , install the required dependencies: Copy Ask AI pip install "pipecat-ai[websocket]" ​ Configuration ​ Constructor Parameters ​ host str default: "localhost" Host address to bind the WebSocket server ​ port int default: "8765" Port number for the WebSocket server ​ params WebsocketServerParams default: "WebsocketServerParams()" Transport configuration parameters ​ WebsocketServerParams Configuration ​ add_wav_header bool default: "False" Add WAV header to audio frames ​ serializer FrameSerializer default: "ProtobufFrameSerializer()" Frame serializer for WebSocket messages ​ session_timeout int | None default: "None" Session timeout in seconds. If set, triggers timeout event when no activity is detected ​ Audio Configuration ​ audio_in_enabled bool default: false Enable audio input from the WebRTC client ​ audio_in_passthrough bool default: "False" When enabled, incoming audio frames are pushed downstream ​ audio_out_enabled bool default: "False" Enable audio output capabilities ​ audio_out_sample_rate int default: "None" Audio output sample rate in Hz ​ audio_out_channels int default: "1" Number of audio output channels ​ Voice Activity Detection (VAD) ​ vad_analyzer VADAnalyzer | None default: "None" Voice Activity Detection analyzer. You can set this to either SileroVADAnalyzer() or WebRTCVADAnalyzer() . SileroVADAnalyzer is the recommended option. Learn more about the SileroVADAnalyzer . ​ Basic Usage Copy Ask AI from pipecat.transports.network.websocket_server import ( WebsocketServerTransport, WebsocketServerParams ) from pipecat.audio.vad.silero import SileroVADAnalyzer from pipecat.pipeline.pipeline import Pipeline # Configure transport transport = WebsocketServerTransport( host = "localhost" , port = 8765 , params = WebsocketServerParams( audio_in_enabled = True , audio_out_enabled = True , add_wav_header = True , vad_analyzer = SileroVADAnalyzer(), session_timeout = 180 # 3 minutes ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), # Handle incoming audio stt, # Speech-to-text llm, # Language model tts, # Text-to-speech transport.output() # Handle outgoing audio ]) Check out the Websocket Server example to see how to use this transport in a pipeline. ​ Event Callbacks WebsocketServerTransport provides callbacks for handling client connection events. Register callbacks using the @transport.event_handler() decorator. ​ Connection Events ​ on_client_connected async callback Called when a client connects to the WebSocket server. Parameters: transport : The WebsocketServerTransport instance client : WebSocket client connection object Copy Ask AI @transport.event_handler ( "on_client_connected" ) async def on_client_connected ( transport , client ): logger.info( f "Client connected: { client.remote_address } " ) # Initialize conversation await task.queue_frames([LLMMessagesFrame(initial_messages)]) ​ on_client_disconnected async callback Called when a client disconnects from the WebSocket server. Parameters: transport : The WebsocketServerTransport instance client : WebSocket client connection object Copy Ask AI @transport.event_handler ( "on_client_disconnected" ) async def on_client_disconnected ( transport , client ): logger.info( f "Client disconnected: { client.remote_address } " ) ​ on_session_timeout async callback Called when a session times out (if session_timeout is configured). Parameters: transport : The WebsocketServerTransport instance client : WebSocket client connection object Copy Ask AI @transport.event_handler ( "on_session_timeout" ) async def on_session_timeout ( transport , client ): logger.info( f "Session timeout for client: { client.remote_address } " ) # Handle timeout (e.g., send message, close connection) ​ Frame Types ​ Input Frames ​ InputAudioRawFrame Frame Raw audio data from the WebSocket client ​ Output Frames ​ OutputAudioRawFrame Frame Audio data to be sent to the WebSocket client ​ Notes Supports real-time audio communication Best suited for server-side applications Handles WebSocket connection management Provides voice activity detection Supports session timeouts Single client per server (new connections replace existing ones) All callbacks are asynchronous FastAPI WebSocket Frame Serializer Overview On this page Overview Installation Configuration Constructor Parameters WebsocketServerParams Configuration Audio Configuration Voice Activity Detection (VAD) Basic Usage Event Callbacks Connection Events Frame Types Input Frames Output Frames Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/transport_websocket-server_9d8eb940.txt b/transport_websocket-server_9d8eb940.txt
new file mode 100644
index 0000000000000000000000000000000000000000..4df05973fecb76c17c31f8e0da0a7a06f392aa69
--- /dev/null
+++ b/transport_websocket-server_9d8eb940.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/transport/websocket-server#websocketserverparams-configuration
+Title: WebSocket Server - Pipecat
+==================================================
+
+WebSocket Server - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Transport WebSocket Server Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Daily WebRTC SmallWebRTCTransport FastAPI WebSocket WebSocket Server Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview WebsocketServerTransport provides a WebSocket server implementation for real-time audio communication. It supports bidirectional audio streams and voice activity detection (VAD). WebsocketServerTransport is best suited for server-side applications and prototyping client/server apps. For client/server production applications, we strongly recommend using a WebRTC-based transport for robust network and media handling. ​ Installation To use WebsocketServerTransport , install the required dependencies: Copy Ask AI pip install "pipecat-ai[websocket]" ​ Configuration ​ Constructor Parameters ​ host str default: "localhost" Host address to bind the WebSocket server ​ port int default: "8765" Port number for the WebSocket server ​ params WebsocketServerParams default: "WebsocketServerParams()" Transport configuration parameters ​ WebsocketServerParams Configuration ​ add_wav_header bool default: "False" Add WAV header to audio frames ​ serializer FrameSerializer default: "ProtobufFrameSerializer()" Frame serializer for WebSocket messages ​ session_timeout int | None default: "None" Session timeout in seconds. If set, triggers timeout event when no activity is detected ​ Audio Configuration ​ audio_in_enabled bool default: false Enable audio input from the WebRTC client ​ audio_in_passthrough bool default: "False" When enabled, incoming audio frames are pushed downstream ​ audio_out_enabled bool default: "False" Enable audio output capabilities ​ audio_out_sample_rate int default: "None" Audio output sample rate in Hz ​ audio_out_channels int default: "1" Number of audio output channels ​ Voice Activity Detection (VAD) ​ vad_analyzer VADAnalyzer | None default: "None" Voice Activity Detection analyzer. You can set this to either SileroVADAnalyzer() or WebRTCVADAnalyzer() . SileroVADAnalyzer is the recommended option. Learn more about the SileroVADAnalyzer . ​ Basic Usage Copy Ask AI from pipecat.transports.network.websocket_server import ( WebsocketServerTransport, WebsocketServerParams ) from pipecat.audio.vad.silero import SileroVADAnalyzer from pipecat.pipeline.pipeline import Pipeline # Configure transport transport = WebsocketServerTransport( host = "localhost" , port = 8765 , params = WebsocketServerParams( audio_in_enabled = True , audio_out_enabled = True , add_wav_header = True , vad_analyzer = SileroVADAnalyzer(), session_timeout = 180 # 3 minutes ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), # Handle incoming audio stt, # Speech-to-text llm, # Language model tts, # Text-to-speech transport.output() # Handle outgoing audio ]) Check out the Websocket Server example to see how to use this transport in a pipeline. ​ Event Callbacks WebsocketServerTransport provides callbacks for handling client connection events. Register callbacks using the @transport.event_handler() decorator. ​ Connection Events ​ on_client_connected async callback Called when a client connects to the WebSocket server. Parameters: transport : The WebsocketServerTransport instance client : WebSocket client connection object Copy Ask AI @transport.event_handler ( "on_client_connected" ) async def on_client_connected ( transport , client ): logger.info( f "Client connected: { client.remote_address } " ) # Initialize conversation await task.queue_frames([LLMMessagesFrame(initial_messages)]) ​ on_client_disconnected async callback Called when a client disconnects from the WebSocket server. Parameters: transport : The WebsocketServerTransport instance client : WebSocket client connection object Copy Ask AI @transport.event_handler ( "on_client_disconnected" ) async def on_client_disconnected ( transport , client ): logger.info( f "Client disconnected: { client.remote_address } " ) ​ on_session_timeout async callback Called when a session times out (if session_timeout is configured). Parameters: transport : The WebsocketServerTransport instance client : WebSocket client connection object Copy Ask AI @transport.event_handler ( "on_session_timeout" ) async def on_session_timeout ( transport , client ): logger.info( f "Session timeout for client: { client.remote_address } " ) # Handle timeout (e.g., send message, close connection) ​ Frame Types ​ Input Frames ​ InputAudioRawFrame Frame Raw audio data from the WebSocket client ​ Output Frames ​ OutputAudioRawFrame Frame Audio data to be sent to the WebSocket client ​ Notes Supports real-time audio communication Best suited for server-side applications Handles WebSocket connection management Provides voice activity detection Supports session timeouts Single client per server (new connections replace existing ones) All callbacks are asynchronous FastAPI WebSocket Frame Serializer Overview On this page Overview Installation Configuration Constructor Parameters WebsocketServerParams Configuration Audio Configuration Voice Activity Detection (VAD) Basic Usage Event Callbacks Connection Events Frame Types Input Frames Output Frames Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/transport_websocket-server_a7b44ae6.txt b/transport_websocket-server_a7b44ae6.txt
new file mode 100644
index 0000000000000000000000000000000000000000..b623103729f0ca67360ba2b1d9935080f800912e
--- /dev/null
+++ b/transport_websocket-server_a7b44ae6.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/transport/websocket-server#notes
+Title: WebSocket Server - Pipecat
+==================================================
+
+WebSocket Server - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Transport WebSocket Server Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Daily WebRTC SmallWebRTCTransport FastAPI WebSocket WebSocket Server Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview WebsocketServerTransport provides a WebSocket server implementation for real-time audio communication. It supports bidirectional audio streams and voice activity detection (VAD). WebsocketServerTransport is best suited for server-side applications and prototyping client/server apps. For client/server production applications, we strongly recommend using a WebRTC-based transport for robust network and media handling. ​ Installation To use WebsocketServerTransport , install the required dependencies: Copy Ask AI pip install "pipecat-ai[websocket]" ​ Configuration ​ Constructor Parameters ​ host str default: "localhost" Host address to bind the WebSocket server ​ port int default: "8765" Port number for the WebSocket server ​ params WebsocketServerParams default: "WebsocketServerParams()" Transport configuration parameters ​ WebsocketServerParams Configuration ​ add_wav_header bool default: "False" Add WAV header to audio frames ​ serializer FrameSerializer default: "ProtobufFrameSerializer()" Frame serializer for WebSocket messages ​ session_timeout int | None default: "None" Session timeout in seconds. If set, triggers timeout event when no activity is detected ​ Audio Configuration ​ audio_in_enabled bool default: false Enable audio input from the WebRTC client ​ audio_in_passthrough bool default: "False" When enabled, incoming audio frames are pushed downstream ​ audio_out_enabled bool default: "False" Enable audio output capabilities ​ audio_out_sample_rate int default: "None" Audio output sample rate in Hz ​ audio_out_channels int default: "1" Number of audio output channels ​ Voice Activity Detection (VAD) ​ vad_analyzer VADAnalyzer | None default: "None" Voice Activity Detection analyzer. You can set this to either SileroVADAnalyzer() or WebRTCVADAnalyzer() . SileroVADAnalyzer is the recommended option. Learn more about the SileroVADAnalyzer . ​ Basic Usage Copy Ask AI from pipecat.transports.network.websocket_server import ( WebsocketServerTransport, WebsocketServerParams ) from pipecat.audio.vad.silero import SileroVADAnalyzer from pipecat.pipeline.pipeline import Pipeline # Configure transport transport = WebsocketServerTransport( host = "localhost" , port = 8765 , params = WebsocketServerParams( audio_in_enabled = True , audio_out_enabled = True , add_wav_header = True , vad_analyzer = SileroVADAnalyzer(), session_timeout = 180 # 3 minutes ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), # Handle incoming audio stt, # Speech-to-text llm, # Language model tts, # Text-to-speech transport.output() # Handle outgoing audio ]) Check out the Websocket Server example to see how to use this transport in a pipeline. ​ Event Callbacks WebsocketServerTransport provides callbacks for handling client connection events. Register callbacks using the @transport.event_handler() decorator. ​ Connection Events ​ on_client_connected async callback Called when a client connects to the WebSocket server. Parameters: transport : The WebsocketServerTransport instance client : WebSocket client connection object Copy Ask AI @transport.event_handler ( "on_client_connected" ) async def on_client_connected ( transport , client ): logger.info( f "Client connected: { client.remote_address } " ) # Initialize conversation await task.queue_frames([LLMMessagesFrame(initial_messages)]) ​ on_client_disconnected async callback Called when a client disconnects from the WebSocket server. Parameters: transport : The WebsocketServerTransport instance client : WebSocket client connection object Copy Ask AI @transport.event_handler ( "on_client_disconnected" ) async def on_client_disconnected ( transport , client ): logger.info( f "Client disconnected: { client.remote_address } " ) ​ on_session_timeout async callback Called when a session times out (if session_timeout is configured). Parameters: transport : The WebsocketServerTransport instance client : WebSocket client connection object Copy Ask AI @transport.event_handler ( "on_session_timeout" ) async def on_session_timeout ( transport , client ): logger.info( f "Session timeout for client: { client.remote_address } " ) # Handle timeout (e.g., send message, close connection) ​ Frame Types ​ Input Frames ​ InputAudioRawFrame Frame Raw audio data from the WebSocket client ​ Output Frames ​ OutputAudioRawFrame Frame Audio data to be sent to the WebSocket client ​ Notes Supports real-time audio communication Best suited for server-side applications Handles WebSocket connection management Provides voice activity detection Supports session timeouts Single client per server (new connections replace existing ones) All callbacks are asynchronous FastAPI WebSocket Frame Serializer Overview On this page Overview Installation Configuration Constructor Parameters WebsocketServerParams Configuration Audio Configuration Voice Activity Detection (VAD) Basic Usage Event Callbacks Connection Events Frame Types Input Frames Output Frames Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/transport_websocket-server_d2494ffd.txt b/transport_websocket-server_d2494ffd.txt
new file mode 100644
index 0000000000000000000000000000000000000000..2e9f46cc427db88f651ce638f5ef78e63baca658
--- /dev/null
+++ b/transport_websocket-server_d2494ffd.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/transport/websocket-server#param-on-client-connected
+Title: WebSocket Server - Pipecat
+==================================================
+
+WebSocket Server - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Transport WebSocket Server Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Daily WebRTC SmallWebRTCTransport FastAPI WebSocket WebSocket Server Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview WebsocketServerTransport provides a WebSocket server implementation for real-time audio communication. It supports bidirectional audio streams and voice activity detection (VAD). WebsocketServerTransport is best suited for server-side applications and prototyping client/server apps. For client/server production applications, we strongly recommend using a WebRTC-based transport for robust network and media handling. ​ Installation To use WebsocketServerTransport , install the required dependencies: Copy Ask AI pip install "pipecat-ai[websocket]" ​ Configuration ​ Constructor Parameters ​ host str default: "localhost" Host address to bind the WebSocket server ​ port int default: "8765" Port number for the WebSocket server ​ params WebsocketServerParams default: "WebsocketServerParams()" Transport configuration parameters ​ WebsocketServerParams Configuration ​ add_wav_header bool default: "False" Add WAV header to audio frames ​ serializer FrameSerializer default: "ProtobufFrameSerializer()" Frame serializer for WebSocket messages ​ session_timeout int | None default: "None" Session timeout in seconds. If set, triggers timeout event when no activity is detected ​ Audio Configuration ​ audio_in_enabled bool default: false Enable audio input from the WebRTC client ​ audio_in_passthrough bool default: "False" When enabled, incoming audio frames are pushed downstream ​ audio_out_enabled bool default: "False" Enable audio output capabilities ​ audio_out_sample_rate int default: "None" Audio output sample rate in Hz ​ audio_out_channels int default: "1" Number of audio output channels ​ Voice Activity Detection (VAD) ​ vad_analyzer VADAnalyzer | None default: "None" Voice Activity Detection analyzer. You can set this to either SileroVADAnalyzer() or WebRTCVADAnalyzer() . SileroVADAnalyzer is the recommended option. Learn more about the SileroVADAnalyzer . ​ Basic Usage Copy Ask AI from pipecat.transports.network.websocket_server import ( WebsocketServerTransport, WebsocketServerParams ) from pipecat.audio.vad.silero import SileroVADAnalyzer from pipecat.pipeline.pipeline import Pipeline # Configure transport transport = WebsocketServerTransport( host = "localhost" , port = 8765 , params = WebsocketServerParams( audio_in_enabled = True , audio_out_enabled = True , add_wav_header = True , vad_analyzer = SileroVADAnalyzer(), session_timeout = 180 # 3 minutes ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), # Handle incoming audio stt, # Speech-to-text llm, # Language model tts, # Text-to-speech transport.output() # Handle outgoing audio ]) Check out the Websocket Server example to see how to use this transport in a pipeline. ​ Event Callbacks WebsocketServerTransport provides callbacks for handling client connection events. Register callbacks using the @transport.event_handler() decorator. ​ Connection Events ​ on_client_connected async callback Called when a client connects to the WebSocket server. Parameters: transport : The WebsocketServerTransport instance client : WebSocket client connection object Copy Ask AI @transport.event_handler ( "on_client_connected" ) async def on_client_connected ( transport , client ): logger.info( f "Client connected: { client.remote_address } " ) # Initialize conversation await task.queue_frames([LLMMessagesFrame(initial_messages)]) ​ on_client_disconnected async callback Called when a client disconnects from the WebSocket server. Parameters: transport : The WebsocketServerTransport instance client : WebSocket client connection object Copy Ask AI @transport.event_handler ( "on_client_disconnected" ) async def on_client_disconnected ( transport , client ): logger.info( f "Client disconnected: { client.remote_address } " ) ​ on_session_timeout async callback Called when a session times out (if session_timeout is configured). Parameters: transport : The WebsocketServerTransport instance client : WebSocket client connection object Copy Ask AI @transport.event_handler ( "on_session_timeout" ) async def on_session_timeout ( transport , client ): logger.info( f "Session timeout for client: { client.remote_address } " ) # Handle timeout (e.g., send message, close connection) ​ Frame Types ​ Input Frames ​ InputAudioRawFrame Frame Raw audio data from the WebSocket client ​ Output Frames ​ OutputAudioRawFrame Frame Audio data to be sent to the WebSocket client ​ Notes Supports real-time audio communication Best suited for server-side applications Handles WebSocket connection management Provides voice activity detection Supports session timeouts Single client per server (new connections replace existing ones) All callbacks are asynchronous FastAPI WebSocket Frame Serializer Overview On this page Overview Installation Configuration Constructor Parameters WebsocketServerParams Configuration Audio Configuration Voice Activity Detection (VAD) Basic Usage Event Callbacks Connection Events Frame Types Input Frames Output Frames Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/transport_websocket-server_db474ddd.txt b/transport_websocket-server_db474ddd.txt
new file mode 100644
index 0000000000000000000000000000000000000000..f6e5c370b73364d5481de1a968f1eb0cbf82af83
--- /dev/null
+++ b/transport_websocket-server_db474ddd.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/transport/websocket-server#param-audio-out-sample-rate
+Title: WebSocket Server - Pipecat
+==================================================
+
+WebSocket Server - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Transport WebSocket Server Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Daily WebRTC SmallWebRTCTransport FastAPI WebSocket WebSocket Server Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview WebsocketServerTransport provides a WebSocket server implementation for real-time audio communication. It supports bidirectional audio streams and voice activity detection (VAD). WebsocketServerTransport is best suited for server-side applications and prototyping client/server apps. For client/server production applications, we strongly recommend using a WebRTC-based transport for robust network and media handling. ​ Installation To use WebsocketServerTransport , install the required dependencies: Copy Ask AI pip install "pipecat-ai[websocket]" ​ Configuration ​ Constructor Parameters ​ host str default: "localhost" Host address to bind the WebSocket server ​ port int default: "8765" Port number for the WebSocket server ​ params WebsocketServerParams default: "WebsocketServerParams()" Transport configuration parameters ​ WebsocketServerParams Configuration ​ add_wav_header bool default: "False" Add WAV header to audio frames ​ serializer FrameSerializer default: "ProtobufFrameSerializer()" Frame serializer for WebSocket messages ​ session_timeout int | None default: "None" Session timeout in seconds. If set, triggers timeout event when no activity is detected ​ Audio Configuration ​ audio_in_enabled bool default: false Enable audio input from the WebRTC client ​ audio_in_passthrough bool default: "False" When enabled, incoming audio frames are pushed downstream ​ audio_out_enabled bool default: "False" Enable audio output capabilities ​ audio_out_sample_rate int default: "None" Audio output sample rate in Hz ​ audio_out_channels int default: "1" Number of audio output channels ​ Voice Activity Detection (VAD) ​ vad_analyzer VADAnalyzer | None default: "None" Voice Activity Detection analyzer. You can set this to either SileroVADAnalyzer() or WebRTCVADAnalyzer() . SileroVADAnalyzer is the recommended option. Learn more about the SileroVADAnalyzer . ​ Basic Usage Copy Ask AI from pipecat.transports.network.websocket_server import ( WebsocketServerTransport, WebsocketServerParams ) from pipecat.audio.vad.silero import SileroVADAnalyzer from pipecat.pipeline.pipeline import Pipeline # Configure transport transport = WebsocketServerTransport( host = "localhost" , port = 8765 , params = WebsocketServerParams( audio_in_enabled = True , audio_out_enabled = True , add_wav_header = True , vad_analyzer = SileroVADAnalyzer(), session_timeout = 180 # 3 minutes ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), # Handle incoming audio stt, # Speech-to-text llm, # Language model tts, # Text-to-speech transport.output() # Handle outgoing audio ]) Check out the Websocket Server example to see how to use this transport in a pipeline. ​ Event Callbacks WebsocketServerTransport provides callbacks for handling client connection events. Register callbacks using the @transport.event_handler() decorator. ​ Connection Events ​ on_client_connected async callback Called when a client connects to the WebSocket server. Parameters: transport : The WebsocketServerTransport instance client : WebSocket client connection object Copy Ask AI @transport.event_handler ( "on_client_connected" ) async def on_client_connected ( transport , client ): logger.info( f "Client connected: { client.remote_address } " ) # Initialize conversation await task.queue_frames([LLMMessagesFrame(initial_messages)]) ​ on_client_disconnected async callback Called when a client disconnects from the WebSocket server. Parameters: transport : The WebsocketServerTransport instance client : WebSocket client connection object Copy Ask AI @transport.event_handler ( "on_client_disconnected" ) async def on_client_disconnected ( transport , client ): logger.info( f "Client disconnected: { client.remote_address } " ) ​ on_session_timeout async callback Called when a session times out (if session_timeout is configured). Parameters: transport : The WebsocketServerTransport instance client : WebSocket client connection object Copy Ask AI @transport.event_handler ( "on_session_timeout" ) async def on_session_timeout ( transport , client ): logger.info( f "Session timeout for client: { client.remote_address } " ) # Handle timeout (e.g., send message, close connection) ​ Frame Types ​ Input Frames ​ InputAudioRawFrame Frame Raw audio data from the WebSocket client ​ Output Frames ​ OutputAudioRawFrame Frame Audio data to be sent to the WebSocket client ​ Notes Supports real-time audio communication Best suited for server-side applications Handles WebSocket connection management Provides voice activity detection Supports session timeouts Single client per server (new connections replace existing ones) All callbacks are asynchronous FastAPI WebSocket Frame Serializer Overview On this page Overview Installation Configuration Constructor Parameters WebsocketServerParams Configuration Audio Configuration Voice Activity Detection (VAD) Basic Usage Event Callbacks Connection Events Frame Types Input Frames Output Frames Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/transport_websocket-server_e92df993.txt b/transport_websocket-server_e92df993.txt
new file mode 100644
index 0000000000000000000000000000000000000000..3970f7cccea2b54c3212f7a154afc79a8d511d1f
--- /dev/null
+++ b/transport_websocket-server_e92df993.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/transport/websocket-server#param-add-wav-header
+Title: WebSocket Server - Pipecat
+==================================================
+
+WebSocket Server - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Transport WebSocket Server Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Daily WebRTC SmallWebRTCTransport FastAPI WebSocket WebSocket Server Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview WebsocketServerTransport provides a WebSocket server implementation for real-time audio communication. It supports bidirectional audio streams and voice activity detection (VAD). WebsocketServerTransport is best suited for server-side applications and prototyping client/server apps. For client/server production applications, we strongly recommend using a WebRTC-based transport for robust network and media handling. ​ Installation To use WebsocketServerTransport , install the required dependencies: Copy Ask AI pip install "pipecat-ai[websocket]" ​ Configuration ​ Constructor Parameters ​ host str default: "localhost" Host address to bind the WebSocket server ​ port int default: "8765" Port number for the WebSocket server ​ params WebsocketServerParams default: "WebsocketServerParams()" Transport configuration parameters ​ WebsocketServerParams Configuration ​ add_wav_header bool default: "False" Add WAV header to audio frames ​ serializer FrameSerializer default: "ProtobufFrameSerializer()" Frame serializer for WebSocket messages ​ session_timeout int | None default: "None" Session timeout in seconds. If set, triggers timeout event when no activity is detected ​ Audio Configuration ​ audio_in_enabled bool default: false Enable audio input from the WebRTC client ​ audio_in_passthrough bool default: "False" When enabled, incoming audio frames are pushed downstream ​ audio_out_enabled bool default: "False" Enable audio output capabilities ​ audio_out_sample_rate int default: "None" Audio output sample rate in Hz ​ audio_out_channels int default: "1" Number of audio output channels ​ Voice Activity Detection (VAD) ​ vad_analyzer VADAnalyzer | None default: "None" Voice Activity Detection analyzer. You can set this to either SileroVADAnalyzer() or WebRTCVADAnalyzer() . SileroVADAnalyzer is the recommended option. Learn more about the SileroVADAnalyzer . ​ Basic Usage Copy Ask AI from pipecat.transports.network.websocket_server import ( WebsocketServerTransport, WebsocketServerParams ) from pipecat.audio.vad.silero import SileroVADAnalyzer from pipecat.pipeline.pipeline import Pipeline # Configure transport transport = WebsocketServerTransport( host = "localhost" , port = 8765 , params = WebsocketServerParams( audio_in_enabled = True , audio_out_enabled = True , add_wav_header = True , vad_analyzer = SileroVADAnalyzer(), session_timeout = 180 # 3 minutes ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), # Handle incoming audio stt, # Speech-to-text llm, # Language model tts, # Text-to-speech transport.output() # Handle outgoing audio ]) Check out the Websocket Server example to see how to use this transport in a pipeline. ​ Event Callbacks WebsocketServerTransport provides callbacks for handling client connection events. Register callbacks using the @transport.event_handler() decorator. ​ Connection Events ​ on_client_connected async callback Called when a client connects to the WebSocket server. Parameters: transport : The WebsocketServerTransport instance client : WebSocket client connection object Copy Ask AI @transport.event_handler ( "on_client_connected" ) async def on_client_connected ( transport , client ): logger.info( f "Client connected: { client.remote_address } " ) # Initialize conversation await task.queue_frames([LLMMessagesFrame(initial_messages)]) ​ on_client_disconnected async callback Called when a client disconnects from the WebSocket server. Parameters: transport : The WebsocketServerTransport instance client : WebSocket client connection object Copy Ask AI @transport.event_handler ( "on_client_disconnected" ) async def on_client_disconnected ( transport , client ): logger.info( f "Client disconnected: { client.remote_address } " ) ​ on_session_timeout async callback Called when a session times out (if session_timeout is configured). Parameters: transport : The WebsocketServerTransport instance client : WebSocket client connection object Copy Ask AI @transport.event_handler ( "on_session_timeout" ) async def on_session_timeout ( transport , client ): logger.info( f "Session timeout for client: { client.remote_address } " ) # Handle timeout (e.g., send message, close connection) ​ Frame Types ​ Input Frames ​ InputAudioRawFrame Frame Raw audio data from the WebSocket client ​ Output Frames ​ OutputAudioRawFrame Frame Audio data to be sent to the WebSocket client ​ Notes Supports real-time audio communication Best suited for server-side applications Handles WebSocket connection management Provides voice activity detection Supports session timeouts Single client per server (new connections replace existing ones) All callbacks are asynchronous FastAPI WebSocket Frame Serializer Overview On this page Overview Installation Configuration Constructor Parameters WebsocketServerParams Configuration Audio Configuration Voice Activity Detection (VAD) Basic Usage Event Callbacks Connection Events Frame Types Input Frames Output Frames Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/transport_websocket-server_f83e8396.txt b/transport_websocket-server_f83e8396.txt
new file mode 100644
index 0000000000000000000000000000000000000000..2c8e39c3212245b5e53318120f690253d1a3369e
--- /dev/null
+++ b/transport_websocket-server_f83e8396.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/transport/websocket-server#input-frames
+Title: WebSocket Server - Pipecat
+==================================================
+
+WebSocket Server - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Transport WebSocket Server Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Daily WebRTC SmallWebRTCTransport FastAPI WebSocket WebSocket Server Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview WebsocketServerTransport provides a WebSocket server implementation for real-time audio communication. It supports bidirectional audio streams and voice activity detection (VAD). WebsocketServerTransport is best suited for server-side applications and prototyping client/server apps. For client/server production applications, we strongly recommend using a WebRTC-based transport for robust network and media handling. ​ Installation To use WebsocketServerTransport , install the required dependencies: Copy Ask AI pip install "pipecat-ai[websocket]" ​ Configuration ​ Constructor Parameters ​ host str default: "localhost" Host address to bind the WebSocket server ​ port int default: "8765" Port number for the WebSocket server ​ params WebsocketServerParams default: "WebsocketServerParams()" Transport configuration parameters ​ WebsocketServerParams Configuration ​ add_wav_header bool default: "False" Add WAV header to audio frames ​ serializer FrameSerializer default: "ProtobufFrameSerializer()" Frame serializer for WebSocket messages ​ session_timeout int | None default: "None" Session timeout in seconds. If set, triggers timeout event when no activity is detected ​ Audio Configuration ​ audio_in_enabled bool default: false Enable audio input from the WebRTC client ​ audio_in_passthrough bool default: "False" When enabled, incoming audio frames are pushed downstream ​ audio_out_enabled bool default: "False" Enable audio output capabilities ​ audio_out_sample_rate int default: "None" Audio output sample rate in Hz ​ audio_out_channels int default: "1" Number of audio output channels ​ Voice Activity Detection (VAD) ​ vad_analyzer VADAnalyzer | None default: "None" Voice Activity Detection analyzer. You can set this to either SileroVADAnalyzer() or WebRTCVADAnalyzer() . SileroVADAnalyzer is the recommended option. Learn more about the SileroVADAnalyzer . ​ Basic Usage Copy Ask AI from pipecat.transports.network.websocket_server import ( WebsocketServerTransport, WebsocketServerParams ) from pipecat.audio.vad.silero import SileroVADAnalyzer from pipecat.pipeline.pipeline import Pipeline # Configure transport transport = WebsocketServerTransport( host = "localhost" , port = 8765 , params = WebsocketServerParams( audio_in_enabled = True , audio_out_enabled = True , add_wav_header = True , vad_analyzer = SileroVADAnalyzer(), session_timeout = 180 # 3 minutes ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), # Handle incoming audio stt, # Speech-to-text llm, # Language model tts, # Text-to-speech transport.output() # Handle outgoing audio ]) Check out the Websocket Server example to see how to use this transport in a pipeline. ​ Event Callbacks WebsocketServerTransport provides callbacks for handling client connection events. Register callbacks using the @transport.event_handler() decorator. ​ Connection Events ​ on_client_connected async callback Called when a client connects to the WebSocket server. Parameters: transport : The WebsocketServerTransport instance client : WebSocket client connection object Copy Ask AI @transport.event_handler ( "on_client_connected" ) async def on_client_connected ( transport , client ): logger.info( f "Client connected: { client.remote_address } " ) # Initialize conversation await task.queue_frames([LLMMessagesFrame(initial_messages)]) ​ on_client_disconnected async callback Called when a client disconnects from the WebSocket server. Parameters: transport : The WebsocketServerTransport instance client : WebSocket client connection object Copy Ask AI @transport.event_handler ( "on_client_disconnected" ) async def on_client_disconnected ( transport , client ): logger.info( f "Client disconnected: { client.remote_address } " ) ​ on_session_timeout async callback Called when a session times out (if session_timeout is configured). Parameters: transport : The WebsocketServerTransport instance client : WebSocket client connection object Copy Ask AI @transport.event_handler ( "on_session_timeout" ) async def on_session_timeout ( transport , client ): logger.info( f "Session timeout for client: { client.remote_address } " ) # Handle timeout (e.g., send message, close connection) ​ Frame Types ​ Input Frames ​ InputAudioRawFrame Frame Raw audio data from the WebSocket client ​ Output Frames ​ OutputAudioRawFrame Frame Audio data to be sent to the WebSocket client ​ Notes Supports real-time audio communication Best suited for server-side applications Handles WebSocket connection management Provides voice activity detection Supports session timeouts Single client per server (new connections replace existing ones) All callbacks are asynchronous FastAPI WebSocket Frame Serializer Overview On this page Overview Installation Configuration Constructor Parameters WebsocketServerParams Configuration Audio Configuration Voice Activity Detection (VAD) Basic Usage Event Callbacks Connection Events Frame Types Input Frames Output Frames Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/transports_daily_27e5cecd.txt b/transports_daily_27e5cecd.txt
new file mode 100644
index 0000000000000000000000000000000000000000..0f20fe60f46c9087601db4db9e4187c93d0f504e
--- /dev/null
+++ b/transports_daily_27e5cecd.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/client/android/transports/daily#configuration
+Title: Daily WebRTC Transport - Pipecat
+==================================================
+
+Daily WebRTC Transport - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Transport packages Daily WebRTC Transport Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Client SDKs The RTVI Standard RTVIClient Migration Guide Javascript SDK SDK Introduction API Reference Transport packages React SDK SDK Introduction API Reference React Native SDK SDK Introduction API Reference iOS SDK SDK Introduction API Reference Transport packages Android SDK SDK Introduction API Reference Transport packages Daily WebRTC Transport Gemini Live Websocket Transport OpenAI Realtime WebRTC Transport Small WebRTC Transport C++ SDK SDK Introduction Daily WebRTC Transport The Daily transport implementation enables real-time audio and video communication in your Pipecat Android applications using Daily’s WebRTC infrastructure. ​ Installation Add the Daily transport dependency to your build.gradle : Copy Ask AI implementation "ai.pipecat:daily-transport:0.3.7" ​ Usage Create a client using the Daily transport: Copy Ask AI val callbacks = object : RTVIEventCallbacks () { override fun onBackendError (message: String ) { Log. e (TAG, "Error from backend: $message " ) } } val options = RTVIClientOptions ( services = listOf ( ServiceRegistration ( "llm" , "together" ), ServiceRegistration ( "tts" , "cartesia" )), params = RTVIClientParams (baseUrl = "<your API url>" , config = clientConfig) ) val client = RTVIClient (DailyTransport. Factory (context), callbacks, options) client. connect (). await () ​ Configuration Your server endpoint should return Daily-specific configuration: Copy Ask AI { "url" : "https://your-domain.daily.co/room-name" , "token" : "your-daily-token" } ​ Resources Demo Simple Chatbot Demo Source Client Transports Daily Transport Reference Complete API documentation for the Daily transport implementation API Reference Gemini Live Websocket Transport On this page Installation Usage Configuration Resources Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/transports_daily_5a3870bd.txt b/transports_daily_5a3870bd.txt
new file mode 100644
index 0000000000000000000000000000000000000000..014563ea44c3b6f97579fc3fc83cc2bd01a57011
--- /dev/null
+++ b/transports_daily_5a3870bd.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/client/android/transports/daily
+Title: Daily WebRTC Transport - Pipecat
+==================================================
+
+Daily WebRTC Transport - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Transport packages Daily WebRTC Transport Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Client SDKs The RTVI Standard RTVIClient Migration Guide Javascript SDK SDK Introduction API Reference Transport packages React SDK SDK Introduction API Reference React Native SDK SDK Introduction API Reference iOS SDK SDK Introduction API Reference Transport packages Android SDK SDK Introduction API Reference Transport packages Daily WebRTC Transport Gemini Live Websocket Transport OpenAI Realtime WebRTC Transport Small WebRTC Transport C++ SDK SDK Introduction Daily WebRTC Transport The Daily transport implementation enables real-time audio and video communication in your Pipecat Android applications using Daily’s WebRTC infrastructure. ​ Installation Add the Daily transport dependency to your build.gradle : Copy Ask AI implementation "ai.pipecat:daily-transport:0.3.7" ​ Usage Create a client using the Daily transport: Copy Ask AI val callbacks = object : RTVIEventCallbacks () { override fun onBackendError (message: String ) { Log. e (TAG, "Error from backend: $message " ) } } val options = RTVIClientOptions ( services = listOf ( ServiceRegistration ( "llm" , "together" ), ServiceRegistration ( "tts" , "cartesia" )), params = RTVIClientParams (baseUrl = "<your API url>" , config = clientConfig) ) val client = RTVIClient (DailyTransport. Factory (context), callbacks, options) client. connect (). await () ​ Configuration Your server endpoint should return Daily-specific configuration: Copy Ask AI { "url" : "https://your-domain.daily.co/room-name" , "token" : "your-daily-token" } ​ Resources Demo Simple Chatbot Demo Source Client Transports Daily Transport Reference Complete API documentation for the Daily transport implementation API Reference Gemini Live Websocket Transport On this page Installation Usage Configuration Resources Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/transports_daily_c5ef21a6.txt b/transports_daily_c5ef21a6.txt
new file mode 100644
index 0000000000000000000000000000000000000000..46c69b02a3ccbbc37a4c33694beebcdbca16d889
--- /dev/null
+++ b/transports_daily_c5ef21a6.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/client/ios/transports/daily#installation
+Title: Daily WebRTC Transport - Pipecat
+==================================================
+
+Daily WebRTC Transport - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Transport packages Daily WebRTC Transport Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Client SDKs The RTVI Standard RTVIClient Migration Guide Javascript SDK SDK Introduction API Reference Transport packages React SDK SDK Introduction API Reference React Native SDK SDK Introduction API Reference iOS SDK SDK Introduction API Reference Transport packages Daily WebRTC Transport Gemini Live Websocket Transport OpenAIRealTimeWebRTCTransport Android SDK SDK Introduction API Reference Transport packages C++ SDK SDK Introduction Daily WebRTC Transport The Daily transport implementation enables real-time audio and video communication in your Pipecat iOS applications using Daily’s WebRTC infrastructure. ​ Installation Add the Daily transport package to your project: Copy Ask AI . package ( url : "https://github.com/pipecat-ai/pipecat-client-ios-daily.git" , from : "0.3.0" ) // Add to your target dependencies . target ( name : "YourApp" , dependencies : [ . product ( name : "PipecatClientIOSDaily" , package : "pipecat-client-ios-daily" ) ]) ​ Usage Create a client using the Daily transport: Copy Ask AI import PipecatClientIOS import PipecatClientIOSDaily let configOptions = RTVIClientOptions. init ( enableMic : true , params : RTVIClientParams ( baseUrl : $PIPECAT_API_URL ) ) let client = RTVIClient. init ( transport : DailyTransport. init ( options : configOptions), options : configOptions ) try await client. start () ​ Configuration Your server endpoint should return Daily-specific configuration: Copy Ask AI // Example server response { "url" : "https://your-domain.daily.co/room-name" , "token" : "your-daily-token" } ​ API Reference Demo Simple Chatbot Demo Source Daily Transport Daily Transport Reference Complete API documentation for the Daily transport implementation API Reference Gemini Live Websocket Transport On this page Installation Usage Configuration API Reference Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/transports_daily_f249e137.txt b/transports_daily_f249e137.txt
new file mode 100644
index 0000000000000000000000000000000000000000..d5e6bb5e5b01c1da26aa32a1f68367ef250a6ed2
--- /dev/null
+++ b/transports_daily_f249e137.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/client/ios/transports/daily
+Title: Daily WebRTC Transport - Pipecat
+==================================================
+
+Daily WebRTC Transport - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Transport packages Daily WebRTC Transport Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Client SDKs The RTVI Standard RTVIClient Migration Guide Javascript SDK SDK Introduction API Reference Transport packages React SDK SDK Introduction API Reference React Native SDK SDK Introduction API Reference iOS SDK SDK Introduction API Reference Transport packages Daily WebRTC Transport Gemini Live Websocket Transport OpenAIRealTimeWebRTCTransport Android SDK SDK Introduction API Reference Transport packages C++ SDK SDK Introduction Daily WebRTC Transport The Daily transport implementation enables real-time audio and video communication in your Pipecat iOS applications using Daily’s WebRTC infrastructure. ​ Installation Add the Daily transport package to your project: Copy Ask AI . package ( url : "https://github.com/pipecat-ai/pipecat-client-ios-daily.git" , from : "0.3.0" ) // Add to your target dependencies . target ( name : "YourApp" , dependencies : [ . product ( name : "PipecatClientIOSDaily" , package : "pipecat-client-ios-daily" ) ]) ​ Usage Create a client using the Daily transport: Copy Ask AI import PipecatClientIOS import PipecatClientIOSDaily let configOptions = RTVIClientOptions. init ( enableMic : true , params : RTVIClientParams ( baseUrl : $PIPECAT_API_URL ) ) let client = RTVIClient. init ( transport : DailyTransport. init ( options : configOptions), options : configOptions ) try await client. start () ​ Configuration Your server endpoint should return Daily-specific configuration: Copy Ask AI // Example server response { "url" : "https://your-domain.daily.co/room-name" , "token" : "your-daily-token" } ​ API Reference Demo Simple Chatbot Demo Source Daily Transport Daily Transport Reference Complete API documentation for the Daily transport implementation API Reference Gemini Live Websocket Transport On this page Installation Usage Configuration API Reference Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/transports_daily_fd6754ce.txt b/transports_daily_fd6754ce.txt
new file mode 100644
index 0000000000000000000000000000000000000000..92c3a9896e5262ab0a741c99aeff754e13af08a3
--- /dev/null
+++ b/transports_daily_fd6754ce.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/client/ios/transports/daily#usage
+Title: Daily WebRTC Transport - Pipecat
+==================================================
+
+Daily WebRTC Transport - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Transport packages Daily WebRTC Transport Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Client SDKs The RTVI Standard RTVIClient Migration Guide Javascript SDK SDK Introduction API Reference Transport packages React SDK SDK Introduction API Reference React Native SDK SDK Introduction API Reference iOS SDK SDK Introduction API Reference Transport packages Daily WebRTC Transport Gemini Live Websocket Transport OpenAIRealTimeWebRTCTransport Android SDK SDK Introduction API Reference Transport packages C++ SDK SDK Introduction Daily WebRTC Transport The Daily transport implementation enables real-time audio and video communication in your Pipecat iOS applications using Daily’s WebRTC infrastructure. ​ Installation Add the Daily transport package to your project: Copy Ask AI . package ( url : "https://github.com/pipecat-ai/pipecat-client-ios-daily.git" , from : "0.3.0" ) // Add to your target dependencies . target ( name : "YourApp" , dependencies : [ . product ( name : "PipecatClientIOSDaily" , package : "pipecat-client-ios-daily" ) ]) ​ Usage Create a client using the Daily transport: Copy Ask AI import PipecatClientIOS import PipecatClientIOSDaily let configOptions = RTVIClientOptions. init ( enableMic : true , params : RTVIClientParams ( baseUrl : $PIPECAT_API_URL ) ) let client = RTVIClient. init ( transport : DailyTransport. init ( options : configOptions), options : configOptions ) try await client. start () ​ Configuration Your server endpoint should return Daily-specific configuration: Copy Ask AI // Example server response { "url" : "https://your-domain.daily.co/room-name" , "token" : "your-daily-token" } ​ API Reference Demo Simple Chatbot Demo Source Daily Transport Daily Transport Reference Complete API documentation for the Daily transport implementation API Reference Gemini Live Websocket Transport On this page Installation Usage Configuration API Reference Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/transports_daily_fe338524.txt b/transports_daily_fe338524.txt
new file mode 100644
index 0000000000000000000000000000000000000000..5eea6e75c921615f47d387ee8bd3e2bc327d3f0f
--- /dev/null
+++ b/transports_daily_fe338524.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/client/android/transports/daily#usage
+Title: Daily WebRTC Transport - Pipecat
+==================================================
+
+Daily WebRTC Transport - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Transport packages Daily WebRTC Transport Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Client SDKs The RTVI Standard RTVIClient Migration Guide Javascript SDK SDK Introduction API Reference Transport packages React SDK SDK Introduction API Reference React Native SDK SDK Introduction API Reference iOS SDK SDK Introduction API Reference Transport packages Android SDK SDK Introduction API Reference Transport packages Daily WebRTC Transport Gemini Live Websocket Transport OpenAI Realtime WebRTC Transport Small WebRTC Transport C++ SDK SDK Introduction Daily WebRTC Transport The Daily transport implementation enables real-time audio and video communication in your Pipecat Android applications using Daily’s WebRTC infrastructure. ​ Installation Add the Daily transport dependency to your build.gradle : Copy Ask AI implementation "ai.pipecat:daily-transport:0.3.7" ​ Usage Create a client using the Daily transport: Copy Ask AI val callbacks = object : RTVIEventCallbacks () { override fun onBackendError (message: String ) { Log. e (TAG, "Error from backend: $message " ) } } val options = RTVIClientOptions ( services = listOf ( ServiceRegistration ( "llm" , "together" ), ServiceRegistration ( "tts" , "cartesia" )), params = RTVIClientParams (baseUrl = "<your API url>" , config = clientConfig) ) val client = RTVIClient (DailyTransport. Factory (context), callbacks, options) client. connect (). await () ​ Configuration Your server endpoint should return Daily-specific configuration: Copy Ask AI { "url" : "https://your-domain.daily.co/room-name" , "token" : "your-daily-token" } ​ Resources Demo Simple Chatbot Demo Source Client Transports Daily Transport Reference Complete API documentation for the Daily transport implementation API Reference Gemini Live Websocket Transport On this page Installation Usage Configuration Resources Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/transports_gemini-websocket_0b58dabe.txt b/transports_gemini-websocket_0b58dabe.txt
new file mode 100644
index 0000000000000000000000000000000000000000..f6e41e38a0abae8cf53571337af492a4efe00dbc
--- /dev/null
+++ b/transports_gemini-websocket_0b58dabe.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/client/ios/transports/gemini-websocket#installation
+Title: Gemini Live Websocket Transport - Pipecat
+==================================================
+
+Gemini Live Websocket Transport - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Transport packages Gemini Live Websocket Transport Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Client SDKs The RTVI Standard RTVIClient Migration Guide Javascript SDK SDK Introduction API Reference Transport packages React SDK SDK Introduction API Reference React Native SDK SDK Introduction API Reference iOS SDK SDK Introduction API Reference Transport packages Daily WebRTC Transport Gemini Live Websocket Transport OpenAIRealTimeWebRTCTransport Android SDK SDK Introduction API Reference Transport packages C++ SDK SDK Introduction Daily WebRTC Transport The Gemini Live Websocket transport implementation enables real-time audio communication with the Gemini Multimodal Live service, using a direct websocket connection. Transports of this type are designed primarily for development and testing purposes. For production applications, you will need to build a server component with a server-friendly transport, like the DailyTransport , to securely handle API keys. ​ Installation Add the Gemini transport package to your project: Copy Ask AI . package ( url : "https://github.com/pipecat-ai/pipecat-client-ios-gemini-live-websocket.git" , from : "0.3.1" ), // Add to your target dependencies . target ( name : "YourApp" , dependencies : [ . product ( name : "PipecatClientIOSGeminiLiveWebSocket" , package : "pipecat-client-ios-gemini-live-websocket" ) ], ​ Usage Create a client: Copy Ask AI let options: RTVIClientOptions = . init ( params : . init ( config : [ . init ( service : "llm" , options : [ . init ( name : "api_key" , value : . string ( "<your Gemini api key>" )), . init ( name : "initial_messages" , value : . array ([ . object ([ "role" : . string ( "user" ), // "user" | "system" "content" : . string ( "I need your help planning my next vacation." ) ]) ])), . init ( name : "generation_config" , value : . object ([ "speech_config" : . object ([ "voice_config" : . object ([ "prebuilt_voice_config" : . object ([ "voice_name" : . string ( "Puck" ) // "Puck" | "Charon" | "Kore" | "Fenrir" | "Aoede" ]) ]) ]) ])) ] ) ]) ) let client = GeminiLiveWebSocketVoiceClient ( options : options) try await client. start () ​ API Reference Demo Simple Chatbot Gemini Demo Source iOS Gemini Live WebSocket Pipecat iOS Client Reference Complete API documentation for the Gemini transport implementation Daily WebRTC Transport OpenAIRealTimeWebRTCTransport On this page Installation Usage API Reference Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/transports_gemini-websocket_9e2c6297.txt b/transports_gemini-websocket_9e2c6297.txt
new file mode 100644
index 0000000000000000000000000000000000000000..f05ea3ca51e1ebe1e3a99c0b13f13d28c8814f6f
--- /dev/null
+++ b/transports_gemini-websocket_9e2c6297.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/client/ios/transports/gemini-websocket
+Title: Gemini Live Websocket Transport - Pipecat
+==================================================
+
+Gemini Live Websocket Transport - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Transport packages Gemini Live Websocket Transport Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Client SDKs The RTVI Standard RTVIClient Migration Guide Javascript SDK SDK Introduction API Reference Transport packages React SDK SDK Introduction API Reference React Native SDK SDK Introduction API Reference iOS SDK SDK Introduction API Reference Transport packages Daily WebRTC Transport Gemini Live Websocket Transport OpenAIRealTimeWebRTCTransport Android SDK SDK Introduction API Reference Transport packages C++ SDK SDK Introduction Daily WebRTC Transport The Gemini Live Websocket transport implementation enables real-time audio communication with the Gemini Multimodal Live service, using a direct websocket connection. Transports of this type are designed primarily for development and testing purposes. For production applications, you will need to build a server component with a server-friendly transport, like the DailyTransport , to securely handle API keys. ​ Installation Add the Gemini transport package to your project: Copy Ask AI . package ( url : "https://github.com/pipecat-ai/pipecat-client-ios-gemini-live-websocket.git" , from : "0.3.1" ), // Add to your target dependencies . target ( name : "YourApp" , dependencies : [ . product ( name : "PipecatClientIOSGeminiLiveWebSocket" , package : "pipecat-client-ios-gemini-live-websocket" ) ], ​ Usage Create a client: Copy Ask AI let options: RTVIClientOptions = . init ( params : . init ( config : [ . init ( service : "llm" , options : [ . init ( name : "api_key" , value : . string ( "<your Gemini api key>" )), . init ( name : "initial_messages" , value : . array ([ . object ([ "role" : . string ( "user" ), // "user" | "system" "content" : . string ( "I need your help planning my next vacation." ) ]) ])), . init ( name : "generation_config" , value : . object ([ "speech_config" : . object ([ "voice_config" : . object ([ "prebuilt_voice_config" : . object ([ "voice_name" : . string ( "Puck" ) // "Puck" | "Charon" | "Kore" | "Fenrir" | "Aoede" ]) ]) ]) ])) ] ) ]) ) let client = GeminiLiveWebSocketVoiceClient ( options : options) try await client. start () ​ API Reference Demo Simple Chatbot Gemini Demo Source iOS Gemini Live WebSocket Pipecat iOS Client Reference Complete API documentation for the Gemini transport implementation Daily WebRTC Transport OpenAIRealTimeWebRTCTransport On this page Installation Usage API Reference Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/transports_openai-webrtc_3da73649.txt b/transports_openai-webrtc_3da73649.txt
new file mode 100644
index 0000000000000000000000000000000000000000..23f5cdbdb61e596b97ca445eda12c62c0b96b4fb
--- /dev/null
+++ b/transports_openai-webrtc_3da73649.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/client/android/transports/openai-webrtc#usage
+Title: OpenAI Realtime WebRTC Transport - Pipecat
+==================================================
+
+OpenAI Realtime WebRTC Transport - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Transport packages OpenAI Realtime WebRTC Transport Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Client SDKs The RTVI Standard RTVIClient Migration Guide Javascript SDK SDK Introduction API Reference Transport packages React SDK SDK Introduction API Reference React Native SDK SDK Introduction API Reference iOS SDK SDK Introduction API Reference Transport packages Android SDK SDK Introduction API Reference Transport packages Daily WebRTC Transport Gemini Live Websocket Transport OpenAI Realtime WebRTC Transport Small WebRTC Transport C++ SDK SDK Introduction Daily WebRTC Transport The OpenAI Realtime WebRTC transport implementation enables real-time audio communication with the OpenAI Realtime service, using a direct WebRTC connection. ​ Installation Add the transport dependency to your build.gradle : Copy Ask AI implementation "ai.pipecat:openai-realtime-webrtc-transport:0.3.7" ​ Usage Create a client: Copy Ask AI val transport = OpenAIRealtimeWebRTCTransport. Factory (context) val options = RTVIClientOptions ( params = RTVIClientParams ( baseUrl = null , config = OpenAIRealtimeWebRTCTransport. buildConfig ( apiKey = apiKey, initialMessages = listOf ( LLMContextMessage (role = "user" , content = "How tall is the Eiffel Tower?" ) ), initialConfig = OpenAIRealtimeSessionConfig ( voice = "ballad" , turnDetection = Value. Object ( "type" to Value. Str ( "semantic_vad" )), inputAudioNoiseReduction = Value. Object ( "type" to Value. Str ( "near_field" )), inputAudioTranscription = Value. Object ( "model" to Value. Str ( "gpt-4o-transcribe" )) ) ) ) ) val client = RTVIClient (transport, callbacks, options) client. start (). withCallback { // ... } ​ Resources Demo Simple Chatbot Demo Source Client Transports Pipecat Android Client Reference Complete API documentation for the Pipecat Android client. Gemini Live Websocket Transport Small WebRTC Transport On this page Installation Usage Resources Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/transports_openai-webrtc_4867a1b6.txt b/transports_openai-webrtc_4867a1b6.txt
new file mode 100644
index 0000000000000000000000000000000000000000..1455015cbcb79352f1fa3eded6b5014671874c9b
--- /dev/null
+++ b/transports_openai-webrtc_4867a1b6.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/client/ios/transports/openai-webrtc#overview
+Title: OpenAIRealTimeWebRTCTransport - Pipecat
+==================================================
+
+OpenAIRealTimeWebRTCTransport - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Transport packages OpenAIRealTimeWebRTCTransport Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Client SDKs The RTVI Standard RTVIClient Migration Guide Javascript SDK SDK Introduction API Reference Transport packages React SDK SDK Introduction API Reference React Native SDK SDK Introduction API Reference iOS SDK SDK Introduction API Reference Transport packages Daily WebRTC Transport Gemini Live Websocket Transport OpenAIRealTimeWebRTCTransport Android SDK SDK Introduction API Reference Transport packages C++ SDK SDK Introduction Daily WebRTC Transport ​ Overview The OpenAI Realtime WebRTC transport implementation enables real-time audio communication directly with the OpenAI Realtime API using WebRTC voice-to-voice service. It handles media device management, audio/video streams, and state management for the connection. ​ Installation Add the OpenAI transport package to your project: Copy Ask AI . package ( url : "https://github.com/pipecat-ai/pipecat-client-ios-openai-realtime.git" , from : "0.0.1" ), // Add to your target dependencies . target ( name : "YourApp" , dependencies : [ . product ( name : "PipecatClientIOSOpenAIRealtimeWebrtc" , package : "pipecat-client-ios-openai-realtime" ) ], ​ Usage Create a client: Copy Ask AI let rtviClientOptions = RTVIClientOptions. init ( enableMic : currentSettings. enableMic , enableCam : false , params : . init ( config : [ . init ( service : "llm" , options : [ . init ( name : "api_key" , value : . string (openaiAPIKey)), . init ( name : "initial_messages" , value : . array ([ . object ([ "role" : . string ( "user" ), // "user" | "system" "content" : . string ( "Start by introducing yourself." ) ]) ])), . init ( name : "session_config" , value : . object ([ "instructions" : . string ( "You are Chatbot, a friendly and helpful assistant who provides useful information, including weather updates." ), "voice" : . string ( "echo" ), "input_audio_noise_reduction" : . object ([ "type" : . string ( "near_field" ) ]), "turn_detection" : . object ([ "type" : . string ( "semantic_vad" ) ]) ])), ] ) ]) ) self . rtviClientIOS = RTVIClient. init ( transport : OpenAIRealtimeTransport. init ( options : rtviClientOptions), options : rtviClientOptions ) try await rtviClientIOS. start () Currently, invalid session configurations will result in the OpenAI connection being failed. ​ API Reference Demo Simple Chatbot OpenAI Demo Source iOS OpenAI Realtime WebRTC Pipecat iOS Client Reference Complete API documentation for the OpenAI transport implementation Gemini Live Websocket Transport SDK Introduction On this page Overview Installation Usage API Reference Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_aws_02a03c44.txt b/tts_aws_02a03c44.txt
new file mode 100644
index 0000000000000000000000000000000000000000..cbbadeae23b398b0f2f5e315b0c4a652c19712e8
--- /dev/null
+++ b/tts_aws_02a03c44.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/aws#output-frames
+Title: AWS Polly - Pipecat
+==================================================
+
+AWS Polly - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech AWS Polly Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview AWSPollyTTSService provides text-to-speech capabilities using AWS’s Polly service. It supports multiple voices, languages, and speech customization options through SSML. The older PollyTTSService class is still available but has been deprecated. Use AWSPollyTTSService instead. ​ Installation To use AWSPollyTTSService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[aws]" You’ll also need to set up your AWS credentials as environment variables: AWS_SECRET_ACCESS_KEY AWS_ACCESS_KEY_ID AWS_SESSION_TOKEN (if using temporary credentials) AWS_REGION (defaults to “us-east-1”) ​ Configuration ​ Constructor Parameters ​ api_key str AWS secret access key (can also use environment variable) ​ aws_access_key_id str AWS access key ID (can also use environment variable) ​ aws_session_token str AWS session token for temporary credentials (can also use environment variable) ​ region str AWS region name (defaults to “us-east-1” if not provided) ​ voice_id str default: "Joanna" AWS Polly voice identifier ​ sample_rate int default: "None" Output audio sample rate in Hz (resampled from Polly’s 16kHz) ​ text_filter BaseTextFilter default: "None" Modifies text provided to the TTS. Learn more about the available filters. ​ params InputParams TTS configuration parameters ​ Input Parameters Copy Ask AI class InputParams ( BaseModel ): engine: Optional[ str ] = None # Polly engine type ("standard", "neural", or "generative") language: Optional[Language] = Language. EN pitch: Optional[ str ] = None # SSML pitch adjustment rate: Optional[ str ] = None # SSML rate adjustment volume: Optional[ str ] = None # SSML volume adjustment ​ Output Frames ​ Control Frames ​ TTSStartedFrame Frame Signals start of speech synthesis ​ TTSStoppedFrame Frame Signals completion of speech synthesis ​ Audio Frames ​ TTSAudioRawFrame Frame Contains generated audio data with: PCM audio format Sample rate as specified (resampled from 16kHz) Single channel (mono) ​ Error Frames ​ ErrorFrame Frame Contains AWS Polly error information ​ Methods See the TTS base class methods for additional functionality. ​ Language Support Supports an extensive range of languages and regional variants: Language Code Description Service Code Language.AR Arabic arb Language.AR_AE Arabic (UAE) ar-AE Language.CA Catalan ca-ES Language.ZH Chinese (Mandarin) cmn-CN Language.YUE Chinese (Cantonese) yue-CN Language.YUE_CN Chinese (Cantonese) yue-CN Language.CS Czech cs-CZ Language.DA Danish da-DK Language.NL Dutch nl-NL Language.NL_BE Dutch (Belgium) nl-BE Language.EN English (US) en-US Language.EN_AU English (Australia) en-AU Language.EN_GB English (UK) en-GB Language.EN_IN English (India) en-IN Language.EN_NZ English (New Zealand) en-NZ Language.EN_US English (US) en-US Language.EN_ZA English (South Africa) en-ZA Language.FI Finnish fi-FI Language.FR French fr-FR Language.FR_BE French (Belgium) fr-BE Language.FR_CA French (Canada) fr-CA Language.DE German de-DE Language.DE_AT German (Austria) de-AT Language.DE_CH German (Switzerland) de-CH Language.HI Hindi hi-IN Language.IS Icelandic is-IS Language.IT Italian it-IT Language.JA Japanese ja-JP Language.KO Korean ko-KR Language.NO Norwegian nb-NO Language.NB Norwegian (Bokmål) nb-NO Language.NB_NO Norwegian (Bokmål) nb-NO Language.PL Polish pl-PL Language.PT Portuguese pt-PT Language.PT_BR Portuguese (Brazil) pt-BR Language.PT_PT Portuguese (Portugal) pt-PT Language.RO Romanian ro-RO Language.RU Russian ru-RU Language.ES Spanish es-ES Language.ES_MX Spanish (Mexico) es-MX Language.ES_US Spanish (US) es-US Language.SV Swedish sv-SE Language.TR Turkish tr-TR Language.CY Welsh cy-GB Language.CY_GB Welsh cy-GB ​ Usage Example Copy Ask AI from pipecat.services.aws.tts import AWSPollyTTSService from pipecat.transcriptions.language import Language # Configure service using environment variables for credentials tts = AWSPollyTTSService( region = "us-west-2" , voice_id = "Joanna" , params = AWSPollyTTSService.InputParams( engine = "neural" , language = Language. EN , rate = "+10%" , volume = "loud" ) ) # Or provide credentials directly tts = AWSPollyTTSService( aws_access_key_id = "YOUR_ACCESS_KEY_ID" , api_key = "YOUR_SECRET_ACCESS_KEY" , region = "us-west-2" , voice_id = "Joanna" , params = AWSPollyTTSService.InputParams( engine = "generative" , # For newer generative voices language = Language. EN , rate = "1.1" # Generative engine rate format ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ SSML Support The service automatically constructs SSML tags for advanced speech control: Copy Ask AI # Example with SSML controls service = AWSPollyTTSService( # ... other params ... params = AWSPollyTTSService.InputParams( engine = "neural" , rate = "+20%" , # Increase speed pitch = "low" , # Lower pitch volume = "loud" # Increase volume ) ) Prosody tags (pitch, rate, volume) have different behaviors based on the engine: - Standard engine: Supports all prosody tags - Neural engine: Full prosody support - Generative engine: Only rate is supported, with a different format (e.g., “1.1” for 10% faster) ​ Frame Flow ​ Metrics Support The service collects processing metrics: Time to First Byte (TTFB) Processing duration Character usage API calls ​ Notes Supports all AWS Polly engines: Standard (non-neural voices) Neural (improved quality voices) Generative (high-quality, natural-sounding voices) Automatic audio resampling from 16kHz to any desired rate Thread-safe processing Automatic error handling Manages AWS client lifecycle Together AI Azure On this page Overview Installation Configuration Constructor Parameters Input Parameters Output Frames Control Frames Audio Frames Error Frames Methods Language Support Usage Example SSML Support Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_aws_0b913f18.txt b/tts_aws_0b913f18.txt
new file mode 100644
index 0000000000000000000000000000000000000000..b6cb5098338a11dca1a8a73783809c51e7d0d302
--- /dev/null
+++ b/tts_aws_0b913f18.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/aws#input-parameters
+Title: AWS Polly - Pipecat
+==================================================
+
+AWS Polly - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech AWS Polly Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview AWSPollyTTSService provides text-to-speech capabilities using AWS’s Polly service. It supports multiple voices, languages, and speech customization options through SSML. The older PollyTTSService class is still available but has been deprecated. Use AWSPollyTTSService instead. ​ Installation To use AWSPollyTTSService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[aws]" You’ll also need to set up your AWS credentials as environment variables: AWS_SECRET_ACCESS_KEY AWS_ACCESS_KEY_ID AWS_SESSION_TOKEN (if using temporary credentials) AWS_REGION (defaults to “us-east-1”) ​ Configuration ​ Constructor Parameters ​ api_key str AWS secret access key (can also use environment variable) ​ aws_access_key_id str AWS access key ID (can also use environment variable) ​ aws_session_token str AWS session token for temporary credentials (can also use environment variable) ​ region str AWS region name (defaults to “us-east-1” if not provided) ​ voice_id str default: "Joanna" AWS Polly voice identifier ​ sample_rate int default: "None" Output audio sample rate in Hz (resampled from Polly’s 16kHz) ​ text_filter BaseTextFilter default: "None" Modifies text provided to the TTS. Learn more about the available filters. ​ params InputParams TTS configuration parameters ​ Input Parameters Copy Ask AI class InputParams ( BaseModel ): engine: Optional[ str ] = None # Polly engine type ("standard", "neural", or "generative") language: Optional[Language] = Language. EN pitch: Optional[ str ] = None # SSML pitch adjustment rate: Optional[ str ] = None # SSML rate adjustment volume: Optional[ str ] = None # SSML volume adjustment ​ Output Frames ​ Control Frames ​ TTSStartedFrame Frame Signals start of speech synthesis ​ TTSStoppedFrame Frame Signals completion of speech synthesis ​ Audio Frames ​ TTSAudioRawFrame Frame Contains generated audio data with: PCM audio format Sample rate as specified (resampled from 16kHz) Single channel (mono) ​ Error Frames ​ ErrorFrame Frame Contains AWS Polly error information ​ Methods See the TTS base class methods for additional functionality. ​ Language Support Supports an extensive range of languages and regional variants: Language Code Description Service Code Language.AR Arabic arb Language.AR_AE Arabic (UAE) ar-AE Language.CA Catalan ca-ES Language.ZH Chinese (Mandarin) cmn-CN Language.YUE Chinese (Cantonese) yue-CN Language.YUE_CN Chinese (Cantonese) yue-CN Language.CS Czech cs-CZ Language.DA Danish da-DK Language.NL Dutch nl-NL Language.NL_BE Dutch (Belgium) nl-BE Language.EN English (US) en-US Language.EN_AU English (Australia) en-AU Language.EN_GB English (UK) en-GB Language.EN_IN English (India) en-IN Language.EN_NZ English (New Zealand) en-NZ Language.EN_US English (US) en-US Language.EN_ZA English (South Africa) en-ZA Language.FI Finnish fi-FI Language.FR French fr-FR Language.FR_BE French (Belgium) fr-BE Language.FR_CA French (Canada) fr-CA Language.DE German de-DE Language.DE_AT German (Austria) de-AT Language.DE_CH German (Switzerland) de-CH Language.HI Hindi hi-IN Language.IS Icelandic is-IS Language.IT Italian it-IT Language.JA Japanese ja-JP Language.KO Korean ko-KR Language.NO Norwegian nb-NO Language.NB Norwegian (Bokmål) nb-NO Language.NB_NO Norwegian (Bokmål) nb-NO Language.PL Polish pl-PL Language.PT Portuguese pt-PT Language.PT_BR Portuguese (Brazil) pt-BR Language.PT_PT Portuguese (Portugal) pt-PT Language.RO Romanian ro-RO Language.RU Russian ru-RU Language.ES Spanish es-ES Language.ES_MX Spanish (Mexico) es-MX Language.ES_US Spanish (US) es-US Language.SV Swedish sv-SE Language.TR Turkish tr-TR Language.CY Welsh cy-GB Language.CY_GB Welsh cy-GB ​ Usage Example Copy Ask AI from pipecat.services.aws.tts import AWSPollyTTSService from pipecat.transcriptions.language import Language # Configure service using environment variables for credentials tts = AWSPollyTTSService( region = "us-west-2" , voice_id = "Joanna" , params = AWSPollyTTSService.InputParams( engine = "neural" , language = Language. EN , rate = "+10%" , volume = "loud" ) ) # Or provide credentials directly tts = AWSPollyTTSService( aws_access_key_id = "YOUR_ACCESS_KEY_ID" , api_key = "YOUR_SECRET_ACCESS_KEY" , region = "us-west-2" , voice_id = "Joanna" , params = AWSPollyTTSService.InputParams( engine = "generative" , # For newer generative voices language = Language. EN , rate = "1.1" # Generative engine rate format ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ SSML Support The service automatically constructs SSML tags for advanced speech control: Copy Ask AI # Example with SSML controls service = AWSPollyTTSService( # ... other params ... params = AWSPollyTTSService.InputParams( engine = "neural" , rate = "+20%" , # Increase speed pitch = "low" , # Lower pitch volume = "loud" # Increase volume ) ) Prosody tags (pitch, rate, volume) have different behaviors based on the engine: - Standard engine: Supports all prosody tags - Neural engine: Full prosody support - Generative engine: Only rate is supported, with a different format (e.g., “1.1” for 10% faster) ​ Frame Flow ​ Metrics Support The service collects processing metrics: Time to First Byte (TTFB) Processing duration Character usage API calls ​ Notes Supports all AWS Polly engines: Standard (non-neural voices) Neural (improved quality voices) Generative (high-quality, natural-sounding voices) Automatic audio resampling from 16kHz to any desired rate Thread-safe processing Automatic error handling Manages AWS client lifecycle Together AI Azure On this page Overview Installation Configuration Constructor Parameters Input Parameters Output Frames Control Frames Audio Frames Error Frames Methods Language Support Usage Example SSML Support Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_aws_130946c0.txt b/tts_aws_130946c0.txt
new file mode 100644
index 0000000000000000000000000000000000000000..c8285ccd88127249afa8f66e657ceec2d43740c7
--- /dev/null
+++ b/tts_aws_130946c0.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/aws#configuration
+Title: AWS Polly - Pipecat
+==================================================
+
+AWS Polly - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech AWS Polly Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview AWSPollyTTSService provides text-to-speech capabilities using AWS’s Polly service. It supports multiple voices, languages, and speech customization options through SSML. The older PollyTTSService class is still available but has been deprecated. Use AWSPollyTTSService instead. ​ Installation To use AWSPollyTTSService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[aws]" You’ll also need to set up your AWS credentials as environment variables: AWS_SECRET_ACCESS_KEY AWS_ACCESS_KEY_ID AWS_SESSION_TOKEN (if using temporary credentials) AWS_REGION (defaults to “us-east-1”) ​ Configuration ​ Constructor Parameters ​ api_key str AWS secret access key (can also use environment variable) ​ aws_access_key_id str AWS access key ID (can also use environment variable) ​ aws_session_token str AWS session token for temporary credentials (can also use environment variable) ​ region str AWS region name (defaults to “us-east-1” if not provided) ​ voice_id str default: "Joanna" AWS Polly voice identifier ​ sample_rate int default: "None" Output audio sample rate in Hz (resampled from Polly’s 16kHz) ​ text_filter BaseTextFilter default: "None" Modifies text provided to the TTS. Learn more about the available filters. ​ params InputParams TTS configuration parameters ​ Input Parameters Copy Ask AI class InputParams ( BaseModel ): engine: Optional[ str ] = None # Polly engine type ("standard", "neural", or "generative") language: Optional[Language] = Language. EN pitch: Optional[ str ] = None # SSML pitch adjustment rate: Optional[ str ] = None # SSML rate adjustment volume: Optional[ str ] = None # SSML volume adjustment ​ Output Frames ​ Control Frames ​ TTSStartedFrame Frame Signals start of speech synthesis ​ TTSStoppedFrame Frame Signals completion of speech synthesis ​ Audio Frames ​ TTSAudioRawFrame Frame Contains generated audio data with: PCM audio format Sample rate as specified (resampled from 16kHz) Single channel (mono) ​ Error Frames ​ ErrorFrame Frame Contains AWS Polly error information ​ Methods See the TTS base class methods for additional functionality. ​ Language Support Supports an extensive range of languages and regional variants: Language Code Description Service Code Language.AR Arabic arb Language.AR_AE Arabic (UAE) ar-AE Language.CA Catalan ca-ES Language.ZH Chinese (Mandarin) cmn-CN Language.YUE Chinese (Cantonese) yue-CN Language.YUE_CN Chinese (Cantonese) yue-CN Language.CS Czech cs-CZ Language.DA Danish da-DK Language.NL Dutch nl-NL Language.NL_BE Dutch (Belgium) nl-BE Language.EN English (US) en-US Language.EN_AU English (Australia) en-AU Language.EN_GB English (UK) en-GB Language.EN_IN English (India) en-IN Language.EN_NZ English (New Zealand) en-NZ Language.EN_US English (US) en-US Language.EN_ZA English (South Africa) en-ZA Language.FI Finnish fi-FI Language.FR French fr-FR Language.FR_BE French (Belgium) fr-BE Language.FR_CA French (Canada) fr-CA Language.DE German de-DE Language.DE_AT German (Austria) de-AT Language.DE_CH German (Switzerland) de-CH Language.HI Hindi hi-IN Language.IS Icelandic is-IS Language.IT Italian it-IT Language.JA Japanese ja-JP Language.KO Korean ko-KR Language.NO Norwegian nb-NO Language.NB Norwegian (Bokmål) nb-NO Language.NB_NO Norwegian (Bokmål) nb-NO Language.PL Polish pl-PL Language.PT Portuguese pt-PT Language.PT_BR Portuguese (Brazil) pt-BR Language.PT_PT Portuguese (Portugal) pt-PT Language.RO Romanian ro-RO Language.RU Russian ru-RU Language.ES Spanish es-ES Language.ES_MX Spanish (Mexico) es-MX Language.ES_US Spanish (US) es-US Language.SV Swedish sv-SE Language.TR Turkish tr-TR Language.CY Welsh cy-GB Language.CY_GB Welsh cy-GB ​ Usage Example Copy Ask AI from pipecat.services.aws.tts import AWSPollyTTSService from pipecat.transcriptions.language import Language # Configure service using environment variables for credentials tts = AWSPollyTTSService( region = "us-west-2" , voice_id = "Joanna" , params = AWSPollyTTSService.InputParams( engine = "neural" , language = Language. EN , rate = "+10%" , volume = "loud" ) ) # Or provide credentials directly tts = AWSPollyTTSService( aws_access_key_id = "YOUR_ACCESS_KEY_ID" , api_key = "YOUR_SECRET_ACCESS_KEY" , region = "us-west-2" , voice_id = "Joanna" , params = AWSPollyTTSService.InputParams( engine = "generative" , # For newer generative voices language = Language. EN , rate = "1.1" # Generative engine rate format ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ SSML Support The service automatically constructs SSML tags for advanced speech control: Copy Ask AI # Example with SSML controls service = AWSPollyTTSService( # ... other params ... params = AWSPollyTTSService.InputParams( engine = "neural" , rate = "+20%" , # Increase speed pitch = "low" , # Lower pitch volume = "loud" # Increase volume ) ) Prosody tags (pitch, rate, volume) have different behaviors based on the engine: - Standard engine: Supports all prosody tags - Neural engine: Full prosody support - Generative engine: Only rate is supported, with a different format (e.g., “1.1” for 10% faster) ​ Frame Flow ​ Metrics Support The service collects processing metrics: Time to First Byte (TTFB) Processing duration Character usage API calls ​ Notes Supports all AWS Polly engines: Standard (non-neural voices) Neural (improved quality voices) Generative (high-quality, natural-sounding voices) Automatic audio resampling from 16kHz to any desired rate Thread-safe processing Automatic error handling Manages AWS client lifecycle Together AI Azure On this page Overview Installation Configuration Constructor Parameters Input Parameters Output Frames Control Frames Audio Frames Error Frames Methods Language Support Usage Example SSML Support Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_aws_20996048.txt b/tts_aws_20996048.txt
new file mode 100644
index 0000000000000000000000000000000000000000..cbf2f52b31060366135f11aff0bcd879d4b594b3
--- /dev/null
+++ b/tts_aws_20996048.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/aws#param-error-frame
+Title: AWS Polly - Pipecat
+==================================================
+
+AWS Polly - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech AWS Polly Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview AWSPollyTTSService provides text-to-speech capabilities using AWS’s Polly service. It supports multiple voices, languages, and speech customization options through SSML. The older PollyTTSService class is still available but has been deprecated. Use AWSPollyTTSService instead. ​ Installation To use AWSPollyTTSService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[aws]" You’ll also need to set up your AWS credentials as environment variables: AWS_SECRET_ACCESS_KEY AWS_ACCESS_KEY_ID AWS_SESSION_TOKEN (if using temporary credentials) AWS_REGION (defaults to “us-east-1”) ​ Configuration ​ Constructor Parameters ​ api_key str AWS secret access key (can also use environment variable) ​ aws_access_key_id str AWS access key ID (can also use environment variable) ​ aws_session_token str AWS session token for temporary credentials (can also use environment variable) ​ region str AWS region name (defaults to “us-east-1” if not provided) ​ voice_id str default: "Joanna" AWS Polly voice identifier ​ sample_rate int default: "None" Output audio sample rate in Hz (resampled from Polly’s 16kHz) ​ text_filter BaseTextFilter default: "None" Modifies text provided to the TTS. Learn more about the available filters. ​ params InputParams TTS configuration parameters ​ Input Parameters Copy Ask AI class InputParams ( BaseModel ): engine: Optional[ str ] = None # Polly engine type ("standard", "neural", or "generative") language: Optional[Language] = Language. EN pitch: Optional[ str ] = None # SSML pitch adjustment rate: Optional[ str ] = None # SSML rate adjustment volume: Optional[ str ] = None # SSML volume adjustment ​ Output Frames ​ Control Frames ​ TTSStartedFrame Frame Signals start of speech synthesis ​ TTSStoppedFrame Frame Signals completion of speech synthesis ​ Audio Frames ​ TTSAudioRawFrame Frame Contains generated audio data with: PCM audio format Sample rate as specified (resampled from 16kHz) Single channel (mono) ​ Error Frames ​ ErrorFrame Frame Contains AWS Polly error information ​ Methods See the TTS base class methods for additional functionality. ​ Language Support Supports an extensive range of languages and regional variants: Language Code Description Service Code Language.AR Arabic arb Language.AR_AE Arabic (UAE) ar-AE Language.CA Catalan ca-ES Language.ZH Chinese (Mandarin) cmn-CN Language.YUE Chinese (Cantonese) yue-CN Language.YUE_CN Chinese (Cantonese) yue-CN Language.CS Czech cs-CZ Language.DA Danish da-DK Language.NL Dutch nl-NL Language.NL_BE Dutch (Belgium) nl-BE Language.EN English (US) en-US Language.EN_AU English (Australia) en-AU Language.EN_GB English (UK) en-GB Language.EN_IN English (India) en-IN Language.EN_NZ English (New Zealand) en-NZ Language.EN_US English (US) en-US Language.EN_ZA English (South Africa) en-ZA Language.FI Finnish fi-FI Language.FR French fr-FR Language.FR_BE French (Belgium) fr-BE Language.FR_CA French (Canada) fr-CA Language.DE German de-DE Language.DE_AT German (Austria) de-AT Language.DE_CH German (Switzerland) de-CH Language.HI Hindi hi-IN Language.IS Icelandic is-IS Language.IT Italian it-IT Language.JA Japanese ja-JP Language.KO Korean ko-KR Language.NO Norwegian nb-NO Language.NB Norwegian (Bokmål) nb-NO Language.NB_NO Norwegian (Bokmål) nb-NO Language.PL Polish pl-PL Language.PT Portuguese pt-PT Language.PT_BR Portuguese (Brazil) pt-BR Language.PT_PT Portuguese (Portugal) pt-PT Language.RO Romanian ro-RO Language.RU Russian ru-RU Language.ES Spanish es-ES Language.ES_MX Spanish (Mexico) es-MX Language.ES_US Spanish (US) es-US Language.SV Swedish sv-SE Language.TR Turkish tr-TR Language.CY Welsh cy-GB Language.CY_GB Welsh cy-GB ​ Usage Example Copy Ask AI from pipecat.services.aws.tts import AWSPollyTTSService from pipecat.transcriptions.language import Language # Configure service using environment variables for credentials tts = AWSPollyTTSService( region = "us-west-2" , voice_id = "Joanna" , params = AWSPollyTTSService.InputParams( engine = "neural" , language = Language. EN , rate = "+10%" , volume = "loud" ) ) # Or provide credentials directly tts = AWSPollyTTSService( aws_access_key_id = "YOUR_ACCESS_KEY_ID" , api_key = "YOUR_SECRET_ACCESS_KEY" , region = "us-west-2" , voice_id = "Joanna" , params = AWSPollyTTSService.InputParams( engine = "generative" , # For newer generative voices language = Language. EN , rate = "1.1" # Generative engine rate format ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ SSML Support The service automatically constructs SSML tags for advanced speech control: Copy Ask AI # Example with SSML controls service = AWSPollyTTSService( # ... other params ... params = AWSPollyTTSService.InputParams( engine = "neural" , rate = "+20%" , # Increase speed pitch = "low" , # Lower pitch volume = "loud" # Increase volume ) ) Prosody tags (pitch, rate, volume) have different behaviors based on the engine: - Standard engine: Supports all prosody tags - Neural engine: Full prosody support - Generative engine: Only rate is supported, with a different format (e.g., “1.1” for 10% faster) ​ Frame Flow ​ Metrics Support The service collects processing metrics: Time to First Byte (TTFB) Processing duration Character usage API calls ​ Notes Supports all AWS Polly engines: Standard (non-neural voices) Neural (improved quality voices) Generative (high-quality, natural-sounding voices) Automatic audio resampling from 16kHz to any desired rate Thread-safe processing Automatic error handling Manages AWS client lifecycle Together AI Azure On this page Overview Installation Configuration Constructor Parameters Input Parameters Output Frames Control Frames Audio Frames Error Frames Methods Language Support Usage Example SSML Support Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_aws_3d9cd762.txt b/tts_aws_3d9cd762.txt
new file mode 100644
index 0000000000000000000000000000000000000000..923ac0bfa8e0673206d83092df66df2a4985950d
--- /dev/null
+++ b/tts_aws_3d9cd762.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/aws#language-support
+Title: AWS Polly - Pipecat
+==================================================
+
+AWS Polly - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech AWS Polly Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview AWSPollyTTSService provides text-to-speech capabilities using AWS’s Polly service. It supports multiple voices, languages, and speech customization options through SSML. The older PollyTTSService class is still available but has been deprecated. Use AWSPollyTTSService instead. ​ Installation To use AWSPollyTTSService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[aws]" You’ll also need to set up your AWS credentials as environment variables: AWS_SECRET_ACCESS_KEY AWS_ACCESS_KEY_ID AWS_SESSION_TOKEN (if using temporary credentials) AWS_REGION (defaults to “us-east-1”) ​ Configuration ​ Constructor Parameters ​ api_key str AWS secret access key (can also use environment variable) ​ aws_access_key_id str AWS access key ID (can also use environment variable) ​ aws_session_token str AWS session token for temporary credentials (can also use environment variable) ​ region str AWS region name (defaults to “us-east-1” if not provided) ​ voice_id str default: "Joanna" AWS Polly voice identifier ​ sample_rate int default: "None" Output audio sample rate in Hz (resampled from Polly’s 16kHz) ​ text_filter BaseTextFilter default: "None" Modifies text provided to the TTS. Learn more about the available filters. ​ params InputParams TTS configuration parameters ​ Input Parameters Copy Ask AI class InputParams ( BaseModel ): engine: Optional[ str ] = None # Polly engine type ("standard", "neural", or "generative") language: Optional[Language] = Language. EN pitch: Optional[ str ] = None # SSML pitch adjustment rate: Optional[ str ] = None # SSML rate adjustment volume: Optional[ str ] = None # SSML volume adjustment ​ Output Frames ​ Control Frames ​ TTSStartedFrame Frame Signals start of speech synthesis ​ TTSStoppedFrame Frame Signals completion of speech synthesis ​ Audio Frames ​ TTSAudioRawFrame Frame Contains generated audio data with: PCM audio format Sample rate as specified (resampled from 16kHz) Single channel (mono) ​ Error Frames ​ ErrorFrame Frame Contains AWS Polly error information ​ Methods See the TTS base class methods for additional functionality. ​ Language Support Supports an extensive range of languages and regional variants: Language Code Description Service Code Language.AR Arabic arb Language.AR_AE Arabic (UAE) ar-AE Language.CA Catalan ca-ES Language.ZH Chinese (Mandarin) cmn-CN Language.YUE Chinese (Cantonese) yue-CN Language.YUE_CN Chinese (Cantonese) yue-CN Language.CS Czech cs-CZ Language.DA Danish da-DK Language.NL Dutch nl-NL Language.NL_BE Dutch (Belgium) nl-BE Language.EN English (US) en-US Language.EN_AU English (Australia) en-AU Language.EN_GB English (UK) en-GB Language.EN_IN English (India) en-IN Language.EN_NZ English (New Zealand) en-NZ Language.EN_US English (US) en-US Language.EN_ZA English (South Africa) en-ZA Language.FI Finnish fi-FI Language.FR French fr-FR Language.FR_BE French (Belgium) fr-BE Language.FR_CA French (Canada) fr-CA Language.DE German de-DE Language.DE_AT German (Austria) de-AT Language.DE_CH German (Switzerland) de-CH Language.HI Hindi hi-IN Language.IS Icelandic is-IS Language.IT Italian it-IT Language.JA Japanese ja-JP Language.KO Korean ko-KR Language.NO Norwegian nb-NO Language.NB Norwegian (Bokmål) nb-NO Language.NB_NO Norwegian (Bokmål) nb-NO Language.PL Polish pl-PL Language.PT Portuguese pt-PT Language.PT_BR Portuguese (Brazil) pt-BR Language.PT_PT Portuguese (Portugal) pt-PT Language.RO Romanian ro-RO Language.RU Russian ru-RU Language.ES Spanish es-ES Language.ES_MX Spanish (Mexico) es-MX Language.ES_US Spanish (US) es-US Language.SV Swedish sv-SE Language.TR Turkish tr-TR Language.CY Welsh cy-GB Language.CY_GB Welsh cy-GB ​ Usage Example Copy Ask AI from pipecat.services.aws.tts import AWSPollyTTSService from pipecat.transcriptions.language import Language # Configure service using environment variables for credentials tts = AWSPollyTTSService( region = "us-west-2" , voice_id = "Joanna" , params = AWSPollyTTSService.InputParams( engine = "neural" , language = Language. EN , rate = "+10%" , volume = "loud" ) ) # Or provide credentials directly tts = AWSPollyTTSService( aws_access_key_id = "YOUR_ACCESS_KEY_ID" , api_key = "YOUR_SECRET_ACCESS_KEY" , region = "us-west-2" , voice_id = "Joanna" , params = AWSPollyTTSService.InputParams( engine = "generative" , # For newer generative voices language = Language. EN , rate = "1.1" # Generative engine rate format ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ SSML Support The service automatically constructs SSML tags for advanced speech control: Copy Ask AI # Example with SSML controls service = AWSPollyTTSService( # ... other params ... params = AWSPollyTTSService.InputParams( engine = "neural" , rate = "+20%" , # Increase speed pitch = "low" , # Lower pitch volume = "loud" # Increase volume ) ) Prosody tags (pitch, rate, volume) have different behaviors based on the engine: - Standard engine: Supports all prosody tags - Neural engine: Full prosody support - Generative engine: Only rate is supported, with a different format (e.g., “1.1” for 10% faster) ​ Frame Flow ​ Metrics Support The service collects processing metrics: Time to First Byte (TTFB) Processing duration Character usage API calls ​ Notes Supports all AWS Polly engines: Standard (non-neural voices) Neural (improved quality voices) Generative (high-quality, natural-sounding voices) Automatic audio resampling from 16kHz to any desired rate Thread-safe processing Automatic error handling Manages AWS client lifecycle Together AI Azure On this page Overview Installation Configuration Constructor Parameters Input Parameters Output Frames Control Frames Audio Frames Error Frames Methods Language Support Usage Example SSML Support Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_aws_47e5c228.txt b/tts_aws_47e5c228.txt
new file mode 100644
index 0000000000000000000000000000000000000000..fd33d77bb32826bb98734147b149ce27b39d76e9
--- /dev/null
+++ b/tts_aws_47e5c228.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/aws#constructor-parameters
+Title: AWS Polly - Pipecat
+==================================================
+
+AWS Polly - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech AWS Polly Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview AWSPollyTTSService provides text-to-speech capabilities using AWS’s Polly service. It supports multiple voices, languages, and speech customization options through SSML. The older PollyTTSService class is still available but has been deprecated. Use AWSPollyTTSService instead. ​ Installation To use AWSPollyTTSService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[aws]" You’ll also need to set up your AWS credentials as environment variables: AWS_SECRET_ACCESS_KEY AWS_ACCESS_KEY_ID AWS_SESSION_TOKEN (if using temporary credentials) AWS_REGION (defaults to “us-east-1”) ​ Configuration ​ Constructor Parameters ​ api_key str AWS secret access key (can also use environment variable) ​ aws_access_key_id str AWS access key ID (can also use environment variable) ​ aws_session_token str AWS session token for temporary credentials (can also use environment variable) ​ region str AWS region name (defaults to “us-east-1” if not provided) ​ voice_id str default: "Joanna" AWS Polly voice identifier ​ sample_rate int default: "None" Output audio sample rate in Hz (resampled from Polly’s 16kHz) ​ text_filter BaseTextFilter default: "None" Modifies text provided to the TTS. Learn more about the available filters. ​ params InputParams TTS configuration parameters ​ Input Parameters Copy Ask AI class InputParams ( BaseModel ): engine: Optional[ str ] = None # Polly engine type ("standard", "neural", or "generative") language: Optional[Language] = Language. EN pitch: Optional[ str ] = None # SSML pitch adjustment rate: Optional[ str ] = None # SSML rate adjustment volume: Optional[ str ] = None # SSML volume adjustment ​ Output Frames ​ Control Frames ​ TTSStartedFrame Frame Signals start of speech synthesis ​ TTSStoppedFrame Frame Signals completion of speech synthesis ​ Audio Frames ​ TTSAudioRawFrame Frame Contains generated audio data with: PCM audio format Sample rate as specified (resampled from 16kHz) Single channel (mono) ​ Error Frames ​ ErrorFrame Frame Contains AWS Polly error information ​ Methods See the TTS base class methods for additional functionality. ​ Language Support Supports an extensive range of languages and regional variants: Language Code Description Service Code Language.AR Arabic arb Language.AR_AE Arabic (UAE) ar-AE Language.CA Catalan ca-ES Language.ZH Chinese (Mandarin) cmn-CN Language.YUE Chinese (Cantonese) yue-CN Language.YUE_CN Chinese (Cantonese) yue-CN Language.CS Czech cs-CZ Language.DA Danish da-DK Language.NL Dutch nl-NL Language.NL_BE Dutch (Belgium) nl-BE Language.EN English (US) en-US Language.EN_AU English (Australia) en-AU Language.EN_GB English (UK) en-GB Language.EN_IN English (India) en-IN Language.EN_NZ English (New Zealand) en-NZ Language.EN_US English (US) en-US Language.EN_ZA English (South Africa) en-ZA Language.FI Finnish fi-FI Language.FR French fr-FR Language.FR_BE French (Belgium) fr-BE Language.FR_CA French (Canada) fr-CA Language.DE German de-DE Language.DE_AT German (Austria) de-AT Language.DE_CH German (Switzerland) de-CH Language.HI Hindi hi-IN Language.IS Icelandic is-IS Language.IT Italian it-IT Language.JA Japanese ja-JP Language.KO Korean ko-KR Language.NO Norwegian nb-NO Language.NB Norwegian (Bokmål) nb-NO Language.NB_NO Norwegian (Bokmål) nb-NO Language.PL Polish pl-PL Language.PT Portuguese pt-PT Language.PT_BR Portuguese (Brazil) pt-BR Language.PT_PT Portuguese (Portugal) pt-PT Language.RO Romanian ro-RO Language.RU Russian ru-RU Language.ES Spanish es-ES Language.ES_MX Spanish (Mexico) es-MX Language.ES_US Spanish (US) es-US Language.SV Swedish sv-SE Language.TR Turkish tr-TR Language.CY Welsh cy-GB Language.CY_GB Welsh cy-GB ​ Usage Example Copy Ask AI from pipecat.services.aws.tts import AWSPollyTTSService from pipecat.transcriptions.language import Language # Configure service using environment variables for credentials tts = AWSPollyTTSService( region = "us-west-2" , voice_id = "Joanna" , params = AWSPollyTTSService.InputParams( engine = "neural" , language = Language. EN , rate = "+10%" , volume = "loud" ) ) # Or provide credentials directly tts = AWSPollyTTSService( aws_access_key_id = "YOUR_ACCESS_KEY_ID" , api_key = "YOUR_SECRET_ACCESS_KEY" , region = "us-west-2" , voice_id = "Joanna" , params = AWSPollyTTSService.InputParams( engine = "generative" , # For newer generative voices language = Language. EN , rate = "1.1" # Generative engine rate format ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ SSML Support The service automatically constructs SSML tags for advanced speech control: Copy Ask AI # Example with SSML controls service = AWSPollyTTSService( # ... other params ... params = AWSPollyTTSService.InputParams( engine = "neural" , rate = "+20%" , # Increase speed pitch = "low" , # Lower pitch volume = "loud" # Increase volume ) ) Prosody tags (pitch, rate, volume) have different behaviors based on the engine: - Standard engine: Supports all prosody tags - Neural engine: Full prosody support - Generative engine: Only rate is supported, with a different format (e.g., “1.1” for 10% faster) ​ Frame Flow ​ Metrics Support The service collects processing metrics: Time to First Byte (TTFB) Processing duration Character usage API calls ​ Notes Supports all AWS Polly engines: Standard (non-neural voices) Neural (improved quality voices) Generative (high-quality, natural-sounding voices) Automatic audio resampling from 16kHz to any desired rate Thread-safe processing Automatic error handling Manages AWS client lifecycle Together AI Azure On this page Overview Installation Configuration Constructor Parameters Input Parameters Output Frames Control Frames Audio Frames Error Frames Methods Language Support Usage Example SSML Support Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_aws_4edf275a.txt b/tts_aws_4edf275a.txt
new file mode 100644
index 0000000000000000000000000000000000000000..a1ea401ab6e23b66274b8f87c1c0fe79e2e2f4f8
--- /dev/null
+++ b/tts_aws_4edf275a.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/aws#frame-flow
+Title: AWS Polly - Pipecat
+==================================================
+
+AWS Polly - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech AWS Polly Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview AWSPollyTTSService provides text-to-speech capabilities using AWS’s Polly service. It supports multiple voices, languages, and speech customization options through SSML. The older PollyTTSService class is still available but has been deprecated. Use AWSPollyTTSService instead. ​ Installation To use AWSPollyTTSService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[aws]" You’ll also need to set up your AWS credentials as environment variables: AWS_SECRET_ACCESS_KEY AWS_ACCESS_KEY_ID AWS_SESSION_TOKEN (if using temporary credentials) AWS_REGION (defaults to “us-east-1”) ​ Configuration ​ Constructor Parameters ​ api_key str AWS secret access key (can also use environment variable) ​ aws_access_key_id str AWS access key ID (can also use environment variable) ​ aws_session_token str AWS session token for temporary credentials (can also use environment variable) ​ region str AWS region name (defaults to “us-east-1” if not provided) ​ voice_id str default: "Joanna" AWS Polly voice identifier ​ sample_rate int default: "None" Output audio sample rate in Hz (resampled from Polly’s 16kHz) ​ text_filter BaseTextFilter default: "None" Modifies text provided to the TTS. Learn more about the available filters. ​ params InputParams TTS configuration parameters ​ Input Parameters Copy Ask AI class InputParams ( BaseModel ): engine: Optional[ str ] = None # Polly engine type ("standard", "neural", or "generative") language: Optional[Language] = Language. EN pitch: Optional[ str ] = None # SSML pitch adjustment rate: Optional[ str ] = None # SSML rate adjustment volume: Optional[ str ] = None # SSML volume adjustment ​ Output Frames ​ Control Frames ​ TTSStartedFrame Frame Signals start of speech synthesis ​ TTSStoppedFrame Frame Signals completion of speech synthesis ​ Audio Frames ​ TTSAudioRawFrame Frame Contains generated audio data with: PCM audio format Sample rate as specified (resampled from 16kHz) Single channel (mono) ​ Error Frames ​ ErrorFrame Frame Contains AWS Polly error information ​ Methods See the TTS base class methods for additional functionality. ​ Language Support Supports an extensive range of languages and regional variants: Language Code Description Service Code Language.AR Arabic arb Language.AR_AE Arabic (UAE) ar-AE Language.CA Catalan ca-ES Language.ZH Chinese (Mandarin) cmn-CN Language.YUE Chinese (Cantonese) yue-CN Language.YUE_CN Chinese (Cantonese) yue-CN Language.CS Czech cs-CZ Language.DA Danish da-DK Language.NL Dutch nl-NL Language.NL_BE Dutch (Belgium) nl-BE Language.EN English (US) en-US Language.EN_AU English (Australia) en-AU Language.EN_GB English (UK) en-GB Language.EN_IN English (India) en-IN Language.EN_NZ English (New Zealand) en-NZ Language.EN_US English (US) en-US Language.EN_ZA English (South Africa) en-ZA Language.FI Finnish fi-FI Language.FR French fr-FR Language.FR_BE French (Belgium) fr-BE Language.FR_CA French (Canada) fr-CA Language.DE German de-DE Language.DE_AT German (Austria) de-AT Language.DE_CH German (Switzerland) de-CH Language.HI Hindi hi-IN Language.IS Icelandic is-IS Language.IT Italian it-IT Language.JA Japanese ja-JP Language.KO Korean ko-KR Language.NO Norwegian nb-NO Language.NB Norwegian (Bokmål) nb-NO Language.NB_NO Norwegian (Bokmål) nb-NO Language.PL Polish pl-PL Language.PT Portuguese pt-PT Language.PT_BR Portuguese (Brazil) pt-BR Language.PT_PT Portuguese (Portugal) pt-PT Language.RO Romanian ro-RO Language.RU Russian ru-RU Language.ES Spanish es-ES Language.ES_MX Spanish (Mexico) es-MX Language.ES_US Spanish (US) es-US Language.SV Swedish sv-SE Language.TR Turkish tr-TR Language.CY Welsh cy-GB Language.CY_GB Welsh cy-GB ​ Usage Example Copy Ask AI from pipecat.services.aws.tts import AWSPollyTTSService from pipecat.transcriptions.language import Language # Configure service using environment variables for credentials tts = AWSPollyTTSService( region = "us-west-2" , voice_id = "Joanna" , params = AWSPollyTTSService.InputParams( engine = "neural" , language = Language. EN , rate = "+10%" , volume = "loud" ) ) # Or provide credentials directly tts = AWSPollyTTSService( aws_access_key_id = "YOUR_ACCESS_KEY_ID" , api_key = "YOUR_SECRET_ACCESS_KEY" , region = "us-west-2" , voice_id = "Joanna" , params = AWSPollyTTSService.InputParams( engine = "generative" , # For newer generative voices language = Language. EN , rate = "1.1" # Generative engine rate format ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ SSML Support The service automatically constructs SSML tags for advanced speech control: Copy Ask AI # Example with SSML controls service = AWSPollyTTSService( # ... other params ... params = AWSPollyTTSService.InputParams( engine = "neural" , rate = "+20%" , # Increase speed pitch = "low" , # Lower pitch volume = "loud" # Increase volume ) ) Prosody tags (pitch, rate, volume) have different behaviors based on the engine: - Standard engine: Supports all prosody tags - Neural engine: Full prosody support - Generative engine: Only rate is supported, with a different format (e.g., “1.1” for 10% faster) ​ Frame Flow ​ Metrics Support The service collects processing metrics: Time to First Byte (TTFB) Processing duration Character usage API calls ​ Notes Supports all AWS Polly engines: Standard (non-neural voices) Neural (improved quality voices) Generative (high-quality, natural-sounding voices) Automatic audio resampling from 16kHz to any desired rate Thread-safe processing Automatic error handling Manages AWS client lifecycle Together AI Azure On this page Overview Installation Configuration Constructor Parameters Input Parameters Output Frames Control Frames Audio Frames Error Frames Methods Language Support Usage Example SSML Support Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_aws_7e6f2ec7.txt b/tts_aws_7e6f2ec7.txt
new file mode 100644
index 0000000000000000000000000000000000000000..39acd5fedbbb81b437f1a9868be066d6f7570850
--- /dev/null
+++ b/tts_aws_7e6f2ec7.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/aws#param-aws-session-token
+Title: AWS Polly - Pipecat
+==================================================
+
+AWS Polly - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech AWS Polly Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview AWSPollyTTSService provides text-to-speech capabilities using AWS’s Polly service. It supports multiple voices, languages, and speech customization options through SSML. The older PollyTTSService class is still available but has been deprecated. Use AWSPollyTTSService instead. ​ Installation To use AWSPollyTTSService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[aws]" You’ll also need to set up your AWS credentials as environment variables: AWS_SECRET_ACCESS_KEY AWS_ACCESS_KEY_ID AWS_SESSION_TOKEN (if using temporary credentials) AWS_REGION (defaults to “us-east-1”) ​ Configuration ​ Constructor Parameters ​ api_key str AWS secret access key (can also use environment variable) ​ aws_access_key_id str AWS access key ID (can also use environment variable) ​ aws_session_token str AWS session token for temporary credentials (can also use environment variable) ​ region str AWS region name (defaults to “us-east-1” if not provided) ​ voice_id str default: "Joanna" AWS Polly voice identifier ​ sample_rate int default: "None" Output audio sample rate in Hz (resampled from Polly’s 16kHz) ​ text_filter BaseTextFilter default: "None" Modifies text provided to the TTS. Learn more about the available filters. ​ params InputParams TTS configuration parameters ​ Input Parameters Copy Ask AI class InputParams ( BaseModel ): engine: Optional[ str ] = None # Polly engine type ("standard", "neural", or "generative") language: Optional[Language] = Language. EN pitch: Optional[ str ] = None # SSML pitch adjustment rate: Optional[ str ] = None # SSML rate adjustment volume: Optional[ str ] = None # SSML volume adjustment ​ Output Frames ​ Control Frames ​ TTSStartedFrame Frame Signals start of speech synthesis ​ TTSStoppedFrame Frame Signals completion of speech synthesis ​ Audio Frames ​ TTSAudioRawFrame Frame Contains generated audio data with: PCM audio format Sample rate as specified (resampled from 16kHz) Single channel (mono) ​ Error Frames ​ ErrorFrame Frame Contains AWS Polly error information ​ Methods See the TTS base class methods for additional functionality. ​ Language Support Supports an extensive range of languages and regional variants: Language Code Description Service Code Language.AR Arabic arb Language.AR_AE Arabic (UAE) ar-AE Language.CA Catalan ca-ES Language.ZH Chinese (Mandarin) cmn-CN Language.YUE Chinese (Cantonese) yue-CN Language.YUE_CN Chinese (Cantonese) yue-CN Language.CS Czech cs-CZ Language.DA Danish da-DK Language.NL Dutch nl-NL Language.NL_BE Dutch (Belgium) nl-BE Language.EN English (US) en-US Language.EN_AU English (Australia) en-AU Language.EN_GB English (UK) en-GB Language.EN_IN English (India) en-IN Language.EN_NZ English (New Zealand) en-NZ Language.EN_US English (US) en-US Language.EN_ZA English (South Africa) en-ZA Language.FI Finnish fi-FI Language.FR French fr-FR Language.FR_BE French (Belgium) fr-BE Language.FR_CA French (Canada) fr-CA Language.DE German de-DE Language.DE_AT German (Austria) de-AT Language.DE_CH German (Switzerland) de-CH Language.HI Hindi hi-IN Language.IS Icelandic is-IS Language.IT Italian it-IT Language.JA Japanese ja-JP Language.KO Korean ko-KR Language.NO Norwegian nb-NO Language.NB Norwegian (Bokmål) nb-NO Language.NB_NO Norwegian (Bokmål) nb-NO Language.PL Polish pl-PL Language.PT Portuguese pt-PT Language.PT_BR Portuguese (Brazil) pt-BR Language.PT_PT Portuguese (Portugal) pt-PT Language.RO Romanian ro-RO Language.RU Russian ru-RU Language.ES Spanish es-ES Language.ES_MX Spanish (Mexico) es-MX Language.ES_US Spanish (US) es-US Language.SV Swedish sv-SE Language.TR Turkish tr-TR Language.CY Welsh cy-GB Language.CY_GB Welsh cy-GB ​ Usage Example Copy Ask AI from pipecat.services.aws.tts import AWSPollyTTSService from pipecat.transcriptions.language import Language # Configure service using environment variables for credentials tts = AWSPollyTTSService( region = "us-west-2" , voice_id = "Joanna" , params = AWSPollyTTSService.InputParams( engine = "neural" , language = Language. EN , rate = "+10%" , volume = "loud" ) ) # Or provide credentials directly tts = AWSPollyTTSService( aws_access_key_id = "YOUR_ACCESS_KEY_ID" , api_key = "YOUR_SECRET_ACCESS_KEY" , region = "us-west-2" , voice_id = "Joanna" , params = AWSPollyTTSService.InputParams( engine = "generative" , # For newer generative voices language = Language. EN , rate = "1.1" # Generative engine rate format ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ SSML Support The service automatically constructs SSML tags for advanced speech control: Copy Ask AI # Example with SSML controls service = AWSPollyTTSService( # ... other params ... params = AWSPollyTTSService.InputParams( engine = "neural" , rate = "+20%" , # Increase speed pitch = "low" , # Lower pitch volume = "loud" # Increase volume ) ) Prosody tags (pitch, rate, volume) have different behaviors based on the engine: - Standard engine: Supports all prosody tags - Neural engine: Full prosody support - Generative engine: Only rate is supported, with a different format (e.g., “1.1” for 10% faster) ​ Frame Flow ​ Metrics Support The service collects processing metrics: Time to First Byte (TTFB) Processing duration Character usage API calls ​ Notes Supports all AWS Polly engines: Standard (non-neural voices) Neural (improved quality voices) Generative (high-quality, natural-sounding voices) Automatic audio resampling from 16kHz to any desired rate Thread-safe processing Automatic error handling Manages AWS client lifecycle Together AI Azure On this page Overview Installation Configuration Constructor Parameters Input Parameters Output Frames Control Frames Audio Frames Error Frames Methods Language Support Usage Example SSML Support Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_aws_826f0513.txt b/tts_aws_826f0513.txt
new file mode 100644
index 0000000000000000000000000000000000000000..bcfa08f670245655defc8185b2a41c4887a30b04
--- /dev/null
+++ b/tts_aws_826f0513.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/aws#metrics-support
+Title: AWS Polly - Pipecat
+==================================================
+
+AWS Polly - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech AWS Polly Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview AWSPollyTTSService provides text-to-speech capabilities using AWS’s Polly service. It supports multiple voices, languages, and speech customization options through SSML. The older PollyTTSService class is still available but has been deprecated. Use AWSPollyTTSService instead. ​ Installation To use AWSPollyTTSService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[aws]" You’ll also need to set up your AWS credentials as environment variables: AWS_SECRET_ACCESS_KEY AWS_ACCESS_KEY_ID AWS_SESSION_TOKEN (if using temporary credentials) AWS_REGION (defaults to “us-east-1”) ​ Configuration ​ Constructor Parameters ​ api_key str AWS secret access key (can also use environment variable) ​ aws_access_key_id str AWS access key ID (can also use environment variable) ​ aws_session_token str AWS session token for temporary credentials (can also use environment variable) ​ region str AWS region name (defaults to “us-east-1” if not provided) ​ voice_id str default: "Joanna" AWS Polly voice identifier ​ sample_rate int default: "None" Output audio sample rate in Hz (resampled from Polly’s 16kHz) ​ text_filter BaseTextFilter default: "None" Modifies text provided to the TTS. Learn more about the available filters. ​ params InputParams TTS configuration parameters ​ Input Parameters Copy Ask AI class InputParams ( BaseModel ): engine: Optional[ str ] = None # Polly engine type ("standard", "neural", or "generative") language: Optional[Language] = Language. EN pitch: Optional[ str ] = None # SSML pitch adjustment rate: Optional[ str ] = None # SSML rate adjustment volume: Optional[ str ] = None # SSML volume adjustment ​ Output Frames ​ Control Frames ​ TTSStartedFrame Frame Signals start of speech synthesis ​ TTSStoppedFrame Frame Signals completion of speech synthesis ​ Audio Frames ​ TTSAudioRawFrame Frame Contains generated audio data with: PCM audio format Sample rate as specified (resampled from 16kHz) Single channel (mono) ​ Error Frames ​ ErrorFrame Frame Contains AWS Polly error information ​ Methods See the TTS base class methods for additional functionality. ​ Language Support Supports an extensive range of languages and regional variants: Language Code Description Service Code Language.AR Arabic arb Language.AR_AE Arabic (UAE) ar-AE Language.CA Catalan ca-ES Language.ZH Chinese (Mandarin) cmn-CN Language.YUE Chinese (Cantonese) yue-CN Language.YUE_CN Chinese (Cantonese) yue-CN Language.CS Czech cs-CZ Language.DA Danish da-DK Language.NL Dutch nl-NL Language.NL_BE Dutch (Belgium) nl-BE Language.EN English (US) en-US Language.EN_AU English (Australia) en-AU Language.EN_GB English (UK) en-GB Language.EN_IN English (India) en-IN Language.EN_NZ English (New Zealand) en-NZ Language.EN_US English (US) en-US Language.EN_ZA English (South Africa) en-ZA Language.FI Finnish fi-FI Language.FR French fr-FR Language.FR_BE French (Belgium) fr-BE Language.FR_CA French (Canada) fr-CA Language.DE German de-DE Language.DE_AT German (Austria) de-AT Language.DE_CH German (Switzerland) de-CH Language.HI Hindi hi-IN Language.IS Icelandic is-IS Language.IT Italian it-IT Language.JA Japanese ja-JP Language.KO Korean ko-KR Language.NO Norwegian nb-NO Language.NB Norwegian (Bokmål) nb-NO Language.NB_NO Norwegian (Bokmål) nb-NO Language.PL Polish pl-PL Language.PT Portuguese pt-PT Language.PT_BR Portuguese (Brazil) pt-BR Language.PT_PT Portuguese (Portugal) pt-PT Language.RO Romanian ro-RO Language.RU Russian ru-RU Language.ES Spanish es-ES Language.ES_MX Spanish (Mexico) es-MX Language.ES_US Spanish (US) es-US Language.SV Swedish sv-SE Language.TR Turkish tr-TR Language.CY Welsh cy-GB Language.CY_GB Welsh cy-GB ​ Usage Example Copy Ask AI from pipecat.services.aws.tts import AWSPollyTTSService from pipecat.transcriptions.language import Language # Configure service using environment variables for credentials tts = AWSPollyTTSService( region = "us-west-2" , voice_id = "Joanna" , params = AWSPollyTTSService.InputParams( engine = "neural" , language = Language. EN , rate = "+10%" , volume = "loud" ) ) # Or provide credentials directly tts = AWSPollyTTSService( aws_access_key_id = "YOUR_ACCESS_KEY_ID" , api_key = "YOUR_SECRET_ACCESS_KEY" , region = "us-west-2" , voice_id = "Joanna" , params = AWSPollyTTSService.InputParams( engine = "generative" , # For newer generative voices language = Language. EN , rate = "1.1" # Generative engine rate format ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ SSML Support The service automatically constructs SSML tags for advanced speech control: Copy Ask AI # Example with SSML controls service = AWSPollyTTSService( # ... other params ... params = AWSPollyTTSService.InputParams( engine = "neural" , rate = "+20%" , # Increase speed pitch = "low" , # Lower pitch volume = "loud" # Increase volume ) ) Prosody tags (pitch, rate, volume) have different behaviors based on the engine: - Standard engine: Supports all prosody tags - Neural engine: Full prosody support - Generative engine: Only rate is supported, with a different format (e.g., “1.1” for 10% faster) ​ Frame Flow ​ Metrics Support The service collects processing metrics: Time to First Byte (TTFB) Processing duration Character usage API calls ​ Notes Supports all AWS Polly engines: Standard (non-neural voices) Neural (improved quality voices) Generative (high-quality, natural-sounding voices) Automatic audio resampling from 16kHz to any desired rate Thread-safe processing Automatic error handling Manages AWS client lifecycle Together AI Azure On this page Overview Installation Configuration Constructor Parameters Input Parameters Output Frames Control Frames Audio Frames Error Frames Methods Language Support Usage Example SSML Support Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_aws_87fc0d41.txt b/tts_aws_87fc0d41.txt
new file mode 100644
index 0000000000000000000000000000000000000000..be6498ffdddcc06b9e769909e1b6e6ce7fac59a9
--- /dev/null
+++ b/tts_aws_87fc0d41.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/aws#param-sample-rate
+Title: AWS Polly - Pipecat
+==================================================
+
+AWS Polly - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech AWS Polly Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview AWSPollyTTSService provides text-to-speech capabilities using AWS’s Polly service. It supports multiple voices, languages, and speech customization options through SSML. The older PollyTTSService class is still available but has been deprecated. Use AWSPollyTTSService instead. ​ Installation To use AWSPollyTTSService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[aws]" You’ll also need to set up your AWS credentials as environment variables: AWS_SECRET_ACCESS_KEY AWS_ACCESS_KEY_ID AWS_SESSION_TOKEN (if using temporary credentials) AWS_REGION (defaults to “us-east-1”) ​ Configuration ​ Constructor Parameters ​ api_key str AWS secret access key (can also use environment variable) ​ aws_access_key_id str AWS access key ID (can also use environment variable) ​ aws_session_token str AWS session token for temporary credentials (can also use environment variable) ​ region str AWS region name (defaults to “us-east-1” if not provided) ​ voice_id str default: "Joanna" AWS Polly voice identifier ​ sample_rate int default: "None" Output audio sample rate in Hz (resampled from Polly’s 16kHz) ​ text_filter BaseTextFilter default: "None" Modifies text provided to the TTS. Learn more about the available filters. ​ params InputParams TTS configuration parameters ​ Input Parameters Copy Ask AI class InputParams ( BaseModel ): engine: Optional[ str ] = None # Polly engine type ("standard", "neural", or "generative") language: Optional[Language] = Language. EN pitch: Optional[ str ] = None # SSML pitch adjustment rate: Optional[ str ] = None # SSML rate adjustment volume: Optional[ str ] = None # SSML volume adjustment ​ Output Frames ​ Control Frames ​ TTSStartedFrame Frame Signals start of speech synthesis ​ TTSStoppedFrame Frame Signals completion of speech synthesis ​ Audio Frames ​ TTSAudioRawFrame Frame Contains generated audio data with: PCM audio format Sample rate as specified (resampled from 16kHz) Single channel (mono) ​ Error Frames ​ ErrorFrame Frame Contains AWS Polly error information ​ Methods See the TTS base class methods for additional functionality. ​ Language Support Supports an extensive range of languages and regional variants: Language Code Description Service Code Language.AR Arabic arb Language.AR_AE Arabic (UAE) ar-AE Language.CA Catalan ca-ES Language.ZH Chinese (Mandarin) cmn-CN Language.YUE Chinese (Cantonese) yue-CN Language.YUE_CN Chinese (Cantonese) yue-CN Language.CS Czech cs-CZ Language.DA Danish da-DK Language.NL Dutch nl-NL Language.NL_BE Dutch (Belgium) nl-BE Language.EN English (US) en-US Language.EN_AU English (Australia) en-AU Language.EN_GB English (UK) en-GB Language.EN_IN English (India) en-IN Language.EN_NZ English (New Zealand) en-NZ Language.EN_US English (US) en-US Language.EN_ZA English (South Africa) en-ZA Language.FI Finnish fi-FI Language.FR French fr-FR Language.FR_BE French (Belgium) fr-BE Language.FR_CA French (Canada) fr-CA Language.DE German de-DE Language.DE_AT German (Austria) de-AT Language.DE_CH German (Switzerland) de-CH Language.HI Hindi hi-IN Language.IS Icelandic is-IS Language.IT Italian it-IT Language.JA Japanese ja-JP Language.KO Korean ko-KR Language.NO Norwegian nb-NO Language.NB Norwegian (Bokmål) nb-NO Language.NB_NO Norwegian (Bokmål) nb-NO Language.PL Polish pl-PL Language.PT Portuguese pt-PT Language.PT_BR Portuguese (Brazil) pt-BR Language.PT_PT Portuguese (Portugal) pt-PT Language.RO Romanian ro-RO Language.RU Russian ru-RU Language.ES Spanish es-ES Language.ES_MX Spanish (Mexico) es-MX Language.ES_US Spanish (US) es-US Language.SV Swedish sv-SE Language.TR Turkish tr-TR Language.CY Welsh cy-GB Language.CY_GB Welsh cy-GB ​ Usage Example Copy Ask AI from pipecat.services.aws.tts import AWSPollyTTSService from pipecat.transcriptions.language import Language # Configure service using environment variables for credentials tts = AWSPollyTTSService( region = "us-west-2" , voice_id = "Joanna" , params = AWSPollyTTSService.InputParams( engine = "neural" , language = Language. EN , rate = "+10%" , volume = "loud" ) ) # Or provide credentials directly tts = AWSPollyTTSService( aws_access_key_id = "YOUR_ACCESS_KEY_ID" , api_key = "YOUR_SECRET_ACCESS_KEY" , region = "us-west-2" , voice_id = "Joanna" , params = AWSPollyTTSService.InputParams( engine = "generative" , # For newer generative voices language = Language. EN , rate = "1.1" # Generative engine rate format ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ SSML Support The service automatically constructs SSML tags for advanced speech control: Copy Ask AI # Example with SSML controls service = AWSPollyTTSService( # ... other params ... params = AWSPollyTTSService.InputParams( engine = "neural" , rate = "+20%" , # Increase speed pitch = "low" , # Lower pitch volume = "loud" # Increase volume ) ) Prosody tags (pitch, rate, volume) have different behaviors based on the engine: - Standard engine: Supports all prosody tags - Neural engine: Full prosody support - Generative engine: Only rate is supported, with a different format (e.g., “1.1” for 10% faster) ​ Frame Flow ​ Metrics Support The service collects processing metrics: Time to First Byte (TTFB) Processing duration Character usage API calls ​ Notes Supports all AWS Polly engines: Standard (non-neural voices) Neural (improved quality voices) Generative (high-quality, natural-sounding voices) Automatic audio resampling from 16kHz to any desired rate Thread-safe processing Automatic error handling Manages AWS client lifecycle Together AI Azure On this page Overview Installation Configuration Constructor Parameters Input Parameters Output Frames Control Frames Audio Frames Error Frames Methods Language Support Usage Example SSML Support Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_aws_ef1f37f9.txt b/tts_aws_ef1f37f9.txt
new file mode 100644
index 0000000000000000000000000000000000000000..c58dc09ba7f3bd84789dbd2805a18766c9bc35f6
--- /dev/null
+++ b/tts_aws_ef1f37f9.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/aws#audio-frames
+Title: AWS Polly - Pipecat
+==================================================
+
+AWS Polly - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech AWS Polly Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview AWSPollyTTSService provides text-to-speech capabilities using AWS’s Polly service. It supports multiple voices, languages, and speech customization options through SSML. The older PollyTTSService class is still available but has been deprecated. Use AWSPollyTTSService instead. ​ Installation To use AWSPollyTTSService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[aws]" You’ll also need to set up your AWS credentials as environment variables: AWS_SECRET_ACCESS_KEY AWS_ACCESS_KEY_ID AWS_SESSION_TOKEN (if using temporary credentials) AWS_REGION (defaults to “us-east-1”) ​ Configuration ​ Constructor Parameters ​ api_key str AWS secret access key (can also use environment variable) ​ aws_access_key_id str AWS access key ID (can also use environment variable) ​ aws_session_token str AWS session token for temporary credentials (can also use environment variable) ​ region str AWS region name (defaults to “us-east-1” if not provided) ​ voice_id str default: "Joanna" AWS Polly voice identifier ​ sample_rate int default: "None" Output audio sample rate in Hz (resampled from Polly’s 16kHz) ​ text_filter BaseTextFilter default: "None" Modifies text provided to the TTS. Learn more about the available filters. ​ params InputParams TTS configuration parameters ​ Input Parameters Copy Ask AI class InputParams ( BaseModel ): engine: Optional[ str ] = None # Polly engine type ("standard", "neural", or "generative") language: Optional[Language] = Language. EN pitch: Optional[ str ] = None # SSML pitch adjustment rate: Optional[ str ] = None # SSML rate adjustment volume: Optional[ str ] = None # SSML volume adjustment ​ Output Frames ​ Control Frames ​ TTSStartedFrame Frame Signals start of speech synthesis ​ TTSStoppedFrame Frame Signals completion of speech synthesis ​ Audio Frames ​ TTSAudioRawFrame Frame Contains generated audio data with: PCM audio format Sample rate as specified (resampled from 16kHz) Single channel (mono) ​ Error Frames ​ ErrorFrame Frame Contains AWS Polly error information ​ Methods See the TTS base class methods for additional functionality. ​ Language Support Supports an extensive range of languages and regional variants: Language Code Description Service Code Language.AR Arabic arb Language.AR_AE Arabic (UAE) ar-AE Language.CA Catalan ca-ES Language.ZH Chinese (Mandarin) cmn-CN Language.YUE Chinese (Cantonese) yue-CN Language.YUE_CN Chinese (Cantonese) yue-CN Language.CS Czech cs-CZ Language.DA Danish da-DK Language.NL Dutch nl-NL Language.NL_BE Dutch (Belgium) nl-BE Language.EN English (US) en-US Language.EN_AU English (Australia) en-AU Language.EN_GB English (UK) en-GB Language.EN_IN English (India) en-IN Language.EN_NZ English (New Zealand) en-NZ Language.EN_US English (US) en-US Language.EN_ZA English (South Africa) en-ZA Language.FI Finnish fi-FI Language.FR French fr-FR Language.FR_BE French (Belgium) fr-BE Language.FR_CA French (Canada) fr-CA Language.DE German de-DE Language.DE_AT German (Austria) de-AT Language.DE_CH German (Switzerland) de-CH Language.HI Hindi hi-IN Language.IS Icelandic is-IS Language.IT Italian it-IT Language.JA Japanese ja-JP Language.KO Korean ko-KR Language.NO Norwegian nb-NO Language.NB Norwegian (Bokmål) nb-NO Language.NB_NO Norwegian (Bokmål) nb-NO Language.PL Polish pl-PL Language.PT Portuguese pt-PT Language.PT_BR Portuguese (Brazil) pt-BR Language.PT_PT Portuguese (Portugal) pt-PT Language.RO Romanian ro-RO Language.RU Russian ru-RU Language.ES Spanish es-ES Language.ES_MX Spanish (Mexico) es-MX Language.ES_US Spanish (US) es-US Language.SV Swedish sv-SE Language.TR Turkish tr-TR Language.CY Welsh cy-GB Language.CY_GB Welsh cy-GB ​ Usage Example Copy Ask AI from pipecat.services.aws.tts import AWSPollyTTSService from pipecat.transcriptions.language import Language # Configure service using environment variables for credentials tts = AWSPollyTTSService( region = "us-west-2" , voice_id = "Joanna" , params = AWSPollyTTSService.InputParams( engine = "neural" , language = Language. EN , rate = "+10%" , volume = "loud" ) ) # Or provide credentials directly tts = AWSPollyTTSService( aws_access_key_id = "YOUR_ACCESS_KEY_ID" , api_key = "YOUR_SECRET_ACCESS_KEY" , region = "us-west-2" , voice_id = "Joanna" , params = AWSPollyTTSService.InputParams( engine = "generative" , # For newer generative voices language = Language. EN , rate = "1.1" # Generative engine rate format ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ SSML Support The service automatically constructs SSML tags for advanced speech control: Copy Ask AI # Example with SSML controls service = AWSPollyTTSService( # ... other params ... params = AWSPollyTTSService.InputParams( engine = "neural" , rate = "+20%" , # Increase speed pitch = "low" , # Lower pitch volume = "loud" # Increase volume ) ) Prosody tags (pitch, rate, volume) have different behaviors based on the engine: - Standard engine: Supports all prosody tags - Neural engine: Full prosody support - Generative engine: Only rate is supported, with a different format (e.g., “1.1” for 10% faster) ​ Frame Flow ​ Metrics Support The service collects processing metrics: Time to First Byte (TTFB) Processing duration Character usage API calls ​ Notes Supports all AWS Polly engines: Standard (non-neural voices) Neural (improved quality voices) Generative (high-quality, natural-sounding voices) Automatic audio resampling from 16kHz to any desired rate Thread-safe processing Automatic error handling Manages AWS client lifecycle Together AI Azure On this page Overview Installation Configuration Constructor Parameters Input Parameters Output Frames Control Frames Audio Frames Error Frames Methods Language Support Usage Example SSML Support Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_aws_f075526e.txt b/tts_aws_f075526e.txt
new file mode 100644
index 0000000000000000000000000000000000000000..b07f961a7806b0f4235ea21bc671ceb2a01f0e02
--- /dev/null
+++ b/tts_aws_f075526e.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/aws#param-voice-id
+Title: AWS Polly - Pipecat
+==================================================
+
+AWS Polly - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech AWS Polly Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview AWSPollyTTSService provides text-to-speech capabilities using AWS’s Polly service. It supports multiple voices, languages, and speech customization options through SSML. The older PollyTTSService class is still available but has been deprecated. Use AWSPollyTTSService instead. ​ Installation To use AWSPollyTTSService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[aws]" You’ll also need to set up your AWS credentials as environment variables: AWS_SECRET_ACCESS_KEY AWS_ACCESS_KEY_ID AWS_SESSION_TOKEN (if using temporary credentials) AWS_REGION (defaults to “us-east-1”) ​ Configuration ​ Constructor Parameters ​ api_key str AWS secret access key (can also use environment variable) ​ aws_access_key_id str AWS access key ID (can also use environment variable) ​ aws_session_token str AWS session token for temporary credentials (can also use environment variable) ​ region str AWS region name (defaults to “us-east-1” if not provided) ​ voice_id str default: "Joanna" AWS Polly voice identifier ​ sample_rate int default: "None" Output audio sample rate in Hz (resampled from Polly’s 16kHz) ​ text_filter BaseTextFilter default: "None" Modifies text provided to the TTS. Learn more about the available filters. ​ params InputParams TTS configuration parameters ​ Input Parameters Copy Ask AI class InputParams ( BaseModel ): engine: Optional[ str ] = None # Polly engine type ("standard", "neural", or "generative") language: Optional[Language] = Language. EN pitch: Optional[ str ] = None # SSML pitch adjustment rate: Optional[ str ] = None # SSML rate adjustment volume: Optional[ str ] = None # SSML volume adjustment ​ Output Frames ​ Control Frames ​ TTSStartedFrame Frame Signals start of speech synthesis ​ TTSStoppedFrame Frame Signals completion of speech synthesis ​ Audio Frames ​ TTSAudioRawFrame Frame Contains generated audio data with: PCM audio format Sample rate as specified (resampled from 16kHz) Single channel (mono) ​ Error Frames ​ ErrorFrame Frame Contains AWS Polly error information ​ Methods See the TTS base class methods for additional functionality. ​ Language Support Supports an extensive range of languages and regional variants: Language Code Description Service Code Language.AR Arabic arb Language.AR_AE Arabic (UAE) ar-AE Language.CA Catalan ca-ES Language.ZH Chinese (Mandarin) cmn-CN Language.YUE Chinese (Cantonese) yue-CN Language.YUE_CN Chinese (Cantonese) yue-CN Language.CS Czech cs-CZ Language.DA Danish da-DK Language.NL Dutch nl-NL Language.NL_BE Dutch (Belgium) nl-BE Language.EN English (US) en-US Language.EN_AU English (Australia) en-AU Language.EN_GB English (UK) en-GB Language.EN_IN English (India) en-IN Language.EN_NZ English (New Zealand) en-NZ Language.EN_US English (US) en-US Language.EN_ZA English (South Africa) en-ZA Language.FI Finnish fi-FI Language.FR French fr-FR Language.FR_BE French (Belgium) fr-BE Language.FR_CA French (Canada) fr-CA Language.DE German de-DE Language.DE_AT German (Austria) de-AT Language.DE_CH German (Switzerland) de-CH Language.HI Hindi hi-IN Language.IS Icelandic is-IS Language.IT Italian it-IT Language.JA Japanese ja-JP Language.KO Korean ko-KR Language.NO Norwegian nb-NO Language.NB Norwegian (Bokmål) nb-NO Language.NB_NO Norwegian (Bokmål) nb-NO Language.PL Polish pl-PL Language.PT Portuguese pt-PT Language.PT_BR Portuguese (Brazil) pt-BR Language.PT_PT Portuguese (Portugal) pt-PT Language.RO Romanian ro-RO Language.RU Russian ru-RU Language.ES Spanish es-ES Language.ES_MX Spanish (Mexico) es-MX Language.ES_US Spanish (US) es-US Language.SV Swedish sv-SE Language.TR Turkish tr-TR Language.CY Welsh cy-GB Language.CY_GB Welsh cy-GB ​ Usage Example Copy Ask AI from pipecat.services.aws.tts import AWSPollyTTSService from pipecat.transcriptions.language import Language # Configure service using environment variables for credentials tts = AWSPollyTTSService( region = "us-west-2" , voice_id = "Joanna" , params = AWSPollyTTSService.InputParams( engine = "neural" , language = Language. EN , rate = "+10%" , volume = "loud" ) ) # Or provide credentials directly tts = AWSPollyTTSService( aws_access_key_id = "YOUR_ACCESS_KEY_ID" , api_key = "YOUR_SECRET_ACCESS_KEY" , region = "us-west-2" , voice_id = "Joanna" , params = AWSPollyTTSService.InputParams( engine = "generative" , # For newer generative voices language = Language. EN , rate = "1.1" # Generative engine rate format ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ SSML Support The service automatically constructs SSML tags for advanced speech control: Copy Ask AI # Example with SSML controls service = AWSPollyTTSService( # ... other params ... params = AWSPollyTTSService.InputParams( engine = "neural" , rate = "+20%" , # Increase speed pitch = "low" , # Lower pitch volume = "loud" # Increase volume ) ) Prosody tags (pitch, rate, volume) have different behaviors based on the engine: - Standard engine: Supports all prosody tags - Neural engine: Full prosody support - Generative engine: Only rate is supported, with a different format (e.g., “1.1” for 10% faster) ​ Frame Flow ​ Metrics Support The service collects processing metrics: Time to First Byte (TTFB) Processing duration Character usage API calls ​ Notes Supports all AWS Polly engines: Standard (non-neural voices) Neural (improved quality voices) Generative (high-quality, natural-sounding voices) Automatic audio resampling from 16kHz to any desired rate Thread-safe processing Automatic error handling Manages AWS client lifecycle Together AI Azure On this page Overview Installation Configuration Constructor Parameters Input Parameters Output Frames Control Frames Audio Frames Error Frames Methods Language Support Usage Example SSML Support Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_azure_0a569b86.txt b/tts_azure_0a569b86.txt
new file mode 100644
index 0000000000000000000000000000000000000000..80ff2a864253eb6db76f497f9ffa68056dc299bf
--- /dev/null
+++ b/tts_azure_0a569b86.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/azure#methods
+Title: Azure - Pipecat
+==================================================
+
+Azure - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech Azure Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview AzureTTSService provides high-quality text-to-speech synthesis using Azure’s Cognitive Services. It supports SSML for advanced voice control and multiple languages. ​ Installation To use AzureTTSService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[azure]" You’ll also need to set up the following environment variables: AZURE_API_KEY AZURE_REGION ​ Configuration ​ Constructor Parameters ​ api_key str required Azure Speech Service API key ​ region str required Azure region identifier ​ voice str default: "en-US-SaraNeural" Voice identifier ​ sample_rate int default: "None" Output audio sample rate in Hz ​ text_filter BaseTextFilter default: "None" Modifies text provided to the TTS. Learn more about the available filters. ​ Input Parameters Copy Ask AI class InputParams ( BaseModel ): emphasis: Optional[ str ] language: Optional[Language] = Language. EN_US pitch: Optional[ str ] rate: Optional[ str ] = "1.05" role: Optional[ str ] style: Optional[ str ] style_degree: Optional[ str ] volume: Optional[ str ] ​ Supported Sample Rates 8000 Hz: Raw8Khz16BitMonoPcm 16000 Hz: Raw16Khz16BitMonoPcm 22050 Hz: Raw22050Hz16BitMonoPcm 24000 Hz: Raw24Khz16BitMonoPcm 44100 Hz: Raw44100Hz16BitMonoPcm 48000 Hz: Raw48Khz16BitMonoPcm ​ Usage Example Copy Ask AI # Configure service tts = AzureTTSService( api_key = "your-api-key" , region = "eastus" , voice = "en-US-JennyNeural" , params = AzureTTSService.InputParams( language = Language. EN_US , rate = "1.1" , style = "cheerful" ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ Methods See the TTS base class methods for additional functionality. ​ Language Support Azure Speech Services support the following languages and regional variants: Language Code Description Service Code Language.BG Bulgarian bg-BG Language.CA Catalan ca-ES Language.ZH Chinese (Simplified) zh-CN Language.ZH_TW Chinese (Traditional) zh-TW Language.CS Czech cs-CZ Language.DA Danish da-DK Language.NL Dutch (Netherlands) nl-NL Language.NL_BE Dutch (Belgium) nl-BE Language.EN English (US) en-US Language.EN_US English (US) en-US Language.EN_AU English (Australia) en-AU Language.EN_GB English (UK) en-GB Language.EN_NZ English (New Zealand) en-NZ Language.EN_IN English (India) en-IN Language.ET Estonian et-EE Language.FI Finnish fi-FI Language.FR French (France) fr-FR Language.FR_CA French (Canada) fr-CA Language.DE German (Germany) de-DE Language.DE_CH German (Switzerland) de-CH Language.EL Greek el-GR Language.HI Hindi hi-IN Language.HU Hungarian hu-HU Language.ID Indonesian id-ID Language.IT Italian it-IT Language.JA Japanese ja-JP Language.KO Korean ko-KR Language.LV Latvian lv-LV Language.LT Lithuanian lt-LT Language.MS Malay ms-MY Language.NO Norwegian nb-NO Language.PL Polish pl-PL Language.PT Portuguese (Portugal) pt-PT Language.PT_BR Portuguese (Brazil) pt-BR Language.RO Romanian ro-RO Language.RU Russian ru-RU Language.SK Slovak sk-SK Language.ES Spanish es-ES Language.SV Swedish sv-SE Language.TH Thai th-TH Language.TR Turkish tr-TR Language.UK Ukrainian uk-UA Language.VI Vietnamese vi-VN ​ Usage Examples ​ TTS Configuration Copy Ask AI # Configure TTS with specific language tts = AzureTTSService( api_key = "your-api-key" , region = "eastus" , params = AzureTTSService.InputParams( language = Language. FR_CA , # Canadian French voice = "fr-CA-SylvieNeural" ) ) ​ Regional Considerations Each language code includes both language and region (e.g., fr-FR for French in France) Some languages have multiple regional variants (e.g., English has US, UK, Australian, Indian, and New Zealand variants) Voice availability may vary by region and language Neural voices are available for most language/region combinations Some features (like custom pronunciation) may be limited to specific languages Note: Voice selection should match the specified language code for optimal results. Check Azure’s documentation for the latest list of available voices for each language/region combination. ​ SSML Support The service supports rich SSML customization: Copy Ask AI # Example with multiple SSML features params = AzureTTSService.InputParams( emphasis = "strong" , pitch = "+2st" , rate = "1.2" , style = "cheerful" , style_degree = "2" , volume = "loud" ) ​ Frame Flow ​ Metrics Support The service collects processing metrics: Time to First Byte (TTFB) Processing duration Character usage API calls ​ Notes SSML-based speech customization Chunked audio delivery Thread-safe processing Automatic error handling Manages Azure client lifecycle AWS Polly Cartesia On this page Overview Installation Configuration Constructor Parameters Input Parameters Supported Sample Rates Usage Example Methods Language Support Usage Examples TTS Configuration Regional Considerations SSML Support Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_azure_1651c045.txt b/tts_azure_1651c045.txt
new file mode 100644
index 0000000000000000000000000000000000000000..18b655d5934346a13fcd9efa561ce339d0a0eb17
--- /dev/null
+++ b/tts_azure_1651c045.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/azure#input-parameters
+Title: Azure - Pipecat
+==================================================
+
+Azure - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech Azure Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview AzureTTSService provides high-quality text-to-speech synthesis using Azure’s Cognitive Services. It supports SSML for advanced voice control and multiple languages. ​ Installation To use AzureTTSService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[azure]" You’ll also need to set up the following environment variables: AZURE_API_KEY AZURE_REGION ​ Configuration ​ Constructor Parameters ​ api_key str required Azure Speech Service API key ​ region str required Azure region identifier ​ voice str default: "en-US-SaraNeural" Voice identifier ​ sample_rate int default: "None" Output audio sample rate in Hz ​ text_filter BaseTextFilter default: "None" Modifies text provided to the TTS. Learn more about the available filters. ​ Input Parameters Copy Ask AI class InputParams ( BaseModel ): emphasis: Optional[ str ] language: Optional[Language] = Language. EN_US pitch: Optional[ str ] rate: Optional[ str ] = "1.05" role: Optional[ str ] style: Optional[ str ] style_degree: Optional[ str ] volume: Optional[ str ] ​ Supported Sample Rates 8000 Hz: Raw8Khz16BitMonoPcm 16000 Hz: Raw16Khz16BitMonoPcm 22050 Hz: Raw22050Hz16BitMonoPcm 24000 Hz: Raw24Khz16BitMonoPcm 44100 Hz: Raw44100Hz16BitMonoPcm 48000 Hz: Raw48Khz16BitMonoPcm ​ Usage Example Copy Ask AI # Configure service tts = AzureTTSService( api_key = "your-api-key" , region = "eastus" , voice = "en-US-JennyNeural" , params = AzureTTSService.InputParams( language = Language. EN_US , rate = "1.1" , style = "cheerful" ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ Methods See the TTS base class methods for additional functionality. ​ Language Support Azure Speech Services support the following languages and regional variants: Language Code Description Service Code Language.BG Bulgarian bg-BG Language.CA Catalan ca-ES Language.ZH Chinese (Simplified) zh-CN Language.ZH_TW Chinese (Traditional) zh-TW Language.CS Czech cs-CZ Language.DA Danish da-DK Language.NL Dutch (Netherlands) nl-NL Language.NL_BE Dutch (Belgium) nl-BE Language.EN English (US) en-US Language.EN_US English (US) en-US Language.EN_AU English (Australia) en-AU Language.EN_GB English (UK) en-GB Language.EN_NZ English (New Zealand) en-NZ Language.EN_IN English (India) en-IN Language.ET Estonian et-EE Language.FI Finnish fi-FI Language.FR French (France) fr-FR Language.FR_CA French (Canada) fr-CA Language.DE German (Germany) de-DE Language.DE_CH German (Switzerland) de-CH Language.EL Greek el-GR Language.HI Hindi hi-IN Language.HU Hungarian hu-HU Language.ID Indonesian id-ID Language.IT Italian it-IT Language.JA Japanese ja-JP Language.KO Korean ko-KR Language.LV Latvian lv-LV Language.LT Lithuanian lt-LT Language.MS Malay ms-MY Language.NO Norwegian nb-NO Language.PL Polish pl-PL Language.PT Portuguese (Portugal) pt-PT Language.PT_BR Portuguese (Brazil) pt-BR Language.RO Romanian ro-RO Language.RU Russian ru-RU Language.SK Slovak sk-SK Language.ES Spanish es-ES Language.SV Swedish sv-SE Language.TH Thai th-TH Language.TR Turkish tr-TR Language.UK Ukrainian uk-UA Language.VI Vietnamese vi-VN ​ Usage Examples ​ TTS Configuration Copy Ask AI # Configure TTS with specific language tts = AzureTTSService( api_key = "your-api-key" , region = "eastus" , params = AzureTTSService.InputParams( language = Language. FR_CA , # Canadian French voice = "fr-CA-SylvieNeural" ) ) ​ Regional Considerations Each language code includes both language and region (e.g., fr-FR for French in France) Some languages have multiple regional variants (e.g., English has US, UK, Australian, Indian, and New Zealand variants) Voice availability may vary by region and language Neural voices are available for most language/region combinations Some features (like custom pronunciation) may be limited to specific languages Note: Voice selection should match the specified language code for optimal results. Check Azure’s documentation for the latest list of available voices for each language/region combination. ​ SSML Support The service supports rich SSML customization: Copy Ask AI # Example with multiple SSML features params = AzureTTSService.InputParams( emphasis = "strong" , pitch = "+2st" , rate = "1.2" , style = "cheerful" , style_degree = "2" , volume = "loud" ) ​ Frame Flow ​ Metrics Support The service collects processing metrics: Time to First Byte (TTFB) Processing duration Character usage API calls ​ Notes SSML-based speech customization Chunked audio delivery Thread-safe processing Automatic error handling Manages Azure client lifecycle AWS Polly Cartesia On this page Overview Installation Configuration Constructor Parameters Input Parameters Supported Sample Rates Usage Example Methods Language Support Usage Examples TTS Configuration Regional Considerations SSML Support Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_azure_1eea80a2.txt b/tts_azure_1eea80a2.txt
new file mode 100644
index 0000000000000000000000000000000000000000..959b47da576b468a52604193f9b38cadc98b18e6
--- /dev/null
+++ b/tts_azure_1eea80a2.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/azure#param-voice
+Title: Azure - Pipecat
+==================================================
+
+Azure - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech Azure Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview AzureTTSService provides high-quality text-to-speech synthesis using Azure’s Cognitive Services. It supports SSML for advanced voice control and multiple languages. ​ Installation To use AzureTTSService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[azure]" You’ll also need to set up the following environment variables: AZURE_API_KEY AZURE_REGION ​ Configuration ​ Constructor Parameters ​ api_key str required Azure Speech Service API key ​ region str required Azure region identifier ​ voice str default: "en-US-SaraNeural" Voice identifier ​ sample_rate int default: "None" Output audio sample rate in Hz ​ text_filter BaseTextFilter default: "None" Modifies text provided to the TTS. Learn more about the available filters. ​ Input Parameters Copy Ask AI class InputParams ( BaseModel ): emphasis: Optional[ str ] language: Optional[Language] = Language. EN_US pitch: Optional[ str ] rate: Optional[ str ] = "1.05" role: Optional[ str ] style: Optional[ str ] style_degree: Optional[ str ] volume: Optional[ str ] ​ Supported Sample Rates 8000 Hz: Raw8Khz16BitMonoPcm 16000 Hz: Raw16Khz16BitMonoPcm 22050 Hz: Raw22050Hz16BitMonoPcm 24000 Hz: Raw24Khz16BitMonoPcm 44100 Hz: Raw44100Hz16BitMonoPcm 48000 Hz: Raw48Khz16BitMonoPcm ​ Usage Example Copy Ask AI # Configure service tts = AzureTTSService( api_key = "your-api-key" , region = "eastus" , voice = "en-US-JennyNeural" , params = AzureTTSService.InputParams( language = Language. EN_US , rate = "1.1" , style = "cheerful" ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ Methods See the TTS base class methods for additional functionality. ​ Language Support Azure Speech Services support the following languages and regional variants: Language Code Description Service Code Language.BG Bulgarian bg-BG Language.CA Catalan ca-ES Language.ZH Chinese (Simplified) zh-CN Language.ZH_TW Chinese (Traditional) zh-TW Language.CS Czech cs-CZ Language.DA Danish da-DK Language.NL Dutch (Netherlands) nl-NL Language.NL_BE Dutch (Belgium) nl-BE Language.EN English (US) en-US Language.EN_US English (US) en-US Language.EN_AU English (Australia) en-AU Language.EN_GB English (UK) en-GB Language.EN_NZ English (New Zealand) en-NZ Language.EN_IN English (India) en-IN Language.ET Estonian et-EE Language.FI Finnish fi-FI Language.FR French (France) fr-FR Language.FR_CA French (Canada) fr-CA Language.DE German (Germany) de-DE Language.DE_CH German (Switzerland) de-CH Language.EL Greek el-GR Language.HI Hindi hi-IN Language.HU Hungarian hu-HU Language.ID Indonesian id-ID Language.IT Italian it-IT Language.JA Japanese ja-JP Language.KO Korean ko-KR Language.LV Latvian lv-LV Language.LT Lithuanian lt-LT Language.MS Malay ms-MY Language.NO Norwegian nb-NO Language.PL Polish pl-PL Language.PT Portuguese (Portugal) pt-PT Language.PT_BR Portuguese (Brazil) pt-BR Language.RO Romanian ro-RO Language.RU Russian ru-RU Language.SK Slovak sk-SK Language.ES Spanish es-ES Language.SV Swedish sv-SE Language.TH Thai th-TH Language.TR Turkish tr-TR Language.UK Ukrainian uk-UA Language.VI Vietnamese vi-VN ​ Usage Examples ​ TTS Configuration Copy Ask AI # Configure TTS with specific language tts = AzureTTSService( api_key = "your-api-key" , region = "eastus" , params = AzureTTSService.InputParams( language = Language. FR_CA , # Canadian French voice = "fr-CA-SylvieNeural" ) ) ​ Regional Considerations Each language code includes both language and region (e.g., fr-FR for French in France) Some languages have multiple regional variants (e.g., English has US, UK, Australian, Indian, and New Zealand variants) Voice availability may vary by region and language Neural voices are available for most language/region combinations Some features (like custom pronunciation) may be limited to specific languages Note: Voice selection should match the specified language code for optimal results. Check Azure’s documentation for the latest list of available voices for each language/region combination. ​ SSML Support The service supports rich SSML customization: Copy Ask AI # Example with multiple SSML features params = AzureTTSService.InputParams( emphasis = "strong" , pitch = "+2st" , rate = "1.2" , style = "cheerful" , style_degree = "2" , volume = "loud" ) ​ Frame Flow ​ Metrics Support The service collects processing metrics: Time to First Byte (TTFB) Processing duration Character usage API calls ​ Notes SSML-based speech customization Chunked audio delivery Thread-safe processing Automatic error handling Manages Azure client lifecycle AWS Polly Cartesia On this page Overview Installation Configuration Constructor Parameters Input Parameters Supported Sample Rates Usage Example Methods Language Support Usage Examples TTS Configuration Regional Considerations SSML Support Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_azure_3b572d24.txt b/tts_azure_3b572d24.txt
new file mode 100644
index 0000000000000000000000000000000000000000..8570b6f464e1936c79b935de0f1755f47d399baa
--- /dev/null
+++ b/tts_azure_3b572d24.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/azure#notes
+Title: Azure - Pipecat
+==================================================
+
+Azure - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech Azure Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview AzureTTSService provides high-quality text-to-speech synthesis using Azure’s Cognitive Services. It supports SSML for advanced voice control and multiple languages. ​ Installation To use AzureTTSService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[azure]" You’ll also need to set up the following environment variables: AZURE_API_KEY AZURE_REGION ​ Configuration ​ Constructor Parameters ​ api_key str required Azure Speech Service API key ​ region str required Azure region identifier ​ voice str default: "en-US-SaraNeural" Voice identifier ​ sample_rate int default: "None" Output audio sample rate in Hz ​ text_filter BaseTextFilter default: "None" Modifies text provided to the TTS. Learn more about the available filters. ​ Input Parameters Copy Ask AI class InputParams ( BaseModel ): emphasis: Optional[ str ] language: Optional[Language] = Language. EN_US pitch: Optional[ str ] rate: Optional[ str ] = "1.05" role: Optional[ str ] style: Optional[ str ] style_degree: Optional[ str ] volume: Optional[ str ] ​ Supported Sample Rates 8000 Hz: Raw8Khz16BitMonoPcm 16000 Hz: Raw16Khz16BitMonoPcm 22050 Hz: Raw22050Hz16BitMonoPcm 24000 Hz: Raw24Khz16BitMonoPcm 44100 Hz: Raw44100Hz16BitMonoPcm 48000 Hz: Raw48Khz16BitMonoPcm ​ Usage Example Copy Ask AI # Configure service tts = AzureTTSService( api_key = "your-api-key" , region = "eastus" , voice = "en-US-JennyNeural" , params = AzureTTSService.InputParams( language = Language. EN_US , rate = "1.1" , style = "cheerful" ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ Methods See the TTS base class methods for additional functionality. ​ Language Support Azure Speech Services support the following languages and regional variants: Language Code Description Service Code Language.BG Bulgarian bg-BG Language.CA Catalan ca-ES Language.ZH Chinese (Simplified) zh-CN Language.ZH_TW Chinese (Traditional) zh-TW Language.CS Czech cs-CZ Language.DA Danish da-DK Language.NL Dutch (Netherlands) nl-NL Language.NL_BE Dutch (Belgium) nl-BE Language.EN English (US) en-US Language.EN_US English (US) en-US Language.EN_AU English (Australia) en-AU Language.EN_GB English (UK) en-GB Language.EN_NZ English (New Zealand) en-NZ Language.EN_IN English (India) en-IN Language.ET Estonian et-EE Language.FI Finnish fi-FI Language.FR French (France) fr-FR Language.FR_CA French (Canada) fr-CA Language.DE German (Germany) de-DE Language.DE_CH German (Switzerland) de-CH Language.EL Greek el-GR Language.HI Hindi hi-IN Language.HU Hungarian hu-HU Language.ID Indonesian id-ID Language.IT Italian it-IT Language.JA Japanese ja-JP Language.KO Korean ko-KR Language.LV Latvian lv-LV Language.LT Lithuanian lt-LT Language.MS Malay ms-MY Language.NO Norwegian nb-NO Language.PL Polish pl-PL Language.PT Portuguese (Portugal) pt-PT Language.PT_BR Portuguese (Brazil) pt-BR Language.RO Romanian ro-RO Language.RU Russian ru-RU Language.SK Slovak sk-SK Language.ES Spanish es-ES Language.SV Swedish sv-SE Language.TH Thai th-TH Language.TR Turkish tr-TR Language.UK Ukrainian uk-UA Language.VI Vietnamese vi-VN ​ Usage Examples ​ TTS Configuration Copy Ask AI # Configure TTS with specific language tts = AzureTTSService( api_key = "your-api-key" , region = "eastus" , params = AzureTTSService.InputParams( language = Language. FR_CA , # Canadian French voice = "fr-CA-SylvieNeural" ) ) ​ Regional Considerations Each language code includes both language and region (e.g., fr-FR for French in France) Some languages have multiple regional variants (e.g., English has US, UK, Australian, Indian, and New Zealand variants) Voice availability may vary by region and language Neural voices are available for most language/region combinations Some features (like custom pronunciation) may be limited to specific languages Note: Voice selection should match the specified language code for optimal results. Check Azure’s documentation for the latest list of available voices for each language/region combination. ​ SSML Support The service supports rich SSML customization: Copy Ask AI # Example with multiple SSML features params = AzureTTSService.InputParams( emphasis = "strong" , pitch = "+2st" , rate = "1.2" , style = "cheerful" , style_degree = "2" , volume = "loud" ) ​ Frame Flow ​ Metrics Support The service collects processing metrics: Time to First Byte (TTFB) Processing duration Character usage API calls ​ Notes SSML-based speech customization Chunked audio delivery Thread-safe processing Automatic error handling Manages Azure client lifecycle AWS Polly Cartesia On this page Overview Installation Configuration Constructor Parameters Input Parameters Supported Sample Rates Usage Example Methods Language Support Usage Examples TTS Configuration Regional Considerations SSML Support Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_azure_75fc764f.txt b/tts_azure_75fc764f.txt
new file mode 100644
index 0000000000000000000000000000000000000000..95e219c7374eb58b30ee17d30c4e51306390e15f
--- /dev/null
+++ b/tts_azure_75fc764f.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/azure#usage-example
+Title: Azure - Pipecat
+==================================================
+
+Azure - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech Azure Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview AzureTTSService provides high-quality text-to-speech synthesis using Azure’s Cognitive Services. It supports SSML for advanced voice control and multiple languages. ​ Installation To use AzureTTSService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[azure]" You’ll also need to set up the following environment variables: AZURE_API_KEY AZURE_REGION ​ Configuration ​ Constructor Parameters ​ api_key str required Azure Speech Service API key ​ region str required Azure region identifier ​ voice str default: "en-US-SaraNeural" Voice identifier ​ sample_rate int default: "None" Output audio sample rate in Hz ​ text_filter BaseTextFilter default: "None" Modifies text provided to the TTS. Learn more about the available filters. ​ Input Parameters Copy Ask AI class InputParams ( BaseModel ): emphasis: Optional[ str ] language: Optional[Language] = Language. EN_US pitch: Optional[ str ] rate: Optional[ str ] = "1.05" role: Optional[ str ] style: Optional[ str ] style_degree: Optional[ str ] volume: Optional[ str ] ​ Supported Sample Rates 8000 Hz: Raw8Khz16BitMonoPcm 16000 Hz: Raw16Khz16BitMonoPcm 22050 Hz: Raw22050Hz16BitMonoPcm 24000 Hz: Raw24Khz16BitMonoPcm 44100 Hz: Raw44100Hz16BitMonoPcm 48000 Hz: Raw48Khz16BitMonoPcm ​ Usage Example Copy Ask AI # Configure service tts = AzureTTSService( api_key = "your-api-key" , region = "eastus" , voice = "en-US-JennyNeural" , params = AzureTTSService.InputParams( language = Language. EN_US , rate = "1.1" , style = "cheerful" ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ Methods See the TTS base class methods for additional functionality. ​ Language Support Azure Speech Services support the following languages and regional variants: Language Code Description Service Code Language.BG Bulgarian bg-BG Language.CA Catalan ca-ES Language.ZH Chinese (Simplified) zh-CN Language.ZH_TW Chinese (Traditional) zh-TW Language.CS Czech cs-CZ Language.DA Danish da-DK Language.NL Dutch (Netherlands) nl-NL Language.NL_BE Dutch (Belgium) nl-BE Language.EN English (US) en-US Language.EN_US English (US) en-US Language.EN_AU English (Australia) en-AU Language.EN_GB English (UK) en-GB Language.EN_NZ English (New Zealand) en-NZ Language.EN_IN English (India) en-IN Language.ET Estonian et-EE Language.FI Finnish fi-FI Language.FR French (France) fr-FR Language.FR_CA French (Canada) fr-CA Language.DE German (Germany) de-DE Language.DE_CH German (Switzerland) de-CH Language.EL Greek el-GR Language.HI Hindi hi-IN Language.HU Hungarian hu-HU Language.ID Indonesian id-ID Language.IT Italian it-IT Language.JA Japanese ja-JP Language.KO Korean ko-KR Language.LV Latvian lv-LV Language.LT Lithuanian lt-LT Language.MS Malay ms-MY Language.NO Norwegian nb-NO Language.PL Polish pl-PL Language.PT Portuguese (Portugal) pt-PT Language.PT_BR Portuguese (Brazil) pt-BR Language.RO Romanian ro-RO Language.RU Russian ru-RU Language.SK Slovak sk-SK Language.ES Spanish es-ES Language.SV Swedish sv-SE Language.TH Thai th-TH Language.TR Turkish tr-TR Language.UK Ukrainian uk-UA Language.VI Vietnamese vi-VN ​ Usage Examples ​ TTS Configuration Copy Ask AI # Configure TTS with specific language tts = AzureTTSService( api_key = "your-api-key" , region = "eastus" , params = AzureTTSService.InputParams( language = Language. FR_CA , # Canadian French voice = "fr-CA-SylvieNeural" ) ) ​ Regional Considerations Each language code includes both language and region (e.g., fr-FR for French in France) Some languages have multiple regional variants (e.g., English has US, UK, Australian, Indian, and New Zealand variants) Voice availability may vary by region and language Neural voices are available for most language/region combinations Some features (like custom pronunciation) may be limited to specific languages Note: Voice selection should match the specified language code for optimal results. Check Azure’s documentation for the latest list of available voices for each language/region combination. ​ SSML Support The service supports rich SSML customization: Copy Ask AI # Example with multiple SSML features params = AzureTTSService.InputParams( emphasis = "strong" , pitch = "+2st" , rate = "1.2" , style = "cheerful" , style_degree = "2" , volume = "loud" ) ​ Frame Flow ​ Metrics Support The service collects processing metrics: Time to First Byte (TTFB) Processing duration Character usage API calls ​ Notes SSML-based speech customization Chunked audio delivery Thread-safe processing Automatic error handling Manages Azure client lifecycle AWS Polly Cartesia On this page Overview Installation Configuration Constructor Parameters Input Parameters Supported Sample Rates Usage Example Methods Language Support Usage Examples TTS Configuration Regional Considerations SSML Support Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_azure_81614a43.txt b/tts_azure_81614a43.txt
new file mode 100644
index 0000000000000000000000000000000000000000..7cdb2887178771da06541fd3945357dbe792c0e6
--- /dev/null
+++ b/tts_azure_81614a43.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/azure#overview
+Title: Azure - Pipecat
+==================================================
+
+Azure - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech Azure Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview AzureTTSService provides high-quality text-to-speech synthesis using Azure’s Cognitive Services. It supports SSML for advanced voice control and multiple languages. ​ Installation To use AzureTTSService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[azure]" You’ll also need to set up the following environment variables: AZURE_API_KEY AZURE_REGION ​ Configuration ​ Constructor Parameters ​ api_key str required Azure Speech Service API key ​ region str required Azure region identifier ​ voice str default: "en-US-SaraNeural" Voice identifier ​ sample_rate int default: "None" Output audio sample rate in Hz ​ text_filter BaseTextFilter default: "None" Modifies text provided to the TTS. Learn more about the available filters. ​ Input Parameters Copy Ask AI class InputParams ( BaseModel ): emphasis: Optional[ str ] language: Optional[Language] = Language. EN_US pitch: Optional[ str ] rate: Optional[ str ] = "1.05" role: Optional[ str ] style: Optional[ str ] style_degree: Optional[ str ] volume: Optional[ str ] ​ Supported Sample Rates 8000 Hz: Raw8Khz16BitMonoPcm 16000 Hz: Raw16Khz16BitMonoPcm 22050 Hz: Raw22050Hz16BitMonoPcm 24000 Hz: Raw24Khz16BitMonoPcm 44100 Hz: Raw44100Hz16BitMonoPcm 48000 Hz: Raw48Khz16BitMonoPcm ​ Usage Example Copy Ask AI # Configure service tts = AzureTTSService( api_key = "your-api-key" , region = "eastus" , voice = "en-US-JennyNeural" , params = AzureTTSService.InputParams( language = Language. EN_US , rate = "1.1" , style = "cheerful" ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ Methods See the TTS base class methods for additional functionality. ​ Language Support Azure Speech Services support the following languages and regional variants: Language Code Description Service Code Language.BG Bulgarian bg-BG Language.CA Catalan ca-ES Language.ZH Chinese (Simplified) zh-CN Language.ZH_TW Chinese (Traditional) zh-TW Language.CS Czech cs-CZ Language.DA Danish da-DK Language.NL Dutch (Netherlands) nl-NL Language.NL_BE Dutch (Belgium) nl-BE Language.EN English (US) en-US Language.EN_US English (US) en-US Language.EN_AU English (Australia) en-AU Language.EN_GB English (UK) en-GB Language.EN_NZ English (New Zealand) en-NZ Language.EN_IN English (India) en-IN Language.ET Estonian et-EE Language.FI Finnish fi-FI Language.FR French (France) fr-FR Language.FR_CA French (Canada) fr-CA Language.DE German (Germany) de-DE Language.DE_CH German (Switzerland) de-CH Language.EL Greek el-GR Language.HI Hindi hi-IN Language.HU Hungarian hu-HU Language.ID Indonesian id-ID Language.IT Italian it-IT Language.JA Japanese ja-JP Language.KO Korean ko-KR Language.LV Latvian lv-LV Language.LT Lithuanian lt-LT Language.MS Malay ms-MY Language.NO Norwegian nb-NO Language.PL Polish pl-PL Language.PT Portuguese (Portugal) pt-PT Language.PT_BR Portuguese (Brazil) pt-BR Language.RO Romanian ro-RO Language.RU Russian ru-RU Language.SK Slovak sk-SK Language.ES Spanish es-ES Language.SV Swedish sv-SE Language.TH Thai th-TH Language.TR Turkish tr-TR Language.UK Ukrainian uk-UA Language.VI Vietnamese vi-VN ​ Usage Examples ​ TTS Configuration Copy Ask AI # Configure TTS with specific language tts = AzureTTSService( api_key = "your-api-key" , region = "eastus" , params = AzureTTSService.InputParams( language = Language. FR_CA , # Canadian French voice = "fr-CA-SylvieNeural" ) ) ​ Regional Considerations Each language code includes both language and region (e.g., fr-FR for French in France) Some languages have multiple regional variants (e.g., English has US, UK, Australian, Indian, and New Zealand variants) Voice availability may vary by region and language Neural voices are available for most language/region combinations Some features (like custom pronunciation) may be limited to specific languages Note: Voice selection should match the specified language code for optimal results. Check Azure’s documentation for the latest list of available voices for each language/region combination. ​ SSML Support The service supports rich SSML customization: Copy Ask AI # Example with multiple SSML features params = AzureTTSService.InputParams( emphasis = "strong" , pitch = "+2st" , rate = "1.2" , style = "cheerful" , style_degree = "2" , volume = "loud" ) ​ Frame Flow ​ Metrics Support The service collects processing metrics: Time to First Byte (TTFB) Processing duration Character usage API calls ​ Notes SSML-based speech customization Chunked audio delivery Thread-safe processing Automatic error handling Manages Azure client lifecycle AWS Polly Cartesia On this page Overview Installation Configuration Constructor Parameters Input Parameters Supported Sample Rates Usage Example Methods Language Support Usage Examples TTS Configuration Regional Considerations SSML Support Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_azure_e30436f6.txt b/tts_azure_e30436f6.txt
new file mode 100644
index 0000000000000000000000000000000000000000..6b8e5c9561a0db97a449867101564565a020c1f9
--- /dev/null
+++ b/tts_azure_e30436f6.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/azure#ssml-support
+Title: Azure - Pipecat
+==================================================
+
+Azure - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech Azure Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview AzureTTSService provides high-quality text-to-speech synthesis using Azure’s Cognitive Services. It supports SSML for advanced voice control and multiple languages. ​ Installation To use AzureTTSService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[azure]" You’ll also need to set up the following environment variables: AZURE_API_KEY AZURE_REGION ​ Configuration ​ Constructor Parameters ​ api_key str required Azure Speech Service API key ​ region str required Azure region identifier ​ voice str default: "en-US-SaraNeural" Voice identifier ​ sample_rate int default: "None" Output audio sample rate in Hz ​ text_filter BaseTextFilter default: "None" Modifies text provided to the TTS. Learn more about the available filters. ​ Input Parameters Copy Ask AI class InputParams ( BaseModel ): emphasis: Optional[ str ] language: Optional[Language] = Language. EN_US pitch: Optional[ str ] rate: Optional[ str ] = "1.05" role: Optional[ str ] style: Optional[ str ] style_degree: Optional[ str ] volume: Optional[ str ] ​ Supported Sample Rates 8000 Hz: Raw8Khz16BitMonoPcm 16000 Hz: Raw16Khz16BitMonoPcm 22050 Hz: Raw22050Hz16BitMonoPcm 24000 Hz: Raw24Khz16BitMonoPcm 44100 Hz: Raw44100Hz16BitMonoPcm 48000 Hz: Raw48Khz16BitMonoPcm ​ Usage Example Copy Ask AI # Configure service tts = AzureTTSService( api_key = "your-api-key" , region = "eastus" , voice = "en-US-JennyNeural" , params = AzureTTSService.InputParams( language = Language. EN_US , rate = "1.1" , style = "cheerful" ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ Methods See the TTS base class methods for additional functionality. ​ Language Support Azure Speech Services support the following languages and regional variants: Language Code Description Service Code Language.BG Bulgarian bg-BG Language.CA Catalan ca-ES Language.ZH Chinese (Simplified) zh-CN Language.ZH_TW Chinese (Traditional) zh-TW Language.CS Czech cs-CZ Language.DA Danish da-DK Language.NL Dutch (Netherlands) nl-NL Language.NL_BE Dutch (Belgium) nl-BE Language.EN English (US) en-US Language.EN_US English (US) en-US Language.EN_AU English (Australia) en-AU Language.EN_GB English (UK) en-GB Language.EN_NZ English (New Zealand) en-NZ Language.EN_IN English (India) en-IN Language.ET Estonian et-EE Language.FI Finnish fi-FI Language.FR French (France) fr-FR Language.FR_CA French (Canada) fr-CA Language.DE German (Germany) de-DE Language.DE_CH German (Switzerland) de-CH Language.EL Greek el-GR Language.HI Hindi hi-IN Language.HU Hungarian hu-HU Language.ID Indonesian id-ID Language.IT Italian it-IT Language.JA Japanese ja-JP Language.KO Korean ko-KR Language.LV Latvian lv-LV Language.LT Lithuanian lt-LT Language.MS Malay ms-MY Language.NO Norwegian nb-NO Language.PL Polish pl-PL Language.PT Portuguese (Portugal) pt-PT Language.PT_BR Portuguese (Brazil) pt-BR Language.RO Romanian ro-RO Language.RU Russian ru-RU Language.SK Slovak sk-SK Language.ES Spanish es-ES Language.SV Swedish sv-SE Language.TH Thai th-TH Language.TR Turkish tr-TR Language.UK Ukrainian uk-UA Language.VI Vietnamese vi-VN ​ Usage Examples ​ TTS Configuration Copy Ask AI # Configure TTS with specific language tts = AzureTTSService( api_key = "your-api-key" , region = "eastus" , params = AzureTTSService.InputParams( language = Language. FR_CA , # Canadian French voice = "fr-CA-SylvieNeural" ) ) ​ Regional Considerations Each language code includes both language and region (e.g., fr-FR for French in France) Some languages have multiple regional variants (e.g., English has US, UK, Australian, Indian, and New Zealand variants) Voice availability may vary by region and language Neural voices are available for most language/region combinations Some features (like custom pronunciation) may be limited to specific languages Note: Voice selection should match the specified language code for optimal results. Check Azure’s documentation for the latest list of available voices for each language/region combination. ​ SSML Support The service supports rich SSML customization: Copy Ask AI # Example with multiple SSML features params = AzureTTSService.InputParams( emphasis = "strong" , pitch = "+2st" , rate = "1.2" , style = "cheerful" , style_degree = "2" , volume = "loud" ) ​ Frame Flow ​ Metrics Support The service collects processing metrics: Time to First Byte (TTFB) Processing duration Character usage API calls ​ Notes SSML-based speech customization Chunked audio delivery Thread-safe processing Automatic error handling Manages Azure client lifecycle AWS Polly Cartesia On this page Overview Installation Configuration Constructor Parameters Input Parameters Supported Sample Rates Usage Example Methods Language Support Usage Examples TTS Configuration Regional Considerations SSML Support Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_cartesia_1c68b735.txt b/tts_cartesia_1c68b735.txt
new file mode 100644
index 0000000000000000000000000000000000000000..9cf682e9893174f456993cd15ce437d131ee85e5
--- /dev/null
+++ b/tts_cartesia_1c68b735.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/cartesia#language-support
+Title: Cartesia - Pipecat
+==================================================
+
+Cartesia - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech Cartesia Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview Cartesia provides two TTS service implementations: CartesiaTTSService (WebSocket-based with streaming and word timestamps) and CartesiaHttpTTSService (HTTP-based for simpler synthesis). The WebSocket service is recommended for real-time applications. API Reference Complete API documentation and method details Cartesia Docs Official Cartesia documentation and features Example Code Working example with interruption handling ​ Installation To use Cartesia services, install the required dependencies: Copy Ask AI pip install "pipecat-ai[cartesia]" You’ll also need to set up your Cartesia API key as an environment variable: CARTESIA_API_KEY . Get your API key by signing up at Cartesia . ​ Frames ​ Input TextFrame - Text content to synthesize into speech TTSSpeakFrame - Text that the TTS service should speak TTSUpdateSettingsFrame - Runtime configuration updates (e.g., voice) LLMFullResponseStartFrame / LLMFullResponseEndFrame - LLM response boundaries ​ Output TTSStartedFrame - Signals start of synthesis TTSAudioRawFrame - Generated audio data chunks TTSStoppedFrame - Signals completion of synthesis ErrorFrame - Connection or processing errors ​ Service Comparison Feature CartesiaTTSService (WebSocket) CartesiaHttpTTSService (HTTP) Streaming ✅ Real-time chunks ❌ Single audio block Word Timestamps ✅ Precise timing ❌ Not available Interruption ✅ Advanced handling ⚠️ Basic support Latency 🚀 Low 📈 Higher Best For Interactive apps Batch processing ​ Language Support Supports multiple languages through the Language enum: Language Code Description Service Code Language.DE German de Language.EN English en Language.ES Spanish es Language.FR French fr Language.HI Hindi hi Language.IT Italian it Language.JA Japanese ja Language.KO Korean ko Language.NL Dutch nl Language.PL Polish pl Language.PT Portuguese pt Language.RU Russian ru Language.SV Swedish sv Language.TR Turkish tr Language.ZH Chinese (Mandarin) zh ​ Usage Example ​ WebSocket Service (Recommended) Copy Ask AI from pipecat.services.cartesia.tts import CartesiaTTSService from pipecat.transcriptions.language import Language import os # Configure WebSocket service tts = CartesiaTTSService( api_key = os.getenv( "CARTESIA_API_KEY" ), voice_id = "your-voice-id" , model = "sonic-2" , params = CartesiaTTSService.InputParams( language = Language. EN , speed = "normal" ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, tts, # Word timestamps enable precise context updates transport.output() ]) ​ HTTP Service Copy Ask AI # For simpler, non-streaming use cases http_tts = CartesiaHttpTTSService( api_key = os.getenv( "CARTESIA_API_KEY" ), voice_id = "your-voice-id" , model = "sonic-2" , params = CartesiaHttpTTSService.InputParams( language = Language. EN ) ) ​ Metrics Both services provide: Time to First Byte (TTFB) - Latency from text input to first audio Processing Duration - Total synthesis time Usage Metrics - Character count and synthesis statistics ​ Additional Notes WebSocket Recommended : Use CartesiaTTSService for low-latency streaming and accurate context updates with word timestamps Connection Management : WebSocket lifecycle is handled automatically with reconnection support Sample Rate : Set globally in PipelineParams rather than per-service for consistency Azure Deepgram On this page Overview Installation Frames Input Output Service Comparison Language Support Usage Example WebSocket Service (Recommended) HTTP Service Metrics Additional Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_cartesia_3a6bfe79.txt b/tts_cartesia_3a6bfe79.txt
new file mode 100644
index 0000000000000000000000000000000000000000..54e23e72ff76f738203c013c619a1bfec9a4fbc3
--- /dev/null
+++ b/tts_cartesia_3a6bfe79.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/cartesia#input
+Title: Cartesia - Pipecat
+==================================================
+
+Cartesia - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech Cartesia Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview Cartesia provides two TTS service implementations: CartesiaTTSService (WebSocket-based with streaming and word timestamps) and CartesiaHttpTTSService (HTTP-based for simpler synthesis). The WebSocket service is recommended for real-time applications. API Reference Complete API documentation and method details Cartesia Docs Official Cartesia documentation and features Example Code Working example with interruption handling ​ Installation To use Cartesia services, install the required dependencies: Copy Ask AI pip install "pipecat-ai[cartesia]" You’ll also need to set up your Cartesia API key as an environment variable: CARTESIA_API_KEY . Get your API key by signing up at Cartesia . ​ Frames ​ Input TextFrame - Text content to synthesize into speech TTSSpeakFrame - Text that the TTS service should speak TTSUpdateSettingsFrame - Runtime configuration updates (e.g., voice) LLMFullResponseStartFrame / LLMFullResponseEndFrame - LLM response boundaries ​ Output TTSStartedFrame - Signals start of synthesis TTSAudioRawFrame - Generated audio data chunks TTSStoppedFrame - Signals completion of synthesis ErrorFrame - Connection or processing errors ​ Service Comparison Feature CartesiaTTSService (WebSocket) CartesiaHttpTTSService (HTTP) Streaming ✅ Real-time chunks ❌ Single audio block Word Timestamps ✅ Precise timing ❌ Not available Interruption ✅ Advanced handling ⚠️ Basic support Latency 🚀 Low 📈 Higher Best For Interactive apps Batch processing ​ Language Support Supports multiple languages through the Language enum: Language Code Description Service Code Language.DE German de Language.EN English en Language.ES Spanish es Language.FR French fr Language.HI Hindi hi Language.IT Italian it Language.JA Japanese ja Language.KO Korean ko Language.NL Dutch nl Language.PL Polish pl Language.PT Portuguese pt Language.RU Russian ru Language.SV Swedish sv Language.TR Turkish tr Language.ZH Chinese (Mandarin) zh ​ Usage Example ​ WebSocket Service (Recommended) Copy Ask AI from pipecat.services.cartesia.tts import CartesiaTTSService from pipecat.transcriptions.language import Language import os # Configure WebSocket service tts = CartesiaTTSService( api_key = os.getenv( "CARTESIA_API_KEY" ), voice_id = "your-voice-id" , model = "sonic-2" , params = CartesiaTTSService.InputParams( language = Language. EN , speed = "normal" ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, tts, # Word timestamps enable precise context updates transport.output() ]) ​ HTTP Service Copy Ask AI # For simpler, non-streaming use cases http_tts = CartesiaHttpTTSService( api_key = os.getenv( "CARTESIA_API_KEY" ), voice_id = "your-voice-id" , model = "sonic-2" , params = CartesiaHttpTTSService.InputParams( language = Language. EN ) ) ​ Metrics Both services provide: Time to First Byte (TTFB) - Latency from text input to first audio Processing Duration - Total synthesis time Usage Metrics - Character count and synthesis statistics ​ Additional Notes WebSocket Recommended : Use CartesiaTTSService for low-latency streaming and accurate context updates with word timestamps Connection Management : WebSocket lifecycle is handled automatically with reconnection support Sample Rate : Set globally in PipelineParams rather than per-service for consistency Azure Deepgram On this page Overview Installation Frames Input Output Service Comparison Language Support Usage Example WebSocket Service (Recommended) HTTP Service Metrics Additional Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_cartesia_3fb03c4a.txt b/tts_cartesia_3fb03c4a.txt
new file mode 100644
index 0000000000000000000000000000000000000000..57a26c40ffec8bfa77b7604a97143fefae81b9bd
--- /dev/null
+++ b/tts_cartesia_3fb03c4a.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/cartesia#output
+Title: Cartesia - Pipecat
+==================================================
+
+Cartesia - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech Cartesia Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview Cartesia provides two TTS service implementations: CartesiaTTSService (WebSocket-based with streaming and word timestamps) and CartesiaHttpTTSService (HTTP-based for simpler synthesis). The WebSocket service is recommended for real-time applications. API Reference Complete API documentation and method details Cartesia Docs Official Cartesia documentation and features Example Code Working example with interruption handling ​ Installation To use Cartesia services, install the required dependencies: Copy Ask AI pip install "pipecat-ai[cartesia]" You’ll also need to set up your Cartesia API key as an environment variable: CARTESIA_API_KEY . Get your API key by signing up at Cartesia . ​ Frames ​ Input TextFrame - Text content to synthesize into speech TTSSpeakFrame - Text that the TTS service should speak TTSUpdateSettingsFrame - Runtime configuration updates (e.g., voice) LLMFullResponseStartFrame / LLMFullResponseEndFrame - LLM response boundaries ​ Output TTSStartedFrame - Signals start of synthesis TTSAudioRawFrame - Generated audio data chunks TTSStoppedFrame - Signals completion of synthesis ErrorFrame - Connection or processing errors ​ Service Comparison Feature CartesiaTTSService (WebSocket) CartesiaHttpTTSService (HTTP) Streaming ✅ Real-time chunks ❌ Single audio block Word Timestamps ✅ Precise timing ❌ Not available Interruption ✅ Advanced handling ⚠️ Basic support Latency 🚀 Low 📈 Higher Best For Interactive apps Batch processing ​ Language Support Supports multiple languages through the Language enum: Language Code Description Service Code Language.DE German de Language.EN English en Language.ES Spanish es Language.FR French fr Language.HI Hindi hi Language.IT Italian it Language.JA Japanese ja Language.KO Korean ko Language.NL Dutch nl Language.PL Polish pl Language.PT Portuguese pt Language.RU Russian ru Language.SV Swedish sv Language.TR Turkish tr Language.ZH Chinese (Mandarin) zh ​ Usage Example ​ WebSocket Service (Recommended) Copy Ask AI from pipecat.services.cartesia.tts import CartesiaTTSService from pipecat.transcriptions.language import Language import os # Configure WebSocket service tts = CartesiaTTSService( api_key = os.getenv( "CARTESIA_API_KEY" ), voice_id = "your-voice-id" , model = "sonic-2" , params = CartesiaTTSService.InputParams( language = Language. EN , speed = "normal" ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, tts, # Word timestamps enable precise context updates transport.output() ]) ​ HTTP Service Copy Ask AI # For simpler, non-streaming use cases http_tts = CartesiaHttpTTSService( api_key = os.getenv( "CARTESIA_API_KEY" ), voice_id = "your-voice-id" , model = "sonic-2" , params = CartesiaHttpTTSService.InputParams( language = Language. EN ) ) ​ Metrics Both services provide: Time to First Byte (TTFB) - Latency from text input to first audio Processing Duration - Total synthesis time Usage Metrics - Character count and synthesis statistics ​ Additional Notes WebSocket Recommended : Use CartesiaTTSService for low-latency streaming and accurate context updates with word timestamps Connection Management : WebSocket lifecycle is handled automatically with reconnection support Sample Rate : Set globally in PipelineParams rather than per-service for consistency Azure Deepgram On this page Overview Installation Frames Input Output Service Comparison Language Support Usage Example WebSocket Service (Recommended) HTTP Service Metrics Additional Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_cartesia_4f26a783.txt b/tts_cartesia_4f26a783.txt
new file mode 100644
index 0000000000000000000000000000000000000000..ee72a0e48a0634636c431cc9651100d84b8a70bf
--- /dev/null
+++ b/tts_cartesia_4f26a783.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/cartesia#service-comparison
+Title: Cartesia - Pipecat
+==================================================
+
+Cartesia - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech Cartesia Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview Cartesia provides two TTS service implementations: CartesiaTTSService (WebSocket-based with streaming and word timestamps) and CartesiaHttpTTSService (HTTP-based for simpler synthesis). The WebSocket service is recommended for real-time applications. API Reference Complete API documentation and method details Cartesia Docs Official Cartesia documentation and features Example Code Working example with interruption handling ​ Installation To use Cartesia services, install the required dependencies: Copy Ask AI pip install "pipecat-ai[cartesia]" You’ll also need to set up your Cartesia API key as an environment variable: CARTESIA_API_KEY . Get your API key by signing up at Cartesia . ​ Frames ​ Input TextFrame - Text content to synthesize into speech TTSSpeakFrame - Text that the TTS service should speak TTSUpdateSettingsFrame - Runtime configuration updates (e.g., voice) LLMFullResponseStartFrame / LLMFullResponseEndFrame - LLM response boundaries ​ Output TTSStartedFrame - Signals start of synthesis TTSAudioRawFrame - Generated audio data chunks TTSStoppedFrame - Signals completion of synthesis ErrorFrame - Connection or processing errors ​ Service Comparison Feature CartesiaTTSService (WebSocket) CartesiaHttpTTSService (HTTP) Streaming ✅ Real-time chunks ❌ Single audio block Word Timestamps ✅ Precise timing ❌ Not available Interruption ✅ Advanced handling ⚠️ Basic support Latency 🚀 Low 📈 Higher Best For Interactive apps Batch processing ​ Language Support Supports multiple languages through the Language enum: Language Code Description Service Code Language.DE German de Language.EN English en Language.ES Spanish es Language.FR French fr Language.HI Hindi hi Language.IT Italian it Language.JA Japanese ja Language.KO Korean ko Language.NL Dutch nl Language.PL Polish pl Language.PT Portuguese pt Language.RU Russian ru Language.SV Swedish sv Language.TR Turkish tr Language.ZH Chinese (Mandarin) zh ​ Usage Example ​ WebSocket Service (Recommended) Copy Ask AI from pipecat.services.cartesia.tts import CartesiaTTSService from pipecat.transcriptions.language import Language import os # Configure WebSocket service tts = CartesiaTTSService( api_key = os.getenv( "CARTESIA_API_KEY" ), voice_id = "your-voice-id" , model = "sonic-2" , params = CartesiaTTSService.InputParams( language = Language. EN , speed = "normal" ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, tts, # Word timestamps enable precise context updates transport.output() ]) ​ HTTP Service Copy Ask AI # For simpler, non-streaming use cases http_tts = CartesiaHttpTTSService( api_key = os.getenv( "CARTESIA_API_KEY" ), voice_id = "your-voice-id" , model = "sonic-2" , params = CartesiaHttpTTSService.InputParams( language = Language. EN ) ) ​ Metrics Both services provide: Time to First Byte (TTFB) - Latency from text input to first audio Processing Duration - Total synthesis time Usage Metrics - Character count and synthesis statistics ​ Additional Notes WebSocket Recommended : Use CartesiaTTSService for low-latency streaming and accurate context updates with word timestamps Connection Management : WebSocket lifecycle is handled automatically with reconnection support Sample Rate : Set globally in PipelineParams rather than per-service for consistency Azure Deepgram On this page Overview Installation Frames Input Output Service Comparison Language Support Usage Example WebSocket Service (Recommended) HTTP Service Metrics Additional Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_cartesia_9a71eb42.txt b/tts_cartesia_9a71eb42.txt
new file mode 100644
index 0000000000000000000000000000000000000000..ed4d9379d64408ed763aa988f977cc1bb80688e5
--- /dev/null
+++ b/tts_cartesia_9a71eb42.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/cartesia#websocket-service-recommended
+Title: Cartesia - Pipecat
+==================================================
+
+Cartesia - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech Cartesia Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview Cartesia provides two TTS service implementations: CartesiaTTSService (WebSocket-based with streaming and word timestamps) and CartesiaHttpTTSService (HTTP-based for simpler synthesis). The WebSocket service is recommended for real-time applications. API Reference Complete API documentation and method details Cartesia Docs Official Cartesia documentation and features Example Code Working example with interruption handling ​ Installation To use Cartesia services, install the required dependencies: Copy Ask AI pip install "pipecat-ai[cartesia]" You’ll also need to set up your Cartesia API key as an environment variable: CARTESIA_API_KEY . Get your API key by signing up at Cartesia . ​ Frames ​ Input TextFrame - Text content to synthesize into speech TTSSpeakFrame - Text that the TTS service should speak TTSUpdateSettingsFrame - Runtime configuration updates (e.g., voice) LLMFullResponseStartFrame / LLMFullResponseEndFrame - LLM response boundaries ​ Output TTSStartedFrame - Signals start of synthesis TTSAudioRawFrame - Generated audio data chunks TTSStoppedFrame - Signals completion of synthesis ErrorFrame - Connection or processing errors ​ Service Comparison Feature CartesiaTTSService (WebSocket) CartesiaHttpTTSService (HTTP) Streaming ✅ Real-time chunks ❌ Single audio block Word Timestamps ✅ Precise timing ❌ Not available Interruption ✅ Advanced handling ⚠️ Basic support Latency 🚀 Low 📈 Higher Best For Interactive apps Batch processing ​ Language Support Supports multiple languages through the Language enum: Language Code Description Service Code Language.DE German de Language.EN English en Language.ES Spanish es Language.FR French fr Language.HI Hindi hi Language.IT Italian it Language.JA Japanese ja Language.KO Korean ko Language.NL Dutch nl Language.PL Polish pl Language.PT Portuguese pt Language.RU Russian ru Language.SV Swedish sv Language.TR Turkish tr Language.ZH Chinese (Mandarin) zh ​ Usage Example ​ WebSocket Service (Recommended) Copy Ask AI from pipecat.services.cartesia.tts import CartesiaTTSService from pipecat.transcriptions.language import Language import os # Configure WebSocket service tts = CartesiaTTSService( api_key = os.getenv( "CARTESIA_API_KEY" ), voice_id = "your-voice-id" , model = "sonic-2" , params = CartesiaTTSService.InputParams( language = Language. EN , speed = "normal" ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, tts, # Word timestamps enable precise context updates transport.output() ]) ​ HTTP Service Copy Ask AI # For simpler, non-streaming use cases http_tts = CartesiaHttpTTSService( api_key = os.getenv( "CARTESIA_API_KEY" ), voice_id = "your-voice-id" , model = "sonic-2" , params = CartesiaHttpTTSService.InputParams( language = Language. EN ) ) ​ Metrics Both services provide: Time to First Byte (TTFB) - Latency from text input to first audio Processing Duration - Total synthesis time Usage Metrics - Character count and synthesis statistics ​ Additional Notes WebSocket Recommended : Use CartesiaTTSService for low-latency streaming and accurate context updates with word timestamps Connection Management : WebSocket lifecycle is handled automatically with reconnection support Sample Rate : Set globally in PipelineParams rather than per-service for consistency Azure Deepgram On this page Overview Installation Frames Input Output Service Comparison Language Support Usage Example WebSocket Service (Recommended) HTTP Service Metrics Additional Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_cartesia_9bc5a949.txt b/tts_cartesia_9bc5a949.txt
new file mode 100644
index 0000000000000000000000000000000000000000..95e5bad136ee557383aa8bbe5a099c77c4378bf4
--- /dev/null
+++ b/tts_cartesia_9bc5a949.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/cartesia#metrics
+Title: Cartesia - Pipecat
+==================================================
+
+Cartesia - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech Cartesia Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview Cartesia provides two TTS service implementations: CartesiaTTSService (WebSocket-based with streaming and word timestamps) and CartesiaHttpTTSService (HTTP-based for simpler synthesis). The WebSocket service is recommended for real-time applications. API Reference Complete API documentation and method details Cartesia Docs Official Cartesia documentation and features Example Code Working example with interruption handling ​ Installation To use Cartesia services, install the required dependencies: Copy Ask AI pip install "pipecat-ai[cartesia]" You’ll also need to set up your Cartesia API key as an environment variable: CARTESIA_API_KEY . Get your API key by signing up at Cartesia . ​ Frames ​ Input TextFrame - Text content to synthesize into speech TTSSpeakFrame - Text that the TTS service should speak TTSUpdateSettingsFrame - Runtime configuration updates (e.g., voice) LLMFullResponseStartFrame / LLMFullResponseEndFrame - LLM response boundaries ​ Output TTSStartedFrame - Signals start of synthesis TTSAudioRawFrame - Generated audio data chunks TTSStoppedFrame - Signals completion of synthesis ErrorFrame - Connection or processing errors ​ Service Comparison Feature CartesiaTTSService (WebSocket) CartesiaHttpTTSService (HTTP) Streaming ✅ Real-time chunks ❌ Single audio block Word Timestamps ✅ Precise timing ❌ Not available Interruption ✅ Advanced handling ⚠️ Basic support Latency 🚀 Low 📈 Higher Best For Interactive apps Batch processing ​ Language Support Supports multiple languages through the Language enum: Language Code Description Service Code Language.DE German de Language.EN English en Language.ES Spanish es Language.FR French fr Language.HI Hindi hi Language.IT Italian it Language.JA Japanese ja Language.KO Korean ko Language.NL Dutch nl Language.PL Polish pl Language.PT Portuguese pt Language.RU Russian ru Language.SV Swedish sv Language.TR Turkish tr Language.ZH Chinese (Mandarin) zh ​ Usage Example ​ WebSocket Service (Recommended) Copy Ask AI from pipecat.services.cartesia.tts import CartesiaTTSService from pipecat.transcriptions.language import Language import os # Configure WebSocket service tts = CartesiaTTSService( api_key = os.getenv( "CARTESIA_API_KEY" ), voice_id = "your-voice-id" , model = "sonic-2" , params = CartesiaTTSService.InputParams( language = Language. EN , speed = "normal" ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, tts, # Word timestamps enable precise context updates transport.output() ]) ​ HTTP Service Copy Ask AI # For simpler, non-streaming use cases http_tts = CartesiaHttpTTSService( api_key = os.getenv( "CARTESIA_API_KEY" ), voice_id = "your-voice-id" , model = "sonic-2" , params = CartesiaHttpTTSService.InputParams( language = Language. EN ) ) ​ Metrics Both services provide: Time to First Byte (TTFB) - Latency from text input to first audio Processing Duration - Total synthesis time Usage Metrics - Character count and synthesis statistics ​ Additional Notes WebSocket Recommended : Use CartesiaTTSService for low-latency streaming and accurate context updates with word timestamps Connection Management : WebSocket lifecycle is handled automatically with reconnection support Sample Rate : Set globally in PipelineParams rather than per-service for consistency Azure Deepgram On this page Overview Installation Frames Input Output Service Comparison Language Support Usage Example WebSocket Service (Recommended) HTTP Service Metrics Additional Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_cartesia_a258f0cf.txt b/tts_cartesia_a258f0cf.txt
new file mode 100644
index 0000000000000000000000000000000000000000..3af652dab69e9c359c99d5e734aa0966bf870daa
--- /dev/null
+++ b/tts_cartesia_a258f0cf.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/cartesia#overview
+Title: Cartesia - Pipecat
+==================================================
+
+Cartesia - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech Cartesia Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview Cartesia provides two TTS service implementations: CartesiaTTSService (WebSocket-based with streaming and word timestamps) and CartesiaHttpTTSService (HTTP-based for simpler synthesis). The WebSocket service is recommended for real-time applications. API Reference Complete API documentation and method details Cartesia Docs Official Cartesia documentation and features Example Code Working example with interruption handling ​ Installation To use Cartesia services, install the required dependencies: Copy Ask AI pip install "pipecat-ai[cartesia]" You’ll also need to set up your Cartesia API key as an environment variable: CARTESIA_API_KEY . Get your API key by signing up at Cartesia . ​ Frames ​ Input TextFrame - Text content to synthesize into speech TTSSpeakFrame - Text that the TTS service should speak TTSUpdateSettingsFrame - Runtime configuration updates (e.g., voice) LLMFullResponseStartFrame / LLMFullResponseEndFrame - LLM response boundaries ​ Output TTSStartedFrame - Signals start of synthesis TTSAudioRawFrame - Generated audio data chunks TTSStoppedFrame - Signals completion of synthesis ErrorFrame - Connection or processing errors ​ Service Comparison Feature CartesiaTTSService (WebSocket) CartesiaHttpTTSService (HTTP) Streaming ✅ Real-time chunks ❌ Single audio block Word Timestamps ✅ Precise timing ❌ Not available Interruption ✅ Advanced handling ⚠️ Basic support Latency 🚀 Low 📈 Higher Best For Interactive apps Batch processing ​ Language Support Supports multiple languages through the Language enum: Language Code Description Service Code Language.DE German de Language.EN English en Language.ES Spanish es Language.FR French fr Language.HI Hindi hi Language.IT Italian it Language.JA Japanese ja Language.KO Korean ko Language.NL Dutch nl Language.PL Polish pl Language.PT Portuguese pt Language.RU Russian ru Language.SV Swedish sv Language.TR Turkish tr Language.ZH Chinese (Mandarin) zh ​ Usage Example ​ WebSocket Service (Recommended) Copy Ask AI from pipecat.services.cartesia.tts import CartesiaTTSService from pipecat.transcriptions.language import Language import os # Configure WebSocket service tts = CartesiaTTSService( api_key = os.getenv( "CARTESIA_API_KEY" ), voice_id = "your-voice-id" , model = "sonic-2" , params = CartesiaTTSService.InputParams( language = Language. EN , speed = "normal" ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, tts, # Word timestamps enable precise context updates transport.output() ]) ​ HTTP Service Copy Ask AI # For simpler, non-streaming use cases http_tts = CartesiaHttpTTSService( api_key = os.getenv( "CARTESIA_API_KEY" ), voice_id = "your-voice-id" , model = "sonic-2" , params = CartesiaHttpTTSService.InputParams( language = Language. EN ) ) ​ Metrics Both services provide: Time to First Byte (TTFB) - Latency from text input to first audio Processing Duration - Total synthesis time Usage Metrics - Character count and synthesis statistics ​ Additional Notes WebSocket Recommended : Use CartesiaTTSService for low-latency streaming and accurate context updates with word timestamps Connection Management : WebSocket lifecycle is handled automatically with reconnection support Sample Rate : Set globally in PipelineParams rather than per-service for consistency Azure Deepgram On this page Overview Installation Frames Input Output Service Comparison Language Support Usage Example WebSocket Service (Recommended) HTTP Service Metrics Additional Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_cartesia_ce94ef60.txt b/tts_cartesia_ce94ef60.txt
new file mode 100644
index 0000000000000000000000000000000000000000..ca58086c564ce1e71ae563b2b1b72cf6cdf385ac
--- /dev/null
+++ b/tts_cartesia_ce94ef60.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/cartesia#installation
+Title: Cartesia - Pipecat
+==================================================
+
+Cartesia - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech Cartesia Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview Cartesia provides two TTS service implementations: CartesiaTTSService (WebSocket-based with streaming and word timestamps) and CartesiaHttpTTSService (HTTP-based for simpler synthesis). The WebSocket service is recommended for real-time applications. API Reference Complete API documentation and method details Cartesia Docs Official Cartesia documentation and features Example Code Working example with interruption handling ​ Installation To use Cartesia services, install the required dependencies: Copy Ask AI pip install "pipecat-ai[cartesia]" You’ll also need to set up your Cartesia API key as an environment variable: CARTESIA_API_KEY . Get your API key by signing up at Cartesia . ​ Frames ​ Input TextFrame - Text content to synthesize into speech TTSSpeakFrame - Text that the TTS service should speak TTSUpdateSettingsFrame - Runtime configuration updates (e.g., voice) LLMFullResponseStartFrame / LLMFullResponseEndFrame - LLM response boundaries ​ Output TTSStartedFrame - Signals start of synthesis TTSAudioRawFrame - Generated audio data chunks TTSStoppedFrame - Signals completion of synthesis ErrorFrame - Connection or processing errors ​ Service Comparison Feature CartesiaTTSService (WebSocket) CartesiaHttpTTSService (HTTP) Streaming ✅ Real-time chunks ❌ Single audio block Word Timestamps ✅ Precise timing ❌ Not available Interruption ✅ Advanced handling ⚠️ Basic support Latency 🚀 Low 📈 Higher Best For Interactive apps Batch processing ​ Language Support Supports multiple languages through the Language enum: Language Code Description Service Code Language.DE German de Language.EN English en Language.ES Spanish es Language.FR French fr Language.HI Hindi hi Language.IT Italian it Language.JA Japanese ja Language.KO Korean ko Language.NL Dutch nl Language.PL Polish pl Language.PT Portuguese pt Language.RU Russian ru Language.SV Swedish sv Language.TR Turkish tr Language.ZH Chinese (Mandarin) zh ​ Usage Example ​ WebSocket Service (Recommended) Copy Ask AI from pipecat.services.cartesia.tts import CartesiaTTSService from pipecat.transcriptions.language import Language import os # Configure WebSocket service tts = CartesiaTTSService( api_key = os.getenv( "CARTESIA_API_KEY" ), voice_id = "your-voice-id" , model = "sonic-2" , params = CartesiaTTSService.InputParams( language = Language. EN , speed = "normal" ) ) # Use in pipeline pipeline = Pipeline([ transport.input(), stt, llm, tts, # Word timestamps enable precise context updates transport.output() ]) ​ HTTP Service Copy Ask AI # For simpler, non-streaming use cases http_tts = CartesiaHttpTTSService( api_key = os.getenv( "CARTESIA_API_KEY" ), voice_id = "your-voice-id" , model = "sonic-2" , params = CartesiaHttpTTSService.InputParams( language = Language. EN ) ) ​ Metrics Both services provide: Time to First Byte (TTFB) - Latency from text input to first audio Processing Duration - Total synthesis time Usage Metrics - Character count and synthesis statistics ​ Additional Notes WebSocket Recommended : Use CartesiaTTSService for low-latency streaming and accurate context updates with word timestamps Connection Management : WebSocket lifecycle is handled automatically with reconnection support Sample Rate : Set globally in PipelineParams rather than per-service for consistency Azure Deepgram On this page Overview Installation Frames Input Output Service Comparison Language Support Usage Example WebSocket Service (Recommended) HTTP Service Metrics Additional Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_deepgram_1cb60756.txt b/tts_deepgram_1cb60756.txt
new file mode 100644
index 0000000000000000000000000000000000000000..2cdff3b724f1bd9f7085820d1e6d21cd002fdd96
--- /dev/null
+++ b/tts_deepgram_1cb60756.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/deepgram#installation
+Title: Deepgram - Pipecat
+==================================================
+
+Deepgram - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech Deepgram Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview DeepgramTTSService converts text to speech using Deepgram’s Aura API. It supports various voices and audio configurations. ​ Installation To use DeepgramTTSService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[deepgram]" You’ll also need to set up your Deepgram API key as an environment variable: DEEPGRAM_API_KEY ​ Configuration ​ Constructor Parameters ​ api_key str required Your Deepgram API key ​ voice str default: "aura-helios-en" Voice identifier to use for synthesis ​ sample_rate int default: "None" Output audio sample rate in Hz ​ encoding str default: "linear16" Audio encoding format ​ text_filter BaseTextFilter default: "None" Modifies text provided to the TTS. Learn more about the available filters. ​ Input The service accepts text input through its TTS pipeline. ​ Output Frames ​ TTSStartedFrame Signals the start of audio generation. ​ TTSAudioRawFrame Contains generated audio data: ​ audio bytes Raw audio data chunk ​ sample_rate int Audio sample rate (24kHz default) ​ num_channels int Number of audio channels (1 for mono) ​ TTSStoppedFrame Signals the completion of audio generation. ​ Methods See the TTS base class methods for additional functionality. ​ Language Support Deepgram TTS supports the following languages and regional variants: Language Code Description Service Codes Language.EN English en ​ Usage Example Copy Ask AI from pipecat.services.deepgram.tts import DeepgramTTSService # Configure service tts = DeepgramTTSService( api_key = "your-api-key" , voice = "aura-helios-en" , sample_rate = 24000 ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ Frame Flow ​ Metrics Support The service supports metrics collection: Time to First Byte (TTFB) TTS usage metrics Processing duration ​ Audio Processing Streams audio in 8KB chunks Supports 16-bit PCM format Generates mono audio output Handles memory buffering ​ Notes Requires valid Deepgram API key Streams audio in chunks Supports various voices Provides metrics collection Handles memory efficiently Thread-safe processing Cartesia ElevenLabs On this page Overview Installation Configuration Constructor Parameters Input Output Frames TTSStartedFrame TTSAudioRawFrame TTSStoppedFrame Methods Language Support Usage Example Frame Flow Metrics Support Audio Processing Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_deepgram_45a08303.txt b/tts_deepgram_45a08303.txt
new file mode 100644
index 0000000000000000000000000000000000000000..630df9353fafd2701244ddfc4771753a780c55e2
--- /dev/null
+++ b/tts_deepgram_45a08303.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/deepgram
+Title: Deepgram - Pipecat
+==================================================
+
+Deepgram - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech Deepgram Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview DeepgramTTSService converts text to speech using Deepgram’s Aura API. It supports various voices and audio configurations. ​ Installation To use DeepgramTTSService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[deepgram]" You’ll also need to set up your Deepgram API key as an environment variable: DEEPGRAM_API_KEY ​ Configuration ​ Constructor Parameters ​ api_key str required Your Deepgram API key ​ voice str default: "aura-helios-en" Voice identifier to use for synthesis ​ sample_rate int default: "None" Output audio sample rate in Hz ​ encoding str default: "linear16" Audio encoding format ​ text_filter BaseTextFilter default: "None" Modifies text provided to the TTS. Learn more about the available filters. ​ Input The service accepts text input through its TTS pipeline. ​ Output Frames ​ TTSStartedFrame Signals the start of audio generation. ​ TTSAudioRawFrame Contains generated audio data: ​ audio bytes Raw audio data chunk ​ sample_rate int Audio sample rate (24kHz default) ​ num_channels int Number of audio channels (1 for mono) ​ TTSStoppedFrame Signals the completion of audio generation. ​ Methods See the TTS base class methods for additional functionality. ​ Language Support Deepgram TTS supports the following languages and regional variants: Language Code Description Service Codes Language.EN English en ​ Usage Example Copy Ask AI from pipecat.services.deepgram.tts import DeepgramTTSService # Configure service tts = DeepgramTTSService( api_key = "your-api-key" , voice = "aura-helios-en" , sample_rate = 24000 ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ Frame Flow ​ Metrics Support The service supports metrics collection: Time to First Byte (TTFB) TTS usage metrics Processing duration ​ Audio Processing Streams audio in 8KB chunks Supports 16-bit PCM format Generates mono audio output Handles memory buffering ​ Notes Requires valid Deepgram API key Streams audio in chunks Supports various voices Provides metrics collection Handles memory efficiently Thread-safe processing Cartesia ElevenLabs On this page Overview Installation Configuration Constructor Parameters Input Output Frames TTSStartedFrame TTSAudioRawFrame TTSStoppedFrame Methods Language Support Usage Example Frame Flow Metrics Support Audio Processing Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_deepgram_95cc2499.txt b/tts_deepgram_95cc2499.txt
new file mode 100644
index 0000000000000000000000000000000000000000..44b6a5390dfb1a640a05b26b291ff342b26f8ec1
--- /dev/null
+++ b/tts_deepgram_95cc2499.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/deepgram#param-audio
+Title: Deepgram - Pipecat
+==================================================
+
+Deepgram - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech Deepgram Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview DeepgramTTSService converts text to speech using Deepgram’s Aura API. It supports various voices and audio configurations. ​ Installation To use DeepgramTTSService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[deepgram]" You’ll also need to set up your Deepgram API key as an environment variable: DEEPGRAM_API_KEY ​ Configuration ​ Constructor Parameters ​ api_key str required Your Deepgram API key ​ voice str default: "aura-helios-en" Voice identifier to use for synthesis ​ sample_rate int default: "None" Output audio sample rate in Hz ​ encoding str default: "linear16" Audio encoding format ​ text_filter BaseTextFilter default: "None" Modifies text provided to the TTS. Learn more about the available filters. ​ Input The service accepts text input through its TTS pipeline. ​ Output Frames ​ TTSStartedFrame Signals the start of audio generation. ​ TTSAudioRawFrame Contains generated audio data: ​ audio bytes Raw audio data chunk ​ sample_rate int Audio sample rate (24kHz default) ​ num_channels int Number of audio channels (1 for mono) ​ TTSStoppedFrame Signals the completion of audio generation. ​ Methods See the TTS base class methods for additional functionality. ​ Language Support Deepgram TTS supports the following languages and regional variants: Language Code Description Service Codes Language.EN English en ​ Usage Example Copy Ask AI from pipecat.services.deepgram.tts import DeepgramTTSService # Configure service tts = DeepgramTTSService( api_key = "your-api-key" , voice = "aura-helios-en" , sample_rate = 24000 ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ Frame Flow ​ Metrics Support The service supports metrics collection: Time to First Byte (TTFB) TTS usage metrics Processing duration ​ Audio Processing Streams audio in 8KB chunks Supports 16-bit PCM format Generates mono audio output Handles memory buffering ​ Notes Requires valid Deepgram API key Streams audio in chunks Supports various voices Provides metrics collection Handles memory efficiently Thread-safe processing Cartesia ElevenLabs On this page Overview Installation Configuration Constructor Parameters Input Output Frames TTSStartedFrame TTSAudioRawFrame TTSStoppedFrame Methods Language Support Usage Example Frame Flow Metrics Support Audio Processing Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_deepgram_c84d7190.txt b/tts_deepgram_c84d7190.txt
new file mode 100644
index 0000000000000000000000000000000000000000..11b66a0bc4871ec18e2092671b7315536e7e0320
--- /dev/null
+++ b/tts_deepgram_c84d7190.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/deepgram#output-frames
+Title: Deepgram - Pipecat
+==================================================
+
+Deepgram - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech Deepgram Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview DeepgramTTSService converts text to speech using Deepgram’s Aura API. It supports various voices and audio configurations. ​ Installation To use DeepgramTTSService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[deepgram]" You’ll also need to set up your Deepgram API key as an environment variable: DEEPGRAM_API_KEY ​ Configuration ​ Constructor Parameters ​ api_key str required Your Deepgram API key ​ voice str default: "aura-helios-en" Voice identifier to use for synthesis ​ sample_rate int default: "None" Output audio sample rate in Hz ​ encoding str default: "linear16" Audio encoding format ​ text_filter BaseTextFilter default: "None" Modifies text provided to the TTS. Learn more about the available filters. ​ Input The service accepts text input through its TTS pipeline. ​ Output Frames ​ TTSStartedFrame Signals the start of audio generation. ​ TTSAudioRawFrame Contains generated audio data: ​ audio bytes Raw audio data chunk ​ sample_rate int Audio sample rate (24kHz default) ​ num_channels int Number of audio channels (1 for mono) ​ TTSStoppedFrame Signals the completion of audio generation. ​ Methods See the TTS base class methods for additional functionality. ​ Language Support Deepgram TTS supports the following languages and regional variants: Language Code Description Service Codes Language.EN English en ​ Usage Example Copy Ask AI from pipecat.services.deepgram.tts import DeepgramTTSService # Configure service tts = DeepgramTTSService( api_key = "your-api-key" , voice = "aura-helios-en" , sample_rate = 24000 ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ Frame Flow ​ Metrics Support The service supports metrics collection: Time to First Byte (TTFB) TTS usage metrics Processing duration ​ Audio Processing Streams audio in 8KB chunks Supports 16-bit PCM format Generates mono audio output Handles memory buffering ​ Notes Requires valid Deepgram API key Streams audio in chunks Supports various voices Provides metrics collection Handles memory efficiently Thread-safe processing Cartesia ElevenLabs On this page Overview Installation Configuration Constructor Parameters Input Output Frames TTSStartedFrame TTSAudioRawFrame TTSStoppedFrame Methods Language Support Usage Example Frame Flow Metrics Support Audio Processing Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_deepgram_f3c7aad8.txt b/tts_deepgram_f3c7aad8.txt
new file mode 100644
index 0000000000000000000000000000000000000000..1c7b234e8180af7d3ac359f0a3e0db70f2204b8f
--- /dev/null
+++ b/tts_deepgram_f3c7aad8.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/deepgram#ttsaudiorawframe
+Title: Deepgram - Pipecat
+==================================================
+
+Deepgram - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech Deepgram Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview DeepgramTTSService converts text to speech using Deepgram’s Aura API. It supports various voices and audio configurations. ​ Installation To use DeepgramTTSService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[deepgram]" You’ll also need to set up your Deepgram API key as an environment variable: DEEPGRAM_API_KEY ​ Configuration ​ Constructor Parameters ​ api_key str required Your Deepgram API key ​ voice str default: "aura-helios-en" Voice identifier to use for synthesis ​ sample_rate int default: "None" Output audio sample rate in Hz ​ encoding str default: "linear16" Audio encoding format ​ text_filter BaseTextFilter default: "None" Modifies text provided to the TTS. Learn more about the available filters. ​ Input The service accepts text input through its TTS pipeline. ​ Output Frames ​ TTSStartedFrame Signals the start of audio generation. ​ TTSAudioRawFrame Contains generated audio data: ​ audio bytes Raw audio data chunk ​ sample_rate int Audio sample rate (24kHz default) ​ num_channels int Number of audio channels (1 for mono) ​ TTSStoppedFrame Signals the completion of audio generation. ​ Methods See the TTS base class methods for additional functionality. ​ Language Support Deepgram TTS supports the following languages and regional variants: Language Code Description Service Codes Language.EN English en ​ Usage Example Copy Ask AI from pipecat.services.deepgram.tts import DeepgramTTSService # Configure service tts = DeepgramTTSService( api_key = "your-api-key" , voice = "aura-helios-en" , sample_rate = 24000 ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ Frame Flow ​ Metrics Support The service supports metrics collection: Time to First Byte (TTFB) TTS usage metrics Processing duration ​ Audio Processing Streams audio in 8KB chunks Supports 16-bit PCM format Generates mono audio output Handles memory buffering ​ Notes Requires valid Deepgram API key Streams audio in chunks Supports various voices Provides metrics collection Handles memory efficiently Thread-safe processing Cartesia ElevenLabs On this page Overview Installation Configuration Constructor Parameters Input Output Frames TTSStartedFrame TTSAudioRawFrame TTSStoppedFrame Methods Language Support Usage Example Frame Flow Metrics Support Audio Processing Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_elevenlabs_02a52beb.txt b/tts_elevenlabs_02a52beb.txt
new file mode 100644
index 0000000000000000000000000000000000000000..d17c1f863157edcd4519b1c9bc18d2f6f2b56852
--- /dev/null
+++ b/tts_elevenlabs_02a52beb.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/elevenlabs#param-text-filter
+Title: ElevenLabs - Pipecat
+==================================================
+
+ElevenLabs - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech ElevenLabs Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview ElevenLabs TTS provides high-quality text-to-speech synthesis through two service implementations: ElevenLabsTTSService : WebSocket-based implementation with word-level timing and interruption support ElevenLabsHttpTTSService : HTTP-based implementation for simpler use cases ​ Installation To use ElevenLabsTTSService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[elevenlabs]" You’ll also need to set up your ElevenLabs API key as an environment variable: ELEVENLABS_API_KEY . You can obtain a ElevenLabs API key by signing up at ElevenLabs . ​ ElevenLabsTTSService (WebSocket) ​ Configuration ​ api_key str required ElevenLabs API key ​ voice_id str required Voice identifier ​ model str default: "eleven_flash_v2_5" Model identifier ​ url str default: "wss://api.elevenlabs.io" API endpoint URL ​ sample_rate int default: "None" Output audio sample rate in Hz ​ params InputParams default: "InputParams()" Additional configuration parameters ​ text_filter BaseTextFilter default: "None" Modifies text provided to the TTS. Learn more about the available filters. ​ InputParams ​ language Language default: "None" The language of the text to be synthesized ​ optimize_streaming_latency str default: "None" Optimization level for streaming latency ​ stability float default: "None" Defines the stability for voice settings ​ similarity_boost float default: "None" Defines the similarity boost for voice settings ​ style float default: "None" Defines the style for voice settings. Available on V2+ models ​ use_speaker_boost bool default: "None" Defines whether to use speaker boost for voice settings. Available on V2+ models ​ speed float default: "None" Speech rate multiplier. Higher values increase speech speed ​ auto_mode bool default: "True" This parameter focuses on reducing the latency by disabling the chunk schedule and buffers. Recommended when sending full sentences or phrases ​ ElevenLabsHttpTTSService (HTTP) ​ Configuration ​ api_key str required ElevenLabs API key ​ voice_id str required Voice identifier ​ aiohttp_session aiohttp.ClientSession required aiohttp ClientSession for HTTP requests ​ model str default: "eleven_flash_v2_5" Model identifier ​ base_url str default: "https://api.elevenlabs.io" API base URL ​ sample_rate int default: "None" Output audio sample rate in Hz ​ params InputParams default: "InputParams()" Additional configuration parameters (similar to WebSocket implementation) ​ Output Frames ​ TTSStartedFrame Signals the start of audio generation. ​ TTSAudioRawFrame Contains generated audio data: ​ audio bytes Raw audio data chunk ​ sample_rate int Audio sample rate ​ num_channels int Number of audio channels (1 for mono) ​ TTSStoppedFrame Signals the completion of audio generation. ​ ErrorFrame (HTTP implementation) Sent when an error occurs during HTTP TTS generation: ​ error str Error message describing what went wrong ​ Usage Examples ​ Basic Usage Copy Ask AI # Configure service tts = ElevenLabsTTSService( api_key = "your-api-key" , voice_id = "voice-id" , sample_rate = 24000 , params = ElevenLabsTTSService.InputParams( language = Language. EN ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output() ]) ​ With Voice Settings Copy Ask AI # Configure with voice customization tts = ElevenLabsTTSService( api_key = "your-api-key" , voice_id = "voice-id" , params = ElevenLabsTTSService.InputParams( stability = 0.7 , similarity_boost = 0.8 , style = 0.5 , use_speaker_boost = True ) ) ​ Methods See the TTS base class methods for additional functionality. ​ Language Support ElevenLabs supports the following languages and their variants: Language Code Description Service Code Language.AR Arabic ar Language.BG Bulgarian bg Language.CS Czech cs Language.DA Danish da Language.DE German de Language.EL Greek el Language.EN English en Language.ES Spanish es Language.FI Finnish fi Language.FIL Filipino fil Language.FR French fr Language.HI Hindi hi Language.HR Croatian hr Language.HU Hungarian hu Language.ID Indonesian id Language.IT Italian it Language.JA Japanese ja Language.KO Korean ko Language.MS Malay ms Language.NL Dutch nl Language.NO Norwegian no Language.PL Polish pl Language.PT Portuguese pt Language.RO Romanian ro Language.RU Russian ru Language.SK Slovak sk Language.SV Swedish sv Language.TA Tamil ta Language.TR Turkish tr Language.UK Ukrainian uk Language.VI Vietnamese vi Language.ZH Chinese zh Note: Language support may vary based on the selected model. See the ElevenLabs docs for more details. ​ Usage Example Copy Ask AI # Configure service with specific language service = ElevenLabsTTSService( api_key = "your-api-key" , voice_id = "voice-id" , params = ElevenLabsTTSService.InputParams( language = Language. FR # French ) ) ​ Frame Flow ​ Notes WebSocket implementation includes a 10-second keepalive mechanism Sample rate must be one of: 16000, 22050, 24000, or 44100 Hz Voice settings require both stability and similarity_boost to be set The language parameter only works with multilingual models WebSocket implementation pauses frame processing during speech generation HTTP implementation requires an external aiohttp ClientSession Deepgram Fish Audio On this page Overview Installation ElevenLabsTTSService (WebSocket) Configuration InputParams ElevenLabsHttpTTSService (HTTP) Configuration Output Frames TTSStartedFrame TTSAudioRawFrame TTSStoppedFrame ErrorFrame (HTTP implementation) Usage Examples Basic Usage With Voice Settings Methods Language Support Usage Example Frame Flow Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_elevenlabs_03b56ed2.txt b/tts_elevenlabs_03b56ed2.txt
new file mode 100644
index 0000000000000000000000000000000000000000..5dd9ded857219888fdb7c01879da0323f47c84d5
--- /dev/null
+++ b/tts_elevenlabs_03b56ed2.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/elevenlabs#elevenlabsttsservice-websocket
+Title: ElevenLabs - Pipecat
+==================================================
+
+ElevenLabs - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech ElevenLabs Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview ElevenLabs TTS provides high-quality text-to-speech synthesis through two service implementations: ElevenLabsTTSService : WebSocket-based implementation with word-level timing and interruption support ElevenLabsHttpTTSService : HTTP-based implementation for simpler use cases ​ Installation To use ElevenLabsTTSService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[elevenlabs]" You’ll also need to set up your ElevenLabs API key as an environment variable: ELEVENLABS_API_KEY . You can obtain a ElevenLabs API key by signing up at ElevenLabs . ​ ElevenLabsTTSService (WebSocket) ​ Configuration ​ api_key str required ElevenLabs API key ​ voice_id str required Voice identifier ​ model str default: "eleven_flash_v2_5" Model identifier ​ url str default: "wss://api.elevenlabs.io" API endpoint URL ​ sample_rate int default: "None" Output audio sample rate in Hz ​ params InputParams default: "InputParams()" Additional configuration parameters ​ text_filter BaseTextFilter default: "None" Modifies text provided to the TTS. Learn more about the available filters. ​ InputParams ​ language Language default: "None" The language of the text to be synthesized ​ optimize_streaming_latency str default: "None" Optimization level for streaming latency ​ stability float default: "None" Defines the stability for voice settings ​ similarity_boost float default: "None" Defines the similarity boost for voice settings ​ style float default: "None" Defines the style for voice settings. Available on V2+ models ​ use_speaker_boost bool default: "None" Defines whether to use speaker boost for voice settings. Available on V2+ models ​ speed float default: "None" Speech rate multiplier. Higher values increase speech speed ​ auto_mode bool default: "True" This parameter focuses on reducing the latency by disabling the chunk schedule and buffers. Recommended when sending full sentences or phrases ​ ElevenLabsHttpTTSService (HTTP) ​ Configuration ​ api_key str required ElevenLabs API key ​ voice_id str required Voice identifier ​ aiohttp_session aiohttp.ClientSession required aiohttp ClientSession for HTTP requests ​ model str default: "eleven_flash_v2_5" Model identifier ​ base_url str default: "https://api.elevenlabs.io" API base URL ​ sample_rate int default: "None" Output audio sample rate in Hz ​ params InputParams default: "InputParams()" Additional configuration parameters (similar to WebSocket implementation) ​ Output Frames ​ TTSStartedFrame Signals the start of audio generation. ​ TTSAudioRawFrame Contains generated audio data: ​ audio bytes Raw audio data chunk ​ sample_rate int Audio sample rate ​ num_channels int Number of audio channels (1 for mono) ​ TTSStoppedFrame Signals the completion of audio generation. ​ ErrorFrame (HTTP implementation) Sent when an error occurs during HTTP TTS generation: ​ error str Error message describing what went wrong ​ Usage Examples ​ Basic Usage Copy Ask AI # Configure service tts = ElevenLabsTTSService( api_key = "your-api-key" , voice_id = "voice-id" , sample_rate = 24000 , params = ElevenLabsTTSService.InputParams( language = Language. EN ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output() ]) ​ With Voice Settings Copy Ask AI # Configure with voice customization tts = ElevenLabsTTSService( api_key = "your-api-key" , voice_id = "voice-id" , params = ElevenLabsTTSService.InputParams( stability = 0.7 , similarity_boost = 0.8 , style = 0.5 , use_speaker_boost = True ) ) ​ Methods See the TTS base class methods for additional functionality. ​ Language Support ElevenLabs supports the following languages and their variants: Language Code Description Service Code Language.AR Arabic ar Language.BG Bulgarian bg Language.CS Czech cs Language.DA Danish da Language.DE German de Language.EL Greek el Language.EN English en Language.ES Spanish es Language.FI Finnish fi Language.FIL Filipino fil Language.FR French fr Language.HI Hindi hi Language.HR Croatian hr Language.HU Hungarian hu Language.ID Indonesian id Language.IT Italian it Language.JA Japanese ja Language.KO Korean ko Language.MS Malay ms Language.NL Dutch nl Language.NO Norwegian no Language.PL Polish pl Language.PT Portuguese pt Language.RO Romanian ro Language.RU Russian ru Language.SK Slovak sk Language.SV Swedish sv Language.TA Tamil ta Language.TR Turkish tr Language.UK Ukrainian uk Language.VI Vietnamese vi Language.ZH Chinese zh Note: Language support may vary based on the selected model. See the ElevenLabs docs for more details. ​ Usage Example Copy Ask AI # Configure service with specific language service = ElevenLabsTTSService( api_key = "your-api-key" , voice_id = "voice-id" , params = ElevenLabsTTSService.InputParams( language = Language. FR # French ) ) ​ Frame Flow ​ Notes WebSocket implementation includes a 10-second keepalive mechanism Sample rate must be one of: 16000, 22050, 24000, or 44100 Hz Voice settings require both stability and similarity_boost to be set The language parameter only works with multilingual models WebSocket implementation pauses frame processing during speech generation HTTP implementation requires an external aiohttp ClientSession Deepgram Fish Audio On this page Overview Installation ElevenLabsTTSService (WebSocket) Configuration InputParams ElevenLabsHttpTTSService (HTTP) Configuration Output Frames TTSStartedFrame TTSAudioRawFrame TTSStoppedFrame ErrorFrame (HTTP implementation) Usage Examples Basic Usage With Voice Settings Methods Language Support Usage Example Frame Flow Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_elevenlabs_0698b276.txt b/tts_elevenlabs_0698b276.txt
new file mode 100644
index 0000000000000000000000000000000000000000..377be46a7f2f4c00745c880c1ee545d294362b05
--- /dev/null
+++ b/tts_elevenlabs_0698b276.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/elevenlabs#param-style
+Title: ElevenLabs - Pipecat
+==================================================
+
+ElevenLabs - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech ElevenLabs Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview ElevenLabs TTS provides high-quality text-to-speech synthesis through two service implementations: ElevenLabsTTSService : WebSocket-based implementation with word-level timing and interruption support ElevenLabsHttpTTSService : HTTP-based implementation for simpler use cases ​ Installation To use ElevenLabsTTSService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[elevenlabs]" You’ll also need to set up your ElevenLabs API key as an environment variable: ELEVENLABS_API_KEY . You can obtain a ElevenLabs API key by signing up at ElevenLabs . ​ ElevenLabsTTSService (WebSocket) ​ Configuration ​ api_key str required ElevenLabs API key ​ voice_id str required Voice identifier ​ model str default: "eleven_flash_v2_5" Model identifier ​ url str default: "wss://api.elevenlabs.io" API endpoint URL ​ sample_rate int default: "None" Output audio sample rate in Hz ​ params InputParams default: "InputParams()" Additional configuration parameters ​ text_filter BaseTextFilter default: "None" Modifies text provided to the TTS. Learn more about the available filters. ​ InputParams ​ language Language default: "None" The language of the text to be synthesized ​ optimize_streaming_latency str default: "None" Optimization level for streaming latency ​ stability float default: "None" Defines the stability for voice settings ​ similarity_boost float default: "None" Defines the similarity boost for voice settings ​ style float default: "None" Defines the style for voice settings. Available on V2+ models ​ use_speaker_boost bool default: "None" Defines whether to use speaker boost for voice settings. Available on V2+ models ​ speed float default: "None" Speech rate multiplier. Higher values increase speech speed ​ auto_mode bool default: "True" This parameter focuses on reducing the latency by disabling the chunk schedule and buffers. Recommended when sending full sentences or phrases ​ ElevenLabsHttpTTSService (HTTP) ​ Configuration ​ api_key str required ElevenLabs API key ​ voice_id str required Voice identifier ​ aiohttp_session aiohttp.ClientSession required aiohttp ClientSession for HTTP requests ​ model str default: "eleven_flash_v2_5" Model identifier ​ base_url str default: "https://api.elevenlabs.io" API base URL ​ sample_rate int default: "None" Output audio sample rate in Hz ​ params InputParams default: "InputParams()" Additional configuration parameters (similar to WebSocket implementation) ​ Output Frames ​ TTSStartedFrame Signals the start of audio generation. ​ TTSAudioRawFrame Contains generated audio data: ​ audio bytes Raw audio data chunk ​ sample_rate int Audio sample rate ​ num_channels int Number of audio channels (1 for mono) ​ TTSStoppedFrame Signals the completion of audio generation. ​ ErrorFrame (HTTP implementation) Sent when an error occurs during HTTP TTS generation: ​ error str Error message describing what went wrong ​ Usage Examples ​ Basic Usage Copy Ask AI # Configure service tts = ElevenLabsTTSService( api_key = "your-api-key" , voice_id = "voice-id" , sample_rate = 24000 , params = ElevenLabsTTSService.InputParams( language = Language. EN ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output() ]) ​ With Voice Settings Copy Ask AI # Configure with voice customization tts = ElevenLabsTTSService( api_key = "your-api-key" , voice_id = "voice-id" , params = ElevenLabsTTSService.InputParams( stability = 0.7 , similarity_boost = 0.8 , style = 0.5 , use_speaker_boost = True ) ) ​ Methods See the TTS base class methods for additional functionality. ​ Language Support ElevenLabs supports the following languages and their variants: Language Code Description Service Code Language.AR Arabic ar Language.BG Bulgarian bg Language.CS Czech cs Language.DA Danish da Language.DE German de Language.EL Greek el Language.EN English en Language.ES Spanish es Language.FI Finnish fi Language.FIL Filipino fil Language.FR French fr Language.HI Hindi hi Language.HR Croatian hr Language.HU Hungarian hu Language.ID Indonesian id Language.IT Italian it Language.JA Japanese ja Language.KO Korean ko Language.MS Malay ms Language.NL Dutch nl Language.NO Norwegian no Language.PL Polish pl Language.PT Portuguese pt Language.RO Romanian ro Language.RU Russian ru Language.SK Slovak sk Language.SV Swedish sv Language.TA Tamil ta Language.TR Turkish tr Language.UK Ukrainian uk Language.VI Vietnamese vi Language.ZH Chinese zh Note: Language support may vary based on the selected model. See the ElevenLabs docs for more details. ​ Usage Example Copy Ask AI # Configure service with specific language service = ElevenLabsTTSService( api_key = "your-api-key" , voice_id = "voice-id" , params = ElevenLabsTTSService.InputParams( language = Language. FR # French ) ) ​ Frame Flow ​ Notes WebSocket implementation includes a 10-second keepalive mechanism Sample rate must be one of: 16000, 22050, 24000, or 44100 Hz Voice settings require both stability and similarity_boost to be set The language parameter only works with multilingual models WebSocket implementation pauses frame processing during speech generation HTTP implementation requires an external aiohttp ClientSession Deepgram Fish Audio On this page Overview Installation ElevenLabsTTSService (WebSocket) Configuration InputParams ElevenLabsHttpTTSService (HTTP) Configuration Output Frames TTSStartedFrame TTSAudioRawFrame TTSStoppedFrame ErrorFrame (HTTP implementation) Usage Examples Basic Usage With Voice Settings Methods Language Support Usage Example Frame Flow Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_elevenlabs_324f6ddc.txt b/tts_elevenlabs_324f6ddc.txt
new file mode 100644
index 0000000000000000000000000000000000000000..dd46a28957c84262fb19840babfdec29856b8ea4
--- /dev/null
+++ b/tts_elevenlabs_324f6ddc.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/elevenlabs#overview
+Title: ElevenLabs - Pipecat
+==================================================
+
+ElevenLabs - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech ElevenLabs Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview ElevenLabs TTS provides high-quality text-to-speech synthesis through two service implementations: ElevenLabsTTSService : WebSocket-based implementation with word-level timing and interruption support ElevenLabsHttpTTSService : HTTP-based implementation for simpler use cases ​ Installation To use ElevenLabsTTSService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[elevenlabs]" You’ll also need to set up your ElevenLabs API key as an environment variable: ELEVENLABS_API_KEY . You can obtain a ElevenLabs API key by signing up at ElevenLabs . ​ ElevenLabsTTSService (WebSocket) ​ Configuration ​ api_key str required ElevenLabs API key ​ voice_id str required Voice identifier ​ model str default: "eleven_flash_v2_5" Model identifier ​ url str default: "wss://api.elevenlabs.io" API endpoint URL ​ sample_rate int default: "None" Output audio sample rate in Hz ​ params InputParams default: "InputParams()" Additional configuration parameters ​ text_filter BaseTextFilter default: "None" Modifies text provided to the TTS. Learn more about the available filters. ​ InputParams ​ language Language default: "None" The language of the text to be synthesized ​ optimize_streaming_latency str default: "None" Optimization level for streaming latency ​ stability float default: "None" Defines the stability for voice settings ​ similarity_boost float default: "None" Defines the similarity boost for voice settings ​ style float default: "None" Defines the style for voice settings. Available on V2+ models ​ use_speaker_boost bool default: "None" Defines whether to use speaker boost for voice settings. Available on V2+ models ​ speed float default: "None" Speech rate multiplier. Higher values increase speech speed ​ auto_mode bool default: "True" This parameter focuses on reducing the latency by disabling the chunk schedule and buffers. Recommended when sending full sentences or phrases ​ ElevenLabsHttpTTSService (HTTP) ​ Configuration ​ api_key str required ElevenLabs API key ​ voice_id str required Voice identifier ​ aiohttp_session aiohttp.ClientSession required aiohttp ClientSession for HTTP requests ​ model str default: "eleven_flash_v2_5" Model identifier ​ base_url str default: "https://api.elevenlabs.io" API base URL ​ sample_rate int default: "None" Output audio sample rate in Hz ​ params InputParams default: "InputParams()" Additional configuration parameters (similar to WebSocket implementation) ​ Output Frames ​ TTSStartedFrame Signals the start of audio generation. ​ TTSAudioRawFrame Contains generated audio data: ​ audio bytes Raw audio data chunk ​ sample_rate int Audio sample rate ​ num_channels int Number of audio channels (1 for mono) ​ TTSStoppedFrame Signals the completion of audio generation. ​ ErrorFrame (HTTP implementation) Sent when an error occurs during HTTP TTS generation: ​ error str Error message describing what went wrong ​ Usage Examples ​ Basic Usage Copy Ask AI # Configure service tts = ElevenLabsTTSService( api_key = "your-api-key" , voice_id = "voice-id" , sample_rate = 24000 , params = ElevenLabsTTSService.InputParams( language = Language. EN ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output() ]) ​ With Voice Settings Copy Ask AI # Configure with voice customization tts = ElevenLabsTTSService( api_key = "your-api-key" , voice_id = "voice-id" , params = ElevenLabsTTSService.InputParams( stability = 0.7 , similarity_boost = 0.8 , style = 0.5 , use_speaker_boost = True ) ) ​ Methods See the TTS base class methods for additional functionality. ​ Language Support ElevenLabs supports the following languages and their variants: Language Code Description Service Code Language.AR Arabic ar Language.BG Bulgarian bg Language.CS Czech cs Language.DA Danish da Language.DE German de Language.EL Greek el Language.EN English en Language.ES Spanish es Language.FI Finnish fi Language.FIL Filipino fil Language.FR French fr Language.HI Hindi hi Language.HR Croatian hr Language.HU Hungarian hu Language.ID Indonesian id Language.IT Italian it Language.JA Japanese ja Language.KO Korean ko Language.MS Malay ms Language.NL Dutch nl Language.NO Norwegian no Language.PL Polish pl Language.PT Portuguese pt Language.RO Romanian ro Language.RU Russian ru Language.SK Slovak sk Language.SV Swedish sv Language.TA Tamil ta Language.TR Turkish tr Language.UK Ukrainian uk Language.VI Vietnamese vi Language.ZH Chinese zh Note: Language support may vary based on the selected model. See the ElevenLabs docs for more details. ​ Usage Example Copy Ask AI # Configure service with specific language service = ElevenLabsTTSService( api_key = "your-api-key" , voice_id = "voice-id" , params = ElevenLabsTTSService.InputParams( language = Language. FR # French ) ) ​ Frame Flow ​ Notes WebSocket implementation includes a 10-second keepalive mechanism Sample rate must be one of: 16000, 22050, 24000, or 44100 Hz Voice settings require both stability and similarity_boost to be set The language parameter only works with multilingual models WebSocket implementation pauses frame processing during speech generation HTTP implementation requires an external aiohttp ClientSession Deepgram Fish Audio On this page Overview Installation ElevenLabsTTSService (WebSocket) Configuration InputParams ElevenLabsHttpTTSService (HTTP) Configuration Output Frames TTSStartedFrame TTSAudioRawFrame TTSStoppedFrame ErrorFrame (HTTP implementation) Usage Examples Basic Usage With Voice Settings Methods Language Support Usage Example Frame Flow Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_elevenlabs_36cd92b3.txt b/tts_elevenlabs_36cd92b3.txt
new file mode 100644
index 0000000000000000000000000000000000000000..7c5201620c5583c9de7b8ac24fadb00ec0b4c7ce
--- /dev/null
+++ b/tts_elevenlabs_36cd92b3.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/elevenlabs#language-support
+Title: ElevenLabs - Pipecat
+==================================================
+
+ElevenLabs - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech ElevenLabs Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview ElevenLabs TTS provides high-quality text-to-speech synthesis through two service implementations: ElevenLabsTTSService : WebSocket-based implementation with word-level timing and interruption support ElevenLabsHttpTTSService : HTTP-based implementation for simpler use cases ​ Installation To use ElevenLabsTTSService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[elevenlabs]" You’ll also need to set up your ElevenLabs API key as an environment variable: ELEVENLABS_API_KEY . You can obtain a ElevenLabs API key by signing up at ElevenLabs . ​ ElevenLabsTTSService (WebSocket) ​ Configuration ​ api_key str required ElevenLabs API key ​ voice_id str required Voice identifier ​ model str default: "eleven_flash_v2_5" Model identifier ​ url str default: "wss://api.elevenlabs.io" API endpoint URL ​ sample_rate int default: "None" Output audio sample rate in Hz ​ params InputParams default: "InputParams()" Additional configuration parameters ​ text_filter BaseTextFilter default: "None" Modifies text provided to the TTS. Learn more about the available filters. ​ InputParams ​ language Language default: "None" The language of the text to be synthesized ​ optimize_streaming_latency str default: "None" Optimization level for streaming latency ​ stability float default: "None" Defines the stability for voice settings ​ similarity_boost float default: "None" Defines the similarity boost for voice settings ​ style float default: "None" Defines the style for voice settings. Available on V2+ models ​ use_speaker_boost bool default: "None" Defines whether to use speaker boost for voice settings. Available on V2+ models ​ speed float default: "None" Speech rate multiplier. Higher values increase speech speed ​ auto_mode bool default: "True" This parameter focuses on reducing the latency by disabling the chunk schedule and buffers. Recommended when sending full sentences or phrases ​ ElevenLabsHttpTTSService (HTTP) ​ Configuration ​ api_key str required ElevenLabs API key ​ voice_id str required Voice identifier ​ aiohttp_session aiohttp.ClientSession required aiohttp ClientSession for HTTP requests ​ model str default: "eleven_flash_v2_5" Model identifier ​ base_url str default: "https://api.elevenlabs.io" API base URL ​ sample_rate int default: "None" Output audio sample rate in Hz ​ params InputParams default: "InputParams()" Additional configuration parameters (similar to WebSocket implementation) ​ Output Frames ​ TTSStartedFrame Signals the start of audio generation. ​ TTSAudioRawFrame Contains generated audio data: ​ audio bytes Raw audio data chunk ​ sample_rate int Audio sample rate ​ num_channels int Number of audio channels (1 for mono) ​ TTSStoppedFrame Signals the completion of audio generation. ​ ErrorFrame (HTTP implementation) Sent when an error occurs during HTTP TTS generation: ​ error str Error message describing what went wrong ​ Usage Examples ​ Basic Usage Copy Ask AI # Configure service tts = ElevenLabsTTSService( api_key = "your-api-key" , voice_id = "voice-id" , sample_rate = 24000 , params = ElevenLabsTTSService.InputParams( language = Language. EN ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output() ]) ​ With Voice Settings Copy Ask AI # Configure with voice customization tts = ElevenLabsTTSService( api_key = "your-api-key" , voice_id = "voice-id" , params = ElevenLabsTTSService.InputParams( stability = 0.7 , similarity_boost = 0.8 , style = 0.5 , use_speaker_boost = True ) ) ​ Methods See the TTS base class methods for additional functionality. ​ Language Support ElevenLabs supports the following languages and their variants: Language Code Description Service Code Language.AR Arabic ar Language.BG Bulgarian bg Language.CS Czech cs Language.DA Danish da Language.DE German de Language.EL Greek el Language.EN English en Language.ES Spanish es Language.FI Finnish fi Language.FIL Filipino fil Language.FR French fr Language.HI Hindi hi Language.HR Croatian hr Language.HU Hungarian hu Language.ID Indonesian id Language.IT Italian it Language.JA Japanese ja Language.KO Korean ko Language.MS Malay ms Language.NL Dutch nl Language.NO Norwegian no Language.PL Polish pl Language.PT Portuguese pt Language.RO Romanian ro Language.RU Russian ru Language.SK Slovak sk Language.SV Swedish sv Language.TA Tamil ta Language.TR Turkish tr Language.UK Ukrainian uk Language.VI Vietnamese vi Language.ZH Chinese zh Note: Language support may vary based on the selected model. See the ElevenLabs docs for more details. ​ Usage Example Copy Ask AI # Configure service with specific language service = ElevenLabsTTSService( api_key = "your-api-key" , voice_id = "voice-id" , params = ElevenLabsTTSService.InputParams( language = Language. FR # French ) ) ​ Frame Flow ​ Notes WebSocket implementation includes a 10-second keepalive mechanism Sample rate must be one of: 16000, 22050, 24000, or 44100 Hz Voice settings require both stability and similarity_boost to be set The language parameter only works with multilingual models WebSocket implementation pauses frame processing during speech generation HTTP implementation requires an external aiohttp ClientSession Deepgram Fish Audio On this page Overview Installation ElevenLabsTTSService (WebSocket) Configuration InputParams ElevenLabsHttpTTSService (HTTP) Configuration Output Frames TTSStartedFrame TTSAudioRawFrame TTSStoppedFrame ErrorFrame (HTTP implementation) Usage Examples Basic Usage With Voice Settings Methods Language Support Usage Example Frame Flow Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_elevenlabs_558ccbe6.txt b/tts_elevenlabs_558ccbe6.txt
new file mode 100644
index 0000000000000000000000000000000000000000..778fb0e6edafe85fcd9cd9a72c52055935c78f55
--- /dev/null
+++ b/tts_elevenlabs_558ccbe6.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/elevenlabs#ttsaudiorawframe
+Title: ElevenLabs - Pipecat
+==================================================
+
+ElevenLabs - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech ElevenLabs Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview ElevenLabs TTS provides high-quality text-to-speech synthesis through two service implementations: ElevenLabsTTSService : WebSocket-based implementation with word-level timing and interruption support ElevenLabsHttpTTSService : HTTP-based implementation for simpler use cases ​ Installation To use ElevenLabsTTSService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[elevenlabs]" You’ll also need to set up your ElevenLabs API key as an environment variable: ELEVENLABS_API_KEY . You can obtain a ElevenLabs API key by signing up at ElevenLabs . ​ ElevenLabsTTSService (WebSocket) ​ Configuration ​ api_key str required ElevenLabs API key ​ voice_id str required Voice identifier ​ model str default: "eleven_flash_v2_5" Model identifier ​ url str default: "wss://api.elevenlabs.io" API endpoint URL ​ sample_rate int default: "None" Output audio sample rate in Hz ​ params InputParams default: "InputParams()" Additional configuration parameters ​ text_filter BaseTextFilter default: "None" Modifies text provided to the TTS. Learn more about the available filters. ​ InputParams ​ language Language default: "None" The language of the text to be synthesized ​ optimize_streaming_latency str default: "None" Optimization level for streaming latency ​ stability float default: "None" Defines the stability for voice settings ​ similarity_boost float default: "None" Defines the similarity boost for voice settings ​ style float default: "None" Defines the style for voice settings. Available on V2+ models ​ use_speaker_boost bool default: "None" Defines whether to use speaker boost for voice settings. Available on V2+ models ​ speed float default: "None" Speech rate multiplier. Higher values increase speech speed ​ auto_mode bool default: "True" This parameter focuses on reducing the latency by disabling the chunk schedule and buffers. Recommended when sending full sentences or phrases ​ ElevenLabsHttpTTSService (HTTP) ​ Configuration ​ api_key str required ElevenLabs API key ​ voice_id str required Voice identifier ​ aiohttp_session aiohttp.ClientSession required aiohttp ClientSession for HTTP requests ​ model str default: "eleven_flash_v2_5" Model identifier ​ base_url str default: "https://api.elevenlabs.io" API base URL ​ sample_rate int default: "None" Output audio sample rate in Hz ​ params InputParams default: "InputParams()" Additional configuration parameters (similar to WebSocket implementation) ​ Output Frames ​ TTSStartedFrame Signals the start of audio generation. ​ TTSAudioRawFrame Contains generated audio data: ​ audio bytes Raw audio data chunk ​ sample_rate int Audio sample rate ​ num_channels int Number of audio channels (1 for mono) ​ TTSStoppedFrame Signals the completion of audio generation. ​ ErrorFrame (HTTP implementation) Sent when an error occurs during HTTP TTS generation: ​ error str Error message describing what went wrong ​ Usage Examples ​ Basic Usage Copy Ask AI # Configure service tts = ElevenLabsTTSService( api_key = "your-api-key" , voice_id = "voice-id" , sample_rate = 24000 , params = ElevenLabsTTSService.InputParams( language = Language. EN ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output() ]) ​ With Voice Settings Copy Ask AI # Configure with voice customization tts = ElevenLabsTTSService( api_key = "your-api-key" , voice_id = "voice-id" , params = ElevenLabsTTSService.InputParams( stability = 0.7 , similarity_boost = 0.8 , style = 0.5 , use_speaker_boost = True ) ) ​ Methods See the TTS base class methods for additional functionality. ​ Language Support ElevenLabs supports the following languages and their variants: Language Code Description Service Code Language.AR Arabic ar Language.BG Bulgarian bg Language.CS Czech cs Language.DA Danish da Language.DE German de Language.EL Greek el Language.EN English en Language.ES Spanish es Language.FI Finnish fi Language.FIL Filipino fil Language.FR French fr Language.HI Hindi hi Language.HR Croatian hr Language.HU Hungarian hu Language.ID Indonesian id Language.IT Italian it Language.JA Japanese ja Language.KO Korean ko Language.MS Malay ms Language.NL Dutch nl Language.NO Norwegian no Language.PL Polish pl Language.PT Portuguese pt Language.RO Romanian ro Language.RU Russian ru Language.SK Slovak sk Language.SV Swedish sv Language.TA Tamil ta Language.TR Turkish tr Language.UK Ukrainian uk Language.VI Vietnamese vi Language.ZH Chinese zh Note: Language support may vary based on the selected model. See the ElevenLabs docs for more details. ​ Usage Example Copy Ask AI # Configure service with specific language service = ElevenLabsTTSService( api_key = "your-api-key" , voice_id = "voice-id" , params = ElevenLabsTTSService.InputParams( language = Language. FR # French ) ) ​ Frame Flow ​ Notes WebSocket implementation includes a 10-second keepalive mechanism Sample rate must be one of: 16000, 22050, 24000, or 44100 Hz Voice settings require both stability and similarity_boost to be set The language parameter only works with multilingual models WebSocket implementation pauses frame processing during speech generation HTTP implementation requires an external aiohttp ClientSession Deepgram Fish Audio On this page Overview Installation ElevenLabsTTSService (WebSocket) Configuration InputParams ElevenLabsHttpTTSService (HTTP) Configuration Output Frames TTSStartedFrame TTSAudioRawFrame TTSStoppedFrame ErrorFrame (HTTP implementation) Usage Examples Basic Usage With Voice Settings Methods Language Support Usage Example Frame Flow Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_elevenlabs_8525fa52.txt b/tts_elevenlabs_8525fa52.txt
new file mode 100644
index 0000000000000000000000000000000000000000..15bbfe15c2efc8ba39bc48624a77e9dfb3ee09cd
--- /dev/null
+++ b/tts_elevenlabs_8525fa52.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/elevenlabs#usage-example
+Title: ElevenLabs - Pipecat
+==================================================
+
+ElevenLabs - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech ElevenLabs Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview ElevenLabs TTS provides high-quality text-to-speech synthesis through two service implementations: ElevenLabsTTSService : WebSocket-based implementation with word-level timing and interruption support ElevenLabsHttpTTSService : HTTP-based implementation for simpler use cases ​ Installation To use ElevenLabsTTSService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[elevenlabs]" You’ll also need to set up your ElevenLabs API key as an environment variable: ELEVENLABS_API_KEY . You can obtain a ElevenLabs API key by signing up at ElevenLabs . ​ ElevenLabsTTSService (WebSocket) ​ Configuration ​ api_key str required ElevenLabs API key ​ voice_id str required Voice identifier ​ model str default: "eleven_flash_v2_5" Model identifier ​ url str default: "wss://api.elevenlabs.io" API endpoint URL ​ sample_rate int default: "None" Output audio sample rate in Hz ​ params InputParams default: "InputParams()" Additional configuration parameters ​ text_filter BaseTextFilter default: "None" Modifies text provided to the TTS. Learn more about the available filters. ​ InputParams ​ language Language default: "None" The language of the text to be synthesized ​ optimize_streaming_latency str default: "None" Optimization level for streaming latency ​ stability float default: "None" Defines the stability for voice settings ​ similarity_boost float default: "None" Defines the similarity boost for voice settings ​ style float default: "None" Defines the style for voice settings. Available on V2+ models ​ use_speaker_boost bool default: "None" Defines whether to use speaker boost for voice settings. Available on V2+ models ​ speed float default: "None" Speech rate multiplier. Higher values increase speech speed ​ auto_mode bool default: "True" This parameter focuses on reducing the latency by disabling the chunk schedule and buffers. Recommended when sending full sentences or phrases ​ ElevenLabsHttpTTSService (HTTP) ​ Configuration ​ api_key str required ElevenLabs API key ​ voice_id str required Voice identifier ​ aiohttp_session aiohttp.ClientSession required aiohttp ClientSession for HTTP requests ​ model str default: "eleven_flash_v2_5" Model identifier ​ base_url str default: "https://api.elevenlabs.io" API base URL ​ sample_rate int default: "None" Output audio sample rate in Hz ​ params InputParams default: "InputParams()" Additional configuration parameters (similar to WebSocket implementation) ​ Output Frames ​ TTSStartedFrame Signals the start of audio generation. ​ TTSAudioRawFrame Contains generated audio data: ​ audio bytes Raw audio data chunk ​ sample_rate int Audio sample rate ​ num_channels int Number of audio channels (1 for mono) ​ TTSStoppedFrame Signals the completion of audio generation. ​ ErrorFrame (HTTP implementation) Sent when an error occurs during HTTP TTS generation: ​ error str Error message describing what went wrong ​ Usage Examples ​ Basic Usage Copy Ask AI # Configure service tts = ElevenLabsTTSService( api_key = "your-api-key" , voice_id = "voice-id" , sample_rate = 24000 , params = ElevenLabsTTSService.InputParams( language = Language. EN ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output() ]) ​ With Voice Settings Copy Ask AI # Configure with voice customization tts = ElevenLabsTTSService( api_key = "your-api-key" , voice_id = "voice-id" , params = ElevenLabsTTSService.InputParams( stability = 0.7 , similarity_boost = 0.8 , style = 0.5 , use_speaker_boost = True ) ) ​ Methods See the TTS base class methods for additional functionality. ​ Language Support ElevenLabs supports the following languages and their variants: Language Code Description Service Code Language.AR Arabic ar Language.BG Bulgarian bg Language.CS Czech cs Language.DA Danish da Language.DE German de Language.EL Greek el Language.EN English en Language.ES Spanish es Language.FI Finnish fi Language.FIL Filipino fil Language.FR French fr Language.HI Hindi hi Language.HR Croatian hr Language.HU Hungarian hu Language.ID Indonesian id Language.IT Italian it Language.JA Japanese ja Language.KO Korean ko Language.MS Malay ms Language.NL Dutch nl Language.NO Norwegian no Language.PL Polish pl Language.PT Portuguese pt Language.RO Romanian ro Language.RU Russian ru Language.SK Slovak sk Language.SV Swedish sv Language.TA Tamil ta Language.TR Turkish tr Language.UK Ukrainian uk Language.VI Vietnamese vi Language.ZH Chinese zh Note: Language support may vary based on the selected model. See the ElevenLabs docs for more details. ​ Usage Example Copy Ask AI # Configure service with specific language service = ElevenLabsTTSService( api_key = "your-api-key" , voice_id = "voice-id" , params = ElevenLabsTTSService.InputParams( language = Language. FR # French ) ) ​ Frame Flow ​ Notes WebSocket implementation includes a 10-second keepalive mechanism Sample rate must be one of: 16000, 22050, 24000, or 44100 Hz Voice settings require both stability and similarity_boost to be set The language parameter only works with multilingual models WebSocket implementation pauses frame processing during speech generation HTTP implementation requires an external aiohttp ClientSession Deepgram Fish Audio On this page Overview Installation ElevenLabsTTSService (WebSocket) Configuration InputParams ElevenLabsHttpTTSService (HTTP) Configuration Output Frames TTSStartedFrame TTSAudioRawFrame TTSStoppedFrame ErrorFrame (HTTP implementation) Usage Examples Basic Usage With Voice Settings Methods Language Support Usage Example Frame Flow Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_elevenlabs_85888c15.txt b/tts_elevenlabs_85888c15.txt
new file mode 100644
index 0000000000000000000000000000000000000000..87827377f2f93d093a0b456b75b80d39fbecbae8
--- /dev/null
+++ b/tts_elevenlabs_85888c15.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/elevenlabs#param-model-1
+Title: ElevenLabs - Pipecat
+==================================================
+
+ElevenLabs - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech ElevenLabs Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview ElevenLabs TTS provides high-quality text-to-speech synthesis through two service implementations: ElevenLabsTTSService : WebSocket-based implementation with word-level timing and interruption support ElevenLabsHttpTTSService : HTTP-based implementation for simpler use cases ​ Installation To use ElevenLabsTTSService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[elevenlabs]" You’ll also need to set up your ElevenLabs API key as an environment variable: ELEVENLABS_API_KEY . You can obtain a ElevenLabs API key by signing up at ElevenLabs . ​ ElevenLabsTTSService (WebSocket) ​ Configuration ​ api_key str required ElevenLabs API key ​ voice_id str required Voice identifier ​ model str default: "eleven_flash_v2_5" Model identifier ​ url str default: "wss://api.elevenlabs.io" API endpoint URL ​ sample_rate int default: "None" Output audio sample rate in Hz ​ params InputParams default: "InputParams()" Additional configuration parameters ​ text_filter BaseTextFilter default: "None" Modifies text provided to the TTS. Learn more about the available filters. ​ InputParams ​ language Language default: "None" The language of the text to be synthesized ​ optimize_streaming_latency str default: "None" Optimization level for streaming latency ​ stability float default: "None" Defines the stability for voice settings ​ similarity_boost float default: "None" Defines the similarity boost for voice settings ​ style float default: "None" Defines the style for voice settings. Available on V2+ models ​ use_speaker_boost bool default: "None" Defines whether to use speaker boost for voice settings. Available on V2+ models ​ speed float default: "None" Speech rate multiplier. Higher values increase speech speed ​ auto_mode bool default: "True" This parameter focuses on reducing the latency by disabling the chunk schedule and buffers. Recommended when sending full sentences or phrases ​ ElevenLabsHttpTTSService (HTTP) ​ Configuration ​ api_key str required ElevenLabs API key ​ voice_id str required Voice identifier ​ aiohttp_session aiohttp.ClientSession required aiohttp ClientSession for HTTP requests ​ model str default: "eleven_flash_v2_5" Model identifier ​ base_url str default: "https://api.elevenlabs.io" API base URL ​ sample_rate int default: "None" Output audio sample rate in Hz ​ params InputParams default: "InputParams()" Additional configuration parameters (similar to WebSocket implementation) ​ Output Frames ​ TTSStartedFrame Signals the start of audio generation. ​ TTSAudioRawFrame Contains generated audio data: ​ audio bytes Raw audio data chunk ​ sample_rate int Audio sample rate ​ num_channels int Number of audio channels (1 for mono) ​ TTSStoppedFrame Signals the completion of audio generation. ​ ErrorFrame (HTTP implementation) Sent when an error occurs during HTTP TTS generation: ​ error str Error message describing what went wrong ​ Usage Examples ​ Basic Usage Copy Ask AI # Configure service tts = ElevenLabsTTSService( api_key = "your-api-key" , voice_id = "voice-id" , sample_rate = 24000 , params = ElevenLabsTTSService.InputParams( language = Language. EN ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output() ]) ​ With Voice Settings Copy Ask AI # Configure with voice customization tts = ElevenLabsTTSService( api_key = "your-api-key" , voice_id = "voice-id" , params = ElevenLabsTTSService.InputParams( stability = 0.7 , similarity_boost = 0.8 , style = 0.5 , use_speaker_boost = True ) ) ​ Methods See the TTS base class methods for additional functionality. ​ Language Support ElevenLabs supports the following languages and their variants: Language Code Description Service Code Language.AR Arabic ar Language.BG Bulgarian bg Language.CS Czech cs Language.DA Danish da Language.DE German de Language.EL Greek el Language.EN English en Language.ES Spanish es Language.FI Finnish fi Language.FIL Filipino fil Language.FR French fr Language.HI Hindi hi Language.HR Croatian hr Language.HU Hungarian hu Language.ID Indonesian id Language.IT Italian it Language.JA Japanese ja Language.KO Korean ko Language.MS Malay ms Language.NL Dutch nl Language.NO Norwegian no Language.PL Polish pl Language.PT Portuguese pt Language.RO Romanian ro Language.RU Russian ru Language.SK Slovak sk Language.SV Swedish sv Language.TA Tamil ta Language.TR Turkish tr Language.UK Ukrainian uk Language.VI Vietnamese vi Language.ZH Chinese zh Note: Language support may vary based on the selected model. See the ElevenLabs docs for more details. ​ Usage Example Copy Ask AI # Configure service with specific language service = ElevenLabsTTSService( api_key = "your-api-key" , voice_id = "voice-id" , params = ElevenLabsTTSService.InputParams( language = Language. FR # French ) ) ​ Frame Flow ​ Notes WebSocket implementation includes a 10-second keepalive mechanism Sample rate must be one of: 16000, 22050, 24000, or 44100 Hz Voice settings require both stability and similarity_boost to be set The language parameter only works with multilingual models WebSocket implementation pauses frame processing during speech generation HTTP implementation requires an external aiohttp ClientSession Deepgram Fish Audio On this page Overview Installation ElevenLabsTTSService (WebSocket) Configuration InputParams ElevenLabsHttpTTSService (HTTP) Configuration Output Frames TTSStartedFrame TTSAudioRawFrame TTSStoppedFrame ErrorFrame (HTTP implementation) Usage Examples Basic Usage With Voice Settings Methods Language Support Usage Example Frame Flow Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_elevenlabs_87023363.txt b/tts_elevenlabs_87023363.txt
new file mode 100644
index 0000000000000000000000000000000000000000..ce5a572f78a39c5831a2c607bc64f588fd2e56b5
--- /dev/null
+++ b/tts_elevenlabs_87023363.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/elevenlabs#with-voice-settings
+Title: ElevenLabs - Pipecat
+==================================================
+
+ElevenLabs - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech ElevenLabs Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview ElevenLabs TTS provides high-quality text-to-speech synthesis through two service implementations: ElevenLabsTTSService : WebSocket-based implementation with word-level timing and interruption support ElevenLabsHttpTTSService : HTTP-based implementation for simpler use cases ​ Installation To use ElevenLabsTTSService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[elevenlabs]" You’ll also need to set up your ElevenLabs API key as an environment variable: ELEVENLABS_API_KEY . You can obtain a ElevenLabs API key by signing up at ElevenLabs . ​ ElevenLabsTTSService (WebSocket) ​ Configuration ​ api_key str required ElevenLabs API key ​ voice_id str required Voice identifier ​ model str default: "eleven_flash_v2_5" Model identifier ​ url str default: "wss://api.elevenlabs.io" API endpoint URL ​ sample_rate int default: "None" Output audio sample rate in Hz ​ params InputParams default: "InputParams()" Additional configuration parameters ​ text_filter BaseTextFilter default: "None" Modifies text provided to the TTS. Learn more about the available filters. ​ InputParams ​ language Language default: "None" The language of the text to be synthesized ​ optimize_streaming_latency str default: "None" Optimization level for streaming latency ​ stability float default: "None" Defines the stability for voice settings ​ similarity_boost float default: "None" Defines the similarity boost for voice settings ​ style float default: "None" Defines the style for voice settings. Available on V2+ models ​ use_speaker_boost bool default: "None" Defines whether to use speaker boost for voice settings. Available on V2+ models ​ speed float default: "None" Speech rate multiplier. Higher values increase speech speed ​ auto_mode bool default: "True" This parameter focuses on reducing the latency by disabling the chunk schedule and buffers. Recommended when sending full sentences or phrases ​ ElevenLabsHttpTTSService (HTTP) ​ Configuration ​ api_key str required ElevenLabs API key ​ voice_id str required Voice identifier ​ aiohttp_session aiohttp.ClientSession required aiohttp ClientSession for HTTP requests ​ model str default: "eleven_flash_v2_5" Model identifier ​ base_url str default: "https://api.elevenlabs.io" API base URL ​ sample_rate int default: "None" Output audio sample rate in Hz ​ params InputParams default: "InputParams()" Additional configuration parameters (similar to WebSocket implementation) ​ Output Frames ​ TTSStartedFrame Signals the start of audio generation. ​ TTSAudioRawFrame Contains generated audio data: ​ audio bytes Raw audio data chunk ​ sample_rate int Audio sample rate ​ num_channels int Number of audio channels (1 for mono) ​ TTSStoppedFrame Signals the completion of audio generation. ​ ErrorFrame (HTTP implementation) Sent when an error occurs during HTTP TTS generation: ​ error str Error message describing what went wrong ​ Usage Examples ​ Basic Usage Copy Ask AI # Configure service tts = ElevenLabsTTSService( api_key = "your-api-key" , voice_id = "voice-id" , sample_rate = 24000 , params = ElevenLabsTTSService.InputParams( language = Language. EN ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output() ]) ​ With Voice Settings Copy Ask AI # Configure with voice customization tts = ElevenLabsTTSService( api_key = "your-api-key" , voice_id = "voice-id" , params = ElevenLabsTTSService.InputParams( stability = 0.7 , similarity_boost = 0.8 , style = 0.5 , use_speaker_boost = True ) ) ​ Methods See the TTS base class methods for additional functionality. ​ Language Support ElevenLabs supports the following languages and their variants: Language Code Description Service Code Language.AR Arabic ar Language.BG Bulgarian bg Language.CS Czech cs Language.DA Danish da Language.DE German de Language.EL Greek el Language.EN English en Language.ES Spanish es Language.FI Finnish fi Language.FIL Filipino fil Language.FR French fr Language.HI Hindi hi Language.HR Croatian hr Language.HU Hungarian hu Language.ID Indonesian id Language.IT Italian it Language.JA Japanese ja Language.KO Korean ko Language.MS Malay ms Language.NL Dutch nl Language.NO Norwegian no Language.PL Polish pl Language.PT Portuguese pt Language.RO Romanian ro Language.RU Russian ru Language.SK Slovak sk Language.SV Swedish sv Language.TA Tamil ta Language.TR Turkish tr Language.UK Ukrainian uk Language.VI Vietnamese vi Language.ZH Chinese zh Note: Language support may vary based on the selected model. See the ElevenLabs docs for more details. ​ Usage Example Copy Ask AI # Configure service with specific language service = ElevenLabsTTSService( api_key = "your-api-key" , voice_id = "voice-id" , params = ElevenLabsTTSService.InputParams( language = Language. FR # French ) ) ​ Frame Flow ​ Notes WebSocket implementation includes a 10-second keepalive mechanism Sample rate must be one of: 16000, 22050, 24000, or 44100 Hz Voice settings require both stability and similarity_boost to be set The language parameter only works with multilingual models WebSocket implementation pauses frame processing during speech generation HTTP implementation requires an external aiohttp ClientSession Deepgram Fish Audio On this page Overview Installation ElevenLabsTTSService (WebSocket) Configuration InputParams ElevenLabsHttpTTSService (HTTP) Configuration Output Frames TTSStartedFrame TTSAudioRawFrame TTSStoppedFrame ErrorFrame (HTTP implementation) Usage Examples Basic Usage With Voice Settings Methods Language Support Usage Example Frame Flow Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_elevenlabs_92438ade.txt b/tts_elevenlabs_92438ade.txt
new file mode 100644
index 0000000000000000000000000000000000000000..c90076e11d2beb97548e3b1fd308a263faec46fb
--- /dev/null
+++ b/tts_elevenlabs_92438ade.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/elevenlabs#frame-flow
+Title: ElevenLabs - Pipecat
+==================================================
+
+ElevenLabs - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech ElevenLabs Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview ElevenLabs TTS provides high-quality text-to-speech synthesis through two service implementations: ElevenLabsTTSService : WebSocket-based implementation with word-level timing and interruption support ElevenLabsHttpTTSService : HTTP-based implementation for simpler use cases ​ Installation To use ElevenLabsTTSService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[elevenlabs]" You’ll also need to set up your ElevenLabs API key as an environment variable: ELEVENLABS_API_KEY . You can obtain a ElevenLabs API key by signing up at ElevenLabs . ​ ElevenLabsTTSService (WebSocket) ​ Configuration ​ api_key str required ElevenLabs API key ​ voice_id str required Voice identifier ​ model str default: "eleven_flash_v2_5" Model identifier ​ url str default: "wss://api.elevenlabs.io" API endpoint URL ​ sample_rate int default: "None" Output audio sample rate in Hz ​ params InputParams default: "InputParams()" Additional configuration parameters ​ text_filter BaseTextFilter default: "None" Modifies text provided to the TTS. Learn more about the available filters. ​ InputParams ​ language Language default: "None" The language of the text to be synthesized ​ optimize_streaming_latency str default: "None" Optimization level for streaming latency ​ stability float default: "None" Defines the stability for voice settings ​ similarity_boost float default: "None" Defines the similarity boost for voice settings ​ style float default: "None" Defines the style for voice settings. Available on V2+ models ​ use_speaker_boost bool default: "None" Defines whether to use speaker boost for voice settings. Available on V2+ models ​ speed float default: "None" Speech rate multiplier. Higher values increase speech speed ​ auto_mode bool default: "True" This parameter focuses on reducing the latency by disabling the chunk schedule and buffers. Recommended when sending full sentences or phrases ​ ElevenLabsHttpTTSService (HTTP) ​ Configuration ​ api_key str required ElevenLabs API key ​ voice_id str required Voice identifier ​ aiohttp_session aiohttp.ClientSession required aiohttp ClientSession for HTTP requests ​ model str default: "eleven_flash_v2_5" Model identifier ​ base_url str default: "https://api.elevenlabs.io" API base URL ​ sample_rate int default: "None" Output audio sample rate in Hz ​ params InputParams default: "InputParams()" Additional configuration parameters (similar to WebSocket implementation) ​ Output Frames ​ TTSStartedFrame Signals the start of audio generation. ​ TTSAudioRawFrame Contains generated audio data: ​ audio bytes Raw audio data chunk ​ sample_rate int Audio sample rate ​ num_channels int Number of audio channels (1 for mono) ​ TTSStoppedFrame Signals the completion of audio generation. ​ ErrorFrame (HTTP implementation) Sent when an error occurs during HTTP TTS generation: ​ error str Error message describing what went wrong ​ Usage Examples ​ Basic Usage Copy Ask AI # Configure service tts = ElevenLabsTTSService( api_key = "your-api-key" , voice_id = "voice-id" , sample_rate = 24000 , params = ElevenLabsTTSService.InputParams( language = Language. EN ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output() ]) ​ With Voice Settings Copy Ask AI # Configure with voice customization tts = ElevenLabsTTSService( api_key = "your-api-key" , voice_id = "voice-id" , params = ElevenLabsTTSService.InputParams( stability = 0.7 , similarity_boost = 0.8 , style = 0.5 , use_speaker_boost = True ) ) ​ Methods See the TTS base class methods for additional functionality. ​ Language Support ElevenLabs supports the following languages and their variants: Language Code Description Service Code Language.AR Arabic ar Language.BG Bulgarian bg Language.CS Czech cs Language.DA Danish da Language.DE German de Language.EL Greek el Language.EN English en Language.ES Spanish es Language.FI Finnish fi Language.FIL Filipino fil Language.FR French fr Language.HI Hindi hi Language.HR Croatian hr Language.HU Hungarian hu Language.ID Indonesian id Language.IT Italian it Language.JA Japanese ja Language.KO Korean ko Language.MS Malay ms Language.NL Dutch nl Language.NO Norwegian no Language.PL Polish pl Language.PT Portuguese pt Language.RO Romanian ro Language.RU Russian ru Language.SK Slovak sk Language.SV Swedish sv Language.TA Tamil ta Language.TR Turkish tr Language.UK Ukrainian uk Language.VI Vietnamese vi Language.ZH Chinese zh Note: Language support may vary based on the selected model. See the ElevenLabs docs for more details. ​ Usage Example Copy Ask AI # Configure service with specific language service = ElevenLabsTTSService( api_key = "your-api-key" , voice_id = "voice-id" , params = ElevenLabsTTSService.InputParams( language = Language. FR # French ) ) ​ Frame Flow ​ Notes WebSocket implementation includes a 10-second keepalive mechanism Sample rate must be one of: 16000, 22050, 24000, or 44100 Hz Voice settings require both stability and similarity_boost to be set The language parameter only works with multilingual models WebSocket implementation pauses frame processing during speech generation HTTP implementation requires an external aiohttp ClientSession Deepgram Fish Audio On this page Overview Installation ElevenLabsTTSService (WebSocket) Configuration InputParams ElevenLabsHttpTTSService (HTTP) Configuration Output Frames TTSStartedFrame TTSAudioRawFrame TTSStoppedFrame ErrorFrame (HTTP implementation) Usage Examples Basic Usage With Voice Settings Methods Language Support Usage Example Frame Flow Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_elevenlabs_92ed8306.txt b/tts_elevenlabs_92ed8306.txt
new file mode 100644
index 0000000000000000000000000000000000000000..61d258c75a5bd03f47a3b3835baad8b24500f9c6
--- /dev/null
+++ b/tts_elevenlabs_92ed8306.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/elevenlabs#methods
+Title: ElevenLabs - Pipecat
+==================================================
+
+ElevenLabs - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech ElevenLabs Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview ElevenLabs TTS provides high-quality text-to-speech synthesis through two service implementations: ElevenLabsTTSService : WebSocket-based implementation with word-level timing and interruption support ElevenLabsHttpTTSService : HTTP-based implementation for simpler use cases ​ Installation To use ElevenLabsTTSService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[elevenlabs]" You’ll also need to set up your ElevenLabs API key as an environment variable: ELEVENLABS_API_KEY . You can obtain a ElevenLabs API key by signing up at ElevenLabs . ​ ElevenLabsTTSService (WebSocket) ​ Configuration ​ api_key str required ElevenLabs API key ​ voice_id str required Voice identifier ​ model str default: "eleven_flash_v2_5" Model identifier ​ url str default: "wss://api.elevenlabs.io" API endpoint URL ​ sample_rate int default: "None" Output audio sample rate in Hz ​ params InputParams default: "InputParams()" Additional configuration parameters ​ text_filter BaseTextFilter default: "None" Modifies text provided to the TTS. Learn more about the available filters. ​ InputParams ​ language Language default: "None" The language of the text to be synthesized ​ optimize_streaming_latency str default: "None" Optimization level for streaming latency ​ stability float default: "None" Defines the stability for voice settings ​ similarity_boost float default: "None" Defines the similarity boost for voice settings ​ style float default: "None" Defines the style for voice settings. Available on V2+ models ​ use_speaker_boost bool default: "None" Defines whether to use speaker boost for voice settings. Available on V2+ models ​ speed float default: "None" Speech rate multiplier. Higher values increase speech speed ​ auto_mode bool default: "True" This parameter focuses on reducing the latency by disabling the chunk schedule and buffers. Recommended when sending full sentences or phrases ​ ElevenLabsHttpTTSService (HTTP) ​ Configuration ​ api_key str required ElevenLabs API key ​ voice_id str required Voice identifier ​ aiohttp_session aiohttp.ClientSession required aiohttp ClientSession for HTTP requests ​ model str default: "eleven_flash_v2_5" Model identifier ​ base_url str default: "https://api.elevenlabs.io" API base URL ​ sample_rate int default: "None" Output audio sample rate in Hz ​ params InputParams default: "InputParams()" Additional configuration parameters (similar to WebSocket implementation) ​ Output Frames ​ TTSStartedFrame Signals the start of audio generation. ​ TTSAudioRawFrame Contains generated audio data: ​ audio bytes Raw audio data chunk ​ sample_rate int Audio sample rate ​ num_channels int Number of audio channels (1 for mono) ​ TTSStoppedFrame Signals the completion of audio generation. ​ ErrorFrame (HTTP implementation) Sent when an error occurs during HTTP TTS generation: ​ error str Error message describing what went wrong ​ Usage Examples ​ Basic Usage Copy Ask AI # Configure service tts = ElevenLabsTTSService( api_key = "your-api-key" , voice_id = "voice-id" , sample_rate = 24000 , params = ElevenLabsTTSService.InputParams( language = Language. EN ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output() ]) ​ With Voice Settings Copy Ask AI # Configure with voice customization tts = ElevenLabsTTSService( api_key = "your-api-key" , voice_id = "voice-id" , params = ElevenLabsTTSService.InputParams( stability = 0.7 , similarity_boost = 0.8 , style = 0.5 , use_speaker_boost = True ) ) ​ Methods See the TTS base class methods for additional functionality. ​ Language Support ElevenLabs supports the following languages and their variants: Language Code Description Service Code Language.AR Arabic ar Language.BG Bulgarian bg Language.CS Czech cs Language.DA Danish da Language.DE German de Language.EL Greek el Language.EN English en Language.ES Spanish es Language.FI Finnish fi Language.FIL Filipino fil Language.FR French fr Language.HI Hindi hi Language.HR Croatian hr Language.HU Hungarian hu Language.ID Indonesian id Language.IT Italian it Language.JA Japanese ja Language.KO Korean ko Language.MS Malay ms Language.NL Dutch nl Language.NO Norwegian no Language.PL Polish pl Language.PT Portuguese pt Language.RO Romanian ro Language.RU Russian ru Language.SK Slovak sk Language.SV Swedish sv Language.TA Tamil ta Language.TR Turkish tr Language.UK Ukrainian uk Language.VI Vietnamese vi Language.ZH Chinese zh Note: Language support may vary based on the selected model. See the ElevenLabs docs for more details. ​ Usage Example Copy Ask AI # Configure service with specific language service = ElevenLabsTTSService( api_key = "your-api-key" , voice_id = "voice-id" , params = ElevenLabsTTSService.InputParams( language = Language. FR # French ) ) ​ Frame Flow ​ Notes WebSocket implementation includes a 10-second keepalive mechanism Sample rate must be one of: 16000, 22050, 24000, or 44100 Hz Voice settings require both stability and similarity_boost to be set The language parameter only works with multilingual models WebSocket implementation pauses frame processing during speech generation HTTP implementation requires an external aiohttp ClientSession Deepgram Fish Audio On this page Overview Installation ElevenLabsTTSService (WebSocket) Configuration InputParams ElevenLabsHttpTTSService (HTTP) Configuration Output Frames TTSStartedFrame TTSAudioRawFrame TTSStoppedFrame ErrorFrame (HTTP implementation) Usage Examples Basic Usage With Voice Settings Methods Language Support Usage Example Frame Flow Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_elevenlabs_955b8f2f.txt b/tts_elevenlabs_955b8f2f.txt
new file mode 100644
index 0000000000000000000000000000000000000000..9b8d5c3d5468f2967c48455c9d4c6759079c1881
--- /dev/null
+++ b/tts_elevenlabs_955b8f2f.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/elevenlabs#inputparams
+Title: ElevenLabs - Pipecat
+==================================================
+
+ElevenLabs - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech ElevenLabs Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview ElevenLabs TTS provides high-quality text-to-speech synthesis through two service implementations: ElevenLabsTTSService : WebSocket-based implementation with word-level timing and interruption support ElevenLabsHttpTTSService : HTTP-based implementation for simpler use cases ​ Installation To use ElevenLabsTTSService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[elevenlabs]" You’ll also need to set up your ElevenLabs API key as an environment variable: ELEVENLABS_API_KEY . You can obtain a ElevenLabs API key by signing up at ElevenLabs . ​ ElevenLabsTTSService (WebSocket) ​ Configuration ​ api_key str required ElevenLabs API key ​ voice_id str required Voice identifier ​ model str default: "eleven_flash_v2_5" Model identifier ​ url str default: "wss://api.elevenlabs.io" API endpoint URL ​ sample_rate int default: "None" Output audio sample rate in Hz ​ params InputParams default: "InputParams()" Additional configuration parameters ​ text_filter BaseTextFilter default: "None" Modifies text provided to the TTS. Learn more about the available filters. ​ InputParams ​ language Language default: "None" The language of the text to be synthesized ​ optimize_streaming_latency str default: "None" Optimization level for streaming latency ​ stability float default: "None" Defines the stability for voice settings ​ similarity_boost float default: "None" Defines the similarity boost for voice settings ​ style float default: "None" Defines the style for voice settings. Available on V2+ models ​ use_speaker_boost bool default: "None" Defines whether to use speaker boost for voice settings. Available on V2+ models ​ speed float default: "None" Speech rate multiplier. Higher values increase speech speed ​ auto_mode bool default: "True" This parameter focuses on reducing the latency by disabling the chunk schedule and buffers. Recommended when sending full sentences or phrases ​ ElevenLabsHttpTTSService (HTTP) ​ Configuration ​ api_key str required ElevenLabs API key ​ voice_id str required Voice identifier ​ aiohttp_session aiohttp.ClientSession required aiohttp ClientSession for HTTP requests ​ model str default: "eleven_flash_v2_5" Model identifier ​ base_url str default: "https://api.elevenlabs.io" API base URL ​ sample_rate int default: "None" Output audio sample rate in Hz ​ params InputParams default: "InputParams()" Additional configuration parameters (similar to WebSocket implementation) ​ Output Frames ​ TTSStartedFrame Signals the start of audio generation. ​ TTSAudioRawFrame Contains generated audio data: ​ audio bytes Raw audio data chunk ​ sample_rate int Audio sample rate ​ num_channels int Number of audio channels (1 for mono) ​ TTSStoppedFrame Signals the completion of audio generation. ​ ErrorFrame (HTTP implementation) Sent when an error occurs during HTTP TTS generation: ​ error str Error message describing what went wrong ​ Usage Examples ​ Basic Usage Copy Ask AI # Configure service tts = ElevenLabsTTSService( api_key = "your-api-key" , voice_id = "voice-id" , sample_rate = 24000 , params = ElevenLabsTTSService.InputParams( language = Language. EN ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output() ]) ​ With Voice Settings Copy Ask AI # Configure with voice customization tts = ElevenLabsTTSService( api_key = "your-api-key" , voice_id = "voice-id" , params = ElevenLabsTTSService.InputParams( stability = 0.7 , similarity_boost = 0.8 , style = 0.5 , use_speaker_boost = True ) ) ​ Methods See the TTS base class methods for additional functionality. ​ Language Support ElevenLabs supports the following languages and their variants: Language Code Description Service Code Language.AR Arabic ar Language.BG Bulgarian bg Language.CS Czech cs Language.DA Danish da Language.DE German de Language.EL Greek el Language.EN English en Language.ES Spanish es Language.FI Finnish fi Language.FIL Filipino fil Language.FR French fr Language.HI Hindi hi Language.HR Croatian hr Language.HU Hungarian hu Language.ID Indonesian id Language.IT Italian it Language.JA Japanese ja Language.KO Korean ko Language.MS Malay ms Language.NL Dutch nl Language.NO Norwegian no Language.PL Polish pl Language.PT Portuguese pt Language.RO Romanian ro Language.RU Russian ru Language.SK Slovak sk Language.SV Swedish sv Language.TA Tamil ta Language.TR Turkish tr Language.UK Ukrainian uk Language.VI Vietnamese vi Language.ZH Chinese zh Note: Language support may vary based on the selected model. See the ElevenLabs docs for more details. ​ Usage Example Copy Ask AI # Configure service with specific language service = ElevenLabsTTSService( api_key = "your-api-key" , voice_id = "voice-id" , params = ElevenLabsTTSService.InputParams( language = Language. FR # French ) ) ​ Frame Flow ​ Notes WebSocket implementation includes a 10-second keepalive mechanism Sample rate must be one of: 16000, 22050, 24000, or 44100 Hz Voice settings require both stability and similarity_boost to be set The language parameter only works with multilingual models WebSocket implementation pauses frame processing during speech generation HTTP implementation requires an external aiohttp ClientSession Deepgram Fish Audio On this page Overview Installation ElevenLabsTTSService (WebSocket) Configuration InputParams ElevenLabsHttpTTSService (HTTP) Configuration Output Frames TTSStartedFrame TTSAudioRawFrame TTSStoppedFrame ErrorFrame (HTTP implementation) Usage Examples Basic Usage With Voice Settings Methods Language Support Usage Example Frame Flow Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_elevenlabs_cfb3d4d8.txt b/tts_elevenlabs_cfb3d4d8.txt
new file mode 100644
index 0000000000000000000000000000000000000000..27b6a6bf8abf471a1d77655f26c2a358e2fad86b
--- /dev/null
+++ b/tts_elevenlabs_cfb3d4d8.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/elevenlabs#configuration
+Title: ElevenLabs - Pipecat
+==================================================
+
+ElevenLabs - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech ElevenLabs Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview ElevenLabs TTS provides high-quality text-to-speech synthesis through two service implementations: ElevenLabsTTSService : WebSocket-based implementation with word-level timing and interruption support ElevenLabsHttpTTSService : HTTP-based implementation for simpler use cases ​ Installation To use ElevenLabsTTSService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[elevenlabs]" You’ll also need to set up your ElevenLabs API key as an environment variable: ELEVENLABS_API_KEY . You can obtain a ElevenLabs API key by signing up at ElevenLabs . ​ ElevenLabsTTSService (WebSocket) ​ Configuration ​ api_key str required ElevenLabs API key ​ voice_id str required Voice identifier ​ model str default: "eleven_flash_v2_5" Model identifier ​ url str default: "wss://api.elevenlabs.io" API endpoint URL ​ sample_rate int default: "None" Output audio sample rate in Hz ​ params InputParams default: "InputParams()" Additional configuration parameters ​ text_filter BaseTextFilter default: "None" Modifies text provided to the TTS. Learn more about the available filters. ​ InputParams ​ language Language default: "None" The language of the text to be synthesized ​ optimize_streaming_latency str default: "None" Optimization level for streaming latency ​ stability float default: "None" Defines the stability for voice settings ​ similarity_boost float default: "None" Defines the similarity boost for voice settings ​ style float default: "None" Defines the style for voice settings. Available on V2+ models ​ use_speaker_boost bool default: "None" Defines whether to use speaker boost for voice settings. Available on V2+ models ​ speed float default: "None" Speech rate multiplier. Higher values increase speech speed ​ auto_mode bool default: "True" This parameter focuses on reducing the latency by disabling the chunk schedule and buffers. Recommended when sending full sentences or phrases ​ ElevenLabsHttpTTSService (HTTP) ​ Configuration ​ api_key str required ElevenLabs API key ​ voice_id str required Voice identifier ​ aiohttp_session aiohttp.ClientSession required aiohttp ClientSession for HTTP requests ​ model str default: "eleven_flash_v2_5" Model identifier ​ base_url str default: "https://api.elevenlabs.io" API base URL ​ sample_rate int default: "None" Output audio sample rate in Hz ​ params InputParams default: "InputParams()" Additional configuration parameters (similar to WebSocket implementation) ​ Output Frames ​ TTSStartedFrame Signals the start of audio generation. ​ TTSAudioRawFrame Contains generated audio data: ​ audio bytes Raw audio data chunk ​ sample_rate int Audio sample rate ​ num_channels int Number of audio channels (1 for mono) ​ TTSStoppedFrame Signals the completion of audio generation. ​ ErrorFrame (HTTP implementation) Sent when an error occurs during HTTP TTS generation: ​ error str Error message describing what went wrong ​ Usage Examples ​ Basic Usage Copy Ask AI # Configure service tts = ElevenLabsTTSService( api_key = "your-api-key" , voice_id = "voice-id" , sample_rate = 24000 , params = ElevenLabsTTSService.InputParams( language = Language. EN ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output() ]) ​ With Voice Settings Copy Ask AI # Configure with voice customization tts = ElevenLabsTTSService( api_key = "your-api-key" , voice_id = "voice-id" , params = ElevenLabsTTSService.InputParams( stability = 0.7 , similarity_boost = 0.8 , style = 0.5 , use_speaker_boost = True ) ) ​ Methods See the TTS base class methods for additional functionality. ​ Language Support ElevenLabs supports the following languages and their variants: Language Code Description Service Code Language.AR Arabic ar Language.BG Bulgarian bg Language.CS Czech cs Language.DA Danish da Language.DE German de Language.EL Greek el Language.EN English en Language.ES Spanish es Language.FI Finnish fi Language.FIL Filipino fil Language.FR French fr Language.HI Hindi hi Language.HR Croatian hr Language.HU Hungarian hu Language.ID Indonesian id Language.IT Italian it Language.JA Japanese ja Language.KO Korean ko Language.MS Malay ms Language.NL Dutch nl Language.NO Norwegian no Language.PL Polish pl Language.PT Portuguese pt Language.RO Romanian ro Language.RU Russian ru Language.SK Slovak sk Language.SV Swedish sv Language.TA Tamil ta Language.TR Turkish tr Language.UK Ukrainian uk Language.VI Vietnamese vi Language.ZH Chinese zh Note: Language support may vary based on the selected model. See the ElevenLabs docs for more details. ​ Usage Example Copy Ask AI # Configure service with specific language service = ElevenLabsTTSService( api_key = "your-api-key" , voice_id = "voice-id" , params = ElevenLabsTTSService.InputParams( language = Language. FR # French ) ) ​ Frame Flow ​ Notes WebSocket implementation includes a 10-second keepalive mechanism Sample rate must be one of: 16000, 22050, 24000, or 44100 Hz Voice settings require both stability and similarity_boost to be set The language parameter only works with multilingual models WebSocket implementation pauses frame processing during speech generation HTTP implementation requires an external aiohttp ClientSession Deepgram Fish Audio On this page Overview Installation ElevenLabsTTSService (WebSocket) Configuration InputParams ElevenLabsHttpTTSService (HTTP) Configuration Output Frames TTSStartedFrame TTSAudioRawFrame TTSStoppedFrame ErrorFrame (HTTP implementation) Usage Examples Basic Usage With Voice Settings Methods Language Support Usage Example Frame Flow Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_elevenlabs_dd94e4db.txt b/tts_elevenlabs_dd94e4db.txt
new file mode 100644
index 0000000000000000000000000000000000000000..4fce01edba6488eb1b3f5b8a6d437ba4c3ab5fd8
--- /dev/null
+++ b/tts_elevenlabs_dd94e4db.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/elevenlabs#param-api-key-1
+Title: ElevenLabs - Pipecat
+==================================================
+
+ElevenLabs - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech ElevenLabs Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview ElevenLabs TTS provides high-quality text-to-speech synthesis through two service implementations: ElevenLabsTTSService : WebSocket-based implementation with word-level timing and interruption support ElevenLabsHttpTTSService : HTTP-based implementation for simpler use cases ​ Installation To use ElevenLabsTTSService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[elevenlabs]" You’ll also need to set up your ElevenLabs API key as an environment variable: ELEVENLABS_API_KEY . You can obtain a ElevenLabs API key by signing up at ElevenLabs . ​ ElevenLabsTTSService (WebSocket) ​ Configuration ​ api_key str required ElevenLabs API key ​ voice_id str required Voice identifier ​ model str default: "eleven_flash_v2_5" Model identifier ​ url str default: "wss://api.elevenlabs.io" API endpoint URL ​ sample_rate int default: "None" Output audio sample rate in Hz ​ params InputParams default: "InputParams()" Additional configuration parameters ​ text_filter BaseTextFilter default: "None" Modifies text provided to the TTS. Learn more about the available filters. ​ InputParams ​ language Language default: "None" The language of the text to be synthesized ​ optimize_streaming_latency str default: "None" Optimization level for streaming latency ​ stability float default: "None" Defines the stability for voice settings ​ similarity_boost float default: "None" Defines the similarity boost for voice settings ​ style float default: "None" Defines the style for voice settings. Available on V2+ models ​ use_speaker_boost bool default: "None" Defines whether to use speaker boost for voice settings. Available on V2+ models ​ speed float default: "None" Speech rate multiplier. Higher values increase speech speed ​ auto_mode bool default: "True" This parameter focuses on reducing the latency by disabling the chunk schedule and buffers. Recommended when sending full sentences or phrases ​ ElevenLabsHttpTTSService (HTTP) ​ Configuration ​ api_key str required ElevenLabs API key ​ voice_id str required Voice identifier ​ aiohttp_session aiohttp.ClientSession required aiohttp ClientSession for HTTP requests ​ model str default: "eleven_flash_v2_5" Model identifier ​ base_url str default: "https://api.elevenlabs.io" API base URL ​ sample_rate int default: "None" Output audio sample rate in Hz ​ params InputParams default: "InputParams()" Additional configuration parameters (similar to WebSocket implementation) ​ Output Frames ​ TTSStartedFrame Signals the start of audio generation. ​ TTSAudioRawFrame Contains generated audio data: ​ audio bytes Raw audio data chunk ​ sample_rate int Audio sample rate ​ num_channels int Number of audio channels (1 for mono) ​ TTSStoppedFrame Signals the completion of audio generation. ​ ErrorFrame (HTTP implementation) Sent when an error occurs during HTTP TTS generation: ​ error str Error message describing what went wrong ​ Usage Examples ​ Basic Usage Copy Ask AI # Configure service tts = ElevenLabsTTSService( api_key = "your-api-key" , voice_id = "voice-id" , sample_rate = 24000 , params = ElevenLabsTTSService.InputParams( language = Language. EN ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output() ]) ​ With Voice Settings Copy Ask AI # Configure with voice customization tts = ElevenLabsTTSService( api_key = "your-api-key" , voice_id = "voice-id" , params = ElevenLabsTTSService.InputParams( stability = 0.7 , similarity_boost = 0.8 , style = 0.5 , use_speaker_boost = True ) ) ​ Methods See the TTS base class methods for additional functionality. ​ Language Support ElevenLabs supports the following languages and their variants: Language Code Description Service Code Language.AR Arabic ar Language.BG Bulgarian bg Language.CS Czech cs Language.DA Danish da Language.DE German de Language.EL Greek el Language.EN English en Language.ES Spanish es Language.FI Finnish fi Language.FIL Filipino fil Language.FR French fr Language.HI Hindi hi Language.HR Croatian hr Language.HU Hungarian hu Language.ID Indonesian id Language.IT Italian it Language.JA Japanese ja Language.KO Korean ko Language.MS Malay ms Language.NL Dutch nl Language.NO Norwegian no Language.PL Polish pl Language.PT Portuguese pt Language.RO Romanian ro Language.RU Russian ru Language.SK Slovak sk Language.SV Swedish sv Language.TA Tamil ta Language.TR Turkish tr Language.UK Ukrainian uk Language.VI Vietnamese vi Language.ZH Chinese zh Note: Language support may vary based on the selected model. See the ElevenLabs docs for more details. ​ Usage Example Copy Ask AI # Configure service with specific language service = ElevenLabsTTSService( api_key = "your-api-key" , voice_id = "voice-id" , params = ElevenLabsTTSService.InputParams( language = Language. FR # French ) ) ​ Frame Flow ​ Notes WebSocket implementation includes a 10-second keepalive mechanism Sample rate must be one of: 16000, 22050, 24000, or 44100 Hz Voice settings require both stability and similarity_boost to be set The language parameter only works with multilingual models WebSocket implementation pauses frame processing during speech generation HTTP implementation requires an external aiohttp ClientSession Deepgram Fish Audio On this page Overview Installation ElevenLabsTTSService (WebSocket) Configuration InputParams ElevenLabsHttpTTSService (HTTP) Configuration Output Frames TTSStartedFrame TTSAudioRawFrame TTSStoppedFrame ErrorFrame (HTTP implementation) Usage Examples Basic Usage With Voice Settings Methods Language Support Usage Example Frame Flow Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_elevenlabs_f482b965.txt b/tts_elevenlabs_f482b965.txt
new file mode 100644
index 0000000000000000000000000000000000000000..5aea09d496f2b3812f1f76f4dedb320e87569abb
--- /dev/null
+++ b/tts_elevenlabs_f482b965.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/elevenlabs#param-use-speaker-boost
+Title: ElevenLabs - Pipecat
+==================================================
+
+ElevenLabs - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech ElevenLabs Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview ElevenLabs TTS provides high-quality text-to-speech synthesis through two service implementations: ElevenLabsTTSService : WebSocket-based implementation with word-level timing and interruption support ElevenLabsHttpTTSService : HTTP-based implementation for simpler use cases ​ Installation To use ElevenLabsTTSService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[elevenlabs]" You’ll also need to set up your ElevenLabs API key as an environment variable: ELEVENLABS_API_KEY . You can obtain a ElevenLabs API key by signing up at ElevenLabs . ​ ElevenLabsTTSService (WebSocket) ​ Configuration ​ api_key str required ElevenLabs API key ​ voice_id str required Voice identifier ​ model str default: "eleven_flash_v2_5" Model identifier ​ url str default: "wss://api.elevenlabs.io" API endpoint URL ​ sample_rate int default: "None" Output audio sample rate in Hz ​ params InputParams default: "InputParams()" Additional configuration parameters ​ text_filter BaseTextFilter default: "None" Modifies text provided to the TTS. Learn more about the available filters. ​ InputParams ​ language Language default: "None" The language of the text to be synthesized ​ optimize_streaming_latency str default: "None" Optimization level for streaming latency ​ stability float default: "None" Defines the stability for voice settings ​ similarity_boost float default: "None" Defines the similarity boost for voice settings ​ style float default: "None" Defines the style for voice settings. Available on V2+ models ​ use_speaker_boost bool default: "None" Defines whether to use speaker boost for voice settings. Available on V2+ models ​ speed float default: "None" Speech rate multiplier. Higher values increase speech speed ​ auto_mode bool default: "True" This parameter focuses on reducing the latency by disabling the chunk schedule and buffers. Recommended when sending full sentences or phrases ​ ElevenLabsHttpTTSService (HTTP) ​ Configuration ​ api_key str required ElevenLabs API key ​ voice_id str required Voice identifier ​ aiohttp_session aiohttp.ClientSession required aiohttp ClientSession for HTTP requests ​ model str default: "eleven_flash_v2_5" Model identifier ​ base_url str default: "https://api.elevenlabs.io" API base URL ​ sample_rate int default: "None" Output audio sample rate in Hz ​ params InputParams default: "InputParams()" Additional configuration parameters (similar to WebSocket implementation) ​ Output Frames ​ TTSStartedFrame Signals the start of audio generation. ​ TTSAudioRawFrame Contains generated audio data: ​ audio bytes Raw audio data chunk ​ sample_rate int Audio sample rate ​ num_channels int Number of audio channels (1 for mono) ​ TTSStoppedFrame Signals the completion of audio generation. ​ ErrorFrame (HTTP implementation) Sent when an error occurs during HTTP TTS generation: ​ error str Error message describing what went wrong ​ Usage Examples ​ Basic Usage Copy Ask AI # Configure service tts = ElevenLabsTTSService( api_key = "your-api-key" , voice_id = "voice-id" , sample_rate = 24000 , params = ElevenLabsTTSService.InputParams( language = Language. EN ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output() ]) ​ With Voice Settings Copy Ask AI # Configure with voice customization tts = ElevenLabsTTSService( api_key = "your-api-key" , voice_id = "voice-id" , params = ElevenLabsTTSService.InputParams( stability = 0.7 , similarity_boost = 0.8 , style = 0.5 , use_speaker_boost = True ) ) ​ Methods See the TTS base class methods for additional functionality. ​ Language Support ElevenLabs supports the following languages and their variants: Language Code Description Service Code Language.AR Arabic ar Language.BG Bulgarian bg Language.CS Czech cs Language.DA Danish da Language.DE German de Language.EL Greek el Language.EN English en Language.ES Spanish es Language.FI Finnish fi Language.FIL Filipino fil Language.FR French fr Language.HI Hindi hi Language.HR Croatian hr Language.HU Hungarian hu Language.ID Indonesian id Language.IT Italian it Language.JA Japanese ja Language.KO Korean ko Language.MS Malay ms Language.NL Dutch nl Language.NO Norwegian no Language.PL Polish pl Language.PT Portuguese pt Language.RO Romanian ro Language.RU Russian ru Language.SK Slovak sk Language.SV Swedish sv Language.TA Tamil ta Language.TR Turkish tr Language.UK Ukrainian uk Language.VI Vietnamese vi Language.ZH Chinese zh Note: Language support may vary based on the selected model. See the ElevenLabs docs for more details. ​ Usage Example Copy Ask AI # Configure service with specific language service = ElevenLabsTTSService( api_key = "your-api-key" , voice_id = "voice-id" , params = ElevenLabsTTSService.InputParams( language = Language. FR # French ) ) ​ Frame Flow ​ Notes WebSocket implementation includes a 10-second keepalive mechanism Sample rate must be one of: 16000, 22050, 24000, or 44100 Hz Voice settings require both stability and similarity_boost to be set The language parameter only works with multilingual models WebSocket implementation pauses frame processing during speech generation HTTP implementation requires an external aiohttp ClientSession Deepgram Fish Audio On this page Overview Installation ElevenLabsTTSService (WebSocket) Configuration InputParams ElevenLabsHttpTTSService (HTTP) Configuration Output Frames TTSStartedFrame TTSAudioRawFrame TTSStoppedFrame ErrorFrame (HTTP implementation) Usage Examples Basic Usage With Voice Settings Methods Language Support Usage Example Frame Flow Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_fish_0a201790.txt b/tts_fish_0a201790.txt
new file mode 100644
index 0000000000000000000000000000000000000000..152637ffecf1b85ac15c13a8be6cf590458ef87f
--- /dev/null
+++ b/tts_fish_0a201790.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/fish#overview
+Title: Fish Audio - Pipecat
+==================================================
+
+Fish Audio - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech Fish Audio Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview The FishAudioTTSService provides real-time text-to-speech synthesis using Fish Audio’s WebSocket API. It supports streaming audio output, multiple voices, and various audio formats. ​ Installation To use Fish Audio, install the required dependencies: Copy Ask AI pip install "pipecat-ai[fish]" You’ll need to set up your Fish Audio API key as an environment variable: FISH_API_KEY . ​ Constructor Parameters ​ api_key str required Fish Audio API key ​ model str required Reference ID for the voice model ​ output_format str default: "pcm" Audio output format. Options: “opus”, “mp3”, “pcm”, “wav” ​ sample_rate int default: "None" Output audio sample rate in Hz ​ Basic Usage Copy Ask AI tts = FishAudioTTSService( api_key = os.getenv( "FISH_API_KEY" ), model = "your-model-id" , # Get this from Fish Audio playground output_format = "pcm" , # Choose output format sample_rate = 24000 , # Set sample rate params = FishAudioTTSService.InputParams( latency = "normal" , prosody_speed = 1.0 ) ) ​ Input Parameters ​ language Language default: "Language.EN" Language for speech synthesis. See Language Support section for available options. ​ latency str default: "normal" Latency mode for synthesis. Options: “normal” or “balanced” ​ prosody_speed float default: "1.0" Speech speed adjustment. Range: 0.5 to 2.0 ​ prosody_volume int default: "0" Volume adjustment in decibels (dB) Copy Ask AI tts = FishAudioTTSService( api_key = os.getenv( "FISH_API_KEY" ), model = "your-model-id" , params = InputParams( language = Language. EN , latency = "normal" , # Balance between quality and speed prosody_speed = 1.2 , # Slightly faster speech prosody_volume = 0 # Default volume ) ) ​ Output Frames ​ Control Frames ​ TTSStartedFrame Frame Signals start of synthesis ​ TTSStoppedFrame Frame Signals completion of synthesis ​ Audio Frames ​ TTSAudioRawFrame Frame Contains generated audio data with: - Specified format (PCM, WAV, MP3, or Opus) - Configured sample rate - Single channel (mono) ​ Error Frames ​ ErrorFrame Frame Contains Fish Audio error information ​ Language Support Supports multiple languages through the Language enum: Language Code Service Code Language.EN en-US Language.ZH zh-CN ​ Usage Example Copy Ask AI from pipecat.services.fish.tts import FishAudioTTSService from pipecat.transcriptions.language import Language # Configure service tts = FishAudioTTSService( api_key = os.getenv( "FISH_API_KEY" ), model = "e58b0d7efca34eb38d5c4985e378abcb" , # Example model ID output_format = "pcm" , params = FishAudioTTSService.InputParams( language = Language. EN , latency = "normal" , prosody_speed = 1.0 , prosody_volume = 0 ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ Frame Flow ​ Metrics Support The service collects processing metrics: Time to First Byte (TTFB) Processing duration Character usage API calls ElevenLabs Google On this page Overview Installation Constructor Parameters Basic Usage Input Parameters Output Frames Control Frames Audio Frames Error Frames Language Support Usage Example Frame Flow Metrics Support Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_fish_17e8b0f5.txt b/tts_fish_17e8b0f5.txt
new file mode 100644
index 0000000000000000000000000000000000000000..402167065790dc7bb512a9d87ebf3336daa700d7
--- /dev/null
+++ b/tts_fish_17e8b0f5.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/fish#param-tts-started-frame
+Title: Fish Audio - Pipecat
+==================================================
+
+Fish Audio - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech Fish Audio Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview The FishAudioTTSService provides real-time text-to-speech synthesis using Fish Audio’s WebSocket API. It supports streaming audio output, multiple voices, and various audio formats. ​ Installation To use Fish Audio, install the required dependencies: Copy Ask AI pip install "pipecat-ai[fish]" You’ll need to set up your Fish Audio API key as an environment variable: FISH_API_KEY . ​ Constructor Parameters ​ api_key str required Fish Audio API key ​ model str required Reference ID for the voice model ​ output_format str default: "pcm" Audio output format. Options: “opus”, “mp3”, “pcm”, “wav” ​ sample_rate int default: "None" Output audio sample rate in Hz ​ Basic Usage Copy Ask AI tts = FishAudioTTSService( api_key = os.getenv( "FISH_API_KEY" ), model = "your-model-id" , # Get this from Fish Audio playground output_format = "pcm" , # Choose output format sample_rate = 24000 , # Set sample rate params = FishAudioTTSService.InputParams( latency = "normal" , prosody_speed = 1.0 ) ) ​ Input Parameters ​ language Language default: "Language.EN" Language for speech synthesis. See Language Support section for available options. ​ latency str default: "normal" Latency mode for synthesis. Options: “normal” or “balanced” ​ prosody_speed float default: "1.0" Speech speed adjustment. Range: 0.5 to 2.0 ​ prosody_volume int default: "0" Volume adjustment in decibels (dB) Copy Ask AI tts = FishAudioTTSService( api_key = os.getenv( "FISH_API_KEY" ), model = "your-model-id" , params = InputParams( language = Language. EN , latency = "normal" , # Balance between quality and speed prosody_speed = 1.2 , # Slightly faster speech prosody_volume = 0 # Default volume ) ) ​ Output Frames ​ Control Frames ​ TTSStartedFrame Frame Signals start of synthesis ​ TTSStoppedFrame Frame Signals completion of synthesis ​ Audio Frames ​ TTSAudioRawFrame Frame Contains generated audio data with: - Specified format (PCM, WAV, MP3, or Opus) - Configured sample rate - Single channel (mono) ​ Error Frames ​ ErrorFrame Frame Contains Fish Audio error information ​ Language Support Supports multiple languages through the Language enum: Language Code Service Code Language.EN en-US Language.ZH zh-CN ​ Usage Example Copy Ask AI from pipecat.services.fish.tts import FishAudioTTSService from pipecat.transcriptions.language import Language # Configure service tts = FishAudioTTSService( api_key = os.getenv( "FISH_API_KEY" ), model = "e58b0d7efca34eb38d5c4985e378abcb" , # Example model ID output_format = "pcm" , params = FishAudioTTSService.InputParams( language = Language. EN , latency = "normal" , prosody_speed = 1.0 , prosody_volume = 0 ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ Frame Flow ​ Metrics Support The service collects processing metrics: Time to First Byte (TTFB) Processing duration Character usage API calls ElevenLabs Google On this page Overview Installation Constructor Parameters Basic Usage Input Parameters Output Frames Control Frames Audio Frames Error Frames Language Support Usage Example Frame Flow Metrics Support Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_fish_469e44e0.txt b/tts_fish_469e44e0.txt
new file mode 100644
index 0000000000000000000000000000000000000000..5bbb24a188e35ef90ed3bf99d7e85835e7779e99
--- /dev/null
+++ b/tts_fish_469e44e0.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/fish#input-parameters
+Title: Fish Audio - Pipecat
+==================================================
+
+Fish Audio - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech Fish Audio Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview The FishAudioTTSService provides real-time text-to-speech synthesis using Fish Audio’s WebSocket API. It supports streaming audio output, multiple voices, and various audio formats. ​ Installation To use Fish Audio, install the required dependencies: Copy Ask AI pip install "pipecat-ai[fish]" You’ll need to set up your Fish Audio API key as an environment variable: FISH_API_KEY . ​ Constructor Parameters ​ api_key str required Fish Audio API key ​ model str required Reference ID for the voice model ​ output_format str default: "pcm" Audio output format. Options: “opus”, “mp3”, “pcm”, “wav” ​ sample_rate int default: "None" Output audio sample rate in Hz ​ Basic Usage Copy Ask AI tts = FishAudioTTSService( api_key = os.getenv( "FISH_API_KEY" ), model = "your-model-id" , # Get this from Fish Audio playground output_format = "pcm" , # Choose output format sample_rate = 24000 , # Set sample rate params = FishAudioTTSService.InputParams( latency = "normal" , prosody_speed = 1.0 ) ) ​ Input Parameters ​ language Language default: "Language.EN" Language for speech synthesis. See Language Support section for available options. ​ latency str default: "normal" Latency mode for synthesis. Options: “normal” or “balanced” ​ prosody_speed float default: "1.0" Speech speed adjustment. Range: 0.5 to 2.0 ​ prosody_volume int default: "0" Volume adjustment in decibels (dB) Copy Ask AI tts = FishAudioTTSService( api_key = os.getenv( "FISH_API_KEY" ), model = "your-model-id" , params = InputParams( language = Language. EN , latency = "normal" , # Balance between quality and speed prosody_speed = 1.2 , # Slightly faster speech prosody_volume = 0 # Default volume ) ) ​ Output Frames ​ Control Frames ​ TTSStartedFrame Frame Signals start of synthesis ​ TTSStoppedFrame Frame Signals completion of synthesis ​ Audio Frames ​ TTSAudioRawFrame Frame Contains generated audio data with: - Specified format (PCM, WAV, MP3, or Opus) - Configured sample rate - Single channel (mono) ​ Error Frames ​ ErrorFrame Frame Contains Fish Audio error information ​ Language Support Supports multiple languages through the Language enum: Language Code Service Code Language.EN en-US Language.ZH zh-CN ​ Usage Example Copy Ask AI from pipecat.services.fish.tts import FishAudioTTSService from pipecat.transcriptions.language import Language # Configure service tts = FishAudioTTSService( api_key = os.getenv( "FISH_API_KEY" ), model = "e58b0d7efca34eb38d5c4985e378abcb" , # Example model ID output_format = "pcm" , params = FishAudioTTSService.InputParams( language = Language. EN , latency = "normal" , prosody_speed = 1.0 , prosody_volume = 0 ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ Frame Flow ​ Metrics Support The service collects processing metrics: Time to First Byte (TTFB) Processing duration Character usage API calls ElevenLabs Google On this page Overview Installation Constructor Parameters Basic Usage Input Parameters Output Frames Control Frames Audio Frames Error Frames Language Support Usage Example Frame Flow Metrics Support Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_fish_4d4056c5.txt b/tts_fish_4d4056c5.txt
new file mode 100644
index 0000000000000000000000000000000000000000..e378d7e6146179dfcaf1513b11df931caefe0159
--- /dev/null
+++ b/tts_fish_4d4056c5.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/fish#param-language
+Title: Fish Audio - Pipecat
+==================================================
+
+Fish Audio - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech Fish Audio Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview The FishAudioTTSService provides real-time text-to-speech synthesis using Fish Audio’s WebSocket API. It supports streaming audio output, multiple voices, and various audio formats. ​ Installation To use Fish Audio, install the required dependencies: Copy Ask AI pip install "pipecat-ai[fish]" You’ll need to set up your Fish Audio API key as an environment variable: FISH_API_KEY . ​ Constructor Parameters ​ api_key str required Fish Audio API key ​ model str required Reference ID for the voice model ​ output_format str default: "pcm" Audio output format. Options: “opus”, “mp3”, “pcm”, “wav” ​ sample_rate int default: "None" Output audio sample rate in Hz ​ Basic Usage Copy Ask AI tts = FishAudioTTSService( api_key = os.getenv( "FISH_API_KEY" ), model = "your-model-id" , # Get this from Fish Audio playground output_format = "pcm" , # Choose output format sample_rate = 24000 , # Set sample rate params = FishAudioTTSService.InputParams( latency = "normal" , prosody_speed = 1.0 ) ) ​ Input Parameters ​ language Language default: "Language.EN" Language for speech synthesis. See Language Support section for available options. ​ latency str default: "normal" Latency mode for synthesis. Options: “normal” or “balanced” ​ prosody_speed float default: "1.0" Speech speed adjustment. Range: 0.5 to 2.0 ​ prosody_volume int default: "0" Volume adjustment in decibels (dB) Copy Ask AI tts = FishAudioTTSService( api_key = os.getenv( "FISH_API_KEY" ), model = "your-model-id" , params = InputParams( language = Language. EN , latency = "normal" , # Balance between quality and speed prosody_speed = 1.2 , # Slightly faster speech prosody_volume = 0 # Default volume ) ) ​ Output Frames ​ Control Frames ​ TTSStartedFrame Frame Signals start of synthesis ​ TTSStoppedFrame Frame Signals completion of synthesis ​ Audio Frames ​ TTSAudioRawFrame Frame Contains generated audio data with: - Specified format (PCM, WAV, MP3, or Opus) - Configured sample rate - Single channel (mono) ​ Error Frames ​ ErrorFrame Frame Contains Fish Audio error information ​ Language Support Supports multiple languages through the Language enum: Language Code Service Code Language.EN en-US Language.ZH zh-CN ​ Usage Example Copy Ask AI from pipecat.services.fish.tts import FishAudioTTSService from pipecat.transcriptions.language import Language # Configure service tts = FishAudioTTSService( api_key = os.getenv( "FISH_API_KEY" ), model = "e58b0d7efca34eb38d5c4985e378abcb" , # Example model ID output_format = "pcm" , params = FishAudioTTSService.InputParams( language = Language. EN , latency = "normal" , prosody_speed = 1.0 , prosody_volume = 0 ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ Frame Flow ​ Metrics Support The service collects processing metrics: Time to First Byte (TTFB) Processing duration Character usage API calls ElevenLabs Google On this page Overview Installation Constructor Parameters Basic Usage Input Parameters Output Frames Control Frames Audio Frames Error Frames Language Support Usage Example Frame Flow Metrics Support Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_fish_79e62a87.txt b/tts_fish_79e62a87.txt
new file mode 100644
index 0000000000000000000000000000000000000000..008d598432c28f062c0e47814d5b61a16e162fdb
--- /dev/null
+++ b/tts_fish_79e62a87.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/fish#constructor-parameters
+Title: Fish Audio - Pipecat
+==================================================
+
+Fish Audio - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech Fish Audio Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview The FishAudioTTSService provides real-time text-to-speech synthesis using Fish Audio’s WebSocket API. It supports streaming audio output, multiple voices, and various audio formats. ​ Installation To use Fish Audio, install the required dependencies: Copy Ask AI pip install "pipecat-ai[fish]" You’ll need to set up your Fish Audio API key as an environment variable: FISH_API_KEY . ​ Constructor Parameters ​ api_key str required Fish Audio API key ​ model str required Reference ID for the voice model ​ output_format str default: "pcm" Audio output format. Options: “opus”, “mp3”, “pcm”, “wav” ​ sample_rate int default: "None" Output audio sample rate in Hz ​ Basic Usage Copy Ask AI tts = FishAudioTTSService( api_key = os.getenv( "FISH_API_KEY" ), model = "your-model-id" , # Get this from Fish Audio playground output_format = "pcm" , # Choose output format sample_rate = 24000 , # Set sample rate params = FishAudioTTSService.InputParams( latency = "normal" , prosody_speed = 1.0 ) ) ​ Input Parameters ​ language Language default: "Language.EN" Language for speech synthesis. See Language Support section for available options. ​ latency str default: "normal" Latency mode for synthesis. Options: “normal” or “balanced” ​ prosody_speed float default: "1.0" Speech speed adjustment. Range: 0.5 to 2.0 ​ prosody_volume int default: "0" Volume adjustment in decibels (dB) Copy Ask AI tts = FishAudioTTSService( api_key = os.getenv( "FISH_API_KEY" ), model = "your-model-id" , params = InputParams( language = Language. EN , latency = "normal" , # Balance between quality and speed prosody_speed = 1.2 , # Slightly faster speech prosody_volume = 0 # Default volume ) ) ​ Output Frames ​ Control Frames ​ TTSStartedFrame Frame Signals start of synthesis ​ TTSStoppedFrame Frame Signals completion of synthesis ​ Audio Frames ​ TTSAudioRawFrame Frame Contains generated audio data with: - Specified format (PCM, WAV, MP3, or Opus) - Configured sample rate - Single channel (mono) ​ Error Frames ​ ErrorFrame Frame Contains Fish Audio error information ​ Language Support Supports multiple languages through the Language enum: Language Code Service Code Language.EN en-US Language.ZH zh-CN ​ Usage Example Copy Ask AI from pipecat.services.fish.tts import FishAudioTTSService from pipecat.transcriptions.language import Language # Configure service tts = FishAudioTTSService( api_key = os.getenv( "FISH_API_KEY" ), model = "e58b0d7efca34eb38d5c4985e378abcb" , # Example model ID output_format = "pcm" , params = FishAudioTTSService.InputParams( language = Language. EN , latency = "normal" , prosody_speed = 1.0 , prosody_volume = 0 ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ Frame Flow ​ Metrics Support The service collects processing metrics: Time to First Byte (TTFB) Processing duration Character usage API calls ElevenLabs Google On this page Overview Installation Constructor Parameters Basic Usage Input Parameters Output Frames Control Frames Audio Frames Error Frames Language Support Usage Example Frame Flow Metrics Support Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_fish_7cac0901.txt b/tts_fish_7cac0901.txt
new file mode 100644
index 0000000000000000000000000000000000000000..95c3f49776115d421945a231cb9eefd68665294c
--- /dev/null
+++ b/tts_fish_7cac0901.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/fish#param-error-frame
+Title: Fish Audio - Pipecat
+==================================================
+
+Fish Audio - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech Fish Audio Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview The FishAudioTTSService provides real-time text-to-speech synthesis using Fish Audio’s WebSocket API. It supports streaming audio output, multiple voices, and various audio formats. ​ Installation To use Fish Audio, install the required dependencies: Copy Ask AI pip install "pipecat-ai[fish]" You’ll need to set up your Fish Audio API key as an environment variable: FISH_API_KEY . ​ Constructor Parameters ​ api_key str required Fish Audio API key ​ model str required Reference ID for the voice model ​ output_format str default: "pcm" Audio output format. Options: “opus”, “mp3”, “pcm”, “wav” ​ sample_rate int default: "None" Output audio sample rate in Hz ​ Basic Usage Copy Ask AI tts = FishAudioTTSService( api_key = os.getenv( "FISH_API_KEY" ), model = "your-model-id" , # Get this from Fish Audio playground output_format = "pcm" , # Choose output format sample_rate = 24000 , # Set sample rate params = FishAudioTTSService.InputParams( latency = "normal" , prosody_speed = 1.0 ) ) ​ Input Parameters ​ language Language default: "Language.EN" Language for speech synthesis. See Language Support section for available options. ​ latency str default: "normal" Latency mode for synthesis. Options: “normal” or “balanced” ​ prosody_speed float default: "1.0" Speech speed adjustment. Range: 0.5 to 2.0 ​ prosody_volume int default: "0" Volume adjustment in decibels (dB) Copy Ask AI tts = FishAudioTTSService( api_key = os.getenv( "FISH_API_KEY" ), model = "your-model-id" , params = InputParams( language = Language. EN , latency = "normal" , # Balance between quality and speed prosody_speed = 1.2 , # Slightly faster speech prosody_volume = 0 # Default volume ) ) ​ Output Frames ​ Control Frames ​ TTSStartedFrame Frame Signals start of synthesis ​ TTSStoppedFrame Frame Signals completion of synthesis ​ Audio Frames ​ TTSAudioRawFrame Frame Contains generated audio data with: - Specified format (PCM, WAV, MP3, or Opus) - Configured sample rate - Single channel (mono) ​ Error Frames ​ ErrorFrame Frame Contains Fish Audio error information ​ Language Support Supports multiple languages through the Language enum: Language Code Service Code Language.EN en-US Language.ZH zh-CN ​ Usage Example Copy Ask AI from pipecat.services.fish.tts import FishAudioTTSService from pipecat.transcriptions.language import Language # Configure service tts = FishAudioTTSService( api_key = os.getenv( "FISH_API_KEY" ), model = "e58b0d7efca34eb38d5c4985e378abcb" , # Example model ID output_format = "pcm" , params = FishAudioTTSService.InputParams( language = Language. EN , latency = "normal" , prosody_speed = 1.0 , prosody_volume = 0 ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ Frame Flow ​ Metrics Support The service collects processing metrics: Time to First Byte (TTFB) Processing duration Character usage API calls ElevenLabs Google On this page Overview Installation Constructor Parameters Basic Usage Input Parameters Output Frames Control Frames Audio Frames Error Frames Language Support Usage Example Frame Flow Metrics Support Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_fish_b55a5a25.txt b/tts_fish_b55a5a25.txt
new file mode 100644
index 0000000000000000000000000000000000000000..68bc4e3adc61a02d2da30310b5d95680845de314
--- /dev/null
+++ b/tts_fish_b55a5a25.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/fish#param-api-key
+Title: Fish Audio - Pipecat
+==================================================
+
+Fish Audio - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech Fish Audio Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview The FishAudioTTSService provides real-time text-to-speech synthesis using Fish Audio’s WebSocket API. It supports streaming audio output, multiple voices, and various audio formats. ​ Installation To use Fish Audio, install the required dependencies: Copy Ask AI pip install "pipecat-ai[fish]" You’ll need to set up your Fish Audio API key as an environment variable: FISH_API_KEY . ​ Constructor Parameters ​ api_key str required Fish Audio API key ​ model str required Reference ID for the voice model ​ output_format str default: "pcm" Audio output format. Options: “opus”, “mp3”, “pcm”, “wav” ​ sample_rate int default: "None" Output audio sample rate in Hz ​ Basic Usage Copy Ask AI tts = FishAudioTTSService( api_key = os.getenv( "FISH_API_KEY" ), model = "your-model-id" , # Get this from Fish Audio playground output_format = "pcm" , # Choose output format sample_rate = 24000 , # Set sample rate params = FishAudioTTSService.InputParams( latency = "normal" , prosody_speed = 1.0 ) ) ​ Input Parameters ​ language Language default: "Language.EN" Language for speech synthesis. See Language Support section for available options. ​ latency str default: "normal" Latency mode for synthesis. Options: “normal” or “balanced” ​ prosody_speed float default: "1.0" Speech speed adjustment. Range: 0.5 to 2.0 ​ prosody_volume int default: "0" Volume adjustment in decibels (dB) Copy Ask AI tts = FishAudioTTSService( api_key = os.getenv( "FISH_API_KEY" ), model = "your-model-id" , params = InputParams( language = Language. EN , latency = "normal" , # Balance between quality and speed prosody_speed = 1.2 , # Slightly faster speech prosody_volume = 0 # Default volume ) ) ​ Output Frames ​ Control Frames ​ TTSStartedFrame Frame Signals start of synthesis ​ TTSStoppedFrame Frame Signals completion of synthesis ​ Audio Frames ​ TTSAudioRawFrame Frame Contains generated audio data with: - Specified format (PCM, WAV, MP3, or Opus) - Configured sample rate - Single channel (mono) ​ Error Frames ​ ErrorFrame Frame Contains Fish Audio error information ​ Language Support Supports multiple languages through the Language enum: Language Code Service Code Language.EN en-US Language.ZH zh-CN ​ Usage Example Copy Ask AI from pipecat.services.fish.tts import FishAudioTTSService from pipecat.transcriptions.language import Language # Configure service tts = FishAudioTTSService( api_key = os.getenv( "FISH_API_KEY" ), model = "e58b0d7efca34eb38d5c4985e378abcb" , # Example model ID output_format = "pcm" , params = FishAudioTTSService.InputParams( language = Language. EN , latency = "normal" , prosody_speed = 1.0 , prosody_volume = 0 ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ Frame Flow ​ Metrics Support The service collects processing metrics: Time to First Byte (TTFB) Processing duration Character usage API calls ElevenLabs Google On this page Overview Installation Constructor Parameters Basic Usage Input Parameters Output Frames Control Frames Audio Frames Error Frames Language Support Usage Example Frame Flow Metrics Support Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_fish_b5c8635b.txt b/tts_fish_b5c8635b.txt
new file mode 100644
index 0000000000000000000000000000000000000000..b6249984cd207c43802ea4de32564ed4044a59cc
--- /dev/null
+++ b/tts_fish_b5c8635b.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/fish#output-frames
+Title: Fish Audio - Pipecat
+==================================================
+
+Fish Audio - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech Fish Audio Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview The FishAudioTTSService provides real-time text-to-speech synthesis using Fish Audio’s WebSocket API. It supports streaming audio output, multiple voices, and various audio formats. ​ Installation To use Fish Audio, install the required dependencies: Copy Ask AI pip install "pipecat-ai[fish]" You’ll need to set up your Fish Audio API key as an environment variable: FISH_API_KEY . ​ Constructor Parameters ​ api_key str required Fish Audio API key ​ model str required Reference ID for the voice model ​ output_format str default: "pcm" Audio output format. Options: “opus”, “mp3”, “pcm”, “wav” ​ sample_rate int default: "None" Output audio sample rate in Hz ​ Basic Usage Copy Ask AI tts = FishAudioTTSService( api_key = os.getenv( "FISH_API_KEY" ), model = "your-model-id" , # Get this from Fish Audio playground output_format = "pcm" , # Choose output format sample_rate = 24000 , # Set sample rate params = FishAudioTTSService.InputParams( latency = "normal" , prosody_speed = 1.0 ) ) ​ Input Parameters ​ language Language default: "Language.EN" Language for speech synthesis. See Language Support section for available options. ​ latency str default: "normal" Latency mode for synthesis. Options: “normal” or “balanced” ​ prosody_speed float default: "1.0" Speech speed adjustment. Range: 0.5 to 2.0 ​ prosody_volume int default: "0" Volume adjustment in decibels (dB) Copy Ask AI tts = FishAudioTTSService( api_key = os.getenv( "FISH_API_KEY" ), model = "your-model-id" , params = InputParams( language = Language. EN , latency = "normal" , # Balance between quality and speed prosody_speed = 1.2 , # Slightly faster speech prosody_volume = 0 # Default volume ) ) ​ Output Frames ​ Control Frames ​ TTSStartedFrame Frame Signals start of synthesis ​ TTSStoppedFrame Frame Signals completion of synthesis ​ Audio Frames ​ TTSAudioRawFrame Frame Contains generated audio data with: - Specified format (PCM, WAV, MP3, or Opus) - Configured sample rate - Single channel (mono) ​ Error Frames ​ ErrorFrame Frame Contains Fish Audio error information ​ Language Support Supports multiple languages through the Language enum: Language Code Service Code Language.EN en-US Language.ZH zh-CN ​ Usage Example Copy Ask AI from pipecat.services.fish.tts import FishAudioTTSService from pipecat.transcriptions.language import Language # Configure service tts = FishAudioTTSService( api_key = os.getenv( "FISH_API_KEY" ), model = "e58b0d7efca34eb38d5c4985e378abcb" , # Example model ID output_format = "pcm" , params = FishAudioTTSService.InputParams( language = Language. EN , latency = "normal" , prosody_speed = 1.0 , prosody_volume = 0 ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ Frame Flow ​ Metrics Support The service collects processing metrics: Time to First Byte (TTFB) Processing duration Character usage API calls ElevenLabs Google On this page Overview Installation Constructor Parameters Basic Usage Input Parameters Output Frames Control Frames Audio Frames Error Frames Language Support Usage Example Frame Flow Metrics Support Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_fish_bea7f7fa.txt b/tts_fish_bea7f7fa.txt
new file mode 100644
index 0000000000000000000000000000000000000000..f245d2be601381c5dc5cd2c28d7db6ca4ed9771f
--- /dev/null
+++ b/tts_fish_bea7f7fa.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/fish#metrics-support
+Title: Fish Audio - Pipecat
+==================================================
+
+Fish Audio - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech Fish Audio Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview The FishAudioTTSService provides real-time text-to-speech synthesis using Fish Audio’s WebSocket API. It supports streaming audio output, multiple voices, and various audio formats. ​ Installation To use Fish Audio, install the required dependencies: Copy Ask AI pip install "pipecat-ai[fish]" You’ll need to set up your Fish Audio API key as an environment variable: FISH_API_KEY . ​ Constructor Parameters ​ api_key str required Fish Audio API key ​ model str required Reference ID for the voice model ​ output_format str default: "pcm" Audio output format. Options: “opus”, “mp3”, “pcm”, “wav” ​ sample_rate int default: "None" Output audio sample rate in Hz ​ Basic Usage Copy Ask AI tts = FishAudioTTSService( api_key = os.getenv( "FISH_API_KEY" ), model = "your-model-id" , # Get this from Fish Audio playground output_format = "pcm" , # Choose output format sample_rate = 24000 , # Set sample rate params = FishAudioTTSService.InputParams( latency = "normal" , prosody_speed = 1.0 ) ) ​ Input Parameters ​ language Language default: "Language.EN" Language for speech synthesis. See Language Support section for available options. ​ latency str default: "normal" Latency mode for synthesis. Options: “normal” or “balanced” ​ prosody_speed float default: "1.0" Speech speed adjustment. Range: 0.5 to 2.0 ​ prosody_volume int default: "0" Volume adjustment in decibels (dB) Copy Ask AI tts = FishAudioTTSService( api_key = os.getenv( "FISH_API_KEY" ), model = "your-model-id" , params = InputParams( language = Language. EN , latency = "normal" , # Balance between quality and speed prosody_speed = 1.2 , # Slightly faster speech prosody_volume = 0 # Default volume ) ) ​ Output Frames ​ Control Frames ​ TTSStartedFrame Frame Signals start of synthesis ​ TTSStoppedFrame Frame Signals completion of synthesis ​ Audio Frames ​ TTSAudioRawFrame Frame Contains generated audio data with: - Specified format (PCM, WAV, MP3, or Opus) - Configured sample rate - Single channel (mono) ​ Error Frames ​ ErrorFrame Frame Contains Fish Audio error information ​ Language Support Supports multiple languages through the Language enum: Language Code Service Code Language.EN en-US Language.ZH zh-CN ​ Usage Example Copy Ask AI from pipecat.services.fish.tts import FishAudioTTSService from pipecat.transcriptions.language import Language # Configure service tts = FishAudioTTSService( api_key = os.getenv( "FISH_API_KEY" ), model = "e58b0d7efca34eb38d5c4985e378abcb" , # Example model ID output_format = "pcm" , params = FishAudioTTSService.InputParams( language = Language. EN , latency = "normal" , prosody_speed = 1.0 , prosody_volume = 0 ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ Frame Flow ​ Metrics Support The service collects processing metrics: Time to First Byte (TTFB) Processing duration Character usage API calls ElevenLabs Google On this page Overview Installation Constructor Parameters Basic Usage Input Parameters Output Frames Control Frames Audio Frames Error Frames Language Support Usage Example Frame Flow Metrics Support Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_google_030d4f72.txt b/tts_google_030d4f72.txt
new file mode 100644
index 0000000000000000000000000000000000000000..d8f0513f56ec4ef9d70546a98fc3998eb92f82a1
--- /dev/null
+++ b/tts_google_030d4f72.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/google#param-credentials
+Title: Google - Pipecat
+==================================================
+
+Google - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech Google Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview GoogleTTSService provides high-quality text-to-speech synthesis using Google Cloud’s Text-to-Speech API. It supports SSML for advanced voice control and multiple languages. ​ Installation To use GoogleTTSService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[google]" You’ll also need to set up Google Cloud credentials through either: Environment variable: GOOGLE_APPLICATION_CREDENTIALS Direct credentials JSON Credentials file path ​ Configuration ​ Constructor Parameters ​ credentials str | None Google Cloud credentials JSON string ​ credentials_path str | None Path to credentials JSON file ​ voice_id str default: "en-US-Neural2-A" Voice identifier ​ sample_rate int default: "None" Output audio sample rate in Hz ​ text_filter BaseTextFilter default: "None" Modifies text provided to the TTS. Learn more about the available filters. ​ Input Parameters Copy Ask AI class InputParams ( BaseModel ): pitch: Optional[ str ] rate: Optional[ str ] volume: Optional[ str ] emphasis: Optional[Literal[ "strong" , "moderate" , "reduced" , "none" ]] language: Optional[Language] = Language. EN gender: Optional[Literal[ "male" , "female" , "neutral" ]] google_style: Optional[Literal[ "apologetic" , "calm" , "empathetic" , "firm" , "lively" ]] ​ Output Frames ​ Control Frames ​ TTSStartedFrame Frame Signals start of synthesis ​ TTSStoppedFrame Frame Signals completion of synthesis ​ Audio Frames ​ TTSAudioRawFrame Frame Contains generated audio data: - PCM encoded audio - Configured sample rate - Mono channel ​ Error Frames ​ ErrorFrame Frame Contains error information ​ Usage Examples ​ Basic Usage Copy Ask AI # Configure service tts = GoogleTTSService( credentials_path = "path/to/credentials.json" , voice_id = "en-US-Neural2-A" , params = GoogleTTSService.InputParams( language = Language. EN , gender = "female" , google_style = "empathetic" ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ With SSML Controls Copy Ask AI # Configure with voice controls service = GoogleTTSService( credentials = credentials_json, params = GoogleTTSService.InputParams( pitch = "+2st" , rate = "1.2" , volume = "loud" , emphasis = "moderate" ) ) ​ Methods See the TTS base class methods for additional functionality. ​ Language Support Google Cloud Text-to-Speech supports the following languages and regional variants: Language Code Description Service Code Language.BG Bulgarian bg-BG Language.CA Catalan ca-ES Language.ZH Chinese (Mandarin) cmn-CN Language.ZH_TW Chinese (Taiwan) cmn-TW Language.CS Czech cs-CZ Language.DA Danish da-DK Language.NL Dutch (Netherlands) nl-NL Language.NL_BE Dutch (Belgium) nl-BE Language.EN English (US) en-US Language.EN_US English (US) en-US Language.EN_AU English (Australia) en-AU Language.EN_GB English (UK) en-GB Language.EN_IN English (India) en-IN Language.ET Estonian et-EE Language.FI Finnish fi-FI Language.FR French (France) fr-FR Language.FR_CA French (Canada) fr-CA Language.DE German de-DE Language.EL Greek el-GR Language.HI Hindi hi-IN Language.HU Hungarian hu-HU Language.ID Indonesian id-ID Language.IT Italian it-IT Language.JA Japanese ja-JP Language.KO Korean ko-KR Language.LV Latvian lv-LV Language.LT Lithuanian lt-LT Language.MS Malay ms-MY Language.NO Norwegian nb-NO Language.PL Polish pl-PL Language.PT Portuguese (Portugal) pt-PT Language.PT_BR Portuguese (Brazil) pt-BR Language.RO Romanian ro-RO Language.RU Russian ru-RU Language.SK Slovak sk-SK Language.ES Spanish es-ES Language.SV Swedish sv-SE Language.TH Thai th-TH Language.TR Turkish tr-TR Language.UK Ukrainian uk-UA Language.VI Vietnamese vi-VN ​ Usage Example Copy Ask AI # Configure service with specific language and region service = GoogleTTSService( credentials_path = "path/to/credentials.json" , voice_id = "en-US-Neural2-A" , params = GoogleTTSService.InputParams( language = Language. EN_GB , # British English gender = "female" ) ) ​ Regional Considerations Each language code includes both language and region (e.g., fr-FR for French in France) Some languages have multiple regional variants (e.g., English has US, UK, Australian, and Indian variants) Voice availability may vary by region Neural voices may not be available for all language/region combinations Note: Voice selection should match the specified language code for optimal results. ​ Frame Flow ​ Notes Supports SSML markup Multiple voice styles Gender selection Prosody control Emphasis levels Regional language variants Metrics collection Chunked audio output Thread-safe processing Fish Audio Groq On this page Overview Installation Configuration Constructor Parameters Input Parameters Output Frames Control Frames Audio Frames Error Frames Usage Examples Basic Usage With SSML Controls Methods Language Support Usage Example Regional Considerations Frame Flow Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_google_0e691248.txt b/tts_google_0e691248.txt
new file mode 100644
index 0000000000000000000000000000000000000000..209ac46f7a98aacb7720d7c76f20ecad47b6d2b8
--- /dev/null
+++ b/tts_google_0e691248.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/google#installation
+Title: Google - Pipecat
+==================================================
+
+Google - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech Google Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview GoogleTTSService provides high-quality text-to-speech synthesis using Google Cloud’s Text-to-Speech API. It supports SSML for advanced voice control and multiple languages. ​ Installation To use GoogleTTSService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[google]" You’ll also need to set up Google Cloud credentials through either: Environment variable: GOOGLE_APPLICATION_CREDENTIALS Direct credentials JSON Credentials file path ​ Configuration ​ Constructor Parameters ​ credentials str | None Google Cloud credentials JSON string ​ credentials_path str | None Path to credentials JSON file ​ voice_id str default: "en-US-Neural2-A" Voice identifier ​ sample_rate int default: "None" Output audio sample rate in Hz ​ text_filter BaseTextFilter default: "None" Modifies text provided to the TTS. Learn more about the available filters. ​ Input Parameters Copy Ask AI class InputParams ( BaseModel ): pitch: Optional[ str ] rate: Optional[ str ] volume: Optional[ str ] emphasis: Optional[Literal[ "strong" , "moderate" , "reduced" , "none" ]] language: Optional[Language] = Language. EN gender: Optional[Literal[ "male" , "female" , "neutral" ]] google_style: Optional[Literal[ "apologetic" , "calm" , "empathetic" , "firm" , "lively" ]] ​ Output Frames ​ Control Frames ​ TTSStartedFrame Frame Signals start of synthesis ​ TTSStoppedFrame Frame Signals completion of synthesis ​ Audio Frames ​ TTSAudioRawFrame Frame Contains generated audio data: - PCM encoded audio - Configured sample rate - Mono channel ​ Error Frames ​ ErrorFrame Frame Contains error information ​ Usage Examples ​ Basic Usage Copy Ask AI # Configure service tts = GoogleTTSService( credentials_path = "path/to/credentials.json" , voice_id = "en-US-Neural2-A" , params = GoogleTTSService.InputParams( language = Language. EN , gender = "female" , google_style = "empathetic" ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ With SSML Controls Copy Ask AI # Configure with voice controls service = GoogleTTSService( credentials = credentials_json, params = GoogleTTSService.InputParams( pitch = "+2st" , rate = "1.2" , volume = "loud" , emphasis = "moderate" ) ) ​ Methods See the TTS base class methods for additional functionality. ​ Language Support Google Cloud Text-to-Speech supports the following languages and regional variants: Language Code Description Service Code Language.BG Bulgarian bg-BG Language.CA Catalan ca-ES Language.ZH Chinese (Mandarin) cmn-CN Language.ZH_TW Chinese (Taiwan) cmn-TW Language.CS Czech cs-CZ Language.DA Danish da-DK Language.NL Dutch (Netherlands) nl-NL Language.NL_BE Dutch (Belgium) nl-BE Language.EN English (US) en-US Language.EN_US English (US) en-US Language.EN_AU English (Australia) en-AU Language.EN_GB English (UK) en-GB Language.EN_IN English (India) en-IN Language.ET Estonian et-EE Language.FI Finnish fi-FI Language.FR French (France) fr-FR Language.FR_CA French (Canada) fr-CA Language.DE German de-DE Language.EL Greek el-GR Language.HI Hindi hi-IN Language.HU Hungarian hu-HU Language.ID Indonesian id-ID Language.IT Italian it-IT Language.JA Japanese ja-JP Language.KO Korean ko-KR Language.LV Latvian lv-LV Language.LT Lithuanian lt-LT Language.MS Malay ms-MY Language.NO Norwegian nb-NO Language.PL Polish pl-PL Language.PT Portuguese (Portugal) pt-PT Language.PT_BR Portuguese (Brazil) pt-BR Language.RO Romanian ro-RO Language.RU Russian ru-RU Language.SK Slovak sk-SK Language.ES Spanish es-ES Language.SV Swedish sv-SE Language.TH Thai th-TH Language.TR Turkish tr-TR Language.UK Ukrainian uk-UA Language.VI Vietnamese vi-VN ​ Usage Example Copy Ask AI # Configure service with specific language and region service = GoogleTTSService( credentials_path = "path/to/credentials.json" , voice_id = "en-US-Neural2-A" , params = GoogleTTSService.InputParams( language = Language. EN_GB , # British English gender = "female" ) ) ​ Regional Considerations Each language code includes both language and region (e.g., fr-FR for French in France) Some languages have multiple regional variants (e.g., English has US, UK, Australian, and Indian variants) Voice availability may vary by region Neural voices may not be available for all language/region combinations Note: Voice selection should match the specified language code for optimal results. ​ Frame Flow ​ Notes Supports SSML markup Multiple voice styles Gender selection Prosody control Emphasis levels Regional language variants Metrics collection Chunked audio output Thread-safe processing Fish Audio Groq On this page Overview Installation Configuration Constructor Parameters Input Parameters Output Frames Control Frames Audio Frames Error Frames Usage Examples Basic Usage With SSML Controls Methods Language Support Usage Example Regional Considerations Frame Flow Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_google_17b9020a.txt b/tts_google_17b9020a.txt
new file mode 100644
index 0000000000000000000000000000000000000000..7b5656e398e4b06a32dff391fa9359c6c8d9f41d
--- /dev/null
+++ b/tts_google_17b9020a.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/google#configuration
+Title: Google - Pipecat
+==================================================
+
+Google - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech Google Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview GoogleTTSService provides high-quality text-to-speech synthesis using Google Cloud’s Text-to-Speech API. It supports SSML for advanced voice control and multiple languages. ​ Installation To use GoogleTTSService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[google]" You’ll also need to set up Google Cloud credentials through either: Environment variable: GOOGLE_APPLICATION_CREDENTIALS Direct credentials JSON Credentials file path ​ Configuration ​ Constructor Parameters ​ credentials str | None Google Cloud credentials JSON string ​ credentials_path str | None Path to credentials JSON file ​ voice_id str default: "en-US-Neural2-A" Voice identifier ​ sample_rate int default: "None" Output audio sample rate in Hz ​ text_filter BaseTextFilter default: "None" Modifies text provided to the TTS. Learn more about the available filters. ​ Input Parameters Copy Ask AI class InputParams ( BaseModel ): pitch: Optional[ str ] rate: Optional[ str ] volume: Optional[ str ] emphasis: Optional[Literal[ "strong" , "moderate" , "reduced" , "none" ]] language: Optional[Language] = Language. EN gender: Optional[Literal[ "male" , "female" , "neutral" ]] google_style: Optional[Literal[ "apologetic" , "calm" , "empathetic" , "firm" , "lively" ]] ​ Output Frames ​ Control Frames ​ TTSStartedFrame Frame Signals start of synthesis ​ TTSStoppedFrame Frame Signals completion of synthesis ​ Audio Frames ​ TTSAudioRawFrame Frame Contains generated audio data: - PCM encoded audio - Configured sample rate - Mono channel ​ Error Frames ​ ErrorFrame Frame Contains error information ​ Usage Examples ​ Basic Usage Copy Ask AI # Configure service tts = GoogleTTSService( credentials_path = "path/to/credentials.json" , voice_id = "en-US-Neural2-A" , params = GoogleTTSService.InputParams( language = Language. EN , gender = "female" , google_style = "empathetic" ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ With SSML Controls Copy Ask AI # Configure with voice controls service = GoogleTTSService( credentials = credentials_json, params = GoogleTTSService.InputParams( pitch = "+2st" , rate = "1.2" , volume = "loud" , emphasis = "moderate" ) ) ​ Methods See the TTS base class methods for additional functionality. ​ Language Support Google Cloud Text-to-Speech supports the following languages and regional variants: Language Code Description Service Code Language.BG Bulgarian bg-BG Language.CA Catalan ca-ES Language.ZH Chinese (Mandarin) cmn-CN Language.ZH_TW Chinese (Taiwan) cmn-TW Language.CS Czech cs-CZ Language.DA Danish da-DK Language.NL Dutch (Netherlands) nl-NL Language.NL_BE Dutch (Belgium) nl-BE Language.EN English (US) en-US Language.EN_US English (US) en-US Language.EN_AU English (Australia) en-AU Language.EN_GB English (UK) en-GB Language.EN_IN English (India) en-IN Language.ET Estonian et-EE Language.FI Finnish fi-FI Language.FR French (France) fr-FR Language.FR_CA French (Canada) fr-CA Language.DE German de-DE Language.EL Greek el-GR Language.HI Hindi hi-IN Language.HU Hungarian hu-HU Language.ID Indonesian id-ID Language.IT Italian it-IT Language.JA Japanese ja-JP Language.KO Korean ko-KR Language.LV Latvian lv-LV Language.LT Lithuanian lt-LT Language.MS Malay ms-MY Language.NO Norwegian nb-NO Language.PL Polish pl-PL Language.PT Portuguese (Portugal) pt-PT Language.PT_BR Portuguese (Brazil) pt-BR Language.RO Romanian ro-RO Language.RU Russian ru-RU Language.SK Slovak sk-SK Language.ES Spanish es-ES Language.SV Swedish sv-SE Language.TH Thai th-TH Language.TR Turkish tr-TR Language.UK Ukrainian uk-UA Language.VI Vietnamese vi-VN ​ Usage Example Copy Ask AI # Configure service with specific language and region service = GoogleTTSService( credentials_path = "path/to/credentials.json" , voice_id = "en-US-Neural2-A" , params = GoogleTTSService.InputParams( language = Language. EN_GB , # British English gender = "female" ) ) ​ Regional Considerations Each language code includes both language and region (e.g., fr-FR for French in France) Some languages have multiple regional variants (e.g., English has US, UK, Australian, and Indian variants) Voice availability may vary by region Neural voices may not be available for all language/region combinations Note: Voice selection should match the specified language code for optimal results. ​ Frame Flow ​ Notes Supports SSML markup Multiple voice styles Gender selection Prosody control Emphasis levels Regional language variants Metrics collection Chunked audio output Thread-safe processing Fish Audio Groq On this page Overview Installation Configuration Constructor Parameters Input Parameters Output Frames Control Frames Audio Frames Error Frames Usage Examples Basic Usage With SSML Controls Methods Language Support Usage Example Regional Considerations Frame Flow Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_google_2db9019e.txt b/tts_google_2db9019e.txt
new file mode 100644
index 0000000000000000000000000000000000000000..6a5d8fa04576c8d20dfb8ff6c814fc4ed7517c2a
--- /dev/null
+++ b/tts_google_2db9019e.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/google#output-frames
+Title: Google - Pipecat
+==================================================
+
+Google - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech Google Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview GoogleTTSService provides high-quality text-to-speech synthesis using Google Cloud’s Text-to-Speech API. It supports SSML for advanced voice control and multiple languages. ​ Installation To use GoogleTTSService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[google]" You’ll also need to set up Google Cloud credentials through either: Environment variable: GOOGLE_APPLICATION_CREDENTIALS Direct credentials JSON Credentials file path ​ Configuration ​ Constructor Parameters ​ credentials str | None Google Cloud credentials JSON string ​ credentials_path str | None Path to credentials JSON file ​ voice_id str default: "en-US-Neural2-A" Voice identifier ​ sample_rate int default: "None" Output audio sample rate in Hz ​ text_filter BaseTextFilter default: "None" Modifies text provided to the TTS. Learn more about the available filters. ​ Input Parameters Copy Ask AI class InputParams ( BaseModel ): pitch: Optional[ str ] rate: Optional[ str ] volume: Optional[ str ] emphasis: Optional[Literal[ "strong" , "moderate" , "reduced" , "none" ]] language: Optional[Language] = Language. EN gender: Optional[Literal[ "male" , "female" , "neutral" ]] google_style: Optional[Literal[ "apologetic" , "calm" , "empathetic" , "firm" , "lively" ]] ​ Output Frames ​ Control Frames ​ TTSStartedFrame Frame Signals start of synthesis ​ TTSStoppedFrame Frame Signals completion of synthesis ​ Audio Frames ​ TTSAudioRawFrame Frame Contains generated audio data: - PCM encoded audio - Configured sample rate - Mono channel ​ Error Frames ​ ErrorFrame Frame Contains error information ​ Usage Examples ​ Basic Usage Copy Ask AI # Configure service tts = GoogleTTSService( credentials_path = "path/to/credentials.json" , voice_id = "en-US-Neural2-A" , params = GoogleTTSService.InputParams( language = Language. EN , gender = "female" , google_style = "empathetic" ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ With SSML Controls Copy Ask AI # Configure with voice controls service = GoogleTTSService( credentials = credentials_json, params = GoogleTTSService.InputParams( pitch = "+2st" , rate = "1.2" , volume = "loud" , emphasis = "moderate" ) ) ​ Methods See the TTS base class methods for additional functionality. ​ Language Support Google Cloud Text-to-Speech supports the following languages and regional variants: Language Code Description Service Code Language.BG Bulgarian bg-BG Language.CA Catalan ca-ES Language.ZH Chinese (Mandarin) cmn-CN Language.ZH_TW Chinese (Taiwan) cmn-TW Language.CS Czech cs-CZ Language.DA Danish da-DK Language.NL Dutch (Netherlands) nl-NL Language.NL_BE Dutch (Belgium) nl-BE Language.EN English (US) en-US Language.EN_US English (US) en-US Language.EN_AU English (Australia) en-AU Language.EN_GB English (UK) en-GB Language.EN_IN English (India) en-IN Language.ET Estonian et-EE Language.FI Finnish fi-FI Language.FR French (France) fr-FR Language.FR_CA French (Canada) fr-CA Language.DE German de-DE Language.EL Greek el-GR Language.HI Hindi hi-IN Language.HU Hungarian hu-HU Language.ID Indonesian id-ID Language.IT Italian it-IT Language.JA Japanese ja-JP Language.KO Korean ko-KR Language.LV Latvian lv-LV Language.LT Lithuanian lt-LT Language.MS Malay ms-MY Language.NO Norwegian nb-NO Language.PL Polish pl-PL Language.PT Portuguese (Portugal) pt-PT Language.PT_BR Portuguese (Brazil) pt-BR Language.RO Romanian ro-RO Language.RU Russian ru-RU Language.SK Slovak sk-SK Language.ES Spanish es-ES Language.SV Swedish sv-SE Language.TH Thai th-TH Language.TR Turkish tr-TR Language.UK Ukrainian uk-UA Language.VI Vietnamese vi-VN ​ Usage Example Copy Ask AI # Configure service with specific language and region service = GoogleTTSService( credentials_path = "path/to/credentials.json" , voice_id = "en-US-Neural2-A" , params = GoogleTTSService.InputParams( language = Language. EN_GB , # British English gender = "female" ) ) ​ Regional Considerations Each language code includes both language and region (e.g., fr-FR for French in France) Some languages have multiple regional variants (e.g., English has US, UK, Australian, and Indian variants) Voice availability may vary by region Neural voices may not be available for all language/region combinations Note: Voice selection should match the specified language code for optimal results. ​ Frame Flow ​ Notes Supports SSML markup Multiple voice styles Gender selection Prosody control Emphasis levels Regional language variants Metrics collection Chunked audio output Thread-safe processing Fish Audio Groq On this page Overview Installation Configuration Constructor Parameters Input Parameters Output Frames Control Frames Audio Frames Error Frames Usage Examples Basic Usage With SSML Controls Methods Language Support Usage Example Regional Considerations Frame Flow Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_google_ac4e95c8.txt b/tts_google_ac4e95c8.txt
new file mode 100644
index 0000000000000000000000000000000000000000..1a5ae1645a48d8b485a7df1221a9a4db936924ca
--- /dev/null
+++ b/tts_google_ac4e95c8.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/google#error-frames
+Title: Google - Pipecat
+==================================================
+
+Google - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech Google Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview GoogleTTSService provides high-quality text-to-speech synthesis using Google Cloud’s Text-to-Speech API. It supports SSML for advanced voice control and multiple languages. ​ Installation To use GoogleTTSService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[google]" You’ll also need to set up Google Cloud credentials through either: Environment variable: GOOGLE_APPLICATION_CREDENTIALS Direct credentials JSON Credentials file path ​ Configuration ​ Constructor Parameters ​ credentials str | None Google Cloud credentials JSON string ​ credentials_path str | None Path to credentials JSON file ​ voice_id str default: "en-US-Neural2-A" Voice identifier ​ sample_rate int default: "None" Output audio sample rate in Hz ​ text_filter BaseTextFilter default: "None" Modifies text provided to the TTS. Learn more about the available filters. ​ Input Parameters Copy Ask AI class InputParams ( BaseModel ): pitch: Optional[ str ] rate: Optional[ str ] volume: Optional[ str ] emphasis: Optional[Literal[ "strong" , "moderate" , "reduced" , "none" ]] language: Optional[Language] = Language. EN gender: Optional[Literal[ "male" , "female" , "neutral" ]] google_style: Optional[Literal[ "apologetic" , "calm" , "empathetic" , "firm" , "lively" ]] ​ Output Frames ​ Control Frames ​ TTSStartedFrame Frame Signals start of synthesis ​ TTSStoppedFrame Frame Signals completion of synthesis ​ Audio Frames ​ TTSAudioRawFrame Frame Contains generated audio data: - PCM encoded audio - Configured sample rate - Mono channel ​ Error Frames ​ ErrorFrame Frame Contains error information ​ Usage Examples ​ Basic Usage Copy Ask AI # Configure service tts = GoogleTTSService( credentials_path = "path/to/credentials.json" , voice_id = "en-US-Neural2-A" , params = GoogleTTSService.InputParams( language = Language. EN , gender = "female" , google_style = "empathetic" ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ With SSML Controls Copy Ask AI # Configure with voice controls service = GoogleTTSService( credentials = credentials_json, params = GoogleTTSService.InputParams( pitch = "+2st" , rate = "1.2" , volume = "loud" , emphasis = "moderate" ) ) ​ Methods See the TTS base class methods for additional functionality. ​ Language Support Google Cloud Text-to-Speech supports the following languages and regional variants: Language Code Description Service Code Language.BG Bulgarian bg-BG Language.CA Catalan ca-ES Language.ZH Chinese (Mandarin) cmn-CN Language.ZH_TW Chinese (Taiwan) cmn-TW Language.CS Czech cs-CZ Language.DA Danish da-DK Language.NL Dutch (Netherlands) nl-NL Language.NL_BE Dutch (Belgium) nl-BE Language.EN English (US) en-US Language.EN_US English (US) en-US Language.EN_AU English (Australia) en-AU Language.EN_GB English (UK) en-GB Language.EN_IN English (India) en-IN Language.ET Estonian et-EE Language.FI Finnish fi-FI Language.FR French (France) fr-FR Language.FR_CA French (Canada) fr-CA Language.DE German de-DE Language.EL Greek el-GR Language.HI Hindi hi-IN Language.HU Hungarian hu-HU Language.ID Indonesian id-ID Language.IT Italian it-IT Language.JA Japanese ja-JP Language.KO Korean ko-KR Language.LV Latvian lv-LV Language.LT Lithuanian lt-LT Language.MS Malay ms-MY Language.NO Norwegian nb-NO Language.PL Polish pl-PL Language.PT Portuguese (Portugal) pt-PT Language.PT_BR Portuguese (Brazil) pt-BR Language.RO Romanian ro-RO Language.RU Russian ru-RU Language.SK Slovak sk-SK Language.ES Spanish es-ES Language.SV Swedish sv-SE Language.TH Thai th-TH Language.TR Turkish tr-TR Language.UK Ukrainian uk-UA Language.VI Vietnamese vi-VN ​ Usage Example Copy Ask AI # Configure service with specific language and region service = GoogleTTSService( credentials_path = "path/to/credentials.json" , voice_id = "en-US-Neural2-A" , params = GoogleTTSService.InputParams( language = Language. EN_GB , # British English gender = "female" ) ) ​ Regional Considerations Each language code includes both language and region (e.g., fr-FR for French in France) Some languages have multiple regional variants (e.g., English has US, UK, Australian, and Indian variants) Voice availability may vary by region Neural voices may not be available for all language/region combinations Note: Voice selection should match the specified language code for optimal results. ​ Frame Flow ​ Notes Supports SSML markup Multiple voice styles Gender selection Prosody control Emphasis levels Regional language variants Metrics collection Chunked audio output Thread-safe processing Fish Audio Groq On this page Overview Installation Configuration Constructor Parameters Input Parameters Output Frames Control Frames Audio Frames Error Frames Usage Examples Basic Usage With SSML Controls Methods Language Support Usage Example Regional Considerations Frame Flow Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_google_bf7de62a.txt b/tts_google_bf7de62a.txt
new file mode 100644
index 0000000000000000000000000000000000000000..9d98af119c6f6d1022bd7b0cc24906b848fe6a75
--- /dev/null
+++ b/tts_google_bf7de62a.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/google#param-text-filter
+Title: Google - Pipecat
+==================================================
+
+Google - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech Google Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview GoogleTTSService provides high-quality text-to-speech synthesis using Google Cloud’s Text-to-Speech API. It supports SSML for advanced voice control and multiple languages. ​ Installation To use GoogleTTSService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[google]" You’ll also need to set up Google Cloud credentials through either: Environment variable: GOOGLE_APPLICATION_CREDENTIALS Direct credentials JSON Credentials file path ​ Configuration ​ Constructor Parameters ​ credentials str | None Google Cloud credentials JSON string ​ credentials_path str | None Path to credentials JSON file ​ voice_id str default: "en-US-Neural2-A" Voice identifier ​ sample_rate int default: "None" Output audio sample rate in Hz ​ text_filter BaseTextFilter default: "None" Modifies text provided to the TTS. Learn more about the available filters. ​ Input Parameters Copy Ask AI class InputParams ( BaseModel ): pitch: Optional[ str ] rate: Optional[ str ] volume: Optional[ str ] emphasis: Optional[Literal[ "strong" , "moderate" , "reduced" , "none" ]] language: Optional[Language] = Language. EN gender: Optional[Literal[ "male" , "female" , "neutral" ]] google_style: Optional[Literal[ "apologetic" , "calm" , "empathetic" , "firm" , "lively" ]] ​ Output Frames ​ Control Frames ​ TTSStartedFrame Frame Signals start of synthesis ​ TTSStoppedFrame Frame Signals completion of synthesis ​ Audio Frames ​ TTSAudioRawFrame Frame Contains generated audio data: - PCM encoded audio - Configured sample rate - Mono channel ​ Error Frames ​ ErrorFrame Frame Contains error information ​ Usage Examples ​ Basic Usage Copy Ask AI # Configure service tts = GoogleTTSService( credentials_path = "path/to/credentials.json" , voice_id = "en-US-Neural2-A" , params = GoogleTTSService.InputParams( language = Language. EN , gender = "female" , google_style = "empathetic" ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ With SSML Controls Copy Ask AI # Configure with voice controls service = GoogleTTSService( credentials = credentials_json, params = GoogleTTSService.InputParams( pitch = "+2st" , rate = "1.2" , volume = "loud" , emphasis = "moderate" ) ) ​ Methods See the TTS base class methods for additional functionality. ​ Language Support Google Cloud Text-to-Speech supports the following languages and regional variants: Language Code Description Service Code Language.BG Bulgarian bg-BG Language.CA Catalan ca-ES Language.ZH Chinese (Mandarin) cmn-CN Language.ZH_TW Chinese (Taiwan) cmn-TW Language.CS Czech cs-CZ Language.DA Danish da-DK Language.NL Dutch (Netherlands) nl-NL Language.NL_BE Dutch (Belgium) nl-BE Language.EN English (US) en-US Language.EN_US English (US) en-US Language.EN_AU English (Australia) en-AU Language.EN_GB English (UK) en-GB Language.EN_IN English (India) en-IN Language.ET Estonian et-EE Language.FI Finnish fi-FI Language.FR French (France) fr-FR Language.FR_CA French (Canada) fr-CA Language.DE German de-DE Language.EL Greek el-GR Language.HI Hindi hi-IN Language.HU Hungarian hu-HU Language.ID Indonesian id-ID Language.IT Italian it-IT Language.JA Japanese ja-JP Language.KO Korean ko-KR Language.LV Latvian lv-LV Language.LT Lithuanian lt-LT Language.MS Malay ms-MY Language.NO Norwegian nb-NO Language.PL Polish pl-PL Language.PT Portuguese (Portugal) pt-PT Language.PT_BR Portuguese (Brazil) pt-BR Language.RO Romanian ro-RO Language.RU Russian ru-RU Language.SK Slovak sk-SK Language.ES Spanish es-ES Language.SV Swedish sv-SE Language.TH Thai th-TH Language.TR Turkish tr-TR Language.UK Ukrainian uk-UA Language.VI Vietnamese vi-VN ​ Usage Example Copy Ask AI # Configure service with specific language and region service = GoogleTTSService( credentials_path = "path/to/credentials.json" , voice_id = "en-US-Neural2-A" , params = GoogleTTSService.InputParams( language = Language. EN_GB , # British English gender = "female" ) ) ​ Regional Considerations Each language code includes both language and region (e.g., fr-FR for French in France) Some languages have multiple regional variants (e.g., English has US, UK, Australian, and Indian variants) Voice availability may vary by region Neural voices may not be available for all language/region combinations Note: Voice selection should match the specified language code for optimal results. ​ Frame Flow ​ Notes Supports SSML markup Multiple voice styles Gender selection Prosody control Emphasis levels Regional language variants Metrics collection Chunked audio output Thread-safe processing Fish Audio Groq On this page Overview Installation Configuration Constructor Parameters Input Parameters Output Frames Control Frames Audio Frames Error Frames Usage Examples Basic Usage With SSML Controls Methods Language Support Usage Example Regional Considerations Frame Flow Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_google_bf7fd4f8.txt b/tts_google_bf7fd4f8.txt
new file mode 100644
index 0000000000000000000000000000000000000000..946b5717cc32c8f7f95ba24e30704dc7d8348b1c
--- /dev/null
+++ b/tts_google_bf7fd4f8.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/google#param-tts-audio-raw-frame
+Title: Google - Pipecat
+==================================================
+
+Google - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech Google Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview GoogleTTSService provides high-quality text-to-speech synthesis using Google Cloud’s Text-to-Speech API. It supports SSML for advanced voice control and multiple languages. ​ Installation To use GoogleTTSService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[google]" You’ll also need to set up Google Cloud credentials through either: Environment variable: GOOGLE_APPLICATION_CREDENTIALS Direct credentials JSON Credentials file path ​ Configuration ​ Constructor Parameters ​ credentials str | None Google Cloud credentials JSON string ​ credentials_path str | None Path to credentials JSON file ​ voice_id str default: "en-US-Neural2-A" Voice identifier ​ sample_rate int default: "None" Output audio sample rate in Hz ​ text_filter BaseTextFilter default: "None" Modifies text provided to the TTS. Learn more about the available filters. ​ Input Parameters Copy Ask AI class InputParams ( BaseModel ): pitch: Optional[ str ] rate: Optional[ str ] volume: Optional[ str ] emphasis: Optional[Literal[ "strong" , "moderate" , "reduced" , "none" ]] language: Optional[Language] = Language. EN gender: Optional[Literal[ "male" , "female" , "neutral" ]] google_style: Optional[Literal[ "apologetic" , "calm" , "empathetic" , "firm" , "lively" ]] ​ Output Frames ​ Control Frames ​ TTSStartedFrame Frame Signals start of synthesis ​ TTSStoppedFrame Frame Signals completion of synthesis ​ Audio Frames ​ TTSAudioRawFrame Frame Contains generated audio data: - PCM encoded audio - Configured sample rate - Mono channel ​ Error Frames ​ ErrorFrame Frame Contains error information ​ Usage Examples ​ Basic Usage Copy Ask AI # Configure service tts = GoogleTTSService( credentials_path = "path/to/credentials.json" , voice_id = "en-US-Neural2-A" , params = GoogleTTSService.InputParams( language = Language. EN , gender = "female" , google_style = "empathetic" ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ With SSML Controls Copy Ask AI # Configure with voice controls service = GoogleTTSService( credentials = credentials_json, params = GoogleTTSService.InputParams( pitch = "+2st" , rate = "1.2" , volume = "loud" , emphasis = "moderate" ) ) ​ Methods See the TTS base class methods for additional functionality. ​ Language Support Google Cloud Text-to-Speech supports the following languages and regional variants: Language Code Description Service Code Language.BG Bulgarian bg-BG Language.CA Catalan ca-ES Language.ZH Chinese (Mandarin) cmn-CN Language.ZH_TW Chinese (Taiwan) cmn-TW Language.CS Czech cs-CZ Language.DA Danish da-DK Language.NL Dutch (Netherlands) nl-NL Language.NL_BE Dutch (Belgium) nl-BE Language.EN English (US) en-US Language.EN_US English (US) en-US Language.EN_AU English (Australia) en-AU Language.EN_GB English (UK) en-GB Language.EN_IN English (India) en-IN Language.ET Estonian et-EE Language.FI Finnish fi-FI Language.FR French (France) fr-FR Language.FR_CA French (Canada) fr-CA Language.DE German de-DE Language.EL Greek el-GR Language.HI Hindi hi-IN Language.HU Hungarian hu-HU Language.ID Indonesian id-ID Language.IT Italian it-IT Language.JA Japanese ja-JP Language.KO Korean ko-KR Language.LV Latvian lv-LV Language.LT Lithuanian lt-LT Language.MS Malay ms-MY Language.NO Norwegian nb-NO Language.PL Polish pl-PL Language.PT Portuguese (Portugal) pt-PT Language.PT_BR Portuguese (Brazil) pt-BR Language.RO Romanian ro-RO Language.RU Russian ru-RU Language.SK Slovak sk-SK Language.ES Spanish es-ES Language.SV Swedish sv-SE Language.TH Thai th-TH Language.TR Turkish tr-TR Language.UK Ukrainian uk-UA Language.VI Vietnamese vi-VN ​ Usage Example Copy Ask AI # Configure service with specific language and region service = GoogleTTSService( credentials_path = "path/to/credentials.json" , voice_id = "en-US-Neural2-A" , params = GoogleTTSService.InputParams( language = Language. EN_GB , # British English gender = "female" ) ) ​ Regional Considerations Each language code includes both language and region (e.g., fr-FR for French in France) Some languages have multiple regional variants (e.g., English has US, UK, Australian, and Indian variants) Voice availability may vary by region Neural voices may not be available for all language/region combinations Note: Voice selection should match the specified language code for optimal results. ​ Frame Flow ​ Notes Supports SSML markup Multiple voice styles Gender selection Prosody control Emphasis levels Regional language variants Metrics collection Chunked audio output Thread-safe processing Fish Audio Groq On this page Overview Installation Configuration Constructor Parameters Input Parameters Output Frames Control Frames Audio Frames Error Frames Usage Examples Basic Usage With SSML Controls Methods Language Support Usage Example Regional Considerations Frame Flow Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_google_c49d69b9.txt b/tts_google_c49d69b9.txt
new file mode 100644
index 0000000000000000000000000000000000000000..c43b416829b613df05dc2214ce25f524356815bb
--- /dev/null
+++ b/tts_google_c49d69b9.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/google#with-ssml-controls
+Title: Google - Pipecat
+==================================================
+
+Google - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech Google Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview GoogleTTSService provides high-quality text-to-speech synthesis using Google Cloud’s Text-to-Speech API. It supports SSML for advanced voice control and multiple languages. ​ Installation To use GoogleTTSService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[google]" You’ll also need to set up Google Cloud credentials through either: Environment variable: GOOGLE_APPLICATION_CREDENTIALS Direct credentials JSON Credentials file path ​ Configuration ​ Constructor Parameters ​ credentials str | None Google Cloud credentials JSON string ​ credentials_path str | None Path to credentials JSON file ​ voice_id str default: "en-US-Neural2-A" Voice identifier ​ sample_rate int default: "None" Output audio sample rate in Hz ​ text_filter BaseTextFilter default: "None" Modifies text provided to the TTS. Learn more about the available filters. ​ Input Parameters Copy Ask AI class InputParams ( BaseModel ): pitch: Optional[ str ] rate: Optional[ str ] volume: Optional[ str ] emphasis: Optional[Literal[ "strong" , "moderate" , "reduced" , "none" ]] language: Optional[Language] = Language. EN gender: Optional[Literal[ "male" , "female" , "neutral" ]] google_style: Optional[Literal[ "apologetic" , "calm" , "empathetic" , "firm" , "lively" ]] ​ Output Frames ​ Control Frames ​ TTSStartedFrame Frame Signals start of synthesis ​ TTSStoppedFrame Frame Signals completion of synthesis ​ Audio Frames ​ TTSAudioRawFrame Frame Contains generated audio data: - PCM encoded audio - Configured sample rate - Mono channel ​ Error Frames ​ ErrorFrame Frame Contains error information ​ Usage Examples ​ Basic Usage Copy Ask AI # Configure service tts = GoogleTTSService( credentials_path = "path/to/credentials.json" , voice_id = "en-US-Neural2-A" , params = GoogleTTSService.InputParams( language = Language. EN , gender = "female" , google_style = "empathetic" ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ With SSML Controls Copy Ask AI # Configure with voice controls service = GoogleTTSService( credentials = credentials_json, params = GoogleTTSService.InputParams( pitch = "+2st" , rate = "1.2" , volume = "loud" , emphasis = "moderate" ) ) ​ Methods See the TTS base class methods for additional functionality. ​ Language Support Google Cloud Text-to-Speech supports the following languages and regional variants: Language Code Description Service Code Language.BG Bulgarian bg-BG Language.CA Catalan ca-ES Language.ZH Chinese (Mandarin) cmn-CN Language.ZH_TW Chinese (Taiwan) cmn-TW Language.CS Czech cs-CZ Language.DA Danish da-DK Language.NL Dutch (Netherlands) nl-NL Language.NL_BE Dutch (Belgium) nl-BE Language.EN English (US) en-US Language.EN_US English (US) en-US Language.EN_AU English (Australia) en-AU Language.EN_GB English (UK) en-GB Language.EN_IN English (India) en-IN Language.ET Estonian et-EE Language.FI Finnish fi-FI Language.FR French (France) fr-FR Language.FR_CA French (Canada) fr-CA Language.DE German de-DE Language.EL Greek el-GR Language.HI Hindi hi-IN Language.HU Hungarian hu-HU Language.ID Indonesian id-ID Language.IT Italian it-IT Language.JA Japanese ja-JP Language.KO Korean ko-KR Language.LV Latvian lv-LV Language.LT Lithuanian lt-LT Language.MS Malay ms-MY Language.NO Norwegian nb-NO Language.PL Polish pl-PL Language.PT Portuguese (Portugal) pt-PT Language.PT_BR Portuguese (Brazil) pt-BR Language.RO Romanian ro-RO Language.RU Russian ru-RU Language.SK Slovak sk-SK Language.ES Spanish es-ES Language.SV Swedish sv-SE Language.TH Thai th-TH Language.TR Turkish tr-TR Language.UK Ukrainian uk-UA Language.VI Vietnamese vi-VN ​ Usage Example Copy Ask AI # Configure service with specific language and region service = GoogleTTSService( credentials_path = "path/to/credentials.json" , voice_id = "en-US-Neural2-A" , params = GoogleTTSService.InputParams( language = Language. EN_GB , # British English gender = "female" ) ) ​ Regional Considerations Each language code includes both language and region (e.g., fr-FR for French in France) Some languages have multiple regional variants (e.g., English has US, UK, Australian, and Indian variants) Voice availability may vary by region Neural voices may not be available for all language/region combinations Note: Voice selection should match the specified language code for optimal results. ​ Frame Flow ​ Notes Supports SSML markup Multiple voice styles Gender selection Prosody control Emphasis levels Regional language variants Metrics collection Chunked audio output Thread-safe processing Fish Audio Groq On this page Overview Installation Configuration Constructor Parameters Input Parameters Output Frames Control Frames Audio Frames Error Frames Usage Examples Basic Usage With SSML Controls Methods Language Support Usage Example Regional Considerations Frame Flow Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_groq_214d0a00.txt b/tts_groq_214d0a00.txt
new file mode 100644
index 0000000000000000000000000000000000000000..2699ca1260afe6e197a28a839fb5dc63249ff592
--- /dev/null
+++ b/tts_groq_214d0a00.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/groq#param-output-format
+Title: Groq - Pipecat
+==================================================
+
+Groq - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech Groq Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview GroqTTSService converts text to speech using Groq’s TTS API. It supports real-time audio generation with multiple voices. ​ Installation To use GroqTTSService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[groq]" You’ll also need to set up your Groq API key as an environment variable: GROQ_API_KEY . You can obtain a Groq Cloud API key by signing up at Groq . ​ Configuration ​ Constructor Parameters ​ api_key str required Your Groq API key ​ output_format str default: "wav" Audio output format ​ params InputParams default: "InputParams()" Configuration parameters for speech generation ​ model_name str default: "playai-tts" TTS model to use. See the Groq Cloud docs for available models . ​ voice_id str default: "Celeste-PlayAI" Voice identifier to use for synthesis ​ Input Parameters ​ language Language default: "Language.EN" Language for speech synthesis ​ speed float default: "1.0" Speech rate multiplier (higher values produce faster speech) ​ seed Optional[int] default: "None" Random seed for reproducible audio generation ​ Input The service accepts text input through the pipeline, including streaming text from an LLM service. ​ Output Frames ​ TTSStartedFrame Signals the start of audio generation. ​ TTSAudioRawFrame Contains generated audio data: ​ audio bytes Raw audio data chunk ​ sample_rate int Audio sample rate, based on the constructor setting ​ num_channels int Number of audio channels (1 for mono) ​ TTSStoppedFrame Signals the completion of audio generation. ​ Methods See the TTS base class methods for additional functionality. ​ Language Support GroqTTSService supports the following languages: Language Code Description Service Codes Language.EN English en ​ Usage Example Copy Ask AI from pipecat.services.groq.tts import GroqTTSService from pipecat.transcriptions.language import Language # Configure service tts = GroqTTSService( api_key = "your-api-key" , model_name = "playai-tts" , voice_id = "Celeste-PlayAI" , params = GroqTTSService.InputParams( language = Language. EN , speed = 1.0 , seed = 42 ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ Frame Flow ​ Metrics Support The service supports metrics collection: Time to First Byte (TTFB) Processing duration ​ Audio Processing Streams audio in chunks Outputs mono audio at the defined sample rate Handles WAV header removal automatically Supports WAV format by default ​ Notes Requires a Groq Cloud API key Streams audio in chunks for efficient processing Automatically handles WAV headers in the response Provides metrics collection Supports configurable speech parameters Google LMNT On this page Overview Installation Configuration Constructor Parameters Input Parameters Input Output Frames TTSStartedFrame TTSAudioRawFrame TTSStoppedFrame Methods Language Support Usage Example Frame Flow Metrics Support Audio Processing Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_groq_6f9b4ce3.txt b/tts_groq_6f9b4ce3.txt
new file mode 100644
index 0000000000000000000000000000000000000000..0f60cf28eeb41f355ef9142efe5a39a0ca16bbd0
--- /dev/null
+++ b/tts_groq_6f9b4ce3.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/groq#frame-flow
+Title: Groq - Pipecat
+==================================================
+
+Groq - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech Groq Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview GroqTTSService converts text to speech using Groq’s TTS API. It supports real-time audio generation with multiple voices. ​ Installation To use GroqTTSService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[groq]" You’ll also need to set up your Groq API key as an environment variable: GROQ_API_KEY . You can obtain a Groq Cloud API key by signing up at Groq . ​ Configuration ​ Constructor Parameters ​ api_key str required Your Groq API key ​ output_format str default: "wav" Audio output format ​ params InputParams default: "InputParams()" Configuration parameters for speech generation ​ model_name str default: "playai-tts" TTS model to use. See the Groq Cloud docs for available models . ​ voice_id str default: "Celeste-PlayAI" Voice identifier to use for synthesis ​ Input Parameters ​ language Language default: "Language.EN" Language for speech synthesis ​ speed float default: "1.0" Speech rate multiplier (higher values produce faster speech) ​ seed Optional[int] default: "None" Random seed for reproducible audio generation ​ Input The service accepts text input through the pipeline, including streaming text from an LLM service. ​ Output Frames ​ TTSStartedFrame Signals the start of audio generation. ​ TTSAudioRawFrame Contains generated audio data: ​ audio bytes Raw audio data chunk ​ sample_rate int Audio sample rate, based on the constructor setting ​ num_channels int Number of audio channels (1 for mono) ​ TTSStoppedFrame Signals the completion of audio generation. ​ Methods See the TTS base class methods for additional functionality. ​ Language Support GroqTTSService supports the following languages: Language Code Description Service Codes Language.EN English en ​ Usage Example Copy Ask AI from pipecat.services.groq.tts import GroqTTSService from pipecat.transcriptions.language import Language # Configure service tts = GroqTTSService( api_key = "your-api-key" , model_name = "playai-tts" , voice_id = "Celeste-PlayAI" , params = GroqTTSService.InputParams( language = Language. EN , speed = 1.0 , seed = 42 ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ Frame Flow ​ Metrics Support The service supports metrics collection: Time to First Byte (TTFB) Processing duration ​ Audio Processing Streams audio in chunks Outputs mono audio at the defined sample rate Handles WAV header removal automatically Supports WAV format by default ​ Notes Requires a Groq Cloud API key Streams audio in chunks for efficient processing Automatically handles WAV headers in the response Provides metrics collection Supports configurable speech parameters Google LMNT On this page Overview Installation Configuration Constructor Parameters Input Parameters Input Output Frames TTSStartedFrame TTSAudioRawFrame TTSStoppedFrame Methods Language Support Usage Example Frame Flow Metrics Support Audio Processing Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_groq_7cf40764.txt b/tts_groq_7cf40764.txt
new file mode 100644
index 0000000000000000000000000000000000000000..03c12ffa11f37b7e40f5427ee33333b8e9b655e7
--- /dev/null
+++ b/tts_groq_7cf40764.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/groq#ttsaudiorawframe
+Title: Groq - Pipecat
+==================================================
+
+Groq - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech Groq Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview GroqTTSService converts text to speech using Groq’s TTS API. It supports real-time audio generation with multiple voices. ​ Installation To use GroqTTSService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[groq]" You’ll also need to set up your Groq API key as an environment variable: GROQ_API_KEY . You can obtain a Groq Cloud API key by signing up at Groq . ​ Configuration ​ Constructor Parameters ​ api_key str required Your Groq API key ​ output_format str default: "wav" Audio output format ​ params InputParams default: "InputParams()" Configuration parameters for speech generation ​ model_name str default: "playai-tts" TTS model to use. See the Groq Cloud docs for available models . ​ voice_id str default: "Celeste-PlayAI" Voice identifier to use for synthesis ​ Input Parameters ​ language Language default: "Language.EN" Language for speech synthesis ​ speed float default: "1.0" Speech rate multiplier (higher values produce faster speech) ​ seed Optional[int] default: "None" Random seed for reproducible audio generation ​ Input The service accepts text input through the pipeline, including streaming text from an LLM service. ​ Output Frames ​ TTSStartedFrame Signals the start of audio generation. ​ TTSAudioRawFrame Contains generated audio data: ​ audio bytes Raw audio data chunk ​ sample_rate int Audio sample rate, based on the constructor setting ​ num_channels int Number of audio channels (1 for mono) ​ TTSStoppedFrame Signals the completion of audio generation. ​ Methods See the TTS base class methods for additional functionality. ​ Language Support GroqTTSService supports the following languages: Language Code Description Service Codes Language.EN English en ​ Usage Example Copy Ask AI from pipecat.services.groq.tts import GroqTTSService from pipecat.transcriptions.language import Language # Configure service tts = GroqTTSService( api_key = "your-api-key" , model_name = "playai-tts" , voice_id = "Celeste-PlayAI" , params = GroqTTSService.InputParams( language = Language. EN , speed = 1.0 , seed = 42 ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ Frame Flow ​ Metrics Support The service supports metrics collection: Time to First Byte (TTFB) Processing duration ​ Audio Processing Streams audio in chunks Outputs mono audio at the defined sample rate Handles WAV header removal automatically Supports WAV format by default ​ Notes Requires a Groq Cloud API key Streams audio in chunks for efficient processing Automatically handles WAV headers in the response Provides metrics collection Supports configurable speech parameters Google LMNT On this page Overview Installation Configuration Constructor Parameters Input Parameters Input Output Frames TTSStartedFrame TTSAudioRawFrame TTSStoppedFrame Methods Language Support Usage Example Frame Flow Metrics Support Audio Processing Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_groq_7d2da01d.txt b/tts_groq_7d2da01d.txt
new file mode 100644
index 0000000000000000000000000000000000000000..888e89e0e076a872e2d16051e5bae7b7a415edd2
--- /dev/null
+++ b/tts_groq_7d2da01d.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/groq#configuration
+Title: Groq - Pipecat
+==================================================
+
+Groq - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech Groq Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview GroqTTSService converts text to speech using Groq’s TTS API. It supports real-time audio generation with multiple voices. ​ Installation To use GroqTTSService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[groq]" You’ll also need to set up your Groq API key as an environment variable: GROQ_API_KEY . You can obtain a Groq Cloud API key by signing up at Groq . ​ Configuration ​ Constructor Parameters ​ api_key str required Your Groq API key ​ output_format str default: "wav" Audio output format ​ params InputParams default: "InputParams()" Configuration parameters for speech generation ​ model_name str default: "playai-tts" TTS model to use. See the Groq Cloud docs for available models . ​ voice_id str default: "Celeste-PlayAI" Voice identifier to use for synthesis ​ Input Parameters ​ language Language default: "Language.EN" Language for speech synthesis ​ speed float default: "1.0" Speech rate multiplier (higher values produce faster speech) ​ seed Optional[int] default: "None" Random seed for reproducible audio generation ​ Input The service accepts text input through the pipeline, including streaming text from an LLM service. ​ Output Frames ​ TTSStartedFrame Signals the start of audio generation. ​ TTSAudioRawFrame Contains generated audio data: ​ audio bytes Raw audio data chunk ​ sample_rate int Audio sample rate, based on the constructor setting ​ num_channels int Number of audio channels (1 for mono) ​ TTSStoppedFrame Signals the completion of audio generation. ​ Methods See the TTS base class methods for additional functionality. ​ Language Support GroqTTSService supports the following languages: Language Code Description Service Codes Language.EN English en ​ Usage Example Copy Ask AI from pipecat.services.groq.tts import GroqTTSService from pipecat.transcriptions.language import Language # Configure service tts = GroqTTSService( api_key = "your-api-key" , model_name = "playai-tts" , voice_id = "Celeste-PlayAI" , params = GroqTTSService.InputParams( language = Language. EN , speed = 1.0 , seed = 42 ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ Frame Flow ​ Metrics Support The service supports metrics collection: Time to First Byte (TTFB) Processing duration ​ Audio Processing Streams audio in chunks Outputs mono audio at the defined sample rate Handles WAV header removal automatically Supports WAV format by default ​ Notes Requires a Groq Cloud API key Streams audio in chunks for efficient processing Automatically handles WAV headers in the response Provides metrics collection Supports configurable speech parameters Google LMNT On this page Overview Installation Configuration Constructor Parameters Input Parameters Input Output Frames TTSStartedFrame TTSAudioRawFrame TTSStoppedFrame Methods Language Support Usage Example Frame Flow Metrics Support Audio Processing Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_groq_817d4009.txt b/tts_groq_817d4009.txt
new file mode 100644
index 0000000000000000000000000000000000000000..ea64499392f4909383bd2685bb3a111461c2a11f
--- /dev/null
+++ b/tts_groq_817d4009.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/groq#ttsstoppedframe
+Title: Groq - Pipecat
+==================================================
+
+Groq - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech Groq Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview GroqTTSService converts text to speech using Groq’s TTS API. It supports real-time audio generation with multiple voices. ​ Installation To use GroqTTSService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[groq]" You’ll also need to set up your Groq API key as an environment variable: GROQ_API_KEY . You can obtain a Groq Cloud API key by signing up at Groq . ​ Configuration ​ Constructor Parameters ​ api_key str required Your Groq API key ​ output_format str default: "wav" Audio output format ​ params InputParams default: "InputParams()" Configuration parameters for speech generation ​ model_name str default: "playai-tts" TTS model to use. See the Groq Cloud docs for available models . ​ voice_id str default: "Celeste-PlayAI" Voice identifier to use for synthesis ​ Input Parameters ​ language Language default: "Language.EN" Language for speech synthesis ​ speed float default: "1.0" Speech rate multiplier (higher values produce faster speech) ​ seed Optional[int] default: "None" Random seed for reproducible audio generation ​ Input The service accepts text input through the pipeline, including streaming text from an LLM service. ​ Output Frames ​ TTSStartedFrame Signals the start of audio generation. ​ TTSAudioRawFrame Contains generated audio data: ​ audio bytes Raw audio data chunk ​ sample_rate int Audio sample rate, based on the constructor setting ​ num_channels int Number of audio channels (1 for mono) ​ TTSStoppedFrame Signals the completion of audio generation. ​ Methods See the TTS base class methods for additional functionality. ​ Language Support GroqTTSService supports the following languages: Language Code Description Service Codes Language.EN English en ​ Usage Example Copy Ask AI from pipecat.services.groq.tts import GroqTTSService from pipecat.transcriptions.language import Language # Configure service tts = GroqTTSService( api_key = "your-api-key" , model_name = "playai-tts" , voice_id = "Celeste-PlayAI" , params = GroqTTSService.InputParams( language = Language. EN , speed = 1.0 , seed = 42 ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ Frame Flow ​ Metrics Support The service supports metrics collection: Time to First Byte (TTFB) Processing duration ​ Audio Processing Streams audio in chunks Outputs mono audio at the defined sample rate Handles WAV header removal automatically Supports WAV format by default ​ Notes Requires a Groq Cloud API key Streams audio in chunks for efficient processing Automatically handles WAV headers in the response Provides metrics collection Supports configurable speech parameters Google LMNT On this page Overview Installation Configuration Constructor Parameters Input Parameters Input Output Frames TTSStartedFrame TTSAudioRawFrame TTSStoppedFrame Methods Language Support Usage Example Frame Flow Metrics Support Audio Processing Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_groq_a6f9c255.txt b/tts_groq_a6f9c255.txt
new file mode 100644
index 0000000000000000000000000000000000000000..94c4c89260b11e0d896f536d36bed2cb8f140069
--- /dev/null
+++ b/tts_groq_a6f9c255.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/groq#param-audio
+Title: Groq - Pipecat
+==================================================
+
+Groq - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech Groq Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview GroqTTSService converts text to speech using Groq’s TTS API. It supports real-time audio generation with multiple voices. ​ Installation To use GroqTTSService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[groq]" You’ll also need to set up your Groq API key as an environment variable: GROQ_API_KEY . You can obtain a Groq Cloud API key by signing up at Groq . ​ Configuration ​ Constructor Parameters ​ api_key str required Your Groq API key ​ output_format str default: "wav" Audio output format ​ params InputParams default: "InputParams()" Configuration parameters for speech generation ​ model_name str default: "playai-tts" TTS model to use. See the Groq Cloud docs for available models . ​ voice_id str default: "Celeste-PlayAI" Voice identifier to use for synthesis ​ Input Parameters ​ language Language default: "Language.EN" Language for speech synthesis ​ speed float default: "1.0" Speech rate multiplier (higher values produce faster speech) ​ seed Optional[int] default: "None" Random seed for reproducible audio generation ​ Input The service accepts text input through the pipeline, including streaming text from an LLM service. ​ Output Frames ​ TTSStartedFrame Signals the start of audio generation. ​ TTSAudioRawFrame Contains generated audio data: ​ audio bytes Raw audio data chunk ​ sample_rate int Audio sample rate, based on the constructor setting ​ num_channels int Number of audio channels (1 for mono) ​ TTSStoppedFrame Signals the completion of audio generation. ​ Methods See the TTS base class methods for additional functionality. ​ Language Support GroqTTSService supports the following languages: Language Code Description Service Codes Language.EN English en ​ Usage Example Copy Ask AI from pipecat.services.groq.tts import GroqTTSService from pipecat.transcriptions.language import Language # Configure service tts = GroqTTSService( api_key = "your-api-key" , model_name = "playai-tts" , voice_id = "Celeste-PlayAI" , params = GroqTTSService.InputParams( language = Language. EN , speed = 1.0 , seed = 42 ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ Frame Flow ​ Metrics Support The service supports metrics collection: Time to First Byte (TTFB) Processing duration ​ Audio Processing Streams audio in chunks Outputs mono audio at the defined sample rate Handles WAV header removal automatically Supports WAV format by default ​ Notes Requires a Groq Cloud API key Streams audio in chunks for efficient processing Automatically handles WAV headers in the response Provides metrics collection Supports configurable speech parameters Google LMNT On this page Overview Installation Configuration Constructor Parameters Input Parameters Input Output Frames TTSStartedFrame TTSAudioRawFrame TTSStoppedFrame Methods Language Support Usage Example Frame Flow Metrics Support Audio Processing Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_groq_f6d2195e.txt b/tts_groq_f6d2195e.txt
new file mode 100644
index 0000000000000000000000000000000000000000..ef5c0a935c72b7df2353fbec6dd34fc6bae0430b
--- /dev/null
+++ b/tts_groq_f6d2195e.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/groq
+Title: Groq - Pipecat
+==================================================
+
+Groq - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech Groq Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview GroqTTSService converts text to speech using Groq’s TTS API. It supports real-time audio generation with multiple voices. ​ Installation To use GroqTTSService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[groq]" You’ll also need to set up your Groq API key as an environment variable: GROQ_API_KEY . You can obtain a Groq Cloud API key by signing up at Groq . ​ Configuration ​ Constructor Parameters ​ api_key str required Your Groq API key ​ output_format str default: "wav" Audio output format ​ params InputParams default: "InputParams()" Configuration parameters for speech generation ​ model_name str default: "playai-tts" TTS model to use. See the Groq Cloud docs for available models . ​ voice_id str default: "Celeste-PlayAI" Voice identifier to use for synthesis ​ Input Parameters ​ language Language default: "Language.EN" Language for speech synthesis ​ speed float default: "1.0" Speech rate multiplier (higher values produce faster speech) ​ seed Optional[int] default: "None" Random seed for reproducible audio generation ​ Input The service accepts text input through the pipeline, including streaming text from an LLM service. ​ Output Frames ​ TTSStartedFrame Signals the start of audio generation. ​ TTSAudioRawFrame Contains generated audio data: ​ audio bytes Raw audio data chunk ​ sample_rate int Audio sample rate, based on the constructor setting ​ num_channels int Number of audio channels (1 for mono) ​ TTSStoppedFrame Signals the completion of audio generation. ​ Methods See the TTS base class methods for additional functionality. ​ Language Support GroqTTSService supports the following languages: Language Code Description Service Codes Language.EN English en ​ Usage Example Copy Ask AI from pipecat.services.groq.tts import GroqTTSService from pipecat.transcriptions.language import Language # Configure service tts = GroqTTSService( api_key = "your-api-key" , model_name = "playai-tts" , voice_id = "Celeste-PlayAI" , params = GroqTTSService.InputParams( language = Language. EN , speed = 1.0 , seed = 42 ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ Frame Flow ​ Metrics Support The service supports metrics collection: Time to First Byte (TTFB) Processing duration ​ Audio Processing Streams audio in chunks Outputs mono audio at the defined sample rate Handles WAV header removal automatically Supports WAV format by default ​ Notes Requires a Groq Cloud API key Streams audio in chunks for efficient processing Automatically handles WAV headers in the response Provides metrics collection Supports configurable speech parameters Google LMNT On this page Overview Installation Configuration Constructor Parameters Input Parameters Input Output Frames TTSStartedFrame TTSAudioRawFrame TTSStoppedFrame Methods Language Support Usage Example Frame Flow Metrics Support Audio Processing Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_lmnt_319ec92a.txt b/tts_lmnt_319ec92a.txt
new file mode 100644
index 0000000000000000000000000000000000000000..108577b2a73e4c824e3ad133b925f975a64552fd
--- /dev/null
+++ b/tts_lmnt_319ec92a.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/lmnt#metrics-support
+Title: LMNT - Pipecat
+==================================================
+
+LMNT - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech LMNT Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview LmntTTSService provides text-to-speech capabilities using LMNT’s WebSocket-based streaming API. It supports real-time audio generation with multiple voices and languages. ​ Installation To use LmntTTSService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[lmnt]" You’ll also need to set up your LMNT API key as an environment variable: LMNT_API_KEY ​ Configuration ​ Constructor Parameters ​ api_key str required Your LMNT API key ​ voice_id str required LMNT voice identifier ​ sample_rate int default: "None" Output audio sample rate in Hz ​ language Language default: "Language.EN" Synthesis language ​ model str default: "aurora" Model identifier ​ text_filter BaseTextFilter default: "None" Modifies text provided to the TTS. Learn more about the available filters. ​ Output Frames ​ Control Frames ​ TTSStartedFrame Frame Signals start of speech synthesis ​ TTSStoppedFrame Frame Signals completion of speech synthesis ​ Audio Frames ​ TTSAudioRawFrame Frame Contains generated audio data with: - PCM audio format (16-bit) - Specified sample rate - Single channel (mono) ​ Error Frames ​ ErrorFrame Frame Contains LMNT API error information ​ Methods See the TTS base class methods for additional functionality. ​ Language Support Supports multiple languages and their variants: Language Code Description Service Code Supported Models Language.DE German de Aurora, Blizzard Language.EN English en Aurora, Blizzard Language.ES Spanish es Aurora, Blizzard Language.FR French fr Aurora, Blizzard Language.HI Hindi hi Aurora, Blizzard Language.ID Indonesian id Blizzard Language.IT Italian it Blizzard Language.JA Japanese ja Blizzard Language.KO Korean ko Aurora, Blizzard Language.NL Dutch nl Blizzard Language.PL Polish pl Blizzard Language.PT Portuguese pt Aurora, Blizzard Language.RU Russian ru Blizzard Language.SV Swedish sv Blizzard Language.TH Thai th Blizzard Language.TR Turkish tr Blizzard Language.UK Ukrainian uk Blizzard Language.VI Vietnamese vi Blizzard Language.ZH Chinese zh Aurora, Blizzard ​ Usage Example Copy Ask AI from pipecat.services.lmnt.tts import LmntTTSService from pipecat.transcriptions.language import Language # Configure service tts = LmntTTSService( api_key = "your-lmnt-api-key" , voice_id = "your-voice-id" , sample_rate = 24000 , language = Language. EN , model = 'blizzard' ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ Frame Flow ​ Metrics Support The service collects processing metrics: Time to First Byte (TTFB) Processing duration Character usage WebSocket connection status ​ Notes Supports streaming synthesis Handles WebSocket lifecycle Provides real-time audio chunks Manages connection state Supports interruptions Thread-safe processing Automatic error handling Includes metrics collection Groq MiniMax On this page Overview Installation Configuration Constructor Parameters Output Frames Control Frames Audio Frames Error Frames Methods Language Support Usage Example Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_lmnt_33747e6b.txt b/tts_lmnt_33747e6b.txt
new file mode 100644
index 0000000000000000000000000000000000000000..aad71569fe87f66aa7225a9f7531b72b316e245a
--- /dev/null
+++ b/tts_lmnt_33747e6b.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/lmnt#notes
+Title: LMNT - Pipecat
+==================================================
+
+LMNT - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech LMNT Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview LmntTTSService provides text-to-speech capabilities using LMNT’s WebSocket-based streaming API. It supports real-time audio generation with multiple voices and languages. ​ Installation To use LmntTTSService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[lmnt]" You’ll also need to set up your LMNT API key as an environment variable: LMNT_API_KEY ​ Configuration ​ Constructor Parameters ​ api_key str required Your LMNT API key ​ voice_id str required LMNT voice identifier ​ sample_rate int default: "None" Output audio sample rate in Hz ​ language Language default: "Language.EN" Synthesis language ​ model str default: "aurora" Model identifier ​ text_filter BaseTextFilter default: "None" Modifies text provided to the TTS. Learn more about the available filters. ​ Output Frames ​ Control Frames ​ TTSStartedFrame Frame Signals start of speech synthesis ​ TTSStoppedFrame Frame Signals completion of speech synthesis ​ Audio Frames ​ TTSAudioRawFrame Frame Contains generated audio data with: - PCM audio format (16-bit) - Specified sample rate - Single channel (mono) ​ Error Frames ​ ErrorFrame Frame Contains LMNT API error information ​ Methods See the TTS base class methods for additional functionality. ​ Language Support Supports multiple languages and their variants: Language Code Description Service Code Supported Models Language.DE German de Aurora, Blizzard Language.EN English en Aurora, Blizzard Language.ES Spanish es Aurora, Blizzard Language.FR French fr Aurora, Blizzard Language.HI Hindi hi Aurora, Blizzard Language.ID Indonesian id Blizzard Language.IT Italian it Blizzard Language.JA Japanese ja Blizzard Language.KO Korean ko Aurora, Blizzard Language.NL Dutch nl Blizzard Language.PL Polish pl Blizzard Language.PT Portuguese pt Aurora, Blizzard Language.RU Russian ru Blizzard Language.SV Swedish sv Blizzard Language.TH Thai th Blizzard Language.TR Turkish tr Blizzard Language.UK Ukrainian uk Blizzard Language.VI Vietnamese vi Blizzard Language.ZH Chinese zh Aurora, Blizzard ​ Usage Example Copy Ask AI from pipecat.services.lmnt.tts import LmntTTSService from pipecat.transcriptions.language import Language # Configure service tts = LmntTTSService( api_key = "your-lmnt-api-key" , voice_id = "your-voice-id" , sample_rate = 24000 , language = Language. EN , model = 'blizzard' ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ Frame Flow ​ Metrics Support The service collects processing metrics: Time to First Byte (TTFB) Processing duration Character usage WebSocket connection status ​ Notes Supports streaming synthesis Handles WebSocket lifecycle Provides real-time audio chunks Manages connection state Supports interruptions Thread-safe processing Automatic error handling Includes metrics collection Groq MiniMax On this page Overview Installation Configuration Constructor Parameters Output Frames Control Frames Audio Frames Error Frames Methods Language Support Usage Example Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_lmnt_3aa0fd3c.txt b/tts_lmnt_3aa0fd3c.txt
new file mode 100644
index 0000000000000000000000000000000000000000..9f91cc95bace8b1bd380bbfc273115d6f67b3451
--- /dev/null
+++ b/tts_lmnt_3aa0fd3c.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/lmnt#param-tts-stopped-frame
+Title: LMNT - Pipecat
+==================================================
+
+LMNT - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech LMNT Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview LmntTTSService provides text-to-speech capabilities using LMNT’s WebSocket-based streaming API. It supports real-time audio generation with multiple voices and languages. ​ Installation To use LmntTTSService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[lmnt]" You’ll also need to set up your LMNT API key as an environment variable: LMNT_API_KEY ​ Configuration ​ Constructor Parameters ​ api_key str required Your LMNT API key ​ voice_id str required LMNT voice identifier ​ sample_rate int default: "None" Output audio sample rate in Hz ​ language Language default: "Language.EN" Synthesis language ​ model str default: "aurora" Model identifier ​ text_filter BaseTextFilter default: "None" Modifies text provided to the TTS. Learn more about the available filters. ​ Output Frames ​ Control Frames ​ TTSStartedFrame Frame Signals start of speech synthesis ​ TTSStoppedFrame Frame Signals completion of speech synthesis ​ Audio Frames ​ TTSAudioRawFrame Frame Contains generated audio data with: - PCM audio format (16-bit) - Specified sample rate - Single channel (mono) ​ Error Frames ​ ErrorFrame Frame Contains LMNT API error information ​ Methods See the TTS base class methods for additional functionality. ​ Language Support Supports multiple languages and their variants: Language Code Description Service Code Supported Models Language.DE German de Aurora, Blizzard Language.EN English en Aurora, Blizzard Language.ES Spanish es Aurora, Blizzard Language.FR French fr Aurora, Blizzard Language.HI Hindi hi Aurora, Blizzard Language.ID Indonesian id Blizzard Language.IT Italian it Blizzard Language.JA Japanese ja Blizzard Language.KO Korean ko Aurora, Blizzard Language.NL Dutch nl Blizzard Language.PL Polish pl Blizzard Language.PT Portuguese pt Aurora, Blizzard Language.RU Russian ru Blizzard Language.SV Swedish sv Blizzard Language.TH Thai th Blizzard Language.TR Turkish tr Blizzard Language.UK Ukrainian uk Blizzard Language.VI Vietnamese vi Blizzard Language.ZH Chinese zh Aurora, Blizzard ​ Usage Example Copy Ask AI from pipecat.services.lmnt.tts import LmntTTSService from pipecat.transcriptions.language import Language # Configure service tts = LmntTTSService( api_key = "your-lmnt-api-key" , voice_id = "your-voice-id" , sample_rate = 24000 , language = Language. EN , model = 'blizzard' ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ Frame Flow ​ Metrics Support The service collects processing metrics: Time to First Byte (TTFB) Processing duration Character usage WebSocket connection status ​ Notes Supports streaming synthesis Handles WebSocket lifecycle Provides real-time audio chunks Manages connection state Supports interruptions Thread-safe processing Automatic error handling Includes metrics collection Groq MiniMax On this page Overview Installation Configuration Constructor Parameters Output Frames Control Frames Audio Frames Error Frames Methods Language Support Usage Example Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_lmnt_479dcae9.txt b/tts_lmnt_479dcae9.txt
new file mode 100644
index 0000000000000000000000000000000000000000..c8d950b670fa351aa32dbc366d4a410831a9e394
--- /dev/null
+++ b/tts_lmnt_479dcae9.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/lmnt#param-language
+Title: LMNT - Pipecat
+==================================================
+
+LMNT - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech LMNT Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview LmntTTSService provides text-to-speech capabilities using LMNT’s WebSocket-based streaming API. It supports real-time audio generation with multiple voices and languages. ​ Installation To use LmntTTSService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[lmnt]" You’ll also need to set up your LMNT API key as an environment variable: LMNT_API_KEY ​ Configuration ​ Constructor Parameters ​ api_key str required Your LMNT API key ​ voice_id str required LMNT voice identifier ​ sample_rate int default: "None" Output audio sample rate in Hz ​ language Language default: "Language.EN" Synthesis language ​ model str default: "aurora" Model identifier ​ text_filter BaseTextFilter default: "None" Modifies text provided to the TTS. Learn more about the available filters. ​ Output Frames ​ Control Frames ​ TTSStartedFrame Frame Signals start of speech synthesis ​ TTSStoppedFrame Frame Signals completion of speech synthesis ​ Audio Frames ​ TTSAudioRawFrame Frame Contains generated audio data with: - PCM audio format (16-bit) - Specified sample rate - Single channel (mono) ​ Error Frames ​ ErrorFrame Frame Contains LMNT API error information ​ Methods See the TTS base class methods for additional functionality. ​ Language Support Supports multiple languages and their variants: Language Code Description Service Code Supported Models Language.DE German de Aurora, Blizzard Language.EN English en Aurora, Blizzard Language.ES Spanish es Aurora, Blizzard Language.FR French fr Aurora, Blizzard Language.HI Hindi hi Aurora, Blizzard Language.ID Indonesian id Blizzard Language.IT Italian it Blizzard Language.JA Japanese ja Blizzard Language.KO Korean ko Aurora, Blizzard Language.NL Dutch nl Blizzard Language.PL Polish pl Blizzard Language.PT Portuguese pt Aurora, Blizzard Language.RU Russian ru Blizzard Language.SV Swedish sv Blizzard Language.TH Thai th Blizzard Language.TR Turkish tr Blizzard Language.UK Ukrainian uk Blizzard Language.VI Vietnamese vi Blizzard Language.ZH Chinese zh Aurora, Blizzard ​ Usage Example Copy Ask AI from pipecat.services.lmnt.tts import LmntTTSService from pipecat.transcriptions.language import Language # Configure service tts = LmntTTSService( api_key = "your-lmnt-api-key" , voice_id = "your-voice-id" , sample_rate = 24000 , language = Language. EN , model = 'blizzard' ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ Frame Flow ​ Metrics Support The service collects processing metrics: Time to First Byte (TTFB) Processing duration Character usage WebSocket connection status ​ Notes Supports streaming synthesis Handles WebSocket lifecycle Provides real-time audio chunks Manages connection state Supports interruptions Thread-safe processing Automatic error handling Includes metrics collection Groq MiniMax On this page Overview Installation Configuration Constructor Parameters Output Frames Control Frames Audio Frames Error Frames Methods Language Support Usage Example Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_lmnt_c11e0ebe.txt b/tts_lmnt_c11e0ebe.txt
new file mode 100644
index 0000000000000000000000000000000000000000..4e92d5df6b4aa7028051d5fda9524d07664d34be
--- /dev/null
+++ b/tts_lmnt_c11e0ebe.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/lmnt#param-sample-rate
+Title: LMNT - Pipecat
+==================================================
+
+LMNT - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech LMNT Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview LmntTTSService provides text-to-speech capabilities using LMNT’s WebSocket-based streaming API. It supports real-time audio generation with multiple voices and languages. ​ Installation To use LmntTTSService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[lmnt]" You’ll also need to set up your LMNT API key as an environment variable: LMNT_API_KEY ​ Configuration ​ Constructor Parameters ​ api_key str required Your LMNT API key ​ voice_id str required LMNT voice identifier ​ sample_rate int default: "None" Output audio sample rate in Hz ​ language Language default: "Language.EN" Synthesis language ​ model str default: "aurora" Model identifier ​ text_filter BaseTextFilter default: "None" Modifies text provided to the TTS. Learn more about the available filters. ​ Output Frames ​ Control Frames ​ TTSStartedFrame Frame Signals start of speech synthesis ​ TTSStoppedFrame Frame Signals completion of speech synthesis ​ Audio Frames ​ TTSAudioRawFrame Frame Contains generated audio data with: - PCM audio format (16-bit) - Specified sample rate - Single channel (mono) ​ Error Frames ​ ErrorFrame Frame Contains LMNT API error information ​ Methods See the TTS base class methods for additional functionality. ​ Language Support Supports multiple languages and their variants: Language Code Description Service Code Supported Models Language.DE German de Aurora, Blizzard Language.EN English en Aurora, Blizzard Language.ES Spanish es Aurora, Blizzard Language.FR French fr Aurora, Blizzard Language.HI Hindi hi Aurora, Blizzard Language.ID Indonesian id Blizzard Language.IT Italian it Blizzard Language.JA Japanese ja Blizzard Language.KO Korean ko Aurora, Blizzard Language.NL Dutch nl Blizzard Language.PL Polish pl Blizzard Language.PT Portuguese pt Aurora, Blizzard Language.RU Russian ru Blizzard Language.SV Swedish sv Blizzard Language.TH Thai th Blizzard Language.TR Turkish tr Blizzard Language.UK Ukrainian uk Blizzard Language.VI Vietnamese vi Blizzard Language.ZH Chinese zh Aurora, Blizzard ​ Usage Example Copy Ask AI from pipecat.services.lmnt.tts import LmntTTSService from pipecat.transcriptions.language import Language # Configure service tts = LmntTTSService( api_key = "your-lmnt-api-key" , voice_id = "your-voice-id" , sample_rate = 24000 , language = Language. EN , model = 'blizzard' ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ Frame Flow ​ Metrics Support The service collects processing metrics: Time to First Byte (TTFB) Processing duration Character usage WebSocket connection status ​ Notes Supports streaming synthesis Handles WebSocket lifecycle Provides real-time audio chunks Manages connection state Supports interruptions Thread-safe processing Automatic error handling Includes metrics collection Groq MiniMax On this page Overview Installation Configuration Constructor Parameters Output Frames Control Frames Audio Frames Error Frames Methods Language Support Usage Example Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_lmnt_d0aa8c9b.txt b/tts_lmnt_d0aa8c9b.txt
new file mode 100644
index 0000000000000000000000000000000000000000..ec312bceea33d6b2e5336aa5ed333579047e5c52
--- /dev/null
+++ b/tts_lmnt_d0aa8c9b.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/lmnt#param-error-frame
+Title: LMNT - Pipecat
+==================================================
+
+LMNT - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech LMNT Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview LmntTTSService provides text-to-speech capabilities using LMNT’s WebSocket-based streaming API. It supports real-time audio generation with multiple voices and languages. ​ Installation To use LmntTTSService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[lmnt]" You’ll also need to set up your LMNT API key as an environment variable: LMNT_API_KEY ​ Configuration ​ Constructor Parameters ​ api_key str required Your LMNT API key ​ voice_id str required LMNT voice identifier ​ sample_rate int default: "None" Output audio sample rate in Hz ​ language Language default: "Language.EN" Synthesis language ​ model str default: "aurora" Model identifier ​ text_filter BaseTextFilter default: "None" Modifies text provided to the TTS. Learn more about the available filters. ​ Output Frames ​ Control Frames ​ TTSStartedFrame Frame Signals start of speech synthesis ​ TTSStoppedFrame Frame Signals completion of speech synthesis ​ Audio Frames ​ TTSAudioRawFrame Frame Contains generated audio data with: - PCM audio format (16-bit) - Specified sample rate - Single channel (mono) ​ Error Frames ​ ErrorFrame Frame Contains LMNT API error information ​ Methods See the TTS base class methods for additional functionality. ​ Language Support Supports multiple languages and their variants: Language Code Description Service Code Supported Models Language.DE German de Aurora, Blizzard Language.EN English en Aurora, Blizzard Language.ES Spanish es Aurora, Blizzard Language.FR French fr Aurora, Blizzard Language.HI Hindi hi Aurora, Blizzard Language.ID Indonesian id Blizzard Language.IT Italian it Blizzard Language.JA Japanese ja Blizzard Language.KO Korean ko Aurora, Blizzard Language.NL Dutch nl Blizzard Language.PL Polish pl Blizzard Language.PT Portuguese pt Aurora, Blizzard Language.RU Russian ru Blizzard Language.SV Swedish sv Blizzard Language.TH Thai th Blizzard Language.TR Turkish tr Blizzard Language.UK Ukrainian uk Blizzard Language.VI Vietnamese vi Blizzard Language.ZH Chinese zh Aurora, Blizzard ​ Usage Example Copy Ask AI from pipecat.services.lmnt.tts import LmntTTSService from pipecat.transcriptions.language import Language # Configure service tts = LmntTTSService( api_key = "your-lmnt-api-key" , voice_id = "your-voice-id" , sample_rate = 24000 , language = Language. EN , model = 'blizzard' ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ Frame Flow ​ Metrics Support The service collects processing metrics: Time to First Byte (TTFB) Processing duration Character usage WebSocket connection status ​ Notes Supports streaming synthesis Handles WebSocket lifecycle Provides real-time audio chunks Manages connection state Supports interruptions Thread-safe processing Automatic error handling Includes metrics collection Groq MiniMax On this page Overview Installation Configuration Constructor Parameters Output Frames Control Frames Audio Frames Error Frames Methods Language Support Usage Example Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_minimax_01ce8137.txt b/tts_minimax_01ce8137.txt
new file mode 100644
index 0000000000000000000000000000000000000000..7f34199b806c73d9b9f353d5621b67f5e05a9db8
--- /dev/null
+++ b/tts_minimax_01ce8137.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/minimax#constructor-parameters
+Title: MiniMax - Pipecat
+==================================================
+
+MiniMax - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech MiniMax Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview MiniMaxHttpTTSService provides text-to-speech capabilities using MiniMax’s T2A (Text-to-Audio) API. It supports multiple voices, emotions, languages, and speech customization options. ​ Installation To use MiniMaxHttpTTSService , no additional dependencies are required. You’ll also need MiniMax API credentials (API key and Group ID). ​ Configuration ​ Constructor Parameters ​ api_key str required MiniMax API key for authentication ​ group_id str required MiniMax Group ID to identify your project ​ model str default: "speech-02-turbo" MiniMax TTS model to use. Available options include: speech-02-hd : HD model with superior rhythm and stability speech-02-turbo : Turbo model with enhanced multilingual capabilities speech-01-hd : Rich voices with expressive emotions speech-01-turbo : Low-latency model with regular updates ​ voice_id str default: "Calm_Woman" MiniMax voice identifier. Options include: Wise_Woman Friendly_Person Inspirational_girl Deep_Voice_Man Calm_Woman Casual_Guy Lively_Girl Patient_Man Young_Knight Determined_Man Lovely_Girl Decent_Boy Imposing_Manner Elegant_Man Abbess Sweet_Girl_2 Exuberant_Girl See the MiniMax documentation for a complete list of available voices. ​ aiohttp_session aiohttp.ClientSession required Aiohttp session for API communication ​ sample_rate int default: "None" Output audio sample rate in Hz ​ params InputParams TTS configuration parameters ​ Input Parameters ​ language Language default: "Language.EN" Language for TTS generation ​ speed float default: "1.0" Speech speed (range: 0.5 to 2.0). Values greater than 1.0 increase speed, less than 1.0 decrease speed. ​ volume float default: "1.0" Speech volume (range: 0 to 10). Values greater than 1.0 increase volume. ​ pitch float default: "0" Pitch adjustment (range: -12 to 12). Positive values raise pitch, negative values lower pitch. ​ emotion str Emotional tone of the speech. Options include: “happy”, “sad”, “angry”, “fearful”, “disgusted”, “surprised”, and “neutral”. ​ english_normalization bool Whether to apply English text normalization, which improves performance in number-reading scenarios at the cost of slightly increased latency. ​ Output Frames ​ Control Frames ​ TTSStartedFrame Frame Signals start of speech synthesis ​ TTSStoppedFrame Frame Signals completion of speech synthesis ​ Audio Frames ​ TTSAudioRawFrame Frame Contains generated audio data with: PCM audio format Sample rate as specified Single channel (mono) ​ Error Frames ​ ErrorFrame Frame Contains MiniMax API error information ​ Methods See the TTS base class methods for additional functionality. ​ Language Support Supports a wide range of languages through the language_boost parameter: Language Code Service Code Description Language.AR Arabic Arabic Language.CS Czech Czech Language.DE German German Language.EL Greek Greek Language.EN English English Language.ES Spanish Spanish Language.FI Finnish Finnish Language.FR French French Language.HI Hindi Hindi Language.ID Indonesian Indonesian Language.IT Italian Italian Language.JA Japanese Japanese Language.KO Korean Korean Language.NL Dutch Dutch Language.PL Polish Polish Language.PT Portuguese Portuguese Language.RO Romanian Romanian Language.RU Russian Russian Language.TH Thai Thai Language.TR Turkish Turkish Language.UK Ukrainian Ukrainian Language.VI Vietnamese Vietnamese Language.YUE Chinese,Yue Chinese (Cantonese) Language.ZH Chinese Chinese (Mandarin) ​ Usage Example Copy Ask AI import aiohttp import os from pipecat.services.minimax.tts import MiniMaxHttpTTSService from pipecat.transcriptions.language import Language async def create_tts_service (): # Create an HTTP session session = aiohttp.ClientSession() # Configure service with credentials tts = MiniMaxHttpTTSService( api_key = os.getenv( "MINIMAX_API_KEY" ), group_id = os.getenv( "MINIMAX_GROUP_ID" ), model = "speech-02-turbo" , voice_id = "Patient_Man" , aiohttp_session = session, params = MiniMaxHttpTTSService.InputParams( language = Language. EN , speed = 1.1 , # Slightly faster speech volume = 1.2 , # Slightly louder pitch = 0 , # Default pitch emotion = "neutral" # Neutral emotional tone ) ) return tts # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ Frame Flow ​ Metrics Support The service collects processing metrics: Time to First Byte (TTFB) Processing duration Character usage ​ Notes Uses streaming audio generation for faster initial response Processes audio in chunks for efficient memory usage Supports real-time applications with low latency Automatically handles API authentication Provides PCM audio compatible with most audio pipelines LMNT Neuphonic On this page Overview Installation Configuration Constructor Parameters Input Parameters Output Frames Control Frames Audio Frames Error Frames Methods Language Support Usage Example Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_minimax_0cc98ff5.txt b/tts_minimax_0cc98ff5.txt
new file mode 100644
index 0000000000000000000000000000000000000000..46711474d9811d56acd3d7d58551761c6335c720
--- /dev/null
+++ b/tts_minimax_0cc98ff5.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/minimax#param-volume
+Title: MiniMax - Pipecat
+==================================================
+
+MiniMax - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech MiniMax Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview MiniMaxHttpTTSService provides text-to-speech capabilities using MiniMax’s T2A (Text-to-Audio) API. It supports multiple voices, emotions, languages, and speech customization options. ​ Installation To use MiniMaxHttpTTSService , no additional dependencies are required. You’ll also need MiniMax API credentials (API key and Group ID). ​ Configuration ​ Constructor Parameters ​ api_key str required MiniMax API key for authentication ​ group_id str required MiniMax Group ID to identify your project ​ model str default: "speech-02-turbo" MiniMax TTS model to use. Available options include: speech-02-hd : HD model with superior rhythm and stability speech-02-turbo : Turbo model with enhanced multilingual capabilities speech-01-hd : Rich voices with expressive emotions speech-01-turbo : Low-latency model with regular updates ​ voice_id str default: "Calm_Woman" MiniMax voice identifier. Options include: Wise_Woman Friendly_Person Inspirational_girl Deep_Voice_Man Calm_Woman Casual_Guy Lively_Girl Patient_Man Young_Knight Determined_Man Lovely_Girl Decent_Boy Imposing_Manner Elegant_Man Abbess Sweet_Girl_2 Exuberant_Girl See the MiniMax documentation for a complete list of available voices. ​ aiohttp_session aiohttp.ClientSession required Aiohttp session for API communication ​ sample_rate int default: "None" Output audio sample rate in Hz ​ params InputParams TTS configuration parameters ​ Input Parameters ​ language Language default: "Language.EN" Language for TTS generation ​ speed float default: "1.0" Speech speed (range: 0.5 to 2.0). Values greater than 1.0 increase speed, less than 1.0 decrease speed. ​ volume float default: "1.0" Speech volume (range: 0 to 10). Values greater than 1.0 increase volume. ​ pitch float default: "0" Pitch adjustment (range: -12 to 12). Positive values raise pitch, negative values lower pitch. ​ emotion str Emotional tone of the speech. Options include: “happy”, “sad”, “angry”, “fearful”, “disgusted”, “surprised”, and “neutral”. ​ english_normalization bool Whether to apply English text normalization, which improves performance in number-reading scenarios at the cost of slightly increased latency. ​ Output Frames ​ Control Frames ​ TTSStartedFrame Frame Signals start of speech synthesis ​ TTSStoppedFrame Frame Signals completion of speech synthesis ​ Audio Frames ​ TTSAudioRawFrame Frame Contains generated audio data with: PCM audio format Sample rate as specified Single channel (mono) ​ Error Frames ​ ErrorFrame Frame Contains MiniMax API error information ​ Methods See the TTS base class methods for additional functionality. ​ Language Support Supports a wide range of languages through the language_boost parameter: Language Code Service Code Description Language.AR Arabic Arabic Language.CS Czech Czech Language.DE German German Language.EL Greek Greek Language.EN English English Language.ES Spanish Spanish Language.FI Finnish Finnish Language.FR French French Language.HI Hindi Hindi Language.ID Indonesian Indonesian Language.IT Italian Italian Language.JA Japanese Japanese Language.KO Korean Korean Language.NL Dutch Dutch Language.PL Polish Polish Language.PT Portuguese Portuguese Language.RO Romanian Romanian Language.RU Russian Russian Language.TH Thai Thai Language.TR Turkish Turkish Language.UK Ukrainian Ukrainian Language.VI Vietnamese Vietnamese Language.YUE Chinese,Yue Chinese (Cantonese) Language.ZH Chinese Chinese (Mandarin) ​ Usage Example Copy Ask AI import aiohttp import os from pipecat.services.minimax.tts import MiniMaxHttpTTSService from pipecat.transcriptions.language import Language async def create_tts_service (): # Create an HTTP session session = aiohttp.ClientSession() # Configure service with credentials tts = MiniMaxHttpTTSService( api_key = os.getenv( "MINIMAX_API_KEY" ), group_id = os.getenv( "MINIMAX_GROUP_ID" ), model = "speech-02-turbo" , voice_id = "Patient_Man" , aiohttp_session = session, params = MiniMaxHttpTTSService.InputParams( language = Language. EN , speed = 1.1 , # Slightly faster speech volume = 1.2 , # Slightly louder pitch = 0 , # Default pitch emotion = "neutral" # Neutral emotional tone ) ) return tts # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ Frame Flow ​ Metrics Support The service collects processing metrics: Time to First Byte (TTFB) Processing duration Character usage ​ Notes Uses streaming audio generation for faster initial response Processes audio in chunks for efficient memory usage Supports real-time applications with low latency Automatically handles API authentication Provides PCM audio compatible with most audio pipelines LMNT Neuphonic On this page Overview Installation Configuration Constructor Parameters Input Parameters Output Frames Control Frames Audio Frames Error Frames Methods Language Support Usage Example Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_minimax_1f4c9697.txt b/tts_minimax_1f4c9697.txt
new file mode 100644
index 0000000000000000000000000000000000000000..704e1a8024b3c1fab541ec7ae976d9219d747a4d
--- /dev/null
+++ b/tts_minimax_1f4c9697.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/minimax#param-aiohttp-session
+Title: MiniMax - Pipecat
+==================================================
+
+MiniMax - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech MiniMax Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview MiniMaxHttpTTSService provides text-to-speech capabilities using MiniMax’s T2A (Text-to-Audio) API. It supports multiple voices, emotions, languages, and speech customization options. ​ Installation To use MiniMaxHttpTTSService , no additional dependencies are required. You’ll also need MiniMax API credentials (API key and Group ID). ​ Configuration ​ Constructor Parameters ​ api_key str required MiniMax API key for authentication ​ group_id str required MiniMax Group ID to identify your project ​ model str default: "speech-02-turbo" MiniMax TTS model to use. Available options include: speech-02-hd : HD model with superior rhythm and stability speech-02-turbo : Turbo model with enhanced multilingual capabilities speech-01-hd : Rich voices with expressive emotions speech-01-turbo : Low-latency model with regular updates ​ voice_id str default: "Calm_Woman" MiniMax voice identifier. Options include: Wise_Woman Friendly_Person Inspirational_girl Deep_Voice_Man Calm_Woman Casual_Guy Lively_Girl Patient_Man Young_Knight Determined_Man Lovely_Girl Decent_Boy Imposing_Manner Elegant_Man Abbess Sweet_Girl_2 Exuberant_Girl See the MiniMax documentation for a complete list of available voices. ​ aiohttp_session aiohttp.ClientSession required Aiohttp session for API communication ​ sample_rate int default: "None" Output audio sample rate in Hz ​ params InputParams TTS configuration parameters ​ Input Parameters ​ language Language default: "Language.EN" Language for TTS generation ​ speed float default: "1.0" Speech speed (range: 0.5 to 2.0). Values greater than 1.0 increase speed, less than 1.0 decrease speed. ​ volume float default: "1.0" Speech volume (range: 0 to 10). Values greater than 1.0 increase volume. ​ pitch float default: "0" Pitch adjustment (range: -12 to 12). Positive values raise pitch, negative values lower pitch. ​ emotion str Emotional tone of the speech. Options include: “happy”, “sad”, “angry”, “fearful”, “disgusted”, “surprised”, and “neutral”. ​ english_normalization bool Whether to apply English text normalization, which improves performance in number-reading scenarios at the cost of slightly increased latency. ​ Output Frames ​ Control Frames ​ TTSStartedFrame Frame Signals start of speech synthesis ​ TTSStoppedFrame Frame Signals completion of speech synthesis ​ Audio Frames ​ TTSAudioRawFrame Frame Contains generated audio data with: PCM audio format Sample rate as specified Single channel (mono) ​ Error Frames ​ ErrorFrame Frame Contains MiniMax API error information ​ Methods See the TTS base class methods for additional functionality. ​ Language Support Supports a wide range of languages through the language_boost parameter: Language Code Service Code Description Language.AR Arabic Arabic Language.CS Czech Czech Language.DE German German Language.EL Greek Greek Language.EN English English Language.ES Spanish Spanish Language.FI Finnish Finnish Language.FR French French Language.HI Hindi Hindi Language.ID Indonesian Indonesian Language.IT Italian Italian Language.JA Japanese Japanese Language.KO Korean Korean Language.NL Dutch Dutch Language.PL Polish Polish Language.PT Portuguese Portuguese Language.RO Romanian Romanian Language.RU Russian Russian Language.TH Thai Thai Language.TR Turkish Turkish Language.UK Ukrainian Ukrainian Language.VI Vietnamese Vietnamese Language.YUE Chinese,Yue Chinese (Cantonese) Language.ZH Chinese Chinese (Mandarin) ​ Usage Example Copy Ask AI import aiohttp import os from pipecat.services.minimax.tts import MiniMaxHttpTTSService from pipecat.transcriptions.language import Language async def create_tts_service (): # Create an HTTP session session = aiohttp.ClientSession() # Configure service with credentials tts = MiniMaxHttpTTSService( api_key = os.getenv( "MINIMAX_API_KEY" ), group_id = os.getenv( "MINIMAX_GROUP_ID" ), model = "speech-02-turbo" , voice_id = "Patient_Man" , aiohttp_session = session, params = MiniMaxHttpTTSService.InputParams( language = Language. EN , speed = 1.1 , # Slightly faster speech volume = 1.2 , # Slightly louder pitch = 0 , # Default pitch emotion = "neutral" # Neutral emotional tone ) ) return tts # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ Frame Flow ​ Metrics Support The service collects processing metrics: Time to First Byte (TTFB) Processing duration Character usage ​ Notes Uses streaming audio generation for faster initial response Processes audio in chunks for efficient memory usage Supports real-time applications with low latency Automatically handles API authentication Provides PCM audio compatible with most audio pipelines LMNT Neuphonic On this page Overview Installation Configuration Constructor Parameters Input Parameters Output Frames Control Frames Audio Frames Error Frames Methods Language Support Usage Example Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_minimax_50d506b1.txt b/tts_minimax_50d506b1.txt
new file mode 100644
index 0000000000000000000000000000000000000000..bde866a6da0bdd62a1e51feb8ea4a97a2388d576
--- /dev/null
+++ b/tts_minimax_50d506b1.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/minimax#param-api-key
+Title: MiniMax - Pipecat
+==================================================
+
+MiniMax - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech MiniMax Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview MiniMaxHttpTTSService provides text-to-speech capabilities using MiniMax’s T2A (Text-to-Audio) API. It supports multiple voices, emotions, languages, and speech customization options. ​ Installation To use MiniMaxHttpTTSService , no additional dependencies are required. You’ll also need MiniMax API credentials (API key and Group ID). ​ Configuration ​ Constructor Parameters ​ api_key str required MiniMax API key for authentication ​ group_id str required MiniMax Group ID to identify your project ​ model str default: "speech-02-turbo" MiniMax TTS model to use. Available options include: speech-02-hd : HD model with superior rhythm and stability speech-02-turbo : Turbo model with enhanced multilingual capabilities speech-01-hd : Rich voices with expressive emotions speech-01-turbo : Low-latency model with regular updates ​ voice_id str default: "Calm_Woman" MiniMax voice identifier. Options include: Wise_Woman Friendly_Person Inspirational_girl Deep_Voice_Man Calm_Woman Casual_Guy Lively_Girl Patient_Man Young_Knight Determined_Man Lovely_Girl Decent_Boy Imposing_Manner Elegant_Man Abbess Sweet_Girl_2 Exuberant_Girl See the MiniMax documentation for a complete list of available voices. ​ aiohttp_session aiohttp.ClientSession required Aiohttp session for API communication ​ sample_rate int default: "None" Output audio sample rate in Hz ​ params InputParams TTS configuration parameters ​ Input Parameters ​ language Language default: "Language.EN" Language for TTS generation ​ speed float default: "1.0" Speech speed (range: 0.5 to 2.0). Values greater than 1.0 increase speed, less than 1.0 decrease speed. ​ volume float default: "1.0" Speech volume (range: 0 to 10). Values greater than 1.0 increase volume. ​ pitch float default: "0" Pitch adjustment (range: -12 to 12). Positive values raise pitch, negative values lower pitch. ​ emotion str Emotional tone of the speech. Options include: “happy”, “sad”, “angry”, “fearful”, “disgusted”, “surprised”, and “neutral”. ​ english_normalization bool Whether to apply English text normalization, which improves performance in number-reading scenarios at the cost of slightly increased latency. ​ Output Frames ​ Control Frames ​ TTSStartedFrame Frame Signals start of speech synthesis ​ TTSStoppedFrame Frame Signals completion of speech synthesis ​ Audio Frames ​ TTSAudioRawFrame Frame Contains generated audio data with: PCM audio format Sample rate as specified Single channel (mono) ​ Error Frames ​ ErrorFrame Frame Contains MiniMax API error information ​ Methods See the TTS base class methods for additional functionality. ​ Language Support Supports a wide range of languages through the language_boost parameter: Language Code Service Code Description Language.AR Arabic Arabic Language.CS Czech Czech Language.DE German German Language.EL Greek Greek Language.EN English English Language.ES Spanish Spanish Language.FI Finnish Finnish Language.FR French French Language.HI Hindi Hindi Language.ID Indonesian Indonesian Language.IT Italian Italian Language.JA Japanese Japanese Language.KO Korean Korean Language.NL Dutch Dutch Language.PL Polish Polish Language.PT Portuguese Portuguese Language.RO Romanian Romanian Language.RU Russian Russian Language.TH Thai Thai Language.TR Turkish Turkish Language.UK Ukrainian Ukrainian Language.VI Vietnamese Vietnamese Language.YUE Chinese,Yue Chinese (Cantonese) Language.ZH Chinese Chinese (Mandarin) ​ Usage Example Copy Ask AI import aiohttp import os from pipecat.services.minimax.tts import MiniMaxHttpTTSService from pipecat.transcriptions.language import Language async def create_tts_service (): # Create an HTTP session session = aiohttp.ClientSession() # Configure service with credentials tts = MiniMaxHttpTTSService( api_key = os.getenv( "MINIMAX_API_KEY" ), group_id = os.getenv( "MINIMAX_GROUP_ID" ), model = "speech-02-turbo" , voice_id = "Patient_Man" , aiohttp_session = session, params = MiniMaxHttpTTSService.InputParams( language = Language. EN , speed = 1.1 , # Slightly faster speech volume = 1.2 , # Slightly louder pitch = 0 , # Default pitch emotion = "neutral" # Neutral emotional tone ) ) return tts # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ Frame Flow ​ Metrics Support The service collects processing metrics: Time to First Byte (TTFB) Processing duration Character usage ​ Notes Uses streaming audio generation for faster initial response Processes audio in chunks for efficient memory usage Supports real-time applications with low latency Automatically handles API authentication Provides PCM audio compatible with most audio pipelines LMNT Neuphonic On this page Overview Installation Configuration Constructor Parameters Input Parameters Output Frames Control Frames Audio Frames Error Frames Methods Language Support Usage Example Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_minimax_7929f6a0.txt b/tts_minimax_7929f6a0.txt
new file mode 100644
index 0000000000000000000000000000000000000000..a6737e4419a59537c55544ebdafbb445030f0dd2
--- /dev/null
+++ b/tts_minimax_7929f6a0.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/minimax#param-params
+Title: MiniMax - Pipecat
+==================================================
+
+MiniMax - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech MiniMax Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview MiniMaxHttpTTSService provides text-to-speech capabilities using MiniMax’s T2A (Text-to-Audio) API. It supports multiple voices, emotions, languages, and speech customization options. ​ Installation To use MiniMaxHttpTTSService , no additional dependencies are required. You’ll also need MiniMax API credentials (API key and Group ID). ​ Configuration ​ Constructor Parameters ​ api_key str required MiniMax API key for authentication ​ group_id str required MiniMax Group ID to identify your project ​ model str default: "speech-02-turbo" MiniMax TTS model to use. Available options include: speech-02-hd : HD model with superior rhythm and stability speech-02-turbo : Turbo model with enhanced multilingual capabilities speech-01-hd : Rich voices with expressive emotions speech-01-turbo : Low-latency model with regular updates ​ voice_id str default: "Calm_Woman" MiniMax voice identifier. Options include: Wise_Woman Friendly_Person Inspirational_girl Deep_Voice_Man Calm_Woman Casual_Guy Lively_Girl Patient_Man Young_Knight Determined_Man Lovely_Girl Decent_Boy Imposing_Manner Elegant_Man Abbess Sweet_Girl_2 Exuberant_Girl See the MiniMax documentation for a complete list of available voices. ​ aiohttp_session aiohttp.ClientSession required Aiohttp session for API communication ​ sample_rate int default: "None" Output audio sample rate in Hz ​ params InputParams TTS configuration parameters ​ Input Parameters ​ language Language default: "Language.EN" Language for TTS generation ​ speed float default: "1.0" Speech speed (range: 0.5 to 2.0). Values greater than 1.0 increase speed, less than 1.0 decrease speed. ​ volume float default: "1.0" Speech volume (range: 0 to 10). Values greater than 1.0 increase volume. ​ pitch float default: "0" Pitch adjustment (range: -12 to 12). Positive values raise pitch, negative values lower pitch. ​ emotion str Emotional tone of the speech. Options include: “happy”, “sad”, “angry”, “fearful”, “disgusted”, “surprised”, and “neutral”. ​ english_normalization bool Whether to apply English text normalization, which improves performance in number-reading scenarios at the cost of slightly increased latency. ​ Output Frames ​ Control Frames ​ TTSStartedFrame Frame Signals start of speech synthesis ​ TTSStoppedFrame Frame Signals completion of speech synthesis ​ Audio Frames ​ TTSAudioRawFrame Frame Contains generated audio data with: PCM audio format Sample rate as specified Single channel (mono) ​ Error Frames ​ ErrorFrame Frame Contains MiniMax API error information ​ Methods See the TTS base class methods for additional functionality. ​ Language Support Supports a wide range of languages through the language_boost parameter: Language Code Service Code Description Language.AR Arabic Arabic Language.CS Czech Czech Language.DE German German Language.EL Greek Greek Language.EN English English Language.ES Spanish Spanish Language.FI Finnish Finnish Language.FR French French Language.HI Hindi Hindi Language.ID Indonesian Indonesian Language.IT Italian Italian Language.JA Japanese Japanese Language.KO Korean Korean Language.NL Dutch Dutch Language.PL Polish Polish Language.PT Portuguese Portuguese Language.RO Romanian Romanian Language.RU Russian Russian Language.TH Thai Thai Language.TR Turkish Turkish Language.UK Ukrainian Ukrainian Language.VI Vietnamese Vietnamese Language.YUE Chinese,Yue Chinese (Cantonese) Language.ZH Chinese Chinese (Mandarin) ​ Usage Example Copy Ask AI import aiohttp import os from pipecat.services.minimax.tts import MiniMaxHttpTTSService from pipecat.transcriptions.language import Language async def create_tts_service (): # Create an HTTP session session = aiohttp.ClientSession() # Configure service with credentials tts = MiniMaxHttpTTSService( api_key = os.getenv( "MINIMAX_API_KEY" ), group_id = os.getenv( "MINIMAX_GROUP_ID" ), model = "speech-02-turbo" , voice_id = "Patient_Man" , aiohttp_session = session, params = MiniMaxHttpTTSService.InputParams( language = Language. EN , speed = 1.1 , # Slightly faster speech volume = 1.2 , # Slightly louder pitch = 0 , # Default pitch emotion = "neutral" # Neutral emotional tone ) ) return tts # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ Frame Flow ​ Metrics Support The service collects processing metrics: Time to First Byte (TTFB) Processing duration Character usage ​ Notes Uses streaming audio generation for faster initial response Processes audio in chunks for efficient memory usage Supports real-time applications with low latency Automatically handles API authentication Provides PCM audio compatible with most audio pipelines LMNT Neuphonic On this page Overview Installation Configuration Constructor Parameters Input Parameters Output Frames Control Frames Audio Frames Error Frames Methods Language Support Usage Example Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_minimax_88f944bd.txt b/tts_minimax_88f944bd.txt
new file mode 100644
index 0000000000000000000000000000000000000000..115f5c0cb11219dc808ed9b60acc15d710dfc5b3
--- /dev/null
+++ b/tts_minimax_88f944bd.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/minimax#overview
+Title: MiniMax - Pipecat
+==================================================
+
+MiniMax - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech MiniMax Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview MiniMaxHttpTTSService provides text-to-speech capabilities using MiniMax’s T2A (Text-to-Audio) API. It supports multiple voices, emotions, languages, and speech customization options. ​ Installation To use MiniMaxHttpTTSService , no additional dependencies are required. You’ll also need MiniMax API credentials (API key and Group ID). ​ Configuration ​ Constructor Parameters ​ api_key str required MiniMax API key for authentication ​ group_id str required MiniMax Group ID to identify your project ​ model str default: "speech-02-turbo" MiniMax TTS model to use. Available options include: speech-02-hd : HD model with superior rhythm and stability speech-02-turbo : Turbo model with enhanced multilingual capabilities speech-01-hd : Rich voices with expressive emotions speech-01-turbo : Low-latency model with regular updates ​ voice_id str default: "Calm_Woman" MiniMax voice identifier. Options include: Wise_Woman Friendly_Person Inspirational_girl Deep_Voice_Man Calm_Woman Casual_Guy Lively_Girl Patient_Man Young_Knight Determined_Man Lovely_Girl Decent_Boy Imposing_Manner Elegant_Man Abbess Sweet_Girl_2 Exuberant_Girl See the MiniMax documentation for a complete list of available voices. ​ aiohttp_session aiohttp.ClientSession required Aiohttp session for API communication ​ sample_rate int default: "None" Output audio sample rate in Hz ​ params InputParams TTS configuration parameters ​ Input Parameters ​ language Language default: "Language.EN" Language for TTS generation ​ speed float default: "1.0" Speech speed (range: 0.5 to 2.0). Values greater than 1.0 increase speed, less than 1.0 decrease speed. ​ volume float default: "1.0" Speech volume (range: 0 to 10). Values greater than 1.0 increase volume. ​ pitch float default: "0" Pitch adjustment (range: -12 to 12). Positive values raise pitch, negative values lower pitch. ​ emotion str Emotional tone of the speech. Options include: “happy”, “sad”, “angry”, “fearful”, “disgusted”, “surprised”, and “neutral”. ​ english_normalization bool Whether to apply English text normalization, which improves performance in number-reading scenarios at the cost of slightly increased latency. ​ Output Frames ​ Control Frames ​ TTSStartedFrame Frame Signals start of speech synthesis ​ TTSStoppedFrame Frame Signals completion of speech synthesis ​ Audio Frames ​ TTSAudioRawFrame Frame Contains generated audio data with: PCM audio format Sample rate as specified Single channel (mono) ​ Error Frames ​ ErrorFrame Frame Contains MiniMax API error information ​ Methods See the TTS base class methods for additional functionality. ​ Language Support Supports a wide range of languages through the language_boost parameter: Language Code Service Code Description Language.AR Arabic Arabic Language.CS Czech Czech Language.DE German German Language.EL Greek Greek Language.EN English English Language.ES Spanish Spanish Language.FI Finnish Finnish Language.FR French French Language.HI Hindi Hindi Language.ID Indonesian Indonesian Language.IT Italian Italian Language.JA Japanese Japanese Language.KO Korean Korean Language.NL Dutch Dutch Language.PL Polish Polish Language.PT Portuguese Portuguese Language.RO Romanian Romanian Language.RU Russian Russian Language.TH Thai Thai Language.TR Turkish Turkish Language.UK Ukrainian Ukrainian Language.VI Vietnamese Vietnamese Language.YUE Chinese,Yue Chinese (Cantonese) Language.ZH Chinese Chinese (Mandarin) ​ Usage Example Copy Ask AI import aiohttp import os from pipecat.services.minimax.tts import MiniMaxHttpTTSService from pipecat.transcriptions.language import Language async def create_tts_service (): # Create an HTTP session session = aiohttp.ClientSession() # Configure service with credentials tts = MiniMaxHttpTTSService( api_key = os.getenv( "MINIMAX_API_KEY" ), group_id = os.getenv( "MINIMAX_GROUP_ID" ), model = "speech-02-turbo" , voice_id = "Patient_Man" , aiohttp_session = session, params = MiniMaxHttpTTSService.InputParams( language = Language. EN , speed = 1.1 , # Slightly faster speech volume = 1.2 , # Slightly louder pitch = 0 , # Default pitch emotion = "neutral" # Neutral emotional tone ) ) return tts # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ Frame Flow ​ Metrics Support The service collects processing metrics: Time to First Byte (TTFB) Processing duration Character usage ​ Notes Uses streaming audio generation for faster initial response Processes audio in chunks for efficient memory usage Supports real-time applications with low latency Automatically handles API authentication Provides PCM audio compatible with most audio pipelines LMNT Neuphonic On this page Overview Installation Configuration Constructor Parameters Input Parameters Output Frames Control Frames Audio Frames Error Frames Methods Language Support Usage Example Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_minimax_89f1088f.txt b/tts_minimax_89f1088f.txt
new file mode 100644
index 0000000000000000000000000000000000000000..e836a8a2fcdd3167d36801ce4f53bbb30c383a69
--- /dev/null
+++ b/tts_minimax_89f1088f.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/minimax#output-frames
+Title: MiniMax - Pipecat
+==================================================
+
+MiniMax - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech MiniMax Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview MiniMaxHttpTTSService provides text-to-speech capabilities using MiniMax’s T2A (Text-to-Audio) API. It supports multiple voices, emotions, languages, and speech customization options. ​ Installation To use MiniMaxHttpTTSService , no additional dependencies are required. You’ll also need MiniMax API credentials (API key and Group ID). ​ Configuration ​ Constructor Parameters ​ api_key str required MiniMax API key for authentication ​ group_id str required MiniMax Group ID to identify your project ​ model str default: "speech-02-turbo" MiniMax TTS model to use. Available options include: speech-02-hd : HD model with superior rhythm and stability speech-02-turbo : Turbo model with enhanced multilingual capabilities speech-01-hd : Rich voices with expressive emotions speech-01-turbo : Low-latency model with regular updates ​ voice_id str default: "Calm_Woman" MiniMax voice identifier. Options include: Wise_Woman Friendly_Person Inspirational_girl Deep_Voice_Man Calm_Woman Casual_Guy Lively_Girl Patient_Man Young_Knight Determined_Man Lovely_Girl Decent_Boy Imposing_Manner Elegant_Man Abbess Sweet_Girl_2 Exuberant_Girl See the MiniMax documentation for a complete list of available voices. ​ aiohttp_session aiohttp.ClientSession required Aiohttp session for API communication ​ sample_rate int default: "None" Output audio sample rate in Hz ​ params InputParams TTS configuration parameters ​ Input Parameters ​ language Language default: "Language.EN" Language for TTS generation ​ speed float default: "1.0" Speech speed (range: 0.5 to 2.0). Values greater than 1.0 increase speed, less than 1.0 decrease speed. ​ volume float default: "1.0" Speech volume (range: 0 to 10). Values greater than 1.0 increase volume. ​ pitch float default: "0" Pitch adjustment (range: -12 to 12). Positive values raise pitch, negative values lower pitch. ​ emotion str Emotional tone of the speech. Options include: “happy”, “sad”, “angry”, “fearful”, “disgusted”, “surprised”, and “neutral”. ​ english_normalization bool Whether to apply English text normalization, which improves performance in number-reading scenarios at the cost of slightly increased latency. ​ Output Frames ​ Control Frames ​ TTSStartedFrame Frame Signals start of speech synthesis ​ TTSStoppedFrame Frame Signals completion of speech synthesis ​ Audio Frames ​ TTSAudioRawFrame Frame Contains generated audio data with: PCM audio format Sample rate as specified Single channel (mono) ​ Error Frames ​ ErrorFrame Frame Contains MiniMax API error information ​ Methods See the TTS base class methods for additional functionality. ​ Language Support Supports a wide range of languages through the language_boost parameter: Language Code Service Code Description Language.AR Arabic Arabic Language.CS Czech Czech Language.DE German German Language.EL Greek Greek Language.EN English English Language.ES Spanish Spanish Language.FI Finnish Finnish Language.FR French French Language.HI Hindi Hindi Language.ID Indonesian Indonesian Language.IT Italian Italian Language.JA Japanese Japanese Language.KO Korean Korean Language.NL Dutch Dutch Language.PL Polish Polish Language.PT Portuguese Portuguese Language.RO Romanian Romanian Language.RU Russian Russian Language.TH Thai Thai Language.TR Turkish Turkish Language.UK Ukrainian Ukrainian Language.VI Vietnamese Vietnamese Language.YUE Chinese,Yue Chinese (Cantonese) Language.ZH Chinese Chinese (Mandarin) ​ Usage Example Copy Ask AI import aiohttp import os from pipecat.services.minimax.tts import MiniMaxHttpTTSService from pipecat.transcriptions.language import Language async def create_tts_service (): # Create an HTTP session session = aiohttp.ClientSession() # Configure service with credentials tts = MiniMaxHttpTTSService( api_key = os.getenv( "MINIMAX_API_KEY" ), group_id = os.getenv( "MINIMAX_GROUP_ID" ), model = "speech-02-turbo" , voice_id = "Patient_Man" , aiohttp_session = session, params = MiniMaxHttpTTSService.InputParams( language = Language. EN , speed = 1.1 , # Slightly faster speech volume = 1.2 , # Slightly louder pitch = 0 , # Default pitch emotion = "neutral" # Neutral emotional tone ) ) return tts # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ Frame Flow ​ Metrics Support The service collects processing metrics: Time to First Byte (TTFB) Processing duration Character usage ​ Notes Uses streaming audio generation for faster initial response Processes audio in chunks for efficient memory usage Supports real-time applications with low latency Automatically handles API authentication Provides PCM audio compatible with most audio pipelines LMNT Neuphonic On this page Overview Installation Configuration Constructor Parameters Input Parameters Output Frames Control Frames Audio Frames Error Frames Methods Language Support Usage Example Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_minimax_d36b0e2b.txt b/tts_minimax_d36b0e2b.txt
new file mode 100644
index 0000000000000000000000000000000000000000..f7f9b71991ba95d7a09ec066066a6cd6c8e127d4
--- /dev/null
+++ b/tts_minimax_d36b0e2b.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/minimax#param-group-id
+Title: MiniMax - Pipecat
+==================================================
+
+MiniMax - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech MiniMax Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview MiniMaxHttpTTSService provides text-to-speech capabilities using MiniMax’s T2A (Text-to-Audio) API. It supports multiple voices, emotions, languages, and speech customization options. ​ Installation To use MiniMaxHttpTTSService , no additional dependencies are required. You’ll also need MiniMax API credentials (API key and Group ID). ​ Configuration ​ Constructor Parameters ​ api_key str required MiniMax API key for authentication ​ group_id str required MiniMax Group ID to identify your project ​ model str default: "speech-02-turbo" MiniMax TTS model to use. Available options include: speech-02-hd : HD model with superior rhythm and stability speech-02-turbo : Turbo model with enhanced multilingual capabilities speech-01-hd : Rich voices with expressive emotions speech-01-turbo : Low-latency model with regular updates ​ voice_id str default: "Calm_Woman" MiniMax voice identifier. Options include: Wise_Woman Friendly_Person Inspirational_girl Deep_Voice_Man Calm_Woman Casual_Guy Lively_Girl Patient_Man Young_Knight Determined_Man Lovely_Girl Decent_Boy Imposing_Manner Elegant_Man Abbess Sweet_Girl_2 Exuberant_Girl See the MiniMax documentation for a complete list of available voices. ​ aiohttp_session aiohttp.ClientSession required Aiohttp session for API communication ​ sample_rate int default: "None" Output audio sample rate in Hz ​ params InputParams TTS configuration parameters ​ Input Parameters ​ language Language default: "Language.EN" Language for TTS generation ​ speed float default: "1.0" Speech speed (range: 0.5 to 2.0). Values greater than 1.0 increase speed, less than 1.0 decrease speed. ​ volume float default: "1.0" Speech volume (range: 0 to 10). Values greater than 1.0 increase volume. ​ pitch float default: "0" Pitch adjustment (range: -12 to 12). Positive values raise pitch, negative values lower pitch. ​ emotion str Emotional tone of the speech. Options include: “happy”, “sad”, “angry”, “fearful”, “disgusted”, “surprised”, and “neutral”. ​ english_normalization bool Whether to apply English text normalization, which improves performance in number-reading scenarios at the cost of slightly increased latency. ​ Output Frames ​ Control Frames ​ TTSStartedFrame Frame Signals start of speech synthesis ​ TTSStoppedFrame Frame Signals completion of speech synthesis ​ Audio Frames ​ TTSAudioRawFrame Frame Contains generated audio data with: PCM audio format Sample rate as specified Single channel (mono) ​ Error Frames ​ ErrorFrame Frame Contains MiniMax API error information ​ Methods See the TTS base class methods for additional functionality. ​ Language Support Supports a wide range of languages through the language_boost parameter: Language Code Service Code Description Language.AR Arabic Arabic Language.CS Czech Czech Language.DE German German Language.EL Greek Greek Language.EN English English Language.ES Spanish Spanish Language.FI Finnish Finnish Language.FR French French Language.HI Hindi Hindi Language.ID Indonesian Indonesian Language.IT Italian Italian Language.JA Japanese Japanese Language.KO Korean Korean Language.NL Dutch Dutch Language.PL Polish Polish Language.PT Portuguese Portuguese Language.RO Romanian Romanian Language.RU Russian Russian Language.TH Thai Thai Language.TR Turkish Turkish Language.UK Ukrainian Ukrainian Language.VI Vietnamese Vietnamese Language.YUE Chinese,Yue Chinese (Cantonese) Language.ZH Chinese Chinese (Mandarin) ​ Usage Example Copy Ask AI import aiohttp import os from pipecat.services.minimax.tts import MiniMaxHttpTTSService from pipecat.transcriptions.language import Language async def create_tts_service (): # Create an HTTP session session = aiohttp.ClientSession() # Configure service with credentials tts = MiniMaxHttpTTSService( api_key = os.getenv( "MINIMAX_API_KEY" ), group_id = os.getenv( "MINIMAX_GROUP_ID" ), model = "speech-02-turbo" , voice_id = "Patient_Man" , aiohttp_session = session, params = MiniMaxHttpTTSService.InputParams( language = Language. EN , speed = 1.1 , # Slightly faster speech volume = 1.2 , # Slightly louder pitch = 0 , # Default pitch emotion = "neutral" # Neutral emotional tone ) ) return tts # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ Frame Flow ​ Metrics Support The service collects processing metrics: Time to First Byte (TTFB) Processing duration Character usage ​ Notes Uses streaming audio generation for faster initial response Processes audio in chunks for efficient memory usage Supports real-time applications with low latency Automatically handles API authentication Provides PCM audio compatible with most audio pipelines LMNT Neuphonic On this page Overview Installation Configuration Constructor Parameters Input Parameters Output Frames Control Frames Audio Frames Error Frames Methods Language Support Usage Example Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_minimax_f469964b.txt b/tts_minimax_f469964b.txt
new file mode 100644
index 0000000000000000000000000000000000000000..d08385a8786e21bbd30beb44c2c9134f21ebc70d
--- /dev/null
+++ b/tts_minimax_f469964b.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/minimax#audio-frames
+Title: MiniMax - Pipecat
+==================================================
+
+MiniMax - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech MiniMax Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview MiniMaxHttpTTSService provides text-to-speech capabilities using MiniMax’s T2A (Text-to-Audio) API. It supports multiple voices, emotions, languages, and speech customization options. ​ Installation To use MiniMaxHttpTTSService , no additional dependencies are required. You’ll also need MiniMax API credentials (API key and Group ID). ​ Configuration ​ Constructor Parameters ​ api_key str required MiniMax API key for authentication ​ group_id str required MiniMax Group ID to identify your project ​ model str default: "speech-02-turbo" MiniMax TTS model to use. Available options include: speech-02-hd : HD model with superior rhythm and stability speech-02-turbo : Turbo model with enhanced multilingual capabilities speech-01-hd : Rich voices with expressive emotions speech-01-turbo : Low-latency model with regular updates ​ voice_id str default: "Calm_Woman" MiniMax voice identifier. Options include: Wise_Woman Friendly_Person Inspirational_girl Deep_Voice_Man Calm_Woman Casual_Guy Lively_Girl Patient_Man Young_Knight Determined_Man Lovely_Girl Decent_Boy Imposing_Manner Elegant_Man Abbess Sweet_Girl_2 Exuberant_Girl See the MiniMax documentation for a complete list of available voices. ​ aiohttp_session aiohttp.ClientSession required Aiohttp session for API communication ​ sample_rate int default: "None" Output audio sample rate in Hz ​ params InputParams TTS configuration parameters ​ Input Parameters ​ language Language default: "Language.EN" Language for TTS generation ​ speed float default: "1.0" Speech speed (range: 0.5 to 2.0). Values greater than 1.0 increase speed, less than 1.0 decrease speed. ​ volume float default: "1.0" Speech volume (range: 0 to 10). Values greater than 1.0 increase volume. ​ pitch float default: "0" Pitch adjustment (range: -12 to 12). Positive values raise pitch, negative values lower pitch. ​ emotion str Emotional tone of the speech. Options include: “happy”, “sad”, “angry”, “fearful”, “disgusted”, “surprised”, and “neutral”. ​ english_normalization bool Whether to apply English text normalization, which improves performance in number-reading scenarios at the cost of slightly increased latency. ​ Output Frames ​ Control Frames ​ TTSStartedFrame Frame Signals start of speech synthesis ​ TTSStoppedFrame Frame Signals completion of speech synthesis ​ Audio Frames ​ TTSAudioRawFrame Frame Contains generated audio data with: PCM audio format Sample rate as specified Single channel (mono) ​ Error Frames ​ ErrorFrame Frame Contains MiniMax API error information ​ Methods See the TTS base class methods for additional functionality. ​ Language Support Supports a wide range of languages through the language_boost parameter: Language Code Service Code Description Language.AR Arabic Arabic Language.CS Czech Czech Language.DE German German Language.EL Greek Greek Language.EN English English Language.ES Spanish Spanish Language.FI Finnish Finnish Language.FR French French Language.HI Hindi Hindi Language.ID Indonesian Indonesian Language.IT Italian Italian Language.JA Japanese Japanese Language.KO Korean Korean Language.NL Dutch Dutch Language.PL Polish Polish Language.PT Portuguese Portuguese Language.RO Romanian Romanian Language.RU Russian Russian Language.TH Thai Thai Language.TR Turkish Turkish Language.UK Ukrainian Ukrainian Language.VI Vietnamese Vietnamese Language.YUE Chinese,Yue Chinese (Cantonese) Language.ZH Chinese Chinese (Mandarin) ​ Usage Example Copy Ask AI import aiohttp import os from pipecat.services.minimax.tts import MiniMaxHttpTTSService from pipecat.transcriptions.language import Language async def create_tts_service (): # Create an HTTP session session = aiohttp.ClientSession() # Configure service with credentials tts = MiniMaxHttpTTSService( api_key = os.getenv( "MINIMAX_API_KEY" ), group_id = os.getenv( "MINIMAX_GROUP_ID" ), model = "speech-02-turbo" , voice_id = "Patient_Man" , aiohttp_session = session, params = MiniMaxHttpTTSService.InputParams( language = Language. EN , speed = 1.1 , # Slightly faster speech volume = 1.2 , # Slightly louder pitch = 0 , # Default pitch emotion = "neutral" # Neutral emotional tone ) ) return tts # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ Frame Flow ​ Metrics Support The service collects processing metrics: Time to First Byte (TTFB) Processing duration Character usage ​ Notes Uses streaming audio generation for faster initial response Processes audio in chunks for efficient memory usage Supports real-time applications with low latency Automatically handles API authentication Provides PCM audio compatible with most audio pipelines LMNT Neuphonic On this page Overview Installation Configuration Constructor Parameters Input Parameters Output Frames Control Frames Audio Frames Error Frames Methods Language Support Usage Example Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_neuphonic_05763667.txt b/tts_neuphonic_05763667.txt
new file mode 100644
index 0000000000000000000000000000000000000000..723900af2e398fba1ea2060cc47cb7dcac124a51
--- /dev/null
+++ b/tts_neuphonic_05763667.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/neuphonic#ttsstoppedframe
+Title: Neuphonic - Pipecat
+==================================================
+
+Neuphonic - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech Neuphonic Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview Neuphonic provides high-quality text-to-speech synthesis through two service implementations: NeuphonicTTSService : WebSocket-based implementation with interruption support NeuphonicHttpTTSService : HTTP-based implementation for simpler use cases Both services support various voices, languages, and customization options. ​ Installation To use Neuphonic TTS services, install the required dependencies: Copy Ask AI pip install "pipecat-ai[neuphonic]" You’ll also need to set up your Neuphonic API key as an environment variable: NEUPHONIC_API_KEY ​ NeuphonicTTSService (WebSocket) ​ Configuration ​ api_key str required Your Neuphonic API key ​ voice_id str default: "None" Voice identifier to use for synthesis ​ url str default: "wss://api.neuphonic.com" Neuphonic WebSocket API endpoint ​ sample_rate int default: "22050" Output audio sample rate in Hz ​ encoding str default: "pcm_linear" Audio encoding format ​ params InputParams default: "InputParams()" Additional configuration parameters ​ InputParams ​ language Language default: "Language.EN" The language for TTS generation ​ speed float default: "1.0" Speech speed multiplier (0.5-2.0) ​ NeuphonicHttpTTSService (HTTP) ​ Configuration ​ api_key str required Your Neuphonic API key ​ voice_id str default: "None" Voice identifier to use for synthesis ​ url str default: "https://api.neuphonic.com" Neuphonic HTTP API endpoint ​ sample_rate int default: "22050" Output audio sample rate in Hz ​ encoding str default: "pcm_linear" Audio encoding format ​ params InputParams default: "InputParams()" Additional configuration parameters (same as WebSocket implementation) ​ Input Both services accept text input through their TTS pipeline. ​ Output Frames ​ TTSStartedFrame Signals the start of audio generation. ​ TTSAudioRawFrame Contains generated audio data: ​ audio bytes Raw audio data chunk ​ sample_rate int Audio sample rate (22050Hz default) ​ num_channels int Number of audio channels (1 for mono) ​ TTSStoppedFrame Signals the completion of audio generation. ​ ErrorFrame Sent when an error occurs during TTS generation: ​ error str Error message describing what went wrong ​ Methods ​ WebSocket Implementation The WebSocket implementation ( NeuphonicTTSService ) inherits from InterruptibleTTSService and provides: Support for interrupting ongoing TTS generation Automatic websocket connection management Keep-alive mechanism for persistent connections Special handling for conversation flows ​ HTTP Implementation The HTTP implementation ( NeuphonicHttpTTSService ) inherits from TTSService and provides: Simpler API integration using HTTP streaming Less overhead for single TTS requests Simplified error handling ​ Language Support Neuphonic TTS supports the following languages: Language Code Description Service Codes Language.EN English en Language.ES Spanish es Language.DE German de Language.NL Dutch nl Language.AR Arabic ar Language.FR French fr Language.PT Portuguese pt Language.RU Russian ru Language.HI Hindi hi Language.ZH Chinese zh Regional variants (e.g., EN_US , ES_ES ) are automatically mapped to their base language. ​ Usage Example ​ WebSocket Implementation Copy Ask AI from pipecat.services.neuphonic.tts import NeuphonicTTSService from pipecat.transcriptions.language import Language # Configure service tts = NeuphonicTTSService( api_key = "your-neuphonic-api-key" , voice_id = "preferred-voice-id" , params = NeuphonicTTSService.InputParams( language = Language. EN , speed = 1.2 ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ HTTP Implementation Copy Ask AI from pipecat.services.neuphonic.tts import NeuphonicHttpTTSService from pipecat.transcriptions.language import Language # Configure service tts = NeuphonicHttpTTSService( api_key = "your-neuphonic-api-key" , voice_id = "preferred-voice-id" , params = NeuphonicHttpTTSService.InputParams( language = Language. ES , speed = 1.0 ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ Metrics Support Both services support metrics collection: Time to First Byte (TTFB) TTS usage metrics Processing duration ​ Audio Processing Configurable sample rate (defaults to 22050Hz) PCM linear encoding Single channel (mono) output Base64 decoding for audio data ​ Notes WebSocket implementation includes a keep-alive mechanism (10-second interval) WebSocket service maintains a persistent connection for faster responses Both services automatically select appropriate language codes The WebSocket implementation pauses frame processing during speech generation to prevent overlapping responses MiniMax NVIDIA Riva On this page Overview Installation NeuphonicTTSService (WebSocket) Configuration InputParams NeuphonicHttpTTSService (HTTP) Configuration Input Output Frames TTSStartedFrame TTSAudioRawFrame TTSStoppedFrame ErrorFrame Methods WebSocket Implementation HTTP Implementation Language Support Usage Example WebSocket Implementation HTTP Implementation Metrics Support Audio Processing Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_neuphonic_0789045f.txt b/tts_neuphonic_0789045f.txt
new file mode 100644
index 0000000000000000000000000000000000000000..0356ca2aaf82e82dd133d0a656e3ffa147b3fcf9
--- /dev/null
+++ b/tts_neuphonic_0789045f.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/neuphonic#param-sample-rate
+Title: Neuphonic - Pipecat
+==================================================
+
+Neuphonic - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech Neuphonic Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview Neuphonic provides high-quality text-to-speech synthesis through two service implementations: NeuphonicTTSService : WebSocket-based implementation with interruption support NeuphonicHttpTTSService : HTTP-based implementation for simpler use cases Both services support various voices, languages, and customization options. ​ Installation To use Neuphonic TTS services, install the required dependencies: Copy Ask AI pip install "pipecat-ai[neuphonic]" You’ll also need to set up your Neuphonic API key as an environment variable: NEUPHONIC_API_KEY ​ NeuphonicTTSService (WebSocket) ​ Configuration ​ api_key str required Your Neuphonic API key ​ voice_id str default: "None" Voice identifier to use for synthesis ​ url str default: "wss://api.neuphonic.com" Neuphonic WebSocket API endpoint ​ sample_rate int default: "22050" Output audio sample rate in Hz ​ encoding str default: "pcm_linear" Audio encoding format ​ params InputParams default: "InputParams()" Additional configuration parameters ​ InputParams ​ language Language default: "Language.EN" The language for TTS generation ​ speed float default: "1.0" Speech speed multiplier (0.5-2.0) ​ NeuphonicHttpTTSService (HTTP) ​ Configuration ​ api_key str required Your Neuphonic API key ​ voice_id str default: "None" Voice identifier to use for synthesis ​ url str default: "https://api.neuphonic.com" Neuphonic HTTP API endpoint ​ sample_rate int default: "22050" Output audio sample rate in Hz ​ encoding str default: "pcm_linear" Audio encoding format ​ params InputParams default: "InputParams()" Additional configuration parameters (same as WebSocket implementation) ​ Input Both services accept text input through their TTS pipeline. ​ Output Frames ​ TTSStartedFrame Signals the start of audio generation. ​ TTSAudioRawFrame Contains generated audio data: ​ audio bytes Raw audio data chunk ​ sample_rate int Audio sample rate (22050Hz default) ​ num_channels int Number of audio channels (1 for mono) ​ TTSStoppedFrame Signals the completion of audio generation. ​ ErrorFrame Sent when an error occurs during TTS generation: ​ error str Error message describing what went wrong ​ Methods ​ WebSocket Implementation The WebSocket implementation ( NeuphonicTTSService ) inherits from InterruptibleTTSService and provides: Support for interrupting ongoing TTS generation Automatic websocket connection management Keep-alive mechanism for persistent connections Special handling for conversation flows ​ HTTP Implementation The HTTP implementation ( NeuphonicHttpTTSService ) inherits from TTSService and provides: Simpler API integration using HTTP streaming Less overhead for single TTS requests Simplified error handling ​ Language Support Neuphonic TTS supports the following languages: Language Code Description Service Codes Language.EN English en Language.ES Spanish es Language.DE German de Language.NL Dutch nl Language.AR Arabic ar Language.FR French fr Language.PT Portuguese pt Language.RU Russian ru Language.HI Hindi hi Language.ZH Chinese zh Regional variants (e.g., EN_US , ES_ES ) are automatically mapped to their base language. ​ Usage Example ​ WebSocket Implementation Copy Ask AI from pipecat.services.neuphonic.tts import NeuphonicTTSService from pipecat.transcriptions.language import Language # Configure service tts = NeuphonicTTSService( api_key = "your-neuphonic-api-key" , voice_id = "preferred-voice-id" , params = NeuphonicTTSService.InputParams( language = Language. EN , speed = 1.2 ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ HTTP Implementation Copy Ask AI from pipecat.services.neuphonic.tts import NeuphonicHttpTTSService from pipecat.transcriptions.language import Language # Configure service tts = NeuphonicHttpTTSService( api_key = "your-neuphonic-api-key" , voice_id = "preferred-voice-id" , params = NeuphonicHttpTTSService.InputParams( language = Language. ES , speed = 1.0 ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ Metrics Support Both services support metrics collection: Time to First Byte (TTFB) TTS usage metrics Processing duration ​ Audio Processing Configurable sample rate (defaults to 22050Hz) PCM linear encoding Single channel (mono) output Base64 decoding for audio data ​ Notes WebSocket implementation includes a keep-alive mechanism (10-second interval) WebSocket service maintains a persistent connection for faster responses Both services automatically select appropriate language codes The WebSocket implementation pauses frame processing during speech generation to prevent overlapping responses MiniMax NVIDIA Riva On this page Overview Installation NeuphonicTTSService (WebSocket) Configuration InputParams NeuphonicHttpTTSService (HTTP) Configuration Input Output Frames TTSStartedFrame TTSAudioRawFrame TTSStoppedFrame ErrorFrame Methods WebSocket Implementation HTTP Implementation Language Support Usage Example WebSocket Implementation HTTP Implementation Metrics Support Audio Processing Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_neuphonic_0cead0c4.txt b/tts_neuphonic_0cead0c4.txt
new file mode 100644
index 0000000000000000000000000000000000000000..843696931a8202b6cc6c87b97ad591868be4133d
--- /dev/null
+++ b/tts_neuphonic_0cead0c4.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/neuphonic#methods
+Title: Neuphonic - Pipecat
+==================================================
+
+Neuphonic - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech Neuphonic Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview Neuphonic provides high-quality text-to-speech synthesis through two service implementations: NeuphonicTTSService : WebSocket-based implementation with interruption support NeuphonicHttpTTSService : HTTP-based implementation for simpler use cases Both services support various voices, languages, and customization options. ​ Installation To use Neuphonic TTS services, install the required dependencies: Copy Ask AI pip install "pipecat-ai[neuphonic]" You’ll also need to set up your Neuphonic API key as an environment variable: NEUPHONIC_API_KEY ​ NeuphonicTTSService (WebSocket) ​ Configuration ​ api_key str required Your Neuphonic API key ​ voice_id str default: "None" Voice identifier to use for synthesis ​ url str default: "wss://api.neuphonic.com" Neuphonic WebSocket API endpoint ​ sample_rate int default: "22050" Output audio sample rate in Hz ​ encoding str default: "pcm_linear" Audio encoding format ​ params InputParams default: "InputParams()" Additional configuration parameters ​ InputParams ​ language Language default: "Language.EN" The language for TTS generation ​ speed float default: "1.0" Speech speed multiplier (0.5-2.0) ​ NeuphonicHttpTTSService (HTTP) ​ Configuration ​ api_key str required Your Neuphonic API key ​ voice_id str default: "None" Voice identifier to use for synthesis ​ url str default: "https://api.neuphonic.com" Neuphonic HTTP API endpoint ​ sample_rate int default: "22050" Output audio sample rate in Hz ​ encoding str default: "pcm_linear" Audio encoding format ​ params InputParams default: "InputParams()" Additional configuration parameters (same as WebSocket implementation) ​ Input Both services accept text input through their TTS pipeline. ​ Output Frames ​ TTSStartedFrame Signals the start of audio generation. ​ TTSAudioRawFrame Contains generated audio data: ​ audio bytes Raw audio data chunk ​ sample_rate int Audio sample rate (22050Hz default) ​ num_channels int Number of audio channels (1 for mono) ​ TTSStoppedFrame Signals the completion of audio generation. ​ ErrorFrame Sent when an error occurs during TTS generation: ​ error str Error message describing what went wrong ​ Methods ​ WebSocket Implementation The WebSocket implementation ( NeuphonicTTSService ) inherits from InterruptibleTTSService and provides: Support for interrupting ongoing TTS generation Automatic websocket connection management Keep-alive mechanism for persistent connections Special handling for conversation flows ​ HTTP Implementation The HTTP implementation ( NeuphonicHttpTTSService ) inherits from TTSService and provides: Simpler API integration using HTTP streaming Less overhead for single TTS requests Simplified error handling ​ Language Support Neuphonic TTS supports the following languages: Language Code Description Service Codes Language.EN English en Language.ES Spanish es Language.DE German de Language.NL Dutch nl Language.AR Arabic ar Language.FR French fr Language.PT Portuguese pt Language.RU Russian ru Language.HI Hindi hi Language.ZH Chinese zh Regional variants (e.g., EN_US , ES_ES ) are automatically mapped to their base language. ​ Usage Example ​ WebSocket Implementation Copy Ask AI from pipecat.services.neuphonic.tts import NeuphonicTTSService from pipecat.transcriptions.language import Language # Configure service tts = NeuphonicTTSService( api_key = "your-neuphonic-api-key" , voice_id = "preferred-voice-id" , params = NeuphonicTTSService.InputParams( language = Language. EN , speed = 1.2 ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ HTTP Implementation Copy Ask AI from pipecat.services.neuphonic.tts import NeuphonicHttpTTSService from pipecat.transcriptions.language import Language # Configure service tts = NeuphonicHttpTTSService( api_key = "your-neuphonic-api-key" , voice_id = "preferred-voice-id" , params = NeuphonicHttpTTSService.InputParams( language = Language. ES , speed = 1.0 ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ Metrics Support Both services support metrics collection: Time to First Byte (TTFB) TTS usage metrics Processing duration ​ Audio Processing Configurable sample rate (defaults to 22050Hz) PCM linear encoding Single channel (mono) output Base64 decoding for audio data ​ Notes WebSocket implementation includes a keep-alive mechanism (10-second interval) WebSocket service maintains a persistent connection for faster responses Both services automatically select appropriate language codes The WebSocket implementation pauses frame processing during speech generation to prevent overlapping responses MiniMax NVIDIA Riva On this page Overview Installation NeuphonicTTSService (WebSocket) Configuration InputParams NeuphonicHttpTTSService (HTTP) Configuration Input Output Frames TTSStartedFrame TTSAudioRawFrame TTSStoppedFrame ErrorFrame Methods WebSocket Implementation HTTP Implementation Language Support Usage Example WebSocket Implementation HTTP Implementation Metrics Support Audio Processing Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_neuphonic_6ee08401.txt b/tts_neuphonic_6ee08401.txt
new file mode 100644
index 0000000000000000000000000000000000000000..53da8f587cc5a953abbc3d5b97052c7e22f534eb
--- /dev/null
+++ b/tts_neuphonic_6ee08401.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/neuphonic#language-support
+Title: Neuphonic - Pipecat
+==================================================
+
+Neuphonic - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech Neuphonic Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview Neuphonic provides high-quality text-to-speech synthesis through two service implementations: NeuphonicTTSService : WebSocket-based implementation with interruption support NeuphonicHttpTTSService : HTTP-based implementation for simpler use cases Both services support various voices, languages, and customization options. ​ Installation To use Neuphonic TTS services, install the required dependencies: Copy Ask AI pip install "pipecat-ai[neuphonic]" You’ll also need to set up your Neuphonic API key as an environment variable: NEUPHONIC_API_KEY ​ NeuphonicTTSService (WebSocket) ​ Configuration ​ api_key str required Your Neuphonic API key ​ voice_id str default: "None" Voice identifier to use for synthesis ​ url str default: "wss://api.neuphonic.com" Neuphonic WebSocket API endpoint ​ sample_rate int default: "22050" Output audio sample rate in Hz ​ encoding str default: "pcm_linear" Audio encoding format ​ params InputParams default: "InputParams()" Additional configuration parameters ​ InputParams ​ language Language default: "Language.EN" The language for TTS generation ​ speed float default: "1.0" Speech speed multiplier (0.5-2.0) ​ NeuphonicHttpTTSService (HTTP) ​ Configuration ​ api_key str required Your Neuphonic API key ​ voice_id str default: "None" Voice identifier to use for synthesis ​ url str default: "https://api.neuphonic.com" Neuphonic HTTP API endpoint ​ sample_rate int default: "22050" Output audio sample rate in Hz ​ encoding str default: "pcm_linear" Audio encoding format ​ params InputParams default: "InputParams()" Additional configuration parameters (same as WebSocket implementation) ​ Input Both services accept text input through their TTS pipeline. ​ Output Frames ​ TTSStartedFrame Signals the start of audio generation. ​ TTSAudioRawFrame Contains generated audio data: ​ audio bytes Raw audio data chunk ​ sample_rate int Audio sample rate (22050Hz default) ​ num_channels int Number of audio channels (1 for mono) ​ TTSStoppedFrame Signals the completion of audio generation. ​ ErrorFrame Sent when an error occurs during TTS generation: ​ error str Error message describing what went wrong ​ Methods ​ WebSocket Implementation The WebSocket implementation ( NeuphonicTTSService ) inherits from InterruptibleTTSService and provides: Support for interrupting ongoing TTS generation Automatic websocket connection management Keep-alive mechanism for persistent connections Special handling for conversation flows ​ HTTP Implementation The HTTP implementation ( NeuphonicHttpTTSService ) inherits from TTSService and provides: Simpler API integration using HTTP streaming Less overhead for single TTS requests Simplified error handling ​ Language Support Neuphonic TTS supports the following languages: Language Code Description Service Codes Language.EN English en Language.ES Spanish es Language.DE German de Language.NL Dutch nl Language.AR Arabic ar Language.FR French fr Language.PT Portuguese pt Language.RU Russian ru Language.HI Hindi hi Language.ZH Chinese zh Regional variants (e.g., EN_US , ES_ES ) are automatically mapped to their base language. ​ Usage Example ​ WebSocket Implementation Copy Ask AI from pipecat.services.neuphonic.tts import NeuphonicTTSService from pipecat.transcriptions.language import Language # Configure service tts = NeuphonicTTSService( api_key = "your-neuphonic-api-key" , voice_id = "preferred-voice-id" , params = NeuphonicTTSService.InputParams( language = Language. EN , speed = 1.2 ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ HTTP Implementation Copy Ask AI from pipecat.services.neuphonic.tts import NeuphonicHttpTTSService from pipecat.transcriptions.language import Language # Configure service tts = NeuphonicHttpTTSService( api_key = "your-neuphonic-api-key" , voice_id = "preferred-voice-id" , params = NeuphonicHttpTTSService.InputParams( language = Language. ES , speed = 1.0 ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ Metrics Support Both services support metrics collection: Time to First Byte (TTFB) TTS usage metrics Processing duration ​ Audio Processing Configurable sample rate (defaults to 22050Hz) PCM linear encoding Single channel (mono) output Base64 decoding for audio data ​ Notes WebSocket implementation includes a keep-alive mechanism (10-second interval) WebSocket service maintains a persistent connection for faster responses Both services automatically select appropriate language codes The WebSocket implementation pauses frame processing during speech generation to prevent overlapping responses MiniMax NVIDIA Riva On this page Overview Installation NeuphonicTTSService (WebSocket) Configuration InputParams NeuphonicHttpTTSService (HTTP) Configuration Input Output Frames TTSStartedFrame TTSAudioRawFrame TTSStoppedFrame ErrorFrame Methods WebSocket Implementation HTTP Implementation Language Support Usage Example WebSocket Implementation HTTP Implementation Metrics Support Audio Processing Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_neuphonic_a44400dd.txt b/tts_neuphonic_a44400dd.txt
new file mode 100644
index 0000000000000000000000000000000000000000..68a71b7ae652bd9e7aee49a93696729d44524261
--- /dev/null
+++ b/tts_neuphonic_a44400dd.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/neuphonic#param-error
+Title: Neuphonic - Pipecat
+==================================================
+
+Neuphonic - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech Neuphonic Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview Neuphonic provides high-quality text-to-speech synthesis through two service implementations: NeuphonicTTSService : WebSocket-based implementation with interruption support NeuphonicHttpTTSService : HTTP-based implementation for simpler use cases Both services support various voices, languages, and customization options. ​ Installation To use Neuphonic TTS services, install the required dependencies: Copy Ask AI pip install "pipecat-ai[neuphonic]" You’ll also need to set up your Neuphonic API key as an environment variable: NEUPHONIC_API_KEY ​ NeuphonicTTSService (WebSocket) ​ Configuration ​ api_key str required Your Neuphonic API key ​ voice_id str default: "None" Voice identifier to use for synthesis ​ url str default: "wss://api.neuphonic.com" Neuphonic WebSocket API endpoint ​ sample_rate int default: "22050" Output audio sample rate in Hz ​ encoding str default: "pcm_linear" Audio encoding format ​ params InputParams default: "InputParams()" Additional configuration parameters ​ InputParams ​ language Language default: "Language.EN" The language for TTS generation ​ speed float default: "1.0" Speech speed multiplier (0.5-2.0) ​ NeuphonicHttpTTSService (HTTP) ​ Configuration ​ api_key str required Your Neuphonic API key ​ voice_id str default: "None" Voice identifier to use for synthesis ​ url str default: "https://api.neuphonic.com" Neuphonic HTTP API endpoint ​ sample_rate int default: "22050" Output audio sample rate in Hz ​ encoding str default: "pcm_linear" Audio encoding format ​ params InputParams default: "InputParams()" Additional configuration parameters (same as WebSocket implementation) ​ Input Both services accept text input through their TTS pipeline. ​ Output Frames ​ TTSStartedFrame Signals the start of audio generation. ​ TTSAudioRawFrame Contains generated audio data: ​ audio bytes Raw audio data chunk ​ sample_rate int Audio sample rate (22050Hz default) ​ num_channels int Number of audio channels (1 for mono) ​ TTSStoppedFrame Signals the completion of audio generation. ​ ErrorFrame Sent when an error occurs during TTS generation: ​ error str Error message describing what went wrong ​ Methods ​ WebSocket Implementation The WebSocket implementation ( NeuphonicTTSService ) inherits from InterruptibleTTSService and provides: Support for interrupting ongoing TTS generation Automatic websocket connection management Keep-alive mechanism for persistent connections Special handling for conversation flows ​ HTTP Implementation The HTTP implementation ( NeuphonicHttpTTSService ) inherits from TTSService and provides: Simpler API integration using HTTP streaming Less overhead for single TTS requests Simplified error handling ​ Language Support Neuphonic TTS supports the following languages: Language Code Description Service Codes Language.EN English en Language.ES Spanish es Language.DE German de Language.NL Dutch nl Language.AR Arabic ar Language.FR French fr Language.PT Portuguese pt Language.RU Russian ru Language.HI Hindi hi Language.ZH Chinese zh Regional variants (e.g., EN_US , ES_ES ) are automatically mapped to their base language. ​ Usage Example ​ WebSocket Implementation Copy Ask AI from pipecat.services.neuphonic.tts import NeuphonicTTSService from pipecat.transcriptions.language import Language # Configure service tts = NeuphonicTTSService( api_key = "your-neuphonic-api-key" , voice_id = "preferred-voice-id" , params = NeuphonicTTSService.InputParams( language = Language. EN , speed = 1.2 ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ HTTP Implementation Copy Ask AI from pipecat.services.neuphonic.tts import NeuphonicHttpTTSService from pipecat.transcriptions.language import Language # Configure service tts = NeuphonicHttpTTSService( api_key = "your-neuphonic-api-key" , voice_id = "preferred-voice-id" , params = NeuphonicHttpTTSService.InputParams( language = Language. ES , speed = 1.0 ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ Metrics Support Both services support metrics collection: Time to First Byte (TTFB) TTS usage metrics Processing duration ​ Audio Processing Configurable sample rate (defaults to 22050Hz) PCM linear encoding Single channel (mono) output Base64 decoding for audio data ​ Notes WebSocket implementation includes a keep-alive mechanism (10-second interval) WebSocket service maintains a persistent connection for faster responses Both services automatically select appropriate language codes The WebSocket implementation pauses frame processing during speech generation to prevent overlapping responses MiniMax NVIDIA Riva On this page Overview Installation NeuphonicTTSService (WebSocket) Configuration InputParams NeuphonicHttpTTSService (HTTP) Configuration Input Output Frames TTSStartedFrame TTSAudioRawFrame TTSStoppedFrame ErrorFrame Methods WebSocket Implementation HTTP Implementation Language Support Usage Example WebSocket Implementation HTTP Implementation Metrics Support Audio Processing Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_neuphonic_a87ade73.txt b/tts_neuphonic_a87ade73.txt
new file mode 100644
index 0000000000000000000000000000000000000000..d5e44d53793bcae823c1b0d1d7034dd8191d3eb1
--- /dev/null
+++ b/tts_neuphonic_a87ade73.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/neuphonic#neuphonichttpttsservice-http
+Title: Neuphonic - Pipecat
+==================================================
+
+Neuphonic - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech Neuphonic Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview Neuphonic provides high-quality text-to-speech synthesis through two service implementations: NeuphonicTTSService : WebSocket-based implementation with interruption support NeuphonicHttpTTSService : HTTP-based implementation for simpler use cases Both services support various voices, languages, and customization options. ​ Installation To use Neuphonic TTS services, install the required dependencies: Copy Ask AI pip install "pipecat-ai[neuphonic]" You’ll also need to set up your Neuphonic API key as an environment variable: NEUPHONIC_API_KEY ​ NeuphonicTTSService (WebSocket) ​ Configuration ​ api_key str required Your Neuphonic API key ​ voice_id str default: "None" Voice identifier to use for synthesis ​ url str default: "wss://api.neuphonic.com" Neuphonic WebSocket API endpoint ​ sample_rate int default: "22050" Output audio sample rate in Hz ​ encoding str default: "pcm_linear" Audio encoding format ​ params InputParams default: "InputParams()" Additional configuration parameters ​ InputParams ​ language Language default: "Language.EN" The language for TTS generation ​ speed float default: "1.0" Speech speed multiplier (0.5-2.0) ​ NeuphonicHttpTTSService (HTTP) ​ Configuration ​ api_key str required Your Neuphonic API key ​ voice_id str default: "None" Voice identifier to use for synthesis ​ url str default: "https://api.neuphonic.com" Neuphonic HTTP API endpoint ​ sample_rate int default: "22050" Output audio sample rate in Hz ​ encoding str default: "pcm_linear" Audio encoding format ​ params InputParams default: "InputParams()" Additional configuration parameters (same as WebSocket implementation) ​ Input Both services accept text input through their TTS pipeline. ​ Output Frames ​ TTSStartedFrame Signals the start of audio generation. ​ TTSAudioRawFrame Contains generated audio data: ​ audio bytes Raw audio data chunk ​ sample_rate int Audio sample rate (22050Hz default) ​ num_channels int Number of audio channels (1 for mono) ​ TTSStoppedFrame Signals the completion of audio generation. ​ ErrorFrame Sent when an error occurs during TTS generation: ​ error str Error message describing what went wrong ​ Methods ​ WebSocket Implementation The WebSocket implementation ( NeuphonicTTSService ) inherits from InterruptibleTTSService and provides: Support for interrupting ongoing TTS generation Automatic websocket connection management Keep-alive mechanism for persistent connections Special handling for conversation flows ​ HTTP Implementation The HTTP implementation ( NeuphonicHttpTTSService ) inherits from TTSService and provides: Simpler API integration using HTTP streaming Less overhead for single TTS requests Simplified error handling ​ Language Support Neuphonic TTS supports the following languages: Language Code Description Service Codes Language.EN English en Language.ES Spanish es Language.DE German de Language.NL Dutch nl Language.AR Arabic ar Language.FR French fr Language.PT Portuguese pt Language.RU Russian ru Language.HI Hindi hi Language.ZH Chinese zh Regional variants (e.g., EN_US , ES_ES ) are automatically mapped to their base language. ​ Usage Example ​ WebSocket Implementation Copy Ask AI from pipecat.services.neuphonic.tts import NeuphonicTTSService from pipecat.transcriptions.language import Language # Configure service tts = NeuphonicTTSService( api_key = "your-neuphonic-api-key" , voice_id = "preferred-voice-id" , params = NeuphonicTTSService.InputParams( language = Language. EN , speed = 1.2 ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ HTTP Implementation Copy Ask AI from pipecat.services.neuphonic.tts import NeuphonicHttpTTSService from pipecat.transcriptions.language import Language # Configure service tts = NeuphonicHttpTTSService( api_key = "your-neuphonic-api-key" , voice_id = "preferred-voice-id" , params = NeuphonicHttpTTSService.InputParams( language = Language. ES , speed = 1.0 ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ Metrics Support Both services support metrics collection: Time to First Byte (TTFB) TTS usage metrics Processing duration ​ Audio Processing Configurable sample rate (defaults to 22050Hz) PCM linear encoding Single channel (mono) output Base64 decoding for audio data ​ Notes WebSocket implementation includes a keep-alive mechanism (10-second interval) WebSocket service maintains a persistent connection for faster responses Both services automatically select appropriate language codes The WebSocket implementation pauses frame processing during speech generation to prevent overlapping responses MiniMax NVIDIA Riva On this page Overview Installation NeuphonicTTSService (WebSocket) Configuration InputParams NeuphonicHttpTTSService (HTTP) Configuration Input Output Frames TTSStartedFrame TTSAudioRawFrame TTSStoppedFrame ErrorFrame Methods WebSocket Implementation HTTP Implementation Language Support Usage Example WebSocket Implementation HTTP Implementation Metrics Support Audio Processing Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_neuphonic_c600746a.txt b/tts_neuphonic_c600746a.txt
new file mode 100644
index 0000000000000000000000000000000000000000..6ffd4e9636fe04bf1daa8714476169a8053ad2e7
--- /dev/null
+++ b/tts_neuphonic_c600746a.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/neuphonic#param-params
+Title: Neuphonic - Pipecat
+==================================================
+
+Neuphonic - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech Neuphonic Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview Neuphonic provides high-quality text-to-speech synthesis through two service implementations: NeuphonicTTSService : WebSocket-based implementation with interruption support NeuphonicHttpTTSService : HTTP-based implementation for simpler use cases Both services support various voices, languages, and customization options. ​ Installation To use Neuphonic TTS services, install the required dependencies: Copy Ask AI pip install "pipecat-ai[neuphonic]" You’ll also need to set up your Neuphonic API key as an environment variable: NEUPHONIC_API_KEY ​ NeuphonicTTSService (WebSocket) ​ Configuration ​ api_key str required Your Neuphonic API key ​ voice_id str default: "None" Voice identifier to use for synthesis ​ url str default: "wss://api.neuphonic.com" Neuphonic WebSocket API endpoint ​ sample_rate int default: "22050" Output audio sample rate in Hz ​ encoding str default: "pcm_linear" Audio encoding format ​ params InputParams default: "InputParams()" Additional configuration parameters ​ InputParams ​ language Language default: "Language.EN" The language for TTS generation ​ speed float default: "1.0" Speech speed multiplier (0.5-2.0) ​ NeuphonicHttpTTSService (HTTP) ​ Configuration ​ api_key str required Your Neuphonic API key ​ voice_id str default: "None" Voice identifier to use for synthesis ​ url str default: "https://api.neuphonic.com" Neuphonic HTTP API endpoint ​ sample_rate int default: "22050" Output audio sample rate in Hz ​ encoding str default: "pcm_linear" Audio encoding format ​ params InputParams default: "InputParams()" Additional configuration parameters (same as WebSocket implementation) ​ Input Both services accept text input through their TTS pipeline. ​ Output Frames ​ TTSStartedFrame Signals the start of audio generation. ​ TTSAudioRawFrame Contains generated audio data: ​ audio bytes Raw audio data chunk ​ sample_rate int Audio sample rate (22050Hz default) ​ num_channels int Number of audio channels (1 for mono) ​ TTSStoppedFrame Signals the completion of audio generation. ​ ErrorFrame Sent when an error occurs during TTS generation: ​ error str Error message describing what went wrong ​ Methods ​ WebSocket Implementation The WebSocket implementation ( NeuphonicTTSService ) inherits from InterruptibleTTSService and provides: Support for interrupting ongoing TTS generation Automatic websocket connection management Keep-alive mechanism for persistent connections Special handling for conversation flows ​ HTTP Implementation The HTTP implementation ( NeuphonicHttpTTSService ) inherits from TTSService and provides: Simpler API integration using HTTP streaming Less overhead for single TTS requests Simplified error handling ​ Language Support Neuphonic TTS supports the following languages: Language Code Description Service Codes Language.EN English en Language.ES Spanish es Language.DE German de Language.NL Dutch nl Language.AR Arabic ar Language.FR French fr Language.PT Portuguese pt Language.RU Russian ru Language.HI Hindi hi Language.ZH Chinese zh Regional variants (e.g., EN_US , ES_ES ) are automatically mapped to their base language. ​ Usage Example ​ WebSocket Implementation Copy Ask AI from pipecat.services.neuphonic.tts import NeuphonicTTSService from pipecat.transcriptions.language import Language # Configure service tts = NeuphonicTTSService( api_key = "your-neuphonic-api-key" , voice_id = "preferred-voice-id" , params = NeuphonicTTSService.InputParams( language = Language. EN , speed = 1.2 ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ HTTP Implementation Copy Ask AI from pipecat.services.neuphonic.tts import NeuphonicHttpTTSService from pipecat.transcriptions.language import Language # Configure service tts = NeuphonicHttpTTSService( api_key = "your-neuphonic-api-key" , voice_id = "preferred-voice-id" , params = NeuphonicHttpTTSService.InputParams( language = Language. ES , speed = 1.0 ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ Metrics Support Both services support metrics collection: Time to First Byte (TTFB) TTS usage metrics Processing duration ​ Audio Processing Configurable sample rate (defaults to 22050Hz) PCM linear encoding Single channel (mono) output Base64 decoding for audio data ​ Notes WebSocket implementation includes a keep-alive mechanism (10-second interval) WebSocket service maintains a persistent connection for faster responses Both services automatically select appropriate language codes The WebSocket implementation pauses frame processing during speech generation to prevent overlapping responses MiniMax NVIDIA Riva On this page Overview Installation NeuphonicTTSService (WebSocket) Configuration InputParams NeuphonicHttpTTSService (HTTP) Configuration Input Output Frames TTSStartedFrame TTSAudioRawFrame TTSStoppedFrame ErrorFrame Methods WebSocket Implementation HTTP Implementation Language Support Usage Example WebSocket Implementation HTTP Implementation Metrics Support Audio Processing Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_neuphonic_d4cb46b6.txt b/tts_neuphonic_d4cb46b6.txt
new file mode 100644
index 0000000000000000000000000000000000000000..3435e54d0615cf80fe45f78c79fd6c60c51ee43e
--- /dev/null
+++ b/tts_neuphonic_d4cb46b6.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/neuphonic#param-num-channels
+Title: Neuphonic - Pipecat
+==================================================
+
+Neuphonic - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech Neuphonic Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview Neuphonic provides high-quality text-to-speech synthesis through two service implementations: NeuphonicTTSService : WebSocket-based implementation with interruption support NeuphonicHttpTTSService : HTTP-based implementation for simpler use cases Both services support various voices, languages, and customization options. ​ Installation To use Neuphonic TTS services, install the required dependencies: Copy Ask AI pip install "pipecat-ai[neuphonic]" You’ll also need to set up your Neuphonic API key as an environment variable: NEUPHONIC_API_KEY ​ NeuphonicTTSService (WebSocket) ​ Configuration ​ api_key str required Your Neuphonic API key ​ voice_id str default: "None" Voice identifier to use for synthesis ​ url str default: "wss://api.neuphonic.com" Neuphonic WebSocket API endpoint ​ sample_rate int default: "22050" Output audio sample rate in Hz ​ encoding str default: "pcm_linear" Audio encoding format ​ params InputParams default: "InputParams()" Additional configuration parameters ​ InputParams ​ language Language default: "Language.EN" The language for TTS generation ​ speed float default: "1.0" Speech speed multiplier (0.5-2.0) ​ NeuphonicHttpTTSService (HTTP) ​ Configuration ​ api_key str required Your Neuphonic API key ​ voice_id str default: "None" Voice identifier to use for synthesis ​ url str default: "https://api.neuphonic.com" Neuphonic HTTP API endpoint ​ sample_rate int default: "22050" Output audio sample rate in Hz ​ encoding str default: "pcm_linear" Audio encoding format ​ params InputParams default: "InputParams()" Additional configuration parameters (same as WebSocket implementation) ​ Input Both services accept text input through their TTS pipeline. ​ Output Frames ​ TTSStartedFrame Signals the start of audio generation. ​ TTSAudioRawFrame Contains generated audio data: ​ audio bytes Raw audio data chunk ​ sample_rate int Audio sample rate (22050Hz default) ​ num_channels int Number of audio channels (1 for mono) ​ TTSStoppedFrame Signals the completion of audio generation. ​ ErrorFrame Sent when an error occurs during TTS generation: ​ error str Error message describing what went wrong ​ Methods ​ WebSocket Implementation The WebSocket implementation ( NeuphonicTTSService ) inherits from InterruptibleTTSService and provides: Support for interrupting ongoing TTS generation Automatic websocket connection management Keep-alive mechanism for persistent connections Special handling for conversation flows ​ HTTP Implementation The HTTP implementation ( NeuphonicHttpTTSService ) inherits from TTSService and provides: Simpler API integration using HTTP streaming Less overhead for single TTS requests Simplified error handling ​ Language Support Neuphonic TTS supports the following languages: Language Code Description Service Codes Language.EN English en Language.ES Spanish es Language.DE German de Language.NL Dutch nl Language.AR Arabic ar Language.FR French fr Language.PT Portuguese pt Language.RU Russian ru Language.HI Hindi hi Language.ZH Chinese zh Regional variants (e.g., EN_US , ES_ES ) are automatically mapped to their base language. ​ Usage Example ​ WebSocket Implementation Copy Ask AI from pipecat.services.neuphonic.tts import NeuphonicTTSService from pipecat.transcriptions.language import Language # Configure service tts = NeuphonicTTSService( api_key = "your-neuphonic-api-key" , voice_id = "preferred-voice-id" , params = NeuphonicTTSService.InputParams( language = Language. EN , speed = 1.2 ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ HTTP Implementation Copy Ask AI from pipecat.services.neuphonic.tts import NeuphonicHttpTTSService from pipecat.transcriptions.language import Language # Configure service tts = NeuphonicHttpTTSService( api_key = "your-neuphonic-api-key" , voice_id = "preferred-voice-id" , params = NeuphonicHttpTTSService.InputParams( language = Language. ES , speed = 1.0 ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ Metrics Support Both services support metrics collection: Time to First Byte (TTFB) TTS usage metrics Processing duration ​ Audio Processing Configurable sample rate (defaults to 22050Hz) PCM linear encoding Single channel (mono) output Base64 decoding for audio data ​ Notes WebSocket implementation includes a keep-alive mechanism (10-second interval) WebSocket service maintains a persistent connection for faster responses Both services automatically select appropriate language codes The WebSocket implementation pauses frame processing during speech generation to prevent overlapping responses MiniMax NVIDIA Riva On this page Overview Installation NeuphonicTTSService (WebSocket) Configuration InputParams NeuphonicHttpTTSService (HTTP) Configuration Input Output Frames TTSStartedFrame TTSAudioRawFrame TTSStoppedFrame ErrorFrame Methods WebSocket Implementation HTTP Implementation Language Support Usage Example WebSocket Implementation HTTP Implementation Metrics Support Audio Processing Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_neuphonic_e1df426c.txt b/tts_neuphonic_e1df426c.txt
new file mode 100644
index 0000000000000000000000000000000000000000..629cc1bc5ae4071767244339d8367366a7cbb39a
--- /dev/null
+++ b/tts_neuphonic_e1df426c.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/neuphonic#neuphonicttsservice-websocket
+Title: Neuphonic - Pipecat
+==================================================
+
+Neuphonic - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech Neuphonic Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview Neuphonic provides high-quality text-to-speech synthesis through two service implementations: NeuphonicTTSService : WebSocket-based implementation with interruption support NeuphonicHttpTTSService : HTTP-based implementation for simpler use cases Both services support various voices, languages, and customization options. ​ Installation To use Neuphonic TTS services, install the required dependencies: Copy Ask AI pip install "pipecat-ai[neuphonic]" You’ll also need to set up your Neuphonic API key as an environment variable: NEUPHONIC_API_KEY ​ NeuphonicTTSService (WebSocket) ​ Configuration ​ api_key str required Your Neuphonic API key ​ voice_id str default: "None" Voice identifier to use for synthesis ​ url str default: "wss://api.neuphonic.com" Neuphonic WebSocket API endpoint ​ sample_rate int default: "22050" Output audio sample rate in Hz ​ encoding str default: "pcm_linear" Audio encoding format ​ params InputParams default: "InputParams()" Additional configuration parameters ​ InputParams ​ language Language default: "Language.EN" The language for TTS generation ​ speed float default: "1.0" Speech speed multiplier (0.5-2.0) ​ NeuphonicHttpTTSService (HTTP) ​ Configuration ​ api_key str required Your Neuphonic API key ​ voice_id str default: "None" Voice identifier to use for synthesis ​ url str default: "https://api.neuphonic.com" Neuphonic HTTP API endpoint ​ sample_rate int default: "22050" Output audio sample rate in Hz ​ encoding str default: "pcm_linear" Audio encoding format ​ params InputParams default: "InputParams()" Additional configuration parameters (same as WebSocket implementation) ​ Input Both services accept text input through their TTS pipeline. ​ Output Frames ​ TTSStartedFrame Signals the start of audio generation. ​ TTSAudioRawFrame Contains generated audio data: ​ audio bytes Raw audio data chunk ​ sample_rate int Audio sample rate (22050Hz default) ​ num_channels int Number of audio channels (1 for mono) ​ TTSStoppedFrame Signals the completion of audio generation. ​ ErrorFrame Sent when an error occurs during TTS generation: ​ error str Error message describing what went wrong ​ Methods ​ WebSocket Implementation The WebSocket implementation ( NeuphonicTTSService ) inherits from InterruptibleTTSService and provides: Support for interrupting ongoing TTS generation Automatic websocket connection management Keep-alive mechanism for persistent connections Special handling for conversation flows ​ HTTP Implementation The HTTP implementation ( NeuphonicHttpTTSService ) inherits from TTSService and provides: Simpler API integration using HTTP streaming Less overhead for single TTS requests Simplified error handling ​ Language Support Neuphonic TTS supports the following languages: Language Code Description Service Codes Language.EN English en Language.ES Spanish es Language.DE German de Language.NL Dutch nl Language.AR Arabic ar Language.FR French fr Language.PT Portuguese pt Language.RU Russian ru Language.HI Hindi hi Language.ZH Chinese zh Regional variants (e.g., EN_US , ES_ES ) are automatically mapped to their base language. ​ Usage Example ​ WebSocket Implementation Copy Ask AI from pipecat.services.neuphonic.tts import NeuphonicTTSService from pipecat.transcriptions.language import Language # Configure service tts = NeuphonicTTSService( api_key = "your-neuphonic-api-key" , voice_id = "preferred-voice-id" , params = NeuphonicTTSService.InputParams( language = Language. EN , speed = 1.2 ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ HTTP Implementation Copy Ask AI from pipecat.services.neuphonic.tts import NeuphonicHttpTTSService from pipecat.transcriptions.language import Language # Configure service tts = NeuphonicHttpTTSService( api_key = "your-neuphonic-api-key" , voice_id = "preferred-voice-id" , params = NeuphonicHttpTTSService.InputParams( language = Language. ES , speed = 1.0 ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ Metrics Support Both services support metrics collection: Time to First Byte (TTFB) TTS usage metrics Processing duration ​ Audio Processing Configurable sample rate (defaults to 22050Hz) PCM linear encoding Single channel (mono) output Base64 decoding for audio data ​ Notes WebSocket implementation includes a keep-alive mechanism (10-second interval) WebSocket service maintains a persistent connection for faster responses Both services automatically select appropriate language codes The WebSocket implementation pauses frame processing during speech generation to prevent overlapping responses MiniMax NVIDIA Riva On this page Overview Installation NeuphonicTTSService (WebSocket) Configuration InputParams NeuphonicHttpTTSService (HTTP) Configuration Input Output Frames TTSStartedFrame TTSAudioRawFrame TTSStoppedFrame ErrorFrame Methods WebSocket Implementation HTTP Implementation Language Support Usage Example WebSocket Implementation HTTP Implementation Metrics Support Audio Processing Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_neuphonic_e59433e9.txt b/tts_neuphonic_e59433e9.txt
new file mode 100644
index 0000000000000000000000000000000000000000..3390bea0b4dfb2646a4186fccba5089273825612
--- /dev/null
+++ b/tts_neuphonic_e59433e9.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/neuphonic#notes
+Title: Neuphonic - Pipecat
+==================================================
+
+Neuphonic - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech Neuphonic Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview Neuphonic provides high-quality text-to-speech synthesis through two service implementations: NeuphonicTTSService : WebSocket-based implementation with interruption support NeuphonicHttpTTSService : HTTP-based implementation for simpler use cases Both services support various voices, languages, and customization options. ​ Installation To use Neuphonic TTS services, install the required dependencies: Copy Ask AI pip install "pipecat-ai[neuphonic]" You’ll also need to set up your Neuphonic API key as an environment variable: NEUPHONIC_API_KEY ​ NeuphonicTTSService (WebSocket) ​ Configuration ​ api_key str required Your Neuphonic API key ​ voice_id str default: "None" Voice identifier to use for synthesis ​ url str default: "wss://api.neuphonic.com" Neuphonic WebSocket API endpoint ​ sample_rate int default: "22050" Output audio sample rate in Hz ​ encoding str default: "pcm_linear" Audio encoding format ​ params InputParams default: "InputParams()" Additional configuration parameters ​ InputParams ​ language Language default: "Language.EN" The language for TTS generation ​ speed float default: "1.0" Speech speed multiplier (0.5-2.0) ​ NeuphonicHttpTTSService (HTTP) ​ Configuration ​ api_key str required Your Neuphonic API key ​ voice_id str default: "None" Voice identifier to use for synthesis ​ url str default: "https://api.neuphonic.com" Neuphonic HTTP API endpoint ​ sample_rate int default: "22050" Output audio sample rate in Hz ​ encoding str default: "pcm_linear" Audio encoding format ​ params InputParams default: "InputParams()" Additional configuration parameters (same as WebSocket implementation) ​ Input Both services accept text input through their TTS pipeline. ​ Output Frames ​ TTSStartedFrame Signals the start of audio generation. ​ TTSAudioRawFrame Contains generated audio data: ​ audio bytes Raw audio data chunk ​ sample_rate int Audio sample rate (22050Hz default) ​ num_channels int Number of audio channels (1 for mono) ​ TTSStoppedFrame Signals the completion of audio generation. ​ ErrorFrame Sent when an error occurs during TTS generation: ​ error str Error message describing what went wrong ​ Methods ​ WebSocket Implementation The WebSocket implementation ( NeuphonicTTSService ) inherits from InterruptibleTTSService and provides: Support for interrupting ongoing TTS generation Automatic websocket connection management Keep-alive mechanism for persistent connections Special handling for conversation flows ​ HTTP Implementation The HTTP implementation ( NeuphonicHttpTTSService ) inherits from TTSService and provides: Simpler API integration using HTTP streaming Less overhead for single TTS requests Simplified error handling ​ Language Support Neuphonic TTS supports the following languages: Language Code Description Service Codes Language.EN English en Language.ES Spanish es Language.DE German de Language.NL Dutch nl Language.AR Arabic ar Language.FR French fr Language.PT Portuguese pt Language.RU Russian ru Language.HI Hindi hi Language.ZH Chinese zh Regional variants (e.g., EN_US , ES_ES ) are automatically mapped to their base language. ​ Usage Example ​ WebSocket Implementation Copy Ask AI from pipecat.services.neuphonic.tts import NeuphonicTTSService from pipecat.transcriptions.language import Language # Configure service tts = NeuphonicTTSService( api_key = "your-neuphonic-api-key" , voice_id = "preferred-voice-id" , params = NeuphonicTTSService.InputParams( language = Language. EN , speed = 1.2 ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ HTTP Implementation Copy Ask AI from pipecat.services.neuphonic.tts import NeuphonicHttpTTSService from pipecat.transcriptions.language import Language # Configure service tts = NeuphonicHttpTTSService( api_key = "your-neuphonic-api-key" , voice_id = "preferred-voice-id" , params = NeuphonicHttpTTSService.InputParams( language = Language. ES , speed = 1.0 ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ Metrics Support Both services support metrics collection: Time to First Byte (TTFB) TTS usage metrics Processing duration ​ Audio Processing Configurable sample rate (defaults to 22050Hz) PCM linear encoding Single channel (mono) output Base64 decoding for audio data ​ Notes WebSocket implementation includes a keep-alive mechanism (10-second interval) WebSocket service maintains a persistent connection for faster responses Both services automatically select appropriate language codes The WebSocket implementation pauses frame processing during speech generation to prevent overlapping responses MiniMax NVIDIA Riva On this page Overview Installation NeuphonicTTSService (WebSocket) Configuration InputParams NeuphonicHttpTTSService (HTTP) Configuration Input Output Frames TTSStartedFrame TTSAudioRawFrame TTSStoppedFrame ErrorFrame Methods WebSocket Implementation HTTP Implementation Language Support Usage Example WebSocket Implementation HTTP Implementation Metrics Support Audio Processing Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_neuphonic_f194a330.txt b/tts_neuphonic_f194a330.txt
new file mode 100644
index 0000000000000000000000000000000000000000..a6961ddcfe5a56a8264f6b491fef8b5c27395fc4
--- /dev/null
+++ b/tts_neuphonic_f194a330.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/neuphonic#input
+Title: Neuphonic - Pipecat
+==================================================
+
+Neuphonic - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech Neuphonic Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview Neuphonic provides high-quality text-to-speech synthesis through two service implementations: NeuphonicTTSService : WebSocket-based implementation with interruption support NeuphonicHttpTTSService : HTTP-based implementation for simpler use cases Both services support various voices, languages, and customization options. ​ Installation To use Neuphonic TTS services, install the required dependencies: Copy Ask AI pip install "pipecat-ai[neuphonic]" You’ll also need to set up your Neuphonic API key as an environment variable: NEUPHONIC_API_KEY ​ NeuphonicTTSService (WebSocket) ​ Configuration ​ api_key str required Your Neuphonic API key ​ voice_id str default: "None" Voice identifier to use for synthesis ​ url str default: "wss://api.neuphonic.com" Neuphonic WebSocket API endpoint ​ sample_rate int default: "22050" Output audio sample rate in Hz ​ encoding str default: "pcm_linear" Audio encoding format ​ params InputParams default: "InputParams()" Additional configuration parameters ​ InputParams ​ language Language default: "Language.EN" The language for TTS generation ​ speed float default: "1.0" Speech speed multiplier (0.5-2.0) ​ NeuphonicHttpTTSService (HTTP) ​ Configuration ​ api_key str required Your Neuphonic API key ​ voice_id str default: "None" Voice identifier to use for synthesis ​ url str default: "https://api.neuphonic.com" Neuphonic HTTP API endpoint ​ sample_rate int default: "22050" Output audio sample rate in Hz ​ encoding str default: "pcm_linear" Audio encoding format ​ params InputParams default: "InputParams()" Additional configuration parameters (same as WebSocket implementation) ​ Input Both services accept text input through their TTS pipeline. ​ Output Frames ​ TTSStartedFrame Signals the start of audio generation. ​ TTSAudioRawFrame Contains generated audio data: ​ audio bytes Raw audio data chunk ​ sample_rate int Audio sample rate (22050Hz default) ​ num_channels int Number of audio channels (1 for mono) ​ TTSStoppedFrame Signals the completion of audio generation. ​ ErrorFrame Sent when an error occurs during TTS generation: ​ error str Error message describing what went wrong ​ Methods ​ WebSocket Implementation The WebSocket implementation ( NeuphonicTTSService ) inherits from InterruptibleTTSService and provides: Support for interrupting ongoing TTS generation Automatic websocket connection management Keep-alive mechanism for persistent connections Special handling for conversation flows ​ HTTP Implementation The HTTP implementation ( NeuphonicHttpTTSService ) inherits from TTSService and provides: Simpler API integration using HTTP streaming Less overhead for single TTS requests Simplified error handling ​ Language Support Neuphonic TTS supports the following languages: Language Code Description Service Codes Language.EN English en Language.ES Spanish es Language.DE German de Language.NL Dutch nl Language.AR Arabic ar Language.FR French fr Language.PT Portuguese pt Language.RU Russian ru Language.HI Hindi hi Language.ZH Chinese zh Regional variants (e.g., EN_US , ES_ES ) are automatically mapped to their base language. ​ Usage Example ​ WebSocket Implementation Copy Ask AI from pipecat.services.neuphonic.tts import NeuphonicTTSService from pipecat.transcriptions.language import Language # Configure service tts = NeuphonicTTSService( api_key = "your-neuphonic-api-key" , voice_id = "preferred-voice-id" , params = NeuphonicTTSService.InputParams( language = Language. EN , speed = 1.2 ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ HTTP Implementation Copy Ask AI from pipecat.services.neuphonic.tts import NeuphonicHttpTTSService from pipecat.transcriptions.language import Language # Configure service tts = NeuphonicHttpTTSService( api_key = "your-neuphonic-api-key" , voice_id = "preferred-voice-id" , params = NeuphonicHttpTTSService.InputParams( language = Language. ES , speed = 1.0 ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ Metrics Support Both services support metrics collection: Time to First Byte (TTFB) TTS usage metrics Processing duration ​ Audio Processing Configurable sample rate (defaults to 22050Hz) PCM linear encoding Single channel (mono) output Base64 decoding for audio data ​ Notes WebSocket implementation includes a keep-alive mechanism (10-second interval) WebSocket service maintains a persistent connection for faster responses Both services automatically select appropriate language codes The WebSocket implementation pauses frame processing during speech generation to prevent overlapping responses MiniMax NVIDIA Riva On this page Overview Installation NeuphonicTTSService (WebSocket) Configuration InputParams NeuphonicHttpTTSService (HTTP) Configuration Input Output Frames TTSStartedFrame TTSAudioRawFrame TTSStoppedFrame ErrorFrame Methods WebSocket Implementation HTTP Implementation Language Support Usage Example WebSocket Implementation HTTP Implementation Metrics Support Audio Processing Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_openai_32b84c44.txt b/tts_openai_32b84c44.txt
new file mode 100644
index 0000000000000000000000000000000000000000..2ce07272c9ff862c27749fe2294ecf4acf80b180
--- /dev/null
+++ b/tts_openai_32b84c44.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/openai#methods
+Title: OpenAI - Pipecat
+==================================================
+
+OpenAI - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech OpenAI Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview OpenAITTSService converts text to speech using OpenAI’s TTS API. It supports multiple voices and provides high-quality audio output at 24kHz using both traditional TTS models and the gpt-4o TTS models. ​ Installation To use OpenAITTSService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[openai]" You’ll also need to set up your OpenAI API key as an environment variable: OPENAI_API_KEY ​ Configuration ​ Constructor Parameters ​ api_key str | None OpenAI API key ​ voice str default: "alloy" Voice identifier. Options: "alloy" "echo" "fable" "onyx" "nova" "shimmer" ​ model str default: "gpt-4o-mini-tts" Model to use. Options: "gpt-4o-mini-tts" "tts-1" "tts-1-hd" ​ sample_rate int default: "None" Output audio sample rate in Hz. Supports only 24000 Hz. ​ text_filter BaseTextFilter default: "None" Modifies text provided to the TTS. Learn more about the available filters. ​ Output Frames ​ Control Frames ​ TTSStartedFrame Frame Signals start of audio generation ​ TTSStoppedFrame Frame Signals completion of audio generation ​ Audio Frames ​ TTSAudioRawFrame Frame Contains generated audio data: PCM encoded audio 24kHz sample rate Mono channel ​ Error Frames ​ ErrorFrame Frame Contains error information if TTS fails ​ Methods See the TTS base class methods for additional functionality. ​ Models Model Description Best For gpt-4o-mini-tts Latest GPT-based TTS model Faster generation, improved prosody, recommended for most use cases tts-1 Original TTS model Standard quality speech tts-1-hd High-definition TTS model Premium quality speech with higher fidelity ​ Language Support OpenAI TTS supports the following languages and regional variants: Language Code Description Service Codes Language.EN English en ​ Usage Example Copy Ask AI from pipecat.services.openai.tts import OpenAITTSService # Configure service tts = OpenAITTSService( voice = "nova" , model = "gpt-4o-mini-tts" , ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ Frame Flow ​ Metrics Support The service supports metrics collection: Time to First Byte (TTFB) TTS usage metrics Processing duration ​ Notes Outputs PCM audio at 24kHz Streams audio in 1KB chunks Supports multiple voices Uses GPT-4o Mini TTS by default for improved quality Includes metrics collection Thread-safe processing Handles empty text gracefully NVIDIA Riva Piper On this page Overview Installation Configuration Constructor Parameters Output Frames Control Frames Audio Frames Error Frames Methods Models Language Support Usage Example Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_openai_5e28da8e.txt b/tts_openai_5e28da8e.txt
new file mode 100644
index 0000000000000000000000000000000000000000..c2923b2815351c50ad7642a4d1758409896ad24b
--- /dev/null
+++ b/tts_openai_5e28da8e.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/openai#param-model
+Title: OpenAI - Pipecat
+==================================================
+
+OpenAI - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech OpenAI Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview OpenAITTSService converts text to speech using OpenAI’s TTS API. It supports multiple voices and provides high-quality audio output at 24kHz using both traditional TTS models and the gpt-4o TTS models. ​ Installation To use OpenAITTSService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[openai]" You’ll also need to set up your OpenAI API key as an environment variable: OPENAI_API_KEY ​ Configuration ​ Constructor Parameters ​ api_key str | None OpenAI API key ​ voice str default: "alloy" Voice identifier. Options: "alloy" "echo" "fable" "onyx" "nova" "shimmer" ​ model str default: "gpt-4o-mini-tts" Model to use. Options: "gpt-4o-mini-tts" "tts-1" "tts-1-hd" ​ sample_rate int default: "None" Output audio sample rate in Hz. Supports only 24000 Hz. ​ text_filter BaseTextFilter default: "None" Modifies text provided to the TTS. Learn more about the available filters. ​ Output Frames ​ Control Frames ​ TTSStartedFrame Frame Signals start of audio generation ​ TTSStoppedFrame Frame Signals completion of audio generation ​ Audio Frames ​ TTSAudioRawFrame Frame Contains generated audio data: PCM encoded audio 24kHz sample rate Mono channel ​ Error Frames ​ ErrorFrame Frame Contains error information if TTS fails ​ Methods See the TTS base class methods for additional functionality. ​ Models Model Description Best For gpt-4o-mini-tts Latest GPT-based TTS model Faster generation, improved prosody, recommended for most use cases tts-1 Original TTS model Standard quality speech tts-1-hd High-definition TTS model Premium quality speech with higher fidelity ​ Language Support OpenAI TTS supports the following languages and regional variants: Language Code Description Service Codes Language.EN English en ​ Usage Example Copy Ask AI from pipecat.services.openai.tts import OpenAITTSService # Configure service tts = OpenAITTSService( voice = "nova" , model = "gpt-4o-mini-tts" , ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ Frame Flow ​ Metrics Support The service supports metrics collection: Time to First Byte (TTFB) TTS usage metrics Processing duration ​ Notes Outputs PCM audio at 24kHz Streams audio in 1KB chunks Supports multiple voices Uses GPT-4o Mini TTS by default for improved quality Includes metrics collection Thread-safe processing Handles empty text gracefully NVIDIA Riva Piper On this page Overview Installation Configuration Constructor Parameters Output Frames Control Frames Audio Frames Error Frames Methods Models Language Support Usage Example Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_openai_6af0fd06.txt b/tts_openai_6af0fd06.txt
new file mode 100644
index 0000000000000000000000000000000000000000..c3797d06daabe803e60012212f3445233b352584
--- /dev/null
+++ b/tts_openai_6af0fd06.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/openai#installation
+Title: OpenAI - Pipecat
+==================================================
+
+OpenAI - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech OpenAI Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview OpenAITTSService converts text to speech using OpenAI’s TTS API. It supports multiple voices and provides high-quality audio output at 24kHz using both traditional TTS models and the gpt-4o TTS models. ​ Installation To use OpenAITTSService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[openai]" You’ll also need to set up your OpenAI API key as an environment variable: OPENAI_API_KEY ​ Configuration ​ Constructor Parameters ​ api_key str | None OpenAI API key ​ voice str default: "alloy" Voice identifier. Options: "alloy" "echo" "fable" "onyx" "nova" "shimmer" ​ model str default: "gpt-4o-mini-tts" Model to use. Options: "gpt-4o-mini-tts" "tts-1" "tts-1-hd" ​ sample_rate int default: "None" Output audio sample rate in Hz. Supports only 24000 Hz. ​ text_filter BaseTextFilter default: "None" Modifies text provided to the TTS. Learn more about the available filters. ​ Output Frames ​ Control Frames ​ TTSStartedFrame Frame Signals start of audio generation ​ TTSStoppedFrame Frame Signals completion of audio generation ​ Audio Frames ​ TTSAudioRawFrame Frame Contains generated audio data: PCM encoded audio 24kHz sample rate Mono channel ​ Error Frames ​ ErrorFrame Frame Contains error information if TTS fails ​ Methods See the TTS base class methods for additional functionality. ​ Models Model Description Best For gpt-4o-mini-tts Latest GPT-based TTS model Faster generation, improved prosody, recommended for most use cases tts-1 Original TTS model Standard quality speech tts-1-hd High-definition TTS model Premium quality speech with higher fidelity ​ Language Support OpenAI TTS supports the following languages and regional variants: Language Code Description Service Codes Language.EN English en ​ Usage Example Copy Ask AI from pipecat.services.openai.tts import OpenAITTSService # Configure service tts = OpenAITTSService( voice = "nova" , model = "gpt-4o-mini-tts" , ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ Frame Flow ​ Metrics Support The service supports metrics collection: Time to First Byte (TTFB) TTS usage metrics Processing duration ​ Notes Outputs PCM audio at 24kHz Streams audio in 1KB chunks Supports multiple voices Uses GPT-4o Mini TTS by default for improved quality Includes metrics collection Thread-safe processing Handles empty text gracefully NVIDIA Riva Piper On this page Overview Installation Configuration Constructor Parameters Output Frames Control Frames Audio Frames Error Frames Methods Models Language Support Usage Example Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_openai_7a74dde3.txt b/tts_openai_7a74dde3.txt
new file mode 100644
index 0000000000000000000000000000000000000000..8fc9ea00651481322bf24fef2ab7e71c857f90d5
--- /dev/null
+++ b/tts_openai_7a74dde3.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/openai#param-sample-rate
+Title: OpenAI - Pipecat
+==================================================
+
+OpenAI - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech OpenAI Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview OpenAITTSService converts text to speech using OpenAI’s TTS API. It supports multiple voices and provides high-quality audio output at 24kHz using both traditional TTS models and the gpt-4o TTS models. ​ Installation To use OpenAITTSService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[openai]" You’ll also need to set up your OpenAI API key as an environment variable: OPENAI_API_KEY ​ Configuration ​ Constructor Parameters ​ api_key str | None OpenAI API key ​ voice str default: "alloy" Voice identifier. Options: "alloy" "echo" "fable" "onyx" "nova" "shimmer" ​ model str default: "gpt-4o-mini-tts" Model to use. Options: "gpt-4o-mini-tts" "tts-1" "tts-1-hd" ​ sample_rate int default: "None" Output audio sample rate in Hz. Supports only 24000 Hz. ​ text_filter BaseTextFilter default: "None" Modifies text provided to the TTS. Learn more about the available filters. ​ Output Frames ​ Control Frames ​ TTSStartedFrame Frame Signals start of audio generation ​ TTSStoppedFrame Frame Signals completion of audio generation ​ Audio Frames ​ TTSAudioRawFrame Frame Contains generated audio data: PCM encoded audio 24kHz sample rate Mono channel ​ Error Frames ​ ErrorFrame Frame Contains error information if TTS fails ​ Methods See the TTS base class methods for additional functionality. ​ Models Model Description Best For gpt-4o-mini-tts Latest GPT-based TTS model Faster generation, improved prosody, recommended for most use cases tts-1 Original TTS model Standard quality speech tts-1-hd High-definition TTS model Premium quality speech with higher fidelity ​ Language Support OpenAI TTS supports the following languages and regional variants: Language Code Description Service Codes Language.EN English en ​ Usage Example Copy Ask AI from pipecat.services.openai.tts import OpenAITTSService # Configure service tts = OpenAITTSService( voice = "nova" , model = "gpt-4o-mini-tts" , ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ Frame Flow ​ Metrics Support The service supports metrics collection: Time to First Byte (TTFB) TTS usage metrics Processing duration ​ Notes Outputs PCM audio at 24kHz Streams audio in 1KB chunks Supports multiple voices Uses GPT-4o Mini TTS by default for improved quality Includes metrics collection Thread-safe processing Handles empty text gracefully NVIDIA Riva Piper On this page Overview Installation Configuration Constructor Parameters Output Frames Control Frames Audio Frames Error Frames Methods Models Language Support Usage Example Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_openai_7be22552.txt b/tts_openai_7be22552.txt
new file mode 100644
index 0000000000000000000000000000000000000000..17d085c07dc7775e416a46b5484c950be5cb126f
--- /dev/null
+++ b/tts_openai_7be22552.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/openai#param-tts-stopped-frame
+Title: OpenAI - Pipecat
+==================================================
+
+OpenAI - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech OpenAI Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview OpenAITTSService converts text to speech using OpenAI’s TTS API. It supports multiple voices and provides high-quality audio output at 24kHz using both traditional TTS models and the gpt-4o TTS models. ​ Installation To use OpenAITTSService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[openai]" You’ll also need to set up your OpenAI API key as an environment variable: OPENAI_API_KEY ​ Configuration ​ Constructor Parameters ​ api_key str | None OpenAI API key ​ voice str default: "alloy" Voice identifier. Options: "alloy" "echo" "fable" "onyx" "nova" "shimmer" ​ model str default: "gpt-4o-mini-tts" Model to use. Options: "gpt-4o-mini-tts" "tts-1" "tts-1-hd" ​ sample_rate int default: "None" Output audio sample rate in Hz. Supports only 24000 Hz. ​ text_filter BaseTextFilter default: "None" Modifies text provided to the TTS. Learn more about the available filters. ​ Output Frames ​ Control Frames ​ TTSStartedFrame Frame Signals start of audio generation ​ TTSStoppedFrame Frame Signals completion of audio generation ​ Audio Frames ​ TTSAudioRawFrame Frame Contains generated audio data: PCM encoded audio 24kHz sample rate Mono channel ​ Error Frames ​ ErrorFrame Frame Contains error information if TTS fails ​ Methods See the TTS base class methods for additional functionality. ​ Models Model Description Best For gpt-4o-mini-tts Latest GPT-based TTS model Faster generation, improved prosody, recommended for most use cases tts-1 Original TTS model Standard quality speech tts-1-hd High-definition TTS model Premium quality speech with higher fidelity ​ Language Support OpenAI TTS supports the following languages and regional variants: Language Code Description Service Codes Language.EN English en ​ Usage Example Copy Ask AI from pipecat.services.openai.tts import OpenAITTSService # Configure service tts = OpenAITTSService( voice = "nova" , model = "gpt-4o-mini-tts" , ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ Frame Flow ​ Metrics Support The service supports metrics collection: Time to First Byte (TTFB) TTS usage metrics Processing duration ​ Notes Outputs PCM audio at 24kHz Streams audio in 1KB chunks Supports multiple voices Uses GPT-4o Mini TTS by default for improved quality Includes metrics collection Thread-safe processing Handles empty text gracefully NVIDIA Riva Piper On this page Overview Installation Configuration Constructor Parameters Output Frames Control Frames Audio Frames Error Frames Methods Models Language Support Usage Example Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_openai_8408e8c3.txt b/tts_openai_8408e8c3.txt
new file mode 100644
index 0000000000000000000000000000000000000000..df2f8559d4e4fb1963786aeb99b6a7bbcdc9bfe3
--- /dev/null
+++ b/tts_openai_8408e8c3.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/openai#language-support
+Title: OpenAI - Pipecat
+==================================================
+
+OpenAI - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech OpenAI Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview OpenAITTSService converts text to speech using OpenAI’s TTS API. It supports multiple voices and provides high-quality audio output at 24kHz using both traditional TTS models and the gpt-4o TTS models. ​ Installation To use OpenAITTSService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[openai]" You’ll also need to set up your OpenAI API key as an environment variable: OPENAI_API_KEY ​ Configuration ​ Constructor Parameters ​ api_key str | None OpenAI API key ​ voice str default: "alloy" Voice identifier. Options: "alloy" "echo" "fable" "onyx" "nova" "shimmer" ​ model str default: "gpt-4o-mini-tts" Model to use. Options: "gpt-4o-mini-tts" "tts-1" "tts-1-hd" ​ sample_rate int default: "None" Output audio sample rate in Hz. Supports only 24000 Hz. ​ text_filter BaseTextFilter default: "None" Modifies text provided to the TTS. Learn more about the available filters. ​ Output Frames ​ Control Frames ​ TTSStartedFrame Frame Signals start of audio generation ​ TTSStoppedFrame Frame Signals completion of audio generation ​ Audio Frames ​ TTSAudioRawFrame Frame Contains generated audio data: PCM encoded audio 24kHz sample rate Mono channel ​ Error Frames ​ ErrorFrame Frame Contains error information if TTS fails ​ Methods See the TTS base class methods for additional functionality. ​ Models Model Description Best For gpt-4o-mini-tts Latest GPT-based TTS model Faster generation, improved prosody, recommended for most use cases tts-1 Original TTS model Standard quality speech tts-1-hd High-definition TTS model Premium quality speech with higher fidelity ​ Language Support OpenAI TTS supports the following languages and regional variants: Language Code Description Service Codes Language.EN English en ​ Usage Example Copy Ask AI from pipecat.services.openai.tts import OpenAITTSService # Configure service tts = OpenAITTSService( voice = "nova" , model = "gpt-4o-mini-tts" , ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ Frame Flow ​ Metrics Support The service supports metrics collection: Time to First Byte (TTFB) TTS usage metrics Processing duration ​ Notes Outputs PCM audio at 24kHz Streams audio in 1KB chunks Supports multiple voices Uses GPT-4o Mini TTS by default for improved quality Includes metrics collection Thread-safe processing Handles empty text gracefully NVIDIA Riva Piper On this page Overview Installation Configuration Constructor Parameters Output Frames Control Frames Audio Frames Error Frames Methods Models Language Support Usage Example Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_openai_ad228a2e.txt b/tts_openai_ad228a2e.txt
new file mode 100644
index 0000000000000000000000000000000000000000..197f2ba8d2f2280a691a4abb0a135fdd56f5cedf
--- /dev/null
+++ b/tts_openai_ad228a2e.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/openai#param-tts-started-frame
+Title: OpenAI - Pipecat
+==================================================
+
+OpenAI - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech OpenAI Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview OpenAITTSService converts text to speech using OpenAI’s TTS API. It supports multiple voices and provides high-quality audio output at 24kHz using both traditional TTS models and the gpt-4o TTS models. ​ Installation To use OpenAITTSService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[openai]" You’ll also need to set up your OpenAI API key as an environment variable: OPENAI_API_KEY ​ Configuration ​ Constructor Parameters ​ api_key str | None OpenAI API key ​ voice str default: "alloy" Voice identifier. Options: "alloy" "echo" "fable" "onyx" "nova" "shimmer" ​ model str default: "gpt-4o-mini-tts" Model to use. Options: "gpt-4o-mini-tts" "tts-1" "tts-1-hd" ​ sample_rate int default: "None" Output audio sample rate in Hz. Supports only 24000 Hz. ​ text_filter BaseTextFilter default: "None" Modifies text provided to the TTS. Learn more about the available filters. ​ Output Frames ​ Control Frames ​ TTSStartedFrame Frame Signals start of audio generation ​ TTSStoppedFrame Frame Signals completion of audio generation ​ Audio Frames ​ TTSAudioRawFrame Frame Contains generated audio data: PCM encoded audio 24kHz sample rate Mono channel ​ Error Frames ​ ErrorFrame Frame Contains error information if TTS fails ​ Methods See the TTS base class methods for additional functionality. ​ Models Model Description Best For gpt-4o-mini-tts Latest GPT-based TTS model Faster generation, improved prosody, recommended for most use cases tts-1 Original TTS model Standard quality speech tts-1-hd High-definition TTS model Premium quality speech with higher fidelity ​ Language Support OpenAI TTS supports the following languages and regional variants: Language Code Description Service Codes Language.EN English en ​ Usage Example Copy Ask AI from pipecat.services.openai.tts import OpenAITTSService # Configure service tts = OpenAITTSService( voice = "nova" , model = "gpt-4o-mini-tts" , ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ Frame Flow ​ Metrics Support The service supports metrics collection: Time to First Byte (TTFB) TTS usage metrics Processing duration ​ Notes Outputs PCM audio at 24kHz Streams audio in 1KB chunks Supports multiple voices Uses GPT-4o Mini TTS by default for improved quality Includes metrics collection Thread-safe processing Handles empty text gracefully NVIDIA Riva Piper On this page Overview Installation Configuration Constructor Parameters Output Frames Control Frames Audio Frames Error Frames Methods Models Language Support Usage Example Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_openai_ceb6cd26.txt b/tts_openai_ceb6cd26.txt
new file mode 100644
index 0000000000000000000000000000000000000000..a231790066fa76c1b9e60ff9b16af2cfb1d17e63
--- /dev/null
+++ b/tts_openai_ceb6cd26.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/openai#param-voice
+Title: OpenAI - Pipecat
+==================================================
+
+OpenAI - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech OpenAI Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview OpenAITTSService converts text to speech using OpenAI’s TTS API. It supports multiple voices and provides high-quality audio output at 24kHz using both traditional TTS models and the gpt-4o TTS models. ​ Installation To use OpenAITTSService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[openai]" You’ll also need to set up your OpenAI API key as an environment variable: OPENAI_API_KEY ​ Configuration ​ Constructor Parameters ​ api_key str | None OpenAI API key ​ voice str default: "alloy" Voice identifier. Options: "alloy" "echo" "fable" "onyx" "nova" "shimmer" ​ model str default: "gpt-4o-mini-tts" Model to use. Options: "gpt-4o-mini-tts" "tts-1" "tts-1-hd" ​ sample_rate int default: "None" Output audio sample rate in Hz. Supports only 24000 Hz. ​ text_filter BaseTextFilter default: "None" Modifies text provided to the TTS. Learn more about the available filters. ​ Output Frames ​ Control Frames ​ TTSStartedFrame Frame Signals start of audio generation ​ TTSStoppedFrame Frame Signals completion of audio generation ​ Audio Frames ​ TTSAudioRawFrame Frame Contains generated audio data: PCM encoded audio 24kHz sample rate Mono channel ​ Error Frames ​ ErrorFrame Frame Contains error information if TTS fails ​ Methods See the TTS base class methods for additional functionality. ​ Models Model Description Best For gpt-4o-mini-tts Latest GPT-based TTS model Faster generation, improved prosody, recommended for most use cases tts-1 Original TTS model Standard quality speech tts-1-hd High-definition TTS model Premium quality speech with higher fidelity ​ Language Support OpenAI TTS supports the following languages and regional variants: Language Code Description Service Codes Language.EN English en ​ Usage Example Copy Ask AI from pipecat.services.openai.tts import OpenAITTSService # Configure service tts = OpenAITTSService( voice = "nova" , model = "gpt-4o-mini-tts" , ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ Frame Flow ​ Metrics Support The service supports metrics collection: Time to First Byte (TTFB) TTS usage metrics Processing duration ​ Notes Outputs PCM audio at 24kHz Streams audio in 1KB chunks Supports multiple voices Uses GPT-4o Mini TTS by default for improved quality Includes metrics collection Thread-safe processing Handles empty text gracefully NVIDIA Riva Piper On this page Overview Installation Configuration Constructor Parameters Output Frames Control Frames Audio Frames Error Frames Methods Models Language Support Usage Example Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_openai_d336bffa.txt b/tts_openai_d336bffa.txt
new file mode 100644
index 0000000000000000000000000000000000000000..3e09a6c92030c0babe78fde37fbe3857a8f09914
--- /dev/null
+++ b/tts_openai_d336bffa.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/openai#notes
+Title: OpenAI - Pipecat
+==================================================
+
+OpenAI - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech OpenAI Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview OpenAITTSService converts text to speech using OpenAI’s TTS API. It supports multiple voices and provides high-quality audio output at 24kHz using both traditional TTS models and the gpt-4o TTS models. ​ Installation To use OpenAITTSService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[openai]" You’ll also need to set up your OpenAI API key as an environment variable: OPENAI_API_KEY ​ Configuration ​ Constructor Parameters ​ api_key str | None OpenAI API key ​ voice str default: "alloy" Voice identifier. Options: "alloy" "echo" "fable" "onyx" "nova" "shimmer" ​ model str default: "gpt-4o-mini-tts" Model to use. Options: "gpt-4o-mini-tts" "tts-1" "tts-1-hd" ​ sample_rate int default: "None" Output audio sample rate in Hz. Supports only 24000 Hz. ​ text_filter BaseTextFilter default: "None" Modifies text provided to the TTS. Learn more about the available filters. ​ Output Frames ​ Control Frames ​ TTSStartedFrame Frame Signals start of audio generation ​ TTSStoppedFrame Frame Signals completion of audio generation ​ Audio Frames ​ TTSAudioRawFrame Frame Contains generated audio data: PCM encoded audio 24kHz sample rate Mono channel ​ Error Frames ​ ErrorFrame Frame Contains error information if TTS fails ​ Methods See the TTS base class methods for additional functionality. ​ Models Model Description Best For gpt-4o-mini-tts Latest GPT-based TTS model Faster generation, improved prosody, recommended for most use cases tts-1 Original TTS model Standard quality speech tts-1-hd High-definition TTS model Premium quality speech with higher fidelity ​ Language Support OpenAI TTS supports the following languages and regional variants: Language Code Description Service Codes Language.EN English en ​ Usage Example Copy Ask AI from pipecat.services.openai.tts import OpenAITTSService # Configure service tts = OpenAITTSService( voice = "nova" , model = "gpt-4o-mini-tts" , ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ Frame Flow ​ Metrics Support The service supports metrics collection: Time to First Byte (TTFB) TTS usage metrics Processing duration ​ Notes Outputs PCM audio at 24kHz Streams audio in 1KB chunks Supports multiple voices Uses GPT-4o Mini TTS by default for improved quality Includes metrics collection Thread-safe processing Handles empty text gracefully NVIDIA Riva Piper On this page Overview Installation Configuration Constructor Parameters Output Frames Control Frames Audio Frames Error Frames Methods Models Language Support Usage Example Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_openai_d5809243.txt b/tts_openai_d5809243.txt
new file mode 100644
index 0000000000000000000000000000000000000000..ca2fe073a639c39d4281ff2cf990dec0aa81acbc
--- /dev/null
+++ b/tts_openai_d5809243.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/openai#error-frames
+Title: OpenAI - Pipecat
+==================================================
+
+OpenAI - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech OpenAI Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview OpenAITTSService converts text to speech using OpenAI’s TTS API. It supports multiple voices and provides high-quality audio output at 24kHz using both traditional TTS models and the gpt-4o TTS models. ​ Installation To use OpenAITTSService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[openai]" You’ll also need to set up your OpenAI API key as an environment variable: OPENAI_API_KEY ​ Configuration ​ Constructor Parameters ​ api_key str | None OpenAI API key ​ voice str default: "alloy" Voice identifier. Options: "alloy" "echo" "fable" "onyx" "nova" "shimmer" ​ model str default: "gpt-4o-mini-tts" Model to use. Options: "gpt-4o-mini-tts" "tts-1" "tts-1-hd" ​ sample_rate int default: "None" Output audio sample rate in Hz. Supports only 24000 Hz. ​ text_filter BaseTextFilter default: "None" Modifies text provided to the TTS. Learn more about the available filters. ​ Output Frames ​ Control Frames ​ TTSStartedFrame Frame Signals start of audio generation ​ TTSStoppedFrame Frame Signals completion of audio generation ​ Audio Frames ​ TTSAudioRawFrame Frame Contains generated audio data: PCM encoded audio 24kHz sample rate Mono channel ​ Error Frames ​ ErrorFrame Frame Contains error information if TTS fails ​ Methods See the TTS base class methods for additional functionality. ​ Models Model Description Best For gpt-4o-mini-tts Latest GPT-based TTS model Faster generation, improved prosody, recommended for most use cases tts-1 Original TTS model Standard quality speech tts-1-hd High-definition TTS model Premium quality speech with higher fidelity ​ Language Support OpenAI TTS supports the following languages and regional variants: Language Code Description Service Codes Language.EN English en ​ Usage Example Copy Ask AI from pipecat.services.openai.tts import OpenAITTSService # Configure service tts = OpenAITTSService( voice = "nova" , model = "gpt-4o-mini-tts" , ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ Frame Flow ​ Metrics Support The service supports metrics collection: Time to First Byte (TTFB) TTS usage metrics Processing duration ​ Notes Outputs PCM audio at 24kHz Streams audio in 1KB chunks Supports multiple voices Uses GPT-4o Mini TTS by default for improved quality Includes metrics collection Thread-safe processing Handles empty text gracefully NVIDIA Riva Piper On this page Overview Installation Configuration Constructor Parameters Output Frames Control Frames Audio Frames Error Frames Methods Models Language Support Usage Example Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_openai_e9b8b0f3.txt b/tts_openai_e9b8b0f3.txt
new file mode 100644
index 0000000000000000000000000000000000000000..41a56421b267365c67ab2f0e850586c0bab95196
--- /dev/null
+++ b/tts_openai_e9b8b0f3.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/openai#audio-frames
+Title: OpenAI - Pipecat
+==================================================
+
+OpenAI - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech OpenAI Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview OpenAITTSService converts text to speech using OpenAI’s TTS API. It supports multiple voices and provides high-quality audio output at 24kHz using both traditional TTS models and the gpt-4o TTS models. ​ Installation To use OpenAITTSService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[openai]" You’ll also need to set up your OpenAI API key as an environment variable: OPENAI_API_KEY ​ Configuration ​ Constructor Parameters ​ api_key str | None OpenAI API key ​ voice str default: "alloy" Voice identifier. Options: "alloy" "echo" "fable" "onyx" "nova" "shimmer" ​ model str default: "gpt-4o-mini-tts" Model to use. Options: "gpt-4o-mini-tts" "tts-1" "tts-1-hd" ​ sample_rate int default: "None" Output audio sample rate in Hz. Supports only 24000 Hz. ​ text_filter BaseTextFilter default: "None" Modifies text provided to the TTS. Learn more about the available filters. ​ Output Frames ​ Control Frames ​ TTSStartedFrame Frame Signals start of audio generation ​ TTSStoppedFrame Frame Signals completion of audio generation ​ Audio Frames ​ TTSAudioRawFrame Frame Contains generated audio data: PCM encoded audio 24kHz sample rate Mono channel ​ Error Frames ​ ErrorFrame Frame Contains error information if TTS fails ​ Methods See the TTS base class methods for additional functionality. ​ Models Model Description Best For gpt-4o-mini-tts Latest GPT-based TTS model Faster generation, improved prosody, recommended for most use cases tts-1 Original TTS model Standard quality speech tts-1-hd High-definition TTS model Premium quality speech with higher fidelity ​ Language Support OpenAI TTS supports the following languages and regional variants: Language Code Description Service Codes Language.EN English en ​ Usage Example Copy Ask AI from pipecat.services.openai.tts import OpenAITTSService # Configure service tts = OpenAITTSService( voice = "nova" , model = "gpt-4o-mini-tts" , ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ Frame Flow ​ Metrics Support The service supports metrics collection: Time to First Byte (TTFB) TTS usage metrics Processing duration ​ Notes Outputs PCM audio at 24kHz Streams audio in 1KB chunks Supports multiple voices Uses GPT-4o Mini TTS by default for improved quality Includes metrics collection Thread-safe processing Handles empty text gracefully NVIDIA Riva Piper On this page Overview Installation Configuration Constructor Parameters Output Frames Control Frames Audio Frames Error Frames Methods Models Language Support Usage Example Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_piper_00b3f0be.txt b/tts_piper_00b3f0be.txt
new file mode 100644
index 0000000000000000000000000000000000000000..8160b14eeabf589257bb7f738b6e9159197000ec
--- /dev/null
+++ b/tts_piper_00b3f0be.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/piper#param-error
+Title: Piper - Pipecat
+==================================================
+
+Piper - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech Piper Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview PiperTTSService converts text to speech using the Piper TTS server. This service provides integration with a locally-running Piper TTS service, offering self-hosted speech synthesis capabilities. ​ Installation To use PiperTTSService , no additional dependencies in Pipecat are required. You’ll also need to set up a running Piper TTS server following the Piper HTTP server documentation . ​ Configuration ​ Constructor Parameters ​ base_url str required API base URL for the Piper TTS server (without a trailing slash) ​ aiohttp_session aiohttp.ClientSession required aiohttp ClientSession for making HTTP requests ​ sample_rate Optional[int] default: "None" Output sample rate in Hz. When None, the sample rate depends on the voice model being used by the Piper server. ​ text_filter BaseTextFilter default: "None" Modifies text provided to the TTS. Learn more about the available filters. ​ Input The service accepts text input through its TTS pipeline. ​ Output Frames ​ TTSStartedFrame Signals the start of audio generation. ​ TTSAudioRawFrame Contains generated audio data: ​ audio bytes Raw audio data chunk ​ sample_rate int Audio sample rate (depends on the Piper model) ​ num_channels int Number of audio channels (1 for mono) ​ TTSStoppedFrame Signals the completion of audio generation. ​ ErrorFrame Signals that an error occurred during audio generation: ​ error str Error message ​ Methods See the TTS base class methods for additional functionality. ​ Usage Example Copy Ask AI import aiohttp from pipecat.services.piper.tts import PiperTTSService # Create aiohttp session session = aiohttp.ClientSession() # Configure service tts = PiperTTSService( base_url = "http://localhost:5000/api/tts" , aiohttp_session = session, sample_rate = 22050 # Optional: specify if you know the model's sample rate ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ Frame Flow ​ Metrics Support The service supports metrics collection: Time to First Byte (TTFB) TTS usage metrics Processing duration ​ Audio Processing Streams audio in 1KB chunks Automatically handles WAV headers in the response Outputs mono audio Supports the sample rate specified by your Piper voice model ​ Notes Requires a running Piper TTS server Self-hosted solution with no external API dependencies Streams audio in chunks for efficient processing Automatically handles WAV headers in the response Provides metrics collection OpenAI PlayHT On this page Overview Installation Configuration Constructor Parameters Input Output Frames TTSStartedFrame TTSAudioRawFrame TTSStoppedFrame ErrorFrame Methods Usage Example Frame Flow Metrics Support Audio Processing Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_piper_05ac7380.txt b/tts_piper_05ac7380.txt
new file mode 100644
index 0000000000000000000000000000000000000000..e1035589bbeeac8b4bf1e84ba9ff2885a647d364
--- /dev/null
+++ b/tts_piper_05ac7380.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/piper#metrics-support
+Title: Piper - Pipecat
+==================================================
+
+Piper - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech Piper Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview PiperTTSService converts text to speech using the Piper TTS server. This service provides integration with a locally-running Piper TTS service, offering self-hosted speech synthesis capabilities. ​ Installation To use PiperTTSService , no additional dependencies in Pipecat are required. You’ll also need to set up a running Piper TTS server following the Piper HTTP server documentation . ​ Configuration ​ Constructor Parameters ​ base_url str required API base URL for the Piper TTS server (without a trailing slash) ​ aiohttp_session aiohttp.ClientSession required aiohttp ClientSession for making HTTP requests ​ sample_rate Optional[int] default: "None" Output sample rate in Hz. When None, the sample rate depends on the voice model being used by the Piper server. ​ text_filter BaseTextFilter default: "None" Modifies text provided to the TTS. Learn more about the available filters. ​ Input The service accepts text input through its TTS pipeline. ​ Output Frames ​ TTSStartedFrame Signals the start of audio generation. ​ TTSAudioRawFrame Contains generated audio data: ​ audio bytes Raw audio data chunk ​ sample_rate int Audio sample rate (depends on the Piper model) ​ num_channels int Number of audio channels (1 for mono) ​ TTSStoppedFrame Signals the completion of audio generation. ​ ErrorFrame Signals that an error occurred during audio generation: ​ error str Error message ​ Methods See the TTS base class methods for additional functionality. ​ Usage Example Copy Ask AI import aiohttp from pipecat.services.piper.tts import PiperTTSService # Create aiohttp session session = aiohttp.ClientSession() # Configure service tts = PiperTTSService( base_url = "http://localhost:5000/api/tts" , aiohttp_session = session, sample_rate = 22050 # Optional: specify if you know the model's sample rate ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ Frame Flow ​ Metrics Support The service supports metrics collection: Time to First Byte (TTFB) TTS usage metrics Processing duration ​ Audio Processing Streams audio in 1KB chunks Automatically handles WAV headers in the response Outputs mono audio Supports the sample rate specified by your Piper voice model ​ Notes Requires a running Piper TTS server Self-hosted solution with no external API dependencies Streams audio in chunks for efficient processing Automatically handles WAV headers in the response Provides metrics collection OpenAI PlayHT On this page Overview Installation Configuration Constructor Parameters Input Output Frames TTSStartedFrame TTSAudioRawFrame TTSStoppedFrame ErrorFrame Methods Usage Example Frame Flow Metrics Support Audio Processing Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_piper_37a234aa.txt b/tts_piper_37a234aa.txt
new file mode 100644
index 0000000000000000000000000000000000000000..1cb1be0e7bf58cc76af53e9416c7c9d22c34d1b4
--- /dev/null
+++ b/tts_piper_37a234aa.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/piper#notes
+Title: Piper - Pipecat
+==================================================
+
+Piper - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech Piper Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview PiperTTSService converts text to speech using the Piper TTS server. This service provides integration with a locally-running Piper TTS service, offering self-hosted speech synthesis capabilities. ​ Installation To use PiperTTSService , no additional dependencies in Pipecat are required. You’ll also need to set up a running Piper TTS server following the Piper HTTP server documentation . ​ Configuration ​ Constructor Parameters ​ base_url str required API base URL for the Piper TTS server (without a trailing slash) ​ aiohttp_session aiohttp.ClientSession required aiohttp ClientSession for making HTTP requests ​ sample_rate Optional[int] default: "None" Output sample rate in Hz. When None, the sample rate depends on the voice model being used by the Piper server. ​ text_filter BaseTextFilter default: "None" Modifies text provided to the TTS. Learn more about the available filters. ​ Input The service accepts text input through its TTS pipeline. ​ Output Frames ​ TTSStartedFrame Signals the start of audio generation. ​ TTSAudioRawFrame Contains generated audio data: ​ audio bytes Raw audio data chunk ​ sample_rate int Audio sample rate (depends on the Piper model) ​ num_channels int Number of audio channels (1 for mono) ​ TTSStoppedFrame Signals the completion of audio generation. ​ ErrorFrame Signals that an error occurred during audio generation: ​ error str Error message ​ Methods See the TTS base class methods for additional functionality. ​ Usage Example Copy Ask AI import aiohttp from pipecat.services.piper.tts import PiperTTSService # Create aiohttp session session = aiohttp.ClientSession() # Configure service tts = PiperTTSService( base_url = "http://localhost:5000/api/tts" , aiohttp_session = session, sample_rate = 22050 # Optional: specify if you know the model's sample rate ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ Frame Flow ​ Metrics Support The service supports metrics collection: Time to First Byte (TTFB) TTS usage metrics Processing duration ​ Audio Processing Streams audio in 1KB chunks Automatically handles WAV headers in the response Outputs mono audio Supports the sample rate specified by your Piper voice model ​ Notes Requires a running Piper TTS server Self-hosted solution with no external API dependencies Streams audio in chunks for efficient processing Automatically handles WAV headers in the response Provides metrics collection OpenAI PlayHT On this page Overview Installation Configuration Constructor Parameters Input Output Frames TTSStartedFrame TTSAudioRawFrame TTSStoppedFrame ErrorFrame Methods Usage Example Frame Flow Metrics Support Audio Processing Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_piper_827d37b9.txt b/tts_piper_827d37b9.txt
new file mode 100644
index 0000000000000000000000000000000000000000..73ddde5887e3a13a6a600708568fff482e478095
--- /dev/null
+++ b/tts_piper_827d37b9.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/piper#usage-example
+Title: Piper - Pipecat
+==================================================
+
+Piper - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech Piper Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview PiperTTSService converts text to speech using the Piper TTS server. This service provides integration with a locally-running Piper TTS service, offering self-hosted speech synthesis capabilities. ​ Installation To use PiperTTSService , no additional dependencies in Pipecat are required. You’ll also need to set up a running Piper TTS server following the Piper HTTP server documentation . ​ Configuration ​ Constructor Parameters ​ base_url str required API base URL for the Piper TTS server (without a trailing slash) ​ aiohttp_session aiohttp.ClientSession required aiohttp ClientSession for making HTTP requests ​ sample_rate Optional[int] default: "None" Output sample rate in Hz. When None, the sample rate depends on the voice model being used by the Piper server. ​ text_filter BaseTextFilter default: "None" Modifies text provided to the TTS. Learn more about the available filters. ​ Input The service accepts text input through its TTS pipeline. ​ Output Frames ​ TTSStartedFrame Signals the start of audio generation. ​ TTSAudioRawFrame Contains generated audio data: ​ audio bytes Raw audio data chunk ​ sample_rate int Audio sample rate (depends on the Piper model) ​ num_channels int Number of audio channels (1 for mono) ​ TTSStoppedFrame Signals the completion of audio generation. ​ ErrorFrame Signals that an error occurred during audio generation: ​ error str Error message ​ Methods See the TTS base class methods for additional functionality. ​ Usage Example Copy Ask AI import aiohttp from pipecat.services.piper.tts import PiperTTSService # Create aiohttp session session = aiohttp.ClientSession() # Configure service tts = PiperTTSService( base_url = "http://localhost:5000/api/tts" , aiohttp_session = session, sample_rate = 22050 # Optional: specify if you know the model's sample rate ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ Frame Flow ​ Metrics Support The service supports metrics collection: Time to First Byte (TTFB) TTS usage metrics Processing duration ​ Audio Processing Streams audio in 1KB chunks Automatically handles WAV headers in the response Outputs mono audio Supports the sample rate specified by your Piper voice model ​ Notes Requires a running Piper TTS server Self-hosted solution with no external API dependencies Streams audio in chunks for efficient processing Automatically handles WAV headers in the response Provides metrics collection OpenAI PlayHT On this page Overview Installation Configuration Constructor Parameters Input Output Frames TTSStartedFrame TTSAudioRawFrame TTSStoppedFrame ErrorFrame Methods Usage Example Frame Flow Metrics Support Audio Processing Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_piper_976de1a6.txt b/tts_piper_976de1a6.txt
new file mode 100644
index 0000000000000000000000000000000000000000..9d11f21424b18623d168fed21efd7bad1de2ffe7
--- /dev/null
+++ b/tts_piper_976de1a6.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/piper#input
+Title: Piper - Pipecat
+==================================================
+
+Piper - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech Piper Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview PiperTTSService converts text to speech using the Piper TTS server. This service provides integration with a locally-running Piper TTS service, offering self-hosted speech synthesis capabilities. ​ Installation To use PiperTTSService , no additional dependencies in Pipecat are required. You’ll also need to set up a running Piper TTS server following the Piper HTTP server documentation . ​ Configuration ​ Constructor Parameters ​ base_url str required API base URL for the Piper TTS server (without a trailing slash) ​ aiohttp_session aiohttp.ClientSession required aiohttp ClientSession for making HTTP requests ​ sample_rate Optional[int] default: "None" Output sample rate in Hz. When None, the sample rate depends on the voice model being used by the Piper server. ​ text_filter BaseTextFilter default: "None" Modifies text provided to the TTS. Learn more about the available filters. ​ Input The service accepts text input through its TTS pipeline. ​ Output Frames ​ TTSStartedFrame Signals the start of audio generation. ​ TTSAudioRawFrame Contains generated audio data: ​ audio bytes Raw audio data chunk ​ sample_rate int Audio sample rate (depends on the Piper model) ​ num_channels int Number of audio channels (1 for mono) ​ TTSStoppedFrame Signals the completion of audio generation. ​ ErrorFrame Signals that an error occurred during audio generation: ​ error str Error message ​ Methods See the TTS base class methods for additional functionality. ​ Usage Example Copy Ask AI import aiohttp from pipecat.services.piper.tts import PiperTTSService # Create aiohttp session session = aiohttp.ClientSession() # Configure service tts = PiperTTSService( base_url = "http://localhost:5000/api/tts" , aiohttp_session = session, sample_rate = 22050 # Optional: specify if you know the model's sample rate ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ Frame Flow ​ Metrics Support The service supports metrics collection: Time to First Byte (TTFB) TTS usage metrics Processing duration ​ Audio Processing Streams audio in 1KB chunks Automatically handles WAV headers in the response Outputs mono audio Supports the sample rate specified by your Piper voice model ​ Notes Requires a running Piper TTS server Self-hosted solution with no external API dependencies Streams audio in chunks for efficient processing Automatically handles WAV headers in the response Provides metrics collection OpenAI PlayHT On this page Overview Installation Configuration Constructor Parameters Input Output Frames TTSStartedFrame TTSAudioRawFrame TTSStoppedFrame ErrorFrame Methods Usage Example Frame Flow Metrics Support Audio Processing Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_piper_bcaf3903.txt b/tts_piper_bcaf3903.txt
new file mode 100644
index 0000000000000000000000000000000000000000..f54056f54ed5401e086d8cd287909d952941b892
--- /dev/null
+++ b/tts_piper_bcaf3903.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/piper#param-audio
+Title: Piper - Pipecat
+==================================================
+
+Piper - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech Piper Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview PiperTTSService converts text to speech using the Piper TTS server. This service provides integration with a locally-running Piper TTS service, offering self-hosted speech synthesis capabilities. ​ Installation To use PiperTTSService , no additional dependencies in Pipecat are required. You’ll also need to set up a running Piper TTS server following the Piper HTTP server documentation . ​ Configuration ​ Constructor Parameters ​ base_url str required API base URL for the Piper TTS server (without a trailing slash) ​ aiohttp_session aiohttp.ClientSession required aiohttp ClientSession for making HTTP requests ​ sample_rate Optional[int] default: "None" Output sample rate in Hz. When None, the sample rate depends on the voice model being used by the Piper server. ​ text_filter BaseTextFilter default: "None" Modifies text provided to the TTS. Learn more about the available filters. ​ Input The service accepts text input through its TTS pipeline. ​ Output Frames ​ TTSStartedFrame Signals the start of audio generation. ​ TTSAudioRawFrame Contains generated audio data: ​ audio bytes Raw audio data chunk ​ sample_rate int Audio sample rate (depends on the Piper model) ​ num_channels int Number of audio channels (1 for mono) ​ TTSStoppedFrame Signals the completion of audio generation. ​ ErrorFrame Signals that an error occurred during audio generation: ​ error str Error message ​ Methods See the TTS base class methods for additional functionality. ​ Usage Example Copy Ask AI import aiohttp from pipecat.services.piper.tts import PiperTTSService # Create aiohttp session session = aiohttp.ClientSession() # Configure service tts = PiperTTSService( base_url = "http://localhost:5000/api/tts" , aiohttp_session = session, sample_rate = 22050 # Optional: specify if you know the model's sample rate ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ Frame Flow ​ Metrics Support The service supports metrics collection: Time to First Byte (TTFB) TTS usage metrics Processing duration ​ Audio Processing Streams audio in 1KB chunks Automatically handles WAV headers in the response Outputs mono audio Supports the sample rate specified by your Piper voice model ​ Notes Requires a running Piper TTS server Self-hosted solution with no external API dependencies Streams audio in chunks for efficient processing Automatically handles WAV headers in the response Provides metrics collection OpenAI PlayHT On this page Overview Installation Configuration Constructor Parameters Input Output Frames TTSStartedFrame TTSAudioRawFrame TTSStoppedFrame ErrorFrame Methods Usage Example Frame Flow Metrics Support Audio Processing Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_playht_08c41bbb.txt b/tts_playht_08c41bbb.txt
new file mode 100644
index 0000000000000000000000000000000000000000..e772d0b9b805ed9daa1f56f0e31d96a53e48c92b
--- /dev/null
+++ b/tts_playht_08c41bbb.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/playht#param-tts-stopped-frame
+Title: PlayHT - Pipecat
+==================================================
+
+PlayHT - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech PlayHT Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview PlayHT provides two TTS service implementations: PlayHTTTSService : WebSocket-based service with real-time streaming PlayHTHttpTTSService : HTTP-based service for simpler, non-streaming synthesis ​ Installation To use PlayHT services, install the required dependencies: Copy Ask AI pip install "pipecat-ai[playht]" You’ll also need to set up your PlayHT credentials as environment variables: PLAY_HT_USER_ID PLAY_HT_API_KEY ​ PlayHTTTSService WebSocket-based implementation supporting real-time streaming synthesis. ​ Constructor Parameters ​ api_key str required PlayHT API key ​ user_id str required PlayHT user ID ​ voice_url str required Voice identifier URL ​ voice_engine str default: "PlayHT3.0-mini" TTS engine identifier. See the PlayHT docs for available engines. ​ sample_rate int default: "None" Output audio sample rate in Hz ​ output_format str default: "wav" Audio output format ​ text_filter BaseTextFilter default: "None" Modifies text provided to the TTS. Learn more about the available filters. ​ Input Parameters Copy Ask AI class InputParams ( BaseModel ): language: Optional[Language] = Language. EN speed: Optional[ float ] = 1.0 seed: Optional[ int ] = None ​ PlayHTHttpTTSService HTTP-based implementation for simpler synthesis requirements. ​ Constructor Parameters ​ api_key str required PlayHT API key ​ user_id str required PlayHT user ID ​ voice_url str required Voice identifier URL ​ voice_engine str default: "Play3.0-mini-http" TTS engine identifier. The PlayHTHttpTTSService supports either Play3.0-mini-http or Play3.0-mini-ws . ​ sample_rate int default: "None" Output audio sample rate in Hz ​ Input Parameters Copy Ask AI class InputParams ( BaseModel ): language: Optional[Language] = Language. EN speed: Optional[ float ] = 1.0 seed: Optional[ int ] = None ​ Output Frames ​ Control Frames ​ TTSStartedFrame Frame Signals start of synthesis ​ TTSStoppedFrame Frame Signals completion of synthesis ​ Audio Frames ​ TTSAudioRawFrame Frame Contains generated audio data with: - WAV format - Specified sample rate - Single channel (mono) ​ Error Frames ​ ErrorFrame Frame Contains PlayHT error information ​ Methods See the TTS base class methods for additional functionality. ​ Language Support Supports multiple languages when using the PlayHT3.0-mini engine: Language Code Description Service Code Language.AF Afrikaans afrikans Language.AM Amharic amharic Language.AR Arabic arabic Language.BN Bengali bengali Language.BG Bulgarian bulgarian Language.CA Catalan catalan Language.CS Czech czech Language.DA Danish danish Language.DE German german Language.EL Greek greek Language.EN English english Language.ES Spanish spanish Language.FR French french Language.GL Galician galician Language.HE Hebrew hebrew Language.HI Hindi hindi Language.HR Croatian croatian Language.HU Hungarian hungarian Language.ID Indonesian indonesian Language.IT Italian italian Language.JA Japanese japanese Language.KO Korean korean Language.MS Malay malay Language.NL Dutch dutch Language.PL Polish polish Language.PT Portuguese portuguese Language.RU Russian russian Language.SQ Albanian albanian Language.SR Serbian serbian Language.SV Swedish swedish Language.TH Thai thai Language.TL Tagalog tagalog Language.TR Turkish turkish Language.UK Ukrainian ukrainian Language.UR Urdu urdu Language.XH Xhosa xhosa Language.ZH Mandarin mandarin See the PlayHT docs for a complete list of languages and options. ​ Usage Examples ​ WebSocket Service Copy Ask AI # Configure WebSocket service ws_service = PlayHTTTSService( api_key = "your-api-key" , user_id = "your-user-id" , voice_url = "voice-url" , voice_engine = "PlayHT3.0-mini" , params = PlayHTTTSService.InputParams( language = Language. EN , speed = 1.2 ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ HTTP Service Copy Ask AI # Configure HTTP service http_service = PlayHTHttpTTSService( api_key = "your-api-key" , user_id = "your-user-id" , voice_url = "voice-url" , voice_engine = "PlayHT3.0-mini" , params = PlayHTHttpTTSService.InputParams( language = Language. EN , speed = 1.0 ) ) ​ Frame Flow ​ WebSocket Service ​ HTTP Service ​ Metrics Support Both services collect processing metrics: Time to First Byte (TTFB) Processing duration Character usage API calls ​ Notes ​ WebSocket Service Real-time streaming support Automatic reconnection Interruption handling WAV header management Thread-safe processing ​ HTTP Service Simpler implementation Complete audio delivery WAV header parsing Chunked audio delivery Lower latency for short texts ​ Common Features Multiple voice engines Speed control Language support Seed-based consistency Error handling Metrics collection Piper Rime On this page Overview Installation PlayHTTTSService Constructor Parameters Input Parameters PlayHTHttpTTSService Constructor Parameters Input Parameters Output Frames Control Frames Audio Frames Error Frames Methods Language Support Usage Examples WebSocket Service HTTP Service Frame Flow WebSocket Service HTTP Service Metrics Support Notes WebSocket Service HTTP Service Common Features Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_playht_0c87f71e.txt b/tts_playht_0c87f71e.txt
new file mode 100644
index 0000000000000000000000000000000000000000..e8a126c2c848b6425ed7c1c28e31f60f5eeb4f12
--- /dev/null
+++ b/tts_playht_0c87f71e.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/playht#websocket-service-2
+Title: PlayHT - Pipecat
+==================================================
+
+PlayHT - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech PlayHT Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview PlayHT provides two TTS service implementations: PlayHTTTSService : WebSocket-based service with real-time streaming PlayHTHttpTTSService : HTTP-based service for simpler, non-streaming synthesis ​ Installation To use PlayHT services, install the required dependencies: Copy Ask AI pip install "pipecat-ai[playht]" You’ll also need to set up your PlayHT credentials as environment variables: PLAY_HT_USER_ID PLAY_HT_API_KEY ​ PlayHTTTSService WebSocket-based implementation supporting real-time streaming synthesis. ​ Constructor Parameters ​ api_key str required PlayHT API key ​ user_id str required PlayHT user ID ​ voice_url str required Voice identifier URL ​ voice_engine str default: "PlayHT3.0-mini" TTS engine identifier. See the PlayHT docs for available engines. ​ sample_rate int default: "None" Output audio sample rate in Hz ​ output_format str default: "wav" Audio output format ​ text_filter BaseTextFilter default: "None" Modifies text provided to the TTS. Learn more about the available filters. ​ Input Parameters Copy Ask AI class InputParams ( BaseModel ): language: Optional[Language] = Language. EN speed: Optional[ float ] = 1.0 seed: Optional[ int ] = None ​ PlayHTHttpTTSService HTTP-based implementation for simpler synthesis requirements. ​ Constructor Parameters ​ api_key str required PlayHT API key ​ user_id str required PlayHT user ID ​ voice_url str required Voice identifier URL ​ voice_engine str default: "Play3.0-mini-http" TTS engine identifier. The PlayHTHttpTTSService supports either Play3.0-mini-http or Play3.0-mini-ws . ​ sample_rate int default: "None" Output audio sample rate in Hz ​ Input Parameters Copy Ask AI class InputParams ( BaseModel ): language: Optional[Language] = Language. EN speed: Optional[ float ] = 1.0 seed: Optional[ int ] = None ​ Output Frames ​ Control Frames ​ TTSStartedFrame Frame Signals start of synthesis ​ TTSStoppedFrame Frame Signals completion of synthesis ​ Audio Frames ​ TTSAudioRawFrame Frame Contains generated audio data with: - WAV format - Specified sample rate - Single channel (mono) ​ Error Frames ​ ErrorFrame Frame Contains PlayHT error information ​ Methods See the TTS base class methods for additional functionality. ​ Language Support Supports multiple languages when using the PlayHT3.0-mini engine: Language Code Description Service Code Language.AF Afrikaans afrikans Language.AM Amharic amharic Language.AR Arabic arabic Language.BN Bengali bengali Language.BG Bulgarian bulgarian Language.CA Catalan catalan Language.CS Czech czech Language.DA Danish danish Language.DE German german Language.EL Greek greek Language.EN English english Language.ES Spanish spanish Language.FR French french Language.GL Galician galician Language.HE Hebrew hebrew Language.HI Hindi hindi Language.HR Croatian croatian Language.HU Hungarian hungarian Language.ID Indonesian indonesian Language.IT Italian italian Language.JA Japanese japanese Language.KO Korean korean Language.MS Malay malay Language.NL Dutch dutch Language.PL Polish polish Language.PT Portuguese portuguese Language.RU Russian russian Language.SQ Albanian albanian Language.SR Serbian serbian Language.SV Swedish swedish Language.TH Thai thai Language.TL Tagalog tagalog Language.TR Turkish turkish Language.UK Ukrainian ukrainian Language.UR Urdu urdu Language.XH Xhosa xhosa Language.ZH Mandarin mandarin See the PlayHT docs for a complete list of languages and options. ​ Usage Examples ​ WebSocket Service Copy Ask AI # Configure WebSocket service ws_service = PlayHTTTSService( api_key = "your-api-key" , user_id = "your-user-id" , voice_url = "voice-url" , voice_engine = "PlayHT3.0-mini" , params = PlayHTTTSService.InputParams( language = Language. EN , speed = 1.2 ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ HTTP Service Copy Ask AI # Configure HTTP service http_service = PlayHTHttpTTSService( api_key = "your-api-key" , user_id = "your-user-id" , voice_url = "voice-url" , voice_engine = "PlayHT3.0-mini" , params = PlayHTHttpTTSService.InputParams( language = Language. EN , speed = 1.0 ) ) ​ Frame Flow ​ WebSocket Service ​ HTTP Service ​ Metrics Support Both services collect processing metrics: Time to First Byte (TTFB) Processing duration Character usage API calls ​ Notes ​ WebSocket Service Real-time streaming support Automatic reconnection Interruption handling WAV header management Thread-safe processing ​ HTTP Service Simpler implementation Complete audio delivery WAV header parsing Chunked audio delivery Lower latency for short texts ​ Common Features Multiple voice engines Speed control Language support Seed-based consistency Error handling Metrics collection Piper Rime On this page Overview Installation PlayHTTTSService Constructor Parameters Input Parameters PlayHTHttpTTSService Constructor Parameters Input Parameters Output Frames Control Frames Audio Frames Error Frames Methods Language Support Usage Examples WebSocket Service HTTP Service Frame Flow WebSocket Service HTTP Service Metrics Support Notes WebSocket Service HTTP Service Common Features Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_playht_27e4ab10.txt b/tts_playht_27e4ab10.txt
new file mode 100644
index 0000000000000000000000000000000000000000..3f596b841f18fd7ce328fdc6f525a8796c6abfee
--- /dev/null
+++ b/tts_playht_27e4ab10.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/playht#error-frames
+Title: PlayHT - Pipecat
+==================================================
+
+PlayHT - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech PlayHT Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview PlayHT provides two TTS service implementations: PlayHTTTSService : WebSocket-based service with real-time streaming PlayHTHttpTTSService : HTTP-based service for simpler, non-streaming synthesis ​ Installation To use PlayHT services, install the required dependencies: Copy Ask AI pip install "pipecat-ai[playht]" You’ll also need to set up your PlayHT credentials as environment variables: PLAY_HT_USER_ID PLAY_HT_API_KEY ​ PlayHTTTSService WebSocket-based implementation supporting real-time streaming synthesis. ​ Constructor Parameters ​ api_key str required PlayHT API key ​ user_id str required PlayHT user ID ​ voice_url str required Voice identifier URL ​ voice_engine str default: "PlayHT3.0-mini" TTS engine identifier. See the PlayHT docs for available engines. ​ sample_rate int default: "None" Output audio sample rate in Hz ​ output_format str default: "wav" Audio output format ​ text_filter BaseTextFilter default: "None" Modifies text provided to the TTS. Learn more about the available filters. ​ Input Parameters Copy Ask AI class InputParams ( BaseModel ): language: Optional[Language] = Language. EN speed: Optional[ float ] = 1.0 seed: Optional[ int ] = None ​ PlayHTHttpTTSService HTTP-based implementation for simpler synthesis requirements. ​ Constructor Parameters ​ api_key str required PlayHT API key ​ user_id str required PlayHT user ID ​ voice_url str required Voice identifier URL ​ voice_engine str default: "Play3.0-mini-http" TTS engine identifier. The PlayHTHttpTTSService supports either Play3.0-mini-http or Play3.0-mini-ws . ​ sample_rate int default: "None" Output audio sample rate in Hz ​ Input Parameters Copy Ask AI class InputParams ( BaseModel ): language: Optional[Language] = Language. EN speed: Optional[ float ] = 1.0 seed: Optional[ int ] = None ​ Output Frames ​ Control Frames ​ TTSStartedFrame Frame Signals start of synthesis ​ TTSStoppedFrame Frame Signals completion of synthesis ​ Audio Frames ​ TTSAudioRawFrame Frame Contains generated audio data with: - WAV format - Specified sample rate - Single channel (mono) ​ Error Frames ​ ErrorFrame Frame Contains PlayHT error information ​ Methods See the TTS base class methods for additional functionality. ​ Language Support Supports multiple languages when using the PlayHT3.0-mini engine: Language Code Description Service Code Language.AF Afrikaans afrikans Language.AM Amharic amharic Language.AR Arabic arabic Language.BN Bengali bengali Language.BG Bulgarian bulgarian Language.CA Catalan catalan Language.CS Czech czech Language.DA Danish danish Language.DE German german Language.EL Greek greek Language.EN English english Language.ES Spanish spanish Language.FR French french Language.GL Galician galician Language.HE Hebrew hebrew Language.HI Hindi hindi Language.HR Croatian croatian Language.HU Hungarian hungarian Language.ID Indonesian indonesian Language.IT Italian italian Language.JA Japanese japanese Language.KO Korean korean Language.MS Malay malay Language.NL Dutch dutch Language.PL Polish polish Language.PT Portuguese portuguese Language.RU Russian russian Language.SQ Albanian albanian Language.SR Serbian serbian Language.SV Swedish swedish Language.TH Thai thai Language.TL Tagalog tagalog Language.TR Turkish turkish Language.UK Ukrainian ukrainian Language.UR Urdu urdu Language.XH Xhosa xhosa Language.ZH Mandarin mandarin See the PlayHT docs for a complete list of languages and options. ​ Usage Examples ​ WebSocket Service Copy Ask AI # Configure WebSocket service ws_service = PlayHTTTSService( api_key = "your-api-key" , user_id = "your-user-id" , voice_url = "voice-url" , voice_engine = "PlayHT3.0-mini" , params = PlayHTTTSService.InputParams( language = Language. EN , speed = 1.2 ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ HTTP Service Copy Ask AI # Configure HTTP service http_service = PlayHTHttpTTSService( api_key = "your-api-key" , user_id = "your-user-id" , voice_url = "voice-url" , voice_engine = "PlayHT3.0-mini" , params = PlayHTHttpTTSService.InputParams( language = Language. EN , speed = 1.0 ) ) ​ Frame Flow ​ WebSocket Service ​ HTTP Service ​ Metrics Support Both services collect processing metrics: Time to First Byte (TTFB) Processing duration Character usage API calls ​ Notes ​ WebSocket Service Real-time streaming support Automatic reconnection Interruption handling WAV header management Thread-safe processing ​ HTTP Service Simpler implementation Complete audio delivery WAV header parsing Chunked audio delivery Lower latency for short texts ​ Common Features Multiple voice engines Speed control Language support Seed-based consistency Error handling Metrics collection Piper Rime On this page Overview Installation PlayHTTTSService Constructor Parameters Input Parameters PlayHTHttpTTSService Constructor Parameters Input Parameters Output Frames Control Frames Audio Frames Error Frames Methods Language Support Usage Examples WebSocket Service HTTP Service Frame Flow WebSocket Service HTTP Service Metrics Support Notes WebSocket Service HTTP Service Common Features Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_playht_3615c327.txt b/tts_playht_3615c327.txt
new file mode 100644
index 0000000000000000000000000000000000000000..d34dfc3e0396929d5084d687986e26c77e9aa2ee
--- /dev/null
+++ b/tts_playht_3615c327.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/playht#installation
+Title: PlayHT - Pipecat
+==================================================
+
+PlayHT - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech PlayHT Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview PlayHT provides two TTS service implementations: PlayHTTTSService : WebSocket-based service with real-time streaming PlayHTHttpTTSService : HTTP-based service for simpler, non-streaming synthesis ​ Installation To use PlayHT services, install the required dependencies: Copy Ask AI pip install "pipecat-ai[playht]" You’ll also need to set up your PlayHT credentials as environment variables: PLAY_HT_USER_ID PLAY_HT_API_KEY ​ PlayHTTTSService WebSocket-based implementation supporting real-time streaming synthesis. ​ Constructor Parameters ​ api_key str required PlayHT API key ​ user_id str required PlayHT user ID ​ voice_url str required Voice identifier URL ​ voice_engine str default: "PlayHT3.0-mini" TTS engine identifier. See the PlayHT docs for available engines. ​ sample_rate int default: "None" Output audio sample rate in Hz ​ output_format str default: "wav" Audio output format ​ text_filter BaseTextFilter default: "None" Modifies text provided to the TTS. Learn more about the available filters. ​ Input Parameters Copy Ask AI class InputParams ( BaseModel ): language: Optional[Language] = Language. EN speed: Optional[ float ] = 1.0 seed: Optional[ int ] = None ​ PlayHTHttpTTSService HTTP-based implementation for simpler synthesis requirements. ​ Constructor Parameters ​ api_key str required PlayHT API key ​ user_id str required PlayHT user ID ​ voice_url str required Voice identifier URL ​ voice_engine str default: "Play3.0-mini-http" TTS engine identifier. The PlayHTHttpTTSService supports either Play3.0-mini-http or Play3.0-mini-ws . ​ sample_rate int default: "None" Output audio sample rate in Hz ​ Input Parameters Copy Ask AI class InputParams ( BaseModel ): language: Optional[Language] = Language. EN speed: Optional[ float ] = 1.0 seed: Optional[ int ] = None ​ Output Frames ​ Control Frames ​ TTSStartedFrame Frame Signals start of synthesis ​ TTSStoppedFrame Frame Signals completion of synthesis ​ Audio Frames ​ TTSAudioRawFrame Frame Contains generated audio data with: - WAV format - Specified sample rate - Single channel (mono) ​ Error Frames ​ ErrorFrame Frame Contains PlayHT error information ​ Methods See the TTS base class methods for additional functionality. ​ Language Support Supports multiple languages when using the PlayHT3.0-mini engine: Language Code Description Service Code Language.AF Afrikaans afrikans Language.AM Amharic amharic Language.AR Arabic arabic Language.BN Bengali bengali Language.BG Bulgarian bulgarian Language.CA Catalan catalan Language.CS Czech czech Language.DA Danish danish Language.DE German german Language.EL Greek greek Language.EN English english Language.ES Spanish spanish Language.FR French french Language.GL Galician galician Language.HE Hebrew hebrew Language.HI Hindi hindi Language.HR Croatian croatian Language.HU Hungarian hungarian Language.ID Indonesian indonesian Language.IT Italian italian Language.JA Japanese japanese Language.KO Korean korean Language.MS Malay malay Language.NL Dutch dutch Language.PL Polish polish Language.PT Portuguese portuguese Language.RU Russian russian Language.SQ Albanian albanian Language.SR Serbian serbian Language.SV Swedish swedish Language.TH Thai thai Language.TL Tagalog tagalog Language.TR Turkish turkish Language.UK Ukrainian ukrainian Language.UR Urdu urdu Language.XH Xhosa xhosa Language.ZH Mandarin mandarin See the PlayHT docs for a complete list of languages and options. ​ Usage Examples ​ WebSocket Service Copy Ask AI # Configure WebSocket service ws_service = PlayHTTTSService( api_key = "your-api-key" , user_id = "your-user-id" , voice_url = "voice-url" , voice_engine = "PlayHT3.0-mini" , params = PlayHTTTSService.InputParams( language = Language. EN , speed = 1.2 ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ HTTP Service Copy Ask AI # Configure HTTP service http_service = PlayHTHttpTTSService( api_key = "your-api-key" , user_id = "your-user-id" , voice_url = "voice-url" , voice_engine = "PlayHT3.0-mini" , params = PlayHTHttpTTSService.InputParams( language = Language. EN , speed = 1.0 ) ) ​ Frame Flow ​ WebSocket Service ​ HTTP Service ​ Metrics Support Both services collect processing metrics: Time to First Byte (TTFB) Processing duration Character usage API calls ​ Notes ​ WebSocket Service Real-time streaming support Automatic reconnection Interruption handling WAV header management Thread-safe processing ​ HTTP Service Simpler implementation Complete audio delivery WAV header parsing Chunked audio delivery Lower latency for short texts ​ Common Features Multiple voice engines Speed control Language support Seed-based consistency Error handling Metrics collection Piper Rime On this page Overview Installation PlayHTTTSService Constructor Parameters Input Parameters PlayHTHttpTTSService Constructor Parameters Input Parameters Output Frames Control Frames Audio Frames Error Frames Methods Language Support Usage Examples WebSocket Service HTTP Service Frame Flow WebSocket Service HTTP Service Metrics Support Notes WebSocket Service HTTP Service Common Features Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_playht_4eed3903.txt b/tts_playht_4eed3903.txt
new file mode 100644
index 0000000000000000000000000000000000000000..39023ede6811bebc51909370bbca7cff62d55039
--- /dev/null
+++ b/tts_playht_4eed3903.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/playht
+Title: PlayHT - Pipecat
+==================================================
+
+PlayHT - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech PlayHT Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview PlayHT provides two TTS service implementations: PlayHTTTSService : WebSocket-based service with real-time streaming PlayHTHttpTTSService : HTTP-based service for simpler, non-streaming synthesis ​ Installation To use PlayHT services, install the required dependencies: Copy Ask AI pip install "pipecat-ai[playht]" You’ll also need to set up your PlayHT credentials as environment variables: PLAY_HT_USER_ID PLAY_HT_API_KEY ​ PlayHTTTSService WebSocket-based implementation supporting real-time streaming synthesis. ​ Constructor Parameters ​ api_key str required PlayHT API key ​ user_id str required PlayHT user ID ​ voice_url str required Voice identifier URL ​ voice_engine str default: "PlayHT3.0-mini" TTS engine identifier. See the PlayHT docs for available engines. ​ sample_rate int default: "None" Output audio sample rate in Hz ​ output_format str default: "wav" Audio output format ​ text_filter BaseTextFilter default: "None" Modifies text provided to the TTS. Learn more about the available filters. ​ Input Parameters Copy Ask AI class InputParams ( BaseModel ): language: Optional[Language] = Language. EN speed: Optional[ float ] = 1.0 seed: Optional[ int ] = None ​ PlayHTHttpTTSService HTTP-based implementation for simpler synthesis requirements. ​ Constructor Parameters ​ api_key str required PlayHT API key ​ user_id str required PlayHT user ID ​ voice_url str required Voice identifier URL ​ voice_engine str default: "Play3.0-mini-http" TTS engine identifier. The PlayHTHttpTTSService supports either Play3.0-mini-http or Play3.0-mini-ws . ​ sample_rate int default: "None" Output audio sample rate in Hz ​ Input Parameters Copy Ask AI class InputParams ( BaseModel ): language: Optional[Language] = Language. EN speed: Optional[ float ] = 1.0 seed: Optional[ int ] = None ​ Output Frames ​ Control Frames ​ TTSStartedFrame Frame Signals start of synthesis ​ TTSStoppedFrame Frame Signals completion of synthesis ​ Audio Frames ​ TTSAudioRawFrame Frame Contains generated audio data with: - WAV format - Specified sample rate - Single channel (mono) ​ Error Frames ​ ErrorFrame Frame Contains PlayHT error information ​ Methods See the TTS base class methods for additional functionality. ​ Language Support Supports multiple languages when using the PlayHT3.0-mini engine: Language Code Description Service Code Language.AF Afrikaans afrikans Language.AM Amharic amharic Language.AR Arabic arabic Language.BN Bengali bengali Language.BG Bulgarian bulgarian Language.CA Catalan catalan Language.CS Czech czech Language.DA Danish danish Language.DE German german Language.EL Greek greek Language.EN English english Language.ES Spanish spanish Language.FR French french Language.GL Galician galician Language.HE Hebrew hebrew Language.HI Hindi hindi Language.HR Croatian croatian Language.HU Hungarian hungarian Language.ID Indonesian indonesian Language.IT Italian italian Language.JA Japanese japanese Language.KO Korean korean Language.MS Malay malay Language.NL Dutch dutch Language.PL Polish polish Language.PT Portuguese portuguese Language.RU Russian russian Language.SQ Albanian albanian Language.SR Serbian serbian Language.SV Swedish swedish Language.TH Thai thai Language.TL Tagalog tagalog Language.TR Turkish turkish Language.UK Ukrainian ukrainian Language.UR Urdu urdu Language.XH Xhosa xhosa Language.ZH Mandarin mandarin See the PlayHT docs for a complete list of languages and options. ​ Usage Examples ​ WebSocket Service Copy Ask AI # Configure WebSocket service ws_service = PlayHTTTSService( api_key = "your-api-key" , user_id = "your-user-id" , voice_url = "voice-url" , voice_engine = "PlayHT3.0-mini" , params = PlayHTTTSService.InputParams( language = Language. EN , speed = 1.2 ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ HTTP Service Copy Ask AI # Configure HTTP service http_service = PlayHTHttpTTSService( api_key = "your-api-key" , user_id = "your-user-id" , voice_url = "voice-url" , voice_engine = "PlayHT3.0-mini" , params = PlayHTHttpTTSService.InputParams( language = Language. EN , speed = 1.0 ) ) ​ Frame Flow ​ WebSocket Service ​ HTTP Service ​ Metrics Support Both services collect processing metrics: Time to First Byte (TTFB) Processing duration Character usage API calls ​ Notes ​ WebSocket Service Real-time streaming support Automatic reconnection Interruption handling WAV header management Thread-safe processing ​ HTTP Service Simpler implementation Complete audio delivery WAV header parsing Chunked audio delivery Lower latency for short texts ​ Common Features Multiple voice engines Speed control Language support Seed-based consistency Error handling Metrics collection Piper Rime On this page Overview Installation PlayHTTTSService Constructor Parameters Input Parameters PlayHTHttpTTSService Constructor Parameters Input Parameters Output Frames Control Frames Audio Frames Error Frames Methods Language Support Usage Examples WebSocket Service HTTP Service Frame Flow WebSocket Service HTTP Service Metrics Support Notes WebSocket Service HTTP Service Common Features Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_playht_578ca420.txt b/tts_playht_578ca420.txt
new file mode 100644
index 0000000000000000000000000000000000000000..847a91f0ec50295c26840cd842390eb0d1c504cc
--- /dev/null
+++ b/tts_playht_578ca420.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/playht#output-frames
+Title: PlayHT - Pipecat
+==================================================
+
+PlayHT - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech PlayHT Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview PlayHT provides two TTS service implementations: PlayHTTTSService : WebSocket-based service with real-time streaming PlayHTHttpTTSService : HTTP-based service for simpler, non-streaming synthesis ​ Installation To use PlayHT services, install the required dependencies: Copy Ask AI pip install "pipecat-ai[playht]" You’ll also need to set up your PlayHT credentials as environment variables: PLAY_HT_USER_ID PLAY_HT_API_KEY ​ PlayHTTTSService WebSocket-based implementation supporting real-time streaming synthesis. ​ Constructor Parameters ​ api_key str required PlayHT API key ​ user_id str required PlayHT user ID ​ voice_url str required Voice identifier URL ​ voice_engine str default: "PlayHT3.0-mini" TTS engine identifier. See the PlayHT docs for available engines. ​ sample_rate int default: "None" Output audio sample rate in Hz ​ output_format str default: "wav" Audio output format ​ text_filter BaseTextFilter default: "None" Modifies text provided to the TTS. Learn more about the available filters. ​ Input Parameters Copy Ask AI class InputParams ( BaseModel ): language: Optional[Language] = Language. EN speed: Optional[ float ] = 1.0 seed: Optional[ int ] = None ​ PlayHTHttpTTSService HTTP-based implementation for simpler synthesis requirements. ​ Constructor Parameters ​ api_key str required PlayHT API key ​ user_id str required PlayHT user ID ​ voice_url str required Voice identifier URL ​ voice_engine str default: "Play3.0-mini-http" TTS engine identifier. The PlayHTHttpTTSService supports either Play3.0-mini-http or Play3.0-mini-ws . ​ sample_rate int default: "None" Output audio sample rate in Hz ​ Input Parameters Copy Ask AI class InputParams ( BaseModel ): language: Optional[Language] = Language. EN speed: Optional[ float ] = 1.0 seed: Optional[ int ] = None ​ Output Frames ​ Control Frames ​ TTSStartedFrame Frame Signals start of synthesis ​ TTSStoppedFrame Frame Signals completion of synthesis ​ Audio Frames ​ TTSAudioRawFrame Frame Contains generated audio data with: - WAV format - Specified sample rate - Single channel (mono) ​ Error Frames ​ ErrorFrame Frame Contains PlayHT error information ​ Methods See the TTS base class methods for additional functionality. ​ Language Support Supports multiple languages when using the PlayHT3.0-mini engine: Language Code Description Service Code Language.AF Afrikaans afrikans Language.AM Amharic amharic Language.AR Arabic arabic Language.BN Bengali bengali Language.BG Bulgarian bulgarian Language.CA Catalan catalan Language.CS Czech czech Language.DA Danish danish Language.DE German german Language.EL Greek greek Language.EN English english Language.ES Spanish spanish Language.FR French french Language.GL Galician galician Language.HE Hebrew hebrew Language.HI Hindi hindi Language.HR Croatian croatian Language.HU Hungarian hungarian Language.ID Indonesian indonesian Language.IT Italian italian Language.JA Japanese japanese Language.KO Korean korean Language.MS Malay malay Language.NL Dutch dutch Language.PL Polish polish Language.PT Portuguese portuguese Language.RU Russian russian Language.SQ Albanian albanian Language.SR Serbian serbian Language.SV Swedish swedish Language.TH Thai thai Language.TL Tagalog tagalog Language.TR Turkish turkish Language.UK Ukrainian ukrainian Language.UR Urdu urdu Language.XH Xhosa xhosa Language.ZH Mandarin mandarin See the PlayHT docs for a complete list of languages and options. ​ Usage Examples ​ WebSocket Service Copy Ask AI # Configure WebSocket service ws_service = PlayHTTTSService( api_key = "your-api-key" , user_id = "your-user-id" , voice_url = "voice-url" , voice_engine = "PlayHT3.0-mini" , params = PlayHTTTSService.InputParams( language = Language. EN , speed = 1.2 ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ HTTP Service Copy Ask AI # Configure HTTP service http_service = PlayHTHttpTTSService( api_key = "your-api-key" , user_id = "your-user-id" , voice_url = "voice-url" , voice_engine = "PlayHT3.0-mini" , params = PlayHTHttpTTSService.InputParams( language = Language. EN , speed = 1.0 ) ) ​ Frame Flow ​ WebSocket Service ​ HTTP Service ​ Metrics Support Both services collect processing metrics: Time to First Byte (TTFB) Processing duration Character usage API calls ​ Notes ​ WebSocket Service Real-time streaming support Automatic reconnection Interruption handling WAV header management Thread-safe processing ​ HTTP Service Simpler implementation Complete audio delivery WAV header parsing Chunked audio delivery Lower latency for short texts ​ Common Features Multiple voice engines Speed control Language support Seed-based consistency Error handling Metrics collection Piper Rime On this page Overview Installation PlayHTTTSService Constructor Parameters Input Parameters PlayHTHttpTTSService Constructor Parameters Input Parameters Output Frames Control Frames Audio Frames Error Frames Methods Language Support Usage Examples WebSocket Service HTTP Service Frame Flow WebSocket Service HTTP Service Metrics Support Notes WebSocket Service HTTP Service Common Features Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_playht_60aeccfa.txt b/tts_playht_60aeccfa.txt
new file mode 100644
index 0000000000000000000000000000000000000000..517ae018de1ea56bcddd0f952b046c00d9a2c303
--- /dev/null
+++ b/tts_playht_60aeccfa.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/playht#frame-flow
+Title: PlayHT - Pipecat
+==================================================
+
+PlayHT - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech PlayHT Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview PlayHT provides two TTS service implementations: PlayHTTTSService : WebSocket-based service with real-time streaming PlayHTHttpTTSService : HTTP-based service for simpler, non-streaming synthesis ​ Installation To use PlayHT services, install the required dependencies: Copy Ask AI pip install "pipecat-ai[playht]" You’ll also need to set up your PlayHT credentials as environment variables: PLAY_HT_USER_ID PLAY_HT_API_KEY ​ PlayHTTTSService WebSocket-based implementation supporting real-time streaming synthesis. ​ Constructor Parameters ​ api_key str required PlayHT API key ​ user_id str required PlayHT user ID ​ voice_url str required Voice identifier URL ​ voice_engine str default: "PlayHT3.0-mini" TTS engine identifier. See the PlayHT docs for available engines. ​ sample_rate int default: "None" Output audio sample rate in Hz ​ output_format str default: "wav" Audio output format ​ text_filter BaseTextFilter default: "None" Modifies text provided to the TTS. Learn more about the available filters. ​ Input Parameters Copy Ask AI class InputParams ( BaseModel ): language: Optional[Language] = Language. EN speed: Optional[ float ] = 1.0 seed: Optional[ int ] = None ​ PlayHTHttpTTSService HTTP-based implementation for simpler synthesis requirements. ​ Constructor Parameters ​ api_key str required PlayHT API key ​ user_id str required PlayHT user ID ​ voice_url str required Voice identifier URL ​ voice_engine str default: "Play3.0-mini-http" TTS engine identifier. The PlayHTHttpTTSService supports either Play3.0-mini-http or Play3.0-mini-ws . ​ sample_rate int default: "None" Output audio sample rate in Hz ​ Input Parameters Copy Ask AI class InputParams ( BaseModel ): language: Optional[Language] = Language. EN speed: Optional[ float ] = 1.0 seed: Optional[ int ] = None ​ Output Frames ​ Control Frames ​ TTSStartedFrame Frame Signals start of synthesis ​ TTSStoppedFrame Frame Signals completion of synthesis ​ Audio Frames ​ TTSAudioRawFrame Frame Contains generated audio data with: - WAV format - Specified sample rate - Single channel (mono) ​ Error Frames ​ ErrorFrame Frame Contains PlayHT error information ​ Methods See the TTS base class methods for additional functionality. ​ Language Support Supports multiple languages when using the PlayHT3.0-mini engine: Language Code Description Service Code Language.AF Afrikaans afrikans Language.AM Amharic amharic Language.AR Arabic arabic Language.BN Bengali bengali Language.BG Bulgarian bulgarian Language.CA Catalan catalan Language.CS Czech czech Language.DA Danish danish Language.DE German german Language.EL Greek greek Language.EN English english Language.ES Spanish spanish Language.FR French french Language.GL Galician galician Language.HE Hebrew hebrew Language.HI Hindi hindi Language.HR Croatian croatian Language.HU Hungarian hungarian Language.ID Indonesian indonesian Language.IT Italian italian Language.JA Japanese japanese Language.KO Korean korean Language.MS Malay malay Language.NL Dutch dutch Language.PL Polish polish Language.PT Portuguese portuguese Language.RU Russian russian Language.SQ Albanian albanian Language.SR Serbian serbian Language.SV Swedish swedish Language.TH Thai thai Language.TL Tagalog tagalog Language.TR Turkish turkish Language.UK Ukrainian ukrainian Language.UR Urdu urdu Language.XH Xhosa xhosa Language.ZH Mandarin mandarin See the PlayHT docs for a complete list of languages and options. ​ Usage Examples ​ WebSocket Service Copy Ask AI # Configure WebSocket service ws_service = PlayHTTTSService( api_key = "your-api-key" , user_id = "your-user-id" , voice_url = "voice-url" , voice_engine = "PlayHT3.0-mini" , params = PlayHTTTSService.InputParams( language = Language. EN , speed = 1.2 ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ HTTP Service Copy Ask AI # Configure HTTP service http_service = PlayHTHttpTTSService( api_key = "your-api-key" , user_id = "your-user-id" , voice_url = "voice-url" , voice_engine = "PlayHT3.0-mini" , params = PlayHTHttpTTSService.InputParams( language = Language. EN , speed = 1.0 ) ) ​ Frame Flow ​ WebSocket Service ​ HTTP Service ​ Metrics Support Both services collect processing metrics: Time to First Byte (TTFB) Processing duration Character usage API calls ​ Notes ​ WebSocket Service Real-time streaming support Automatic reconnection Interruption handling WAV header management Thread-safe processing ​ HTTP Service Simpler implementation Complete audio delivery WAV header parsing Chunked audio delivery Lower latency for short texts ​ Common Features Multiple voice engines Speed control Language support Seed-based consistency Error handling Metrics collection Piper Rime On this page Overview Installation PlayHTTTSService Constructor Parameters Input Parameters PlayHTHttpTTSService Constructor Parameters Input Parameters Output Frames Control Frames Audio Frames Error Frames Methods Language Support Usage Examples WebSocket Service HTTP Service Frame Flow WebSocket Service HTTP Service Metrics Support Notes WebSocket Service HTTP Service Common Features Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_playht_646410ba.txt b/tts_playht_646410ba.txt
new file mode 100644
index 0000000000000000000000000000000000000000..525bb94d05b465aa5de060bcdc351f83e07cd0ca
--- /dev/null
+++ b/tts_playht_646410ba.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/playht#playhtttsservice
+Title: PlayHT - Pipecat
+==================================================
+
+PlayHT - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech PlayHT Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview PlayHT provides two TTS service implementations: PlayHTTTSService : WebSocket-based service with real-time streaming PlayHTHttpTTSService : HTTP-based service for simpler, non-streaming synthesis ​ Installation To use PlayHT services, install the required dependencies: Copy Ask AI pip install "pipecat-ai[playht]" You’ll also need to set up your PlayHT credentials as environment variables: PLAY_HT_USER_ID PLAY_HT_API_KEY ​ PlayHTTTSService WebSocket-based implementation supporting real-time streaming synthesis. ​ Constructor Parameters ​ api_key str required PlayHT API key ​ user_id str required PlayHT user ID ​ voice_url str required Voice identifier URL ​ voice_engine str default: "PlayHT3.0-mini" TTS engine identifier. See the PlayHT docs for available engines. ​ sample_rate int default: "None" Output audio sample rate in Hz ​ output_format str default: "wav" Audio output format ​ text_filter BaseTextFilter default: "None" Modifies text provided to the TTS. Learn more about the available filters. ​ Input Parameters Copy Ask AI class InputParams ( BaseModel ): language: Optional[Language] = Language. EN speed: Optional[ float ] = 1.0 seed: Optional[ int ] = None ​ PlayHTHttpTTSService HTTP-based implementation for simpler synthesis requirements. ​ Constructor Parameters ​ api_key str required PlayHT API key ​ user_id str required PlayHT user ID ​ voice_url str required Voice identifier URL ​ voice_engine str default: "Play3.0-mini-http" TTS engine identifier. The PlayHTHttpTTSService supports either Play3.0-mini-http or Play3.0-mini-ws . ​ sample_rate int default: "None" Output audio sample rate in Hz ​ Input Parameters Copy Ask AI class InputParams ( BaseModel ): language: Optional[Language] = Language. EN speed: Optional[ float ] = 1.0 seed: Optional[ int ] = None ​ Output Frames ​ Control Frames ​ TTSStartedFrame Frame Signals start of synthesis ​ TTSStoppedFrame Frame Signals completion of synthesis ​ Audio Frames ​ TTSAudioRawFrame Frame Contains generated audio data with: - WAV format - Specified sample rate - Single channel (mono) ​ Error Frames ​ ErrorFrame Frame Contains PlayHT error information ​ Methods See the TTS base class methods for additional functionality. ​ Language Support Supports multiple languages when using the PlayHT3.0-mini engine: Language Code Description Service Code Language.AF Afrikaans afrikans Language.AM Amharic amharic Language.AR Arabic arabic Language.BN Bengali bengali Language.BG Bulgarian bulgarian Language.CA Catalan catalan Language.CS Czech czech Language.DA Danish danish Language.DE German german Language.EL Greek greek Language.EN English english Language.ES Spanish spanish Language.FR French french Language.GL Galician galician Language.HE Hebrew hebrew Language.HI Hindi hindi Language.HR Croatian croatian Language.HU Hungarian hungarian Language.ID Indonesian indonesian Language.IT Italian italian Language.JA Japanese japanese Language.KO Korean korean Language.MS Malay malay Language.NL Dutch dutch Language.PL Polish polish Language.PT Portuguese portuguese Language.RU Russian russian Language.SQ Albanian albanian Language.SR Serbian serbian Language.SV Swedish swedish Language.TH Thai thai Language.TL Tagalog tagalog Language.TR Turkish turkish Language.UK Ukrainian ukrainian Language.UR Urdu urdu Language.XH Xhosa xhosa Language.ZH Mandarin mandarin See the PlayHT docs for a complete list of languages and options. ​ Usage Examples ​ WebSocket Service Copy Ask AI # Configure WebSocket service ws_service = PlayHTTTSService( api_key = "your-api-key" , user_id = "your-user-id" , voice_url = "voice-url" , voice_engine = "PlayHT3.0-mini" , params = PlayHTTTSService.InputParams( language = Language. EN , speed = 1.2 ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ HTTP Service Copy Ask AI # Configure HTTP service http_service = PlayHTHttpTTSService( api_key = "your-api-key" , user_id = "your-user-id" , voice_url = "voice-url" , voice_engine = "PlayHT3.0-mini" , params = PlayHTHttpTTSService.InputParams( language = Language. EN , speed = 1.0 ) ) ​ Frame Flow ​ WebSocket Service ​ HTTP Service ​ Metrics Support Both services collect processing metrics: Time to First Byte (TTFB) Processing duration Character usage API calls ​ Notes ​ WebSocket Service Real-time streaming support Automatic reconnection Interruption handling WAV header management Thread-safe processing ​ HTTP Service Simpler implementation Complete audio delivery WAV header parsing Chunked audio delivery Lower latency for short texts ​ Common Features Multiple voice engines Speed control Language support Seed-based consistency Error handling Metrics collection Piper Rime On this page Overview Installation PlayHTTTSService Constructor Parameters Input Parameters PlayHTHttpTTSService Constructor Parameters Input Parameters Output Frames Control Frames Audio Frames Error Frames Methods Language Support Usage Examples WebSocket Service HTTP Service Frame Flow WebSocket Service HTTP Service Metrics Support Notes WebSocket Service HTTP Service Common Features Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_playht_68aa69dc.txt b/tts_playht_68aa69dc.txt
new file mode 100644
index 0000000000000000000000000000000000000000..20ff669f85f5c97812621433ff91d04c66aee95a
--- /dev/null
+++ b/tts_playht_68aa69dc.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/playht#common-features
+Title: PlayHT - Pipecat
+==================================================
+
+PlayHT - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech PlayHT Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview PlayHT provides two TTS service implementations: PlayHTTTSService : WebSocket-based service with real-time streaming PlayHTHttpTTSService : HTTP-based service for simpler, non-streaming synthesis ​ Installation To use PlayHT services, install the required dependencies: Copy Ask AI pip install "pipecat-ai[playht]" You’ll also need to set up your PlayHT credentials as environment variables: PLAY_HT_USER_ID PLAY_HT_API_KEY ​ PlayHTTTSService WebSocket-based implementation supporting real-time streaming synthesis. ​ Constructor Parameters ​ api_key str required PlayHT API key ​ user_id str required PlayHT user ID ​ voice_url str required Voice identifier URL ​ voice_engine str default: "PlayHT3.0-mini" TTS engine identifier. See the PlayHT docs for available engines. ​ sample_rate int default: "None" Output audio sample rate in Hz ​ output_format str default: "wav" Audio output format ​ text_filter BaseTextFilter default: "None" Modifies text provided to the TTS. Learn more about the available filters. ​ Input Parameters Copy Ask AI class InputParams ( BaseModel ): language: Optional[Language] = Language. EN speed: Optional[ float ] = 1.0 seed: Optional[ int ] = None ​ PlayHTHttpTTSService HTTP-based implementation for simpler synthesis requirements. ​ Constructor Parameters ​ api_key str required PlayHT API key ​ user_id str required PlayHT user ID ​ voice_url str required Voice identifier URL ​ voice_engine str default: "Play3.0-mini-http" TTS engine identifier. The PlayHTHttpTTSService supports either Play3.0-mini-http or Play3.0-mini-ws . ​ sample_rate int default: "None" Output audio sample rate in Hz ​ Input Parameters Copy Ask AI class InputParams ( BaseModel ): language: Optional[Language] = Language. EN speed: Optional[ float ] = 1.0 seed: Optional[ int ] = None ​ Output Frames ​ Control Frames ​ TTSStartedFrame Frame Signals start of synthesis ​ TTSStoppedFrame Frame Signals completion of synthesis ​ Audio Frames ​ TTSAudioRawFrame Frame Contains generated audio data with: - WAV format - Specified sample rate - Single channel (mono) ​ Error Frames ​ ErrorFrame Frame Contains PlayHT error information ​ Methods See the TTS base class methods for additional functionality. ​ Language Support Supports multiple languages when using the PlayHT3.0-mini engine: Language Code Description Service Code Language.AF Afrikaans afrikans Language.AM Amharic amharic Language.AR Arabic arabic Language.BN Bengali bengali Language.BG Bulgarian bulgarian Language.CA Catalan catalan Language.CS Czech czech Language.DA Danish danish Language.DE German german Language.EL Greek greek Language.EN English english Language.ES Spanish spanish Language.FR French french Language.GL Galician galician Language.HE Hebrew hebrew Language.HI Hindi hindi Language.HR Croatian croatian Language.HU Hungarian hungarian Language.ID Indonesian indonesian Language.IT Italian italian Language.JA Japanese japanese Language.KO Korean korean Language.MS Malay malay Language.NL Dutch dutch Language.PL Polish polish Language.PT Portuguese portuguese Language.RU Russian russian Language.SQ Albanian albanian Language.SR Serbian serbian Language.SV Swedish swedish Language.TH Thai thai Language.TL Tagalog tagalog Language.TR Turkish turkish Language.UK Ukrainian ukrainian Language.UR Urdu urdu Language.XH Xhosa xhosa Language.ZH Mandarin mandarin See the PlayHT docs for a complete list of languages and options. ​ Usage Examples ​ WebSocket Service Copy Ask AI # Configure WebSocket service ws_service = PlayHTTTSService( api_key = "your-api-key" , user_id = "your-user-id" , voice_url = "voice-url" , voice_engine = "PlayHT3.0-mini" , params = PlayHTTTSService.InputParams( language = Language. EN , speed = 1.2 ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ HTTP Service Copy Ask AI # Configure HTTP service http_service = PlayHTHttpTTSService( api_key = "your-api-key" , user_id = "your-user-id" , voice_url = "voice-url" , voice_engine = "PlayHT3.0-mini" , params = PlayHTHttpTTSService.InputParams( language = Language. EN , speed = 1.0 ) ) ​ Frame Flow ​ WebSocket Service ​ HTTP Service ​ Metrics Support Both services collect processing metrics: Time to First Byte (TTFB) Processing duration Character usage API calls ​ Notes ​ WebSocket Service Real-time streaming support Automatic reconnection Interruption handling WAV header management Thread-safe processing ​ HTTP Service Simpler implementation Complete audio delivery WAV header parsing Chunked audio delivery Lower latency for short texts ​ Common Features Multiple voice engines Speed control Language support Seed-based consistency Error handling Metrics collection Piper Rime On this page Overview Installation PlayHTTTSService Constructor Parameters Input Parameters PlayHTHttpTTSService Constructor Parameters Input Parameters Output Frames Control Frames Audio Frames Error Frames Methods Language Support Usage Examples WebSocket Service HTTP Service Frame Flow WebSocket Service HTTP Service Metrics Support Notes WebSocket Service HTTP Service Common Features Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_playht_9dda38d8.txt b/tts_playht_9dda38d8.txt
new file mode 100644
index 0000000000000000000000000000000000000000..d3e8c206a7836b0860c35f8da37947af47a75895
--- /dev/null
+++ b/tts_playht_9dda38d8.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/playht#playhthttpttsservice
+Title: PlayHT - Pipecat
+==================================================
+
+PlayHT - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech PlayHT Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview PlayHT provides two TTS service implementations: PlayHTTTSService : WebSocket-based service with real-time streaming PlayHTHttpTTSService : HTTP-based service for simpler, non-streaming synthesis ​ Installation To use PlayHT services, install the required dependencies: Copy Ask AI pip install "pipecat-ai[playht]" You’ll also need to set up your PlayHT credentials as environment variables: PLAY_HT_USER_ID PLAY_HT_API_KEY ​ PlayHTTTSService WebSocket-based implementation supporting real-time streaming synthesis. ​ Constructor Parameters ​ api_key str required PlayHT API key ​ user_id str required PlayHT user ID ​ voice_url str required Voice identifier URL ​ voice_engine str default: "PlayHT3.0-mini" TTS engine identifier. See the PlayHT docs for available engines. ​ sample_rate int default: "None" Output audio sample rate in Hz ​ output_format str default: "wav" Audio output format ​ text_filter BaseTextFilter default: "None" Modifies text provided to the TTS. Learn more about the available filters. ​ Input Parameters Copy Ask AI class InputParams ( BaseModel ): language: Optional[Language] = Language. EN speed: Optional[ float ] = 1.0 seed: Optional[ int ] = None ​ PlayHTHttpTTSService HTTP-based implementation for simpler synthesis requirements. ​ Constructor Parameters ​ api_key str required PlayHT API key ​ user_id str required PlayHT user ID ​ voice_url str required Voice identifier URL ​ voice_engine str default: "Play3.0-mini-http" TTS engine identifier. The PlayHTHttpTTSService supports either Play3.0-mini-http or Play3.0-mini-ws . ​ sample_rate int default: "None" Output audio sample rate in Hz ​ Input Parameters Copy Ask AI class InputParams ( BaseModel ): language: Optional[Language] = Language. EN speed: Optional[ float ] = 1.0 seed: Optional[ int ] = None ​ Output Frames ​ Control Frames ​ TTSStartedFrame Frame Signals start of synthesis ​ TTSStoppedFrame Frame Signals completion of synthesis ​ Audio Frames ​ TTSAudioRawFrame Frame Contains generated audio data with: - WAV format - Specified sample rate - Single channel (mono) ​ Error Frames ​ ErrorFrame Frame Contains PlayHT error information ​ Methods See the TTS base class methods for additional functionality. ​ Language Support Supports multiple languages when using the PlayHT3.0-mini engine: Language Code Description Service Code Language.AF Afrikaans afrikans Language.AM Amharic amharic Language.AR Arabic arabic Language.BN Bengali bengali Language.BG Bulgarian bulgarian Language.CA Catalan catalan Language.CS Czech czech Language.DA Danish danish Language.DE German german Language.EL Greek greek Language.EN English english Language.ES Spanish spanish Language.FR French french Language.GL Galician galician Language.HE Hebrew hebrew Language.HI Hindi hindi Language.HR Croatian croatian Language.HU Hungarian hungarian Language.ID Indonesian indonesian Language.IT Italian italian Language.JA Japanese japanese Language.KO Korean korean Language.MS Malay malay Language.NL Dutch dutch Language.PL Polish polish Language.PT Portuguese portuguese Language.RU Russian russian Language.SQ Albanian albanian Language.SR Serbian serbian Language.SV Swedish swedish Language.TH Thai thai Language.TL Tagalog tagalog Language.TR Turkish turkish Language.UK Ukrainian ukrainian Language.UR Urdu urdu Language.XH Xhosa xhosa Language.ZH Mandarin mandarin See the PlayHT docs for a complete list of languages and options. ​ Usage Examples ​ WebSocket Service Copy Ask AI # Configure WebSocket service ws_service = PlayHTTTSService( api_key = "your-api-key" , user_id = "your-user-id" , voice_url = "voice-url" , voice_engine = "PlayHT3.0-mini" , params = PlayHTTTSService.InputParams( language = Language. EN , speed = 1.2 ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ HTTP Service Copy Ask AI # Configure HTTP service http_service = PlayHTHttpTTSService( api_key = "your-api-key" , user_id = "your-user-id" , voice_url = "voice-url" , voice_engine = "PlayHT3.0-mini" , params = PlayHTHttpTTSService.InputParams( language = Language. EN , speed = 1.0 ) ) ​ Frame Flow ​ WebSocket Service ​ HTTP Service ​ Metrics Support Both services collect processing metrics: Time to First Byte (TTFB) Processing duration Character usage API calls ​ Notes ​ WebSocket Service Real-time streaming support Automatic reconnection Interruption handling WAV header management Thread-safe processing ​ HTTP Service Simpler implementation Complete audio delivery WAV header parsing Chunked audio delivery Lower latency for short texts ​ Common Features Multiple voice engines Speed control Language support Seed-based consistency Error handling Metrics collection Piper Rime On this page Overview Installation PlayHTTTSService Constructor Parameters Input Parameters PlayHTHttpTTSService Constructor Parameters Input Parameters Output Frames Control Frames Audio Frames Error Frames Methods Language Support Usage Examples WebSocket Service HTTP Service Frame Flow WebSocket Service HTTP Service Metrics Support Notes WebSocket Service HTTP Service Common Features Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_playht_b2631a7f.txt b/tts_playht_b2631a7f.txt
new file mode 100644
index 0000000000000000000000000000000000000000..5f935c55ecdfb97384eae5a5d4b5d6486f26c914
--- /dev/null
+++ b/tts_playht_b2631a7f.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/playht#param-text-filter
+Title: PlayHT - Pipecat
+==================================================
+
+PlayHT - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech PlayHT Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview PlayHT provides two TTS service implementations: PlayHTTTSService : WebSocket-based service with real-time streaming PlayHTHttpTTSService : HTTP-based service for simpler, non-streaming synthesis ​ Installation To use PlayHT services, install the required dependencies: Copy Ask AI pip install "pipecat-ai[playht]" You’ll also need to set up your PlayHT credentials as environment variables: PLAY_HT_USER_ID PLAY_HT_API_KEY ​ PlayHTTTSService WebSocket-based implementation supporting real-time streaming synthesis. ​ Constructor Parameters ​ api_key str required PlayHT API key ​ user_id str required PlayHT user ID ​ voice_url str required Voice identifier URL ​ voice_engine str default: "PlayHT3.0-mini" TTS engine identifier. See the PlayHT docs for available engines. ​ sample_rate int default: "None" Output audio sample rate in Hz ​ output_format str default: "wav" Audio output format ​ text_filter BaseTextFilter default: "None" Modifies text provided to the TTS. Learn more about the available filters. ​ Input Parameters Copy Ask AI class InputParams ( BaseModel ): language: Optional[Language] = Language. EN speed: Optional[ float ] = 1.0 seed: Optional[ int ] = None ​ PlayHTHttpTTSService HTTP-based implementation for simpler synthesis requirements. ​ Constructor Parameters ​ api_key str required PlayHT API key ​ user_id str required PlayHT user ID ​ voice_url str required Voice identifier URL ​ voice_engine str default: "Play3.0-mini-http" TTS engine identifier. The PlayHTHttpTTSService supports either Play3.0-mini-http or Play3.0-mini-ws . ​ sample_rate int default: "None" Output audio sample rate in Hz ​ Input Parameters Copy Ask AI class InputParams ( BaseModel ): language: Optional[Language] = Language. EN speed: Optional[ float ] = 1.0 seed: Optional[ int ] = None ​ Output Frames ​ Control Frames ​ TTSStartedFrame Frame Signals start of synthesis ​ TTSStoppedFrame Frame Signals completion of synthesis ​ Audio Frames ​ TTSAudioRawFrame Frame Contains generated audio data with: - WAV format - Specified sample rate - Single channel (mono) ​ Error Frames ​ ErrorFrame Frame Contains PlayHT error information ​ Methods See the TTS base class methods for additional functionality. ​ Language Support Supports multiple languages when using the PlayHT3.0-mini engine: Language Code Description Service Code Language.AF Afrikaans afrikans Language.AM Amharic amharic Language.AR Arabic arabic Language.BN Bengali bengali Language.BG Bulgarian bulgarian Language.CA Catalan catalan Language.CS Czech czech Language.DA Danish danish Language.DE German german Language.EL Greek greek Language.EN English english Language.ES Spanish spanish Language.FR French french Language.GL Galician galician Language.HE Hebrew hebrew Language.HI Hindi hindi Language.HR Croatian croatian Language.HU Hungarian hungarian Language.ID Indonesian indonesian Language.IT Italian italian Language.JA Japanese japanese Language.KO Korean korean Language.MS Malay malay Language.NL Dutch dutch Language.PL Polish polish Language.PT Portuguese portuguese Language.RU Russian russian Language.SQ Albanian albanian Language.SR Serbian serbian Language.SV Swedish swedish Language.TH Thai thai Language.TL Tagalog tagalog Language.TR Turkish turkish Language.UK Ukrainian ukrainian Language.UR Urdu urdu Language.XH Xhosa xhosa Language.ZH Mandarin mandarin See the PlayHT docs for a complete list of languages and options. ​ Usage Examples ​ WebSocket Service Copy Ask AI # Configure WebSocket service ws_service = PlayHTTTSService( api_key = "your-api-key" , user_id = "your-user-id" , voice_url = "voice-url" , voice_engine = "PlayHT3.0-mini" , params = PlayHTTTSService.InputParams( language = Language. EN , speed = 1.2 ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ HTTP Service Copy Ask AI # Configure HTTP service http_service = PlayHTHttpTTSService( api_key = "your-api-key" , user_id = "your-user-id" , voice_url = "voice-url" , voice_engine = "PlayHT3.0-mini" , params = PlayHTHttpTTSService.InputParams( language = Language. EN , speed = 1.0 ) ) ​ Frame Flow ​ WebSocket Service ​ HTTP Service ​ Metrics Support Both services collect processing metrics: Time to First Byte (TTFB) Processing duration Character usage API calls ​ Notes ​ WebSocket Service Real-time streaming support Automatic reconnection Interruption handling WAV header management Thread-safe processing ​ HTTP Service Simpler implementation Complete audio delivery WAV header parsing Chunked audio delivery Lower latency for short texts ​ Common Features Multiple voice engines Speed control Language support Seed-based consistency Error handling Metrics collection Piper Rime On this page Overview Installation PlayHTTTSService Constructor Parameters Input Parameters PlayHTHttpTTSService Constructor Parameters Input Parameters Output Frames Control Frames Audio Frames Error Frames Methods Language Support Usage Examples WebSocket Service HTTP Service Frame Flow WebSocket Service HTTP Service Metrics Support Notes WebSocket Service HTTP Service Common Features Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_playht_bfce5e52.txt b/tts_playht_bfce5e52.txt
new file mode 100644
index 0000000000000000000000000000000000000000..0f0f6be5d58fb8c2da2bdab5ee0338be4b5106c0
--- /dev/null
+++ b/tts_playht_bfce5e52.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/playht#websocket-service
+Title: PlayHT - Pipecat
+==================================================
+
+PlayHT - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech PlayHT Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview PlayHT provides two TTS service implementations: PlayHTTTSService : WebSocket-based service with real-time streaming PlayHTHttpTTSService : HTTP-based service for simpler, non-streaming synthesis ​ Installation To use PlayHT services, install the required dependencies: Copy Ask AI pip install "pipecat-ai[playht]" You’ll also need to set up your PlayHT credentials as environment variables: PLAY_HT_USER_ID PLAY_HT_API_KEY ​ PlayHTTTSService WebSocket-based implementation supporting real-time streaming synthesis. ​ Constructor Parameters ​ api_key str required PlayHT API key ​ user_id str required PlayHT user ID ​ voice_url str required Voice identifier URL ​ voice_engine str default: "PlayHT3.0-mini" TTS engine identifier. See the PlayHT docs for available engines. ​ sample_rate int default: "None" Output audio sample rate in Hz ​ output_format str default: "wav" Audio output format ​ text_filter BaseTextFilter default: "None" Modifies text provided to the TTS. Learn more about the available filters. ​ Input Parameters Copy Ask AI class InputParams ( BaseModel ): language: Optional[Language] = Language. EN speed: Optional[ float ] = 1.0 seed: Optional[ int ] = None ​ PlayHTHttpTTSService HTTP-based implementation for simpler synthesis requirements. ​ Constructor Parameters ​ api_key str required PlayHT API key ​ user_id str required PlayHT user ID ​ voice_url str required Voice identifier URL ​ voice_engine str default: "Play3.0-mini-http" TTS engine identifier. The PlayHTHttpTTSService supports either Play3.0-mini-http or Play3.0-mini-ws . ​ sample_rate int default: "None" Output audio sample rate in Hz ​ Input Parameters Copy Ask AI class InputParams ( BaseModel ): language: Optional[Language] = Language. EN speed: Optional[ float ] = 1.0 seed: Optional[ int ] = None ​ Output Frames ​ Control Frames ​ TTSStartedFrame Frame Signals start of synthesis ​ TTSStoppedFrame Frame Signals completion of synthesis ​ Audio Frames ​ TTSAudioRawFrame Frame Contains generated audio data with: - WAV format - Specified sample rate - Single channel (mono) ​ Error Frames ​ ErrorFrame Frame Contains PlayHT error information ​ Methods See the TTS base class methods for additional functionality. ​ Language Support Supports multiple languages when using the PlayHT3.0-mini engine: Language Code Description Service Code Language.AF Afrikaans afrikans Language.AM Amharic amharic Language.AR Arabic arabic Language.BN Bengali bengali Language.BG Bulgarian bulgarian Language.CA Catalan catalan Language.CS Czech czech Language.DA Danish danish Language.DE German german Language.EL Greek greek Language.EN English english Language.ES Spanish spanish Language.FR French french Language.GL Galician galician Language.HE Hebrew hebrew Language.HI Hindi hindi Language.HR Croatian croatian Language.HU Hungarian hungarian Language.ID Indonesian indonesian Language.IT Italian italian Language.JA Japanese japanese Language.KO Korean korean Language.MS Malay malay Language.NL Dutch dutch Language.PL Polish polish Language.PT Portuguese portuguese Language.RU Russian russian Language.SQ Albanian albanian Language.SR Serbian serbian Language.SV Swedish swedish Language.TH Thai thai Language.TL Tagalog tagalog Language.TR Turkish turkish Language.UK Ukrainian ukrainian Language.UR Urdu urdu Language.XH Xhosa xhosa Language.ZH Mandarin mandarin See the PlayHT docs for a complete list of languages and options. ​ Usage Examples ​ WebSocket Service Copy Ask AI # Configure WebSocket service ws_service = PlayHTTTSService( api_key = "your-api-key" , user_id = "your-user-id" , voice_url = "voice-url" , voice_engine = "PlayHT3.0-mini" , params = PlayHTTTSService.InputParams( language = Language. EN , speed = 1.2 ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ HTTP Service Copy Ask AI # Configure HTTP service http_service = PlayHTHttpTTSService( api_key = "your-api-key" , user_id = "your-user-id" , voice_url = "voice-url" , voice_engine = "PlayHT3.0-mini" , params = PlayHTHttpTTSService.InputParams( language = Language. EN , speed = 1.0 ) ) ​ Frame Flow ​ WebSocket Service ​ HTTP Service ​ Metrics Support Both services collect processing metrics: Time to First Byte (TTFB) Processing duration Character usage API calls ​ Notes ​ WebSocket Service Real-time streaming support Automatic reconnection Interruption handling WAV header management Thread-safe processing ​ HTTP Service Simpler implementation Complete audio delivery WAV header parsing Chunked audio delivery Lower latency for short texts ​ Common Features Multiple voice engines Speed control Language support Seed-based consistency Error handling Metrics collection Piper Rime On this page Overview Installation PlayHTTTSService Constructor Parameters Input Parameters PlayHTHttpTTSService Constructor Parameters Input Parameters Output Frames Control Frames Audio Frames Error Frames Methods Language Support Usage Examples WebSocket Service HTTP Service Frame Flow WebSocket Service HTTP Service Metrics Support Notes WebSocket Service HTTP Service Common Features Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_playht_f92b099b.txt b/tts_playht_f92b099b.txt
new file mode 100644
index 0000000000000000000000000000000000000000..ea2f8586cf8e792bdd753457b31b4c1333875860
--- /dev/null
+++ b/tts_playht_f92b099b.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/playht#http-service-2
+Title: PlayHT - Pipecat
+==================================================
+
+PlayHT - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech PlayHT Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview PlayHT provides two TTS service implementations: PlayHTTTSService : WebSocket-based service with real-time streaming PlayHTHttpTTSService : HTTP-based service for simpler, non-streaming synthesis ​ Installation To use PlayHT services, install the required dependencies: Copy Ask AI pip install "pipecat-ai[playht]" You’ll also need to set up your PlayHT credentials as environment variables: PLAY_HT_USER_ID PLAY_HT_API_KEY ​ PlayHTTTSService WebSocket-based implementation supporting real-time streaming synthesis. ​ Constructor Parameters ​ api_key str required PlayHT API key ​ user_id str required PlayHT user ID ​ voice_url str required Voice identifier URL ​ voice_engine str default: "PlayHT3.0-mini" TTS engine identifier. See the PlayHT docs for available engines. ​ sample_rate int default: "None" Output audio sample rate in Hz ​ output_format str default: "wav" Audio output format ​ text_filter BaseTextFilter default: "None" Modifies text provided to the TTS. Learn more about the available filters. ​ Input Parameters Copy Ask AI class InputParams ( BaseModel ): language: Optional[Language] = Language. EN speed: Optional[ float ] = 1.0 seed: Optional[ int ] = None ​ PlayHTHttpTTSService HTTP-based implementation for simpler synthesis requirements. ​ Constructor Parameters ​ api_key str required PlayHT API key ​ user_id str required PlayHT user ID ​ voice_url str required Voice identifier URL ​ voice_engine str default: "Play3.0-mini-http" TTS engine identifier. The PlayHTHttpTTSService supports either Play3.0-mini-http or Play3.0-mini-ws . ​ sample_rate int default: "None" Output audio sample rate in Hz ​ Input Parameters Copy Ask AI class InputParams ( BaseModel ): language: Optional[Language] = Language. EN speed: Optional[ float ] = 1.0 seed: Optional[ int ] = None ​ Output Frames ​ Control Frames ​ TTSStartedFrame Frame Signals start of synthesis ​ TTSStoppedFrame Frame Signals completion of synthesis ​ Audio Frames ​ TTSAudioRawFrame Frame Contains generated audio data with: - WAV format - Specified sample rate - Single channel (mono) ​ Error Frames ​ ErrorFrame Frame Contains PlayHT error information ​ Methods See the TTS base class methods for additional functionality. ​ Language Support Supports multiple languages when using the PlayHT3.0-mini engine: Language Code Description Service Code Language.AF Afrikaans afrikans Language.AM Amharic amharic Language.AR Arabic arabic Language.BN Bengali bengali Language.BG Bulgarian bulgarian Language.CA Catalan catalan Language.CS Czech czech Language.DA Danish danish Language.DE German german Language.EL Greek greek Language.EN English english Language.ES Spanish spanish Language.FR French french Language.GL Galician galician Language.HE Hebrew hebrew Language.HI Hindi hindi Language.HR Croatian croatian Language.HU Hungarian hungarian Language.ID Indonesian indonesian Language.IT Italian italian Language.JA Japanese japanese Language.KO Korean korean Language.MS Malay malay Language.NL Dutch dutch Language.PL Polish polish Language.PT Portuguese portuguese Language.RU Russian russian Language.SQ Albanian albanian Language.SR Serbian serbian Language.SV Swedish swedish Language.TH Thai thai Language.TL Tagalog tagalog Language.TR Turkish turkish Language.UK Ukrainian ukrainian Language.UR Urdu urdu Language.XH Xhosa xhosa Language.ZH Mandarin mandarin See the PlayHT docs for a complete list of languages and options. ​ Usage Examples ​ WebSocket Service Copy Ask AI # Configure WebSocket service ws_service = PlayHTTTSService( api_key = "your-api-key" , user_id = "your-user-id" , voice_url = "voice-url" , voice_engine = "PlayHT3.0-mini" , params = PlayHTTTSService.InputParams( language = Language. EN , speed = 1.2 ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ HTTP Service Copy Ask AI # Configure HTTP service http_service = PlayHTHttpTTSService( api_key = "your-api-key" , user_id = "your-user-id" , voice_url = "voice-url" , voice_engine = "PlayHT3.0-mini" , params = PlayHTHttpTTSService.InputParams( language = Language. EN , speed = 1.0 ) ) ​ Frame Flow ​ WebSocket Service ​ HTTP Service ​ Metrics Support Both services collect processing metrics: Time to First Byte (TTFB) Processing duration Character usage API calls ​ Notes ​ WebSocket Service Real-time streaming support Automatic reconnection Interruption handling WAV header management Thread-safe processing ​ HTTP Service Simpler implementation Complete audio delivery WAV header parsing Chunked audio delivery Lower latency for short texts ​ Common Features Multiple voice engines Speed control Language support Seed-based consistency Error handling Metrics collection Piper Rime On this page Overview Installation PlayHTTTSService Constructor Parameters Input Parameters PlayHTHttpTTSService Constructor Parameters Input Parameters Output Frames Control Frames Audio Frames Error Frames Methods Language Support Usage Examples WebSocket Service HTTP Service Frame Flow WebSocket Service HTTP Service Metrics Support Notes WebSocket Service HTTP Service Common Features Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_rime_0658ead3.txt b/tts_rime_0658ead3.txt
new file mode 100644
index 0000000000000000000000000000000000000000..8582b625b0046ac896c340db73b61ed8fc2b924c
--- /dev/null
+++ b/tts_rime_0658ead3.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/rime#param-pause-between-brackets
+Title: Rime - Pipecat
+==================================================
+
+Rime - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech Rime Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview Rime AI provides two TTS service implementations: RimeTTSService : WebSocket-based implementation with word-level timing and interruption support RimeHttpTTSService : HTTP-based implementation for simpler use cases ​ Installation To use Rime services, install the required dependencies: Copy Ask AI pip install "pipecat-ai[rime]" You can obtain a Rime API key by signing up at Rime . ​ Choosing a Rime service Rime has two supported services: RimeTTSService which is a websocket-based implementation RimeHttpTTSService , which is an HTTP-based implementation ​ RimeTTSService The RimeTTSService is recommended for real-time interactive applications. It offers: Word-level timing information for precise synchronization Support for interruptions and context management Context tracking across multiple messages within a turn Non-blocking operation that allows other frames to be processed while audio is being generated ​ RimeHttpTTSService The RimeHttpTTSService is simpler and more suitable for non-interactive use cases. It: Processes the entire text in one request Supports advanced text control features (pauses, phonemes, inline speed) Blocks during the HTTP request, preventing other frames from being processed until the audio is fully generated ​ Input Parameters Both services use the same base input parameters structure: ​ language Language default: "Language.EN" The language to use for synthesis. See Language Support section for available options. ​ speed_alpha float default: "1.0" Speech rate multiplier. Values less than 1.0 increase speed, values greater than 1.0 decrease speed. ​ reduce_latency bool default: "false" Trade accuracy for lower latency ​ pause_between_brackets bool default: "false" When set to true , adds pauses between words enclosed in angle brackets. The number inside the brackets specifies the pause duration in milliseconds. Example: "Hi. <200> I'd love to have a conversation with you." adds a 200ms pause between the first and second sentences. ​ phonemize_between_brackets bool default: "false" When set to true, you can specify the phonemes for a word enclosed in curly brackets. Example: "{h'El.o} World" will pronounce "Hello" as expected. See Rime’s docs for more details. ​ RimeTTSService (WebSocket) Uses Rime’s WebSocket JSON API for real-time speech synthesis with word-level timing information. ​ Constructor Parameters ​ api_key str required Rime API key ​ voice_id str required Rime voice identifier ​ url str default: "wss://users.rime.ai/ws2" Rime WebSocket API endpoint ​ model str default: "mistv2" Model ID to use for synthesis ​ sample_rate int default: "None" Output audio sample rate in Hz ​ params InputParams default: "InputParams()" Speech generation parameters (see Input Parameters section above) ​ text_aggregator BaseTextAggregator default: "SkipTagsAggregator" Text aggregator for processing input text. Defaults to skipping content between spell( and ) tags. ​ Features Word-level timing information with cumulative timing across messages Support for interruptions with context clearing Context tracking across multiple messages within a turn Real-time audio streaming Proper sentence aggregation with skip tags support ​ RimeHttpTTSService (HTTP) HTTP-based implementation for simpler synthesis requirements. ​ Constructor Parameters ​ api_key str required Rime API key ​ voice_id str required Rime voice identifier. See Rime’s documentation for supported voices. ​ aiohttp_session aiohttp.ClientSession required HTTP session for making requests ​ model str default: "mistv2" Choose mistv2 for hyper-realistic conversational voices, mist for Rime’s previous generation model, or the latest arcana model. ​ sample_rate int default: "None" Output audio sample rate in Hz ​ params InputParams default: "InputParams()" Speech generation parameters Show additional properties ​ inline_speed_alpha str default: "None" Comma-separated list of speed values applied to words in square brackets. Values < 1.0 speed up speech, > 1.0 slow it down. Example: "This sentence is [really] [fast]" with inline_speed_alpha set to "0.5, 3" will make “really” slow and “fast” fast. ​ Output Frames Both services generate the following frames: ​ Control Frames ​ TTSStartedFrame Frame Signals start of speech synthesis ​ TTSStoppedFrame Frame Signals completion of speech synthesis ​ Audio Frames ​ TTSAudioRawFrame Frame Contains generated audio data: PCM audio format, specified sample rate, single channel (mono) ​ Error Frames ​ ErrorFrame Frame Contains Rime TTS error information ​ Language Support Supports multiple languages through the Language enum: Language Code Description Service Code Language.DE German ger Language.EN English eng Language.ES Spanish spa Language.FR French fra ​ Usage Examples ​ WebSocket Service Copy Ask AI from pipecat.services.rime.tts import RimeTTSService from pipecat.transcriptions.language import Language # Configure WebSocket service ws_tts = RimeTTSService( api_key = "your-rime-api-key" , voice_id = "cove" , model = "mistv2" , params = RimeTTSService.InputParams( language = Language. EN , speed_alpha = 1.0 , reduce_latency = False ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ HTTP Service Copy Ask AI import aiohttp from pipecat.services.rime.tts import RimeHttpTTSService # Configure HTTP service async with aiohttp.ClientSession() as session: http_tts = RimeHttpTTSService( api_key = "your-rime-api-key" , voice_id = "eva" , aiohttp_session = session, model = "mistv2" , params = RimeHttpTTSService.InputParams( speed_alpha = 1.2 , reduce_latency = True , pause_between_brackets = True , inline_speed_alpha = "0.8,1.5" ) ) ​ Frame Flow ​ Metrics Support Both services collect processing metrics: Time to First Byte (TTFB) Character usage statistics ​ Service Comparison Feature WebSocket HTTP Word timing ✓ - Interruption support ✓ - Context tracking ✓ - Bracket-based pauses ✓ ✓ Phoneme control ✓ ✓ Inline speed control - ✓ Streaming audio ✓ ✓ Arcana model support - ✓ PlayHT Sarvam AI On this page Overview Installation Choosing a Rime service RimeTTSService RimeHttpTTSService Input Parameters RimeTTSService (WebSocket) Constructor Parameters Features RimeHttpTTSService (HTTP) Constructor Parameters Output Frames Control Frames Audio Frames Error Frames Language Support Usage Examples WebSocket Service HTTP Service Frame Flow Metrics Support Service Comparison Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_rime_10e1d8a1.txt b/tts_rime_10e1d8a1.txt
new file mode 100644
index 0000000000000000000000000000000000000000..597e558db45f58a2b4f24682277616bc110f40c4
--- /dev/null
+++ b/tts_rime_10e1d8a1.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/rime#installation
+Title: Rime - Pipecat
+==================================================
+
+Rime - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech Rime Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview Rime AI provides two TTS service implementations: RimeTTSService : WebSocket-based implementation with word-level timing and interruption support RimeHttpTTSService : HTTP-based implementation for simpler use cases ​ Installation To use Rime services, install the required dependencies: Copy Ask AI pip install "pipecat-ai[rime]" You can obtain a Rime API key by signing up at Rime . ​ Choosing a Rime service Rime has two supported services: RimeTTSService which is a websocket-based implementation RimeHttpTTSService , which is an HTTP-based implementation ​ RimeTTSService The RimeTTSService is recommended for real-time interactive applications. It offers: Word-level timing information for precise synchronization Support for interruptions and context management Context tracking across multiple messages within a turn Non-blocking operation that allows other frames to be processed while audio is being generated ​ RimeHttpTTSService The RimeHttpTTSService is simpler and more suitable for non-interactive use cases. It: Processes the entire text in one request Supports advanced text control features (pauses, phonemes, inline speed) Blocks during the HTTP request, preventing other frames from being processed until the audio is fully generated ​ Input Parameters Both services use the same base input parameters structure: ​ language Language default: "Language.EN" The language to use for synthesis. See Language Support section for available options. ​ speed_alpha float default: "1.0" Speech rate multiplier. Values less than 1.0 increase speed, values greater than 1.0 decrease speed. ​ reduce_latency bool default: "false" Trade accuracy for lower latency ​ pause_between_brackets bool default: "false" When set to true , adds pauses between words enclosed in angle brackets. The number inside the brackets specifies the pause duration in milliseconds. Example: "Hi. <200> I'd love to have a conversation with you." adds a 200ms pause between the first and second sentences. ​ phonemize_between_brackets bool default: "false" When set to true, you can specify the phonemes for a word enclosed in curly brackets. Example: "{h'El.o} World" will pronounce "Hello" as expected. See Rime’s docs for more details. ​ RimeTTSService (WebSocket) Uses Rime’s WebSocket JSON API for real-time speech synthesis with word-level timing information. ​ Constructor Parameters ​ api_key str required Rime API key ​ voice_id str required Rime voice identifier ​ url str default: "wss://users.rime.ai/ws2" Rime WebSocket API endpoint ​ model str default: "mistv2" Model ID to use for synthesis ​ sample_rate int default: "None" Output audio sample rate in Hz ​ params InputParams default: "InputParams()" Speech generation parameters (see Input Parameters section above) ​ text_aggregator BaseTextAggregator default: "SkipTagsAggregator" Text aggregator for processing input text. Defaults to skipping content between spell( and ) tags. ​ Features Word-level timing information with cumulative timing across messages Support for interruptions with context clearing Context tracking across multiple messages within a turn Real-time audio streaming Proper sentence aggregation with skip tags support ​ RimeHttpTTSService (HTTP) HTTP-based implementation for simpler synthesis requirements. ​ Constructor Parameters ​ api_key str required Rime API key ​ voice_id str required Rime voice identifier. See Rime’s documentation for supported voices. ​ aiohttp_session aiohttp.ClientSession required HTTP session for making requests ​ model str default: "mistv2" Choose mistv2 for hyper-realistic conversational voices, mist for Rime’s previous generation model, or the latest arcana model. ​ sample_rate int default: "None" Output audio sample rate in Hz ​ params InputParams default: "InputParams()" Speech generation parameters Show additional properties ​ inline_speed_alpha str default: "None" Comma-separated list of speed values applied to words in square brackets. Values < 1.0 speed up speech, > 1.0 slow it down. Example: "This sentence is [really] [fast]" with inline_speed_alpha set to "0.5, 3" will make “really” slow and “fast” fast. ​ Output Frames Both services generate the following frames: ​ Control Frames ​ TTSStartedFrame Frame Signals start of speech synthesis ​ TTSStoppedFrame Frame Signals completion of speech synthesis ​ Audio Frames ​ TTSAudioRawFrame Frame Contains generated audio data: PCM audio format, specified sample rate, single channel (mono) ​ Error Frames ​ ErrorFrame Frame Contains Rime TTS error information ​ Language Support Supports multiple languages through the Language enum: Language Code Description Service Code Language.DE German ger Language.EN English eng Language.ES Spanish spa Language.FR French fra ​ Usage Examples ​ WebSocket Service Copy Ask AI from pipecat.services.rime.tts import RimeTTSService from pipecat.transcriptions.language import Language # Configure WebSocket service ws_tts = RimeTTSService( api_key = "your-rime-api-key" , voice_id = "cove" , model = "mistv2" , params = RimeTTSService.InputParams( language = Language. EN , speed_alpha = 1.0 , reduce_latency = False ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ HTTP Service Copy Ask AI import aiohttp from pipecat.services.rime.tts import RimeHttpTTSService # Configure HTTP service async with aiohttp.ClientSession() as session: http_tts = RimeHttpTTSService( api_key = "your-rime-api-key" , voice_id = "eva" , aiohttp_session = session, model = "mistv2" , params = RimeHttpTTSService.InputParams( speed_alpha = 1.2 , reduce_latency = True , pause_between_brackets = True , inline_speed_alpha = "0.8,1.5" ) ) ​ Frame Flow ​ Metrics Support Both services collect processing metrics: Time to First Byte (TTFB) Character usage statistics ​ Service Comparison Feature WebSocket HTTP Word timing ✓ - Interruption support ✓ - Context tracking ✓ - Bracket-based pauses ✓ ✓ Phoneme control ✓ ✓ Inline speed control - ✓ Streaming audio ✓ ✓ Arcana model support - ✓ PlayHT Sarvam AI On this page Overview Installation Choosing a Rime service RimeTTSService RimeHttpTTSService Input Parameters RimeTTSService (WebSocket) Constructor Parameters Features RimeHttpTTSService (HTTP) Constructor Parameters Output Frames Control Frames Audio Frames Error Frames Language Support Usage Examples WebSocket Service HTTP Service Frame Flow Metrics Support Service Comparison Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_rime_1110a900.txt b/tts_rime_1110a900.txt
new file mode 100644
index 0000000000000000000000000000000000000000..9bd43144ba61578c10560a658dde774dfa6f04ec
--- /dev/null
+++ b/tts_rime_1110a900.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/rime#param-model-1
+Title: Rime - Pipecat
+==================================================
+
+Rime - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech Rime Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview Rime AI provides two TTS service implementations: RimeTTSService : WebSocket-based implementation with word-level timing and interruption support RimeHttpTTSService : HTTP-based implementation for simpler use cases ​ Installation To use Rime services, install the required dependencies: Copy Ask AI pip install "pipecat-ai[rime]" You can obtain a Rime API key by signing up at Rime . ​ Choosing a Rime service Rime has two supported services: RimeTTSService which is a websocket-based implementation RimeHttpTTSService , which is an HTTP-based implementation ​ RimeTTSService The RimeTTSService is recommended for real-time interactive applications. It offers: Word-level timing information for precise synchronization Support for interruptions and context management Context tracking across multiple messages within a turn Non-blocking operation that allows other frames to be processed while audio is being generated ​ RimeHttpTTSService The RimeHttpTTSService is simpler and more suitable for non-interactive use cases. It: Processes the entire text in one request Supports advanced text control features (pauses, phonemes, inline speed) Blocks during the HTTP request, preventing other frames from being processed until the audio is fully generated ​ Input Parameters Both services use the same base input parameters structure: ​ language Language default: "Language.EN" The language to use for synthesis. See Language Support section for available options. ​ speed_alpha float default: "1.0" Speech rate multiplier. Values less than 1.0 increase speed, values greater than 1.0 decrease speed. ​ reduce_latency bool default: "false" Trade accuracy for lower latency ​ pause_between_brackets bool default: "false" When set to true , adds pauses between words enclosed in angle brackets. The number inside the brackets specifies the pause duration in milliseconds. Example: "Hi. <200> I'd love to have a conversation with you." adds a 200ms pause between the first and second sentences. ​ phonemize_between_brackets bool default: "false" When set to true, you can specify the phonemes for a word enclosed in curly brackets. Example: "{h'El.o} World" will pronounce "Hello" as expected. See Rime’s docs for more details. ​ RimeTTSService (WebSocket) Uses Rime’s WebSocket JSON API for real-time speech synthesis with word-level timing information. ​ Constructor Parameters ​ api_key str required Rime API key ​ voice_id str required Rime voice identifier ​ url str default: "wss://users.rime.ai/ws2" Rime WebSocket API endpoint ​ model str default: "mistv2" Model ID to use for synthesis ​ sample_rate int default: "None" Output audio sample rate in Hz ​ params InputParams default: "InputParams()" Speech generation parameters (see Input Parameters section above) ​ text_aggregator BaseTextAggregator default: "SkipTagsAggregator" Text aggregator for processing input text. Defaults to skipping content between spell( and ) tags. ​ Features Word-level timing information with cumulative timing across messages Support for interruptions with context clearing Context tracking across multiple messages within a turn Real-time audio streaming Proper sentence aggregation with skip tags support ​ RimeHttpTTSService (HTTP) HTTP-based implementation for simpler synthesis requirements. ​ Constructor Parameters ​ api_key str required Rime API key ​ voice_id str required Rime voice identifier. See Rime’s documentation for supported voices. ​ aiohttp_session aiohttp.ClientSession required HTTP session for making requests ​ model str default: "mistv2" Choose mistv2 for hyper-realistic conversational voices, mist for Rime’s previous generation model, or the latest arcana model. ​ sample_rate int default: "None" Output audio sample rate in Hz ​ params InputParams default: "InputParams()" Speech generation parameters Show additional properties ​ inline_speed_alpha str default: "None" Comma-separated list of speed values applied to words in square brackets. Values < 1.0 speed up speech, > 1.0 slow it down. Example: "This sentence is [really] [fast]" with inline_speed_alpha set to "0.5, 3" will make “really” slow and “fast” fast. ​ Output Frames Both services generate the following frames: ​ Control Frames ​ TTSStartedFrame Frame Signals start of speech synthesis ​ TTSStoppedFrame Frame Signals completion of speech synthesis ​ Audio Frames ​ TTSAudioRawFrame Frame Contains generated audio data: PCM audio format, specified sample rate, single channel (mono) ​ Error Frames ​ ErrorFrame Frame Contains Rime TTS error information ​ Language Support Supports multiple languages through the Language enum: Language Code Description Service Code Language.DE German ger Language.EN English eng Language.ES Spanish spa Language.FR French fra ​ Usage Examples ​ WebSocket Service Copy Ask AI from pipecat.services.rime.tts import RimeTTSService from pipecat.transcriptions.language import Language # Configure WebSocket service ws_tts = RimeTTSService( api_key = "your-rime-api-key" , voice_id = "cove" , model = "mistv2" , params = RimeTTSService.InputParams( language = Language. EN , speed_alpha = 1.0 , reduce_latency = False ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ HTTP Service Copy Ask AI import aiohttp from pipecat.services.rime.tts import RimeHttpTTSService # Configure HTTP service async with aiohttp.ClientSession() as session: http_tts = RimeHttpTTSService( api_key = "your-rime-api-key" , voice_id = "eva" , aiohttp_session = session, model = "mistv2" , params = RimeHttpTTSService.InputParams( speed_alpha = 1.2 , reduce_latency = True , pause_between_brackets = True , inline_speed_alpha = "0.8,1.5" ) ) ​ Frame Flow ​ Metrics Support Both services collect processing metrics: Time to First Byte (TTFB) Character usage statistics ​ Service Comparison Feature WebSocket HTTP Word timing ✓ - Interruption support ✓ - Context tracking ✓ - Bracket-based pauses ✓ ✓ Phoneme control ✓ ✓ Inline speed control - ✓ Streaming audio ✓ ✓ Arcana model support - ✓ PlayHT Sarvam AI On this page Overview Installation Choosing a Rime service RimeTTSService RimeHttpTTSService Input Parameters RimeTTSService (WebSocket) Constructor Parameters Features RimeHttpTTSService (HTTP) Constructor Parameters Output Frames Control Frames Audio Frames Error Frames Language Support Usage Examples WebSocket Service HTTP Service Frame Flow Metrics Support Service Comparison Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_rime_28874254.txt b/tts_rime_28874254.txt
new file mode 100644
index 0000000000000000000000000000000000000000..1fb7be61310bdcbd5222ed6743316ea2351e39c9
--- /dev/null
+++ b/tts_rime_28874254.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/rime#param-text-aggregator
+Title: Rime - Pipecat
+==================================================
+
+Rime - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech Rime Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview Rime AI provides two TTS service implementations: RimeTTSService : WebSocket-based implementation with word-level timing and interruption support RimeHttpTTSService : HTTP-based implementation for simpler use cases ​ Installation To use Rime services, install the required dependencies: Copy Ask AI pip install "pipecat-ai[rime]" You can obtain a Rime API key by signing up at Rime . ​ Choosing a Rime service Rime has two supported services: RimeTTSService which is a websocket-based implementation RimeHttpTTSService , which is an HTTP-based implementation ​ RimeTTSService The RimeTTSService is recommended for real-time interactive applications. It offers: Word-level timing information for precise synchronization Support for interruptions and context management Context tracking across multiple messages within a turn Non-blocking operation that allows other frames to be processed while audio is being generated ​ RimeHttpTTSService The RimeHttpTTSService is simpler and more suitable for non-interactive use cases. It: Processes the entire text in one request Supports advanced text control features (pauses, phonemes, inline speed) Blocks during the HTTP request, preventing other frames from being processed until the audio is fully generated ​ Input Parameters Both services use the same base input parameters structure: ​ language Language default: "Language.EN" The language to use for synthesis. See Language Support section for available options. ​ speed_alpha float default: "1.0" Speech rate multiplier. Values less than 1.0 increase speed, values greater than 1.0 decrease speed. ​ reduce_latency bool default: "false" Trade accuracy for lower latency ​ pause_between_brackets bool default: "false" When set to true , adds pauses between words enclosed in angle brackets. The number inside the brackets specifies the pause duration in milliseconds. Example: "Hi. <200> I'd love to have a conversation with you." adds a 200ms pause between the first and second sentences. ​ phonemize_between_brackets bool default: "false" When set to true, you can specify the phonemes for a word enclosed in curly brackets. Example: "{h'El.o} World" will pronounce "Hello" as expected. See Rime’s docs for more details. ​ RimeTTSService (WebSocket) Uses Rime’s WebSocket JSON API for real-time speech synthesis with word-level timing information. ​ Constructor Parameters ​ api_key str required Rime API key ​ voice_id str required Rime voice identifier ​ url str default: "wss://users.rime.ai/ws2" Rime WebSocket API endpoint ​ model str default: "mistv2" Model ID to use for synthesis ​ sample_rate int default: "None" Output audio sample rate in Hz ​ params InputParams default: "InputParams()" Speech generation parameters (see Input Parameters section above) ​ text_aggregator BaseTextAggregator default: "SkipTagsAggregator" Text aggregator for processing input text. Defaults to skipping content between spell( and ) tags. ​ Features Word-level timing information with cumulative timing across messages Support for interruptions with context clearing Context tracking across multiple messages within a turn Real-time audio streaming Proper sentence aggregation with skip tags support ​ RimeHttpTTSService (HTTP) HTTP-based implementation for simpler synthesis requirements. ​ Constructor Parameters ​ api_key str required Rime API key ​ voice_id str required Rime voice identifier. See Rime’s documentation for supported voices. ​ aiohttp_session aiohttp.ClientSession required HTTP session for making requests ​ model str default: "mistv2" Choose mistv2 for hyper-realistic conversational voices, mist for Rime’s previous generation model, or the latest arcana model. ​ sample_rate int default: "None" Output audio sample rate in Hz ​ params InputParams default: "InputParams()" Speech generation parameters Show additional properties ​ inline_speed_alpha str default: "None" Comma-separated list of speed values applied to words in square brackets. Values < 1.0 speed up speech, > 1.0 slow it down. Example: "This sentence is [really] [fast]" with inline_speed_alpha set to "0.5, 3" will make “really” slow and “fast” fast. ​ Output Frames Both services generate the following frames: ​ Control Frames ​ TTSStartedFrame Frame Signals start of speech synthesis ​ TTSStoppedFrame Frame Signals completion of speech synthesis ​ Audio Frames ​ TTSAudioRawFrame Frame Contains generated audio data: PCM audio format, specified sample rate, single channel (mono) ​ Error Frames ​ ErrorFrame Frame Contains Rime TTS error information ​ Language Support Supports multiple languages through the Language enum: Language Code Description Service Code Language.DE German ger Language.EN English eng Language.ES Spanish spa Language.FR French fra ​ Usage Examples ​ WebSocket Service Copy Ask AI from pipecat.services.rime.tts import RimeTTSService from pipecat.transcriptions.language import Language # Configure WebSocket service ws_tts = RimeTTSService( api_key = "your-rime-api-key" , voice_id = "cove" , model = "mistv2" , params = RimeTTSService.InputParams( language = Language. EN , speed_alpha = 1.0 , reduce_latency = False ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ HTTP Service Copy Ask AI import aiohttp from pipecat.services.rime.tts import RimeHttpTTSService # Configure HTTP service async with aiohttp.ClientSession() as session: http_tts = RimeHttpTTSService( api_key = "your-rime-api-key" , voice_id = "eva" , aiohttp_session = session, model = "mistv2" , params = RimeHttpTTSService.InputParams( speed_alpha = 1.2 , reduce_latency = True , pause_between_brackets = True , inline_speed_alpha = "0.8,1.5" ) ) ​ Frame Flow ​ Metrics Support Both services collect processing metrics: Time to First Byte (TTFB) Character usage statistics ​ Service Comparison Feature WebSocket HTTP Word timing ✓ - Interruption support ✓ - Context tracking ✓ - Bracket-based pauses ✓ ✓ Phoneme control ✓ ✓ Inline speed control - ✓ Streaming audio ✓ ✓ Arcana model support - ✓ PlayHT Sarvam AI On this page Overview Installation Choosing a Rime service RimeTTSService RimeHttpTTSService Input Parameters RimeTTSService (WebSocket) Constructor Parameters Features RimeHttpTTSService (HTTP) Constructor Parameters Output Frames Control Frames Audio Frames Error Frames Language Support Usage Examples WebSocket Service HTTP Service Frame Flow Metrics Support Service Comparison Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_rime_3c097411.txt b/tts_rime_3c097411.txt
new file mode 100644
index 0000000000000000000000000000000000000000..362c7bd2b9bee1285f8886aaac072286b72389e5
--- /dev/null
+++ b/tts_rime_3c097411.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/rime#service-comparison
+Title: Rime - Pipecat
+==================================================
+
+Rime - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech Rime Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview Rime AI provides two TTS service implementations: RimeTTSService : WebSocket-based implementation with word-level timing and interruption support RimeHttpTTSService : HTTP-based implementation for simpler use cases ​ Installation To use Rime services, install the required dependencies: Copy Ask AI pip install "pipecat-ai[rime]" You can obtain a Rime API key by signing up at Rime . ​ Choosing a Rime service Rime has two supported services: RimeTTSService which is a websocket-based implementation RimeHttpTTSService , which is an HTTP-based implementation ​ RimeTTSService The RimeTTSService is recommended for real-time interactive applications. It offers: Word-level timing information for precise synchronization Support for interruptions and context management Context tracking across multiple messages within a turn Non-blocking operation that allows other frames to be processed while audio is being generated ​ RimeHttpTTSService The RimeHttpTTSService is simpler and more suitable for non-interactive use cases. It: Processes the entire text in one request Supports advanced text control features (pauses, phonemes, inline speed) Blocks during the HTTP request, preventing other frames from being processed until the audio is fully generated ​ Input Parameters Both services use the same base input parameters structure: ​ language Language default: "Language.EN" The language to use for synthesis. See Language Support section for available options. ​ speed_alpha float default: "1.0" Speech rate multiplier. Values less than 1.0 increase speed, values greater than 1.0 decrease speed. ​ reduce_latency bool default: "false" Trade accuracy for lower latency ​ pause_between_brackets bool default: "false" When set to true , adds pauses between words enclosed in angle brackets. The number inside the brackets specifies the pause duration in milliseconds. Example: "Hi. <200> I'd love to have a conversation with you." adds a 200ms pause between the first and second sentences. ​ phonemize_between_brackets bool default: "false" When set to true, you can specify the phonemes for a word enclosed in curly brackets. Example: "{h'El.o} World" will pronounce "Hello" as expected. See Rime’s docs for more details. ​ RimeTTSService (WebSocket) Uses Rime’s WebSocket JSON API for real-time speech synthesis with word-level timing information. ​ Constructor Parameters ​ api_key str required Rime API key ​ voice_id str required Rime voice identifier ​ url str default: "wss://users.rime.ai/ws2" Rime WebSocket API endpoint ​ model str default: "mistv2" Model ID to use for synthesis ​ sample_rate int default: "None" Output audio sample rate in Hz ​ params InputParams default: "InputParams()" Speech generation parameters (see Input Parameters section above) ​ text_aggregator BaseTextAggregator default: "SkipTagsAggregator" Text aggregator for processing input text. Defaults to skipping content between spell( and ) tags. ​ Features Word-level timing information with cumulative timing across messages Support for interruptions with context clearing Context tracking across multiple messages within a turn Real-time audio streaming Proper sentence aggregation with skip tags support ​ RimeHttpTTSService (HTTP) HTTP-based implementation for simpler synthesis requirements. ​ Constructor Parameters ​ api_key str required Rime API key ​ voice_id str required Rime voice identifier. See Rime’s documentation for supported voices. ​ aiohttp_session aiohttp.ClientSession required HTTP session for making requests ​ model str default: "mistv2" Choose mistv2 for hyper-realistic conversational voices, mist for Rime’s previous generation model, or the latest arcana model. ​ sample_rate int default: "None" Output audio sample rate in Hz ​ params InputParams default: "InputParams()" Speech generation parameters Show additional properties ​ inline_speed_alpha str default: "None" Comma-separated list of speed values applied to words in square brackets. Values < 1.0 speed up speech, > 1.0 slow it down. Example: "This sentence is [really] [fast]" with inline_speed_alpha set to "0.5, 3" will make “really” slow and “fast” fast. ​ Output Frames Both services generate the following frames: ​ Control Frames ​ TTSStartedFrame Frame Signals start of speech synthesis ​ TTSStoppedFrame Frame Signals completion of speech synthesis ​ Audio Frames ​ TTSAudioRawFrame Frame Contains generated audio data: PCM audio format, specified sample rate, single channel (mono) ​ Error Frames ​ ErrorFrame Frame Contains Rime TTS error information ​ Language Support Supports multiple languages through the Language enum: Language Code Description Service Code Language.DE German ger Language.EN English eng Language.ES Spanish spa Language.FR French fra ​ Usage Examples ​ WebSocket Service Copy Ask AI from pipecat.services.rime.tts import RimeTTSService from pipecat.transcriptions.language import Language # Configure WebSocket service ws_tts = RimeTTSService( api_key = "your-rime-api-key" , voice_id = "cove" , model = "mistv2" , params = RimeTTSService.InputParams( language = Language. EN , speed_alpha = 1.0 , reduce_latency = False ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ HTTP Service Copy Ask AI import aiohttp from pipecat.services.rime.tts import RimeHttpTTSService # Configure HTTP service async with aiohttp.ClientSession() as session: http_tts = RimeHttpTTSService( api_key = "your-rime-api-key" , voice_id = "eva" , aiohttp_session = session, model = "mistv2" , params = RimeHttpTTSService.InputParams( speed_alpha = 1.2 , reduce_latency = True , pause_between_brackets = True , inline_speed_alpha = "0.8,1.5" ) ) ​ Frame Flow ​ Metrics Support Both services collect processing metrics: Time to First Byte (TTFB) Character usage statistics ​ Service Comparison Feature WebSocket HTTP Word timing ✓ - Interruption support ✓ - Context tracking ✓ - Bracket-based pauses ✓ ✓ Phoneme control ✓ ✓ Inline speed control - ✓ Streaming audio ✓ ✓ Arcana model support - ✓ PlayHT Sarvam AI On this page Overview Installation Choosing a Rime service RimeTTSService RimeHttpTTSService Input Parameters RimeTTSService (WebSocket) Constructor Parameters Features RimeHttpTTSService (HTTP) Constructor Parameters Output Frames Control Frames Audio Frames Error Frames Language Support Usage Examples WebSocket Service HTTP Service Frame Flow Metrics Support Service Comparison Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_rime_3cdb96b6.txt b/tts_rime_3cdb96b6.txt
new file mode 100644
index 0000000000000000000000000000000000000000..5fe32b161f88594ae4ca0252a642f08e04dae2be
--- /dev/null
+++ b/tts_rime_3cdb96b6.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/rime#param-inline-speed-alpha
+Title: Rime - Pipecat
+==================================================
+
+Rime - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech Rime Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview Rime AI provides two TTS service implementations: RimeTTSService : WebSocket-based implementation with word-level timing and interruption support RimeHttpTTSService : HTTP-based implementation for simpler use cases ​ Installation To use Rime services, install the required dependencies: Copy Ask AI pip install "pipecat-ai[rime]" You can obtain a Rime API key by signing up at Rime . ​ Choosing a Rime service Rime has two supported services: RimeTTSService which is a websocket-based implementation RimeHttpTTSService , which is an HTTP-based implementation ​ RimeTTSService The RimeTTSService is recommended for real-time interactive applications. It offers: Word-level timing information for precise synchronization Support for interruptions and context management Context tracking across multiple messages within a turn Non-blocking operation that allows other frames to be processed while audio is being generated ​ RimeHttpTTSService The RimeHttpTTSService is simpler and more suitable for non-interactive use cases. It: Processes the entire text in one request Supports advanced text control features (pauses, phonemes, inline speed) Blocks during the HTTP request, preventing other frames from being processed until the audio is fully generated ​ Input Parameters Both services use the same base input parameters structure: ​ language Language default: "Language.EN" The language to use for synthesis. See Language Support section for available options. ​ speed_alpha float default: "1.0" Speech rate multiplier. Values less than 1.0 increase speed, values greater than 1.0 decrease speed. ​ reduce_latency bool default: "false" Trade accuracy for lower latency ​ pause_between_brackets bool default: "false" When set to true , adds pauses between words enclosed in angle brackets. The number inside the brackets specifies the pause duration in milliseconds. Example: "Hi. <200> I'd love to have a conversation with you." adds a 200ms pause between the first and second sentences. ​ phonemize_between_brackets bool default: "false" When set to true, you can specify the phonemes for a word enclosed in curly brackets. Example: "{h'El.o} World" will pronounce "Hello" as expected. See Rime’s docs for more details. ​ RimeTTSService (WebSocket) Uses Rime’s WebSocket JSON API for real-time speech synthesis with word-level timing information. ​ Constructor Parameters ​ api_key str required Rime API key ​ voice_id str required Rime voice identifier ​ url str default: "wss://users.rime.ai/ws2" Rime WebSocket API endpoint ​ model str default: "mistv2" Model ID to use for synthesis ​ sample_rate int default: "None" Output audio sample rate in Hz ​ params InputParams default: "InputParams()" Speech generation parameters (see Input Parameters section above) ​ text_aggregator BaseTextAggregator default: "SkipTagsAggregator" Text aggregator for processing input text. Defaults to skipping content between spell( and ) tags. ​ Features Word-level timing information with cumulative timing across messages Support for interruptions with context clearing Context tracking across multiple messages within a turn Real-time audio streaming Proper sentence aggregation with skip tags support ​ RimeHttpTTSService (HTTP) HTTP-based implementation for simpler synthesis requirements. ​ Constructor Parameters ​ api_key str required Rime API key ​ voice_id str required Rime voice identifier. See Rime’s documentation for supported voices. ​ aiohttp_session aiohttp.ClientSession required HTTP session for making requests ​ model str default: "mistv2" Choose mistv2 for hyper-realistic conversational voices, mist for Rime’s previous generation model, or the latest arcana model. ​ sample_rate int default: "None" Output audio sample rate in Hz ​ params InputParams default: "InputParams()" Speech generation parameters Show additional properties ​ inline_speed_alpha str default: "None" Comma-separated list of speed values applied to words in square brackets. Values < 1.0 speed up speech, > 1.0 slow it down. Example: "This sentence is [really] [fast]" with inline_speed_alpha set to "0.5, 3" will make “really” slow and “fast” fast. ​ Output Frames Both services generate the following frames: ​ Control Frames ​ TTSStartedFrame Frame Signals start of speech synthesis ​ TTSStoppedFrame Frame Signals completion of speech synthesis ​ Audio Frames ​ TTSAudioRawFrame Frame Contains generated audio data: PCM audio format, specified sample rate, single channel (mono) ​ Error Frames ​ ErrorFrame Frame Contains Rime TTS error information ​ Language Support Supports multiple languages through the Language enum: Language Code Description Service Code Language.DE German ger Language.EN English eng Language.ES Spanish spa Language.FR French fra ​ Usage Examples ​ WebSocket Service Copy Ask AI from pipecat.services.rime.tts import RimeTTSService from pipecat.transcriptions.language import Language # Configure WebSocket service ws_tts = RimeTTSService( api_key = "your-rime-api-key" , voice_id = "cove" , model = "mistv2" , params = RimeTTSService.InputParams( language = Language. EN , speed_alpha = 1.0 , reduce_latency = False ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ HTTP Service Copy Ask AI import aiohttp from pipecat.services.rime.tts import RimeHttpTTSService # Configure HTTP service async with aiohttp.ClientSession() as session: http_tts = RimeHttpTTSService( api_key = "your-rime-api-key" , voice_id = "eva" , aiohttp_session = session, model = "mistv2" , params = RimeHttpTTSService.InputParams( speed_alpha = 1.2 , reduce_latency = True , pause_between_brackets = True , inline_speed_alpha = "0.8,1.5" ) ) ​ Frame Flow ​ Metrics Support Both services collect processing metrics: Time to First Byte (TTFB) Character usage statistics ​ Service Comparison Feature WebSocket HTTP Word timing ✓ - Interruption support ✓ - Context tracking ✓ - Bracket-based pauses ✓ ✓ Phoneme control ✓ ✓ Inline speed control - ✓ Streaming audio ✓ ✓ Arcana model support - ✓ PlayHT Sarvam AI On this page Overview Installation Choosing a Rime service RimeTTSService RimeHttpTTSService Input Parameters RimeTTSService (WebSocket) Constructor Parameters Features RimeHttpTTSService (HTTP) Constructor Parameters Output Frames Control Frames Audio Frames Error Frames Language Support Usage Examples WebSocket Service HTTP Service Frame Flow Metrics Support Service Comparison Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_rime_51f9ef2e.txt b/tts_rime_51f9ef2e.txt
new file mode 100644
index 0000000000000000000000000000000000000000..8a7fe704badda6d57b1f777ab0fdbf9cb0afd12a
--- /dev/null
+++ b/tts_rime_51f9ef2e.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/rime#http-service
+Title: Rime - Pipecat
+==================================================
+
+Rime - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech Rime Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview Rime AI provides two TTS service implementations: RimeTTSService : WebSocket-based implementation with word-level timing and interruption support RimeHttpTTSService : HTTP-based implementation for simpler use cases ​ Installation To use Rime services, install the required dependencies: Copy Ask AI pip install "pipecat-ai[rime]" You can obtain a Rime API key by signing up at Rime . ​ Choosing a Rime service Rime has two supported services: RimeTTSService which is a websocket-based implementation RimeHttpTTSService , which is an HTTP-based implementation ​ RimeTTSService The RimeTTSService is recommended for real-time interactive applications. It offers: Word-level timing information for precise synchronization Support for interruptions and context management Context tracking across multiple messages within a turn Non-blocking operation that allows other frames to be processed while audio is being generated ​ RimeHttpTTSService The RimeHttpTTSService is simpler and more suitable for non-interactive use cases. It: Processes the entire text in one request Supports advanced text control features (pauses, phonemes, inline speed) Blocks during the HTTP request, preventing other frames from being processed until the audio is fully generated ​ Input Parameters Both services use the same base input parameters structure: ​ language Language default: "Language.EN" The language to use for synthesis. See Language Support section for available options. ​ speed_alpha float default: "1.0" Speech rate multiplier. Values less than 1.0 increase speed, values greater than 1.0 decrease speed. ​ reduce_latency bool default: "false" Trade accuracy for lower latency ​ pause_between_brackets bool default: "false" When set to true , adds pauses between words enclosed in angle brackets. The number inside the brackets specifies the pause duration in milliseconds. Example: "Hi. <200> I'd love to have a conversation with you." adds a 200ms pause between the first and second sentences. ​ phonemize_between_brackets bool default: "false" When set to true, you can specify the phonemes for a word enclosed in curly brackets. Example: "{h'El.o} World" will pronounce "Hello" as expected. See Rime’s docs for more details. ​ RimeTTSService (WebSocket) Uses Rime’s WebSocket JSON API for real-time speech synthesis with word-level timing information. ​ Constructor Parameters ​ api_key str required Rime API key ​ voice_id str required Rime voice identifier ​ url str default: "wss://users.rime.ai/ws2" Rime WebSocket API endpoint ​ model str default: "mistv2" Model ID to use for synthesis ​ sample_rate int default: "None" Output audio sample rate in Hz ​ params InputParams default: "InputParams()" Speech generation parameters (see Input Parameters section above) ​ text_aggregator BaseTextAggregator default: "SkipTagsAggregator" Text aggregator for processing input text. Defaults to skipping content between spell( and ) tags. ​ Features Word-level timing information with cumulative timing across messages Support for interruptions with context clearing Context tracking across multiple messages within a turn Real-time audio streaming Proper sentence aggregation with skip tags support ​ RimeHttpTTSService (HTTP) HTTP-based implementation for simpler synthesis requirements. ​ Constructor Parameters ​ api_key str required Rime API key ​ voice_id str required Rime voice identifier. See Rime’s documentation for supported voices. ​ aiohttp_session aiohttp.ClientSession required HTTP session for making requests ​ model str default: "mistv2" Choose mistv2 for hyper-realistic conversational voices, mist for Rime’s previous generation model, or the latest arcana model. ​ sample_rate int default: "None" Output audio sample rate in Hz ​ params InputParams default: "InputParams()" Speech generation parameters Show additional properties ​ inline_speed_alpha str default: "None" Comma-separated list of speed values applied to words in square brackets. Values < 1.0 speed up speech, > 1.0 slow it down. Example: "This sentence is [really] [fast]" with inline_speed_alpha set to "0.5, 3" will make “really” slow and “fast” fast. ​ Output Frames Both services generate the following frames: ​ Control Frames ​ TTSStartedFrame Frame Signals start of speech synthesis ​ TTSStoppedFrame Frame Signals completion of speech synthesis ​ Audio Frames ​ TTSAudioRawFrame Frame Contains generated audio data: PCM audio format, specified sample rate, single channel (mono) ​ Error Frames ​ ErrorFrame Frame Contains Rime TTS error information ​ Language Support Supports multiple languages through the Language enum: Language Code Description Service Code Language.DE German ger Language.EN English eng Language.ES Spanish spa Language.FR French fra ​ Usage Examples ​ WebSocket Service Copy Ask AI from pipecat.services.rime.tts import RimeTTSService from pipecat.transcriptions.language import Language # Configure WebSocket service ws_tts = RimeTTSService( api_key = "your-rime-api-key" , voice_id = "cove" , model = "mistv2" , params = RimeTTSService.InputParams( language = Language. EN , speed_alpha = 1.0 , reduce_latency = False ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ HTTP Service Copy Ask AI import aiohttp from pipecat.services.rime.tts import RimeHttpTTSService # Configure HTTP service async with aiohttp.ClientSession() as session: http_tts = RimeHttpTTSService( api_key = "your-rime-api-key" , voice_id = "eva" , aiohttp_session = session, model = "mistv2" , params = RimeHttpTTSService.InputParams( speed_alpha = 1.2 , reduce_latency = True , pause_between_brackets = True , inline_speed_alpha = "0.8,1.5" ) ) ​ Frame Flow ​ Metrics Support Both services collect processing metrics: Time to First Byte (TTFB) Character usage statistics ​ Service Comparison Feature WebSocket HTTP Word timing ✓ - Interruption support ✓ - Context tracking ✓ - Bracket-based pauses ✓ ✓ Phoneme control ✓ ✓ Inline speed control - ✓ Streaming audio ✓ ✓ Arcana model support - ✓ PlayHT Sarvam AI On this page Overview Installation Choosing a Rime service RimeTTSService RimeHttpTTSService Input Parameters RimeTTSService (WebSocket) Constructor Parameters Features RimeHttpTTSService (HTTP) Constructor Parameters Output Frames Control Frames Audio Frames Error Frames Language Support Usage Examples WebSocket Service HTTP Service Frame Flow Metrics Support Service Comparison Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_rime_54dc60ee.txt b/tts_rime_54dc60ee.txt
new file mode 100644
index 0000000000000000000000000000000000000000..87e68ab412c6182663fe8f9e97167aa86441bc92
--- /dev/null
+++ b/tts_rime_54dc60ee.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/rime#output-frames
+Title: Rime - Pipecat
+==================================================
+
+Rime - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech Rime Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview Rime AI provides two TTS service implementations: RimeTTSService : WebSocket-based implementation with word-level timing and interruption support RimeHttpTTSService : HTTP-based implementation for simpler use cases ​ Installation To use Rime services, install the required dependencies: Copy Ask AI pip install "pipecat-ai[rime]" You can obtain a Rime API key by signing up at Rime . ​ Choosing a Rime service Rime has two supported services: RimeTTSService which is a websocket-based implementation RimeHttpTTSService , which is an HTTP-based implementation ​ RimeTTSService The RimeTTSService is recommended for real-time interactive applications. It offers: Word-level timing information for precise synchronization Support for interruptions and context management Context tracking across multiple messages within a turn Non-blocking operation that allows other frames to be processed while audio is being generated ​ RimeHttpTTSService The RimeHttpTTSService is simpler and more suitable for non-interactive use cases. It: Processes the entire text in one request Supports advanced text control features (pauses, phonemes, inline speed) Blocks during the HTTP request, preventing other frames from being processed until the audio is fully generated ​ Input Parameters Both services use the same base input parameters structure: ​ language Language default: "Language.EN" The language to use for synthesis. See Language Support section for available options. ​ speed_alpha float default: "1.0" Speech rate multiplier. Values less than 1.0 increase speed, values greater than 1.0 decrease speed. ​ reduce_latency bool default: "false" Trade accuracy for lower latency ​ pause_between_brackets bool default: "false" When set to true , adds pauses between words enclosed in angle brackets. The number inside the brackets specifies the pause duration in milliseconds. Example: "Hi. <200> I'd love to have a conversation with you." adds a 200ms pause between the first and second sentences. ​ phonemize_between_brackets bool default: "false" When set to true, you can specify the phonemes for a word enclosed in curly brackets. Example: "{h'El.o} World" will pronounce "Hello" as expected. See Rime’s docs for more details. ​ RimeTTSService (WebSocket) Uses Rime’s WebSocket JSON API for real-time speech synthesis with word-level timing information. ​ Constructor Parameters ​ api_key str required Rime API key ​ voice_id str required Rime voice identifier ​ url str default: "wss://users.rime.ai/ws2" Rime WebSocket API endpoint ​ model str default: "mistv2" Model ID to use for synthesis ​ sample_rate int default: "None" Output audio sample rate in Hz ​ params InputParams default: "InputParams()" Speech generation parameters (see Input Parameters section above) ​ text_aggregator BaseTextAggregator default: "SkipTagsAggregator" Text aggregator for processing input text. Defaults to skipping content between spell( and ) tags. ​ Features Word-level timing information with cumulative timing across messages Support for interruptions with context clearing Context tracking across multiple messages within a turn Real-time audio streaming Proper sentence aggregation with skip tags support ​ RimeHttpTTSService (HTTP) HTTP-based implementation for simpler synthesis requirements. ​ Constructor Parameters ​ api_key str required Rime API key ​ voice_id str required Rime voice identifier. See Rime’s documentation for supported voices. ​ aiohttp_session aiohttp.ClientSession required HTTP session for making requests ​ model str default: "mistv2" Choose mistv2 for hyper-realistic conversational voices, mist for Rime’s previous generation model, or the latest arcana model. ​ sample_rate int default: "None" Output audio sample rate in Hz ​ params InputParams default: "InputParams()" Speech generation parameters Show additional properties ​ inline_speed_alpha str default: "None" Comma-separated list of speed values applied to words in square brackets. Values < 1.0 speed up speech, > 1.0 slow it down. Example: "This sentence is [really] [fast]" with inline_speed_alpha set to "0.5, 3" will make “really” slow and “fast” fast. ​ Output Frames Both services generate the following frames: ​ Control Frames ​ TTSStartedFrame Frame Signals start of speech synthesis ​ TTSStoppedFrame Frame Signals completion of speech synthesis ​ Audio Frames ​ TTSAudioRawFrame Frame Contains generated audio data: PCM audio format, specified sample rate, single channel (mono) ​ Error Frames ​ ErrorFrame Frame Contains Rime TTS error information ​ Language Support Supports multiple languages through the Language enum: Language Code Description Service Code Language.DE German ger Language.EN English eng Language.ES Spanish spa Language.FR French fra ​ Usage Examples ​ WebSocket Service Copy Ask AI from pipecat.services.rime.tts import RimeTTSService from pipecat.transcriptions.language import Language # Configure WebSocket service ws_tts = RimeTTSService( api_key = "your-rime-api-key" , voice_id = "cove" , model = "mistv2" , params = RimeTTSService.InputParams( language = Language. EN , speed_alpha = 1.0 , reduce_latency = False ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ HTTP Service Copy Ask AI import aiohttp from pipecat.services.rime.tts import RimeHttpTTSService # Configure HTTP service async with aiohttp.ClientSession() as session: http_tts = RimeHttpTTSService( api_key = "your-rime-api-key" , voice_id = "eva" , aiohttp_session = session, model = "mistv2" , params = RimeHttpTTSService.InputParams( speed_alpha = 1.2 , reduce_latency = True , pause_between_brackets = True , inline_speed_alpha = "0.8,1.5" ) ) ​ Frame Flow ​ Metrics Support Both services collect processing metrics: Time to First Byte (TTFB) Character usage statistics ​ Service Comparison Feature WebSocket HTTP Word timing ✓ - Interruption support ✓ - Context tracking ✓ - Bracket-based pauses ✓ ✓ Phoneme control ✓ ✓ Inline speed control - ✓ Streaming audio ✓ ✓ Arcana model support - ✓ PlayHT Sarvam AI On this page Overview Installation Choosing a Rime service RimeTTSService RimeHttpTTSService Input Parameters RimeTTSService (WebSocket) Constructor Parameters Features RimeHttpTTSService (HTTP) Constructor Parameters Output Frames Control Frames Audio Frames Error Frames Language Support Usage Examples WebSocket Service HTTP Service Frame Flow Metrics Support Service Comparison Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_rime_775ca231.txt b/tts_rime_775ca231.txt
new file mode 100644
index 0000000000000000000000000000000000000000..f94f77c3168778f57448bbcd37b277bc6597d8e7
--- /dev/null
+++ b/tts_rime_775ca231.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/rime#param-voice-id
+Title: Rime - Pipecat
+==================================================
+
+Rime - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech Rime Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview Rime AI provides two TTS service implementations: RimeTTSService : WebSocket-based implementation with word-level timing and interruption support RimeHttpTTSService : HTTP-based implementation for simpler use cases ​ Installation To use Rime services, install the required dependencies: Copy Ask AI pip install "pipecat-ai[rime]" You can obtain a Rime API key by signing up at Rime . ​ Choosing a Rime service Rime has two supported services: RimeTTSService which is a websocket-based implementation RimeHttpTTSService , which is an HTTP-based implementation ​ RimeTTSService The RimeTTSService is recommended for real-time interactive applications. It offers: Word-level timing information for precise synchronization Support for interruptions and context management Context tracking across multiple messages within a turn Non-blocking operation that allows other frames to be processed while audio is being generated ​ RimeHttpTTSService The RimeHttpTTSService is simpler and more suitable for non-interactive use cases. It: Processes the entire text in one request Supports advanced text control features (pauses, phonemes, inline speed) Blocks during the HTTP request, preventing other frames from being processed until the audio is fully generated ​ Input Parameters Both services use the same base input parameters structure: ​ language Language default: "Language.EN" The language to use for synthesis. See Language Support section for available options. ​ speed_alpha float default: "1.0" Speech rate multiplier. Values less than 1.0 increase speed, values greater than 1.0 decrease speed. ​ reduce_latency bool default: "false" Trade accuracy for lower latency ​ pause_between_brackets bool default: "false" When set to true , adds pauses between words enclosed in angle brackets. The number inside the brackets specifies the pause duration in milliseconds. Example: "Hi. <200> I'd love to have a conversation with you." adds a 200ms pause between the first and second sentences. ​ phonemize_between_brackets bool default: "false" When set to true, you can specify the phonemes for a word enclosed in curly brackets. Example: "{h'El.o} World" will pronounce "Hello" as expected. See Rime’s docs for more details. ​ RimeTTSService (WebSocket) Uses Rime’s WebSocket JSON API for real-time speech synthesis with word-level timing information. ​ Constructor Parameters ​ api_key str required Rime API key ​ voice_id str required Rime voice identifier ​ url str default: "wss://users.rime.ai/ws2" Rime WebSocket API endpoint ​ model str default: "mistv2" Model ID to use for synthesis ​ sample_rate int default: "None" Output audio sample rate in Hz ​ params InputParams default: "InputParams()" Speech generation parameters (see Input Parameters section above) ​ text_aggregator BaseTextAggregator default: "SkipTagsAggregator" Text aggregator for processing input text. Defaults to skipping content between spell( and ) tags. ​ Features Word-level timing information with cumulative timing across messages Support for interruptions with context clearing Context tracking across multiple messages within a turn Real-time audio streaming Proper sentence aggregation with skip tags support ​ RimeHttpTTSService (HTTP) HTTP-based implementation for simpler synthesis requirements. ​ Constructor Parameters ​ api_key str required Rime API key ​ voice_id str required Rime voice identifier. See Rime’s documentation for supported voices. ​ aiohttp_session aiohttp.ClientSession required HTTP session for making requests ​ model str default: "mistv2" Choose mistv2 for hyper-realistic conversational voices, mist for Rime’s previous generation model, or the latest arcana model. ​ sample_rate int default: "None" Output audio sample rate in Hz ​ params InputParams default: "InputParams()" Speech generation parameters Show additional properties ​ inline_speed_alpha str default: "None" Comma-separated list of speed values applied to words in square brackets. Values < 1.0 speed up speech, > 1.0 slow it down. Example: "This sentence is [really] [fast]" with inline_speed_alpha set to "0.5, 3" will make “really” slow and “fast” fast. ​ Output Frames Both services generate the following frames: ​ Control Frames ​ TTSStartedFrame Frame Signals start of speech synthesis ​ TTSStoppedFrame Frame Signals completion of speech synthesis ​ Audio Frames ​ TTSAudioRawFrame Frame Contains generated audio data: PCM audio format, specified sample rate, single channel (mono) ​ Error Frames ​ ErrorFrame Frame Contains Rime TTS error information ​ Language Support Supports multiple languages through the Language enum: Language Code Description Service Code Language.DE German ger Language.EN English eng Language.ES Spanish spa Language.FR French fra ​ Usage Examples ​ WebSocket Service Copy Ask AI from pipecat.services.rime.tts import RimeTTSService from pipecat.transcriptions.language import Language # Configure WebSocket service ws_tts = RimeTTSService( api_key = "your-rime-api-key" , voice_id = "cove" , model = "mistv2" , params = RimeTTSService.InputParams( language = Language. EN , speed_alpha = 1.0 , reduce_latency = False ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ HTTP Service Copy Ask AI import aiohttp from pipecat.services.rime.tts import RimeHttpTTSService # Configure HTTP service async with aiohttp.ClientSession() as session: http_tts = RimeHttpTTSService( api_key = "your-rime-api-key" , voice_id = "eva" , aiohttp_session = session, model = "mistv2" , params = RimeHttpTTSService.InputParams( speed_alpha = 1.2 , reduce_latency = True , pause_between_brackets = True , inline_speed_alpha = "0.8,1.5" ) ) ​ Frame Flow ​ Metrics Support Both services collect processing metrics: Time to First Byte (TTFB) Character usage statistics ​ Service Comparison Feature WebSocket HTTP Word timing ✓ - Interruption support ✓ - Context tracking ✓ - Bracket-based pauses ✓ ✓ Phoneme control ✓ ✓ Inline speed control - ✓ Streaming audio ✓ ✓ Arcana model support - ✓ PlayHT Sarvam AI On this page Overview Installation Choosing a Rime service RimeTTSService RimeHttpTTSService Input Parameters RimeTTSService (WebSocket) Constructor Parameters Features RimeHttpTTSService (HTTP) Constructor Parameters Output Frames Control Frames Audio Frames Error Frames Language Support Usage Examples WebSocket Service HTTP Service Frame Flow Metrics Support Service Comparison Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_rime_8269a7d2.txt b/tts_rime_8269a7d2.txt
new file mode 100644
index 0000000000000000000000000000000000000000..9ff9e9e1e2c203e28470fe2db7d9f5dc2b2e450e
--- /dev/null
+++ b/tts_rime_8269a7d2.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/_sites/docs.pipecat.ai/server/services/tts/rime#next-steps
+Title: Overview - Pipecat
+==================================================
+
+Overview - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Get Started Overview Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Get Started Overview Installation & Setup Quickstart Core Concepts Next Steps & Examples Pipecat is an open source Python framework that handles the complex orchestration of AI services, network transport, audio processing, and multimodal interactions. “Multimodal” means you can use any combination of audio, video, images, and/or text in your interactions. And “real-time” means that things are happening quickly enough that it feels conversational—a “back-and-forth” with a bot, not submitting a query and waiting for results. ​ What You Can Build Voice Assistants Natural, real-time conversations with AI using speech recognition and synthesis Interactive Agents Personal coaches and meeting assistants that can understand context and provide guidance Multimodal Apps Applications that combine voice, video, images, and text for rich interactions Creative Tools Storytelling experiences and social companions that engage users Business Solutions Customer intake flows and support bots for automated business processes Complex Flows Structured conversations using Pipecat Flows for managing complex interactions ​ How It Works The flow of interactions in a Pipecat application is typically straightforward: The bot says something The user says something The bot says something The user says something This continues until the conversation naturally ends. While this flow seems simple, making it feel natural requires sophisticated real-time processing. ​ Real-time Processing Pipecat’s pipeline architecture handles both simple voice interactions and complex multimodal processing. Let’s look at how data flows through the system: Voice app Multimodal app 1 Send Audio Transmit and capture streamed audio from the user 2 Transcribe Speech Convert speech to text as the user is talking 3 Process with LLM Generate responses using a large language model 4 Convert to Speech Transform text responses into natural speech 5 Play Audio Stream the audio response back to the user 1 Send Audio Transmit and capture streamed audio from the user 2 Transcribe Speech Convert speech to text as the user is talking 3 Process with LLM Generate responses using a large language model 4 Convert to Speech Transform text responses into natural speech 5 Play Audio Stream the audio response back to the user 1 Send Audio and Video Transmit and capture audio, video, and image inputs simultaneously 2 Process Streams Handle multiple input streams in parallel 3 Model Processing Send combined inputs to multimodal models (like GPT-4V) 4 Generate Outputs Create various outputs (text, images, audio, etc.) 5 Coordinate Presentation Synchronize and present multiple output types In both cases, Pipecat: Processes responses as they stream in Handles multiple input/output modalities concurrently Manages resource allocation and synchronization Coordinates parallel processing tasks This architecture creates fluid, natural interactions without noticeable delays, whether you’re building a simple voice assistant or a complex multimodal application. Pipecat’s pipeline architecture is particularly valuable for managing the complexity of real-time, multimodal interactions, ensuring smooth data flow and proper synchronization regardless of the input/output types involved. Pipecat handles all this complexity for you, letting you focus on building your application rather than managing the underlying infrastructure. ​ Next Steps Ready to build your first Pipecat application? Installation & Setup Prepare your environment and install required dependencies Quickstart Build and run your first Pipecat application Core Concepts Learn about pipelines, frames, and real-time processing Use Cases Explore example implementations and patterns ​ Join Our Community Discord Community Connect with other developers, share your projects, and get support from the Pipecat team. Installation & Setup On this page What You Can Build How It Works Real-time Processing Next Steps Join Our Community Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_rime_87ee0397.txt b/tts_rime_87ee0397.txt
new file mode 100644
index 0000000000000000000000000000000000000000..b72a728ef8ba6eefd0abd49893ca12f0693a5b46
--- /dev/null
+++ b/tts_rime_87ee0397.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/_sites/docs.pipecat.ai/server/services/tts/rime#how-it-works
+Title: Overview - Pipecat
+==================================================
+
+Overview - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Get Started Overview Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Get Started Overview Installation & Setup Quickstart Core Concepts Next Steps & Examples Pipecat is an open source Python framework that handles the complex orchestration of AI services, network transport, audio processing, and multimodal interactions. “Multimodal” means you can use any combination of audio, video, images, and/or text in your interactions. And “real-time” means that things are happening quickly enough that it feels conversational—a “back-and-forth” with a bot, not submitting a query and waiting for results. ​ What You Can Build Voice Assistants Natural, real-time conversations with AI using speech recognition and synthesis Interactive Agents Personal coaches and meeting assistants that can understand context and provide guidance Multimodal Apps Applications that combine voice, video, images, and text for rich interactions Creative Tools Storytelling experiences and social companions that engage users Business Solutions Customer intake flows and support bots for automated business processes Complex Flows Structured conversations using Pipecat Flows for managing complex interactions ​ How It Works The flow of interactions in a Pipecat application is typically straightforward: The bot says something The user says something The bot says something The user says something This continues until the conversation naturally ends. While this flow seems simple, making it feel natural requires sophisticated real-time processing. ​ Real-time Processing Pipecat’s pipeline architecture handles both simple voice interactions and complex multimodal processing. Let’s look at how data flows through the system: Voice app Multimodal app 1 Send Audio Transmit and capture streamed audio from the user 2 Transcribe Speech Convert speech to text as the user is talking 3 Process with LLM Generate responses using a large language model 4 Convert to Speech Transform text responses into natural speech 5 Play Audio Stream the audio response back to the user 1 Send Audio Transmit and capture streamed audio from the user 2 Transcribe Speech Convert speech to text as the user is talking 3 Process with LLM Generate responses using a large language model 4 Convert to Speech Transform text responses into natural speech 5 Play Audio Stream the audio response back to the user 1 Send Audio and Video Transmit and capture audio, video, and image inputs simultaneously 2 Process Streams Handle multiple input streams in parallel 3 Model Processing Send combined inputs to multimodal models (like GPT-4V) 4 Generate Outputs Create various outputs (text, images, audio, etc.) 5 Coordinate Presentation Synchronize and present multiple output types In both cases, Pipecat: Processes responses as they stream in Handles multiple input/output modalities concurrently Manages resource allocation and synchronization Coordinates parallel processing tasks This architecture creates fluid, natural interactions without noticeable delays, whether you’re building a simple voice assistant or a complex multimodal application. Pipecat’s pipeline architecture is particularly valuable for managing the complexity of real-time, multimodal interactions, ensuring smooth data flow and proper synchronization regardless of the input/output types involved. Pipecat handles all this complexity for you, letting you focus on building your application rather than managing the underlying infrastructure. ​ Next Steps Ready to build your first Pipecat application? Installation & Setup Prepare your environment and install required dependencies Quickstart Build and run your first Pipecat application Core Concepts Learn about pipelines, frames, and real-time processing Use Cases Explore example implementations and patterns ​ Join Our Community Discord Community Connect with other developers, share your projects, and get support from the Pipecat team. Installation & Setup On this page What You Can Build How It Works Real-time Processing Next Steps Join Our Community Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_rime_88776530.txt b/tts_rime_88776530.txt
new file mode 100644
index 0000000000000000000000000000000000000000..036256eaf50541aa053b26186c741952711c99a7
--- /dev/null
+++ b/tts_rime_88776530.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/_sites/docs.pipecat.ai/server/services/tts/rime#join-our-community
+Title: Overview - Pipecat
+==================================================
+
+Overview - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Get Started Overview Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Get Started Overview Installation & Setup Quickstart Core Concepts Next Steps & Examples Pipecat is an open source Python framework that handles the complex orchestration of AI services, network transport, audio processing, and multimodal interactions. “Multimodal” means you can use any combination of audio, video, images, and/or text in your interactions. And “real-time” means that things are happening quickly enough that it feels conversational—a “back-and-forth” with a bot, not submitting a query and waiting for results. ​ What You Can Build Voice Assistants Natural, real-time conversations with AI using speech recognition and synthesis Interactive Agents Personal coaches and meeting assistants that can understand context and provide guidance Multimodal Apps Applications that combine voice, video, images, and text for rich interactions Creative Tools Storytelling experiences and social companions that engage users Business Solutions Customer intake flows and support bots for automated business processes Complex Flows Structured conversations using Pipecat Flows for managing complex interactions ​ How It Works The flow of interactions in a Pipecat application is typically straightforward: The bot says something The user says something The bot says something The user says something This continues until the conversation naturally ends. While this flow seems simple, making it feel natural requires sophisticated real-time processing. ​ Real-time Processing Pipecat’s pipeline architecture handles both simple voice interactions and complex multimodal processing. Let’s look at how data flows through the system: Voice app Multimodal app 1 Send Audio Transmit and capture streamed audio from the user 2 Transcribe Speech Convert speech to text as the user is talking 3 Process with LLM Generate responses using a large language model 4 Convert to Speech Transform text responses into natural speech 5 Play Audio Stream the audio response back to the user 1 Send Audio Transmit and capture streamed audio from the user 2 Transcribe Speech Convert speech to text as the user is talking 3 Process with LLM Generate responses using a large language model 4 Convert to Speech Transform text responses into natural speech 5 Play Audio Stream the audio response back to the user 1 Send Audio and Video Transmit and capture audio, video, and image inputs simultaneously 2 Process Streams Handle multiple input streams in parallel 3 Model Processing Send combined inputs to multimodal models (like GPT-4V) 4 Generate Outputs Create various outputs (text, images, audio, etc.) 5 Coordinate Presentation Synchronize and present multiple output types In both cases, Pipecat: Processes responses as they stream in Handles multiple input/output modalities concurrently Manages resource allocation and synchronization Coordinates parallel processing tasks This architecture creates fluid, natural interactions without noticeable delays, whether you’re building a simple voice assistant or a complex multimodal application. Pipecat’s pipeline architecture is particularly valuable for managing the complexity of real-time, multimodal interactions, ensuring smooth data flow and proper synchronization regardless of the input/output types involved. Pipecat handles all this complexity for you, letting you focus on building your application rather than managing the underlying infrastructure. ​ Next Steps Ready to build your first Pipecat application? Installation & Setup Prepare your environment and install required dependencies Quickstart Build and run your first Pipecat application Core Concepts Learn about pipelines, frames, and real-time processing Use Cases Explore example implementations and patterns ​ Join Our Community Discord Community Connect with other developers, share your projects, and get support from the Pipecat team. Installation & Setup On this page What You Can Build How It Works Real-time Processing Next Steps Join Our Community Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_rime_9a6599dd.txt b/tts_rime_9a6599dd.txt
new file mode 100644
index 0000000000000000000000000000000000000000..6856ce2cfdad891b04b3bf24366030bbe0cd8983
--- /dev/null
+++ b/tts_rime_9a6599dd.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/rime#overview
+Title: Rime - Pipecat
+==================================================
+
+Rime - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech Rime Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview Rime AI provides two TTS service implementations: RimeTTSService : WebSocket-based implementation with word-level timing and interruption support RimeHttpTTSService : HTTP-based implementation for simpler use cases ​ Installation To use Rime services, install the required dependencies: Copy Ask AI pip install "pipecat-ai[rime]" You can obtain a Rime API key by signing up at Rime . ​ Choosing a Rime service Rime has two supported services: RimeTTSService which is a websocket-based implementation RimeHttpTTSService , which is an HTTP-based implementation ​ RimeTTSService The RimeTTSService is recommended for real-time interactive applications. It offers: Word-level timing information for precise synchronization Support for interruptions and context management Context tracking across multiple messages within a turn Non-blocking operation that allows other frames to be processed while audio is being generated ​ RimeHttpTTSService The RimeHttpTTSService is simpler and more suitable for non-interactive use cases. It: Processes the entire text in one request Supports advanced text control features (pauses, phonemes, inline speed) Blocks during the HTTP request, preventing other frames from being processed until the audio is fully generated ​ Input Parameters Both services use the same base input parameters structure: ​ language Language default: "Language.EN" The language to use for synthesis. See Language Support section for available options. ​ speed_alpha float default: "1.0" Speech rate multiplier. Values less than 1.0 increase speed, values greater than 1.0 decrease speed. ​ reduce_latency bool default: "false" Trade accuracy for lower latency ​ pause_between_brackets bool default: "false" When set to true , adds pauses between words enclosed in angle brackets. The number inside the brackets specifies the pause duration in milliseconds. Example: "Hi. <200> I'd love to have a conversation with you." adds a 200ms pause between the first and second sentences. ​ phonemize_between_brackets bool default: "false" When set to true, you can specify the phonemes for a word enclosed in curly brackets. Example: "{h'El.o} World" will pronounce "Hello" as expected. See Rime’s docs for more details. ​ RimeTTSService (WebSocket) Uses Rime’s WebSocket JSON API for real-time speech synthesis with word-level timing information. ​ Constructor Parameters ​ api_key str required Rime API key ​ voice_id str required Rime voice identifier ​ url str default: "wss://users.rime.ai/ws2" Rime WebSocket API endpoint ​ model str default: "mistv2" Model ID to use for synthesis ​ sample_rate int default: "None" Output audio sample rate in Hz ​ params InputParams default: "InputParams()" Speech generation parameters (see Input Parameters section above) ​ text_aggregator BaseTextAggregator default: "SkipTagsAggregator" Text aggregator for processing input text. Defaults to skipping content between spell( and ) tags. ​ Features Word-level timing information with cumulative timing across messages Support for interruptions with context clearing Context tracking across multiple messages within a turn Real-time audio streaming Proper sentence aggregation with skip tags support ​ RimeHttpTTSService (HTTP) HTTP-based implementation for simpler synthesis requirements. ​ Constructor Parameters ​ api_key str required Rime API key ​ voice_id str required Rime voice identifier. See Rime’s documentation for supported voices. ​ aiohttp_session aiohttp.ClientSession required HTTP session for making requests ​ model str default: "mistv2" Choose mistv2 for hyper-realistic conversational voices, mist for Rime’s previous generation model, or the latest arcana model. ​ sample_rate int default: "None" Output audio sample rate in Hz ​ params InputParams default: "InputParams()" Speech generation parameters Show additional properties ​ inline_speed_alpha str default: "None" Comma-separated list of speed values applied to words in square brackets. Values < 1.0 speed up speech, > 1.0 slow it down. Example: "This sentence is [really] [fast]" with inline_speed_alpha set to "0.5, 3" will make “really” slow and “fast” fast. ​ Output Frames Both services generate the following frames: ​ Control Frames ​ TTSStartedFrame Frame Signals start of speech synthesis ​ TTSStoppedFrame Frame Signals completion of speech synthesis ​ Audio Frames ​ TTSAudioRawFrame Frame Contains generated audio data: PCM audio format, specified sample rate, single channel (mono) ​ Error Frames ​ ErrorFrame Frame Contains Rime TTS error information ​ Language Support Supports multiple languages through the Language enum: Language Code Description Service Code Language.DE German ger Language.EN English eng Language.ES Spanish spa Language.FR French fra ​ Usage Examples ​ WebSocket Service Copy Ask AI from pipecat.services.rime.tts import RimeTTSService from pipecat.transcriptions.language import Language # Configure WebSocket service ws_tts = RimeTTSService( api_key = "your-rime-api-key" , voice_id = "cove" , model = "mistv2" , params = RimeTTSService.InputParams( language = Language. EN , speed_alpha = 1.0 , reduce_latency = False ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ HTTP Service Copy Ask AI import aiohttp from pipecat.services.rime.tts import RimeHttpTTSService # Configure HTTP service async with aiohttp.ClientSession() as session: http_tts = RimeHttpTTSService( api_key = "your-rime-api-key" , voice_id = "eva" , aiohttp_session = session, model = "mistv2" , params = RimeHttpTTSService.InputParams( speed_alpha = 1.2 , reduce_latency = True , pause_between_brackets = True , inline_speed_alpha = "0.8,1.5" ) ) ​ Frame Flow ​ Metrics Support Both services collect processing metrics: Time to First Byte (TTFB) Character usage statistics ​ Service Comparison Feature WebSocket HTTP Word timing ✓ - Interruption support ✓ - Context tracking ✓ - Bracket-based pauses ✓ ✓ Phoneme control ✓ ✓ Inline speed control - ✓ Streaming audio ✓ ✓ Arcana model support - ✓ PlayHT Sarvam AI On this page Overview Installation Choosing a Rime service RimeTTSService RimeHttpTTSService Input Parameters RimeTTSService (WebSocket) Constructor Parameters Features RimeHttpTTSService (HTTP) Constructor Parameters Output Frames Control Frames Audio Frames Error Frames Language Support Usage Examples WebSocket Service HTTP Service Frame Flow Metrics Support Service Comparison Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_rime_a27fecc5.txt b/tts_rime_a27fecc5.txt
new file mode 100644
index 0000000000000000000000000000000000000000..5781dc9a82d86d348509c2fe4e7120e13ec36991
--- /dev/null
+++ b/tts_rime_a27fecc5.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/rime#rimehttpttsservice
+Title: Rime - Pipecat
+==================================================
+
+Rime - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech Rime Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview Rime AI provides two TTS service implementations: RimeTTSService : WebSocket-based implementation with word-level timing and interruption support RimeHttpTTSService : HTTP-based implementation for simpler use cases ​ Installation To use Rime services, install the required dependencies: Copy Ask AI pip install "pipecat-ai[rime]" You can obtain a Rime API key by signing up at Rime . ​ Choosing a Rime service Rime has two supported services: RimeTTSService which is a websocket-based implementation RimeHttpTTSService , which is an HTTP-based implementation ​ RimeTTSService The RimeTTSService is recommended for real-time interactive applications. It offers: Word-level timing information for precise synchronization Support for interruptions and context management Context tracking across multiple messages within a turn Non-blocking operation that allows other frames to be processed while audio is being generated ​ RimeHttpTTSService The RimeHttpTTSService is simpler and more suitable for non-interactive use cases. It: Processes the entire text in one request Supports advanced text control features (pauses, phonemes, inline speed) Blocks during the HTTP request, preventing other frames from being processed until the audio is fully generated ​ Input Parameters Both services use the same base input parameters structure: ​ language Language default: "Language.EN" The language to use for synthesis. See Language Support section for available options. ​ speed_alpha float default: "1.0" Speech rate multiplier. Values less than 1.0 increase speed, values greater than 1.0 decrease speed. ​ reduce_latency bool default: "false" Trade accuracy for lower latency ​ pause_between_brackets bool default: "false" When set to true , adds pauses between words enclosed in angle brackets. The number inside the brackets specifies the pause duration in milliseconds. Example: "Hi. <200> I'd love to have a conversation with you." adds a 200ms pause between the first and second sentences. ​ phonemize_between_brackets bool default: "false" When set to true, you can specify the phonemes for a word enclosed in curly brackets. Example: "{h'El.o} World" will pronounce "Hello" as expected. See Rime’s docs for more details. ​ RimeTTSService (WebSocket) Uses Rime’s WebSocket JSON API for real-time speech synthesis with word-level timing information. ​ Constructor Parameters ​ api_key str required Rime API key ​ voice_id str required Rime voice identifier ​ url str default: "wss://users.rime.ai/ws2" Rime WebSocket API endpoint ​ model str default: "mistv2" Model ID to use for synthesis ​ sample_rate int default: "None" Output audio sample rate in Hz ​ params InputParams default: "InputParams()" Speech generation parameters (see Input Parameters section above) ​ text_aggregator BaseTextAggregator default: "SkipTagsAggregator" Text aggregator for processing input text. Defaults to skipping content between spell( and ) tags. ​ Features Word-level timing information with cumulative timing across messages Support for interruptions with context clearing Context tracking across multiple messages within a turn Real-time audio streaming Proper sentence aggregation with skip tags support ​ RimeHttpTTSService (HTTP) HTTP-based implementation for simpler synthesis requirements. ​ Constructor Parameters ​ api_key str required Rime API key ​ voice_id str required Rime voice identifier. See Rime’s documentation for supported voices. ​ aiohttp_session aiohttp.ClientSession required HTTP session for making requests ​ model str default: "mistv2" Choose mistv2 for hyper-realistic conversational voices, mist for Rime’s previous generation model, or the latest arcana model. ​ sample_rate int default: "None" Output audio sample rate in Hz ​ params InputParams default: "InputParams()" Speech generation parameters Show additional properties ​ inline_speed_alpha str default: "None" Comma-separated list of speed values applied to words in square brackets. Values < 1.0 speed up speech, > 1.0 slow it down. Example: "This sentence is [really] [fast]" with inline_speed_alpha set to "0.5, 3" will make “really” slow and “fast” fast. ​ Output Frames Both services generate the following frames: ​ Control Frames ​ TTSStartedFrame Frame Signals start of speech synthesis ​ TTSStoppedFrame Frame Signals completion of speech synthesis ​ Audio Frames ​ TTSAudioRawFrame Frame Contains generated audio data: PCM audio format, specified sample rate, single channel (mono) ​ Error Frames ​ ErrorFrame Frame Contains Rime TTS error information ​ Language Support Supports multiple languages through the Language enum: Language Code Description Service Code Language.DE German ger Language.EN English eng Language.ES Spanish spa Language.FR French fra ​ Usage Examples ​ WebSocket Service Copy Ask AI from pipecat.services.rime.tts import RimeTTSService from pipecat.transcriptions.language import Language # Configure WebSocket service ws_tts = RimeTTSService( api_key = "your-rime-api-key" , voice_id = "cove" , model = "mistv2" , params = RimeTTSService.InputParams( language = Language. EN , speed_alpha = 1.0 , reduce_latency = False ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ HTTP Service Copy Ask AI import aiohttp from pipecat.services.rime.tts import RimeHttpTTSService # Configure HTTP service async with aiohttp.ClientSession() as session: http_tts = RimeHttpTTSService( api_key = "your-rime-api-key" , voice_id = "eva" , aiohttp_session = session, model = "mistv2" , params = RimeHttpTTSService.InputParams( speed_alpha = 1.2 , reduce_latency = True , pause_between_brackets = True , inline_speed_alpha = "0.8,1.5" ) ) ​ Frame Flow ​ Metrics Support Both services collect processing metrics: Time to First Byte (TTFB) Character usage statistics ​ Service Comparison Feature WebSocket HTTP Word timing ✓ - Interruption support ✓ - Context tracking ✓ - Bracket-based pauses ✓ ✓ Phoneme control ✓ ✓ Inline speed control - ✓ Streaming audio ✓ ✓ Arcana model support - ✓ PlayHT Sarvam AI On this page Overview Installation Choosing a Rime service RimeTTSService RimeHttpTTSService Input Parameters RimeTTSService (WebSocket) Constructor Parameters Features RimeHttpTTSService (HTTP) Constructor Parameters Output Frames Control Frames Audio Frames Error Frames Language Support Usage Examples WebSocket Service HTTP Service Frame Flow Metrics Support Service Comparison Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_rime_c4d20444.txt b/tts_rime_c4d20444.txt
new file mode 100644
index 0000000000000000000000000000000000000000..a9a99775a185d2023c3368916cc00cd781f5cdab
--- /dev/null
+++ b/tts_rime_c4d20444.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/rime#constructor-parameters-2
+Title: Rime - Pipecat
+==================================================
+
+Rime - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech Rime Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview Rime AI provides two TTS service implementations: RimeTTSService : WebSocket-based implementation with word-level timing and interruption support RimeHttpTTSService : HTTP-based implementation for simpler use cases ​ Installation To use Rime services, install the required dependencies: Copy Ask AI pip install "pipecat-ai[rime]" You can obtain a Rime API key by signing up at Rime . ​ Choosing a Rime service Rime has two supported services: RimeTTSService which is a websocket-based implementation RimeHttpTTSService , which is an HTTP-based implementation ​ RimeTTSService The RimeTTSService is recommended for real-time interactive applications. It offers: Word-level timing information for precise synchronization Support for interruptions and context management Context tracking across multiple messages within a turn Non-blocking operation that allows other frames to be processed while audio is being generated ​ RimeHttpTTSService The RimeHttpTTSService is simpler and more suitable for non-interactive use cases. It: Processes the entire text in one request Supports advanced text control features (pauses, phonemes, inline speed) Blocks during the HTTP request, preventing other frames from being processed until the audio is fully generated ​ Input Parameters Both services use the same base input parameters structure: ​ language Language default: "Language.EN" The language to use for synthesis. See Language Support section for available options. ​ speed_alpha float default: "1.0" Speech rate multiplier. Values less than 1.0 increase speed, values greater than 1.0 decrease speed. ​ reduce_latency bool default: "false" Trade accuracy for lower latency ​ pause_between_brackets bool default: "false" When set to true , adds pauses between words enclosed in angle brackets. The number inside the brackets specifies the pause duration in milliseconds. Example: "Hi. <200> I'd love to have a conversation with you." adds a 200ms pause between the first and second sentences. ​ phonemize_between_brackets bool default: "false" When set to true, you can specify the phonemes for a word enclosed in curly brackets. Example: "{h'El.o} World" will pronounce "Hello" as expected. See Rime’s docs for more details. ​ RimeTTSService (WebSocket) Uses Rime’s WebSocket JSON API for real-time speech synthesis with word-level timing information. ​ Constructor Parameters ​ api_key str required Rime API key ​ voice_id str required Rime voice identifier ​ url str default: "wss://users.rime.ai/ws2" Rime WebSocket API endpoint ​ model str default: "mistv2" Model ID to use for synthesis ​ sample_rate int default: "None" Output audio sample rate in Hz ​ params InputParams default: "InputParams()" Speech generation parameters (see Input Parameters section above) ​ text_aggregator BaseTextAggregator default: "SkipTagsAggregator" Text aggregator for processing input text. Defaults to skipping content between spell( and ) tags. ​ Features Word-level timing information with cumulative timing across messages Support for interruptions with context clearing Context tracking across multiple messages within a turn Real-time audio streaming Proper sentence aggregation with skip tags support ​ RimeHttpTTSService (HTTP) HTTP-based implementation for simpler synthesis requirements. ​ Constructor Parameters ​ api_key str required Rime API key ​ voice_id str required Rime voice identifier. See Rime’s documentation for supported voices. ​ aiohttp_session aiohttp.ClientSession required HTTP session for making requests ​ model str default: "mistv2" Choose mistv2 for hyper-realistic conversational voices, mist for Rime’s previous generation model, or the latest arcana model. ​ sample_rate int default: "None" Output audio sample rate in Hz ​ params InputParams default: "InputParams()" Speech generation parameters Show additional properties ​ inline_speed_alpha str default: "None" Comma-separated list of speed values applied to words in square brackets. Values < 1.0 speed up speech, > 1.0 slow it down. Example: "This sentence is [really] [fast]" with inline_speed_alpha set to "0.5, 3" will make “really” slow and “fast” fast. ​ Output Frames Both services generate the following frames: ​ Control Frames ​ TTSStartedFrame Frame Signals start of speech synthesis ​ TTSStoppedFrame Frame Signals completion of speech synthesis ​ Audio Frames ​ TTSAudioRawFrame Frame Contains generated audio data: PCM audio format, specified sample rate, single channel (mono) ​ Error Frames ​ ErrorFrame Frame Contains Rime TTS error information ​ Language Support Supports multiple languages through the Language enum: Language Code Description Service Code Language.DE German ger Language.EN English eng Language.ES Spanish spa Language.FR French fra ​ Usage Examples ​ WebSocket Service Copy Ask AI from pipecat.services.rime.tts import RimeTTSService from pipecat.transcriptions.language import Language # Configure WebSocket service ws_tts = RimeTTSService( api_key = "your-rime-api-key" , voice_id = "cove" , model = "mistv2" , params = RimeTTSService.InputParams( language = Language. EN , speed_alpha = 1.0 , reduce_latency = False ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ HTTP Service Copy Ask AI import aiohttp from pipecat.services.rime.tts import RimeHttpTTSService # Configure HTTP service async with aiohttp.ClientSession() as session: http_tts = RimeHttpTTSService( api_key = "your-rime-api-key" , voice_id = "eva" , aiohttp_session = session, model = "mistv2" , params = RimeHttpTTSService.InputParams( speed_alpha = 1.2 , reduce_latency = True , pause_between_brackets = True , inline_speed_alpha = "0.8,1.5" ) ) ​ Frame Flow ​ Metrics Support Both services collect processing metrics: Time to First Byte (TTFB) Character usage statistics ​ Service Comparison Feature WebSocket HTTP Word timing ✓ - Interruption support ✓ - Context tracking ✓ - Bracket-based pauses ✓ ✓ Phoneme control ✓ ✓ Inline speed control - ✓ Streaming audio ✓ ✓ Arcana model support - ✓ PlayHT Sarvam AI On this page Overview Installation Choosing a Rime service RimeTTSService RimeHttpTTSService Input Parameters RimeTTSService (WebSocket) Constructor Parameters Features RimeHttpTTSService (HTTP) Constructor Parameters Output Frames Control Frames Audio Frames Error Frames Language Support Usage Examples WebSocket Service HTTP Service Frame Flow Metrics Support Service Comparison Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_rime_e07bbcdd.txt b/tts_rime_e07bbcdd.txt
new file mode 100644
index 0000000000000000000000000000000000000000..937701dd731100a5e9e831f64f43ca40af09f924
--- /dev/null
+++ b/tts_rime_e07bbcdd.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/rime#param-api-key
+Title: Rime - Pipecat
+==================================================
+
+Rime - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech Rime Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview Rime AI provides two TTS service implementations: RimeTTSService : WebSocket-based implementation with word-level timing and interruption support RimeHttpTTSService : HTTP-based implementation for simpler use cases ​ Installation To use Rime services, install the required dependencies: Copy Ask AI pip install "pipecat-ai[rime]" You can obtain a Rime API key by signing up at Rime . ​ Choosing a Rime service Rime has two supported services: RimeTTSService which is a websocket-based implementation RimeHttpTTSService , which is an HTTP-based implementation ​ RimeTTSService The RimeTTSService is recommended for real-time interactive applications. It offers: Word-level timing information for precise synchronization Support for interruptions and context management Context tracking across multiple messages within a turn Non-blocking operation that allows other frames to be processed while audio is being generated ​ RimeHttpTTSService The RimeHttpTTSService is simpler and more suitable for non-interactive use cases. It: Processes the entire text in one request Supports advanced text control features (pauses, phonemes, inline speed) Blocks during the HTTP request, preventing other frames from being processed until the audio is fully generated ​ Input Parameters Both services use the same base input parameters structure: ​ language Language default: "Language.EN" The language to use for synthesis. See Language Support section for available options. ​ speed_alpha float default: "1.0" Speech rate multiplier. Values less than 1.0 increase speed, values greater than 1.0 decrease speed. ​ reduce_latency bool default: "false" Trade accuracy for lower latency ​ pause_between_brackets bool default: "false" When set to true , adds pauses between words enclosed in angle brackets. The number inside the brackets specifies the pause duration in milliseconds. Example: "Hi. <200> I'd love to have a conversation with you." adds a 200ms pause between the first and second sentences. ​ phonemize_between_brackets bool default: "false" When set to true, you can specify the phonemes for a word enclosed in curly brackets. Example: "{h'El.o} World" will pronounce "Hello" as expected. See Rime’s docs for more details. ​ RimeTTSService (WebSocket) Uses Rime’s WebSocket JSON API for real-time speech synthesis with word-level timing information. ​ Constructor Parameters ​ api_key str required Rime API key ​ voice_id str required Rime voice identifier ​ url str default: "wss://users.rime.ai/ws2" Rime WebSocket API endpoint ​ model str default: "mistv2" Model ID to use for synthesis ​ sample_rate int default: "None" Output audio sample rate in Hz ​ params InputParams default: "InputParams()" Speech generation parameters (see Input Parameters section above) ​ text_aggregator BaseTextAggregator default: "SkipTagsAggregator" Text aggregator for processing input text. Defaults to skipping content between spell( and ) tags. ​ Features Word-level timing information with cumulative timing across messages Support for interruptions with context clearing Context tracking across multiple messages within a turn Real-time audio streaming Proper sentence aggregation with skip tags support ​ RimeHttpTTSService (HTTP) HTTP-based implementation for simpler synthesis requirements. ​ Constructor Parameters ​ api_key str required Rime API key ​ voice_id str required Rime voice identifier. See Rime’s documentation for supported voices. ​ aiohttp_session aiohttp.ClientSession required HTTP session for making requests ​ model str default: "mistv2" Choose mistv2 for hyper-realistic conversational voices, mist for Rime’s previous generation model, or the latest arcana model. ​ sample_rate int default: "None" Output audio sample rate in Hz ​ params InputParams default: "InputParams()" Speech generation parameters Show additional properties ​ inline_speed_alpha str default: "None" Comma-separated list of speed values applied to words in square brackets. Values < 1.0 speed up speech, > 1.0 slow it down. Example: "This sentence is [really] [fast]" with inline_speed_alpha set to "0.5, 3" will make “really” slow and “fast” fast. ​ Output Frames Both services generate the following frames: ​ Control Frames ​ TTSStartedFrame Frame Signals start of speech synthesis ​ TTSStoppedFrame Frame Signals completion of speech synthesis ​ Audio Frames ​ TTSAudioRawFrame Frame Contains generated audio data: PCM audio format, specified sample rate, single channel (mono) ​ Error Frames ​ ErrorFrame Frame Contains Rime TTS error information ​ Language Support Supports multiple languages through the Language enum: Language Code Description Service Code Language.DE German ger Language.EN English eng Language.ES Spanish spa Language.FR French fra ​ Usage Examples ​ WebSocket Service Copy Ask AI from pipecat.services.rime.tts import RimeTTSService from pipecat.transcriptions.language import Language # Configure WebSocket service ws_tts = RimeTTSService( api_key = "your-rime-api-key" , voice_id = "cove" , model = "mistv2" , params = RimeTTSService.InputParams( language = Language. EN , speed_alpha = 1.0 , reduce_latency = False ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ HTTP Service Copy Ask AI import aiohttp from pipecat.services.rime.tts import RimeHttpTTSService # Configure HTTP service async with aiohttp.ClientSession() as session: http_tts = RimeHttpTTSService( api_key = "your-rime-api-key" , voice_id = "eva" , aiohttp_session = session, model = "mistv2" , params = RimeHttpTTSService.InputParams( speed_alpha = 1.2 , reduce_latency = True , pause_between_brackets = True , inline_speed_alpha = "0.8,1.5" ) ) ​ Frame Flow ​ Metrics Support Both services collect processing metrics: Time to First Byte (TTFB) Character usage statistics ​ Service Comparison Feature WebSocket HTTP Word timing ✓ - Interruption support ✓ - Context tracking ✓ - Bracket-based pauses ✓ ✓ Phoneme control ✓ ✓ Inline speed control - ✓ Streaming audio ✓ ✓ Arcana model support - ✓ PlayHT Sarvam AI On this page Overview Installation Choosing a Rime service RimeTTSService RimeHttpTTSService Input Parameters RimeTTSService (WebSocket) Constructor Parameters Features RimeHttpTTSService (HTTP) Constructor Parameters Output Frames Control Frames Audio Frames Error Frames Language Support Usage Examples WebSocket Service HTTP Service Frame Flow Metrics Support Service Comparison Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_rime_f0ab63bb.txt b/tts_rime_f0ab63bb.txt
new file mode 100644
index 0000000000000000000000000000000000000000..cab24d14d6c55b08646dc911f5a512081c97f1d2
--- /dev/null
+++ b/tts_rime_f0ab63bb.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/rime#param-tts-started-frame
+Title: Rime - Pipecat
+==================================================
+
+Rime - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech Rime Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview Rime AI provides two TTS service implementations: RimeTTSService : WebSocket-based implementation with word-level timing and interruption support RimeHttpTTSService : HTTP-based implementation for simpler use cases ​ Installation To use Rime services, install the required dependencies: Copy Ask AI pip install "pipecat-ai[rime]" You can obtain a Rime API key by signing up at Rime . ​ Choosing a Rime service Rime has two supported services: RimeTTSService which is a websocket-based implementation RimeHttpTTSService , which is an HTTP-based implementation ​ RimeTTSService The RimeTTSService is recommended for real-time interactive applications. It offers: Word-level timing information for precise synchronization Support for interruptions and context management Context tracking across multiple messages within a turn Non-blocking operation that allows other frames to be processed while audio is being generated ​ RimeHttpTTSService The RimeHttpTTSService is simpler and more suitable for non-interactive use cases. It: Processes the entire text in one request Supports advanced text control features (pauses, phonemes, inline speed) Blocks during the HTTP request, preventing other frames from being processed until the audio is fully generated ​ Input Parameters Both services use the same base input parameters structure: ​ language Language default: "Language.EN" The language to use for synthesis. See Language Support section for available options. ​ speed_alpha float default: "1.0" Speech rate multiplier. Values less than 1.0 increase speed, values greater than 1.0 decrease speed. ​ reduce_latency bool default: "false" Trade accuracy for lower latency ​ pause_between_brackets bool default: "false" When set to true , adds pauses between words enclosed in angle brackets. The number inside the brackets specifies the pause duration in milliseconds. Example: "Hi. <200> I'd love to have a conversation with you." adds a 200ms pause between the first and second sentences. ​ phonemize_between_brackets bool default: "false" When set to true, you can specify the phonemes for a word enclosed in curly brackets. Example: "{h'El.o} World" will pronounce "Hello" as expected. See Rime’s docs for more details. ​ RimeTTSService (WebSocket) Uses Rime’s WebSocket JSON API for real-time speech synthesis with word-level timing information. ​ Constructor Parameters ​ api_key str required Rime API key ​ voice_id str required Rime voice identifier ​ url str default: "wss://users.rime.ai/ws2" Rime WebSocket API endpoint ​ model str default: "mistv2" Model ID to use for synthesis ​ sample_rate int default: "None" Output audio sample rate in Hz ​ params InputParams default: "InputParams()" Speech generation parameters (see Input Parameters section above) ​ text_aggregator BaseTextAggregator default: "SkipTagsAggregator" Text aggregator for processing input text. Defaults to skipping content between spell( and ) tags. ​ Features Word-level timing information with cumulative timing across messages Support for interruptions with context clearing Context tracking across multiple messages within a turn Real-time audio streaming Proper sentence aggregation with skip tags support ​ RimeHttpTTSService (HTTP) HTTP-based implementation for simpler synthesis requirements. ​ Constructor Parameters ​ api_key str required Rime API key ​ voice_id str required Rime voice identifier. See Rime’s documentation for supported voices. ​ aiohttp_session aiohttp.ClientSession required HTTP session for making requests ​ model str default: "mistv2" Choose mistv2 for hyper-realistic conversational voices, mist for Rime’s previous generation model, or the latest arcana model. ​ sample_rate int default: "None" Output audio sample rate in Hz ​ params InputParams default: "InputParams()" Speech generation parameters Show additional properties ​ inline_speed_alpha str default: "None" Comma-separated list of speed values applied to words in square brackets. Values < 1.0 speed up speech, > 1.0 slow it down. Example: "This sentence is [really] [fast]" with inline_speed_alpha set to "0.5, 3" will make “really” slow and “fast” fast. ​ Output Frames Both services generate the following frames: ​ Control Frames ​ TTSStartedFrame Frame Signals start of speech synthesis ​ TTSStoppedFrame Frame Signals completion of speech synthesis ​ Audio Frames ​ TTSAudioRawFrame Frame Contains generated audio data: PCM audio format, specified sample rate, single channel (mono) ​ Error Frames ​ ErrorFrame Frame Contains Rime TTS error information ​ Language Support Supports multiple languages through the Language enum: Language Code Description Service Code Language.DE German ger Language.EN English eng Language.ES Spanish spa Language.FR French fra ​ Usage Examples ​ WebSocket Service Copy Ask AI from pipecat.services.rime.tts import RimeTTSService from pipecat.transcriptions.language import Language # Configure WebSocket service ws_tts = RimeTTSService( api_key = "your-rime-api-key" , voice_id = "cove" , model = "mistv2" , params = RimeTTSService.InputParams( language = Language. EN , speed_alpha = 1.0 , reduce_latency = False ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ HTTP Service Copy Ask AI import aiohttp from pipecat.services.rime.tts import RimeHttpTTSService # Configure HTTP service async with aiohttp.ClientSession() as session: http_tts = RimeHttpTTSService( api_key = "your-rime-api-key" , voice_id = "eva" , aiohttp_session = session, model = "mistv2" , params = RimeHttpTTSService.InputParams( speed_alpha = 1.2 , reduce_latency = True , pause_between_brackets = True , inline_speed_alpha = "0.8,1.5" ) ) ​ Frame Flow ​ Metrics Support Both services collect processing metrics: Time to First Byte (TTFB) Character usage statistics ​ Service Comparison Feature WebSocket HTTP Word timing ✓ - Interruption support ✓ - Context tracking ✓ - Bracket-based pauses ✓ ✓ Phoneme control ✓ ✓ Inline speed control - ✓ Streaming audio ✓ ✓ Arcana model support - ✓ PlayHT Sarvam AI On this page Overview Installation Choosing a Rime service RimeTTSService RimeHttpTTSService Input Parameters RimeTTSService (WebSocket) Constructor Parameters Features RimeHttpTTSService (HTTP) Constructor Parameters Output Frames Control Frames Audio Frames Error Frames Language Support Usage Examples WebSocket Service HTTP Service Frame Flow Metrics Support Service Comparison Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_rime_f20434c9.txt b/tts_rime_f20434c9.txt
new file mode 100644
index 0000000000000000000000000000000000000000..53f97f037c900cbf74312e70d509496ccbb41ba7
--- /dev/null
+++ b/tts_rime_f20434c9.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/rime#param-sample-rate-1
+Title: Rime - Pipecat
+==================================================
+
+Rime - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech Rime Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview Rime AI provides two TTS service implementations: RimeTTSService : WebSocket-based implementation with word-level timing and interruption support RimeHttpTTSService : HTTP-based implementation for simpler use cases ​ Installation To use Rime services, install the required dependencies: Copy Ask AI pip install "pipecat-ai[rime]" You can obtain a Rime API key by signing up at Rime . ​ Choosing a Rime service Rime has two supported services: RimeTTSService which is a websocket-based implementation RimeHttpTTSService , which is an HTTP-based implementation ​ RimeTTSService The RimeTTSService is recommended for real-time interactive applications. It offers: Word-level timing information for precise synchronization Support for interruptions and context management Context tracking across multiple messages within a turn Non-blocking operation that allows other frames to be processed while audio is being generated ​ RimeHttpTTSService The RimeHttpTTSService is simpler and more suitable for non-interactive use cases. It: Processes the entire text in one request Supports advanced text control features (pauses, phonemes, inline speed) Blocks during the HTTP request, preventing other frames from being processed until the audio is fully generated ​ Input Parameters Both services use the same base input parameters structure: ​ language Language default: "Language.EN" The language to use for synthesis. See Language Support section for available options. ​ speed_alpha float default: "1.0" Speech rate multiplier. Values less than 1.0 increase speed, values greater than 1.0 decrease speed. ​ reduce_latency bool default: "false" Trade accuracy for lower latency ​ pause_between_brackets bool default: "false" When set to true , adds pauses between words enclosed in angle brackets. The number inside the brackets specifies the pause duration in milliseconds. Example: "Hi. <200> I'd love to have a conversation with you." adds a 200ms pause between the first and second sentences. ​ phonemize_between_brackets bool default: "false" When set to true, you can specify the phonemes for a word enclosed in curly brackets. Example: "{h'El.o} World" will pronounce "Hello" as expected. See Rime’s docs for more details. ​ RimeTTSService (WebSocket) Uses Rime’s WebSocket JSON API for real-time speech synthesis with word-level timing information. ​ Constructor Parameters ​ api_key str required Rime API key ​ voice_id str required Rime voice identifier ​ url str default: "wss://users.rime.ai/ws2" Rime WebSocket API endpoint ​ model str default: "mistv2" Model ID to use for synthesis ​ sample_rate int default: "None" Output audio sample rate in Hz ​ params InputParams default: "InputParams()" Speech generation parameters (see Input Parameters section above) ​ text_aggregator BaseTextAggregator default: "SkipTagsAggregator" Text aggregator for processing input text. Defaults to skipping content between spell( and ) tags. ​ Features Word-level timing information with cumulative timing across messages Support for interruptions with context clearing Context tracking across multiple messages within a turn Real-time audio streaming Proper sentence aggregation with skip tags support ​ RimeHttpTTSService (HTTP) HTTP-based implementation for simpler synthesis requirements. ​ Constructor Parameters ​ api_key str required Rime API key ​ voice_id str required Rime voice identifier. See Rime’s documentation for supported voices. ​ aiohttp_session aiohttp.ClientSession required HTTP session for making requests ​ model str default: "mistv2" Choose mistv2 for hyper-realistic conversational voices, mist for Rime’s previous generation model, or the latest arcana model. ​ sample_rate int default: "None" Output audio sample rate in Hz ​ params InputParams default: "InputParams()" Speech generation parameters Show additional properties ​ inline_speed_alpha str default: "None" Comma-separated list of speed values applied to words in square brackets. Values < 1.0 speed up speech, > 1.0 slow it down. Example: "This sentence is [really] [fast]" with inline_speed_alpha set to "0.5, 3" will make “really” slow and “fast” fast. ​ Output Frames Both services generate the following frames: ​ Control Frames ​ TTSStartedFrame Frame Signals start of speech synthesis ​ TTSStoppedFrame Frame Signals completion of speech synthesis ​ Audio Frames ​ TTSAudioRawFrame Frame Contains generated audio data: PCM audio format, specified sample rate, single channel (mono) ​ Error Frames ​ ErrorFrame Frame Contains Rime TTS error information ​ Language Support Supports multiple languages through the Language enum: Language Code Description Service Code Language.DE German ger Language.EN English eng Language.ES Spanish spa Language.FR French fra ​ Usage Examples ​ WebSocket Service Copy Ask AI from pipecat.services.rime.tts import RimeTTSService from pipecat.transcriptions.language import Language # Configure WebSocket service ws_tts = RimeTTSService( api_key = "your-rime-api-key" , voice_id = "cove" , model = "mistv2" , params = RimeTTSService.InputParams( language = Language. EN , speed_alpha = 1.0 , reduce_latency = False ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ HTTP Service Copy Ask AI import aiohttp from pipecat.services.rime.tts import RimeHttpTTSService # Configure HTTP service async with aiohttp.ClientSession() as session: http_tts = RimeHttpTTSService( api_key = "your-rime-api-key" , voice_id = "eva" , aiohttp_session = session, model = "mistv2" , params = RimeHttpTTSService.InputParams( speed_alpha = 1.2 , reduce_latency = True , pause_between_brackets = True , inline_speed_alpha = "0.8,1.5" ) ) ​ Frame Flow ​ Metrics Support Both services collect processing metrics: Time to First Byte (TTFB) Character usage statistics ​ Service Comparison Feature WebSocket HTTP Word timing ✓ - Interruption support ✓ - Context tracking ✓ - Bracket-based pauses ✓ ✓ Phoneme control ✓ ✓ Inline speed control - ✓ Streaming audio ✓ ✓ Arcana model support - ✓ PlayHT Sarvam AI On this page Overview Installation Choosing a Rime service RimeTTSService RimeHttpTTSService Input Parameters RimeTTSService (WebSocket) Constructor Parameters Features RimeHttpTTSService (HTTP) Constructor Parameters Output Frames Control Frames Audio Frames Error Frames Language Support Usage Examples WebSocket Service HTTP Service Frame Flow Metrics Support Service Comparison Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_riva_0b344001.txt b/tts_riva_0b344001.txt
new file mode 100644
index 0000000000000000000000000000000000000000..08e2244afe10a07330e4de389451eefaeffc6eeb
--- /dev/null
+++ b/tts_riva_0b344001.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/riva#audio-processing
+Title: NVIDIA Riva - Pipecat
+==================================================
+
+NVIDIA Riva - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech NVIDIA Riva Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview RivaTTSService converts text to speech using NVIDIA’s Riva. It provides high-quality text-to-speech synthesis with configurable voice options, including multilingual voices. ​ Installation To use RivaTTSService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[riva]" You’ll also need to set up your NVIDIA API key as an environment variable: NVIDIA_API_KEY ​ Configuration ​ Constructor Parameters ​ api_key str required Your NVIDIA API key ​ server str default: "grpc.nvcf.nvidia.com:443" NVIDIA Riva server address ​ voice_id str default: "Magpie-Multilingual.EN-US.Ray" Voice identifier to use for synthesis ​ sample_rate int default: "None" Output audio sample rate in Hz ​ model_function_map Mapping [str, str] A mapping of the NVIDIA function identifier for the TTS service with the model name. ​ params InputParams default: "InputParams()" Additional configuration parameters (language and quality) ​ InputParams ​ language Language default: "Language.EN_US" The language for TTS generation ​ quality int default: "20" Quality level for the generated audio ​ Input The service accepts text input through its TTS pipeline. ​ Output Frames ​ TTSStartedFrame Signals the start of audio generation. ​ TTSAudioRawFrame Contains generated audio data: ​ audio bytes Raw audio data chunk ​ sample_rate int Audio sample rate ​ num_channels int Number of audio channels (1 for mono) ​ TTSStoppedFrame Signals the completion of audio generation. ​ Methods See the TTS base class methods for additional functionality. ​ Models Model Model Card Link magpie-tts-multilingual NVIDIA Model Card fastpitch-hifigan-tts NVIDIA Model Card ​ Language Support Riva model magpie-tts-multilingual (default) supports English, Spanish, and French: Language Code Description Service Codes Language.EN_US English (US) en-US Language.ES-US Spanish (US) es-US Language.FR-FR French (France) fr-FR ​ Usage Examples ​ TTS Language and Voice Configuration Copy Ask AI from pipecat.services.riva.tts import RivaTTSService from pipecat.transcriptions.language import Language # Configure service tts = RivaTTSService( api_key = "your-nvidia-api-key" , voice_id = "Magpie-Multilingual.FR-FR.Louise" , params = RivaTTSService.InputParams( language = Language. FR_FR , quality = 20 ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ Model, function ID, and Voice configuration Copy Ask AI # Configure TTS with specific language tts = RivaTTSService( api_key = "your-nvidia-api-key" , voice_id = "English-US.Female-1" , model_function_map = { "function_id" : "0149dedb-2be8-4195-b9a0-e57e0e14f972" , "model_name" : "fastpitch-hifigan-tts" , } ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ Frame Flow ​ Metrics Support The service supports metrics collection: Time to First Byte (TTFB) TTS usage metrics Processing duration ​ Audio Processing Processes audio through the Riva API Generates mono audio output Handles asynchronous audio streaming Configurable sampling rate ​ Notes Uses NVIDIA’s Riva AI Services platform Streams audio in chunks Requires valid NVIDIA API key Thread-safe processing with asyncio Neuphonic OpenAI On this page Overview Installation Configuration Constructor Parameters InputParams Input Output Frames TTSStartedFrame TTSAudioRawFrame TTSStoppedFrame Methods Models Language Support Usage Examples TTS Language and Voice Configuration Model, function ID, and Voice configuration Frame Flow Metrics Support Audio Processing Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_riva_28d1d327.txt b/tts_riva_28d1d327.txt
new file mode 100644
index 0000000000000000000000000000000000000000..7d2d881daa52417bd0fea3a652e05c4ff76dede8
--- /dev/null
+++ b/tts_riva_28d1d327.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/riva#param-language
+Title: NVIDIA Riva - Pipecat
+==================================================
+
+NVIDIA Riva - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech NVIDIA Riva Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview RivaTTSService converts text to speech using NVIDIA’s Riva. It provides high-quality text-to-speech synthesis with configurable voice options, including multilingual voices. ​ Installation To use RivaTTSService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[riva]" You’ll also need to set up your NVIDIA API key as an environment variable: NVIDIA_API_KEY ​ Configuration ​ Constructor Parameters ​ api_key str required Your NVIDIA API key ​ server str default: "grpc.nvcf.nvidia.com:443" NVIDIA Riva server address ​ voice_id str default: "Magpie-Multilingual.EN-US.Ray" Voice identifier to use for synthesis ​ sample_rate int default: "None" Output audio sample rate in Hz ​ model_function_map Mapping [str, str] A mapping of the NVIDIA function identifier for the TTS service with the model name. ​ params InputParams default: "InputParams()" Additional configuration parameters (language and quality) ​ InputParams ​ language Language default: "Language.EN_US" The language for TTS generation ​ quality int default: "20" Quality level for the generated audio ​ Input The service accepts text input through its TTS pipeline. ​ Output Frames ​ TTSStartedFrame Signals the start of audio generation. ​ TTSAudioRawFrame Contains generated audio data: ​ audio bytes Raw audio data chunk ​ sample_rate int Audio sample rate ​ num_channels int Number of audio channels (1 for mono) ​ TTSStoppedFrame Signals the completion of audio generation. ​ Methods See the TTS base class methods for additional functionality. ​ Models Model Model Card Link magpie-tts-multilingual NVIDIA Model Card fastpitch-hifigan-tts NVIDIA Model Card ​ Language Support Riva model magpie-tts-multilingual (default) supports English, Spanish, and French: Language Code Description Service Codes Language.EN_US English (US) en-US Language.ES-US Spanish (US) es-US Language.FR-FR French (France) fr-FR ​ Usage Examples ​ TTS Language and Voice Configuration Copy Ask AI from pipecat.services.riva.tts import RivaTTSService from pipecat.transcriptions.language import Language # Configure service tts = RivaTTSService( api_key = "your-nvidia-api-key" , voice_id = "Magpie-Multilingual.FR-FR.Louise" , params = RivaTTSService.InputParams( language = Language. FR_FR , quality = 20 ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ Model, function ID, and Voice configuration Copy Ask AI # Configure TTS with specific language tts = RivaTTSService( api_key = "your-nvidia-api-key" , voice_id = "English-US.Female-1" , model_function_map = { "function_id" : "0149dedb-2be8-4195-b9a0-e57e0e14f972" , "model_name" : "fastpitch-hifigan-tts" , } ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ Frame Flow ​ Metrics Support The service supports metrics collection: Time to First Byte (TTFB) TTS usage metrics Processing duration ​ Audio Processing Processes audio through the Riva API Generates mono audio output Handles asynchronous audio streaming Configurable sampling rate ​ Notes Uses NVIDIA’s Riva AI Services platform Streams audio in chunks Requires valid NVIDIA API key Thread-safe processing with asyncio Neuphonic OpenAI On this page Overview Installation Configuration Constructor Parameters InputParams Input Output Frames TTSStartedFrame TTSAudioRawFrame TTSStoppedFrame Methods Models Language Support Usage Examples TTS Language and Voice Configuration Model, function ID, and Voice configuration Frame Flow Metrics Support Audio Processing Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_riva_44dcc448.txt b/tts_riva_44dcc448.txt
new file mode 100644
index 0000000000000000000000000000000000000000..47166ba8479a79b83b460e399a158d6b00ec6186
--- /dev/null
+++ b/tts_riva_44dcc448.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/riva#ttsstartedframe
+Title: NVIDIA Riva - Pipecat
+==================================================
+
+NVIDIA Riva - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech NVIDIA Riva Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview RivaTTSService converts text to speech using NVIDIA’s Riva. It provides high-quality text-to-speech synthesis with configurable voice options, including multilingual voices. ​ Installation To use RivaTTSService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[riva]" You’ll also need to set up your NVIDIA API key as an environment variable: NVIDIA_API_KEY ​ Configuration ​ Constructor Parameters ​ api_key str required Your NVIDIA API key ​ server str default: "grpc.nvcf.nvidia.com:443" NVIDIA Riva server address ​ voice_id str default: "Magpie-Multilingual.EN-US.Ray" Voice identifier to use for synthesis ​ sample_rate int default: "None" Output audio sample rate in Hz ​ model_function_map Mapping [str, str] A mapping of the NVIDIA function identifier for the TTS service with the model name. ​ params InputParams default: "InputParams()" Additional configuration parameters (language and quality) ​ InputParams ​ language Language default: "Language.EN_US" The language for TTS generation ​ quality int default: "20" Quality level for the generated audio ​ Input The service accepts text input through its TTS pipeline. ​ Output Frames ​ TTSStartedFrame Signals the start of audio generation. ​ TTSAudioRawFrame Contains generated audio data: ​ audio bytes Raw audio data chunk ​ sample_rate int Audio sample rate ​ num_channels int Number of audio channels (1 for mono) ​ TTSStoppedFrame Signals the completion of audio generation. ​ Methods See the TTS base class methods for additional functionality. ​ Models Model Model Card Link magpie-tts-multilingual NVIDIA Model Card fastpitch-hifigan-tts NVIDIA Model Card ​ Language Support Riva model magpie-tts-multilingual (default) supports English, Spanish, and French: Language Code Description Service Codes Language.EN_US English (US) en-US Language.ES-US Spanish (US) es-US Language.FR-FR French (France) fr-FR ​ Usage Examples ​ TTS Language and Voice Configuration Copy Ask AI from pipecat.services.riva.tts import RivaTTSService from pipecat.transcriptions.language import Language # Configure service tts = RivaTTSService( api_key = "your-nvidia-api-key" , voice_id = "Magpie-Multilingual.FR-FR.Louise" , params = RivaTTSService.InputParams( language = Language. FR_FR , quality = 20 ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ Model, function ID, and Voice configuration Copy Ask AI # Configure TTS with specific language tts = RivaTTSService( api_key = "your-nvidia-api-key" , voice_id = "English-US.Female-1" , model_function_map = { "function_id" : "0149dedb-2be8-4195-b9a0-e57e0e14f972" , "model_name" : "fastpitch-hifigan-tts" , } ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ Frame Flow ​ Metrics Support The service supports metrics collection: Time to First Byte (TTFB) TTS usage metrics Processing duration ​ Audio Processing Processes audio through the Riva API Generates mono audio output Handles asynchronous audio streaming Configurable sampling rate ​ Notes Uses NVIDIA’s Riva AI Services platform Streams audio in chunks Requires valid NVIDIA API key Thread-safe processing with asyncio Neuphonic OpenAI On this page Overview Installation Configuration Constructor Parameters InputParams Input Output Frames TTSStartedFrame TTSAudioRawFrame TTSStoppedFrame Methods Models Language Support Usage Examples TTS Language and Voice Configuration Model, function ID, and Voice configuration Frame Flow Metrics Support Audio Processing Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_riva_5148aa70.txt b/tts_riva_5148aa70.txt
new file mode 100644
index 0000000000000000000000000000000000000000..43f497439ff42364ac5c99912fb0035d6098b2fc
--- /dev/null
+++ b/tts_riva_5148aa70.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/riva#language-support
+Title: NVIDIA Riva - Pipecat
+==================================================
+
+NVIDIA Riva - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech NVIDIA Riva Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview RivaTTSService converts text to speech using NVIDIA’s Riva. It provides high-quality text-to-speech synthesis with configurable voice options, including multilingual voices. ​ Installation To use RivaTTSService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[riva]" You’ll also need to set up your NVIDIA API key as an environment variable: NVIDIA_API_KEY ​ Configuration ​ Constructor Parameters ​ api_key str required Your NVIDIA API key ​ server str default: "grpc.nvcf.nvidia.com:443" NVIDIA Riva server address ​ voice_id str default: "Magpie-Multilingual.EN-US.Ray" Voice identifier to use for synthesis ​ sample_rate int default: "None" Output audio sample rate in Hz ​ model_function_map Mapping [str, str] A mapping of the NVIDIA function identifier for the TTS service with the model name. ​ params InputParams default: "InputParams()" Additional configuration parameters (language and quality) ​ InputParams ​ language Language default: "Language.EN_US" The language for TTS generation ​ quality int default: "20" Quality level for the generated audio ​ Input The service accepts text input through its TTS pipeline. ​ Output Frames ​ TTSStartedFrame Signals the start of audio generation. ​ TTSAudioRawFrame Contains generated audio data: ​ audio bytes Raw audio data chunk ​ sample_rate int Audio sample rate ​ num_channels int Number of audio channels (1 for mono) ​ TTSStoppedFrame Signals the completion of audio generation. ​ Methods See the TTS base class methods for additional functionality. ​ Models Model Model Card Link magpie-tts-multilingual NVIDIA Model Card fastpitch-hifigan-tts NVIDIA Model Card ​ Language Support Riva model magpie-tts-multilingual (default) supports English, Spanish, and French: Language Code Description Service Codes Language.EN_US English (US) en-US Language.ES-US Spanish (US) es-US Language.FR-FR French (France) fr-FR ​ Usage Examples ​ TTS Language and Voice Configuration Copy Ask AI from pipecat.services.riva.tts import RivaTTSService from pipecat.transcriptions.language import Language # Configure service tts = RivaTTSService( api_key = "your-nvidia-api-key" , voice_id = "Magpie-Multilingual.FR-FR.Louise" , params = RivaTTSService.InputParams( language = Language. FR_FR , quality = 20 ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ Model, function ID, and Voice configuration Copy Ask AI # Configure TTS with specific language tts = RivaTTSService( api_key = "your-nvidia-api-key" , voice_id = "English-US.Female-1" , model_function_map = { "function_id" : "0149dedb-2be8-4195-b9a0-e57e0e14f972" , "model_name" : "fastpitch-hifigan-tts" , } ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ Frame Flow ​ Metrics Support The service supports metrics collection: Time to First Byte (TTFB) TTS usage metrics Processing duration ​ Audio Processing Processes audio through the Riva API Generates mono audio output Handles asynchronous audio streaming Configurable sampling rate ​ Notes Uses NVIDIA’s Riva AI Services platform Streams audio in chunks Requires valid NVIDIA API key Thread-safe processing with asyncio Neuphonic OpenAI On this page Overview Installation Configuration Constructor Parameters InputParams Input Output Frames TTSStartedFrame TTSAudioRawFrame TTSStoppedFrame Methods Models Language Support Usage Examples TTS Language and Voice Configuration Model, function ID, and Voice configuration Frame Flow Metrics Support Audio Processing Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_riva_760665c8.txt b/tts_riva_760665c8.txt
new file mode 100644
index 0000000000000000000000000000000000000000..c85e8d9be4435781776779f4bfb7657b9bbdffbe
--- /dev/null
+++ b/tts_riva_760665c8.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/riva#configuration
+Title: NVIDIA Riva - Pipecat
+==================================================
+
+NVIDIA Riva - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech NVIDIA Riva Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview RivaTTSService converts text to speech using NVIDIA’s Riva. It provides high-quality text-to-speech synthesis with configurable voice options, including multilingual voices. ​ Installation To use RivaTTSService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[riva]" You’ll also need to set up your NVIDIA API key as an environment variable: NVIDIA_API_KEY ​ Configuration ​ Constructor Parameters ​ api_key str required Your NVIDIA API key ​ server str default: "grpc.nvcf.nvidia.com:443" NVIDIA Riva server address ​ voice_id str default: "Magpie-Multilingual.EN-US.Ray" Voice identifier to use for synthesis ​ sample_rate int default: "None" Output audio sample rate in Hz ​ model_function_map Mapping [str, str] A mapping of the NVIDIA function identifier for the TTS service with the model name. ​ params InputParams default: "InputParams()" Additional configuration parameters (language and quality) ​ InputParams ​ language Language default: "Language.EN_US" The language for TTS generation ​ quality int default: "20" Quality level for the generated audio ​ Input The service accepts text input through its TTS pipeline. ​ Output Frames ​ TTSStartedFrame Signals the start of audio generation. ​ TTSAudioRawFrame Contains generated audio data: ​ audio bytes Raw audio data chunk ​ sample_rate int Audio sample rate ​ num_channels int Number of audio channels (1 for mono) ​ TTSStoppedFrame Signals the completion of audio generation. ​ Methods See the TTS base class methods for additional functionality. ​ Models Model Model Card Link magpie-tts-multilingual NVIDIA Model Card fastpitch-hifigan-tts NVIDIA Model Card ​ Language Support Riva model magpie-tts-multilingual (default) supports English, Spanish, and French: Language Code Description Service Codes Language.EN_US English (US) en-US Language.ES-US Spanish (US) es-US Language.FR-FR French (France) fr-FR ​ Usage Examples ​ TTS Language and Voice Configuration Copy Ask AI from pipecat.services.riva.tts import RivaTTSService from pipecat.transcriptions.language import Language # Configure service tts = RivaTTSService( api_key = "your-nvidia-api-key" , voice_id = "Magpie-Multilingual.FR-FR.Louise" , params = RivaTTSService.InputParams( language = Language. FR_FR , quality = 20 ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ Model, function ID, and Voice configuration Copy Ask AI # Configure TTS with specific language tts = RivaTTSService( api_key = "your-nvidia-api-key" , voice_id = "English-US.Female-1" , model_function_map = { "function_id" : "0149dedb-2be8-4195-b9a0-e57e0e14f972" , "model_name" : "fastpitch-hifigan-tts" , } ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ Frame Flow ​ Metrics Support The service supports metrics collection: Time to First Byte (TTFB) TTS usage metrics Processing duration ​ Audio Processing Processes audio through the Riva API Generates mono audio output Handles asynchronous audio streaming Configurable sampling rate ​ Notes Uses NVIDIA’s Riva AI Services platform Streams audio in chunks Requires valid NVIDIA API key Thread-safe processing with asyncio Neuphonic OpenAI On this page Overview Installation Configuration Constructor Parameters InputParams Input Output Frames TTSStartedFrame TTSAudioRawFrame TTSStoppedFrame Methods Models Language Support Usage Examples TTS Language and Voice Configuration Model, function ID, and Voice configuration Frame Flow Metrics Support Audio Processing Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_riva_b7591ce6.txt b/tts_riva_b7591ce6.txt
new file mode 100644
index 0000000000000000000000000000000000000000..722df8d859e912a9fd57d69ddbb3ef8dd57587a3
--- /dev/null
+++ b/tts_riva_b7591ce6.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/riva#models
+Title: NVIDIA Riva - Pipecat
+==================================================
+
+NVIDIA Riva - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech NVIDIA Riva Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview RivaTTSService converts text to speech using NVIDIA’s Riva. It provides high-quality text-to-speech synthesis with configurable voice options, including multilingual voices. ​ Installation To use RivaTTSService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[riva]" You’ll also need to set up your NVIDIA API key as an environment variable: NVIDIA_API_KEY ​ Configuration ​ Constructor Parameters ​ api_key str required Your NVIDIA API key ​ server str default: "grpc.nvcf.nvidia.com:443" NVIDIA Riva server address ​ voice_id str default: "Magpie-Multilingual.EN-US.Ray" Voice identifier to use for synthesis ​ sample_rate int default: "None" Output audio sample rate in Hz ​ model_function_map Mapping [str, str] A mapping of the NVIDIA function identifier for the TTS service with the model name. ​ params InputParams default: "InputParams()" Additional configuration parameters (language and quality) ​ InputParams ​ language Language default: "Language.EN_US" The language for TTS generation ​ quality int default: "20" Quality level for the generated audio ​ Input The service accepts text input through its TTS pipeline. ​ Output Frames ​ TTSStartedFrame Signals the start of audio generation. ​ TTSAudioRawFrame Contains generated audio data: ​ audio bytes Raw audio data chunk ​ sample_rate int Audio sample rate ​ num_channels int Number of audio channels (1 for mono) ​ TTSStoppedFrame Signals the completion of audio generation. ​ Methods See the TTS base class methods for additional functionality. ​ Models Model Model Card Link magpie-tts-multilingual NVIDIA Model Card fastpitch-hifigan-tts NVIDIA Model Card ​ Language Support Riva model magpie-tts-multilingual (default) supports English, Spanish, and French: Language Code Description Service Codes Language.EN_US English (US) en-US Language.ES-US Spanish (US) es-US Language.FR-FR French (France) fr-FR ​ Usage Examples ​ TTS Language and Voice Configuration Copy Ask AI from pipecat.services.riva.tts import RivaTTSService from pipecat.transcriptions.language import Language # Configure service tts = RivaTTSService( api_key = "your-nvidia-api-key" , voice_id = "Magpie-Multilingual.FR-FR.Louise" , params = RivaTTSService.InputParams( language = Language. FR_FR , quality = 20 ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ Model, function ID, and Voice configuration Copy Ask AI # Configure TTS with specific language tts = RivaTTSService( api_key = "your-nvidia-api-key" , voice_id = "English-US.Female-1" , model_function_map = { "function_id" : "0149dedb-2be8-4195-b9a0-e57e0e14f972" , "model_name" : "fastpitch-hifigan-tts" , } ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ Frame Flow ​ Metrics Support The service supports metrics collection: Time to First Byte (TTFB) TTS usage metrics Processing duration ​ Audio Processing Processes audio through the Riva API Generates mono audio output Handles asynchronous audio streaming Configurable sampling rate ​ Notes Uses NVIDIA’s Riva AI Services platform Streams audio in chunks Requires valid NVIDIA API key Thread-safe processing with asyncio Neuphonic OpenAI On this page Overview Installation Configuration Constructor Parameters InputParams Input Output Frames TTSStartedFrame TTSAudioRawFrame TTSStoppedFrame Methods Models Language Support Usage Examples TTS Language and Voice Configuration Model, function ID, and Voice configuration Frame Flow Metrics Support Audio Processing Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_riva_c5091687.txt b/tts_riva_c5091687.txt
new file mode 100644
index 0000000000000000000000000000000000000000..6b5a4157dbd62d8d213af07a5983e023a523be3d
--- /dev/null
+++ b/tts_riva_c5091687.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/riva#inputparams
+Title: NVIDIA Riva - Pipecat
+==================================================
+
+NVIDIA Riva - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech NVIDIA Riva Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview RivaTTSService converts text to speech using NVIDIA’s Riva. It provides high-quality text-to-speech synthesis with configurable voice options, including multilingual voices. ​ Installation To use RivaTTSService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[riva]" You’ll also need to set up your NVIDIA API key as an environment variable: NVIDIA_API_KEY ​ Configuration ​ Constructor Parameters ​ api_key str required Your NVIDIA API key ​ server str default: "grpc.nvcf.nvidia.com:443" NVIDIA Riva server address ​ voice_id str default: "Magpie-Multilingual.EN-US.Ray" Voice identifier to use for synthesis ​ sample_rate int default: "None" Output audio sample rate in Hz ​ model_function_map Mapping [str, str] A mapping of the NVIDIA function identifier for the TTS service with the model name. ​ params InputParams default: "InputParams()" Additional configuration parameters (language and quality) ​ InputParams ​ language Language default: "Language.EN_US" The language for TTS generation ​ quality int default: "20" Quality level for the generated audio ​ Input The service accepts text input through its TTS pipeline. ​ Output Frames ​ TTSStartedFrame Signals the start of audio generation. ​ TTSAudioRawFrame Contains generated audio data: ​ audio bytes Raw audio data chunk ​ sample_rate int Audio sample rate ​ num_channels int Number of audio channels (1 for mono) ​ TTSStoppedFrame Signals the completion of audio generation. ​ Methods See the TTS base class methods for additional functionality. ​ Models Model Model Card Link magpie-tts-multilingual NVIDIA Model Card fastpitch-hifigan-tts NVIDIA Model Card ​ Language Support Riva model magpie-tts-multilingual (default) supports English, Spanish, and French: Language Code Description Service Codes Language.EN_US English (US) en-US Language.ES-US Spanish (US) es-US Language.FR-FR French (France) fr-FR ​ Usage Examples ​ TTS Language and Voice Configuration Copy Ask AI from pipecat.services.riva.tts import RivaTTSService from pipecat.transcriptions.language import Language # Configure service tts = RivaTTSService( api_key = "your-nvidia-api-key" , voice_id = "Magpie-Multilingual.FR-FR.Louise" , params = RivaTTSService.InputParams( language = Language. FR_FR , quality = 20 ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ Model, function ID, and Voice configuration Copy Ask AI # Configure TTS with specific language tts = RivaTTSService( api_key = "your-nvidia-api-key" , voice_id = "English-US.Female-1" , model_function_map = { "function_id" : "0149dedb-2be8-4195-b9a0-e57e0e14f972" , "model_name" : "fastpitch-hifigan-tts" , } ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ Frame Flow ​ Metrics Support The service supports metrics collection: Time to First Byte (TTFB) TTS usage metrics Processing duration ​ Audio Processing Processes audio through the Riva API Generates mono audio output Handles asynchronous audio streaming Configurable sampling rate ​ Notes Uses NVIDIA’s Riva AI Services platform Streams audio in chunks Requires valid NVIDIA API key Thread-safe processing with asyncio Neuphonic OpenAI On this page Overview Installation Configuration Constructor Parameters InputParams Input Output Frames TTSStartedFrame TTSAudioRawFrame TTSStoppedFrame Methods Models Language Support Usage Examples TTS Language and Voice Configuration Model, function ID, and Voice configuration Frame Flow Metrics Support Audio Processing Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_riva_fd6e78cd.txt b/tts_riva_fd6e78cd.txt
new file mode 100644
index 0000000000000000000000000000000000000000..75c0f464edb9f09dada0d317a0447387a4a5ef7b
--- /dev/null
+++ b/tts_riva_fd6e78cd.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/riva#overview
+Title: NVIDIA Riva - Pipecat
+==================================================
+
+NVIDIA Riva - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech NVIDIA Riva Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview RivaTTSService converts text to speech using NVIDIA’s Riva. It provides high-quality text-to-speech synthesis with configurable voice options, including multilingual voices. ​ Installation To use RivaTTSService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[riva]" You’ll also need to set up your NVIDIA API key as an environment variable: NVIDIA_API_KEY ​ Configuration ​ Constructor Parameters ​ api_key str required Your NVIDIA API key ​ server str default: "grpc.nvcf.nvidia.com:443" NVIDIA Riva server address ​ voice_id str default: "Magpie-Multilingual.EN-US.Ray" Voice identifier to use for synthesis ​ sample_rate int default: "None" Output audio sample rate in Hz ​ model_function_map Mapping [str, str] A mapping of the NVIDIA function identifier for the TTS service with the model name. ​ params InputParams default: "InputParams()" Additional configuration parameters (language and quality) ​ InputParams ​ language Language default: "Language.EN_US" The language for TTS generation ​ quality int default: "20" Quality level for the generated audio ​ Input The service accepts text input through its TTS pipeline. ​ Output Frames ​ TTSStartedFrame Signals the start of audio generation. ​ TTSAudioRawFrame Contains generated audio data: ​ audio bytes Raw audio data chunk ​ sample_rate int Audio sample rate ​ num_channels int Number of audio channels (1 for mono) ​ TTSStoppedFrame Signals the completion of audio generation. ​ Methods See the TTS base class methods for additional functionality. ​ Models Model Model Card Link magpie-tts-multilingual NVIDIA Model Card fastpitch-hifigan-tts NVIDIA Model Card ​ Language Support Riva model magpie-tts-multilingual (default) supports English, Spanish, and French: Language Code Description Service Codes Language.EN_US English (US) en-US Language.ES-US Spanish (US) es-US Language.FR-FR French (France) fr-FR ​ Usage Examples ​ TTS Language and Voice Configuration Copy Ask AI from pipecat.services.riva.tts import RivaTTSService from pipecat.transcriptions.language import Language # Configure service tts = RivaTTSService( api_key = "your-nvidia-api-key" , voice_id = "Magpie-Multilingual.FR-FR.Louise" , params = RivaTTSService.InputParams( language = Language. FR_FR , quality = 20 ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ Model, function ID, and Voice configuration Copy Ask AI # Configure TTS with specific language tts = RivaTTSService( api_key = "your-nvidia-api-key" , voice_id = "English-US.Female-1" , model_function_map = { "function_id" : "0149dedb-2be8-4195-b9a0-e57e0e14f972" , "model_name" : "fastpitch-hifigan-tts" , } ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ Frame Flow ​ Metrics Support The service supports metrics collection: Time to First Byte (TTFB) TTS usage metrics Processing duration ​ Audio Processing Processes audio through the Riva API Generates mono audio output Handles asynchronous audio streaming Configurable sampling rate ​ Notes Uses NVIDIA’s Riva AI Services platform Streams audio in chunks Requires valid NVIDIA API key Thread-safe processing with asyncio Neuphonic OpenAI On this page Overview Installation Configuration Constructor Parameters InputParams Input Output Frames TTSStartedFrame TTSAudioRawFrame TTSStoppedFrame Methods Models Language Support Usage Examples TTS Language and Voice Configuration Model, function ID, and Voice configuration Frame Flow Metrics Support Audio Processing Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_riva_fe7abc10.txt b/tts_riva_fe7abc10.txt
new file mode 100644
index 0000000000000000000000000000000000000000..85d8104bc3af43e25dea5e8c40c5906945e209b8
--- /dev/null
+++ b/tts_riva_fe7abc10.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/riva#param-params
+Title: NVIDIA Riva - Pipecat
+==================================================
+
+NVIDIA Riva - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech NVIDIA Riva Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview RivaTTSService converts text to speech using NVIDIA’s Riva. It provides high-quality text-to-speech synthesis with configurable voice options, including multilingual voices. ​ Installation To use RivaTTSService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[riva]" You’ll also need to set up your NVIDIA API key as an environment variable: NVIDIA_API_KEY ​ Configuration ​ Constructor Parameters ​ api_key str required Your NVIDIA API key ​ server str default: "grpc.nvcf.nvidia.com:443" NVIDIA Riva server address ​ voice_id str default: "Magpie-Multilingual.EN-US.Ray" Voice identifier to use for synthesis ​ sample_rate int default: "None" Output audio sample rate in Hz ​ model_function_map Mapping [str, str] A mapping of the NVIDIA function identifier for the TTS service with the model name. ​ params InputParams default: "InputParams()" Additional configuration parameters (language and quality) ​ InputParams ​ language Language default: "Language.EN_US" The language for TTS generation ​ quality int default: "20" Quality level for the generated audio ​ Input The service accepts text input through its TTS pipeline. ​ Output Frames ​ TTSStartedFrame Signals the start of audio generation. ​ TTSAudioRawFrame Contains generated audio data: ​ audio bytes Raw audio data chunk ​ sample_rate int Audio sample rate ​ num_channels int Number of audio channels (1 for mono) ​ TTSStoppedFrame Signals the completion of audio generation. ​ Methods See the TTS base class methods for additional functionality. ​ Models Model Model Card Link magpie-tts-multilingual NVIDIA Model Card fastpitch-hifigan-tts NVIDIA Model Card ​ Language Support Riva model magpie-tts-multilingual (default) supports English, Spanish, and French: Language Code Description Service Codes Language.EN_US English (US) en-US Language.ES-US Spanish (US) es-US Language.FR-FR French (France) fr-FR ​ Usage Examples ​ TTS Language and Voice Configuration Copy Ask AI from pipecat.services.riva.tts import RivaTTSService from pipecat.transcriptions.language import Language # Configure service tts = RivaTTSService( api_key = "your-nvidia-api-key" , voice_id = "Magpie-Multilingual.FR-FR.Louise" , params = RivaTTSService.InputParams( language = Language. FR_FR , quality = 20 ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ Model, function ID, and Voice configuration Copy Ask AI # Configure TTS with specific language tts = RivaTTSService( api_key = "your-nvidia-api-key" , voice_id = "English-US.Female-1" , model_function_map = { "function_id" : "0149dedb-2be8-4195-b9a0-e57e0e14f972" , "model_name" : "fastpitch-hifigan-tts" , } ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ Frame Flow ​ Metrics Support The service supports metrics collection: Time to First Byte (TTFB) TTS usage metrics Processing duration ​ Audio Processing Processes audio through the Riva API Generates mono audio output Handles asynchronous audio streaming Configurable sampling rate ​ Notes Uses NVIDIA’s Riva AI Services platform Streams audio in chunks Requires valid NVIDIA API key Thread-safe processing with asyncio Neuphonic OpenAI On this page Overview Installation Configuration Constructor Parameters InputParams Input Output Frames TTSStartedFrame TTSAudioRawFrame TTSStoppedFrame Methods Models Language Support Usage Examples TTS Language and Voice Configuration Model, function ID, and Voice configuration Frame Flow Metrics Support Audio Processing Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_riva_fe879583.txt b/tts_riva_fe879583.txt
new file mode 100644
index 0000000000000000000000000000000000000000..0e96c353d8c7e31d10544444dc44cdf1711e3ca2
--- /dev/null
+++ b/tts_riva_fe879583.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/riva
+Title: NVIDIA Riva - Pipecat
+==================================================
+
+NVIDIA Riva - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech NVIDIA Riva Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview RivaTTSService converts text to speech using NVIDIA’s Riva. It provides high-quality text-to-speech synthesis with configurable voice options, including multilingual voices. ​ Installation To use RivaTTSService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[riva]" You’ll also need to set up your NVIDIA API key as an environment variable: NVIDIA_API_KEY ​ Configuration ​ Constructor Parameters ​ api_key str required Your NVIDIA API key ​ server str default: "grpc.nvcf.nvidia.com:443" NVIDIA Riva server address ​ voice_id str default: "Magpie-Multilingual.EN-US.Ray" Voice identifier to use for synthesis ​ sample_rate int default: "None" Output audio sample rate in Hz ​ model_function_map Mapping [str, str] A mapping of the NVIDIA function identifier for the TTS service with the model name. ​ params InputParams default: "InputParams()" Additional configuration parameters (language and quality) ​ InputParams ​ language Language default: "Language.EN_US" The language for TTS generation ​ quality int default: "20" Quality level for the generated audio ​ Input The service accepts text input through its TTS pipeline. ​ Output Frames ​ TTSStartedFrame Signals the start of audio generation. ​ TTSAudioRawFrame Contains generated audio data: ​ audio bytes Raw audio data chunk ​ sample_rate int Audio sample rate ​ num_channels int Number of audio channels (1 for mono) ​ TTSStoppedFrame Signals the completion of audio generation. ​ Methods See the TTS base class methods for additional functionality. ​ Models Model Model Card Link magpie-tts-multilingual NVIDIA Model Card fastpitch-hifigan-tts NVIDIA Model Card ​ Language Support Riva model magpie-tts-multilingual (default) supports English, Spanish, and French: Language Code Description Service Codes Language.EN_US English (US) en-US Language.ES-US Spanish (US) es-US Language.FR-FR French (France) fr-FR ​ Usage Examples ​ TTS Language and Voice Configuration Copy Ask AI from pipecat.services.riva.tts import RivaTTSService from pipecat.transcriptions.language import Language # Configure service tts = RivaTTSService( api_key = "your-nvidia-api-key" , voice_id = "Magpie-Multilingual.FR-FR.Louise" , params = RivaTTSService.InputParams( language = Language. FR_FR , quality = 20 ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ Model, function ID, and Voice configuration Copy Ask AI # Configure TTS with specific language tts = RivaTTSService( api_key = "your-nvidia-api-key" , voice_id = "English-US.Female-1" , model_function_map = { "function_id" : "0149dedb-2be8-4195-b9a0-e57e0e14f972" , "model_name" : "fastpitch-hifigan-tts" , } ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ Frame Flow ​ Metrics Support The service supports metrics collection: Time to First Byte (TTFB) TTS usage metrics Processing duration ​ Audio Processing Processes audio through the Riva API Generates mono audio output Handles asynchronous audio streaming Configurable sampling rate ​ Notes Uses NVIDIA’s Riva AI Services platform Streams audio in chunks Requires valid NVIDIA API key Thread-safe processing with asyncio Neuphonic OpenAI On this page Overview Installation Configuration Constructor Parameters InputParams Input Output Frames TTSStartedFrame TTSAudioRawFrame TTSStoppedFrame Methods Models Language Support Usage Examples TTS Language and Voice Configuration Model, function ID, and Voice configuration Frame Flow Metrics Support Audio Processing Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_sarvam_070ea460.txt b/tts_sarvam_070ea460.txt
new file mode 100644
index 0000000000000000000000000000000000000000..7c049c439e03fb9e9ca5d63ea4480f99f6234ba8
--- /dev/null
+++ b/tts_sarvam_070ea460.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/sarvam#param-audio
+Title: Sarvam AI - Pipecat
+==================================================
+
+Sarvam AI - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech Sarvam AI Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview SarvamTTSService converts text to speech using Sarvam AI’s TTS API. It specializes in Indian languages and provides extensive voice customization options including pitch, pace, and loudness control. ​ Installation To use SarvamTTSService , no additional dependencies are required. You’ll also need to set up your Sarvam AI API key as an environment variable: SARVAM_API_KEY ​ Configuration ​ Constructor Parameters ​ api_key str required Your Sarvam AI API subscription key ​ voice_id str default: "anushka" Speaker voice identifier (e.g., “anushka”, “meera”, “abhilash”) ​ model str default: "bulbul:v2" TTS model to use (“bulbul:v1” or “bulbul:v2”) ​ aiohttp_session aiohttp.ClientSession required Shared aiohttp session for making HTTP requests ​ base_url str default: "https://api.sarvam.ai" Sarvam AI API base URL ​ sample_rate int default: "None" Audio sample rate in Hz (8000, 16000, 22050, 24000) ​ params InputParams default: "None" Additional voice and preprocessing parameters ​ InputParams Configuration ​ language Language default: "Language.HI" Target language for synthesis ​ pitch float default: "0.0" Voice pitch adjustment (-0.75 to 0.75) ​ pace float default: "1.0" Speech speed (0.3 to 3.0) ​ loudness float default: "1.0" Audio volume (0.1 to 3.0) ​ enable_preprocessing bool default: "False" Enable text normalization for mixed-language content ​ Input The service accepts text input through its TTS pipeline with automatic WAV header stripping for clean PCM output. ​ Output Frames ​ TTSStartedFrame Signals the start of audio generation. ​ TTSAudioRawFrame Contains generated audio data: ​ audio bytes Raw PCM audio data (WAV header stripped) ​ sample_rate int Audio sample rate (22050Hz default) ​ num_channels int Number of audio channels (1 for mono) ​ TTSStoppedFrame Signals the completion of audio generation. ​ Methods See the TTS base class methods for additional functionality. ​ Language Support Sarvam AI TTS supports the following Indian languages: Language Code Description Service Code Language.BN Bengali bn-IN Language.EN English (India) en-IN Language.GU Gujarati gu-IN Language.HI Hindi hi-IN Language.KN Kannada kn-IN Language.ML Malayalam ml-IN Language.MR Marathi mr-IN Language.OR Odia od-IN Language.PA Punjabi pa-IN Language.TA Tamil ta-IN Language.TE Telugu te-IN ​ Voice Models See the Sarvam docs for the latest information on available voices and models. ​ Usage Example Copy Ask AI from pipecat.services.sarvam.tts import SarvamTTSService from pipecat.transcriptions.language import Language import aiohttp # Configure service async with aiohttp.ClientSession() as session: tts = SarvamTTSService( api_key = "your-api-key" , voice_id = "anushka" , model = "bulbul:v2" , aiohttp_session = session, params = SarvamTTSService.InputParams( language = Language. HI , ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ Frame Flow ​ Metrics Support The service supports metrics collection: Time to First Byte (TTFB) TTS usage metrics Processing duration ​ Audio Processing Returns base64-encoded WAV audio from API Supports multiple sample rates (8000, 16000, 22050, 24000 Hz) Generates mono audio output Handles HTTP-based synthesis ​ Notes Requires valid Sarvam AI API subscription key Specializes in Indian languages and voices Uses HTTP POST requests for synthesis Thread-safe HTTP session management required Rime XTTS On this page Overview Installation Configuration Constructor Parameters InputParams Configuration Input Output Frames TTSStartedFrame TTSAudioRawFrame TTSStoppedFrame Methods Language Support Voice Models Usage Example Frame Flow Metrics Support Audio Processing Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_sarvam_1e537f9e.txt b/tts_sarvam_1e537f9e.txt
new file mode 100644
index 0000000000000000000000000000000000000000..d6da29aff181255ab5051d0748befa7f4675ae7a
--- /dev/null
+++ b/tts_sarvam_1e537f9e.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/sarvam#notes
+Title: Sarvam AI - Pipecat
+==================================================
+
+Sarvam AI - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech Sarvam AI Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview SarvamTTSService converts text to speech using Sarvam AI’s TTS API. It specializes in Indian languages and provides extensive voice customization options including pitch, pace, and loudness control. ​ Installation To use SarvamTTSService , no additional dependencies are required. You’ll also need to set up your Sarvam AI API key as an environment variable: SARVAM_API_KEY ​ Configuration ​ Constructor Parameters ​ api_key str required Your Sarvam AI API subscription key ​ voice_id str default: "anushka" Speaker voice identifier (e.g., “anushka”, “meera”, “abhilash”) ​ model str default: "bulbul:v2" TTS model to use (“bulbul:v1” or “bulbul:v2”) ​ aiohttp_session aiohttp.ClientSession required Shared aiohttp session for making HTTP requests ​ base_url str default: "https://api.sarvam.ai" Sarvam AI API base URL ​ sample_rate int default: "None" Audio sample rate in Hz (8000, 16000, 22050, 24000) ​ params InputParams default: "None" Additional voice and preprocessing parameters ​ InputParams Configuration ​ language Language default: "Language.HI" Target language for synthesis ​ pitch float default: "0.0" Voice pitch adjustment (-0.75 to 0.75) ​ pace float default: "1.0" Speech speed (0.3 to 3.0) ​ loudness float default: "1.0" Audio volume (0.1 to 3.0) ​ enable_preprocessing bool default: "False" Enable text normalization for mixed-language content ​ Input The service accepts text input through its TTS pipeline with automatic WAV header stripping for clean PCM output. ​ Output Frames ​ TTSStartedFrame Signals the start of audio generation. ​ TTSAudioRawFrame Contains generated audio data: ​ audio bytes Raw PCM audio data (WAV header stripped) ​ sample_rate int Audio sample rate (22050Hz default) ​ num_channels int Number of audio channels (1 for mono) ​ TTSStoppedFrame Signals the completion of audio generation. ​ Methods See the TTS base class methods for additional functionality. ​ Language Support Sarvam AI TTS supports the following Indian languages: Language Code Description Service Code Language.BN Bengali bn-IN Language.EN English (India) en-IN Language.GU Gujarati gu-IN Language.HI Hindi hi-IN Language.KN Kannada kn-IN Language.ML Malayalam ml-IN Language.MR Marathi mr-IN Language.OR Odia od-IN Language.PA Punjabi pa-IN Language.TA Tamil ta-IN Language.TE Telugu te-IN ​ Voice Models See the Sarvam docs for the latest information on available voices and models. ​ Usage Example Copy Ask AI from pipecat.services.sarvam.tts import SarvamTTSService from pipecat.transcriptions.language import Language import aiohttp # Configure service async with aiohttp.ClientSession() as session: tts = SarvamTTSService( api_key = "your-api-key" , voice_id = "anushka" , model = "bulbul:v2" , aiohttp_session = session, params = SarvamTTSService.InputParams( language = Language. HI , ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ Frame Flow ​ Metrics Support The service supports metrics collection: Time to First Byte (TTFB) TTS usage metrics Processing duration ​ Audio Processing Returns base64-encoded WAV audio from API Supports multiple sample rates (8000, 16000, 22050, 24000 Hz) Generates mono audio output Handles HTTP-based synthesis ​ Notes Requires valid Sarvam AI API subscription key Specializes in Indian languages and voices Uses HTTP POST requests for synthesis Thread-safe HTTP session management required Rime XTTS On this page Overview Installation Configuration Constructor Parameters InputParams Configuration Input Output Frames TTSStartedFrame TTSAudioRawFrame TTSStoppedFrame Methods Language Support Voice Models Usage Example Frame Flow Metrics Support Audio Processing Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_sarvam_2b8ae187.txt b/tts_sarvam_2b8ae187.txt
new file mode 100644
index 0000000000000000000000000000000000000000..81247c6999b56a6f21a4060c6a717de87049c640
--- /dev/null
+++ b/tts_sarvam_2b8ae187.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/sarvam#overview
+Title: Sarvam AI - Pipecat
+==================================================
+
+Sarvam AI - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech Sarvam AI Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview SarvamTTSService converts text to speech using Sarvam AI’s TTS API. It specializes in Indian languages and provides extensive voice customization options including pitch, pace, and loudness control. ​ Installation To use SarvamTTSService , no additional dependencies are required. You’ll also need to set up your Sarvam AI API key as an environment variable: SARVAM_API_KEY ​ Configuration ​ Constructor Parameters ​ api_key str required Your Sarvam AI API subscription key ​ voice_id str default: "anushka" Speaker voice identifier (e.g., “anushka”, “meera”, “abhilash”) ​ model str default: "bulbul:v2" TTS model to use (“bulbul:v1” or “bulbul:v2”) ​ aiohttp_session aiohttp.ClientSession required Shared aiohttp session for making HTTP requests ​ base_url str default: "https://api.sarvam.ai" Sarvam AI API base URL ​ sample_rate int default: "None" Audio sample rate in Hz (8000, 16000, 22050, 24000) ​ params InputParams default: "None" Additional voice and preprocessing parameters ​ InputParams Configuration ​ language Language default: "Language.HI" Target language for synthesis ​ pitch float default: "0.0" Voice pitch adjustment (-0.75 to 0.75) ​ pace float default: "1.0" Speech speed (0.3 to 3.0) ​ loudness float default: "1.0" Audio volume (0.1 to 3.0) ​ enable_preprocessing bool default: "False" Enable text normalization for mixed-language content ​ Input The service accepts text input through its TTS pipeline with automatic WAV header stripping for clean PCM output. ​ Output Frames ​ TTSStartedFrame Signals the start of audio generation. ​ TTSAudioRawFrame Contains generated audio data: ​ audio bytes Raw PCM audio data (WAV header stripped) ​ sample_rate int Audio sample rate (22050Hz default) ​ num_channels int Number of audio channels (1 for mono) ​ TTSStoppedFrame Signals the completion of audio generation. ​ Methods See the TTS base class methods for additional functionality. ​ Language Support Sarvam AI TTS supports the following Indian languages: Language Code Description Service Code Language.BN Bengali bn-IN Language.EN English (India) en-IN Language.GU Gujarati gu-IN Language.HI Hindi hi-IN Language.KN Kannada kn-IN Language.ML Malayalam ml-IN Language.MR Marathi mr-IN Language.OR Odia od-IN Language.PA Punjabi pa-IN Language.TA Tamil ta-IN Language.TE Telugu te-IN ​ Voice Models See the Sarvam docs for the latest information on available voices and models. ​ Usage Example Copy Ask AI from pipecat.services.sarvam.tts import SarvamTTSService from pipecat.transcriptions.language import Language import aiohttp # Configure service async with aiohttp.ClientSession() as session: tts = SarvamTTSService( api_key = "your-api-key" , voice_id = "anushka" , model = "bulbul:v2" , aiohttp_session = session, params = SarvamTTSService.InputParams( language = Language. HI , ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ Frame Flow ​ Metrics Support The service supports metrics collection: Time to First Byte (TTFB) TTS usage metrics Processing duration ​ Audio Processing Returns base64-encoded WAV audio from API Supports multiple sample rates (8000, 16000, 22050, 24000 Hz) Generates mono audio output Handles HTTP-based synthesis ​ Notes Requires valid Sarvam AI API subscription key Specializes in Indian languages and voices Uses HTTP POST requests for synthesis Thread-safe HTTP session management required Rime XTTS On this page Overview Installation Configuration Constructor Parameters InputParams Configuration Input Output Frames TTSStartedFrame TTSAudioRawFrame TTSStoppedFrame Methods Language Support Voice Models Usage Example Frame Flow Metrics Support Audio Processing Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_sarvam_6a6d0a4d.txt b/tts_sarvam_6a6d0a4d.txt
new file mode 100644
index 0000000000000000000000000000000000000000..e6ed3b6f269b9bd85cbc1144935d23d4da797f39
--- /dev/null
+++ b/tts_sarvam_6a6d0a4d.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/sarvam#ttsstartedframe
+Title: Sarvam AI - Pipecat
+==================================================
+
+Sarvam AI - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech Sarvam AI Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview SarvamTTSService converts text to speech using Sarvam AI’s TTS API. It specializes in Indian languages and provides extensive voice customization options including pitch, pace, and loudness control. ​ Installation To use SarvamTTSService , no additional dependencies are required. You’ll also need to set up your Sarvam AI API key as an environment variable: SARVAM_API_KEY ​ Configuration ​ Constructor Parameters ​ api_key str required Your Sarvam AI API subscription key ​ voice_id str default: "anushka" Speaker voice identifier (e.g., “anushka”, “meera”, “abhilash”) ​ model str default: "bulbul:v2" TTS model to use (“bulbul:v1” or “bulbul:v2”) ​ aiohttp_session aiohttp.ClientSession required Shared aiohttp session for making HTTP requests ​ base_url str default: "https://api.sarvam.ai" Sarvam AI API base URL ​ sample_rate int default: "None" Audio sample rate in Hz (8000, 16000, 22050, 24000) ​ params InputParams default: "None" Additional voice and preprocessing parameters ​ InputParams Configuration ​ language Language default: "Language.HI" Target language for synthesis ​ pitch float default: "0.0" Voice pitch adjustment (-0.75 to 0.75) ​ pace float default: "1.0" Speech speed (0.3 to 3.0) ​ loudness float default: "1.0" Audio volume (0.1 to 3.0) ​ enable_preprocessing bool default: "False" Enable text normalization for mixed-language content ​ Input The service accepts text input through its TTS pipeline with automatic WAV header stripping for clean PCM output. ​ Output Frames ​ TTSStartedFrame Signals the start of audio generation. ​ TTSAudioRawFrame Contains generated audio data: ​ audio bytes Raw PCM audio data (WAV header stripped) ​ sample_rate int Audio sample rate (22050Hz default) ​ num_channels int Number of audio channels (1 for mono) ​ TTSStoppedFrame Signals the completion of audio generation. ​ Methods See the TTS base class methods for additional functionality. ​ Language Support Sarvam AI TTS supports the following Indian languages: Language Code Description Service Code Language.BN Bengali bn-IN Language.EN English (India) en-IN Language.GU Gujarati gu-IN Language.HI Hindi hi-IN Language.KN Kannada kn-IN Language.ML Malayalam ml-IN Language.MR Marathi mr-IN Language.OR Odia od-IN Language.PA Punjabi pa-IN Language.TA Tamil ta-IN Language.TE Telugu te-IN ​ Voice Models See the Sarvam docs for the latest information on available voices and models. ​ Usage Example Copy Ask AI from pipecat.services.sarvam.tts import SarvamTTSService from pipecat.transcriptions.language import Language import aiohttp # Configure service async with aiohttp.ClientSession() as session: tts = SarvamTTSService( api_key = "your-api-key" , voice_id = "anushka" , model = "bulbul:v2" , aiohttp_session = session, params = SarvamTTSService.InputParams( language = Language. HI , ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ Frame Flow ​ Metrics Support The service supports metrics collection: Time to First Byte (TTFB) TTS usage metrics Processing duration ​ Audio Processing Returns base64-encoded WAV audio from API Supports multiple sample rates (8000, 16000, 22050, 24000 Hz) Generates mono audio output Handles HTTP-based synthesis ​ Notes Requires valid Sarvam AI API subscription key Specializes in Indian languages and voices Uses HTTP POST requests for synthesis Thread-safe HTTP session management required Rime XTTS On this page Overview Installation Configuration Constructor Parameters InputParams Configuration Input Output Frames TTSStartedFrame TTSAudioRawFrame TTSStoppedFrame Methods Language Support Voice Models Usage Example Frame Flow Metrics Support Audio Processing Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_sarvam_7dc0b2cd.txt b/tts_sarvam_7dc0b2cd.txt
new file mode 100644
index 0000000000000000000000000000000000000000..7f027081fc21d2b4f635a8c8413e73ad3e8f4283
--- /dev/null
+++ b/tts_sarvam_7dc0b2cd.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/sarvam#param-params
+Title: Sarvam AI - Pipecat
+==================================================
+
+Sarvam AI - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech Sarvam AI Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview SarvamTTSService converts text to speech using Sarvam AI’s TTS API. It specializes in Indian languages and provides extensive voice customization options including pitch, pace, and loudness control. ​ Installation To use SarvamTTSService , no additional dependencies are required. You’ll also need to set up your Sarvam AI API key as an environment variable: SARVAM_API_KEY ​ Configuration ​ Constructor Parameters ​ api_key str required Your Sarvam AI API subscription key ​ voice_id str default: "anushka" Speaker voice identifier (e.g., “anushka”, “meera”, “abhilash”) ​ model str default: "bulbul:v2" TTS model to use (“bulbul:v1” or “bulbul:v2”) ​ aiohttp_session aiohttp.ClientSession required Shared aiohttp session for making HTTP requests ​ base_url str default: "https://api.sarvam.ai" Sarvam AI API base URL ​ sample_rate int default: "None" Audio sample rate in Hz (8000, 16000, 22050, 24000) ​ params InputParams default: "None" Additional voice and preprocessing parameters ​ InputParams Configuration ​ language Language default: "Language.HI" Target language for synthesis ​ pitch float default: "0.0" Voice pitch adjustment (-0.75 to 0.75) ​ pace float default: "1.0" Speech speed (0.3 to 3.0) ​ loudness float default: "1.0" Audio volume (0.1 to 3.0) ​ enable_preprocessing bool default: "False" Enable text normalization for mixed-language content ​ Input The service accepts text input through its TTS pipeline with automatic WAV header stripping for clean PCM output. ​ Output Frames ​ TTSStartedFrame Signals the start of audio generation. ​ TTSAudioRawFrame Contains generated audio data: ​ audio bytes Raw PCM audio data (WAV header stripped) ​ sample_rate int Audio sample rate (22050Hz default) ​ num_channels int Number of audio channels (1 for mono) ​ TTSStoppedFrame Signals the completion of audio generation. ​ Methods See the TTS base class methods for additional functionality. ​ Language Support Sarvam AI TTS supports the following Indian languages: Language Code Description Service Code Language.BN Bengali bn-IN Language.EN English (India) en-IN Language.GU Gujarati gu-IN Language.HI Hindi hi-IN Language.KN Kannada kn-IN Language.ML Malayalam ml-IN Language.MR Marathi mr-IN Language.OR Odia od-IN Language.PA Punjabi pa-IN Language.TA Tamil ta-IN Language.TE Telugu te-IN ​ Voice Models See the Sarvam docs for the latest information on available voices and models. ​ Usage Example Copy Ask AI from pipecat.services.sarvam.tts import SarvamTTSService from pipecat.transcriptions.language import Language import aiohttp # Configure service async with aiohttp.ClientSession() as session: tts = SarvamTTSService( api_key = "your-api-key" , voice_id = "anushka" , model = "bulbul:v2" , aiohttp_session = session, params = SarvamTTSService.InputParams( language = Language. HI , ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ Frame Flow ​ Metrics Support The service supports metrics collection: Time to First Byte (TTFB) TTS usage metrics Processing duration ​ Audio Processing Returns base64-encoded WAV audio from API Supports multiple sample rates (8000, 16000, 22050, 24000 Hz) Generates mono audio output Handles HTTP-based synthesis ​ Notes Requires valid Sarvam AI API subscription key Specializes in Indian languages and voices Uses HTTP POST requests for synthesis Thread-safe HTTP session management required Rime XTTS On this page Overview Installation Configuration Constructor Parameters InputParams Configuration Input Output Frames TTSStartedFrame TTSAudioRawFrame TTSStoppedFrame Methods Language Support Voice Models Usage Example Frame Flow Metrics Support Audio Processing Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_sarvam_9ae67cdc.txt b/tts_sarvam_9ae67cdc.txt
new file mode 100644
index 0000000000000000000000000000000000000000..ddf464b0883e63ad79d486d37152039a43facea2
--- /dev/null
+++ b/tts_sarvam_9ae67cdc.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/sarvam#param-model
+Title: Sarvam AI - Pipecat
+==================================================
+
+Sarvam AI - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech Sarvam AI Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview SarvamTTSService converts text to speech using Sarvam AI’s TTS API. It specializes in Indian languages and provides extensive voice customization options including pitch, pace, and loudness control. ​ Installation To use SarvamTTSService , no additional dependencies are required. You’ll also need to set up your Sarvam AI API key as an environment variable: SARVAM_API_KEY ​ Configuration ​ Constructor Parameters ​ api_key str required Your Sarvam AI API subscription key ​ voice_id str default: "anushka" Speaker voice identifier (e.g., “anushka”, “meera”, “abhilash”) ​ model str default: "bulbul:v2" TTS model to use (“bulbul:v1” or “bulbul:v2”) ​ aiohttp_session aiohttp.ClientSession required Shared aiohttp session for making HTTP requests ​ base_url str default: "https://api.sarvam.ai" Sarvam AI API base URL ​ sample_rate int default: "None" Audio sample rate in Hz (8000, 16000, 22050, 24000) ​ params InputParams default: "None" Additional voice and preprocessing parameters ​ InputParams Configuration ​ language Language default: "Language.HI" Target language for synthesis ​ pitch float default: "0.0" Voice pitch adjustment (-0.75 to 0.75) ​ pace float default: "1.0" Speech speed (0.3 to 3.0) ​ loudness float default: "1.0" Audio volume (0.1 to 3.0) ​ enable_preprocessing bool default: "False" Enable text normalization for mixed-language content ​ Input The service accepts text input through its TTS pipeline with automatic WAV header stripping for clean PCM output. ​ Output Frames ​ TTSStartedFrame Signals the start of audio generation. ​ TTSAudioRawFrame Contains generated audio data: ​ audio bytes Raw PCM audio data (WAV header stripped) ​ sample_rate int Audio sample rate (22050Hz default) ​ num_channels int Number of audio channels (1 for mono) ​ TTSStoppedFrame Signals the completion of audio generation. ​ Methods See the TTS base class methods for additional functionality. ​ Language Support Sarvam AI TTS supports the following Indian languages: Language Code Description Service Code Language.BN Bengali bn-IN Language.EN English (India) en-IN Language.GU Gujarati gu-IN Language.HI Hindi hi-IN Language.KN Kannada kn-IN Language.ML Malayalam ml-IN Language.MR Marathi mr-IN Language.OR Odia od-IN Language.PA Punjabi pa-IN Language.TA Tamil ta-IN Language.TE Telugu te-IN ​ Voice Models See the Sarvam docs for the latest information on available voices and models. ​ Usage Example Copy Ask AI from pipecat.services.sarvam.tts import SarvamTTSService from pipecat.transcriptions.language import Language import aiohttp # Configure service async with aiohttp.ClientSession() as session: tts = SarvamTTSService( api_key = "your-api-key" , voice_id = "anushka" , model = "bulbul:v2" , aiohttp_session = session, params = SarvamTTSService.InputParams( language = Language. HI , ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ Frame Flow ​ Metrics Support The service supports metrics collection: Time to First Byte (TTFB) TTS usage metrics Processing duration ​ Audio Processing Returns base64-encoded WAV audio from API Supports multiple sample rates (8000, 16000, 22050, 24000 Hz) Generates mono audio output Handles HTTP-based synthesis ​ Notes Requires valid Sarvam AI API subscription key Specializes in Indian languages and voices Uses HTTP POST requests for synthesis Thread-safe HTTP session management required Rime XTTS On this page Overview Installation Configuration Constructor Parameters InputParams Configuration Input Output Frames TTSStartedFrame TTSAudioRawFrame TTSStoppedFrame Methods Language Support Voice Models Usage Example Frame Flow Metrics Support Audio Processing Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_sarvam_a54ba4dd.txt b/tts_sarvam_a54ba4dd.txt
new file mode 100644
index 0000000000000000000000000000000000000000..120b435ece3551616d1ff7b3ff9065918b5e74a8
--- /dev/null
+++ b/tts_sarvam_a54ba4dd.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/sarvam#usage-example
+Title: Sarvam AI - Pipecat
+==================================================
+
+Sarvam AI - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech Sarvam AI Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview SarvamTTSService converts text to speech using Sarvam AI’s TTS API. It specializes in Indian languages and provides extensive voice customization options including pitch, pace, and loudness control. ​ Installation To use SarvamTTSService , no additional dependencies are required. You’ll also need to set up your Sarvam AI API key as an environment variable: SARVAM_API_KEY ​ Configuration ​ Constructor Parameters ​ api_key str required Your Sarvam AI API subscription key ​ voice_id str default: "anushka" Speaker voice identifier (e.g., “anushka”, “meera”, “abhilash”) ​ model str default: "bulbul:v2" TTS model to use (“bulbul:v1” or “bulbul:v2”) ​ aiohttp_session aiohttp.ClientSession required Shared aiohttp session for making HTTP requests ​ base_url str default: "https://api.sarvam.ai" Sarvam AI API base URL ​ sample_rate int default: "None" Audio sample rate in Hz (8000, 16000, 22050, 24000) ​ params InputParams default: "None" Additional voice and preprocessing parameters ​ InputParams Configuration ​ language Language default: "Language.HI" Target language for synthesis ​ pitch float default: "0.0" Voice pitch adjustment (-0.75 to 0.75) ​ pace float default: "1.0" Speech speed (0.3 to 3.0) ​ loudness float default: "1.0" Audio volume (0.1 to 3.0) ​ enable_preprocessing bool default: "False" Enable text normalization for mixed-language content ​ Input The service accepts text input through its TTS pipeline with automatic WAV header stripping for clean PCM output. ​ Output Frames ​ TTSStartedFrame Signals the start of audio generation. ​ TTSAudioRawFrame Contains generated audio data: ​ audio bytes Raw PCM audio data (WAV header stripped) ​ sample_rate int Audio sample rate (22050Hz default) ​ num_channels int Number of audio channels (1 for mono) ​ TTSStoppedFrame Signals the completion of audio generation. ​ Methods See the TTS base class methods for additional functionality. ​ Language Support Sarvam AI TTS supports the following Indian languages: Language Code Description Service Code Language.BN Bengali bn-IN Language.EN English (India) en-IN Language.GU Gujarati gu-IN Language.HI Hindi hi-IN Language.KN Kannada kn-IN Language.ML Malayalam ml-IN Language.MR Marathi mr-IN Language.OR Odia od-IN Language.PA Punjabi pa-IN Language.TA Tamil ta-IN Language.TE Telugu te-IN ​ Voice Models See the Sarvam docs for the latest information on available voices and models. ​ Usage Example Copy Ask AI from pipecat.services.sarvam.tts import SarvamTTSService from pipecat.transcriptions.language import Language import aiohttp # Configure service async with aiohttp.ClientSession() as session: tts = SarvamTTSService( api_key = "your-api-key" , voice_id = "anushka" , model = "bulbul:v2" , aiohttp_session = session, params = SarvamTTSService.InputParams( language = Language. HI , ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ Frame Flow ​ Metrics Support The service supports metrics collection: Time to First Byte (TTFB) TTS usage metrics Processing duration ​ Audio Processing Returns base64-encoded WAV audio from API Supports multiple sample rates (8000, 16000, 22050, 24000 Hz) Generates mono audio output Handles HTTP-based synthesis ​ Notes Requires valid Sarvam AI API subscription key Specializes in Indian languages and voices Uses HTTP POST requests for synthesis Thread-safe HTTP session management required Rime XTTS On this page Overview Installation Configuration Constructor Parameters InputParams Configuration Input Output Frames TTSStartedFrame TTSAudioRawFrame TTSStoppedFrame Methods Language Support Voice Models Usage Example Frame Flow Metrics Support Audio Processing Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_sarvam_a80aa1b9.txt b/tts_sarvam_a80aa1b9.txt
new file mode 100644
index 0000000000000000000000000000000000000000..b784e32d3cc45df2a7a005178ede9bc714c4f2df
--- /dev/null
+++ b/tts_sarvam_a80aa1b9.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/sarvam#audio-processing
+Title: Sarvam AI - Pipecat
+==================================================
+
+Sarvam AI - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech Sarvam AI Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview SarvamTTSService converts text to speech using Sarvam AI’s TTS API. It specializes in Indian languages and provides extensive voice customization options including pitch, pace, and loudness control. ​ Installation To use SarvamTTSService , no additional dependencies are required. You’ll also need to set up your Sarvam AI API key as an environment variable: SARVAM_API_KEY ​ Configuration ​ Constructor Parameters ​ api_key str required Your Sarvam AI API subscription key ​ voice_id str default: "anushka" Speaker voice identifier (e.g., “anushka”, “meera”, “abhilash”) ​ model str default: "bulbul:v2" TTS model to use (“bulbul:v1” or “bulbul:v2”) ​ aiohttp_session aiohttp.ClientSession required Shared aiohttp session for making HTTP requests ​ base_url str default: "https://api.sarvam.ai" Sarvam AI API base URL ​ sample_rate int default: "None" Audio sample rate in Hz (8000, 16000, 22050, 24000) ​ params InputParams default: "None" Additional voice and preprocessing parameters ​ InputParams Configuration ​ language Language default: "Language.HI" Target language for synthesis ​ pitch float default: "0.0" Voice pitch adjustment (-0.75 to 0.75) ​ pace float default: "1.0" Speech speed (0.3 to 3.0) ​ loudness float default: "1.0" Audio volume (0.1 to 3.0) ​ enable_preprocessing bool default: "False" Enable text normalization for mixed-language content ​ Input The service accepts text input through its TTS pipeline with automatic WAV header stripping for clean PCM output. ​ Output Frames ​ TTSStartedFrame Signals the start of audio generation. ​ TTSAudioRawFrame Contains generated audio data: ​ audio bytes Raw PCM audio data (WAV header stripped) ​ sample_rate int Audio sample rate (22050Hz default) ​ num_channels int Number of audio channels (1 for mono) ​ TTSStoppedFrame Signals the completion of audio generation. ​ Methods See the TTS base class methods for additional functionality. ​ Language Support Sarvam AI TTS supports the following Indian languages: Language Code Description Service Code Language.BN Bengali bn-IN Language.EN English (India) en-IN Language.GU Gujarati gu-IN Language.HI Hindi hi-IN Language.KN Kannada kn-IN Language.ML Malayalam ml-IN Language.MR Marathi mr-IN Language.OR Odia od-IN Language.PA Punjabi pa-IN Language.TA Tamil ta-IN Language.TE Telugu te-IN ​ Voice Models See the Sarvam docs for the latest information on available voices and models. ​ Usage Example Copy Ask AI from pipecat.services.sarvam.tts import SarvamTTSService from pipecat.transcriptions.language import Language import aiohttp # Configure service async with aiohttp.ClientSession() as session: tts = SarvamTTSService( api_key = "your-api-key" , voice_id = "anushka" , model = "bulbul:v2" , aiohttp_session = session, params = SarvamTTSService.InputParams( language = Language. HI , ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ Frame Flow ​ Metrics Support The service supports metrics collection: Time to First Byte (TTFB) TTS usage metrics Processing duration ​ Audio Processing Returns base64-encoded WAV audio from API Supports multiple sample rates (8000, 16000, 22050, 24000 Hz) Generates mono audio output Handles HTTP-based synthesis ​ Notes Requires valid Sarvam AI API subscription key Specializes in Indian languages and voices Uses HTTP POST requests for synthesis Thread-safe HTTP session management required Rime XTTS On this page Overview Installation Configuration Constructor Parameters InputParams Configuration Input Output Frames TTSStartedFrame TTSAudioRawFrame TTSStoppedFrame Methods Language Support Voice Models Usage Example Frame Flow Metrics Support Audio Processing Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_sarvam_cd86de3f.txt b/tts_sarvam_cd86de3f.txt
new file mode 100644
index 0000000000000000000000000000000000000000..4fff30a52fd503adf3b43671e471cc850807c81b
--- /dev/null
+++ b/tts_sarvam_cd86de3f.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/sarvam#input
+Title: Sarvam AI - Pipecat
+==================================================
+
+Sarvam AI - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech Sarvam AI Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview SarvamTTSService converts text to speech using Sarvam AI’s TTS API. It specializes in Indian languages and provides extensive voice customization options including pitch, pace, and loudness control. ​ Installation To use SarvamTTSService , no additional dependencies are required. You’ll also need to set up your Sarvam AI API key as an environment variable: SARVAM_API_KEY ​ Configuration ​ Constructor Parameters ​ api_key str required Your Sarvam AI API subscription key ​ voice_id str default: "anushka" Speaker voice identifier (e.g., “anushka”, “meera”, “abhilash”) ​ model str default: "bulbul:v2" TTS model to use (“bulbul:v1” or “bulbul:v2”) ​ aiohttp_session aiohttp.ClientSession required Shared aiohttp session for making HTTP requests ​ base_url str default: "https://api.sarvam.ai" Sarvam AI API base URL ​ sample_rate int default: "None" Audio sample rate in Hz (8000, 16000, 22050, 24000) ​ params InputParams default: "None" Additional voice and preprocessing parameters ​ InputParams Configuration ​ language Language default: "Language.HI" Target language for synthesis ​ pitch float default: "0.0" Voice pitch adjustment (-0.75 to 0.75) ​ pace float default: "1.0" Speech speed (0.3 to 3.0) ​ loudness float default: "1.0" Audio volume (0.1 to 3.0) ​ enable_preprocessing bool default: "False" Enable text normalization for mixed-language content ​ Input The service accepts text input through its TTS pipeline with automatic WAV header stripping for clean PCM output. ​ Output Frames ​ TTSStartedFrame Signals the start of audio generation. ​ TTSAudioRawFrame Contains generated audio data: ​ audio bytes Raw PCM audio data (WAV header stripped) ​ sample_rate int Audio sample rate (22050Hz default) ​ num_channels int Number of audio channels (1 for mono) ​ TTSStoppedFrame Signals the completion of audio generation. ​ Methods See the TTS base class methods for additional functionality. ​ Language Support Sarvam AI TTS supports the following Indian languages: Language Code Description Service Code Language.BN Bengali bn-IN Language.EN English (India) en-IN Language.GU Gujarati gu-IN Language.HI Hindi hi-IN Language.KN Kannada kn-IN Language.ML Malayalam ml-IN Language.MR Marathi mr-IN Language.OR Odia od-IN Language.PA Punjabi pa-IN Language.TA Tamil ta-IN Language.TE Telugu te-IN ​ Voice Models See the Sarvam docs for the latest information on available voices and models. ​ Usage Example Copy Ask AI from pipecat.services.sarvam.tts import SarvamTTSService from pipecat.transcriptions.language import Language import aiohttp # Configure service async with aiohttp.ClientSession() as session: tts = SarvamTTSService( api_key = "your-api-key" , voice_id = "anushka" , model = "bulbul:v2" , aiohttp_session = session, params = SarvamTTSService.InputParams( language = Language. HI , ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ Frame Flow ​ Metrics Support The service supports metrics collection: Time to First Byte (TTFB) TTS usage metrics Processing duration ​ Audio Processing Returns base64-encoded WAV audio from API Supports multiple sample rates (8000, 16000, 22050, 24000 Hz) Generates mono audio output Handles HTTP-based synthesis ​ Notes Requires valid Sarvam AI API subscription key Specializes in Indian languages and voices Uses HTTP POST requests for synthesis Thread-safe HTTP session management required Rime XTTS On this page Overview Installation Configuration Constructor Parameters InputParams Configuration Input Output Frames TTSStartedFrame TTSAudioRawFrame TTSStoppedFrame Methods Language Support Voice Models Usage Example Frame Flow Metrics Support Audio Processing Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_sarvam_f35af699.txt b/tts_sarvam_f35af699.txt
new file mode 100644
index 0000000000000000000000000000000000000000..13d1bd6f5011a1b648f47bfdbe1b2533047b1235
--- /dev/null
+++ b/tts_sarvam_f35af699.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/sarvam#ttsstoppedframe
+Title: Sarvam AI - Pipecat
+==================================================
+
+Sarvam AI - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech Sarvam AI Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview SarvamTTSService converts text to speech using Sarvam AI’s TTS API. It specializes in Indian languages and provides extensive voice customization options including pitch, pace, and loudness control. ​ Installation To use SarvamTTSService , no additional dependencies are required. You’ll also need to set up your Sarvam AI API key as an environment variable: SARVAM_API_KEY ​ Configuration ​ Constructor Parameters ​ api_key str required Your Sarvam AI API subscription key ​ voice_id str default: "anushka" Speaker voice identifier (e.g., “anushka”, “meera”, “abhilash”) ​ model str default: "bulbul:v2" TTS model to use (“bulbul:v1” or “bulbul:v2”) ​ aiohttp_session aiohttp.ClientSession required Shared aiohttp session for making HTTP requests ​ base_url str default: "https://api.sarvam.ai" Sarvam AI API base URL ​ sample_rate int default: "None" Audio sample rate in Hz (8000, 16000, 22050, 24000) ​ params InputParams default: "None" Additional voice and preprocessing parameters ​ InputParams Configuration ​ language Language default: "Language.HI" Target language for synthesis ​ pitch float default: "0.0" Voice pitch adjustment (-0.75 to 0.75) ​ pace float default: "1.0" Speech speed (0.3 to 3.0) ​ loudness float default: "1.0" Audio volume (0.1 to 3.0) ​ enable_preprocessing bool default: "False" Enable text normalization for mixed-language content ​ Input The service accepts text input through its TTS pipeline with automatic WAV header stripping for clean PCM output. ​ Output Frames ​ TTSStartedFrame Signals the start of audio generation. ​ TTSAudioRawFrame Contains generated audio data: ​ audio bytes Raw PCM audio data (WAV header stripped) ​ sample_rate int Audio sample rate (22050Hz default) ​ num_channels int Number of audio channels (1 for mono) ​ TTSStoppedFrame Signals the completion of audio generation. ​ Methods See the TTS base class methods for additional functionality. ​ Language Support Sarvam AI TTS supports the following Indian languages: Language Code Description Service Code Language.BN Bengali bn-IN Language.EN English (India) en-IN Language.GU Gujarati gu-IN Language.HI Hindi hi-IN Language.KN Kannada kn-IN Language.ML Malayalam ml-IN Language.MR Marathi mr-IN Language.OR Odia od-IN Language.PA Punjabi pa-IN Language.TA Tamil ta-IN Language.TE Telugu te-IN ​ Voice Models See the Sarvam docs for the latest information on available voices and models. ​ Usage Example Copy Ask AI from pipecat.services.sarvam.tts import SarvamTTSService from pipecat.transcriptions.language import Language import aiohttp # Configure service async with aiohttp.ClientSession() as session: tts = SarvamTTSService( api_key = "your-api-key" , voice_id = "anushka" , model = "bulbul:v2" , aiohttp_session = session, params = SarvamTTSService.InputParams( language = Language. HI , ) ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ Frame Flow ​ Metrics Support The service supports metrics collection: Time to First Byte (TTFB) TTS usage metrics Processing duration ​ Audio Processing Returns base64-encoded WAV audio from API Supports multiple sample rates (8000, 16000, 22050, 24000 Hz) Generates mono audio output Handles HTTP-based synthesis ​ Notes Requires valid Sarvam AI API subscription key Specializes in Indian languages and voices Uses HTTP POST requests for synthesis Thread-safe HTTP session management required Rime XTTS On this page Overview Installation Configuration Constructor Parameters InputParams Configuration Input Output Frames TTSStartedFrame TTSAudioRawFrame TTSStoppedFrame Methods Language Support Voice Models Usage Example Frame Flow Metrics Support Audio Processing Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_xtts_2ec823aa.txt b/tts_xtts_2ec823aa.txt
new file mode 100644
index 0000000000000000000000000000000000000000..f8b73cbf5f6108db9eedccb47303004da6083518
--- /dev/null
+++ b/tts_xtts_2ec823aa.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/xtts#param-aiohttp-session
+Title: XTTS - Pipecat
+==================================================
+
+XTTS - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech XTTS Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline Coqui, the XTTS maintainer, has shut down. XTTS may not receive future updates or support. ​ Overview XTTSService provides text-to-speech capabilities using Coqui’s XTTS (Cross-lingual Text-to-Speech) model through a streaming server. It supports multiple languages and custom voice cloning. ​ Installation The service requires a running XTTS streaming server. You can start one using Docker: Copy Ask AI docker run --gpus=all -e COQUI_TOS_AGREED= 1 --rm -p 8000:80 ghcr.io/coqui-ai/xtts-streaming-server:latest-cuda121 For more information, visit the official repository . ​ Configuration ​ Constructor Parameters ​ voice_id str required Voice identifier from studio speakers ​ language Language required Language for speech synthesis ​ base_url str required XTTS streaming server URL ​ aiohttp_session aiohttp.ClientSession required HTTP client session for API requests ​ sample_rate int default: "None" Output audio sample rate in Hz ​ text_filter BaseTextFilter default: "None" Modifies text provided to the TTS. Learn more about the available filters. ​ Output Frames ​ Control Frames ​ TTSStartedFrame Frame Signals start of speech synthesis ​ TTSStoppedFrame Frame Signals completion of speech synthesis ​ Audio Frames ​ TTSAudioRawFrame Frame Contains generated audio data with: - PCM audio format - Specified sample rate (resampled from 24kHz) - Single channel (mono) ​ Error Frames ​ ErrorFrame Frame Contains XTTS server error information ​ Methods See the TTS base class methods for additional functionality. ​ Language Support Supports multiple languages: Language Code Description Service Code Language.CS Czech cs Language.DE German de Language.EN English en Language.ES Spanish es Language.FR French fr Language.HI Hindi hi Language.HU Hungarian hu Language.IT Italian it Language.JA Japanese ja Language.KO Korean ko Language.NL Dutch nl Language.PL Polish pl Language.PT Portuguese pt Language.RU Russian ru Language.TR Turkish tr Language.ZH Chinese (Simplified) zh-cn ​ Usage Example Copy Ask AI from pipecat.services.xtts.tts import XTTSService from pipecat.transcriptions.language import Language import aiohttp # Configure service async with aiohttp.ClientSession() as session: tts = XTTSService( voice_id = "speaker_1" , language = Language. EN , base_url = "http://localhost:8000" , aiohttp_session = session ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ Streaming Process The service handles audio streaming in chunks: Receives audio chunks from XTTS server Buffers chunks for processing Resamples audio to desired sample rate Delivers audio frames in real-time Copy Ask AI # Streaming configuration payload = { "text" : text, "language" : language_code, "speaker_embedding" : embeddings, "gpt_cond_latent" : latent_data, "add_wav_header" : False , "stream_chunk_size" : 20 } ​ Frame Flow ​ Metrics Support The service collects processing metrics: Time to First Byte (TTFB) Processing duration Character usage Streaming performance ​ Notes Requires GPU for optimal performance Supports real-time streaming Automatic audio resampling Buffer management for smooth playback Thread-safe processing Automatic error handling Manages server connection lifecycle Text preprocessing (removes periods and asterisks) Sarvam AI AWS Nova Sonic On this page Overview Installation Configuration Constructor Parameters Output Frames Control Frames Audio Frames Error Frames Methods Language Support Usage Example Streaming Process Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_xtts_5eb117e0.txt b/tts_xtts_5eb117e0.txt
new file mode 100644
index 0000000000000000000000000000000000000000..d53fa4d9ea5d399d48dc19f62fe1f24988fafe91
--- /dev/null
+++ b/tts_xtts_5eb117e0.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/xtts#param-voice-id
+Title: XTTS - Pipecat
+==================================================
+
+XTTS - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech XTTS Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline Coqui, the XTTS maintainer, has shut down. XTTS may not receive future updates or support. ​ Overview XTTSService provides text-to-speech capabilities using Coqui’s XTTS (Cross-lingual Text-to-Speech) model through a streaming server. It supports multiple languages and custom voice cloning. ​ Installation The service requires a running XTTS streaming server. You can start one using Docker: Copy Ask AI docker run --gpus=all -e COQUI_TOS_AGREED= 1 --rm -p 8000:80 ghcr.io/coqui-ai/xtts-streaming-server:latest-cuda121 For more information, visit the official repository . ​ Configuration ​ Constructor Parameters ​ voice_id str required Voice identifier from studio speakers ​ language Language required Language for speech synthesis ​ base_url str required XTTS streaming server URL ​ aiohttp_session aiohttp.ClientSession required HTTP client session for API requests ​ sample_rate int default: "None" Output audio sample rate in Hz ​ text_filter BaseTextFilter default: "None" Modifies text provided to the TTS. Learn more about the available filters. ​ Output Frames ​ Control Frames ​ TTSStartedFrame Frame Signals start of speech synthesis ​ TTSStoppedFrame Frame Signals completion of speech synthesis ​ Audio Frames ​ TTSAudioRawFrame Frame Contains generated audio data with: - PCM audio format - Specified sample rate (resampled from 24kHz) - Single channel (mono) ​ Error Frames ​ ErrorFrame Frame Contains XTTS server error information ​ Methods See the TTS base class methods for additional functionality. ​ Language Support Supports multiple languages: Language Code Description Service Code Language.CS Czech cs Language.DE German de Language.EN English en Language.ES Spanish es Language.FR French fr Language.HI Hindi hi Language.HU Hungarian hu Language.IT Italian it Language.JA Japanese ja Language.KO Korean ko Language.NL Dutch nl Language.PL Polish pl Language.PT Portuguese pt Language.RU Russian ru Language.TR Turkish tr Language.ZH Chinese (Simplified) zh-cn ​ Usage Example Copy Ask AI from pipecat.services.xtts.tts import XTTSService from pipecat.transcriptions.language import Language import aiohttp # Configure service async with aiohttp.ClientSession() as session: tts = XTTSService( voice_id = "speaker_1" , language = Language. EN , base_url = "http://localhost:8000" , aiohttp_session = session ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ Streaming Process The service handles audio streaming in chunks: Receives audio chunks from XTTS server Buffers chunks for processing Resamples audio to desired sample rate Delivers audio frames in real-time Copy Ask AI # Streaming configuration payload = { "text" : text, "language" : language_code, "speaker_embedding" : embeddings, "gpt_cond_latent" : latent_data, "add_wav_header" : False , "stream_chunk_size" : 20 } ​ Frame Flow ​ Metrics Support The service collects processing metrics: Time to First Byte (TTFB) Processing duration Character usage Streaming performance ​ Notes Requires GPU for optimal performance Supports real-time streaming Automatic audio resampling Buffer management for smooth playback Thread-safe processing Automatic error handling Manages server connection lifecycle Text preprocessing (removes periods and asterisks) Sarvam AI AWS Nova Sonic On this page Overview Installation Configuration Constructor Parameters Output Frames Control Frames Audio Frames Error Frames Methods Language Support Usage Example Streaming Process Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_xtts_6f706f00.txt b/tts_xtts_6f706f00.txt
new file mode 100644
index 0000000000000000000000000000000000000000..a5bd4553b22fbb97eca053d0a64336f4b66c806e
--- /dev/null
+++ b/tts_xtts_6f706f00.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/xtts#frame-flow
+Title: XTTS - Pipecat
+==================================================
+
+XTTS - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech XTTS Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline Coqui, the XTTS maintainer, has shut down. XTTS may not receive future updates or support. ​ Overview XTTSService provides text-to-speech capabilities using Coqui’s XTTS (Cross-lingual Text-to-Speech) model through a streaming server. It supports multiple languages and custom voice cloning. ​ Installation The service requires a running XTTS streaming server. You can start one using Docker: Copy Ask AI docker run --gpus=all -e COQUI_TOS_AGREED= 1 --rm -p 8000:80 ghcr.io/coqui-ai/xtts-streaming-server:latest-cuda121 For more information, visit the official repository . ​ Configuration ​ Constructor Parameters ​ voice_id str required Voice identifier from studio speakers ​ language Language required Language for speech synthesis ​ base_url str required XTTS streaming server URL ​ aiohttp_session aiohttp.ClientSession required HTTP client session for API requests ​ sample_rate int default: "None" Output audio sample rate in Hz ​ text_filter BaseTextFilter default: "None" Modifies text provided to the TTS. Learn more about the available filters. ​ Output Frames ​ Control Frames ​ TTSStartedFrame Frame Signals start of speech synthesis ​ TTSStoppedFrame Frame Signals completion of speech synthesis ​ Audio Frames ​ TTSAudioRawFrame Frame Contains generated audio data with: - PCM audio format - Specified sample rate (resampled from 24kHz) - Single channel (mono) ​ Error Frames ​ ErrorFrame Frame Contains XTTS server error information ​ Methods See the TTS base class methods for additional functionality. ​ Language Support Supports multiple languages: Language Code Description Service Code Language.CS Czech cs Language.DE German de Language.EN English en Language.ES Spanish es Language.FR French fr Language.HI Hindi hi Language.HU Hungarian hu Language.IT Italian it Language.JA Japanese ja Language.KO Korean ko Language.NL Dutch nl Language.PL Polish pl Language.PT Portuguese pt Language.RU Russian ru Language.TR Turkish tr Language.ZH Chinese (Simplified) zh-cn ​ Usage Example Copy Ask AI from pipecat.services.xtts.tts import XTTSService from pipecat.transcriptions.language import Language import aiohttp # Configure service async with aiohttp.ClientSession() as session: tts = XTTSService( voice_id = "speaker_1" , language = Language. EN , base_url = "http://localhost:8000" , aiohttp_session = session ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ Streaming Process The service handles audio streaming in chunks: Receives audio chunks from XTTS server Buffers chunks for processing Resamples audio to desired sample rate Delivers audio frames in real-time Copy Ask AI # Streaming configuration payload = { "text" : text, "language" : language_code, "speaker_embedding" : embeddings, "gpt_cond_latent" : latent_data, "add_wav_header" : False , "stream_chunk_size" : 20 } ​ Frame Flow ​ Metrics Support The service collects processing metrics: Time to First Byte (TTFB) Processing duration Character usage Streaming performance ​ Notes Requires GPU for optimal performance Supports real-time streaming Automatic audio resampling Buffer management for smooth playback Thread-safe processing Automatic error handling Manages server connection lifecycle Text preprocessing (removes periods and asterisks) Sarvam AI AWS Nova Sonic On this page Overview Installation Configuration Constructor Parameters Output Frames Control Frames Audio Frames Error Frames Methods Language Support Usage Example Streaming Process Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_xtts_7ae3caf4.txt b/tts_xtts_7ae3caf4.txt
new file mode 100644
index 0000000000000000000000000000000000000000..f3a7972db45b7ddef3fc497249cf6dad70f31e4b
--- /dev/null
+++ b/tts_xtts_7ae3caf4.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/xtts#configuration
+Title: XTTS - Pipecat
+==================================================
+
+XTTS - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech XTTS Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline Coqui, the XTTS maintainer, has shut down. XTTS may not receive future updates or support. ​ Overview XTTSService provides text-to-speech capabilities using Coqui’s XTTS (Cross-lingual Text-to-Speech) model through a streaming server. It supports multiple languages and custom voice cloning. ​ Installation The service requires a running XTTS streaming server. You can start one using Docker: Copy Ask AI docker run --gpus=all -e COQUI_TOS_AGREED= 1 --rm -p 8000:80 ghcr.io/coqui-ai/xtts-streaming-server:latest-cuda121 For more information, visit the official repository . ​ Configuration ​ Constructor Parameters ​ voice_id str required Voice identifier from studio speakers ​ language Language required Language for speech synthesis ​ base_url str required XTTS streaming server URL ​ aiohttp_session aiohttp.ClientSession required HTTP client session for API requests ​ sample_rate int default: "None" Output audio sample rate in Hz ​ text_filter BaseTextFilter default: "None" Modifies text provided to the TTS. Learn more about the available filters. ​ Output Frames ​ Control Frames ​ TTSStartedFrame Frame Signals start of speech synthesis ​ TTSStoppedFrame Frame Signals completion of speech synthesis ​ Audio Frames ​ TTSAudioRawFrame Frame Contains generated audio data with: - PCM audio format - Specified sample rate (resampled from 24kHz) - Single channel (mono) ​ Error Frames ​ ErrorFrame Frame Contains XTTS server error information ​ Methods See the TTS base class methods for additional functionality. ​ Language Support Supports multiple languages: Language Code Description Service Code Language.CS Czech cs Language.DE German de Language.EN English en Language.ES Spanish es Language.FR French fr Language.HI Hindi hi Language.HU Hungarian hu Language.IT Italian it Language.JA Japanese ja Language.KO Korean ko Language.NL Dutch nl Language.PL Polish pl Language.PT Portuguese pt Language.RU Russian ru Language.TR Turkish tr Language.ZH Chinese (Simplified) zh-cn ​ Usage Example Copy Ask AI from pipecat.services.xtts.tts import XTTSService from pipecat.transcriptions.language import Language import aiohttp # Configure service async with aiohttp.ClientSession() as session: tts = XTTSService( voice_id = "speaker_1" , language = Language. EN , base_url = "http://localhost:8000" , aiohttp_session = session ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ Streaming Process The service handles audio streaming in chunks: Receives audio chunks from XTTS server Buffers chunks for processing Resamples audio to desired sample rate Delivers audio frames in real-time Copy Ask AI # Streaming configuration payload = { "text" : text, "language" : language_code, "speaker_embedding" : embeddings, "gpt_cond_latent" : latent_data, "add_wav_header" : False , "stream_chunk_size" : 20 } ​ Frame Flow ​ Metrics Support The service collects processing metrics: Time to First Byte (TTFB) Processing duration Character usage Streaming performance ​ Notes Requires GPU for optimal performance Supports real-time streaming Automatic audio resampling Buffer management for smooth playback Thread-safe processing Automatic error handling Manages server connection lifecycle Text preprocessing (removes periods and asterisks) Sarvam AI AWS Nova Sonic On this page Overview Installation Configuration Constructor Parameters Output Frames Control Frames Audio Frames Error Frames Methods Language Support Usage Example Streaming Process Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_xtts_90eb7fa8.txt b/tts_xtts_90eb7fa8.txt
new file mode 100644
index 0000000000000000000000000000000000000000..d7fe7c7e063895c671b052753d93867a8dee1ce9
--- /dev/null
+++ b/tts_xtts_90eb7fa8.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/xtts#usage-example
+Title: XTTS - Pipecat
+==================================================
+
+XTTS - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech XTTS Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline Coqui, the XTTS maintainer, has shut down. XTTS may not receive future updates or support. ​ Overview XTTSService provides text-to-speech capabilities using Coqui’s XTTS (Cross-lingual Text-to-Speech) model through a streaming server. It supports multiple languages and custom voice cloning. ​ Installation The service requires a running XTTS streaming server. You can start one using Docker: Copy Ask AI docker run --gpus=all -e COQUI_TOS_AGREED= 1 --rm -p 8000:80 ghcr.io/coqui-ai/xtts-streaming-server:latest-cuda121 For more information, visit the official repository . ​ Configuration ​ Constructor Parameters ​ voice_id str required Voice identifier from studio speakers ​ language Language required Language for speech synthesis ​ base_url str required XTTS streaming server URL ​ aiohttp_session aiohttp.ClientSession required HTTP client session for API requests ​ sample_rate int default: "None" Output audio sample rate in Hz ​ text_filter BaseTextFilter default: "None" Modifies text provided to the TTS. Learn more about the available filters. ​ Output Frames ​ Control Frames ​ TTSStartedFrame Frame Signals start of speech synthesis ​ TTSStoppedFrame Frame Signals completion of speech synthesis ​ Audio Frames ​ TTSAudioRawFrame Frame Contains generated audio data with: - PCM audio format - Specified sample rate (resampled from 24kHz) - Single channel (mono) ​ Error Frames ​ ErrorFrame Frame Contains XTTS server error information ​ Methods See the TTS base class methods for additional functionality. ​ Language Support Supports multiple languages: Language Code Description Service Code Language.CS Czech cs Language.DE German de Language.EN English en Language.ES Spanish es Language.FR French fr Language.HI Hindi hi Language.HU Hungarian hu Language.IT Italian it Language.JA Japanese ja Language.KO Korean ko Language.NL Dutch nl Language.PL Polish pl Language.PT Portuguese pt Language.RU Russian ru Language.TR Turkish tr Language.ZH Chinese (Simplified) zh-cn ​ Usage Example Copy Ask AI from pipecat.services.xtts.tts import XTTSService from pipecat.transcriptions.language import Language import aiohttp # Configure service async with aiohttp.ClientSession() as session: tts = XTTSService( voice_id = "speaker_1" , language = Language. EN , base_url = "http://localhost:8000" , aiohttp_session = session ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ Streaming Process The service handles audio streaming in chunks: Receives audio chunks from XTTS server Buffers chunks for processing Resamples audio to desired sample rate Delivers audio frames in real-time Copy Ask AI # Streaming configuration payload = { "text" : text, "language" : language_code, "speaker_embedding" : embeddings, "gpt_cond_latent" : latent_data, "add_wav_header" : False , "stream_chunk_size" : 20 } ​ Frame Flow ​ Metrics Support The service collects processing metrics: Time to First Byte (TTFB) Processing duration Character usage Streaming performance ​ Notes Requires GPU for optimal performance Supports real-time streaming Automatic audio resampling Buffer management for smooth playback Thread-safe processing Automatic error handling Manages server connection lifecycle Text preprocessing (removes periods and asterisks) Sarvam AI AWS Nova Sonic On this page Overview Installation Configuration Constructor Parameters Output Frames Control Frames Audio Frames Error Frames Methods Language Support Usage Example Streaming Process Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_xtts_991fa1aa.txt b/tts_xtts_991fa1aa.txt
new file mode 100644
index 0000000000000000000000000000000000000000..151e1a93f328e72a94e3a0dc93e091f489f02527
--- /dev/null
+++ b/tts_xtts_991fa1aa.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/xtts#streaming-process
+Title: XTTS - Pipecat
+==================================================
+
+XTTS - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech XTTS Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline Coqui, the XTTS maintainer, has shut down. XTTS may not receive future updates or support. ​ Overview XTTSService provides text-to-speech capabilities using Coqui’s XTTS (Cross-lingual Text-to-Speech) model through a streaming server. It supports multiple languages and custom voice cloning. ​ Installation The service requires a running XTTS streaming server. You can start one using Docker: Copy Ask AI docker run --gpus=all -e COQUI_TOS_AGREED= 1 --rm -p 8000:80 ghcr.io/coqui-ai/xtts-streaming-server:latest-cuda121 For more information, visit the official repository . ​ Configuration ​ Constructor Parameters ​ voice_id str required Voice identifier from studio speakers ​ language Language required Language for speech synthesis ​ base_url str required XTTS streaming server URL ​ aiohttp_session aiohttp.ClientSession required HTTP client session for API requests ​ sample_rate int default: "None" Output audio sample rate in Hz ​ text_filter BaseTextFilter default: "None" Modifies text provided to the TTS. Learn more about the available filters. ​ Output Frames ​ Control Frames ​ TTSStartedFrame Frame Signals start of speech synthesis ​ TTSStoppedFrame Frame Signals completion of speech synthesis ​ Audio Frames ​ TTSAudioRawFrame Frame Contains generated audio data with: - PCM audio format - Specified sample rate (resampled from 24kHz) - Single channel (mono) ​ Error Frames ​ ErrorFrame Frame Contains XTTS server error information ​ Methods See the TTS base class methods for additional functionality. ​ Language Support Supports multiple languages: Language Code Description Service Code Language.CS Czech cs Language.DE German de Language.EN English en Language.ES Spanish es Language.FR French fr Language.HI Hindi hi Language.HU Hungarian hu Language.IT Italian it Language.JA Japanese ja Language.KO Korean ko Language.NL Dutch nl Language.PL Polish pl Language.PT Portuguese pt Language.RU Russian ru Language.TR Turkish tr Language.ZH Chinese (Simplified) zh-cn ​ Usage Example Copy Ask AI from pipecat.services.xtts.tts import XTTSService from pipecat.transcriptions.language import Language import aiohttp # Configure service async with aiohttp.ClientSession() as session: tts = XTTSService( voice_id = "speaker_1" , language = Language. EN , base_url = "http://localhost:8000" , aiohttp_session = session ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ Streaming Process The service handles audio streaming in chunks: Receives audio chunks from XTTS server Buffers chunks for processing Resamples audio to desired sample rate Delivers audio frames in real-time Copy Ask AI # Streaming configuration payload = { "text" : text, "language" : language_code, "speaker_embedding" : embeddings, "gpt_cond_latent" : latent_data, "add_wav_header" : False , "stream_chunk_size" : 20 } ​ Frame Flow ​ Metrics Support The service collects processing metrics: Time to First Byte (TTFB) Processing duration Character usage Streaming performance ​ Notes Requires GPU for optimal performance Supports real-time streaming Automatic audio resampling Buffer management for smooth playback Thread-safe processing Automatic error handling Manages server connection lifecycle Text preprocessing (removes periods and asterisks) Sarvam AI AWS Nova Sonic On this page Overview Installation Configuration Constructor Parameters Output Frames Control Frames Audio Frames Error Frames Methods Language Support Usage Example Streaming Process Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/tts_xtts_aadb795d.txt b/tts_xtts_aadb795d.txt
new file mode 100644
index 0000000000000000000000000000000000000000..b65ae750ae01b25fc0fbb68d61652ac6033d7ef8
--- /dev/null
+++ b/tts_xtts_aadb795d.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/tts/xtts#audio-frames
+Title: XTTS - Pipecat
+==================================================
+
+XTTS - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Text-to-Speech XTTS Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech AWS Polly Azure Cartesia Deepgram ElevenLabs Fish Audio Google Groq LMNT MiniMax Neuphonic NVIDIA Riva OpenAI Piper PlayHT Rime Sarvam AI XTTS Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline Coqui, the XTTS maintainer, has shut down. XTTS may not receive future updates or support. ​ Overview XTTSService provides text-to-speech capabilities using Coqui’s XTTS (Cross-lingual Text-to-Speech) model through a streaming server. It supports multiple languages and custom voice cloning. ​ Installation The service requires a running XTTS streaming server. You can start one using Docker: Copy Ask AI docker run --gpus=all -e COQUI_TOS_AGREED= 1 --rm -p 8000:80 ghcr.io/coqui-ai/xtts-streaming-server:latest-cuda121 For more information, visit the official repository . ​ Configuration ​ Constructor Parameters ​ voice_id str required Voice identifier from studio speakers ​ language Language required Language for speech synthesis ​ base_url str required XTTS streaming server URL ​ aiohttp_session aiohttp.ClientSession required HTTP client session for API requests ​ sample_rate int default: "None" Output audio sample rate in Hz ​ text_filter BaseTextFilter default: "None" Modifies text provided to the TTS. Learn more about the available filters. ​ Output Frames ​ Control Frames ​ TTSStartedFrame Frame Signals start of speech synthesis ​ TTSStoppedFrame Frame Signals completion of speech synthesis ​ Audio Frames ​ TTSAudioRawFrame Frame Contains generated audio data with: - PCM audio format - Specified sample rate (resampled from 24kHz) - Single channel (mono) ​ Error Frames ​ ErrorFrame Frame Contains XTTS server error information ​ Methods See the TTS base class methods for additional functionality. ​ Language Support Supports multiple languages: Language Code Description Service Code Language.CS Czech cs Language.DE German de Language.EN English en Language.ES Spanish es Language.FR French fr Language.HI Hindi hi Language.HU Hungarian hu Language.IT Italian it Language.JA Japanese ja Language.KO Korean ko Language.NL Dutch nl Language.PL Polish pl Language.PT Portuguese pt Language.RU Russian ru Language.TR Turkish tr Language.ZH Chinese (Simplified) zh-cn ​ Usage Example Copy Ask AI from pipecat.services.xtts.tts import XTTSService from pipecat.transcriptions.language import Language import aiohttp # Configure service async with aiohttp.ClientSession() as session: tts = XTTSService( voice_id = "speaker_1" , language = Language. EN , base_url = "http://localhost:8000" , aiohttp_session = session ) # Use in pipeline pipeline = Pipeline([ ... , llm, tts, transport.output(), ]) ​ Streaming Process The service handles audio streaming in chunks: Receives audio chunks from XTTS server Buffers chunks for processing Resamples audio to desired sample rate Delivers audio frames in real-time Copy Ask AI # Streaming configuration payload = { "text" : text, "language" : language_code, "speaker_embedding" : embeddings, "gpt_cond_latent" : latent_data, "add_wav_header" : False , "stream_chunk_size" : 20 } ​ Frame Flow ​ Metrics Support The service collects processing metrics: Time to First Byte (TTFB) Processing duration Character usage Streaming performance ​ Notes Requires GPU for optimal performance Supports real-time streaming Automatic audio resampling Buffer management for smooth playback Thread-safe processing Automatic error handling Manages server connection lifecycle Text preprocessing (removes periods and asterisks) Sarvam AI AWS Nova Sonic On this page Overview Installation Configuration Constructor Parameters Output Frames Control Frames Audio Frames Error Frames Methods Language Support Usage Example Streaming Process Frame Flow Metrics Support Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/utilities_dtmf-aggregator_144925a7.txt b/utilities_dtmf-aggregator_144925a7.txt
new file mode 100644
index 0000000000000000000000000000000000000000..43beb5054e864a01e6f0cf71b6603b4ab56d7ce5
--- /dev/null
+++ b/utilities_dtmf-aggregator_144925a7.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/dtmf-aggregator#param-termination-digit
+Title: DTMFAggregator - Pipecat
+==================================================
+
+DTMFAggregator - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Telephony DTMFAggregator Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony DTMFAggregator Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview DTMFAggregator processes incoming DTMF (Dual-Tone Multi-Frequency) frames from phone keypad input and aggregates them into complete sequences that can be understood by LLM services. It buffers individual digit presses and flushes them as transcription frames when a termination digit is pressed, a timeout occurs, or an interruption happens. This aggregator is essential for telephony applications where users interact via phone keypad buttons, converting raw DTMF input into structured text that LLMs can process alongside voice transcriptions. ​ Constructor Copy Ask AI aggregator = DTMFAggregator( timeout = 2.0 , termination_digit = KeypadEntry. POUND , prefix = "DTMF: " ) ​ timeout float default: "2.0" Idle timeout in seconds before flushing the aggregated digits ​ termination_digit KeypadEntry default: "KeypadEntry.POUND" Digit that triggers immediate flush of the aggregation ​ prefix str default: "DTMF: " Prefix added to DTMF sequence in the output transcription ​ Input Frames ​ InputDTMFFrame Frame Contains a single keypad button press with a KeypadEntry value ​ StartInterruptionFrame Frame Flushes any pending aggregation when user interruption begins ​ EndFrame Frame Flushes pending aggregation and stops the aggregation task ​ Output Frames ​ TranscriptionFrame Frame Contains the aggregated DTMF sequence as text with the configured prefix All input frames are passed through downstream, including the original InputDTMFFrame instances. ​ Keypad Entries The aggregator processes these standard phone keypad entries: KeypadEntry Value Description ZERO through NINE "0" - "9" Numeric digits STAR "*" Star/asterisk key POUND "#" Pound/hash key ​ Aggregation Behavior The aggregator flushes (emits a TranscriptionFrame) when: Termination digit : The configured termination digit is pressed (default: # ) Timeout : No new digits received within the timeout period (default: 2 seconds) Interruption : A StartInterruptionFrame is received Pipeline end : An EndFrame is received ​ Usage Examples ​ Basic Telephony Integration Copy Ask AI from pipecat.processors.aggregators.dtmf_aggregator import DTMFAggregator from pipecat.serializers.twilio import TwilioFrameSerializer # Create DTMF aggregator with default settings dtmf_aggregator = DTMFAggregator() # Set up Twilio serializer for phone integration serializer = TwilioFrameSerializer( stream_sid = stream_sid, call_sid = call_sid, account_sid = os.getenv( "TWILIO_ACCOUNT_SID" ), auth_token = os.getenv( "TWILIO_AUTH_TOKEN" ) ) # Create pipeline with DTMF processing pipeline = Pipeline([ transport.input(), # Websocket input from Twilio dtmf_aggregator, # Process DTMF before STT stt, # Speech-to-text service context_aggregator.user(), llm, # LLM processes both voice and DTMF tts, # Text-to-speech transport.output(), context_aggregator.assistant(), ]) ​ Custom Configuration for Menu Systems Copy Ask AI # Configure for menu system with star termination menu_dtmf = DTMFAggregator( timeout = 5.0 , # Longer timeout for menu selection termination_digit = KeypadEntry. STAR , # Use * to confirm selection prefix = "Menu selection: " # Clear prefix for LLM ) # Update system prompt to handle DTMF input messages = [ { "role" : "system" , "content" : """You are a phone menu assistant. When you receive input starting with "Menu selection:", this represents button presses on the phone keypad: - Single digits (1-9): Menu options - 0: Often "speak to operator" - *: Confirmation or "go back" - #: Usually "repeat menu" Respond appropriately to both voice and keypad input.""" } ] ​ Sequence Examples User Input Aggregation Trigger Output TranscriptionFrame 1 , 2 , 3 , # Termination digit "DTMF: 123#" * , 0 2-second timeout "DTMF: *0" 5 , interruption StartInterruptionFrame "DTMF: 5" 9 , 9 , EndFrame Pipeline shutdown "DTMF: 99" ​ Frame Flow ​ Error Handling The aggregator gracefully handles: Invalid DTMF digits (logged and ignored) Pipeline interruptions (flushes pending sequences) Rapid key presses (buffers efficiently) Mixed voice and DTMF input (processes independently) ​ Best Practices System Prompt Design : Train your LLM to recognize and respond to DTMF prefixed input Timeout Configuration : Use shorter timeouts (1-2s) for rapid entry, longer (3-5s) for menu selection Termination Strategy : Use # for confirmation, * for cancel/back operations Pipeline Placement : Always place before the user context aggregator to ensure proper frame ordering Watchdog Timers MarkdownTextFilter On this page Overview Constructor Input Frames Output Frames Keypad Entries Aggregation Behavior Usage Examples Basic Telephony Integration Custom Configuration for Menu Systems Sequence Examples Frame Flow Error Handling Best Practices Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/utilities_dtmf-aggregator_34f1dfc3.txt b/utilities_dtmf-aggregator_34f1dfc3.txt
new file mode 100644
index 0000000000000000000000000000000000000000..2fc4f6127d3f505c934551b7e71880c3d594a277
--- /dev/null
+++ b/utilities_dtmf-aggregator_34f1dfc3.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/dtmf-aggregator#param-timeout
+Title: DTMFAggregator - Pipecat
+==================================================
+
+DTMFAggregator - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Telephony DTMFAggregator Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony DTMFAggregator Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview DTMFAggregator processes incoming DTMF (Dual-Tone Multi-Frequency) frames from phone keypad input and aggregates them into complete sequences that can be understood by LLM services. It buffers individual digit presses and flushes them as transcription frames when a termination digit is pressed, a timeout occurs, or an interruption happens. This aggregator is essential for telephony applications where users interact via phone keypad buttons, converting raw DTMF input into structured text that LLMs can process alongside voice transcriptions. ​ Constructor Copy Ask AI aggregator = DTMFAggregator( timeout = 2.0 , termination_digit = KeypadEntry. POUND , prefix = "DTMF: " ) ​ timeout float default: "2.0" Idle timeout in seconds before flushing the aggregated digits ​ termination_digit KeypadEntry default: "KeypadEntry.POUND" Digit that triggers immediate flush of the aggregation ​ prefix str default: "DTMF: " Prefix added to DTMF sequence in the output transcription ​ Input Frames ​ InputDTMFFrame Frame Contains a single keypad button press with a KeypadEntry value ​ StartInterruptionFrame Frame Flushes any pending aggregation when user interruption begins ​ EndFrame Frame Flushes pending aggregation and stops the aggregation task ​ Output Frames ​ TranscriptionFrame Frame Contains the aggregated DTMF sequence as text with the configured prefix All input frames are passed through downstream, including the original InputDTMFFrame instances. ​ Keypad Entries The aggregator processes these standard phone keypad entries: KeypadEntry Value Description ZERO through NINE "0" - "9" Numeric digits STAR "*" Star/asterisk key POUND "#" Pound/hash key ​ Aggregation Behavior The aggregator flushes (emits a TranscriptionFrame) when: Termination digit : The configured termination digit is pressed (default: # ) Timeout : No new digits received within the timeout period (default: 2 seconds) Interruption : A StartInterruptionFrame is received Pipeline end : An EndFrame is received ​ Usage Examples ​ Basic Telephony Integration Copy Ask AI from pipecat.processors.aggregators.dtmf_aggregator import DTMFAggregator from pipecat.serializers.twilio import TwilioFrameSerializer # Create DTMF aggregator with default settings dtmf_aggregator = DTMFAggregator() # Set up Twilio serializer for phone integration serializer = TwilioFrameSerializer( stream_sid = stream_sid, call_sid = call_sid, account_sid = os.getenv( "TWILIO_ACCOUNT_SID" ), auth_token = os.getenv( "TWILIO_AUTH_TOKEN" ) ) # Create pipeline with DTMF processing pipeline = Pipeline([ transport.input(), # Websocket input from Twilio dtmf_aggregator, # Process DTMF before STT stt, # Speech-to-text service context_aggregator.user(), llm, # LLM processes both voice and DTMF tts, # Text-to-speech transport.output(), context_aggregator.assistant(), ]) ​ Custom Configuration for Menu Systems Copy Ask AI # Configure for menu system with star termination menu_dtmf = DTMFAggregator( timeout = 5.0 , # Longer timeout for menu selection termination_digit = KeypadEntry. STAR , # Use * to confirm selection prefix = "Menu selection: " # Clear prefix for LLM ) # Update system prompt to handle DTMF input messages = [ { "role" : "system" , "content" : """You are a phone menu assistant. When you receive input starting with "Menu selection:", this represents button presses on the phone keypad: - Single digits (1-9): Menu options - 0: Often "speak to operator" - *: Confirmation or "go back" - #: Usually "repeat menu" Respond appropriately to both voice and keypad input.""" } ] ​ Sequence Examples User Input Aggregation Trigger Output TranscriptionFrame 1 , 2 , 3 , # Termination digit "DTMF: 123#" * , 0 2-second timeout "DTMF: *0" 5 , interruption StartInterruptionFrame "DTMF: 5" 9 , 9 , EndFrame Pipeline shutdown "DTMF: 99" ​ Frame Flow ​ Error Handling The aggregator gracefully handles: Invalid DTMF digits (logged and ignored) Pipeline interruptions (flushes pending sequences) Rapid key presses (buffers efficiently) Mixed voice and DTMF input (processes independently) ​ Best Practices System Prompt Design : Train your LLM to recognize and respond to DTMF prefixed input Timeout Configuration : Use shorter timeouts (1-2s) for rapid entry, longer (3-5s) for menu selection Termination Strategy : Use # for confirmation, * for cancel/back operations Pipeline Placement : Always place before the user context aggregator to ensure proper frame ordering Watchdog Timers MarkdownTextFilter On this page Overview Constructor Input Frames Output Frames Keypad Entries Aggregation Behavior Usage Examples Basic Telephony Integration Custom Configuration for Menu Systems Sequence Examples Frame Flow Error Handling Best Practices Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/utilities_dtmf-aggregator_764e2e96.txt b/utilities_dtmf-aggregator_764e2e96.txt
new file mode 100644
index 0000000000000000000000000000000000000000..e96e5a1a1cc41f02e84f6e28f78d60de67cc8171
--- /dev/null
+++ b/utilities_dtmf-aggregator_764e2e96.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/dtmf-aggregator#custom-configuration-for-menu-systems
+Title: DTMFAggregator - Pipecat
+==================================================
+
+DTMFAggregator - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Telephony DTMFAggregator Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony DTMFAggregator Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview DTMFAggregator processes incoming DTMF (Dual-Tone Multi-Frequency) frames from phone keypad input and aggregates them into complete sequences that can be understood by LLM services. It buffers individual digit presses and flushes them as transcription frames when a termination digit is pressed, a timeout occurs, or an interruption happens. This aggregator is essential for telephony applications where users interact via phone keypad buttons, converting raw DTMF input into structured text that LLMs can process alongside voice transcriptions. ​ Constructor Copy Ask AI aggregator = DTMFAggregator( timeout = 2.0 , termination_digit = KeypadEntry. POUND , prefix = "DTMF: " ) ​ timeout float default: "2.0" Idle timeout in seconds before flushing the aggregated digits ​ termination_digit KeypadEntry default: "KeypadEntry.POUND" Digit that triggers immediate flush of the aggregation ​ prefix str default: "DTMF: " Prefix added to DTMF sequence in the output transcription ​ Input Frames ​ InputDTMFFrame Frame Contains a single keypad button press with a KeypadEntry value ​ StartInterruptionFrame Frame Flushes any pending aggregation when user interruption begins ​ EndFrame Frame Flushes pending aggregation and stops the aggregation task ​ Output Frames ​ TranscriptionFrame Frame Contains the aggregated DTMF sequence as text with the configured prefix All input frames are passed through downstream, including the original InputDTMFFrame instances. ​ Keypad Entries The aggregator processes these standard phone keypad entries: KeypadEntry Value Description ZERO through NINE "0" - "9" Numeric digits STAR "*" Star/asterisk key POUND "#" Pound/hash key ​ Aggregation Behavior The aggregator flushes (emits a TranscriptionFrame) when: Termination digit : The configured termination digit is pressed (default: # ) Timeout : No new digits received within the timeout period (default: 2 seconds) Interruption : A StartInterruptionFrame is received Pipeline end : An EndFrame is received ​ Usage Examples ​ Basic Telephony Integration Copy Ask AI from pipecat.processors.aggregators.dtmf_aggregator import DTMFAggregator from pipecat.serializers.twilio import TwilioFrameSerializer # Create DTMF aggregator with default settings dtmf_aggregator = DTMFAggregator() # Set up Twilio serializer for phone integration serializer = TwilioFrameSerializer( stream_sid = stream_sid, call_sid = call_sid, account_sid = os.getenv( "TWILIO_ACCOUNT_SID" ), auth_token = os.getenv( "TWILIO_AUTH_TOKEN" ) ) # Create pipeline with DTMF processing pipeline = Pipeline([ transport.input(), # Websocket input from Twilio dtmf_aggregator, # Process DTMF before STT stt, # Speech-to-text service context_aggregator.user(), llm, # LLM processes both voice and DTMF tts, # Text-to-speech transport.output(), context_aggregator.assistant(), ]) ​ Custom Configuration for Menu Systems Copy Ask AI # Configure for menu system with star termination menu_dtmf = DTMFAggregator( timeout = 5.0 , # Longer timeout for menu selection termination_digit = KeypadEntry. STAR , # Use * to confirm selection prefix = "Menu selection: " # Clear prefix for LLM ) # Update system prompt to handle DTMF input messages = [ { "role" : "system" , "content" : """You are a phone menu assistant. When you receive input starting with "Menu selection:", this represents button presses on the phone keypad: - Single digits (1-9): Menu options - 0: Often "speak to operator" - *: Confirmation or "go back" - #: Usually "repeat menu" Respond appropriately to both voice and keypad input.""" } ] ​ Sequence Examples User Input Aggregation Trigger Output TranscriptionFrame 1 , 2 , 3 , # Termination digit "DTMF: 123#" * , 0 2-second timeout "DTMF: *0" 5 , interruption StartInterruptionFrame "DTMF: 5" 9 , 9 , EndFrame Pipeline shutdown "DTMF: 99" ​ Frame Flow ​ Error Handling The aggregator gracefully handles: Invalid DTMF digits (logged and ignored) Pipeline interruptions (flushes pending sequences) Rapid key presses (buffers efficiently) Mixed voice and DTMF input (processes independently) ​ Best Practices System Prompt Design : Train your LLM to recognize and respond to DTMF prefixed input Timeout Configuration : Use shorter timeouts (1-2s) for rapid entry, longer (3-5s) for menu selection Termination Strategy : Use # for confirmation, * for cancel/back operations Pipeline Placement : Always place before the user context aggregator to ensure proper frame ordering Watchdog Timers MarkdownTextFilter On this page Overview Constructor Input Frames Output Frames Keypad Entries Aggregation Behavior Usage Examples Basic Telephony Integration Custom Configuration for Menu Systems Sequence Examples Frame Flow Error Handling Best Practices Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/utilities_dtmf-aggregator_be1d9b8d.txt b/utilities_dtmf-aggregator_be1d9b8d.txt
new file mode 100644
index 0000000000000000000000000000000000000000..92444ee47510c1599ed9244d6aa60edb5d0045aa
--- /dev/null
+++ b/utilities_dtmf-aggregator_be1d9b8d.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/dtmf-aggregator#param-end-frame
+Title: DTMFAggregator - Pipecat
+==================================================
+
+DTMFAggregator - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Telephony DTMFAggregator Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony DTMFAggregator Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview DTMFAggregator processes incoming DTMF (Dual-Tone Multi-Frequency) frames from phone keypad input and aggregates them into complete sequences that can be understood by LLM services. It buffers individual digit presses and flushes them as transcription frames when a termination digit is pressed, a timeout occurs, or an interruption happens. This aggregator is essential for telephony applications where users interact via phone keypad buttons, converting raw DTMF input into structured text that LLMs can process alongside voice transcriptions. ​ Constructor Copy Ask AI aggregator = DTMFAggregator( timeout = 2.0 , termination_digit = KeypadEntry. POUND , prefix = "DTMF: " ) ​ timeout float default: "2.0" Idle timeout in seconds before flushing the aggregated digits ​ termination_digit KeypadEntry default: "KeypadEntry.POUND" Digit that triggers immediate flush of the aggregation ​ prefix str default: "DTMF: " Prefix added to DTMF sequence in the output transcription ​ Input Frames ​ InputDTMFFrame Frame Contains a single keypad button press with a KeypadEntry value ​ StartInterruptionFrame Frame Flushes any pending aggregation when user interruption begins ​ EndFrame Frame Flushes pending aggregation and stops the aggregation task ​ Output Frames ​ TranscriptionFrame Frame Contains the aggregated DTMF sequence as text with the configured prefix All input frames are passed through downstream, including the original InputDTMFFrame instances. ​ Keypad Entries The aggregator processes these standard phone keypad entries: KeypadEntry Value Description ZERO through NINE "0" - "9" Numeric digits STAR "*" Star/asterisk key POUND "#" Pound/hash key ​ Aggregation Behavior The aggregator flushes (emits a TranscriptionFrame) when: Termination digit : The configured termination digit is pressed (default: # ) Timeout : No new digits received within the timeout period (default: 2 seconds) Interruption : A StartInterruptionFrame is received Pipeline end : An EndFrame is received ​ Usage Examples ​ Basic Telephony Integration Copy Ask AI from pipecat.processors.aggregators.dtmf_aggregator import DTMFAggregator from pipecat.serializers.twilio import TwilioFrameSerializer # Create DTMF aggregator with default settings dtmf_aggregator = DTMFAggregator() # Set up Twilio serializer for phone integration serializer = TwilioFrameSerializer( stream_sid = stream_sid, call_sid = call_sid, account_sid = os.getenv( "TWILIO_ACCOUNT_SID" ), auth_token = os.getenv( "TWILIO_AUTH_TOKEN" ) ) # Create pipeline with DTMF processing pipeline = Pipeline([ transport.input(), # Websocket input from Twilio dtmf_aggregator, # Process DTMF before STT stt, # Speech-to-text service context_aggregator.user(), llm, # LLM processes both voice and DTMF tts, # Text-to-speech transport.output(), context_aggregator.assistant(), ]) ​ Custom Configuration for Menu Systems Copy Ask AI # Configure for menu system with star termination menu_dtmf = DTMFAggregator( timeout = 5.0 , # Longer timeout for menu selection termination_digit = KeypadEntry. STAR , # Use * to confirm selection prefix = "Menu selection: " # Clear prefix for LLM ) # Update system prompt to handle DTMF input messages = [ { "role" : "system" , "content" : """You are a phone menu assistant. When you receive input starting with "Menu selection:", this represents button presses on the phone keypad: - Single digits (1-9): Menu options - 0: Often "speak to operator" - *: Confirmation or "go back" - #: Usually "repeat menu" Respond appropriately to both voice and keypad input.""" } ] ​ Sequence Examples User Input Aggregation Trigger Output TranscriptionFrame 1 , 2 , 3 , # Termination digit "DTMF: 123#" * , 0 2-second timeout "DTMF: *0" 5 , interruption StartInterruptionFrame "DTMF: 5" 9 , 9 , EndFrame Pipeline shutdown "DTMF: 99" ​ Frame Flow ​ Error Handling The aggregator gracefully handles: Invalid DTMF digits (logged and ignored) Pipeline interruptions (flushes pending sequences) Rapid key presses (buffers efficiently) Mixed voice and DTMF input (processes independently) ​ Best Practices System Prompt Design : Train your LLM to recognize and respond to DTMF prefixed input Timeout Configuration : Use shorter timeouts (1-2s) for rapid entry, longer (3-5s) for menu selection Termination Strategy : Use # for confirmation, * for cancel/back operations Pipeline Placement : Always place before the user context aggregator to ensure proper frame ordering Watchdog Timers MarkdownTextFilter On this page Overview Constructor Input Frames Output Frames Keypad Entries Aggregation Behavior Usage Examples Basic Telephony Integration Custom Configuration for Menu Systems Sequence Examples Frame Flow Error Handling Best Practices Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/utilities_dtmf-aggregator_e5007f18.txt b/utilities_dtmf-aggregator_e5007f18.txt
new file mode 100644
index 0000000000000000000000000000000000000000..4f55f965f97d358a2768ae23c420d984a6b861ab
--- /dev/null
+++ b/utilities_dtmf-aggregator_e5007f18.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/dtmf-aggregator#param-prefix
+Title: DTMFAggregator - Pipecat
+==================================================
+
+DTMFAggregator - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Telephony DTMFAggregator Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony DTMFAggregator Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview DTMFAggregator processes incoming DTMF (Dual-Tone Multi-Frequency) frames from phone keypad input and aggregates them into complete sequences that can be understood by LLM services. It buffers individual digit presses and flushes them as transcription frames when a termination digit is pressed, a timeout occurs, or an interruption happens. This aggregator is essential for telephony applications where users interact via phone keypad buttons, converting raw DTMF input into structured text that LLMs can process alongside voice transcriptions. ​ Constructor Copy Ask AI aggregator = DTMFAggregator( timeout = 2.0 , termination_digit = KeypadEntry. POUND , prefix = "DTMF: " ) ​ timeout float default: "2.0" Idle timeout in seconds before flushing the aggregated digits ​ termination_digit KeypadEntry default: "KeypadEntry.POUND" Digit that triggers immediate flush of the aggregation ​ prefix str default: "DTMF: " Prefix added to DTMF sequence in the output transcription ​ Input Frames ​ InputDTMFFrame Frame Contains a single keypad button press with a KeypadEntry value ​ StartInterruptionFrame Frame Flushes any pending aggregation when user interruption begins ​ EndFrame Frame Flushes pending aggregation and stops the aggregation task ​ Output Frames ​ TranscriptionFrame Frame Contains the aggregated DTMF sequence as text with the configured prefix All input frames are passed through downstream, including the original InputDTMFFrame instances. ​ Keypad Entries The aggregator processes these standard phone keypad entries: KeypadEntry Value Description ZERO through NINE "0" - "9" Numeric digits STAR "*" Star/asterisk key POUND "#" Pound/hash key ​ Aggregation Behavior The aggregator flushes (emits a TranscriptionFrame) when: Termination digit : The configured termination digit is pressed (default: # ) Timeout : No new digits received within the timeout period (default: 2 seconds) Interruption : A StartInterruptionFrame is received Pipeline end : An EndFrame is received ​ Usage Examples ​ Basic Telephony Integration Copy Ask AI from pipecat.processors.aggregators.dtmf_aggregator import DTMFAggregator from pipecat.serializers.twilio import TwilioFrameSerializer # Create DTMF aggregator with default settings dtmf_aggregator = DTMFAggregator() # Set up Twilio serializer for phone integration serializer = TwilioFrameSerializer( stream_sid = stream_sid, call_sid = call_sid, account_sid = os.getenv( "TWILIO_ACCOUNT_SID" ), auth_token = os.getenv( "TWILIO_AUTH_TOKEN" ) ) # Create pipeline with DTMF processing pipeline = Pipeline([ transport.input(), # Websocket input from Twilio dtmf_aggregator, # Process DTMF before STT stt, # Speech-to-text service context_aggregator.user(), llm, # LLM processes both voice and DTMF tts, # Text-to-speech transport.output(), context_aggregator.assistant(), ]) ​ Custom Configuration for Menu Systems Copy Ask AI # Configure for menu system with star termination menu_dtmf = DTMFAggregator( timeout = 5.0 , # Longer timeout for menu selection termination_digit = KeypadEntry. STAR , # Use * to confirm selection prefix = "Menu selection: " # Clear prefix for LLM ) # Update system prompt to handle DTMF input messages = [ { "role" : "system" , "content" : """You are a phone menu assistant. When you receive input starting with "Menu selection:", this represents button presses on the phone keypad: - Single digits (1-9): Menu options - 0: Often "speak to operator" - *: Confirmation or "go back" - #: Usually "repeat menu" Respond appropriately to both voice and keypad input.""" } ] ​ Sequence Examples User Input Aggregation Trigger Output TranscriptionFrame 1 , 2 , 3 , # Termination digit "DTMF: 123#" * , 0 2-second timeout "DTMF: *0" 5 , interruption StartInterruptionFrame "DTMF: 5" 9 , 9 , EndFrame Pipeline shutdown "DTMF: 99" ​ Frame Flow ​ Error Handling The aggregator gracefully handles: Invalid DTMF digits (logged and ignored) Pipeline interruptions (flushes pending sequences) Rapid key presses (buffers efficiently) Mixed voice and DTMF input (processes independently) ​ Best Practices System Prompt Design : Train your LLM to recognize and respond to DTMF prefixed input Timeout Configuration : Use shorter timeouts (1-2s) for rapid entry, longer (3-5s) for menu selection Termination Strategy : Use # for confirmation, * for cancel/back operations Pipeline Placement : Always place before the user context aggregator to ensure proper frame ordering Watchdog Timers MarkdownTextFilter On this page Overview Constructor Input Frames Output Frames Keypad Entries Aggregation Behavior Usage Examples Basic Telephony Integration Custom Configuration for Menu Systems Sequence Examples Frame Flow Error Handling Best Practices Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/utilities_interruption-strategies_1442d9a1.txt b/utilities_interruption-strategies_1442d9a1.txt
new file mode 100644
index 0000000000000000000000000000000000000000..7ccefe7b33b7b8ada0d3a1782eba0b4ecdba7cfa
--- /dev/null
+++ b/utilities_interruption-strategies_1442d9a1.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/interruption-strategies#param-append-audio-audio-sample-rate
+Title: Interruption Strategies - Pipecat
+==================================================
+
+Interruption Strategies - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation User Interruptions Interruption Strategies Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Interruption Strategies Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview Interruption strategies allow you to control when users can interrupt the bot during speech. By default, any user speech immediately interrupts the bot, but this can be problematic when users engage in backchanneling—brief vocal responses like “yeah”, “okay”, or “mm-hmm” that indicate they’re listening without intending to interrupt. With interruption strategies, you can require users to meet specific criteria (such as speaking a minimum number of words or reaching a certain audio volume) before their speech will interrupt the bot, creating a more natural conversation flow. Want to try it out? Check out the interruption strategies foundational demo ​ Configuration Interruption strategies are configured via the interruption_strategies parameter in PipelineParams . When specified, the normal immediate interruption behavior is replaced with conditional interruption based on your criteria. ​ interruption_strategies List[BaseInterruptionStrategy] default: "[]" List of interruption strategies to apply. When multiple strategies are provided, the first one that evaluates to true will trigger the interruption. If empty, normal interruption behavior applies. ​ Base Strategy Interface All interruption strategies inherit from BaseInterruptionStrategy , which provides a common interface for evaluating interruption conditions. ​ append_audio(audio, sample_rate) async method Appends audio data to the strategy for analysis. Not all strategies handle audio. ​ append_text(text) async method Appends text to the strategy for analysis. Not all strategies handle text. ​ should_interrupt() async method Called when the user stops speaking to determine if interruption should occur based on accumulated audio and/or text. ​ reset() async method Resets accumulated text and/or audio data. ​ Available Strategies ​ MinWordsInterruptionStrategy Requires users to speak a minimum number of words before interrupting the bot. ​ min_words int required Minimum number of words the user must speak to interrupt the bot. Must be greater than 0. Copy Ask AI from pipecat.audio.interruptions.min_words_interruption_strategy import MinWordsInterruptionStrategy strategy = MinWordsInterruptionStrategy( min_words = 3 ) ​ How It Works When interruption strategies are configured: Bot not speaking : User speech interrupts immediately (normal behavior) Bot speaking : User speech and audio are collected and fed to strategies User stops speaking : Strategies are evaluated in order First match wins : The first strategy that returns True triggers interruption No matches : User speech is discarded The system automatically handles both audio and text input: Audio frames ( InputAudioRawFrame ) are fed to append_audio() Transcription text is fed to append_text() Strategies can use either or both data types ​ Usage Examples ​ Basic Word Count Interruption Require users to speak at least 3 words to interrupt the bot: Copy Ask AI from pipecat.audio.interruptions.min_words_interruption_strategy import MinWordsInterruptionStrategy from pipecat.pipeline.task import PipelineParams, PipelineTask task = PipelineTask( pipeline, params = PipelineParams( allow_interruptions = True , interruption_strategies = [MinWordsInterruptionStrategy( min_words = 3 )] ) ) ​ Multiple Strategies with Priority Strategies are evaluated in order, with the first match triggering interruption: Copy Ask AI # Prioritize word count, then volume (hypothetical future strategy) task = PipelineTask( pipeline, params = PipelineParams( allow_interruptions = True , interruption_strategies = [ MinWordsInterruptionStrategy( min_words = 2 ), # Check first # VolumeInterruptionStrategy(min_volume=0.8), # Your custom strategy ] ) ) ​ Behavior Comparison Scenario Without Strategy With MinWordsInterruptionStrategy(min_words=3) User says “okay” while bot speaks ✅ Interrupts immediately ❌ Ignored (only 1 word) User says “yes that’s right” while bot speaks ✅ Interrupts immediately ✅ Interrupts (3 words) User speaks while bot is silent ✅ Processed immediately ✅ Processed immediately ​ Notes Interruption strategies only affect behavior when the bot is actively speaking When the bot is not speaking, user input is processed immediately regardless of strategy configuration The allow_interruptions parameter must be True for interruption strategies to work User speech that doesn’t meet interruption criteria is discarded, not queued Strategies are evaluated in order - first match wins Both audio and text data are automatically fed to strategies based on their implementation Word counting uses simple whitespace splitting for word boundaries TranscriptProcessor Overview On this page Overview Configuration Base Strategy Interface Available Strategies MinWordsInterruptionStrategy How It Works Usage Examples Basic Word Count Interruption Multiple Strategies with Priority Behavior Comparison Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/utilities_interruption-strategies_270112a7.txt b/utilities_interruption-strategies_270112a7.txt
new file mode 100644
index 0000000000000000000000000000000000000000..675bb7b827811babc668c292c6bc32634a6fc247
--- /dev/null
+++ b/utilities_interruption-strategies_270112a7.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/interruption-strategies#available-strategies
+Title: Interruption Strategies - Pipecat
+==================================================
+
+Interruption Strategies - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation User Interruptions Interruption Strategies Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Interruption Strategies Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview Interruption strategies allow you to control when users can interrupt the bot during speech. By default, any user speech immediately interrupts the bot, but this can be problematic when users engage in backchanneling—brief vocal responses like “yeah”, “okay”, or “mm-hmm” that indicate they’re listening without intending to interrupt. With interruption strategies, you can require users to meet specific criteria (such as speaking a minimum number of words or reaching a certain audio volume) before their speech will interrupt the bot, creating a more natural conversation flow. Want to try it out? Check out the interruption strategies foundational demo ​ Configuration Interruption strategies are configured via the interruption_strategies parameter in PipelineParams . When specified, the normal immediate interruption behavior is replaced with conditional interruption based on your criteria. ​ interruption_strategies List[BaseInterruptionStrategy] default: "[]" List of interruption strategies to apply. When multiple strategies are provided, the first one that evaluates to true will trigger the interruption. If empty, normal interruption behavior applies. ​ Base Strategy Interface All interruption strategies inherit from BaseInterruptionStrategy , which provides a common interface for evaluating interruption conditions. ​ append_audio(audio, sample_rate) async method Appends audio data to the strategy for analysis. Not all strategies handle audio. ​ append_text(text) async method Appends text to the strategy for analysis. Not all strategies handle text. ​ should_interrupt() async method Called when the user stops speaking to determine if interruption should occur based on accumulated audio and/or text. ​ reset() async method Resets accumulated text and/or audio data. ​ Available Strategies ​ MinWordsInterruptionStrategy Requires users to speak a minimum number of words before interrupting the bot. ​ min_words int required Minimum number of words the user must speak to interrupt the bot. Must be greater than 0. Copy Ask AI from pipecat.audio.interruptions.min_words_interruption_strategy import MinWordsInterruptionStrategy strategy = MinWordsInterruptionStrategy( min_words = 3 ) ​ How It Works When interruption strategies are configured: Bot not speaking : User speech interrupts immediately (normal behavior) Bot speaking : User speech and audio are collected and fed to strategies User stops speaking : Strategies are evaluated in order First match wins : The first strategy that returns True triggers interruption No matches : User speech is discarded The system automatically handles both audio and text input: Audio frames ( InputAudioRawFrame ) are fed to append_audio() Transcription text is fed to append_text() Strategies can use either or both data types ​ Usage Examples ​ Basic Word Count Interruption Require users to speak at least 3 words to interrupt the bot: Copy Ask AI from pipecat.audio.interruptions.min_words_interruption_strategy import MinWordsInterruptionStrategy from pipecat.pipeline.task import PipelineParams, PipelineTask task = PipelineTask( pipeline, params = PipelineParams( allow_interruptions = True , interruption_strategies = [MinWordsInterruptionStrategy( min_words = 3 )] ) ) ​ Multiple Strategies with Priority Strategies are evaluated in order, with the first match triggering interruption: Copy Ask AI # Prioritize word count, then volume (hypothetical future strategy) task = PipelineTask( pipeline, params = PipelineParams( allow_interruptions = True , interruption_strategies = [ MinWordsInterruptionStrategy( min_words = 2 ), # Check first # VolumeInterruptionStrategy(min_volume=0.8), # Your custom strategy ] ) ) ​ Behavior Comparison Scenario Without Strategy With MinWordsInterruptionStrategy(min_words=3) User says “okay” while bot speaks ✅ Interrupts immediately ❌ Ignored (only 1 word) User says “yes that’s right” while bot speaks ✅ Interrupts immediately ✅ Interrupts (3 words) User speaks while bot is silent ✅ Processed immediately ✅ Processed immediately ​ Notes Interruption strategies only affect behavior when the bot is actively speaking When the bot is not speaking, user input is processed immediately regardless of strategy configuration The allow_interruptions parameter must be True for interruption strategies to work User speech that doesn’t meet interruption criteria is discarded, not queued Strategies are evaluated in order - first match wins Both audio and text data are automatically fed to strategies based on their implementation Word counting uses simple whitespace splitting for word boundaries TranscriptProcessor Overview On this page Overview Configuration Base Strategy Interface Available Strategies MinWordsInterruptionStrategy How It Works Usage Examples Basic Word Count Interruption Multiple Strategies with Priority Behavior Comparison Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/utilities_interruption-strategies_4a8ab73c.txt b/utilities_interruption-strategies_4a8ab73c.txt
new file mode 100644
index 0000000000000000000000000000000000000000..ac53d7f888ead86013d309d2abb004bde7533279
--- /dev/null
+++ b/utilities_interruption-strategies_4a8ab73c.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/interruption-strategies#multiple-strategies-with-priority
+Title: Interruption Strategies - Pipecat
+==================================================
+
+Interruption Strategies - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation User Interruptions Interruption Strategies Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Interruption Strategies Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview Interruption strategies allow you to control when users can interrupt the bot during speech. By default, any user speech immediately interrupts the bot, but this can be problematic when users engage in backchanneling—brief vocal responses like “yeah”, “okay”, or “mm-hmm” that indicate they’re listening without intending to interrupt. With interruption strategies, you can require users to meet specific criteria (such as speaking a minimum number of words or reaching a certain audio volume) before their speech will interrupt the bot, creating a more natural conversation flow. Want to try it out? Check out the interruption strategies foundational demo ​ Configuration Interruption strategies are configured via the interruption_strategies parameter in PipelineParams . When specified, the normal immediate interruption behavior is replaced with conditional interruption based on your criteria. ​ interruption_strategies List[BaseInterruptionStrategy] default: "[]" List of interruption strategies to apply. When multiple strategies are provided, the first one that evaluates to true will trigger the interruption. If empty, normal interruption behavior applies. ​ Base Strategy Interface All interruption strategies inherit from BaseInterruptionStrategy , which provides a common interface for evaluating interruption conditions. ​ append_audio(audio, sample_rate) async method Appends audio data to the strategy for analysis. Not all strategies handle audio. ​ append_text(text) async method Appends text to the strategy for analysis. Not all strategies handle text. ​ should_interrupt() async method Called when the user stops speaking to determine if interruption should occur based on accumulated audio and/or text. ​ reset() async method Resets accumulated text and/or audio data. ​ Available Strategies ​ MinWordsInterruptionStrategy Requires users to speak a minimum number of words before interrupting the bot. ​ min_words int required Minimum number of words the user must speak to interrupt the bot. Must be greater than 0. Copy Ask AI from pipecat.audio.interruptions.min_words_interruption_strategy import MinWordsInterruptionStrategy strategy = MinWordsInterruptionStrategy( min_words = 3 ) ​ How It Works When interruption strategies are configured: Bot not speaking : User speech interrupts immediately (normal behavior) Bot speaking : User speech and audio are collected and fed to strategies User stops speaking : Strategies are evaluated in order First match wins : The first strategy that returns True triggers interruption No matches : User speech is discarded The system automatically handles both audio and text input: Audio frames ( InputAudioRawFrame ) are fed to append_audio() Transcription text is fed to append_text() Strategies can use either or both data types ​ Usage Examples ​ Basic Word Count Interruption Require users to speak at least 3 words to interrupt the bot: Copy Ask AI from pipecat.audio.interruptions.min_words_interruption_strategy import MinWordsInterruptionStrategy from pipecat.pipeline.task import PipelineParams, PipelineTask task = PipelineTask( pipeline, params = PipelineParams( allow_interruptions = True , interruption_strategies = [MinWordsInterruptionStrategy( min_words = 3 )] ) ) ​ Multiple Strategies with Priority Strategies are evaluated in order, with the first match triggering interruption: Copy Ask AI # Prioritize word count, then volume (hypothetical future strategy) task = PipelineTask( pipeline, params = PipelineParams( allow_interruptions = True , interruption_strategies = [ MinWordsInterruptionStrategy( min_words = 2 ), # Check first # VolumeInterruptionStrategy(min_volume=0.8), # Your custom strategy ] ) ) ​ Behavior Comparison Scenario Without Strategy With MinWordsInterruptionStrategy(min_words=3) User says “okay” while bot speaks ✅ Interrupts immediately ❌ Ignored (only 1 word) User says “yes that’s right” while bot speaks ✅ Interrupts immediately ✅ Interrupts (3 words) User speaks while bot is silent ✅ Processed immediately ✅ Processed immediately ​ Notes Interruption strategies only affect behavior when the bot is actively speaking When the bot is not speaking, user input is processed immediately regardless of strategy configuration The allow_interruptions parameter must be True for interruption strategies to work User speech that doesn’t meet interruption criteria is discarded, not queued Strategies are evaluated in order - first match wins Both audio and text data are automatically fed to strategies based on their implementation Word counting uses simple whitespace splitting for word boundaries TranscriptProcessor Overview On this page Overview Configuration Base Strategy Interface Available Strategies MinWordsInterruptionStrategy How It Works Usage Examples Basic Word Count Interruption Multiple Strategies with Priority Behavior Comparison Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/utilities_interruption-strategies_54e10a38.txt b/utilities_interruption-strategies_54e10a38.txt
new file mode 100644
index 0000000000000000000000000000000000000000..1405b97db845941d25093001d8b482f95f62e0c3
--- /dev/null
+++ b/utilities_interruption-strategies_54e10a38.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/interruption-strategies#basic-word-count-interruption
+Title: Interruption Strategies - Pipecat
+==================================================
+
+Interruption Strategies - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation User Interruptions Interruption Strategies Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Interruption Strategies Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview Interruption strategies allow you to control when users can interrupt the bot during speech. By default, any user speech immediately interrupts the bot, but this can be problematic when users engage in backchanneling—brief vocal responses like “yeah”, “okay”, or “mm-hmm” that indicate they’re listening without intending to interrupt. With interruption strategies, you can require users to meet specific criteria (such as speaking a minimum number of words or reaching a certain audio volume) before their speech will interrupt the bot, creating a more natural conversation flow. Want to try it out? Check out the interruption strategies foundational demo ​ Configuration Interruption strategies are configured via the interruption_strategies parameter in PipelineParams . When specified, the normal immediate interruption behavior is replaced with conditional interruption based on your criteria. ​ interruption_strategies List[BaseInterruptionStrategy] default: "[]" List of interruption strategies to apply. When multiple strategies are provided, the first one that evaluates to true will trigger the interruption. If empty, normal interruption behavior applies. ​ Base Strategy Interface All interruption strategies inherit from BaseInterruptionStrategy , which provides a common interface for evaluating interruption conditions. ​ append_audio(audio, sample_rate) async method Appends audio data to the strategy for analysis. Not all strategies handle audio. ​ append_text(text) async method Appends text to the strategy for analysis. Not all strategies handle text. ​ should_interrupt() async method Called when the user stops speaking to determine if interruption should occur based on accumulated audio and/or text. ​ reset() async method Resets accumulated text and/or audio data. ​ Available Strategies ​ MinWordsInterruptionStrategy Requires users to speak a minimum number of words before interrupting the bot. ​ min_words int required Minimum number of words the user must speak to interrupt the bot. Must be greater than 0. Copy Ask AI from pipecat.audio.interruptions.min_words_interruption_strategy import MinWordsInterruptionStrategy strategy = MinWordsInterruptionStrategy( min_words = 3 ) ​ How It Works When interruption strategies are configured: Bot not speaking : User speech interrupts immediately (normal behavior) Bot speaking : User speech and audio are collected and fed to strategies User stops speaking : Strategies are evaluated in order First match wins : The first strategy that returns True triggers interruption No matches : User speech is discarded The system automatically handles both audio and text input: Audio frames ( InputAudioRawFrame ) are fed to append_audio() Transcription text is fed to append_text() Strategies can use either or both data types ​ Usage Examples ​ Basic Word Count Interruption Require users to speak at least 3 words to interrupt the bot: Copy Ask AI from pipecat.audio.interruptions.min_words_interruption_strategy import MinWordsInterruptionStrategy from pipecat.pipeline.task import PipelineParams, PipelineTask task = PipelineTask( pipeline, params = PipelineParams( allow_interruptions = True , interruption_strategies = [MinWordsInterruptionStrategy( min_words = 3 )] ) ) ​ Multiple Strategies with Priority Strategies are evaluated in order, with the first match triggering interruption: Copy Ask AI # Prioritize word count, then volume (hypothetical future strategy) task = PipelineTask( pipeline, params = PipelineParams( allow_interruptions = True , interruption_strategies = [ MinWordsInterruptionStrategy( min_words = 2 ), # Check first # VolumeInterruptionStrategy(min_volume=0.8), # Your custom strategy ] ) ) ​ Behavior Comparison Scenario Without Strategy With MinWordsInterruptionStrategy(min_words=3) User says “okay” while bot speaks ✅ Interrupts immediately ❌ Ignored (only 1 word) User says “yes that’s right” while bot speaks ✅ Interrupts immediately ✅ Interrupts (3 words) User speaks while bot is silent ✅ Processed immediately ✅ Processed immediately ​ Notes Interruption strategies only affect behavior when the bot is actively speaking When the bot is not speaking, user input is processed immediately regardless of strategy configuration The allow_interruptions parameter must be True for interruption strategies to work User speech that doesn’t meet interruption criteria is discarded, not queued Strategies are evaluated in order - first match wins Both audio and text data are automatically fed to strategies based on their implementation Word counting uses simple whitespace splitting for word boundaries TranscriptProcessor Overview On this page Overview Configuration Base Strategy Interface Available Strategies MinWordsInterruptionStrategy How It Works Usage Examples Basic Word Count Interruption Multiple Strategies with Priority Behavior Comparison Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/utilities_interruption-strategies_805d2778.txt b/utilities_interruption-strategies_805d2778.txt
new file mode 100644
index 0000000000000000000000000000000000000000..1107f4cdd815360de972c6b9aafa671a62f6ef66
--- /dev/null
+++ b/utilities_interruption-strategies_805d2778.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/interruption-strategies#param-reset
+Title: Interruption Strategies - Pipecat
+==================================================
+
+Interruption Strategies - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation User Interruptions Interruption Strategies Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Interruption Strategies Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview Interruption strategies allow you to control when users can interrupt the bot during speech. By default, any user speech immediately interrupts the bot, but this can be problematic when users engage in backchanneling—brief vocal responses like “yeah”, “okay”, or “mm-hmm” that indicate they’re listening without intending to interrupt. With interruption strategies, you can require users to meet specific criteria (such as speaking a minimum number of words or reaching a certain audio volume) before their speech will interrupt the bot, creating a more natural conversation flow. Want to try it out? Check out the interruption strategies foundational demo ​ Configuration Interruption strategies are configured via the interruption_strategies parameter in PipelineParams . When specified, the normal immediate interruption behavior is replaced with conditional interruption based on your criteria. ​ interruption_strategies List[BaseInterruptionStrategy] default: "[]" List of interruption strategies to apply. When multiple strategies are provided, the first one that evaluates to true will trigger the interruption. If empty, normal interruption behavior applies. ​ Base Strategy Interface All interruption strategies inherit from BaseInterruptionStrategy , which provides a common interface for evaluating interruption conditions. ​ append_audio(audio, sample_rate) async method Appends audio data to the strategy for analysis. Not all strategies handle audio. ​ append_text(text) async method Appends text to the strategy for analysis. Not all strategies handle text. ​ should_interrupt() async method Called when the user stops speaking to determine if interruption should occur based on accumulated audio and/or text. ​ reset() async method Resets accumulated text and/or audio data. ​ Available Strategies ​ MinWordsInterruptionStrategy Requires users to speak a minimum number of words before interrupting the bot. ​ min_words int required Minimum number of words the user must speak to interrupt the bot. Must be greater than 0. Copy Ask AI from pipecat.audio.interruptions.min_words_interruption_strategy import MinWordsInterruptionStrategy strategy = MinWordsInterruptionStrategy( min_words = 3 ) ​ How It Works When interruption strategies are configured: Bot not speaking : User speech interrupts immediately (normal behavior) Bot speaking : User speech and audio are collected and fed to strategies User stops speaking : Strategies are evaluated in order First match wins : The first strategy that returns True triggers interruption No matches : User speech is discarded The system automatically handles both audio and text input: Audio frames ( InputAudioRawFrame ) are fed to append_audio() Transcription text is fed to append_text() Strategies can use either or both data types ​ Usage Examples ​ Basic Word Count Interruption Require users to speak at least 3 words to interrupt the bot: Copy Ask AI from pipecat.audio.interruptions.min_words_interruption_strategy import MinWordsInterruptionStrategy from pipecat.pipeline.task import PipelineParams, PipelineTask task = PipelineTask( pipeline, params = PipelineParams( allow_interruptions = True , interruption_strategies = [MinWordsInterruptionStrategy( min_words = 3 )] ) ) ​ Multiple Strategies with Priority Strategies are evaluated in order, with the first match triggering interruption: Copy Ask AI # Prioritize word count, then volume (hypothetical future strategy) task = PipelineTask( pipeline, params = PipelineParams( allow_interruptions = True , interruption_strategies = [ MinWordsInterruptionStrategy( min_words = 2 ), # Check first # VolumeInterruptionStrategy(min_volume=0.8), # Your custom strategy ] ) ) ​ Behavior Comparison Scenario Without Strategy With MinWordsInterruptionStrategy(min_words=3) User says “okay” while bot speaks ✅ Interrupts immediately ❌ Ignored (only 1 word) User says “yes that’s right” while bot speaks ✅ Interrupts immediately ✅ Interrupts (3 words) User speaks while bot is silent ✅ Processed immediately ✅ Processed immediately ​ Notes Interruption strategies only affect behavior when the bot is actively speaking When the bot is not speaking, user input is processed immediately regardless of strategy configuration The allow_interruptions parameter must be True for interruption strategies to work User speech that doesn’t meet interruption criteria is discarded, not queued Strategies are evaluated in order - first match wins Both audio and text data are automatically fed to strategies based on their implementation Word counting uses simple whitespace splitting for word boundaries TranscriptProcessor Overview On this page Overview Configuration Base Strategy Interface Available Strategies MinWordsInterruptionStrategy How It Works Usage Examples Basic Word Count Interruption Multiple Strategies with Priority Behavior Comparison Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/utilities_interruption-strategies_d5f03c34.txt b/utilities_interruption-strategies_d5f03c34.txt
new file mode 100644
index 0000000000000000000000000000000000000000..06768cb806fd7c9cada0b3aebcfa5cb810e5cf02
--- /dev/null
+++ b/utilities_interruption-strategies_d5f03c34.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/interruption-strategies#notes
+Title: Interruption Strategies - Pipecat
+==================================================
+
+Interruption Strategies - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation User Interruptions Interruption Strategies Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Interruption Strategies Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview Interruption strategies allow you to control when users can interrupt the bot during speech. By default, any user speech immediately interrupts the bot, but this can be problematic when users engage in backchanneling—brief vocal responses like “yeah”, “okay”, or “mm-hmm” that indicate they’re listening without intending to interrupt. With interruption strategies, you can require users to meet specific criteria (such as speaking a minimum number of words or reaching a certain audio volume) before their speech will interrupt the bot, creating a more natural conversation flow. Want to try it out? Check out the interruption strategies foundational demo ​ Configuration Interruption strategies are configured via the interruption_strategies parameter in PipelineParams . When specified, the normal immediate interruption behavior is replaced with conditional interruption based on your criteria. ​ interruption_strategies List[BaseInterruptionStrategy] default: "[]" List of interruption strategies to apply. When multiple strategies are provided, the first one that evaluates to true will trigger the interruption. If empty, normal interruption behavior applies. ​ Base Strategy Interface All interruption strategies inherit from BaseInterruptionStrategy , which provides a common interface for evaluating interruption conditions. ​ append_audio(audio, sample_rate) async method Appends audio data to the strategy for analysis. Not all strategies handle audio. ​ append_text(text) async method Appends text to the strategy for analysis. Not all strategies handle text. ​ should_interrupt() async method Called when the user stops speaking to determine if interruption should occur based on accumulated audio and/or text. ​ reset() async method Resets accumulated text and/or audio data. ​ Available Strategies ​ MinWordsInterruptionStrategy Requires users to speak a minimum number of words before interrupting the bot. ​ min_words int required Minimum number of words the user must speak to interrupt the bot. Must be greater than 0. Copy Ask AI from pipecat.audio.interruptions.min_words_interruption_strategy import MinWordsInterruptionStrategy strategy = MinWordsInterruptionStrategy( min_words = 3 ) ​ How It Works When interruption strategies are configured: Bot not speaking : User speech interrupts immediately (normal behavior) Bot speaking : User speech and audio are collected and fed to strategies User stops speaking : Strategies are evaluated in order First match wins : The first strategy that returns True triggers interruption No matches : User speech is discarded The system automatically handles both audio and text input: Audio frames ( InputAudioRawFrame ) are fed to append_audio() Transcription text is fed to append_text() Strategies can use either or both data types ​ Usage Examples ​ Basic Word Count Interruption Require users to speak at least 3 words to interrupt the bot: Copy Ask AI from pipecat.audio.interruptions.min_words_interruption_strategy import MinWordsInterruptionStrategy from pipecat.pipeline.task import PipelineParams, PipelineTask task = PipelineTask( pipeline, params = PipelineParams( allow_interruptions = True , interruption_strategies = [MinWordsInterruptionStrategy( min_words = 3 )] ) ) ​ Multiple Strategies with Priority Strategies are evaluated in order, with the first match triggering interruption: Copy Ask AI # Prioritize word count, then volume (hypothetical future strategy) task = PipelineTask( pipeline, params = PipelineParams( allow_interruptions = True , interruption_strategies = [ MinWordsInterruptionStrategy( min_words = 2 ), # Check first # VolumeInterruptionStrategy(min_volume=0.8), # Your custom strategy ] ) ) ​ Behavior Comparison Scenario Without Strategy With MinWordsInterruptionStrategy(min_words=3) User says “okay” while bot speaks ✅ Interrupts immediately ❌ Ignored (only 1 word) User says “yes that’s right” while bot speaks ✅ Interrupts immediately ✅ Interrupts (3 words) User speaks while bot is silent ✅ Processed immediately ✅ Processed immediately ​ Notes Interruption strategies only affect behavior when the bot is actively speaking When the bot is not speaking, user input is processed immediately regardless of strategy configuration The allow_interruptions parameter must be True for interruption strategies to work User speech that doesn’t meet interruption criteria is discarded, not queued Strategies are evaluated in order - first match wins Both audio and text data are automatically fed to strategies based on their implementation Word counting uses simple whitespace splitting for word boundaries TranscriptProcessor Overview On this page Overview Configuration Base Strategy Interface Available Strategies MinWordsInterruptionStrategy How It Works Usage Examples Basic Word Count Interruption Multiple Strategies with Priority Behavior Comparison Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/utilities_opentelemetry_1171b467.txt b/utilities_opentelemetry_1171b467.txt
new file mode 100644
index 0000000000000000000000000000000000000000..4d191207c0b44b1f3c71218fa45ce93ae96a96b8
--- /dev/null
+++ b/utilities_opentelemetry_1171b467.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/opentelemetry#param-additional-span-attributes
+Title: OpenTelemetry Tracing - Pipecat
+==================================================
+
+OpenTelemetry Tracing - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Metrics and Telemetry OpenTelemetry Tracing Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry OpenTelemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview Pipecat includes built-in support for OpenTelemetry tracing, allowing you to gain deep visibility into your voice applications. Tracing helps you: Track latency and performance across your conversation pipeline Monitor service health and identify bottlenecks Visualize conversation turns and service dependencies Collect usage metrics and operational analytics ​ Installation To use OpenTelemetry tracing with Pipecat, install the tracing dependencies: Copy Ask AI pip install "pipecat-ai[tracing]" For local development and testing, we recommend using Jaeger as a trace collector. You can run it with Docker: Copy Ask AI docker run -d --name jaeger \ -p 16686:16686 \ -p 4317:4317 \ -p 4318:4318 \ jaegertracing/all-in-one:latest Then access the UI at http://localhost:16686 ​ Basic Setup Enabling tracing in your Pipecat application requires two steps: Initialize the OpenTelemetry SDK with your preferred exporter Enable tracing in your PipelineTask Copy Ask AI import os from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter from pipecat.utils.tracing.setup import setup_tracing from pipecat.pipeline.task import PipelineTask, PipelineParams # Step 1: Initialize OpenTelemetry with your chosen exporter exporter = OTLPSpanExporter( endpoint = "http://localhost:4317" , # Jaeger or other collector endpoint insecure = True , ) setup_tracing( service_name = "my-voice-app" , exporter = exporter, console_export = False , # Set to True for debug output ) # Step 2: Enable tracing in your PipelineTask task = PipelineTask( pipeline, params = PipelineParams( enable_metrics = True , # Required for some service metrics ), enable_tracing = True , # Enable tracing for this task enable_turn_tracking = True , # Enable turn tracking for this task conversation_id = "customer-123" , # Optional - will auto-generate if not provided additional_span_attributes = { "session.id" : "abc-123" } # Optional - additional attributes to attach to the otel span ) For complete working examples, see our sample implementations: Jaeger Tracing Example - Uses gRPC exporter with Jaeger Langfuse Tracing Example - Uses HTTP exporter with Langfuse for LLM-focused observability ​ Trace Structure Pipecat organizes traces hierarchically, following the natural structure of conversations: Copy Ask AI Conversation (conversation) ├── turn │ ├── stt │ ├── llm │ └── tts └── turn ├── stt ├── llm └── tts turn... For real-time multimodal services like Gemini Live and OpenAI Realtime, the structure adapts to their specific patterns: Copy Ask AI Conversation (conversation) ├── turn │ ├── llm_setup (session configuration) │ ├── stt (user input) │ ├── llm_response (complete response with usage) │ └── llm_tool_call/llm_tool_result (for function calls) └── turn ├── stt (user input) └── llm_response (complete response) turn... This hierarchical structure makes it easy to: Track the full lifecycle of a conversation Measure latency for individual turns Identify which services are contributing to delays Compare performance across different conversations ​ Exporter Options Pipecat supports any OpenTelemetry-compatible exporter. Common options include: ​ OTLP Exporter (for Jaeger, Grafana, etc.) Copy Ask AI from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter exporter = OTLPSpanExporter( endpoint = "http://localhost:4317" , # Your collector endpoint insecure = True , # Use False for TLS connections ) ​ HTTP OTLP Exporter (for Langfuse, etc.) Copy Ask AI from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter exporter = OTLPSpanExporter( # Configure with environment variables: # OTEL_EXPORTER_OTLP_ENDPOINT # OTEL_EXPORTER_OTLP_HEADERS ) See our Langfuse example for details on configuring this exporter. ​ Console Exporter (for debugging) The console exporter can be enabled alongside any other exporter by setting console_export=True : Copy Ask AI setup_tracing( service_name = "my-voice-app" , exporter = otlp_exporter, console_export = True , # Prints traces to stdout ) ​ Cloud Provider Exporters Many cloud providers offer OpenTelemetry-compatible observability services: AWS X-Ray Google Cloud Trace Azure Monitor Datadog APM Check the OpenTelemetry documentation for specific exporter configurations: OpenTelemetry Vendors ​ Span Attributes Pipecat enriches spans with detailed attributes about service operations: ​ TTS Service Spans gen_ai.system : Service provider (e.g., “cartesia”) gen_ai.request.model : Model ID/name voice_id : Voice identifier text : The text being synthesized metrics.character_count : Number of characters in the text metrics.ttfb : Time to first byte in seconds settings.* : Service-specific configuration parameters ​ STT Service Spans gen_ai.system : Service provider (e.g., “deepgram”) gen_ai.request.model : Model ID/name transcript : The transcribed text is_final : Whether the transcription is final language : Detected or configured language vad_enabled : Whether voice activity detection is enabled metrics.ttfb : Time to first byte in seconds settings.* : Service-specific configuration parameters ​ LLM Service Spans gen_ai.system : Service provider (e.g., “openai”, “gcp.gemini”) gen_ai.request.model : Model ID/name gen_ai.operation.name : Operation type (e.g., “chat”) stream : Whether streaming is enabled input : JSON-serialized input messages output : Complete response text tools : JSON-serialized tools configuration tools.count : Number of tools available tools.names : Comma-separated tool names system : System message content gen_ai.usage.input_tokens : Number of prompt tokens gen_ai.usage.output_tokens : Number of completion tokens metrics.ttfb : Time to first byte in seconds gen_ai.request.* : Standard parameters (temperature, max_tokens, etc.) ​ Multimodal Service Spans (Gemini Live & OpenAI Realtime) ​ Setup Spans gen_ai.system : “gcp.gemini” or “openai” gen_ai.request.model : Model identifier tools.count : Number of available tools tools.definitions : JSON-serialized tool schemas system_instruction : System prompt (truncated) session.* : Session configuration parameters ​ Request Spans (OpenAI Realtime) input : JSON-serialized context messages being sent gen_ai.operation.name : “llm_request” ​ Response Spans output : Complete assistant response text output_modality : “TEXT” or “AUDIO” (Gemini Live) gen_ai.usage.input_tokens : Prompt tokens used gen_ai.usage.output_tokens : Completion tokens generated function_calls.count : Number of function calls made function_calls.names : Comma-separated function names metrics.ttfb : Time to first response in seconds ​ Tool Call/Result Spans (Gemini Live) tool.function_name : Name of the function being called tool.call_id : Unique identifier for the call tool.arguments : Function arguments (truncated) tool.result : Function execution result (truncated) tool.result_status : “completed”, “error”, or “parse_error” ​ Turn Spans turn.number : Sequential turn number turn.type : Type of turn (e.g., “conversation”) turn.duration_seconds : Duration of the turn turn.was_interrupted : Whether the turn was interrupted conversation.id : ID of the parent conversation ​ Conversation Spans conversation.id : Unique identifier for the conversation conversation.type : Type of conversation (e.g., “voice”) ​ Usage Metrics Pipecat’s tracing implementation automatically captures usage metrics for LLM and TTS services: ​ LLM Token Usage Token usage is captured in LLM spans as: gen_ai.usage.input_tokens gen_ai.usage.output_tokens ​ TTS Character Count Character count is captured in TTS spans as: metrics.character_count ​ Performance Metrics Pipecat traces capture key performance metrics for each service: ​ Time To First Byte (TTFB) The time it takes for a service to produce its first response: metrics.ttfb (in seconds) ​ Processing Duration The total time spent processing in each service is captured in the span duration. ​ Configuration Options ​ PipelineTask Parameters ​ enable_tracing bool default: "True" Enable or disable tracing for the pipeline ​ enable_turn_tracking bool default: "False" Whether to enable turn tracking. ​ conversation_id Optional[str] default: "None" Custom ID for the conversation. If not provided, a UUID will be generated ​ additional_span_attributes Optional[dict] default: "None" Any additional attributes to add to top-level OpenTelemetry conversation span. ​ setup_tracing() Parameters ​ service_name str default: "pipecat" Name of the service for traces ​ exporter Optional[SpanExporter] default: "None" A pre-configured OpenTelemetry span exporter instance ​ console_export bool default: "False" Whether to also export traces to console (useful for debugging) ​ Example Here’s a complete example showing OpenTelemetry tracing setup with Jaeger: Copy Ask AI import os from dotenv import load_dotenv from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter from pipecat.pipeline.pipeline import Pipeline from pipecat.pipeline.task import PipelineParams, PipelineTask from pipecat.services.deepgram.stt import DeepgramSTTService from pipecat.services.openai.llm import OpenAILLMService from pipecat.services.cartesia.tts import CartesiaTTSService from pipecat.utils.tracing.setup import setup_tracing load_dotenv() # Initialize tracing if enabled if os.getenv( "ENABLE_TRACING" ): # Create the exporter otlp_exporter = OTLPSpanExporter( endpoint = os.getenv( "OTEL_EXPORTER_OTLP_ENDPOINT" , "http://localhost:4317" ), insecure = True , ) # Set up tracing with the exporter setup_tracing( service_name = "pipecat-demo" , exporter = otlp_exporter, console_export = bool (os.getenv( "OTEL_CONSOLE_EXPORT" )), ) # Create your services stt = DeepgramSTTService( api_key = os.getenv( "DEEPGRAM_API_KEY" )) llm = OpenAILLMService( api_key = os.getenv( "OPENAI_API_KEY" )) tts = CartesiaTTSService( api_key = os.getenv( "CARTESIA_API_KEY" ), voice_id = "your-voice-id" ) # Build pipeline pipeline = Pipeline([ transport.input(), stt, context_aggregator.user(), llm, tts, transport.output(), context_aggregator.assistant(), ]) # Create pipeline task with tracing enabled task = PipelineTask( pipeline, params = PipelineParams( allow_interruptions = True , enable_metrics = True , enable_usage_metrics = True , ), enable_tracing = True , enable_turn_tracking = True , conversation_id = "customer-123" , # Optional - will auto-generate if not provided additional_span_attributes = { "session.id" : "abc-123" } # Optional - additional attributes to attach to the otel span ) # Run the pipeline runner = PipelineRunner() await runner.run(task) ​ Troubleshooting If you’re having issues with tracing: No Traces Visible : Ensure the OpenTelemetry packages are installed and that your collector endpoint is correct Missing Service Data : Verify that enable_metrics=True is set in PipelineParams Debugging Tracing : Enable console export with console_export=True to view traces in your logs Connection Errors : Check network connectivity to your trace collector Collector Configuration : Verify your collector is properly set up to receive traces ​ References OpenTelemetry Python Documentation OpenTelemetry Tracing Specification Jaeger Documentation Langfuse OpenTelemetry Documentation WakeNotifierFilter MCPClient On this page Overview Installation Basic Setup Trace Structure Exporter Options OTLP Exporter (for Jaeger, Grafana, etc.) HTTP OTLP Exporter (for Langfuse, etc.) Console Exporter (for debugging) Cloud Provider Exporters Span Attributes TTS Service Spans STT Service Spans LLM Service Spans Multimodal Service Spans (Gemini Live & OpenAI Realtime) Setup Spans Request Spans (OpenAI Realtime) Response Spans Tool Call/Result Spans (Gemini Live) Turn Spans Conversation Spans Usage Metrics LLM Token Usage TTS Character Count Performance Metrics Time To First Byte (TTFB) Processing Duration Configuration Options PipelineTask Parameters setup_tracing() Parameters Example Troubleshooting References Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/utilities_opentelemetry_2712a191.txt b/utilities_opentelemetry_2712a191.txt
new file mode 100644
index 0000000000000000000000000000000000000000..983297443aae416fef93efe36e0ec336da77fd29
--- /dev/null
+++ b/utilities_opentelemetry_2712a191.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/opentelemetry#overview
+Title: OpenTelemetry Tracing - Pipecat
+==================================================
+
+OpenTelemetry Tracing - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Metrics and Telemetry OpenTelemetry Tracing Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry OpenTelemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview Pipecat includes built-in support for OpenTelemetry tracing, allowing you to gain deep visibility into your voice applications. Tracing helps you: Track latency and performance across your conversation pipeline Monitor service health and identify bottlenecks Visualize conversation turns and service dependencies Collect usage metrics and operational analytics ​ Installation To use OpenTelemetry tracing with Pipecat, install the tracing dependencies: Copy Ask AI pip install "pipecat-ai[tracing]" For local development and testing, we recommend using Jaeger as a trace collector. You can run it with Docker: Copy Ask AI docker run -d --name jaeger \ -p 16686:16686 \ -p 4317:4317 \ -p 4318:4318 \ jaegertracing/all-in-one:latest Then access the UI at http://localhost:16686 ​ Basic Setup Enabling tracing in your Pipecat application requires two steps: Initialize the OpenTelemetry SDK with your preferred exporter Enable tracing in your PipelineTask Copy Ask AI import os from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter from pipecat.utils.tracing.setup import setup_tracing from pipecat.pipeline.task import PipelineTask, PipelineParams # Step 1: Initialize OpenTelemetry with your chosen exporter exporter = OTLPSpanExporter( endpoint = "http://localhost:4317" , # Jaeger or other collector endpoint insecure = True , ) setup_tracing( service_name = "my-voice-app" , exporter = exporter, console_export = False , # Set to True for debug output ) # Step 2: Enable tracing in your PipelineTask task = PipelineTask( pipeline, params = PipelineParams( enable_metrics = True , # Required for some service metrics ), enable_tracing = True , # Enable tracing for this task enable_turn_tracking = True , # Enable turn tracking for this task conversation_id = "customer-123" , # Optional - will auto-generate if not provided additional_span_attributes = { "session.id" : "abc-123" } # Optional - additional attributes to attach to the otel span ) For complete working examples, see our sample implementations: Jaeger Tracing Example - Uses gRPC exporter with Jaeger Langfuse Tracing Example - Uses HTTP exporter with Langfuse for LLM-focused observability ​ Trace Structure Pipecat organizes traces hierarchically, following the natural structure of conversations: Copy Ask AI Conversation (conversation) ├── turn │ ├── stt │ ├── llm │ └── tts └── turn ├── stt ├── llm └── tts turn... For real-time multimodal services like Gemini Live and OpenAI Realtime, the structure adapts to their specific patterns: Copy Ask AI Conversation (conversation) ├── turn │ ├── llm_setup (session configuration) │ ├── stt (user input) │ ├── llm_response (complete response with usage) │ └── llm_tool_call/llm_tool_result (for function calls) └── turn ├── stt (user input) └── llm_response (complete response) turn... This hierarchical structure makes it easy to: Track the full lifecycle of a conversation Measure latency for individual turns Identify which services are contributing to delays Compare performance across different conversations ​ Exporter Options Pipecat supports any OpenTelemetry-compatible exporter. Common options include: ​ OTLP Exporter (for Jaeger, Grafana, etc.) Copy Ask AI from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter exporter = OTLPSpanExporter( endpoint = "http://localhost:4317" , # Your collector endpoint insecure = True , # Use False for TLS connections ) ​ HTTP OTLP Exporter (for Langfuse, etc.) Copy Ask AI from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter exporter = OTLPSpanExporter( # Configure with environment variables: # OTEL_EXPORTER_OTLP_ENDPOINT # OTEL_EXPORTER_OTLP_HEADERS ) See our Langfuse example for details on configuring this exporter. ​ Console Exporter (for debugging) The console exporter can be enabled alongside any other exporter by setting console_export=True : Copy Ask AI setup_tracing( service_name = "my-voice-app" , exporter = otlp_exporter, console_export = True , # Prints traces to stdout ) ​ Cloud Provider Exporters Many cloud providers offer OpenTelemetry-compatible observability services: AWS X-Ray Google Cloud Trace Azure Monitor Datadog APM Check the OpenTelemetry documentation for specific exporter configurations: OpenTelemetry Vendors ​ Span Attributes Pipecat enriches spans with detailed attributes about service operations: ​ TTS Service Spans gen_ai.system : Service provider (e.g., “cartesia”) gen_ai.request.model : Model ID/name voice_id : Voice identifier text : The text being synthesized metrics.character_count : Number of characters in the text metrics.ttfb : Time to first byte in seconds settings.* : Service-specific configuration parameters ​ STT Service Spans gen_ai.system : Service provider (e.g., “deepgram”) gen_ai.request.model : Model ID/name transcript : The transcribed text is_final : Whether the transcription is final language : Detected or configured language vad_enabled : Whether voice activity detection is enabled metrics.ttfb : Time to first byte in seconds settings.* : Service-specific configuration parameters ​ LLM Service Spans gen_ai.system : Service provider (e.g., “openai”, “gcp.gemini”) gen_ai.request.model : Model ID/name gen_ai.operation.name : Operation type (e.g., “chat”) stream : Whether streaming is enabled input : JSON-serialized input messages output : Complete response text tools : JSON-serialized tools configuration tools.count : Number of tools available tools.names : Comma-separated tool names system : System message content gen_ai.usage.input_tokens : Number of prompt tokens gen_ai.usage.output_tokens : Number of completion tokens metrics.ttfb : Time to first byte in seconds gen_ai.request.* : Standard parameters (temperature, max_tokens, etc.) ​ Multimodal Service Spans (Gemini Live & OpenAI Realtime) ​ Setup Spans gen_ai.system : “gcp.gemini” or “openai” gen_ai.request.model : Model identifier tools.count : Number of available tools tools.definitions : JSON-serialized tool schemas system_instruction : System prompt (truncated) session.* : Session configuration parameters ​ Request Spans (OpenAI Realtime) input : JSON-serialized context messages being sent gen_ai.operation.name : “llm_request” ​ Response Spans output : Complete assistant response text output_modality : “TEXT” or “AUDIO” (Gemini Live) gen_ai.usage.input_tokens : Prompt tokens used gen_ai.usage.output_tokens : Completion tokens generated function_calls.count : Number of function calls made function_calls.names : Comma-separated function names metrics.ttfb : Time to first response in seconds ​ Tool Call/Result Spans (Gemini Live) tool.function_name : Name of the function being called tool.call_id : Unique identifier for the call tool.arguments : Function arguments (truncated) tool.result : Function execution result (truncated) tool.result_status : “completed”, “error”, or “parse_error” ​ Turn Spans turn.number : Sequential turn number turn.type : Type of turn (e.g., “conversation”) turn.duration_seconds : Duration of the turn turn.was_interrupted : Whether the turn was interrupted conversation.id : ID of the parent conversation ​ Conversation Spans conversation.id : Unique identifier for the conversation conversation.type : Type of conversation (e.g., “voice”) ​ Usage Metrics Pipecat’s tracing implementation automatically captures usage metrics for LLM and TTS services: ​ LLM Token Usage Token usage is captured in LLM spans as: gen_ai.usage.input_tokens gen_ai.usage.output_tokens ​ TTS Character Count Character count is captured in TTS spans as: metrics.character_count ​ Performance Metrics Pipecat traces capture key performance metrics for each service: ​ Time To First Byte (TTFB) The time it takes for a service to produce its first response: metrics.ttfb (in seconds) ​ Processing Duration The total time spent processing in each service is captured in the span duration. ​ Configuration Options ​ PipelineTask Parameters ​ enable_tracing bool default: "True" Enable or disable tracing for the pipeline ​ enable_turn_tracking bool default: "False" Whether to enable turn tracking. ​ conversation_id Optional[str] default: "None" Custom ID for the conversation. If not provided, a UUID will be generated ​ additional_span_attributes Optional[dict] default: "None" Any additional attributes to add to top-level OpenTelemetry conversation span. ​ setup_tracing() Parameters ​ service_name str default: "pipecat" Name of the service for traces ​ exporter Optional[SpanExporter] default: "None" A pre-configured OpenTelemetry span exporter instance ​ console_export bool default: "False" Whether to also export traces to console (useful for debugging) ​ Example Here’s a complete example showing OpenTelemetry tracing setup with Jaeger: Copy Ask AI import os from dotenv import load_dotenv from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter from pipecat.pipeline.pipeline import Pipeline from pipecat.pipeline.task import PipelineParams, PipelineTask from pipecat.services.deepgram.stt import DeepgramSTTService from pipecat.services.openai.llm import OpenAILLMService from pipecat.services.cartesia.tts import CartesiaTTSService from pipecat.utils.tracing.setup import setup_tracing load_dotenv() # Initialize tracing if enabled if os.getenv( "ENABLE_TRACING" ): # Create the exporter otlp_exporter = OTLPSpanExporter( endpoint = os.getenv( "OTEL_EXPORTER_OTLP_ENDPOINT" , "http://localhost:4317" ), insecure = True , ) # Set up tracing with the exporter setup_tracing( service_name = "pipecat-demo" , exporter = otlp_exporter, console_export = bool (os.getenv( "OTEL_CONSOLE_EXPORT" )), ) # Create your services stt = DeepgramSTTService( api_key = os.getenv( "DEEPGRAM_API_KEY" )) llm = OpenAILLMService( api_key = os.getenv( "OPENAI_API_KEY" )) tts = CartesiaTTSService( api_key = os.getenv( "CARTESIA_API_KEY" ), voice_id = "your-voice-id" ) # Build pipeline pipeline = Pipeline([ transport.input(), stt, context_aggregator.user(), llm, tts, transport.output(), context_aggregator.assistant(), ]) # Create pipeline task with tracing enabled task = PipelineTask( pipeline, params = PipelineParams( allow_interruptions = True , enable_metrics = True , enable_usage_metrics = True , ), enable_tracing = True , enable_turn_tracking = True , conversation_id = "customer-123" , # Optional - will auto-generate if not provided additional_span_attributes = { "session.id" : "abc-123" } # Optional - additional attributes to attach to the otel span ) # Run the pipeline runner = PipelineRunner() await runner.run(task) ​ Troubleshooting If you’re having issues with tracing: No Traces Visible : Ensure the OpenTelemetry packages are installed and that your collector endpoint is correct Missing Service Data : Verify that enable_metrics=True is set in PipelineParams Debugging Tracing : Enable console export with console_export=True to view traces in your logs Connection Errors : Check network connectivity to your trace collector Collector Configuration : Verify your collector is properly set up to receive traces ​ References OpenTelemetry Python Documentation OpenTelemetry Tracing Specification Jaeger Documentation Langfuse OpenTelemetry Documentation WakeNotifierFilter MCPClient On this page Overview Installation Basic Setup Trace Structure Exporter Options OTLP Exporter (for Jaeger, Grafana, etc.) HTTP OTLP Exporter (for Langfuse, etc.) Console Exporter (for debugging) Cloud Provider Exporters Span Attributes TTS Service Spans STT Service Spans LLM Service Spans Multimodal Service Spans (Gemini Live & OpenAI Realtime) Setup Spans Request Spans (OpenAI Realtime) Response Spans Tool Call/Result Spans (Gemini Live) Turn Spans Conversation Spans Usage Metrics LLM Token Usage TTS Character Count Performance Metrics Time To First Byte (TTFB) Processing Duration Configuration Options PipelineTask Parameters setup_tracing() Parameters Example Troubleshooting References Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/utilities_opentelemetry_436c7e47.txt b/utilities_opentelemetry_436c7e47.txt
new file mode 100644
index 0000000000000000000000000000000000000000..07b583f2e9cd15ed6ef63fc0764279e83e4eb747
--- /dev/null
+++ b/utilities_opentelemetry_436c7e47.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/opentelemetry#processing-duration
+Title: OpenTelemetry Tracing - Pipecat
+==================================================
+
+OpenTelemetry Tracing - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Metrics and Telemetry OpenTelemetry Tracing Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry OpenTelemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview Pipecat includes built-in support for OpenTelemetry tracing, allowing you to gain deep visibility into your voice applications. Tracing helps you: Track latency and performance across your conversation pipeline Monitor service health and identify bottlenecks Visualize conversation turns and service dependencies Collect usage metrics and operational analytics ​ Installation To use OpenTelemetry tracing with Pipecat, install the tracing dependencies: Copy Ask AI pip install "pipecat-ai[tracing]" For local development and testing, we recommend using Jaeger as a trace collector. You can run it with Docker: Copy Ask AI docker run -d --name jaeger \ -p 16686:16686 \ -p 4317:4317 \ -p 4318:4318 \ jaegertracing/all-in-one:latest Then access the UI at http://localhost:16686 ​ Basic Setup Enabling tracing in your Pipecat application requires two steps: Initialize the OpenTelemetry SDK with your preferred exporter Enable tracing in your PipelineTask Copy Ask AI import os from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter from pipecat.utils.tracing.setup import setup_tracing from pipecat.pipeline.task import PipelineTask, PipelineParams # Step 1: Initialize OpenTelemetry with your chosen exporter exporter = OTLPSpanExporter( endpoint = "http://localhost:4317" , # Jaeger or other collector endpoint insecure = True , ) setup_tracing( service_name = "my-voice-app" , exporter = exporter, console_export = False , # Set to True for debug output ) # Step 2: Enable tracing in your PipelineTask task = PipelineTask( pipeline, params = PipelineParams( enable_metrics = True , # Required for some service metrics ), enable_tracing = True , # Enable tracing for this task enable_turn_tracking = True , # Enable turn tracking for this task conversation_id = "customer-123" , # Optional - will auto-generate if not provided additional_span_attributes = { "session.id" : "abc-123" } # Optional - additional attributes to attach to the otel span ) For complete working examples, see our sample implementations: Jaeger Tracing Example - Uses gRPC exporter with Jaeger Langfuse Tracing Example - Uses HTTP exporter with Langfuse for LLM-focused observability ​ Trace Structure Pipecat organizes traces hierarchically, following the natural structure of conversations: Copy Ask AI Conversation (conversation) ├── turn │ ├── stt │ ├── llm │ └── tts └── turn ├── stt ├── llm └── tts turn... For real-time multimodal services like Gemini Live and OpenAI Realtime, the structure adapts to their specific patterns: Copy Ask AI Conversation (conversation) ├── turn │ ├── llm_setup (session configuration) │ ├── stt (user input) │ ├── llm_response (complete response with usage) │ └── llm_tool_call/llm_tool_result (for function calls) └── turn ├── stt (user input) └── llm_response (complete response) turn... This hierarchical structure makes it easy to: Track the full lifecycle of a conversation Measure latency for individual turns Identify which services are contributing to delays Compare performance across different conversations ​ Exporter Options Pipecat supports any OpenTelemetry-compatible exporter. Common options include: ​ OTLP Exporter (for Jaeger, Grafana, etc.) Copy Ask AI from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter exporter = OTLPSpanExporter( endpoint = "http://localhost:4317" , # Your collector endpoint insecure = True , # Use False for TLS connections ) ​ HTTP OTLP Exporter (for Langfuse, etc.) Copy Ask AI from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter exporter = OTLPSpanExporter( # Configure with environment variables: # OTEL_EXPORTER_OTLP_ENDPOINT # OTEL_EXPORTER_OTLP_HEADERS ) See our Langfuse example for details on configuring this exporter. ​ Console Exporter (for debugging) The console exporter can be enabled alongside any other exporter by setting console_export=True : Copy Ask AI setup_tracing( service_name = "my-voice-app" , exporter = otlp_exporter, console_export = True , # Prints traces to stdout ) ​ Cloud Provider Exporters Many cloud providers offer OpenTelemetry-compatible observability services: AWS X-Ray Google Cloud Trace Azure Monitor Datadog APM Check the OpenTelemetry documentation for specific exporter configurations: OpenTelemetry Vendors ​ Span Attributes Pipecat enriches spans with detailed attributes about service operations: ​ TTS Service Spans gen_ai.system : Service provider (e.g., “cartesia”) gen_ai.request.model : Model ID/name voice_id : Voice identifier text : The text being synthesized metrics.character_count : Number of characters in the text metrics.ttfb : Time to first byte in seconds settings.* : Service-specific configuration parameters ​ STT Service Spans gen_ai.system : Service provider (e.g., “deepgram”) gen_ai.request.model : Model ID/name transcript : The transcribed text is_final : Whether the transcription is final language : Detected or configured language vad_enabled : Whether voice activity detection is enabled metrics.ttfb : Time to first byte in seconds settings.* : Service-specific configuration parameters ​ LLM Service Spans gen_ai.system : Service provider (e.g., “openai”, “gcp.gemini”) gen_ai.request.model : Model ID/name gen_ai.operation.name : Operation type (e.g., “chat”) stream : Whether streaming is enabled input : JSON-serialized input messages output : Complete response text tools : JSON-serialized tools configuration tools.count : Number of tools available tools.names : Comma-separated tool names system : System message content gen_ai.usage.input_tokens : Number of prompt tokens gen_ai.usage.output_tokens : Number of completion tokens metrics.ttfb : Time to first byte in seconds gen_ai.request.* : Standard parameters (temperature, max_tokens, etc.) ​ Multimodal Service Spans (Gemini Live & OpenAI Realtime) ​ Setup Spans gen_ai.system : “gcp.gemini” or “openai” gen_ai.request.model : Model identifier tools.count : Number of available tools tools.definitions : JSON-serialized tool schemas system_instruction : System prompt (truncated) session.* : Session configuration parameters ​ Request Spans (OpenAI Realtime) input : JSON-serialized context messages being sent gen_ai.operation.name : “llm_request” ​ Response Spans output : Complete assistant response text output_modality : “TEXT” or “AUDIO” (Gemini Live) gen_ai.usage.input_tokens : Prompt tokens used gen_ai.usage.output_tokens : Completion tokens generated function_calls.count : Number of function calls made function_calls.names : Comma-separated function names metrics.ttfb : Time to first response in seconds ​ Tool Call/Result Spans (Gemini Live) tool.function_name : Name of the function being called tool.call_id : Unique identifier for the call tool.arguments : Function arguments (truncated) tool.result : Function execution result (truncated) tool.result_status : “completed”, “error”, or “parse_error” ​ Turn Spans turn.number : Sequential turn number turn.type : Type of turn (e.g., “conversation”) turn.duration_seconds : Duration of the turn turn.was_interrupted : Whether the turn was interrupted conversation.id : ID of the parent conversation ​ Conversation Spans conversation.id : Unique identifier for the conversation conversation.type : Type of conversation (e.g., “voice”) ​ Usage Metrics Pipecat’s tracing implementation automatically captures usage metrics for LLM and TTS services: ​ LLM Token Usage Token usage is captured in LLM spans as: gen_ai.usage.input_tokens gen_ai.usage.output_tokens ​ TTS Character Count Character count is captured in TTS spans as: metrics.character_count ​ Performance Metrics Pipecat traces capture key performance metrics for each service: ​ Time To First Byte (TTFB) The time it takes for a service to produce its first response: metrics.ttfb (in seconds) ​ Processing Duration The total time spent processing in each service is captured in the span duration. ​ Configuration Options ​ PipelineTask Parameters ​ enable_tracing bool default: "True" Enable or disable tracing for the pipeline ​ enable_turn_tracking bool default: "False" Whether to enable turn tracking. ​ conversation_id Optional[str] default: "None" Custom ID for the conversation. If not provided, a UUID will be generated ​ additional_span_attributes Optional[dict] default: "None" Any additional attributes to add to top-level OpenTelemetry conversation span. ​ setup_tracing() Parameters ​ service_name str default: "pipecat" Name of the service for traces ​ exporter Optional[SpanExporter] default: "None" A pre-configured OpenTelemetry span exporter instance ​ console_export bool default: "False" Whether to also export traces to console (useful for debugging) ​ Example Here’s a complete example showing OpenTelemetry tracing setup with Jaeger: Copy Ask AI import os from dotenv import load_dotenv from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter from pipecat.pipeline.pipeline import Pipeline from pipecat.pipeline.task import PipelineParams, PipelineTask from pipecat.services.deepgram.stt import DeepgramSTTService from pipecat.services.openai.llm import OpenAILLMService from pipecat.services.cartesia.tts import CartesiaTTSService from pipecat.utils.tracing.setup import setup_tracing load_dotenv() # Initialize tracing if enabled if os.getenv( "ENABLE_TRACING" ): # Create the exporter otlp_exporter = OTLPSpanExporter( endpoint = os.getenv( "OTEL_EXPORTER_OTLP_ENDPOINT" , "http://localhost:4317" ), insecure = True , ) # Set up tracing with the exporter setup_tracing( service_name = "pipecat-demo" , exporter = otlp_exporter, console_export = bool (os.getenv( "OTEL_CONSOLE_EXPORT" )), ) # Create your services stt = DeepgramSTTService( api_key = os.getenv( "DEEPGRAM_API_KEY" )) llm = OpenAILLMService( api_key = os.getenv( "OPENAI_API_KEY" )) tts = CartesiaTTSService( api_key = os.getenv( "CARTESIA_API_KEY" ), voice_id = "your-voice-id" ) # Build pipeline pipeline = Pipeline([ transport.input(), stt, context_aggregator.user(), llm, tts, transport.output(), context_aggregator.assistant(), ]) # Create pipeline task with tracing enabled task = PipelineTask( pipeline, params = PipelineParams( allow_interruptions = True , enable_metrics = True , enable_usage_metrics = True , ), enable_tracing = True , enable_turn_tracking = True , conversation_id = "customer-123" , # Optional - will auto-generate if not provided additional_span_attributes = { "session.id" : "abc-123" } # Optional - additional attributes to attach to the otel span ) # Run the pipeline runner = PipelineRunner() await runner.run(task) ​ Troubleshooting If you’re having issues with tracing: No Traces Visible : Ensure the OpenTelemetry packages are installed and that your collector endpoint is correct Missing Service Data : Verify that enable_metrics=True is set in PipelineParams Debugging Tracing : Enable console export with console_export=True to view traces in your logs Connection Errors : Check network connectivity to your trace collector Collector Configuration : Verify your collector is properly set up to receive traces ​ References OpenTelemetry Python Documentation OpenTelemetry Tracing Specification Jaeger Documentation Langfuse OpenTelemetry Documentation WakeNotifierFilter MCPClient On this page Overview Installation Basic Setup Trace Structure Exporter Options OTLP Exporter (for Jaeger, Grafana, etc.) HTTP OTLP Exporter (for Langfuse, etc.) Console Exporter (for debugging) Cloud Provider Exporters Span Attributes TTS Service Spans STT Service Spans LLM Service Spans Multimodal Service Spans (Gemini Live & OpenAI Realtime) Setup Spans Request Spans (OpenAI Realtime) Response Spans Tool Call/Result Spans (Gemini Live) Turn Spans Conversation Spans Usage Metrics LLM Token Usage TTS Character Count Performance Metrics Time To First Byte (TTFB) Processing Duration Configuration Options PipelineTask Parameters setup_tracing() Parameters Example Troubleshooting References Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/utilities_opentelemetry_44a270fc.txt b/utilities_opentelemetry_44a270fc.txt
new file mode 100644
index 0000000000000000000000000000000000000000..842b1ca6a7eecc517fb2a1eee643f216eff7052c
--- /dev/null
+++ b/utilities_opentelemetry_44a270fc.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/opentelemetry#param-conversation-id
+Title: OpenTelemetry Tracing - Pipecat
+==================================================
+
+OpenTelemetry Tracing - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Metrics and Telemetry OpenTelemetry Tracing Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry OpenTelemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview Pipecat includes built-in support for OpenTelemetry tracing, allowing you to gain deep visibility into your voice applications. Tracing helps you: Track latency and performance across your conversation pipeline Monitor service health and identify bottlenecks Visualize conversation turns and service dependencies Collect usage metrics and operational analytics ​ Installation To use OpenTelemetry tracing with Pipecat, install the tracing dependencies: Copy Ask AI pip install "pipecat-ai[tracing]" For local development and testing, we recommend using Jaeger as a trace collector. You can run it with Docker: Copy Ask AI docker run -d --name jaeger \ -p 16686:16686 \ -p 4317:4317 \ -p 4318:4318 \ jaegertracing/all-in-one:latest Then access the UI at http://localhost:16686 ​ Basic Setup Enabling tracing in your Pipecat application requires two steps: Initialize the OpenTelemetry SDK with your preferred exporter Enable tracing in your PipelineTask Copy Ask AI import os from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter from pipecat.utils.tracing.setup import setup_tracing from pipecat.pipeline.task import PipelineTask, PipelineParams # Step 1: Initialize OpenTelemetry with your chosen exporter exporter = OTLPSpanExporter( endpoint = "http://localhost:4317" , # Jaeger or other collector endpoint insecure = True , ) setup_tracing( service_name = "my-voice-app" , exporter = exporter, console_export = False , # Set to True for debug output ) # Step 2: Enable tracing in your PipelineTask task = PipelineTask( pipeline, params = PipelineParams( enable_metrics = True , # Required for some service metrics ), enable_tracing = True , # Enable tracing for this task enable_turn_tracking = True , # Enable turn tracking for this task conversation_id = "customer-123" , # Optional - will auto-generate if not provided additional_span_attributes = { "session.id" : "abc-123" } # Optional - additional attributes to attach to the otel span ) For complete working examples, see our sample implementations: Jaeger Tracing Example - Uses gRPC exporter with Jaeger Langfuse Tracing Example - Uses HTTP exporter with Langfuse for LLM-focused observability ​ Trace Structure Pipecat organizes traces hierarchically, following the natural structure of conversations: Copy Ask AI Conversation (conversation) ├── turn │ ├── stt │ ├── llm │ └── tts └── turn ├── stt ├── llm └── tts turn... For real-time multimodal services like Gemini Live and OpenAI Realtime, the structure adapts to their specific patterns: Copy Ask AI Conversation (conversation) ├── turn │ ├── llm_setup (session configuration) │ ├── stt (user input) │ ├── llm_response (complete response with usage) │ └── llm_tool_call/llm_tool_result (for function calls) └── turn ├── stt (user input) └── llm_response (complete response) turn... This hierarchical structure makes it easy to: Track the full lifecycle of a conversation Measure latency for individual turns Identify which services are contributing to delays Compare performance across different conversations ​ Exporter Options Pipecat supports any OpenTelemetry-compatible exporter. Common options include: ​ OTLP Exporter (for Jaeger, Grafana, etc.) Copy Ask AI from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter exporter = OTLPSpanExporter( endpoint = "http://localhost:4317" , # Your collector endpoint insecure = True , # Use False for TLS connections ) ​ HTTP OTLP Exporter (for Langfuse, etc.) Copy Ask AI from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter exporter = OTLPSpanExporter( # Configure with environment variables: # OTEL_EXPORTER_OTLP_ENDPOINT # OTEL_EXPORTER_OTLP_HEADERS ) See our Langfuse example for details on configuring this exporter. ​ Console Exporter (for debugging) The console exporter can be enabled alongside any other exporter by setting console_export=True : Copy Ask AI setup_tracing( service_name = "my-voice-app" , exporter = otlp_exporter, console_export = True , # Prints traces to stdout ) ​ Cloud Provider Exporters Many cloud providers offer OpenTelemetry-compatible observability services: AWS X-Ray Google Cloud Trace Azure Monitor Datadog APM Check the OpenTelemetry documentation for specific exporter configurations: OpenTelemetry Vendors ​ Span Attributes Pipecat enriches spans with detailed attributes about service operations: ​ TTS Service Spans gen_ai.system : Service provider (e.g., “cartesia”) gen_ai.request.model : Model ID/name voice_id : Voice identifier text : The text being synthesized metrics.character_count : Number of characters in the text metrics.ttfb : Time to first byte in seconds settings.* : Service-specific configuration parameters ​ STT Service Spans gen_ai.system : Service provider (e.g., “deepgram”) gen_ai.request.model : Model ID/name transcript : The transcribed text is_final : Whether the transcription is final language : Detected or configured language vad_enabled : Whether voice activity detection is enabled metrics.ttfb : Time to first byte in seconds settings.* : Service-specific configuration parameters ​ LLM Service Spans gen_ai.system : Service provider (e.g., “openai”, “gcp.gemini”) gen_ai.request.model : Model ID/name gen_ai.operation.name : Operation type (e.g., “chat”) stream : Whether streaming is enabled input : JSON-serialized input messages output : Complete response text tools : JSON-serialized tools configuration tools.count : Number of tools available tools.names : Comma-separated tool names system : System message content gen_ai.usage.input_tokens : Number of prompt tokens gen_ai.usage.output_tokens : Number of completion tokens metrics.ttfb : Time to first byte in seconds gen_ai.request.* : Standard parameters (temperature, max_tokens, etc.) ​ Multimodal Service Spans (Gemini Live & OpenAI Realtime) ​ Setup Spans gen_ai.system : “gcp.gemini” or “openai” gen_ai.request.model : Model identifier tools.count : Number of available tools tools.definitions : JSON-serialized tool schemas system_instruction : System prompt (truncated) session.* : Session configuration parameters ​ Request Spans (OpenAI Realtime) input : JSON-serialized context messages being sent gen_ai.operation.name : “llm_request” ​ Response Spans output : Complete assistant response text output_modality : “TEXT” or “AUDIO” (Gemini Live) gen_ai.usage.input_tokens : Prompt tokens used gen_ai.usage.output_tokens : Completion tokens generated function_calls.count : Number of function calls made function_calls.names : Comma-separated function names metrics.ttfb : Time to first response in seconds ​ Tool Call/Result Spans (Gemini Live) tool.function_name : Name of the function being called tool.call_id : Unique identifier for the call tool.arguments : Function arguments (truncated) tool.result : Function execution result (truncated) tool.result_status : “completed”, “error”, or “parse_error” ​ Turn Spans turn.number : Sequential turn number turn.type : Type of turn (e.g., “conversation”) turn.duration_seconds : Duration of the turn turn.was_interrupted : Whether the turn was interrupted conversation.id : ID of the parent conversation ​ Conversation Spans conversation.id : Unique identifier for the conversation conversation.type : Type of conversation (e.g., “voice”) ​ Usage Metrics Pipecat’s tracing implementation automatically captures usage metrics for LLM and TTS services: ​ LLM Token Usage Token usage is captured in LLM spans as: gen_ai.usage.input_tokens gen_ai.usage.output_tokens ​ TTS Character Count Character count is captured in TTS spans as: metrics.character_count ​ Performance Metrics Pipecat traces capture key performance metrics for each service: ​ Time To First Byte (TTFB) The time it takes for a service to produce its first response: metrics.ttfb (in seconds) ​ Processing Duration The total time spent processing in each service is captured in the span duration. ​ Configuration Options ​ PipelineTask Parameters ​ enable_tracing bool default: "True" Enable or disable tracing for the pipeline ​ enable_turn_tracking bool default: "False" Whether to enable turn tracking. ​ conversation_id Optional[str] default: "None" Custom ID for the conversation. If not provided, a UUID will be generated ​ additional_span_attributes Optional[dict] default: "None" Any additional attributes to add to top-level OpenTelemetry conversation span. ​ setup_tracing() Parameters ​ service_name str default: "pipecat" Name of the service for traces ​ exporter Optional[SpanExporter] default: "None" A pre-configured OpenTelemetry span exporter instance ​ console_export bool default: "False" Whether to also export traces to console (useful for debugging) ​ Example Here’s a complete example showing OpenTelemetry tracing setup with Jaeger: Copy Ask AI import os from dotenv import load_dotenv from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter from pipecat.pipeline.pipeline import Pipeline from pipecat.pipeline.task import PipelineParams, PipelineTask from pipecat.services.deepgram.stt import DeepgramSTTService from pipecat.services.openai.llm import OpenAILLMService from pipecat.services.cartesia.tts import CartesiaTTSService from pipecat.utils.tracing.setup import setup_tracing load_dotenv() # Initialize tracing if enabled if os.getenv( "ENABLE_TRACING" ): # Create the exporter otlp_exporter = OTLPSpanExporter( endpoint = os.getenv( "OTEL_EXPORTER_OTLP_ENDPOINT" , "http://localhost:4317" ), insecure = True , ) # Set up tracing with the exporter setup_tracing( service_name = "pipecat-demo" , exporter = otlp_exporter, console_export = bool (os.getenv( "OTEL_CONSOLE_EXPORT" )), ) # Create your services stt = DeepgramSTTService( api_key = os.getenv( "DEEPGRAM_API_KEY" )) llm = OpenAILLMService( api_key = os.getenv( "OPENAI_API_KEY" )) tts = CartesiaTTSService( api_key = os.getenv( "CARTESIA_API_KEY" ), voice_id = "your-voice-id" ) # Build pipeline pipeline = Pipeline([ transport.input(), stt, context_aggregator.user(), llm, tts, transport.output(), context_aggregator.assistant(), ]) # Create pipeline task with tracing enabled task = PipelineTask( pipeline, params = PipelineParams( allow_interruptions = True , enable_metrics = True , enable_usage_metrics = True , ), enable_tracing = True , enable_turn_tracking = True , conversation_id = "customer-123" , # Optional - will auto-generate if not provided additional_span_attributes = { "session.id" : "abc-123" } # Optional - additional attributes to attach to the otel span ) # Run the pipeline runner = PipelineRunner() await runner.run(task) ​ Troubleshooting If you’re having issues with tracing: No Traces Visible : Ensure the OpenTelemetry packages are installed and that your collector endpoint is correct Missing Service Data : Verify that enable_metrics=True is set in PipelineParams Debugging Tracing : Enable console export with console_export=True to view traces in your logs Connection Errors : Check network connectivity to your trace collector Collector Configuration : Verify your collector is properly set up to receive traces ​ References OpenTelemetry Python Documentation OpenTelemetry Tracing Specification Jaeger Documentation Langfuse OpenTelemetry Documentation WakeNotifierFilter MCPClient On this page Overview Installation Basic Setup Trace Structure Exporter Options OTLP Exporter (for Jaeger, Grafana, etc.) HTTP OTLP Exporter (for Langfuse, etc.) Console Exporter (for debugging) Cloud Provider Exporters Span Attributes TTS Service Spans STT Service Spans LLM Service Spans Multimodal Service Spans (Gemini Live & OpenAI Realtime) Setup Spans Request Spans (OpenAI Realtime) Response Spans Tool Call/Result Spans (Gemini Live) Turn Spans Conversation Spans Usage Metrics LLM Token Usage TTS Character Count Performance Metrics Time To First Byte (TTFB) Processing Duration Configuration Options PipelineTask Parameters setup_tracing() Parameters Example Troubleshooting References Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/utilities_opentelemetry_6a804bc3.txt b/utilities_opentelemetry_6a804bc3.txt
new file mode 100644
index 0000000000000000000000000000000000000000..419d6365c42a1d1e635015f1cdb586bdd7cd39c2
--- /dev/null
+++ b/utilities_opentelemetry_6a804bc3.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/opentelemetry#setup-spans
+Title: OpenTelemetry Tracing - Pipecat
+==================================================
+
+OpenTelemetry Tracing - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Metrics and Telemetry OpenTelemetry Tracing Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry OpenTelemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview Pipecat includes built-in support for OpenTelemetry tracing, allowing you to gain deep visibility into your voice applications. Tracing helps you: Track latency and performance across your conversation pipeline Monitor service health and identify bottlenecks Visualize conversation turns and service dependencies Collect usage metrics and operational analytics ​ Installation To use OpenTelemetry tracing with Pipecat, install the tracing dependencies: Copy Ask AI pip install "pipecat-ai[tracing]" For local development and testing, we recommend using Jaeger as a trace collector. You can run it with Docker: Copy Ask AI docker run -d --name jaeger \ -p 16686:16686 \ -p 4317:4317 \ -p 4318:4318 \ jaegertracing/all-in-one:latest Then access the UI at http://localhost:16686 ​ Basic Setup Enabling tracing in your Pipecat application requires two steps: Initialize the OpenTelemetry SDK with your preferred exporter Enable tracing in your PipelineTask Copy Ask AI import os from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter from pipecat.utils.tracing.setup import setup_tracing from pipecat.pipeline.task import PipelineTask, PipelineParams # Step 1: Initialize OpenTelemetry with your chosen exporter exporter = OTLPSpanExporter( endpoint = "http://localhost:4317" , # Jaeger or other collector endpoint insecure = True , ) setup_tracing( service_name = "my-voice-app" , exporter = exporter, console_export = False , # Set to True for debug output ) # Step 2: Enable tracing in your PipelineTask task = PipelineTask( pipeline, params = PipelineParams( enable_metrics = True , # Required for some service metrics ), enable_tracing = True , # Enable tracing for this task enable_turn_tracking = True , # Enable turn tracking for this task conversation_id = "customer-123" , # Optional - will auto-generate if not provided additional_span_attributes = { "session.id" : "abc-123" } # Optional - additional attributes to attach to the otel span ) For complete working examples, see our sample implementations: Jaeger Tracing Example - Uses gRPC exporter with Jaeger Langfuse Tracing Example - Uses HTTP exporter with Langfuse for LLM-focused observability ​ Trace Structure Pipecat organizes traces hierarchically, following the natural structure of conversations: Copy Ask AI Conversation (conversation) ├── turn │ ├── stt │ ├── llm │ └── tts └── turn ├── stt ├── llm └── tts turn... For real-time multimodal services like Gemini Live and OpenAI Realtime, the structure adapts to their specific patterns: Copy Ask AI Conversation (conversation) ├── turn │ ├── llm_setup (session configuration) │ ├── stt (user input) │ ├── llm_response (complete response with usage) │ └── llm_tool_call/llm_tool_result (for function calls) └── turn ├── stt (user input) └── llm_response (complete response) turn... This hierarchical structure makes it easy to: Track the full lifecycle of a conversation Measure latency for individual turns Identify which services are contributing to delays Compare performance across different conversations ​ Exporter Options Pipecat supports any OpenTelemetry-compatible exporter. Common options include: ​ OTLP Exporter (for Jaeger, Grafana, etc.) Copy Ask AI from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter exporter = OTLPSpanExporter( endpoint = "http://localhost:4317" , # Your collector endpoint insecure = True , # Use False for TLS connections ) ​ HTTP OTLP Exporter (for Langfuse, etc.) Copy Ask AI from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter exporter = OTLPSpanExporter( # Configure with environment variables: # OTEL_EXPORTER_OTLP_ENDPOINT # OTEL_EXPORTER_OTLP_HEADERS ) See our Langfuse example for details on configuring this exporter. ​ Console Exporter (for debugging) The console exporter can be enabled alongside any other exporter by setting console_export=True : Copy Ask AI setup_tracing( service_name = "my-voice-app" , exporter = otlp_exporter, console_export = True , # Prints traces to stdout ) ​ Cloud Provider Exporters Many cloud providers offer OpenTelemetry-compatible observability services: AWS X-Ray Google Cloud Trace Azure Monitor Datadog APM Check the OpenTelemetry documentation for specific exporter configurations: OpenTelemetry Vendors ​ Span Attributes Pipecat enriches spans with detailed attributes about service operations: ​ TTS Service Spans gen_ai.system : Service provider (e.g., “cartesia”) gen_ai.request.model : Model ID/name voice_id : Voice identifier text : The text being synthesized metrics.character_count : Number of characters in the text metrics.ttfb : Time to first byte in seconds settings.* : Service-specific configuration parameters ​ STT Service Spans gen_ai.system : Service provider (e.g., “deepgram”) gen_ai.request.model : Model ID/name transcript : The transcribed text is_final : Whether the transcription is final language : Detected or configured language vad_enabled : Whether voice activity detection is enabled metrics.ttfb : Time to first byte in seconds settings.* : Service-specific configuration parameters ​ LLM Service Spans gen_ai.system : Service provider (e.g., “openai”, “gcp.gemini”) gen_ai.request.model : Model ID/name gen_ai.operation.name : Operation type (e.g., “chat”) stream : Whether streaming is enabled input : JSON-serialized input messages output : Complete response text tools : JSON-serialized tools configuration tools.count : Number of tools available tools.names : Comma-separated tool names system : System message content gen_ai.usage.input_tokens : Number of prompt tokens gen_ai.usage.output_tokens : Number of completion tokens metrics.ttfb : Time to first byte in seconds gen_ai.request.* : Standard parameters (temperature, max_tokens, etc.) ​ Multimodal Service Spans (Gemini Live & OpenAI Realtime) ​ Setup Spans gen_ai.system : “gcp.gemini” or “openai” gen_ai.request.model : Model identifier tools.count : Number of available tools tools.definitions : JSON-serialized tool schemas system_instruction : System prompt (truncated) session.* : Session configuration parameters ​ Request Spans (OpenAI Realtime) input : JSON-serialized context messages being sent gen_ai.operation.name : “llm_request” ​ Response Spans output : Complete assistant response text output_modality : “TEXT” or “AUDIO” (Gemini Live) gen_ai.usage.input_tokens : Prompt tokens used gen_ai.usage.output_tokens : Completion tokens generated function_calls.count : Number of function calls made function_calls.names : Comma-separated function names metrics.ttfb : Time to first response in seconds ​ Tool Call/Result Spans (Gemini Live) tool.function_name : Name of the function being called tool.call_id : Unique identifier for the call tool.arguments : Function arguments (truncated) tool.result : Function execution result (truncated) tool.result_status : “completed”, “error”, or “parse_error” ​ Turn Spans turn.number : Sequential turn number turn.type : Type of turn (e.g., “conversation”) turn.duration_seconds : Duration of the turn turn.was_interrupted : Whether the turn was interrupted conversation.id : ID of the parent conversation ​ Conversation Spans conversation.id : Unique identifier for the conversation conversation.type : Type of conversation (e.g., “voice”) ​ Usage Metrics Pipecat’s tracing implementation automatically captures usage metrics for LLM and TTS services: ​ LLM Token Usage Token usage is captured in LLM spans as: gen_ai.usage.input_tokens gen_ai.usage.output_tokens ​ TTS Character Count Character count is captured in TTS spans as: metrics.character_count ​ Performance Metrics Pipecat traces capture key performance metrics for each service: ​ Time To First Byte (TTFB) The time it takes for a service to produce its first response: metrics.ttfb (in seconds) ​ Processing Duration The total time spent processing in each service is captured in the span duration. ​ Configuration Options ​ PipelineTask Parameters ​ enable_tracing bool default: "True" Enable or disable tracing for the pipeline ​ enable_turn_tracking bool default: "False" Whether to enable turn tracking. ​ conversation_id Optional[str] default: "None" Custom ID for the conversation. If not provided, a UUID will be generated ​ additional_span_attributes Optional[dict] default: "None" Any additional attributes to add to top-level OpenTelemetry conversation span. ​ setup_tracing() Parameters ​ service_name str default: "pipecat" Name of the service for traces ​ exporter Optional[SpanExporter] default: "None" A pre-configured OpenTelemetry span exporter instance ​ console_export bool default: "False" Whether to also export traces to console (useful for debugging) ​ Example Here’s a complete example showing OpenTelemetry tracing setup with Jaeger: Copy Ask AI import os from dotenv import load_dotenv from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter from pipecat.pipeline.pipeline import Pipeline from pipecat.pipeline.task import PipelineParams, PipelineTask from pipecat.services.deepgram.stt import DeepgramSTTService from pipecat.services.openai.llm import OpenAILLMService from pipecat.services.cartesia.tts import CartesiaTTSService from pipecat.utils.tracing.setup import setup_tracing load_dotenv() # Initialize tracing if enabled if os.getenv( "ENABLE_TRACING" ): # Create the exporter otlp_exporter = OTLPSpanExporter( endpoint = os.getenv( "OTEL_EXPORTER_OTLP_ENDPOINT" , "http://localhost:4317" ), insecure = True , ) # Set up tracing with the exporter setup_tracing( service_name = "pipecat-demo" , exporter = otlp_exporter, console_export = bool (os.getenv( "OTEL_CONSOLE_EXPORT" )), ) # Create your services stt = DeepgramSTTService( api_key = os.getenv( "DEEPGRAM_API_KEY" )) llm = OpenAILLMService( api_key = os.getenv( "OPENAI_API_KEY" )) tts = CartesiaTTSService( api_key = os.getenv( "CARTESIA_API_KEY" ), voice_id = "your-voice-id" ) # Build pipeline pipeline = Pipeline([ transport.input(), stt, context_aggregator.user(), llm, tts, transport.output(), context_aggregator.assistant(), ]) # Create pipeline task with tracing enabled task = PipelineTask( pipeline, params = PipelineParams( allow_interruptions = True , enable_metrics = True , enable_usage_metrics = True , ), enable_tracing = True , enable_turn_tracking = True , conversation_id = "customer-123" , # Optional - will auto-generate if not provided additional_span_attributes = { "session.id" : "abc-123" } # Optional - additional attributes to attach to the otel span ) # Run the pipeline runner = PipelineRunner() await runner.run(task) ​ Troubleshooting If you’re having issues with tracing: No Traces Visible : Ensure the OpenTelemetry packages are installed and that your collector endpoint is correct Missing Service Data : Verify that enable_metrics=True is set in PipelineParams Debugging Tracing : Enable console export with console_export=True to view traces in your logs Connection Errors : Check network connectivity to your trace collector Collector Configuration : Verify your collector is properly set up to receive traces ​ References OpenTelemetry Python Documentation OpenTelemetry Tracing Specification Jaeger Documentation Langfuse OpenTelemetry Documentation WakeNotifierFilter MCPClient On this page Overview Installation Basic Setup Trace Structure Exporter Options OTLP Exporter (for Jaeger, Grafana, etc.) HTTP OTLP Exporter (for Langfuse, etc.) Console Exporter (for debugging) Cloud Provider Exporters Span Attributes TTS Service Spans STT Service Spans LLM Service Spans Multimodal Service Spans (Gemini Live & OpenAI Realtime) Setup Spans Request Spans (OpenAI Realtime) Response Spans Tool Call/Result Spans (Gemini Live) Turn Spans Conversation Spans Usage Metrics LLM Token Usage TTS Character Count Performance Metrics Time To First Byte (TTFB) Processing Duration Configuration Options PipelineTask Parameters setup_tracing() Parameters Example Troubleshooting References Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/utilities_opentelemetry_7250d029.txt b/utilities_opentelemetry_7250d029.txt
new file mode 100644
index 0000000000000000000000000000000000000000..6c495eada3b6dbbfe78204ea456f3e1c3b970883
--- /dev/null
+++ b/utilities_opentelemetry_7250d029.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/opentelemetry#turn-spans
+Title: OpenTelemetry Tracing - Pipecat
+==================================================
+
+OpenTelemetry Tracing - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Metrics and Telemetry OpenTelemetry Tracing Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry OpenTelemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview Pipecat includes built-in support for OpenTelemetry tracing, allowing you to gain deep visibility into your voice applications. Tracing helps you: Track latency and performance across your conversation pipeline Monitor service health and identify bottlenecks Visualize conversation turns and service dependencies Collect usage metrics and operational analytics ​ Installation To use OpenTelemetry tracing with Pipecat, install the tracing dependencies: Copy Ask AI pip install "pipecat-ai[tracing]" For local development and testing, we recommend using Jaeger as a trace collector. You can run it with Docker: Copy Ask AI docker run -d --name jaeger \ -p 16686:16686 \ -p 4317:4317 \ -p 4318:4318 \ jaegertracing/all-in-one:latest Then access the UI at http://localhost:16686 ​ Basic Setup Enabling tracing in your Pipecat application requires two steps: Initialize the OpenTelemetry SDK with your preferred exporter Enable tracing in your PipelineTask Copy Ask AI import os from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter from pipecat.utils.tracing.setup import setup_tracing from pipecat.pipeline.task import PipelineTask, PipelineParams # Step 1: Initialize OpenTelemetry with your chosen exporter exporter = OTLPSpanExporter( endpoint = "http://localhost:4317" , # Jaeger or other collector endpoint insecure = True , ) setup_tracing( service_name = "my-voice-app" , exporter = exporter, console_export = False , # Set to True for debug output ) # Step 2: Enable tracing in your PipelineTask task = PipelineTask( pipeline, params = PipelineParams( enable_metrics = True , # Required for some service metrics ), enable_tracing = True , # Enable tracing for this task enable_turn_tracking = True , # Enable turn tracking for this task conversation_id = "customer-123" , # Optional - will auto-generate if not provided additional_span_attributes = { "session.id" : "abc-123" } # Optional - additional attributes to attach to the otel span ) For complete working examples, see our sample implementations: Jaeger Tracing Example - Uses gRPC exporter with Jaeger Langfuse Tracing Example - Uses HTTP exporter with Langfuse for LLM-focused observability ​ Trace Structure Pipecat organizes traces hierarchically, following the natural structure of conversations: Copy Ask AI Conversation (conversation) ├── turn │ ├── stt │ ├── llm │ └── tts └── turn ├── stt ├── llm └── tts turn... For real-time multimodal services like Gemini Live and OpenAI Realtime, the structure adapts to their specific patterns: Copy Ask AI Conversation (conversation) ├── turn │ ├── llm_setup (session configuration) │ ├── stt (user input) │ ├── llm_response (complete response with usage) │ └── llm_tool_call/llm_tool_result (for function calls) └── turn ├── stt (user input) └── llm_response (complete response) turn... This hierarchical structure makes it easy to: Track the full lifecycle of a conversation Measure latency for individual turns Identify which services are contributing to delays Compare performance across different conversations ​ Exporter Options Pipecat supports any OpenTelemetry-compatible exporter. Common options include: ​ OTLP Exporter (for Jaeger, Grafana, etc.) Copy Ask AI from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter exporter = OTLPSpanExporter( endpoint = "http://localhost:4317" , # Your collector endpoint insecure = True , # Use False for TLS connections ) ​ HTTP OTLP Exporter (for Langfuse, etc.) Copy Ask AI from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter exporter = OTLPSpanExporter( # Configure with environment variables: # OTEL_EXPORTER_OTLP_ENDPOINT # OTEL_EXPORTER_OTLP_HEADERS ) See our Langfuse example for details on configuring this exporter. ​ Console Exporter (for debugging) The console exporter can be enabled alongside any other exporter by setting console_export=True : Copy Ask AI setup_tracing( service_name = "my-voice-app" , exporter = otlp_exporter, console_export = True , # Prints traces to stdout ) ​ Cloud Provider Exporters Many cloud providers offer OpenTelemetry-compatible observability services: AWS X-Ray Google Cloud Trace Azure Monitor Datadog APM Check the OpenTelemetry documentation for specific exporter configurations: OpenTelemetry Vendors ​ Span Attributes Pipecat enriches spans with detailed attributes about service operations: ​ TTS Service Spans gen_ai.system : Service provider (e.g., “cartesia”) gen_ai.request.model : Model ID/name voice_id : Voice identifier text : The text being synthesized metrics.character_count : Number of characters in the text metrics.ttfb : Time to first byte in seconds settings.* : Service-specific configuration parameters ​ STT Service Spans gen_ai.system : Service provider (e.g., “deepgram”) gen_ai.request.model : Model ID/name transcript : The transcribed text is_final : Whether the transcription is final language : Detected or configured language vad_enabled : Whether voice activity detection is enabled metrics.ttfb : Time to first byte in seconds settings.* : Service-specific configuration parameters ​ LLM Service Spans gen_ai.system : Service provider (e.g., “openai”, “gcp.gemini”) gen_ai.request.model : Model ID/name gen_ai.operation.name : Operation type (e.g., “chat”) stream : Whether streaming is enabled input : JSON-serialized input messages output : Complete response text tools : JSON-serialized tools configuration tools.count : Number of tools available tools.names : Comma-separated tool names system : System message content gen_ai.usage.input_tokens : Number of prompt tokens gen_ai.usage.output_tokens : Number of completion tokens metrics.ttfb : Time to first byte in seconds gen_ai.request.* : Standard parameters (temperature, max_tokens, etc.) ​ Multimodal Service Spans (Gemini Live & OpenAI Realtime) ​ Setup Spans gen_ai.system : “gcp.gemini” or “openai” gen_ai.request.model : Model identifier tools.count : Number of available tools tools.definitions : JSON-serialized tool schemas system_instruction : System prompt (truncated) session.* : Session configuration parameters ​ Request Spans (OpenAI Realtime) input : JSON-serialized context messages being sent gen_ai.operation.name : “llm_request” ​ Response Spans output : Complete assistant response text output_modality : “TEXT” or “AUDIO” (Gemini Live) gen_ai.usage.input_tokens : Prompt tokens used gen_ai.usage.output_tokens : Completion tokens generated function_calls.count : Number of function calls made function_calls.names : Comma-separated function names metrics.ttfb : Time to first response in seconds ​ Tool Call/Result Spans (Gemini Live) tool.function_name : Name of the function being called tool.call_id : Unique identifier for the call tool.arguments : Function arguments (truncated) tool.result : Function execution result (truncated) tool.result_status : “completed”, “error”, or “parse_error” ​ Turn Spans turn.number : Sequential turn number turn.type : Type of turn (e.g., “conversation”) turn.duration_seconds : Duration of the turn turn.was_interrupted : Whether the turn was interrupted conversation.id : ID of the parent conversation ​ Conversation Spans conversation.id : Unique identifier for the conversation conversation.type : Type of conversation (e.g., “voice”) ​ Usage Metrics Pipecat’s tracing implementation automatically captures usage metrics for LLM and TTS services: ​ LLM Token Usage Token usage is captured in LLM spans as: gen_ai.usage.input_tokens gen_ai.usage.output_tokens ​ TTS Character Count Character count is captured in TTS spans as: metrics.character_count ​ Performance Metrics Pipecat traces capture key performance metrics for each service: ​ Time To First Byte (TTFB) The time it takes for a service to produce its first response: metrics.ttfb (in seconds) ​ Processing Duration The total time spent processing in each service is captured in the span duration. ​ Configuration Options ​ PipelineTask Parameters ​ enable_tracing bool default: "True" Enable or disable tracing for the pipeline ​ enable_turn_tracking bool default: "False" Whether to enable turn tracking. ​ conversation_id Optional[str] default: "None" Custom ID for the conversation. If not provided, a UUID will be generated ​ additional_span_attributes Optional[dict] default: "None" Any additional attributes to add to top-level OpenTelemetry conversation span. ​ setup_tracing() Parameters ​ service_name str default: "pipecat" Name of the service for traces ​ exporter Optional[SpanExporter] default: "None" A pre-configured OpenTelemetry span exporter instance ​ console_export bool default: "False" Whether to also export traces to console (useful for debugging) ​ Example Here’s a complete example showing OpenTelemetry tracing setup with Jaeger: Copy Ask AI import os from dotenv import load_dotenv from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter from pipecat.pipeline.pipeline import Pipeline from pipecat.pipeline.task import PipelineParams, PipelineTask from pipecat.services.deepgram.stt import DeepgramSTTService from pipecat.services.openai.llm import OpenAILLMService from pipecat.services.cartesia.tts import CartesiaTTSService from pipecat.utils.tracing.setup import setup_tracing load_dotenv() # Initialize tracing if enabled if os.getenv( "ENABLE_TRACING" ): # Create the exporter otlp_exporter = OTLPSpanExporter( endpoint = os.getenv( "OTEL_EXPORTER_OTLP_ENDPOINT" , "http://localhost:4317" ), insecure = True , ) # Set up tracing with the exporter setup_tracing( service_name = "pipecat-demo" , exporter = otlp_exporter, console_export = bool (os.getenv( "OTEL_CONSOLE_EXPORT" )), ) # Create your services stt = DeepgramSTTService( api_key = os.getenv( "DEEPGRAM_API_KEY" )) llm = OpenAILLMService( api_key = os.getenv( "OPENAI_API_KEY" )) tts = CartesiaTTSService( api_key = os.getenv( "CARTESIA_API_KEY" ), voice_id = "your-voice-id" ) # Build pipeline pipeline = Pipeline([ transport.input(), stt, context_aggregator.user(), llm, tts, transport.output(), context_aggregator.assistant(), ]) # Create pipeline task with tracing enabled task = PipelineTask( pipeline, params = PipelineParams( allow_interruptions = True , enable_metrics = True , enable_usage_metrics = True , ), enable_tracing = True , enable_turn_tracking = True , conversation_id = "customer-123" , # Optional - will auto-generate if not provided additional_span_attributes = { "session.id" : "abc-123" } # Optional - additional attributes to attach to the otel span ) # Run the pipeline runner = PipelineRunner() await runner.run(task) ​ Troubleshooting If you’re having issues with tracing: No Traces Visible : Ensure the OpenTelemetry packages are installed and that your collector endpoint is correct Missing Service Data : Verify that enable_metrics=True is set in PipelineParams Debugging Tracing : Enable console export with console_export=True to view traces in your logs Connection Errors : Check network connectivity to your trace collector Collector Configuration : Verify your collector is properly set up to receive traces ​ References OpenTelemetry Python Documentation OpenTelemetry Tracing Specification Jaeger Documentation Langfuse OpenTelemetry Documentation WakeNotifierFilter MCPClient On this page Overview Installation Basic Setup Trace Structure Exporter Options OTLP Exporter (for Jaeger, Grafana, etc.) HTTP OTLP Exporter (for Langfuse, etc.) Console Exporter (for debugging) Cloud Provider Exporters Span Attributes TTS Service Spans STT Service Spans LLM Service Spans Multimodal Service Spans (Gemini Live & OpenAI Realtime) Setup Spans Request Spans (OpenAI Realtime) Response Spans Tool Call/Result Spans (Gemini Live) Turn Spans Conversation Spans Usage Metrics LLM Token Usage TTS Character Count Performance Metrics Time To First Byte (TTFB) Processing Duration Configuration Options PipelineTask Parameters setup_tracing() Parameters Example Troubleshooting References Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/utilities_opentelemetry_7633f2a7.txt b/utilities_opentelemetry_7633f2a7.txt
new file mode 100644
index 0000000000000000000000000000000000000000..4a88852b888931ae7f4781715f35c744418057c2
--- /dev/null
+++ b/utilities_opentelemetry_7633f2a7.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/opentelemetry#references
+Title: OpenTelemetry Tracing - Pipecat
+==================================================
+
+OpenTelemetry Tracing - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Metrics and Telemetry OpenTelemetry Tracing Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry OpenTelemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview Pipecat includes built-in support for OpenTelemetry tracing, allowing you to gain deep visibility into your voice applications. Tracing helps you: Track latency and performance across your conversation pipeline Monitor service health and identify bottlenecks Visualize conversation turns and service dependencies Collect usage metrics and operational analytics ​ Installation To use OpenTelemetry tracing with Pipecat, install the tracing dependencies: Copy Ask AI pip install "pipecat-ai[tracing]" For local development and testing, we recommend using Jaeger as a trace collector. You can run it with Docker: Copy Ask AI docker run -d --name jaeger \ -p 16686:16686 \ -p 4317:4317 \ -p 4318:4318 \ jaegertracing/all-in-one:latest Then access the UI at http://localhost:16686 ​ Basic Setup Enabling tracing in your Pipecat application requires two steps: Initialize the OpenTelemetry SDK with your preferred exporter Enable tracing in your PipelineTask Copy Ask AI import os from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter from pipecat.utils.tracing.setup import setup_tracing from pipecat.pipeline.task import PipelineTask, PipelineParams # Step 1: Initialize OpenTelemetry with your chosen exporter exporter = OTLPSpanExporter( endpoint = "http://localhost:4317" , # Jaeger or other collector endpoint insecure = True , ) setup_tracing( service_name = "my-voice-app" , exporter = exporter, console_export = False , # Set to True for debug output ) # Step 2: Enable tracing in your PipelineTask task = PipelineTask( pipeline, params = PipelineParams( enable_metrics = True , # Required for some service metrics ), enable_tracing = True , # Enable tracing for this task enable_turn_tracking = True , # Enable turn tracking for this task conversation_id = "customer-123" , # Optional - will auto-generate if not provided additional_span_attributes = { "session.id" : "abc-123" } # Optional - additional attributes to attach to the otel span ) For complete working examples, see our sample implementations: Jaeger Tracing Example - Uses gRPC exporter with Jaeger Langfuse Tracing Example - Uses HTTP exporter with Langfuse for LLM-focused observability ​ Trace Structure Pipecat organizes traces hierarchically, following the natural structure of conversations: Copy Ask AI Conversation (conversation) ├── turn │ ├── stt │ ├── llm │ └── tts └── turn ├── stt ├── llm └── tts turn... For real-time multimodal services like Gemini Live and OpenAI Realtime, the structure adapts to their specific patterns: Copy Ask AI Conversation (conversation) ├── turn │ ├── llm_setup (session configuration) │ ├── stt (user input) │ ├── llm_response (complete response with usage) │ └── llm_tool_call/llm_tool_result (for function calls) └── turn ├── stt (user input) └── llm_response (complete response) turn... This hierarchical structure makes it easy to: Track the full lifecycle of a conversation Measure latency for individual turns Identify which services are contributing to delays Compare performance across different conversations ​ Exporter Options Pipecat supports any OpenTelemetry-compatible exporter. Common options include: ​ OTLP Exporter (for Jaeger, Grafana, etc.) Copy Ask AI from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter exporter = OTLPSpanExporter( endpoint = "http://localhost:4317" , # Your collector endpoint insecure = True , # Use False for TLS connections ) ​ HTTP OTLP Exporter (for Langfuse, etc.) Copy Ask AI from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter exporter = OTLPSpanExporter( # Configure with environment variables: # OTEL_EXPORTER_OTLP_ENDPOINT # OTEL_EXPORTER_OTLP_HEADERS ) See our Langfuse example for details on configuring this exporter. ​ Console Exporter (for debugging) The console exporter can be enabled alongside any other exporter by setting console_export=True : Copy Ask AI setup_tracing( service_name = "my-voice-app" , exporter = otlp_exporter, console_export = True , # Prints traces to stdout ) ​ Cloud Provider Exporters Many cloud providers offer OpenTelemetry-compatible observability services: AWS X-Ray Google Cloud Trace Azure Monitor Datadog APM Check the OpenTelemetry documentation for specific exporter configurations: OpenTelemetry Vendors ​ Span Attributes Pipecat enriches spans with detailed attributes about service operations: ​ TTS Service Spans gen_ai.system : Service provider (e.g., “cartesia”) gen_ai.request.model : Model ID/name voice_id : Voice identifier text : The text being synthesized metrics.character_count : Number of characters in the text metrics.ttfb : Time to first byte in seconds settings.* : Service-specific configuration parameters ​ STT Service Spans gen_ai.system : Service provider (e.g., “deepgram”) gen_ai.request.model : Model ID/name transcript : The transcribed text is_final : Whether the transcription is final language : Detected or configured language vad_enabled : Whether voice activity detection is enabled metrics.ttfb : Time to first byte in seconds settings.* : Service-specific configuration parameters ​ LLM Service Spans gen_ai.system : Service provider (e.g., “openai”, “gcp.gemini”) gen_ai.request.model : Model ID/name gen_ai.operation.name : Operation type (e.g., “chat”) stream : Whether streaming is enabled input : JSON-serialized input messages output : Complete response text tools : JSON-serialized tools configuration tools.count : Number of tools available tools.names : Comma-separated tool names system : System message content gen_ai.usage.input_tokens : Number of prompt tokens gen_ai.usage.output_tokens : Number of completion tokens metrics.ttfb : Time to first byte in seconds gen_ai.request.* : Standard parameters (temperature, max_tokens, etc.) ​ Multimodal Service Spans (Gemini Live & OpenAI Realtime) ​ Setup Spans gen_ai.system : “gcp.gemini” or “openai” gen_ai.request.model : Model identifier tools.count : Number of available tools tools.definitions : JSON-serialized tool schemas system_instruction : System prompt (truncated) session.* : Session configuration parameters ​ Request Spans (OpenAI Realtime) input : JSON-serialized context messages being sent gen_ai.operation.name : “llm_request” ​ Response Spans output : Complete assistant response text output_modality : “TEXT” or “AUDIO” (Gemini Live) gen_ai.usage.input_tokens : Prompt tokens used gen_ai.usage.output_tokens : Completion tokens generated function_calls.count : Number of function calls made function_calls.names : Comma-separated function names metrics.ttfb : Time to first response in seconds ​ Tool Call/Result Spans (Gemini Live) tool.function_name : Name of the function being called tool.call_id : Unique identifier for the call tool.arguments : Function arguments (truncated) tool.result : Function execution result (truncated) tool.result_status : “completed”, “error”, or “parse_error” ​ Turn Spans turn.number : Sequential turn number turn.type : Type of turn (e.g., “conversation”) turn.duration_seconds : Duration of the turn turn.was_interrupted : Whether the turn was interrupted conversation.id : ID of the parent conversation ​ Conversation Spans conversation.id : Unique identifier for the conversation conversation.type : Type of conversation (e.g., “voice”) ​ Usage Metrics Pipecat’s tracing implementation automatically captures usage metrics for LLM and TTS services: ​ LLM Token Usage Token usage is captured in LLM spans as: gen_ai.usage.input_tokens gen_ai.usage.output_tokens ​ TTS Character Count Character count is captured in TTS spans as: metrics.character_count ​ Performance Metrics Pipecat traces capture key performance metrics for each service: ​ Time To First Byte (TTFB) The time it takes for a service to produce its first response: metrics.ttfb (in seconds) ​ Processing Duration The total time spent processing in each service is captured in the span duration. ​ Configuration Options ​ PipelineTask Parameters ​ enable_tracing bool default: "True" Enable or disable tracing for the pipeline ​ enable_turn_tracking bool default: "False" Whether to enable turn tracking. ​ conversation_id Optional[str] default: "None" Custom ID for the conversation. If not provided, a UUID will be generated ​ additional_span_attributes Optional[dict] default: "None" Any additional attributes to add to top-level OpenTelemetry conversation span. ​ setup_tracing() Parameters ​ service_name str default: "pipecat" Name of the service for traces ​ exporter Optional[SpanExporter] default: "None" A pre-configured OpenTelemetry span exporter instance ​ console_export bool default: "False" Whether to also export traces to console (useful for debugging) ​ Example Here’s a complete example showing OpenTelemetry tracing setup with Jaeger: Copy Ask AI import os from dotenv import load_dotenv from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter from pipecat.pipeline.pipeline import Pipeline from pipecat.pipeline.task import PipelineParams, PipelineTask from pipecat.services.deepgram.stt import DeepgramSTTService from pipecat.services.openai.llm import OpenAILLMService from pipecat.services.cartesia.tts import CartesiaTTSService from pipecat.utils.tracing.setup import setup_tracing load_dotenv() # Initialize tracing if enabled if os.getenv( "ENABLE_TRACING" ): # Create the exporter otlp_exporter = OTLPSpanExporter( endpoint = os.getenv( "OTEL_EXPORTER_OTLP_ENDPOINT" , "http://localhost:4317" ), insecure = True , ) # Set up tracing with the exporter setup_tracing( service_name = "pipecat-demo" , exporter = otlp_exporter, console_export = bool (os.getenv( "OTEL_CONSOLE_EXPORT" )), ) # Create your services stt = DeepgramSTTService( api_key = os.getenv( "DEEPGRAM_API_KEY" )) llm = OpenAILLMService( api_key = os.getenv( "OPENAI_API_KEY" )) tts = CartesiaTTSService( api_key = os.getenv( "CARTESIA_API_KEY" ), voice_id = "your-voice-id" ) # Build pipeline pipeline = Pipeline([ transport.input(), stt, context_aggregator.user(), llm, tts, transport.output(), context_aggregator.assistant(), ]) # Create pipeline task with tracing enabled task = PipelineTask( pipeline, params = PipelineParams( allow_interruptions = True , enable_metrics = True , enable_usage_metrics = True , ), enable_tracing = True , enable_turn_tracking = True , conversation_id = "customer-123" , # Optional - will auto-generate if not provided additional_span_attributes = { "session.id" : "abc-123" } # Optional - additional attributes to attach to the otel span ) # Run the pipeline runner = PipelineRunner() await runner.run(task) ​ Troubleshooting If you’re having issues with tracing: No Traces Visible : Ensure the OpenTelemetry packages are installed and that your collector endpoint is correct Missing Service Data : Verify that enable_metrics=True is set in PipelineParams Debugging Tracing : Enable console export with console_export=True to view traces in your logs Connection Errors : Check network connectivity to your trace collector Collector Configuration : Verify your collector is properly set up to receive traces ​ References OpenTelemetry Python Documentation OpenTelemetry Tracing Specification Jaeger Documentation Langfuse OpenTelemetry Documentation WakeNotifierFilter MCPClient On this page Overview Installation Basic Setup Trace Structure Exporter Options OTLP Exporter (for Jaeger, Grafana, etc.) HTTP OTLP Exporter (for Langfuse, etc.) Console Exporter (for debugging) Cloud Provider Exporters Span Attributes TTS Service Spans STT Service Spans LLM Service Spans Multimodal Service Spans (Gemini Live & OpenAI Realtime) Setup Spans Request Spans (OpenAI Realtime) Response Spans Tool Call/Result Spans (Gemini Live) Turn Spans Conversation Spans Usage Metrics LLM Token Usage TTS Character Count Performance Metrics Time To First Byte (TTFB) Processing Duration Configuration Options PipelineTask Parameters setup_tracing() Parameters Example Troubleshooting References Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/utilities_opentelemetry_8ad4c71f.txt b/utilities_opentelemetry_8ad4c71f.txt
new file mode 100644
index 0000000000000000000000000000000000000000..ed4a046fb077c4ead1c14a661161ac014bb7f9af
--- /dev/null
+++ b/utilities_opentelemetry_8ad4c71f.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/opentelemetry#cloud-provider-exporters
+Title: OpenTelemetry Tracing - Pipecat
+==================================================
+
+OpenTelemetry Tracing - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Metrics and Telemetry OpenTelemetry Tracing Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry OpenTelemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview Pipecat includes built-in support for OpenTelemetry tracing, allowing you to gain deep visibility into your voice applications. Tracing helps you: Track latency and performance across your conversation pipeline Monitor service health and identify bottlenecks Visualize conversation turns and service dependencies Collect usage metrics and operational analytics ​ Installation To use OpenTelemetry tracing with Pipecat, install the tracing dependencies: Copy Ask AI pip install "pipecat-ai[tracing]" For local development and testing, we recommend using Jaeger as a trace collector. You can run it with Docker: Copy Ask AI docker run -d --name jaeger \ -p 16686:16686 \ -p 4317:4317 \ -p 4318:4318 \ jaegertracing/all-in-one:latest Then access the UI at http://localhost:16686 ​ Basic Setup Enabling tracing in your Pipecat application requires two steps: Initialize the OpenTelemetry SDK with your preferred exporter Enable tracing in your PipelineTask Copy Ask AI import os from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter from pipecat.utils.tracing.setup import setup_tracing from pipecat.pipeline.task import PipelineTask, PipelineParams # Step 1: Initialize OpenTelemetry with your chosen exporter exporter = OTLPSpanExporter( endpoint = "http://localhost:4317" , # Jaeger or other collector endpoint insecure = True , ) setup_tracing( service_name = "my-voice-app" , exporter = exporter, console_export = False , # Set to True for debug output ) # Step 2: Enable tracing in your PipelineTask task = PipelineTask( pipeline, params = PipelineParams( enable_metrics = True , # Required for some service metrics ), enable_tracing = True , # Enable tracing for this task enable_turn_tracking = True , # Enable turn tracking for this task conversation_id = "customer-123" , # Optional - will auto-generate if not provided additional_span_attributes = { "session.id" : "abc-123" } # Optional - additional attributes to attach to the otel span ) For complete working examples, see our sample implementations: Jaeger Tracing Example - Uses gRPC exporter with Jaeger Langfuse Tracing Example - Uses HTTP exporter with Langfuse for LLM-focused observability ​ Trace Structure Pipecat organizes traces hierarchically, following the natural structure of conversations: Copy Ask AI Conversation (conversation) ├── turn │ ├── stt │ ├── llm │ └── tts └── turn ├── stt ├── llm └── tts turn... For real-time multimodal services like Gemini Live and OpenAI Realtime, the structure adapts to their specific patterns: Copy Ask AI Conversation (conversation) ├── turn │ ├── llm_setup (session configuration) │ ├── stt (user input) │ ├── llm_response (complete response with usage) │ └── llm_tool_call/llm_tool_result (for function calls) └── turn ├── stt (user input) └── llm_response (complete response) turn... This hierarchical structure makes it easy to: Track the full lifecycle of a conversation Measure latency for individual turns Identify which services are contributing to delays Compare performance across different conversations ​ Exporter Options Pipecat supports any OpenTelemetry-compatible exporter. Common options include: ​ OTLP Exporter (for Jaeger, Grafana, etc.) Copy Ask AI from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter exporter = OTLPSpanExporter( endpoint = "http://localhost:4317" , # Your collector endpoint insecure = True , # Use False for TLS connections ) ​ HTTP OTLP Exporter (for Langfuse, etc.) Copy Ask AI from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter exporter = OTLPSpanExporter( # Configure with environment variables: # OTEL_EXPORTER_OTLP_ENDPOINT # OTEL_EXPORTER_OTLP_HEADERS ) See our Langfuse example for details on configuring this exporter. ​ Console Exporter (for debugging) The console exporter can be enabled alongside any other exporter by setting console_export=True : Copy Ask AI setup_tracing( service_name = "my-voice-app" , exporter = otlp_exporter, console_export = True , # Prints traces to stdout ) ​ Cloud Provider Exporters Many cloud providers offer OpenTelemetry-compatible observability services: AWS X-Ray Google Cloud Trace Azure Monitor Datadog APM Check the OpenTelemetry documentation for specific exporter configurations: OpenTelemetry Vendors ​ Span Attributes Pipecat enriches spans with detailed attributes about service operations: ​ TTS Service Spans gen_ai.system : Service provider (e.g., “cartesia”) gen_ai.request.model : Model ID/name voice_id : Voice identifier text : The text being synthesized metrics.character_count : Number of characters in the text metrics.ttfb : Time to first byte in seconds settings.* : Service-specific configuration parameters ​ STT Service Spans gen_ai.system : Service provider (e.g., “deepgram”) gen_ai.request.model : Model ID/name transcript : The transcribed text is_final : Whether the transcription is final language : Detected or configured language vad_enabled : Whether voice activity detection is enabled metrics.ttfb : Time to first byte in seconds settings.* : Service-specific configuration parameters ​ LLM Service Spans gen_ai.system : Service provider (e.g., “openai”, “gcp.gemini”) gen_ai.request.model : Model ID/name gen_ai.operation.name : Operation type (e.g., “chat”) stream : Whether streaming is enabled input : JSON-serialized input messages output : Complete response text tools : JSON-serialized tools configuration tools.count : Number of tools available tools.names : Comma-separated tool names system : System message content gen_ai.usage.input_tokens : Number of prompt tokens gen_ai.usage.output_tokens : Number of completion tokens metrics.ttfb : Time to first byte in seconds gen_ai.request.* : Standard parameters (temperature, max_tokens, etc.) ​ Multimodal Service Spans (Gemini Live & OpenAI Realtime) ​ Setup Spans gen_ai.system : “gcp.gemini” or “openai” gen_ai.request.model : Model identifier tools.count : Number of available tools tools.definitions : JSON-serialized tool schemas system_instruction : System prompt (truncated) session.* : Session configuration parameters ​ Request Spans (OpenAI Realtime) input : JSON-serialized context messages being sent gen_ai.operation.name : “llm_request” ​ Response Spans output : Complete assistant response text output_modality : “TEXT” or “AUDIO” (Gemini Live) gen_ai.usage.input_tokens : Prompt tokens used gen_ai.usage.output_tokens : Completion tokens generated function_calls.count : Number of function calls made function_calls.names : Comma-separated function names metrics.ttfb : Time to first response in seconds ​ Tool Call/Result Spans (Gemini Live) tool.function_name : Name of the function being called tool.call_id : Unique identifier for the call tool.arguments : Function arguments (truncated) tool.result : Function execution result (truncated) tool.result_status : “completed”, “error”, or “parse_error” ​ Turn Spans turn.number : Sequential turn number turn.type : Type of turn (e.g., “conversation”) turn.duration_seconds : Duration of the turn turn.was_interrupted : Whether the turn was interrupted conversation.id : ID of the parent conversation ​ Conversation Spans conversation.id : Unique identifier for the conversation conversation.type : Type of conversation (e.g., “voice”) ​ Usage Metrics Pipecat’s tracing implementation automatically captures usage metrics for LLM and TTS services: ​ LLM Token Usage Token usage is captured in LLM spans as: gen_ai.usage.input_tokens gen_ai.usage.output_tokens ​ TTS Character Count Character count is captured in TTS spans as: metrics.character_count ​ Performance Metrics Pipecat traces capture key performance metrics for each service: ​ Time To First Byte (TTFB) The time it takes for a service to produce its first response: metrics.ttfb (in seconds) ​ Processing Duration The total time spent processing in each service is captured in the span duration. ​ Configuration Options ​ PipelineTask Parameters ​ enable_tracing bool default: "True" Enable or disable tracing for the pipeline ​ enable_turn_tracking bool default: "False" Whether to enable turn tracking. ​ conversation_id Optional[str] default: "None" Custom ID for the conversation. If not provided, a UUID will be generated ​ additional_span_attributes Optional[dict] default: "None" Any additional attributes to add to top-level OpenTelemetry conversation span. ​ setup_tracing() Parameters ​ service_name str default: "pipecat" Name of the service for traces ​ exporter Optional[SpanExporter] default: "None" A pre-configured OpenTelemetry span exporter instance ​ console_export bool default: "False" Whether to also export traces to console (useful for debugging) ​ Example Here’s a complete example showing OpenTelemetry tracing setup with Jaeger: Copy Ask AI import os from dotenv import load_dotenv from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter from pipecat.pipeline.pipeline import Pipeline from pipecat.pipeline.task import PipelineParams, PipelineTask from pipecat.services.deepgram.stt import DeepgramSTTService from pipecat.services.openai.llm import OpenAILLMService from pipecat.services.cartesia.tts import CartesiaTTSService from pipecat.utils.tracing.setup import setup_tracing load_dotenv() # Initialize tracing if enabled if os.getenv( "ENABLE_TRACING" ): # Create the exporter otlp_exporter = OTLPSpanExporter( endpoint = os.getenv( "OTEL_EXPORTER_OTLP_ENDPOINT" , "http://localhost:4317" ), insecure = True , ) # Set up tracing with the exporter setup_tracing( service_name = "pipecat-demo" , exporter = otlp_exporter, console_export = bool (os.getenv( "OTEL_CONSOLE_EXPORT" )), ) # Create your services stt = DeepgramSTTService( api_key = os.getenv( "DEEPGRAM_API_KEY" )) llm = OpenAILLMService( api_key = os.getenv( "OPENAI_API_KEY" )) tts = CartesiaTTSService( api_key = os.getenv( "CARTESIA_API_KEY" ), voice_id = "your-voice-id" ) # Build pipeline pipeline = Pipeline([ transport.input(), stt, context_aggregator.user(), llm, tts, transport.output(), context_aggregator.assistant(), ]) # Create pipeline task with tracing enabled task = PipelineTask( pipeline, params = PipelineParams( allow_interruptions = True , enable_metrics = True , enable_usage_metrics = True , ), enable_tracing = True , enable_turn_tracking = True , conversation_id = "customer-123" , # Optional - will auto-generate if not provided additional_span_attributes = { "session.id" : "abc-123" } # Optional - additional attributes to attach to the otel span ) # Run the pipeline runner = PipelineRunner() await runner.run(task) ​ Troubleshooting If you’re having issues with tracing: No Traces Visible : Ensure the OpenTelemetry packages are installed and that your collector endpoint is correct Missing Service Data : Verify that enable_metrics=True is set in PipelineParams Debugging Tracing : Enable console export with console_export=True to view traces in your logs Connection Errors : Check network connectivity to your trace collector Collector Configuration : Verify your collector is properly set up to receive traces ​ References OpenTelemetry Python Documentation OpenTelemetry Tracing Specification Jaeger Documentation Langfuse OpenTelemetry Documentation WakeNotifierFilter MCPClient On this page Overview Installation Basic Setup Trace Structure Exporter Options OTLP Exporter (for Jaeger, Grafana, etc.) HTTP OTLP Exporter (for Langfuse, etc.) Console Exporter (for debugging) Cloud Provider Exporters Span Attributes TTS Service Spans STT Service Spans LLM Service Spans Multimodal Service Spans (Gemini Live & OpenAI Realtime) Setup Spans Request Spans (OpenAI Realtime) Response Spans Tool Call/Result Spans (Gemini Live) Turn Spans Conversation Spans Usage Metrics LLM Token Usage TTS Character Count Performance Metrics Time To First Byte (TTFB) Processing Duration Configuration Options PipelineTask Parameters setup_tracing() Parameters Example Troubleshooting References Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/utilities_opentelemetry_e29f4055.txt b/utilities_opentelemetry_e29f4055.txt
new file mode 100644
index 0000000000000000000000000000000000000000..e3f9fbecc0d07ec4ca307b294025f009b2923521
--- /dev/null
+++ b/utilities_opentelemetry_e29f4055.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/opentelemetry#response-spans
+Title: OpenTelemetry Tracing - Pipecat
+==================================================
+
+OpenTelemetry Tracing - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Metrics and Telemetry OpenTelemetry Tracing Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry OpenTelemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview Pipecat includes built-in support for OpenTelemetry tracing, allowing you to gain deep visibility into your voice applications. Tracing helps you: Track latency and performance across your conversation pipeline Monitor service health and identify bottlenecks Visualize conversation turns and service dependencies Collect usage metrics and operational analytics ​ Installation To use OpenTelemetry tracing with Pipecat, install the tracing dependencies: Copy Ask AI pip install "pipecat-ai[tracing]" For local development and testing, we recommend using Jaeger as a trace collector. You can run it with Docker: Copy Ask AI docker run -d --name jaeger \ -p 16686:16686 \ -p 4317:4317 \ -p 4318:4318 \ jaegertracing/all-in-one:latest Then access the UI at http://localhost:16686 ​ Basic Setup Enabling tracing in your Pipecat application requires two steps: Initialize the OpenTelemetry SDK with your preferred exporter Enable tracing in your PipelineTask Copy Ask AI import os from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter from pipecat.utils.tracing.setup import setup_tracing from pipecat.pipeline.task import PipelineTask, PipelineParams # Step 1: Initialize OpenTelemetry with your chosen exporter exporter = OTLPSpanExporter( endpoint = "http://localhost:4317" , # Jaeger or other collector endpoint insecure = True , ) setup_tracing( service_name = "my-voice-app" , exporter = exporter, console_export = False , # Set to True for debug output ) # Step 2: Enable tracing in your PipelineTask task = PipelineTask( pipeline, params = PipelineParams( enable_metrics = True , # Required for some service metrics ), enable_tracing = True , # Enable tracing for this task enable_turn_tracking = True , # Enable turn tracking for this task conversation_id = "customer-123" , # Optional - will auto-generate if not provided additional_span_attributes = { "session.id" : "abc-123" } # Optional - additional attributes to attach to the otel span ) For complete working examples, see our sample implementations: Jaeger Tracing Example - Uses gRPC exporter with Jaeger Langfuse Tracing Example - Uses HTTP exporter with Langfuse for LLM-focused observability ​ Trace Structure Pipecat organizes traces hierarchically, following the natural structure of conversations: Copy Ask AI Conversation (conversation) ├── turn │ ├── stt │ ├── llm │ └── tts └── turn ├── stt ├── llm └── tts turn... For real-time multimodal services like Gemini Live and OpenAI Realtime, the structure adapts to their specific patterns: Copy Ask AI Conversation (conversation) ├── turn │ ├── llm_setup (session configuration) │ ├── stt (user input) │ ├── llm_response (complete response with usage) │ └── llm_tool_call/llm_tool_result (for function calls) └── turn ├── stt (user input) └── llm_response (complete response) turn... This hierarchical structure makes it easy to: Track the full lifecycle of a conversation Measure latency for individual turns Identify which services are contributing to delays Compare performance across different conversations ​ Exporter Options Pipecat supports any OpenTelemetry-compatible exporter. Common options include: ​ OTLP Exporter (for Jaeger, Grafana, etc.) Copy Ask AI from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter exporter = OTLPSpanExporter( endpoint = "http://localhost:4317" , # Your collector endpoint insecure = True , # Use False for TLS connections ) ​ HTTP OTLP Exporter (for Langfuse, etc.) Copy Ask AI from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter exporter = OTLPSpanExporter( # Configure with environment variables: # OTEL_EXPORTER_OTLP_ENDPOINT # OTEL_EXPORTER_OTLP_HEADERS ) See our Langfuse example for details on configuring this exporter. ​ Console Exporter (for debugging) The console exporter can be enabled alongside any other exporter by setting console_export=True : Copy Ask AI setup_tracing( service_name = "my-voice-app" , exporter = otlp_exporter, console_export = True , # Prints traces to stdout ) ​ Cloud Provider Exporters Many cloud providers offer OpenTelemetry-compatible observability services: AWS X-Ray Google Cloud Trace Azure Monitor Datadog APM Check the OpenTelemetry documentation for specific exporter configurations: OpenTelemetry Vendors ​ Span Attributes Pipecat enriches spans with detailed attributes about service operations: ​ TTS Service Spans gen_ai.system : Service provider (e.g., “cartesia”) gen_ai.request.model : Model ID/name voice_id : Voice identifier text : The text being synthesized metrics.character_count : Number of characters in the text metrics.ttfb : Time to first byte in seconds settings.* : Service-specific configuration parameters ​ STT Service Spans gen_ai.system : Service provider (e.g., “deepgram”) gen_ai.request.model : Model ID/name transcript : The transcribed text is_final : Whether the transcription is final language : Detected or configured language vad_enabled : Whether voice activity detection is enabled metrics.ttfb : Time to first byte in seconds settings.* : Service-specific configuration parameters ​ LLM Service Spans gen_ai.system : Service provider (e.g., “openai”, “gcp.gemini”) gen_ai.request.model : Model ID/name gen_ai.operation.name : Operation type (e.g., “chat”) stream : Whether streaming is enabled input : JSON-serialized input messages output : Complete response text tools : JSON-serialized tools configuration tools.count : Number of tools available tools.names : Comma-separated tool names system : System message content gen_ai.usage.input_tokens : Number of prompt tokens gen_ai.usage.output_tokens : Number of completion tokens metrics.ttfb : Time to first byte in seconds gen_ai.request.* : Standard parameters (temperature, max_tokens, etc.) ​ Multimodal Service Spans (Gemini Live & OpenAI Realtime) ​ Setup Spans gen_ai.system : “gcp.gemini” or “openai” gen_ai.request.model : Model identifier tools.count : Number of available tools tools.definitions : JSON-serialized tool schemas system_instruction : System prompt (truncated) session.* : Session configuration parameters ​ Request Spans (OpenAI Realtime) input : JSON-serialized context messages being sent gen_ai.operation.name : “llm_request” ​ Response Spans output : Complete assistant response text output_modality : “TEXT” or “AUDIO” (Gemini Live) gen_ai.usage.input_tokens : Prompt tokens used gen_ai.usage.output_tokens : Completion tokens generated function_calls.count : Number of function calls made function_calls.names : Comma-separated function names metrics.ttfb : Time to first response in seconds ​ Tool Call/Result Spans (Gemini Live) tool.function_name : Name of the function being called tool.call_id : Unique identifier for the call tool.arguments : Function arguments (truncated) tool.result : Function execution result (truncated) tool.result_status : “completed”, “error”, or “parse_error” ​ Turn Spans turn.number : Sequential turn number turn.type : Type of turn (e.g., “conversation”) turn.duration_seconds : Duration of the turn turn.was_interrupted : Whether the turn was interrupted conversation.id : ID of the parent conversation ​ Conversation Spans conversation.id : Unique identifier for the conversation conversation.type : Type of conversation (e.g., “voice”) ​ Usage Metrics Pipecat’s tracing implementation automatically captures usage metrics for LLM and TTS services: ​ LLM Token Usage Token usage is captured in LLM spans as: gen_ai.usage.input_tokens gen_ai.usage.output_tokens ​ TTS Character Count Character count is captured in TTS spans as: metrics.character_count ​ Performance Metrics Pipecat traces capture key performance metrics for each service: ​ Time To First Byte (TTFB) The time it takes for a service to produce its first response: metrics.ttfb (in seconds) ​ Processing Duration The total time spent processing in each service is captured in the span duration. ​ Configuration Options ​ PipelineTask Parameters ​ enable_tracing bool default: "True" Enable or disable tracing for the pipeline ​ enable_turn_tracking bool default: "False" Whether to enable turn tracking. ​ conversation_id Optional[str] default: "None" Custom ID for the conversation. If not provided, a UUID will be generated ​ additional_span_attributes Optional[dict] default: "None" Any additional attributes to add to top-level OpenTelemetry conversation span. ​ setup_tracing() Parameters ​ service_name str default: "pipecat" Name of the service for traces ​ exporter Optional[SpanExporter] default: "None" A pre-configured OpenTelemetry span exporter instance ​ console_export bool default: "False" Whether to also export traces to console (useful for debugging) ​ Example Here’s a complete example showing OpenTelemetry tracing setup with Jaeger: Copy Ask AI import os from dotenv import load_dotenv from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter from pipecat.pipeline.pipeline import Pipeline from pipecat.pipeline.task import PipelineParams, PipelineTask from pipecat.services.deepgram.stt import DeepgramSTTService from pipecat.services.openai.llm import OpenAILLMService from pipecat.services.cartesia.tts import CartesiaTTSService from pipecat.utils.tracing.setup import setup_tracing load_dotenv() # Initialize tracing if enabled if os.getenv( "ENABLE_TRACING" ): # Create the exporter otlp_exporter = OTLPSpanExporter( endpoint = os.getenv( "OTEL_EXPORTER_OTLP_ENDPOINT" , "http://localhost:4317" ), insecure = True , ) # Set up tracing with the exporter setup_tracing( service_name = "pipecat-demo" , exporter = otlp_exporter, console_export = bool (os.getenv( "OTEL_CONSOLE_EXPORT" )), ) # Create your services stt = DeepgramSTTService( api_key = os.getenv( "DEEPGRAM_API_KEY" )) llm = OpenAILLMService( api_key = os.getenv( "OPENAI_API_KEY" )) tts = CartesiaTTSService( api_key = os.getenv( "CARTESIA_API_KEY" ), voice_id = "your-voice-id" ) # Build pipeline pipeline = Pipeline([ transport.input(), stt, context_aggregator.user(), llm, tts, transport.output(), context_aggregator.assistant(), ]) # Create pipeline task with tracing enabled task = PipelineTask( pipeline, params = PipelineParams( allow_interruptions = True , enable_metrics = True , enable_usage_metrics = True , ), enable_tracing = True , enable_turn_tracking = True , conversation_id = "customer-123" , # Optional - will auto-generate if not provided additional_span_attributes = { "session.id" : "abc-123" } # Optional - additional attributes to attach to the otel span ) # Run the pipeline runner = PipelineRunner() await runner.run(task) ​ Troubleshooting If you’re having issues with tracing: No Traces Visible : Ensure the OpenTelemetry packages are installed and that your collector endpoint is correct Missing Service Data : Verify that enable_metrics=True is set in PipelineParams Debugging Tracing : Enable console export with console_export=True to view traces in your logs Connection Errors : Check network connectivity to your trace collector Collector Configuration : Verify your collector is properly set up to receive traces ​ References OpenTelemetry Python Documentation OpenTelemetry Tracing Specification Jaeger Documentation Langfuse OpenTelemetry Documentation WakeNotifierFilter MCPClient On this page Overview Installation Basic Setup Trace Structure Exporter Options OTLP Exporter (for Jaeger, Grafana, etc.) HTTP OTLP Exporter (for Langfuse, etc.) Console Exporter (for debugging) Cloud Provider Exporters Span Attributes TTS Service Spans STT Service Spans LLM Service Spans Multimodal Service Spans (Gemini Live & OpenAI Realtime) Setup Spans Request Spans (OpenAI Realtime) Response Spans Tool Call/Result Spans (Gemini Live) Turn Spans Conversation Spans Usage Metrics LLM Token Usage TTS Character Count Performance Metrics Time To First Byte (TTFB) Processing Duration Configuration Options PipelineTask Parameters setup_tracing() Parameters Example Troubleshooting References Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/utilities_opentelemetry_f5338ca0.txt b/utilities_opentelemetry_f5338ca0.txt
new file mode 100644
index 0000000000000000000000000000000000000000..39cadfdfd4a0942973686e6f3b0ca335573c367d
--- /dev/null
+++ b/utilities_opentelemetry_f5338ca0.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/opentelemetry#usage-metrics
+Title: OpenTelemetry Tracing - Pipecat
+==================================================
+
+OpenTelemetry Tracing - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Metrics and Telemetry OpenTelemetry Tracing Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry OpenTelemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview Pipecat includes built-in support for OpenTelemetry tracing, allowing you to gain deep visibility into your voice applications. Tracing helps you: Track latency and performance across your conversation pipeline Monitor service health and identify bottlenecks Visualize conversation turns and service dependencies Collect usage metrics and operational analytics ​ Installation To use OpenTelemetry tracing with Pipecat, install the tracing dependencies: Copy Ask AI pip install "pipecat-ai[tracing]" For local development and testing, we recommend using Jaeger as a trace collector. You can run it with Docker: Copy Ask AI docker run -d --name jaeger \ -p 16686:16686 \ -p 4317:4317 \ -p 4318:4318 \ jaegertracing/all-in-one:latest Then access the UI at http://localhost:16686 ​ Basic Setup Enabling tracing in your Pipecat application requires two steps: Initialize the OpenTelemetry SDK with your preferred exporter Enable tracing in your PipelineTask Copy Ask AI import os from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter from pipecat.utils.tracing.setup import setup_tracing from pipecat.pipeline.task import PipelineTask, PipelineParams # Step 1: Initialize OpenTelemetry with your chosen exporter exporter = OTLPSpanExporter( endpoint = "http://localhost:4317" , # Jaeger or other collector endpoint insecure = True , ) setup_tracing( service_name = "my-voice-app" , exporter = exporter, console_export = False , # Set to True for debug output ) # Step 2: Enable tracing in your PipelineTask task = PipelineTask( pipeline, params = PipelineParams( enable_metrics = True , # Required for some service metrics ), enable_tracing = True , # Enable tracing for this task enable_turn_tracking = True , # Enable turn tracking for this task conversation_id = "customer-123" , # Optional - will auto-generate if not provided additional_span_attributes = { "session.id" : "abc-123" } # Optional - additional attributes to attach to the otel span ) For complete working examples, see our sample implementations: Jaeger Tracing Example - Uses gRPC exporter with Jaeger Langfuse Tracing Example - Uses HTTP exporter with Langfuse for LLM-focused observability ​ Trace Structure Pipecat organizes traces hierarchically, following the natural structure of conversations: Copy Ask AI Conversation (conversation) ├── turn │ ├── stt │ ├── llm │ └── tts └── turn ├── stt ├── llm └── tts turn... For real-time multimodal services like Gemini Live and OpenAI Realtime, the structure adapts to their specific patterns: Copy Ask AI Conversation (conversation) ├── turn │ ├── llm_setup (session configuration) │ ├── stt (user input) │ ├── llm_response (complete response with usage) │ └── llm_tool_call/llm_tool_result (for function calls) └── turn ├── stt (user input) └── llm_response (complete response) turn... This hierarchical structure makes it easy to: Track the full lifecycle of a conversation Measure latency for individual turns Identify which services are contributing to delays Compare performance across different conversations ​ Exporter Options Pipecat supports any OpenTelemetry-compatible exporter. Common options include: ​ OTLP Exporter (for Jaeger, Grafana, etc.) Copy Ask AI from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter exporter = OTLPSpanExporter( endpoint = "http://localhost:4317" , # Your collector endpoint insecure = True , # Use False for TLS connections ) ​ HTTP OTLP Exporter (for Langfuse, etc.) Copy Ask AI from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter exporter = OTLPSpanExporter( # Configure with environment variables: # OTEL_EXPORTER_OTLP_ENDPOINT # OTEL_EXPORTER_OTLP_HEADERS ) See our Langfuse example for details on configuring this exporter. ​ Console Exporter (for debugging) The console exporter can be enabled alongside any other exporter by setting console_export=True : Copy Ask AI setup_tracing( service_name = "my-voice-app" , exporter = otlp_exporter, console_export = True , # Prints traces to stdout ) ​ Cloud Provider Exporters Many cloud providers offer OpenTelemetry-compatible observability services: AWS X-Ray Google Cloud Trace Azure Monitor Datadog APM Check the OpenTelemetry documentation for specific exporter configurations: OpenTelemetry Vendors ​ Span Attributes Pipecat enriches spans with detailed attributes about service operations: ​ TTS Service Spans gen_ai.system : Service provider (e.g., “cartesia”) gen_ai.request.model : Model ID/name voice_id : Voice identifier text : The text being synthesized metrics.character_count : Number of characters in the text metrics.ttfb : Time to first byte in seconds settings.* : Service-specific configuration parameters ​ STT Service Spans gen_ai.system : Service provider (e.g., “deepgram”) gen_ai.request.model : Model ID/name transcript : The transcribed text is_final : Whether the transcription is final language : Detected or configured language vad_enabled : Whether voice activity detection is enabled metrics.ttfb : Time to first byte in seconds settings.* : Service-specific configuration parameters ​ LLM Service Spans gen_ai.system : Service provider (e.g., “openai”, “gcp.gemini”) gen_ai.request.model : Model ID/name gen_ai.operation.name : Operation type (e.g., “chat”) stream : Whether streaming is enabled input : JSON-serialized input messages output : Complete response text tools : JSON-serialized tools configuration tools.count : Number of tools available tools.names : Comma-separated tool names system : System message content gen_ai.usage.input_tokens : Number of prompt tokens gen_ai.usage.output_tokens : Number of completion tokens metrics.ttfb : Time to first byte in seconds gen_ai.request.* : Standard parameters (temperature, max_tokens, etc.) ​ Multimodal Service Spans (Gemini Live & OpenAI Realtime) ​ Setup Spans gen_ai.system : “gcp.gemini” or “openai” gen_ai.request.model : Model identifier tools.count : Number of available tools tools.definitions : JSON-serialized tool schemas system_instruction : System prompt (truncated) session.* : Session configuration parameters ​ Request Spans (OpenAI Realtime) input : JSON-serialized context messages being sent gen_ai.operation.name : “llm_request” ​ Response Spans output : Complete assistant response text output_modality : “TEXT” or “AUDIO” (Gemini Live) gen_ai.usage.input_tokens : Prompt tokens used gen_ai.usage.output_tokens : Completion tokens generated function_calls.count : Number of function calls made function_calls.names : Comma-separated function names metrics.ttfb : Time to first response in seconds ​ Tool Call/Result Spans (Gemini Live) tool.function_name : Name of the function being called tool.call_id : Unique identifier for the call tool.arguments : Function arguments (truncated) tool.result : Function execution result (truncated) tool.result_status : “completed”, “error”, or “parse_error” ​ Turn Spans turn.number : Sequential turn number turn.type : Type of turn (e.g., “conversation”) turn.duration_seconds : Duration of the turn turn.was_interrupted : Whether the turn was interrupted conversation.id : ID of the parent conversation ​ Conversation Spans conversation.id : Unique identifier for the conversation conversation.type : Type of conversation (e.g., “voice”) ​ Usage Metrics Pipecat’s tracing implementation automatically captures usage metrics for LLM and TTS services: ​ LLM Token Usage Token usage is captured in LLM spans as: gen_ai.usage.input_tokens gen_ai.usage.output_tokens ​ TTS Character Count Character count is captured in TTS spans as: metrics.character_count ​ Performance Metrics Pipecat traces capture key performance metrics for each service: ​ Time To First Byte (TTFB) The time it takes for a service to produce its first response: metrics.ttfb (in seconds) ​ Processing Duration The total time spent processing in each service is captured in the span duration. ​ Configuration Options ​ PipelineTask Parameters ​ enable_tracing bool default: "True" Enable or disable tracing for the pipeline ​ enable_turn_tracking bool default: "False" Whether to enable turn tracking. ​ conversation_id Optional[str] default: "None" Custom ID for the conversation. If not provided, a UUID will be generated ​ additional_span_attributes Optional[dict] default: "None" Any additional attributes to add to top-level OpenTelemetry conversation span. ​ setup_tracing() Parameters ​ service_name str default: "pipecat" Name of the service for traces ​ exporter Optional[SpanExporter] default: "None" A pre-configured OpenTelemetry span exporter instance ​ console_export bool default: "False" Whether to also export traces to console (useful for debugging) ​ Example Here’s a complete example showing OpenTelemetry tracing setup with Jaeger: Copy Ask AI import os from dotenv import load_dotenv from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter from pipecat.pipeline.pipeline import Pipeline from pipecat.pipeline.task import PipelineParams, PipelineTask from pipecat.services.deepgram.stt import DeepgramSTTService from pipecat.services.openai.llm import OpenAILLMService from pipecat.services.cartesia.tts import CartesiaTTSService from pipecat.utils.tracing.setup import setup_tracing load_dotenv() # Initialize tracing if enabled if os.getenv( "ENABLE_TRACING" ): # Create the exporter otlp_exporter = OTLPSpanExporter( endpoint = os.getenv( "OTEL_EXPORTER_OTLP_ENDPOINT" , "http://localhost:4317" ), insecure = True , ) # Set up tracing with the exporter setup_tracing( service_name = "pipecat-demo" , exporter = otlp_exporter, console_export = bool (os.getenv( "OTEL_CONSOLE_EXPORT" )), ) # Create your services stt = DeepgramSTTService( api_key = os.getenv( "DEEPGRAM_API_KEY" )) llm = OpenAILLMService( api_key = os.getenv( "OPENAI_API_KEY" )) tts = CartesiaTTSService( api_key = os.getenv( "CARTESIA_API_KEY" ), voice_id = "your-voice-id" ) # Build pipeline pipeline = Pipeline([ transport.input(), stt, context_aggregator.user(), llm, tts, transport.output(), context_aggregator.assistant(), ]) # Create pipeline task with tracing enabled task = PipelineTask( pipeline, params = PipelineParams( allow_interruptions = True , enable_metrics = True , enable_usage_metrics = True , ), enable_tracing = True , enable_turn_tracking = True , conversation_id = "customer-123" , # Optional - will auto-generate if not provided additional_span_attributes = { "session.id" : "abc-123" } # Optional - additional attributes to attach to the otel span ) # Run the pipeline runner = PipelineRunner() await runner.run(task) ​ Troubleshooting If you’re having issues with tracing: No Traces Visible : Ensure the OpenTelemetry packages are installed and that your collector endpoint is correct Missing Service Data : Verify that enable_metrics=True is set in PipelineParams Debugging Tracing : Enable console export with console_export=True to view traces in your logs Connection Errors : Check network connectivity to your trace collector Collector Configuration : Verify your collector is properly set up to receive traces ​ References OpenTelemetry Python Documentation OpenTelemetry Tracing Specification Jaeger Documentation Langfuse OpenTelemetry Documentation WakeNotifierFilter MCPClient On this page Overview Installation Basic Setup Trace Structure Exporter Options OTLP Exporter (for Jaeger, Grafana, etc.) HTTP OTLP Exporter (for Langfuse, etc.) Console Exporter (for debugging) Cloud Provider Exporters Span Attributes TTS Service Spans STT Service Spans LLM Service Spans Multimodal Service Spans (Gemini Live & OpenAI Realtime) Setup Spans Request Spans (OpenAI Realtime) Response Spans Tool Call/Result Spans (Gemini Live) Turn Spans Conversation Spans Usage Metrics LLM Token Usage TTS Character Count Performance Metrics Time To First Byte (TTFB) Processing Duration Configuration Options PipelineTask Parameters setup_tracing() Parameters Example Troubleshooting References Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/utilities_transcript-processor_3b78fe10.txt b/utilities_transcript-processor_3b78fe10.txt
new file mode 100644
index 0000000000000000000000000000000000000000..005098071233970ba5064b7f8caa78a2f6ce0d40
--- /dev/null
+++ b/utilities_transcript-processor_3b78fe10.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/transcript-processor#event-handler
+Title: TranscriptProcessor - Pipecat
+==================================================
+
+TranscriptProcessor - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation User and Bot Transcriptions TranscriptProcessor Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions TranscriptProcessor User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview The TranscriptProcessor is a factory class that creates and manages processors for handling conversation transcripts from both users and assistants. It provides unified access to transcript processors with shared event handling, making it easy to track and respond to conversation updates in real-time. The processor normalizes messages from various sources into a consistent TranscriptionMessage format and emits events when new messages are added to the conversation. ​ Constructor Copy Ask AI TranscriptProcessor() Creates a new transcript processor factory with no parameters. ​ Methods ​ user() Copy Ask AI def user ( ** kwargs ) -> UserTranscriptProcessor Get or create the user transcript processor instance. This processor handles TranscriptionFrame s from STT services. Parameters: **kwargs : Arguments passed to the UserTranscriptProcessor constructor Returns: UserTranscriptProcessor instance for processing user messages. ​ assistant() Copy Ask AI def assistant ( ** kwargs ) -> AssistantTranscriptProcessor Get or create the assistant transcript processor instance. This processor handles TTSTextFrame s from TTS services and aggregates them into complete utterances. Parameters: **kwargs : Arguments passed to the AssistantTranscriptProcessor constructor Returns: AssistantTranscriptProcessor instance for processing assistant messages. ​ event_handler() Copy Ask AI def event_handler ( event_name : str ) Decorator that registers event handlers for both user and assistant processors. Parameters: event_name : Name of the event to handle Returns: Decorator function that registers the handler with both processors. ​ Event Handlers ​ on_transcript_update Triggered when new messages are added to the conversation transcript. Copy Ask AI @transcript.event_handler ( "on_transcript_update" ) async def handle_transcript_update ( processor , frame ): # Handle transcript updates pass Parameters: processor : The specific processor instance that emitted the event (UserTranscriptProcessor or AssistantTranscriptProcessor) frame : TranscriptionUpdateFrame containing the new messages ​ Data Structures ​ TranscriptionMessage Copy Ask AI @dataclass class TranscriptionMessage : role: Literal[ "user" , "assistant" ] content: str timestamp: str | None = None user_id: str | None = None Fields: role : The message sender type (“user” or “assistant”) content : The transcribed text content timestamp : ISO 8601 timestamp when the message was created user_id : Optional user identifier (for user messages only) ​ TranscriptionUpdateFrame Frame containing new transcript messages, emitted by the on_transcript_update event. Properties: messages : List of TranscriptionMessage objects containing the new transcript content ​ Frames ​ UserTranscriptProcessor Input: TranscriptionFrame from STT services Output: TranscriptionMessage with role “user” ​ AssistantTranscriptProcessor Input: TTSTextFrame from TTS services Output: TranscriptionMessage with role “assistant” ​ Integration Notes ​ Pipeline Placement Place the processors at specific positions in your pipeline for accurate transcript collection: Copy Ask AI pipeline = Pipeline([ transport.input(), stt, # Speech-to-text service transcript.user(), # Place after STT context_aggregator.user(), llm, tts, # Text-to-speech service transport.output(), transcript.assistant(), # Place after transport.output() context_aggregator.assistant(), ]) ​ Event Handler Registration Event handlers are automatically applied to both user and assistant processors: Copy Ask AI transcript = TranscriptProcessor() # This handler will receive events from both processors @transcript.event_handler ( "on_transcript_update" ) async def handle_update ( processor , frame ): for message in frame.messages: print ( f " { message.role } : { message.content } " ) PatternPairAggregator Interruption Strategies On this page Overview Constructor Methods user() assistant() event_handler() Event Handlers on_transcript_update Data Structures TranscriptionMessage TranscriptionUpdateFrame Frames UserTranscriptProcessor AssistantTranscriptProcessor Integration Notes Pipeline Placement Event Handler Registration Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/utilities_transcript-processor_61c115cb.txt b/utilities_transcript-processor_61c115cb.txt
new file mode 100644
index 0000000000000000000000000000000000000000..6c698a1d9554256e180bf3eeeefee5f97c4a93d1
--- /dev/null
+++ b/utilities_transcript-processor_61c115cb.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/transcript-processor#on-transcript-update
+Title: TranscriptProcessor - Pipecat
+==================================================
+
+TranscriptProcessor - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation User and Bot Transcriptions TranscriptProcessor Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions TranscriptProcessor User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview The TranscriptProcessor is a factory class that creates and manages processors for handling conversation transcripts from both users and assistants. It provides unified access to transcript processors with shared event handling, making it easy to track and respond to conversation updates in real-time. The processor normalizes messages from various sources into a consistent TranscriptionMessage format and emits events when new messages are added to the conversation. ​ Constructor Copy Ask AI TranscriptProcessor() Creates a new transcript processor factory with no parameters. ​ Methods ​ user() Copy Ask AI def user ( ** kwargs ) -> UserTranscriptProcessor Get or create the user transcript processor instance. This processor handles TranscriptionFrame s from STT services. Parameters: **kwargs : Arguments passed to the UserTranscriptProcessor constructor Returns: UserTranscriptProcessor instance for processing user messages. ​ assistant() Copy Ask AI def assistant ( ** kwargs ) -> AssistantTranscriptProcessor Get or create the assistant transcript processor instance. This processor handles TTSTextFrame s from TTS services and aggregates them into complete utterances. Parameters: **kwargs : Arguments passed to the AssistantTranscriptProcessor constructor Returns: AssistantTranscriptProcessor instance for processing assistant messages. ​ event_handler() Copy Ask AI def event_handler ( event_name : str ) Decorator that registers event handlers for both user and assistant processors. Parameters: event_name : Name of the event to handle Returns: Decorator function that registers the handler with both processors. ​ Event Handlers ​ on_transcript_update Triggered when new messages are added to the conversation transcript. Copy Ask AI @transcript.event_handler ( "on_transcript_update" ) async def handle_transcript_update ( processor , frame ): # Handle transcript updates pass Parameters: processor : The specific processor instance that emitted the event (UserTranscriptProcessor or AssistantTranscriptProcessor) frame : TranscriptionUpdateFrame containing the new messages ​ Data Structures ​ TranscriptionMessage Copy Ask AI @dataclass class TranscriptionMessage : role: Literal[ "user" , "assistant" ] content: str timestamp: str | None = None user_id: str | None = None Fields: role : The message sender type (“user” or “assistant”) content : The transcribed text content timestamp : ISO 8601 timestamp when the message was created user_id : Optional user identifier (for user messages only) ​ TranscriptionUpdateFrame Frame containing new transcript messages, emitted by the on_transcript_update event. Properties: messages : List of TranscriptionMessage objects containing the new transcript content ​ Frames ​ UserTranscriptProcessor Input: TranscriptionFrame from STT services Output: TranscriptionMessage with role “user” ​ AssistantTranscriptProcessor Input: TTSTextFrame from TTS services Output: TranscriptionMessage with role “assistant” ​ Integration Notes ​ Pipeline Placement Place the processors at specific positions in your pipeline for accurate transcript collection: Copy Ask AI pipeline = Pipeline([ transport.input(), stt, # Speech-to-text service transcript.user(), # Place after STT context_aggregator.user(), llm, tts, # Text-to-speech service transport.output(), transcript.assistant(), # Place after transport.output() context_aggregator.assistant(), ]) ​ Event Handler Registration Event handlers are automatically applied to both user and assistant processors: Copy Ask AI transcript = TranscriptProcessor() # This handler will receive events from both processors @transcript.event_handler ( "on_transcript_update" ) async def handle_update ( processor , frame ): for message in frame.messages: print ( f " { message.role } : { message.content } " ) PatternPairAggregator Interruption Strategies On this page Overview Constructor Methods user() assistant() event_handler() Event Handlers on_transcript_update Data Structures TranscriptionMessage TranscriptionUpdateFrame Frames UserTranscriptProcessor AssistantTranscriptProcessor Integration Notes Pipeline Placement Event Handler Registration Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/utilities_transcript-processor_760662bb.txt b/utilities_transcript-processor_760662bb.txt
new file mode 100644
index 0000000000000000000000000000000000000000..667588491f72897af0031a374c13d9028bff4bf6
--- /dev/null
+++ b/utilities_transcript-processor_760662bb.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/transcript-processor#data-structures
+Title: TranscriptProcessor - Pipecat
+==================================================
+
+TranscriptProcessor - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation User and Bot Transcriptions TranscriptProcessor Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions TranscriptProcessor User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview The TranscriptProcessor is a factory class that creates and manages processors for handling conversation transcripts from both users and assistants. It provides unified access to transcript processors with shared event handling, making it easy to track and respond to conversation updates in real-time. The processor normalizes messages from various sources into a consistent TranscriptionMessage format and emits events when new messages are added to the conversation. ​ Constructor Copy Ask AI TranscriptProcessor() Creates a new transcript processor factory with no parameters. ​ Methods ​ user() Copy Ask AI def user ( ** kwargs ) -> UserTranscriptProcessor Get or create the user transcript processor instance. This processor handles TranscriptionFrame s from STT services. Parameters: **kwargs : Arguments passed to the UserTranscriptProcessor constructor Returns: UserTranscriptProcessor instance for processing user messages. ​ assistant() Copy Ask AI def assistant ( ** kwargs ) -> AssistantTranscriptProcessor Get or create the assistant transcript processor instance. This processor handles TTSTextFrame s from TTS services and aggregates them into complete utterances. Parameters: **kwargs : Arguments passed to the AssistantTranscriptProcessor constructor Returns: AssistantTranscriptProcessor instance for processing assistant messages. ​ event_handler() Copy Ask AI def event_handler ( event_name : str ) Decorator that registers event handlers for both user and assistant processors. Parameters: event_name : Name of the event to handle Returns: Decorator function that registers the handler with both processors. ​ Event Handlers ​ on_transcript_update Triggered when new messages are added to the conversation transcript. Copy Ask AI @transcript.event_handler ( "on_transcript_update" ) async def handle_transcript_update ( processor , frame ): # Handle transcript updates pass Parameters: processor : The specific processor instance that emitted the event (UserTranscriptProcessor or AssistantTranscriptProcessor) frame : TranscriptionUpdateFrame containing the new messages ​ Data Structures ​ TranscriptionMessage Copy Ask AI @dataclass class TranscriptionMessage : role: Literal[ "user" , "assistant" ] content: str timestamp: str | None = None user_id: str | None = None Fields: role : The message sender type (“user” or “assistant”) content : The transcribed text content timestamp : ISO 8601 timestamp when the message was created user_id : Optional user identifier (for user messages only) ​ TranscriptionUpdateFrame Frame containing new transcript messages, emitted by the on_transcript_update event. Properties: messages : List of TranscriptionMessage objects containing the new transcript content ​ Frames ​ UserTranscriptProcessor Input: TranscriptionFrame from STT services Output: TranscriptionMessage with role “user” ​ AssistantTranscriptProcessor Input: TTSTextFrame from TTS services Output: TranscriptionMessage with role “assistant” ​ Integration Notes ​ Pipeline Placement Place the processors at specific positions in your pipeline for accurate transcript collection: Copy Ask AI pipeline = Pipeline([ transport.input(), stt, # Speech-to-text service transcript.user(), # Place after STT context_aggregator.user(), llm, tts, # Text-to-speech service transport.output(), transcript.assistant(), # Place after transport.output() context_aggregator.assistant(), ]) ​ Event Handler Registration Event handlers are automatically applied to both user and assistant processors: Copy Ask AI transcript = TranscriptProcessor() # This handler will receive events from both processors @transcript.event_handler ( "on_transcript_update" ) async def handle_update ( processor , frame ): for message in frame.messages: print ( f " { message.role } : { message.content } " ) PatternPairAggregator Interruption Strategies On this page Overview Constructor Methods user() assistant() event_handler() Event Handlers on_transcript_update Data Structures TranscriptionMessage TranscriptionUpdateFrame Frames UserTranscriptProcessor AssistantTranscriptProcessor Integration Notes Pipeline Placement Event Handler Registration Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/utilities_transcript-processor_c6396a0b.txt b/utilities_transcript-processor_c6396a0b.txt
new file mode 100644
index 0000000000000000000000000000000000000000..45caa9317412eebbae4f2d5c8dbdc3f036ce92ac
--- /dev/null
+++ b/utilities_transcript-processor_c6396a0b.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/transcript-processor#overview
+Title: TranscriptProcessor - Pipecat
+==================================================
+
+TranscriptProcessor - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation User and Bot Transcriptions TranscriptProcessor Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions TranscriptProcessor User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview The TranscriptProcessor is a factory class that creates and manages processors for handling conversation transcripts from both users and assistants. It provides unified access to transcript processors with shared event handling, making it easy to track and respond to conversation updates in real-time. The processor normalizes messages from various sources into a consistent TranscriptionMessage format and emits events when new messages are added to the conversation. ​ Constructor Copy Ask AI TranscriptProcessor() Creates a new transcript processor factory with no parameters. ​ Methods ​ user() Copy Ask AI def user ( ** kwargs ) -> UserTranscriptProcessor Get or create the user transcript processor instance. This processor handles TranscriptionFrame s from STT services. Parameters: **kwargs : Arguments passed to the UserTranscriptProcessor constructor Returns: UserTranscriptProcessor instance for processing user messages. ​ assistant() Copy Ask AI def assistant ( ** kwargs ) -> AssistantTranscriptProcessor Get or create the assistant transcript processor instance. This processor handles TTSTextFrame s from TTS services and aggregates them into complete utterances. Parameters: **kwargs : Arguments passed to the AssistantTranscriptProcessor constructor Returns: AssistantTranscriptProcessor instance for processing assistant messages. ​ event_handler() Copy Ask AI def event_handler ( event_name : str ) Decorator that registers event handlers for both user and assistant processors. Parameters: event_name : Name of the event to handle Returns: Decorator function that registers the handler with both processors. ​ Event Handlers ​ on_transcript_update Triggered when new messages are added to the conversation transcript. Copy Ask AI @transcript.event_handler ( "on_transcript_update" ) async def handle_transcript_update ( processor , frame ): # Handle transcript updates pass Parameters: processor : The specific processor instance that emitted the event (UserTranscriptProcessor or AssistantTranscriptProcessor) frame : TranscriptionUpdateFrame containing the new messages ​ Data Structures ​ TranscriptionMessage Copy Ask AI @dataclass class TranscriptionMessage : role: Literal[ "user" , "assistant" ] content: str timestamp: str | None = None user_id: str | None = None Fields: role : The message sender type (“user” or “assistant”) content : The transcribed text content timestamp : ISO 8601 timestamp when the message was created user_id : Optional user identifier (for user messages only) ​ TranscriptionUpdateFrame Frame containing new transcript messages, emitted by the on_transcript_update event. Properties: messages : List of TranscriptionMessage objects containing the new transcript content ​ Frames ​ UserTranscriptProcessor Input: TranscriptionFrame from STT services Output: TranscriptionMessage with role “user” ​ AssistantTranscriptProcessor Input: TTSTextFrame from TTS services Output: TranscriptionMessage with role “assistant” ​ Integration Notes ​ Pipeline Placement Place the processors at specific positions in your pipeline for accurate transcript collection: Copy Ask AI pipeline = Pipeline([ transport.input(), stt, # Speech-to-text service transcript.user(), # Place after STT context_aggregator.user(), llm, tts, # Text-to-speech service transport.output(), transcript.assistant(), # Place after transport.output() context_aggregator.assistant(), ]) ​ Event Handler Registration Event handlers are automatically applied to both user and assistant processors: Copy Ask AI transcript = TranscriptProcessor() # This handler will receive events from both processors @transcript.event_handler ( "on_transcript_update" ) async def handle_update ( processor , frame ): for message in frame.messages: print ( f " { message.role } : { message.content } " ) PatternPairAggregator Interruption Strategies On this page Overview Constructor Methods user() assistant() event_handler() Event Handlers on_transcript_update Data Structures TranscriptionMessage TranscriptionUpdateFrame Frames UserTranscriptProcessor AssistantTranscriptProcessor Integration Notes Pipeline Placement Event Handler Registration Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/utilities_transcript-processor_cedb9f2a.txt b/utilities_transcript-processor_cedb9f2a.txt
new file mode 100644
index 0000000000000000000000000000000000000000..af86d695a21b1f71cf0cfd65fc8bf18d165a0ba5
--- /dev/null
+++ b/utilities_transcript-processor_cedb9f2a.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/transcript-processor#user
+Title: TranscriptProcessor - Pipecat
+==================================================
+
+TranscriptProcessor - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation User and Bot Transcriptions TranscriptProcessor Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions TranscriptProcessor User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview The TranscriptProcessor is a factory class that creates and manages processors for handling conversation transcripts from both users and assistants. It provides unified access to transcript processors with shared event handling, making it easy to track and respond to conversation updates in real-time. The processor normalizes messages from various sources into a consistent TranscriptionMessage format and emits events when new messages are added to the conversation. ​ Constructor Copy Ask AI TranscriptProcessor() Creates a new transcript processor factory with no parameters. ​ Methods ​ user() Copy Ask AI def user ( ** kwargs ) -> UserTranscriptProcessor Get or create the user transcript processor instance. This processor handles TranscriptionFrame s from STT services. Parameters: **kwargs : Arguments passed to the UserTranscriptProcessor constructor Returns: UserTranscriptProcessor instance for processing user messages. ​ assistant() Copy Ask AI def assistant ( ** kwargs ) -> AssistantTranscriptProcessor Get or create the assistant transcript processor instance. This processor handles TTSTextFrame s from TTS services and aggregates them into complete utterances. Parameters: **kwargs : Arguments passed to the AssistantTranscriptProcessor constructor Returns: AssistantTranscriptProcessor instance for processing assistant messages. ​ event_handler() Copy Ask AI def event_handler ( event_name : str ) Decorator that registers event handlers for both user and assistant processors. Parameters: event_name : Name of the event to handle Returns: Decorator function that registers the handler with both processors. ​ Event Handlers ​ on_transcript_update Triggered when new messages are added to the conversation transcript. Copy Ask AI @transcript.event_handler ( "on_transcript_update" ) async def handle_transcript_update ( processor , frame ): # Handle transcript updates pass Parameters: processor : The specific processor instance that emitted the event (UserTranscriptProcessor or AssistantTranscriptProcessor) frame : TranscriptionUpdateFrame containing the new messages ​ Data Structures ​ TranscriptionMessage Copy Ask AI @dataclass class TranscriptionMessage : role: Literal[ "user" , "assistant" ] content: str timestamp: str | None = None user_id: str | None = None Fields: role : The message sender type (“user” or “assistant”) content : The transcribed text content timestamp : ISO 8601 timestamp when the message was created user_id : Optional user identifier (for user messages only) ​ TranscriptionUpdateFrame Frame containing new transcript messages, emitted by the on_transcript_update event. Properties: messages : List of TranscriptionMessage objects containing the new transcript content ​ Frames ​ UserTranscriptProcessor Input: TranscriptionFrame from STT services Output: TranscriptionMessage with role “user” ​ AssistantTranscriptProcessor Input: TTSTextFrame from TTS services Output: TranscriptionMessage with role “assistant” ​ Integration Notes ​ Pipeline Placement Place the processors at specific positions in your pipeline for accurate transcript collection: Copy Ask AI pipeline = Pipeline([ transport.input(), stt, # Speech-to-text service transcript.user(), # Place after STT context_aggregator.user(), llm, tts, # Text-to-speech service transport.output(), transcript.assistant(), # Place after transport.output() context_aggregator.assistant(), ]) ​ Event Handler Registration Event handlers are automatically applied to both user and assistant processors: Copy Ask AI transcript = TranscriptProcessor() # This handler will receive events from both processors @transcript.event_handler ( "on_transcript_update" ) async def handle_update ( processor , frame ): for message in frame.messages: print ( f " { message.role } : { message.content } " ) PatternPairAggregator Interruption Strategies On this page Overview Constructor Methods user() assistant() event_handler() Event Handlers on_transcript_update Data Structures TranscriptionMessage TranscriptionUpdateFrame Frames UserTranscriptProcessor AssistantTranscriptProcessor Integration Notes Pipeline Placement Event Handler Registration Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/utilities_transcript-processor_dc649fc9.txt b/utilities_transcript-processor_dc649fc9.txt
new file mode 100644
index 0000000000000000000000000000000000000000..3ef26e1b5cea3ba08d317ce991c4c84e20ae2575
--- /dev/null
+++ b/utilities_transcript-processor_dc649fc9.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/transcript-processor#event-handler-registration
+Title: TranscriptProcessor - Pipecat
+==================================================
+
+TranscriptProcessor - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation User and Bot Transcriptions TranscriptProcessor Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions TranscriptProcessor User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview The TranscriptProcessor is a factory class that creates and manages processors for handling conversation transcripts from both users and assistants. It provides unified access to transcript processors with shared event handling, making it easy to track and respond to conversation updates in real-time. The processor normalizes messages from various sources into a consistent TranscriptionMessage format and emits events when new messages are added to the conversation. ​ Constructor Copy Ask AI TranscriptProcessor() Creates a new transcript processor factory with no parameters. ​ Methods ​ user() Copy Ask AI def user ( ** kwargs ) -> UserTranscriptProcessor Get or create the user transcript processor instance. This processor handles TranscriptionFrame s from STT services. Parameters: **kwargs : Arguments passed to the UserTranscriptProcessor constructor Returns: UserTranscriptProcessor instance for processing user messages. ​ assistant() Copy Ask AI def assistant ( ** kwargs ) -> AssistantTranscriptProcessor Get or create the assistant transcript processor instance. This processor handles TTSTextFrame s from TTS services and aggregates them into complete utterances. Parameters: **kwargs : Arguments passed to the AssistantTranscriptProcessor constructor Returns: AssistantTranscriptProcessor instance for processing assistant messages. ​ event_handler() Copy Ask AI def event_handler ( event_name : str ) Decorator that registers event handlers for both user and assistant processors. Parameters: event_name : Name of the event to handle Returns: Decorator function that registers the handler with both processors. ​ Event Handlers ​ on_transcript_update Triggered when new messages are added to the conversation transcript. Copy Ask AI @transcript.event_handler ( "on_transcript_update" ) async def handle_transcript_update ( processor , frame ): # Handle transcript updates pass Parameters: processor : The specific processor instance that emitted the event (UserTranscriptProcessor or AssistantTranscriptProcessor) frame : TranscriptionUpdateFrame containing the new messages ​ Data Structures ​ TranscriptionMessage Copy Ask AI @dataclass class TranscriptionMessage : role: Literal[ "user" , "assistant" ] content: str timestamp: str | None = None user_id: str | None = None Fields: role : The message sender type (“user” or “assistant”) content : The transcribed text content timestamp : ISO 8601 timestamp when the message was created user_id : Optional user identifier (for user messages only) ​ TranscriptionUpdateFrame Frame containing new transcript messages, emitted by the on_transcript_update event. Properties: messages : List of TranscriptionMessage objects containing the new transcript content ​ Frames ​ UserTranscriptProcessor Input: TranscriptionFrame from STT services Output: TranscriptionMessage with role “user” ​ AssistantTranscriptProcessor Input: TTSTextFrame from TTS services Output: TranscriptionMessage with role “assistant” ​ Integration Notes ​ Pipeline Placement Place the processors at specific positions in your pipeline for accurate transcript collection: Copy Ask AI pipeline = Pipeline([ transport.input(), stt, # Speech-to-text service transcript.user(), # Place after STT context_aggregator.user(), llm, tts, # Text-to-speech service transport.output(), transcript.assistant(), # Place after transport.output() context_aggregator.assistant(), ]) ​ Event Handler Registration Event handlers are automatically applied to both user and assistant processors: Copy Ask AI transcript = TranscriptProcessor() # This handler will receive events from both processors @transcript.event_handler ( "on_transcript_update" ) async def handle_update ( processor , frame ): for message in frame.messages: print ( f " { message.role } : { message.content } " ) PatternPairAggregator Interruption Strategies On this page Overview Constructor Methods user() assistant() event_handler() Event Handlers on_transcript_update Data Structures TranscriptionMessage TranscriptionUpdateFrame Frames UserTranscriptProcessor AssistantTranscriptProcessor Integration Notes Pipeline Placement Event Handler Registration Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/utilities_transcript-processor_fe131064.txt b/utilities_transcript-processor_fe131064.txt
new file mode 100644
index 0000000000000000000000000000000000000000..fe07c83dbef80a6544b929b6e4287067b075a627
--- /dev/null
+++ b/utilities_transcript-processor_fe131064.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/transcript-processor#constructor
+Title: TranscriptProcessor - Pipecat
+==================================================
+
+TranscriptProcessor - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation User and Bot Transcriptions TranscriptProcessor Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions TranscriptProcessor User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview The TranscriptProcessor is a factory class that creates and manages processors for handling conversation transcripts from both users and assistants. It provides unified access to transcript processors with shared event handling, making it easy to track and respond to conversation updates in real-time. The processor normalizes messages from various sources into a consistent TranscriptionMessage format and emits events when new messages are added to the conversation. ​ Constructor Copy Ask AI TranscriptProcessor() Creates a new transcript processor factory with no parameters. ​ Methods ​ user() Copy Ask AI def user ( ** kwargs ) -> UserTranscriptProcessor Get or create the user transcript processor instance. This processor handles TranscriptionFrame s from STT services. Parameters: **kwargs : Arguments passed to the UserTranscriptProcessor constructor Returns: UserTranscriptProcessor instance for processing user messages. ​ assistant() Copy Ask AI def assistant ( ** kwargs ) -> AssistantTranscriptProcessor Get or create the assistant transcript processor instance. This processor handles TTSTextFrame s from TTS services and aggregates them into complete utterances. Parameters: **kwargs : Arguments passed to the AssistantTranscriptProcessor constructor Returns: AssistantTranscriptProcessor instance for processing assistant messages. ​ event_handler() Copy Ask AI def event_handler ( event_name : str ) Decorator that registers event handlers for both user and assistant processors. Parameters: event_name : Name of the event to handle Returns: Decorator function that registers the handler with both processors. ​ Event Handlers ​ on_transcript_update Triggered when new messages are added to the conversation transcript. Copy Ask AI @transcript.event_handler ( "on_transcript_update" ) async def handle_transcript_update ( processor , frame ): # Handle transcript updates pass Parameters: processor : The specific processor instance that emitted the event (UserTranscriptProcessor or AssistantTranscriptProcessor) frame : TranscriptionUpdateFrame containing the new messages ​ Data Structures ​ TranscriptionMessage Copy Ask AI @dataclass class TranscriptionMessage : role: Literal[ "user" , "assistant" ] content: str timestamp: str | None = None user_id: str | None = None Fields: role : The message sender type (“user” or “assistant”) content : The transcribed text content timestamp : ISO 8601 timestamp when the message was created user_id : Optional user identifier (for user messages only) ​ TranscriptionUpdateFrame Frame containing new transcript messages, emitted by the on_transcript_update event. Properties: messages : List of TranscriptionMessage objects containing the new transcript content ​ Frames ​ UserTranscriptProcessor Input: TranscriptionFrame from STT services Output: TranscriptionMessage with role “user” ​ AssistantTranscriptProcessor Input: TTSTextFrame from TTS services Output: TranscriptionMessage with role “assistant” ​ Integration Notes ​ Pipeline Placement Place the processors at specific positions in your pipeline for accurate transcript collection: Copy Ask AI pipeline = Pipeline([ transport.input(), stt, # Speech-to-text service transcript.user(), # Place after STT context_aggregator.user(), llm, tts, # Text-to-speech service transport.output(), transcript.assistant(), # Place after transport.output() context_aggregator.assistant(), ]) ​ Event Handler Registration Event handlers are automatically applied to both user and assistant processors: Copy Ask AI transcript = TranscriptProcessor() # This handler will receive events from both processors @transcript.event_handler ( "on_transcript_update" ) async def handle_update ( processor , frame ): for message in frame.messages: print ( f " { message.role } : { message.content } " ) PatternPairAggregator Interruption Strategies On this page Overview Constructor Methods user() assistant() event_handler() Event Handlers on_transcript_update Data Structures TranscriptionMessage TranscriptionUpdateFrame Frames UserTranscriptProcessor AssistantTranscriptProcessor Integration Notes Pipeline Placement Event Handler Registration Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/utilities_user-idle-processor_8369dad5.txt b/utilities_user-idle-processor_8369dad5.txt
new file mode 100644
index 0000000000000000000000000000000000000000..13a7933f2d4bb89a88236addf46d7ab62ef66080
--- /dev/null
+++ b/utilities_user-idle-processor_8369dad5.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/user-idle-processor#constructor-parameters
+Title: UserIdleProcessor - Pipecat
+==================================================
+
+UserIdleProcessor - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Advanced Frame Processors UserIdleProcessor Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Producer & Consumer Processors UserIdleProcessor Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline The UserIdleProcessor is a specialized frame processor that monitors user activity in a conversation and executes callbacks when the user becomes idle. It’s particularly useful for maintaining engagement by detecting periods of user inactivity and providing escalating responses to inactivity. ​ Constructor Parameters ​ callback Union[BasicCallback, RetryCallback] required An async function that will be called when user inactivity is detected. Can be either: Basic callback: async def(processor: UserIdleProcessor) -> None Retry callback: async def(processor: UserIdleProcessor, retry_count: int) -> bool where returning False stops idle monitoring ​ timeout float required The number of seconds to wait before considering the user idle. ​ Behavior The processor starts monitoring for inactivity only after the first conversation activity (either UserStartedSpeakingFrame or BotSpeakingFrame ). It manages idle state based on the following rules: Resets idle timer when user starts or stops speaking Pauses idle monitoring while user is speaking Resets idle timer when bot is speaking Stops monitoring on conversation end or cancellation Manages a retry count for the retry callback Stops monitoring when retry callback returns False ​ Properties ​ retry_count int The current number of retry attempts made to engage the user. ​ Example Implementations Here are two example showing how to use the UserIdleProcessor : one with the basic callback and one with the retry callback: Basic Callback Retry Callback Copy Ask AI from pipecat.frames.frames import LLMMessagesFrame from pipecat.pipeline.pipeline import Pipeline from pipecat.processors.user_idle_processor import UserIdleProcessor async def handle_idle ( user_idle : UserIdleProcessor) -> None : messages.append({ "role" : "system" , "content" : "Ask the user if they are still there and try to prompt for some input." }) await user_idle.push_frame(LLMMessagesFrame(messages)) # Create the processor user_idle = UserIdleProcessor( callback = handle_idle, timeout = 5.0 ) # Add to pipeline pipeline = Pipeline([ transport.input(), user_idle, # Add the processor to monitor user activity context_aggregator.user(), # ... rest of pipeline ]) Copy Ask AI from pipecat.frames.frames import LLMMessagesFrame from pipecat.pipeline.pipeline import Pipeline from pipecat.processors.user_idle_processor import UserIdleProcessor async def handle_idle ( user_idle : UserIdleProcessor) -> None : messages.append({ "role" : "system" , "content" : "Ask the user if they are still there and try to prompt for some input." }) await user_idle.push_frame(LLMMessagesFrame(messages)) # Create the processor user_idle = UserIdleProcessor( callback = handle_idle, timeout = 5.0 ) # Add to pipeline pipeline = Pipeline([ transport.input(), user_idle, # Add the processor to monitor user activity context_aggregator.user(), # ... rest of pipeline ]) Copy Ask AI from pipecat.frames.frames import EndFrame, LLMMessagesFrame, TTSSpeakFrame from pipecat.pipeline.pipeline import Pipeline from pipecat.processors.user_idle_processor import UserIdleProcessor async def handle_user_idle ( user_idle : UserIdleProcessor, retry_count : int ) -> bool : if retry_count == 1 : # First attempt: Gentle reminder messages.append({ "role" : "system" , "content" : "The user has been quiet. Politely and briefly ask if they're still there." }) await user_idle.push_frame(LLMMessagesFrame(messages)) return True elif retry_count == 2 : # Second attempt: Direct prompt messages.append({ "role" : "system" , "content" : "The user is still inactive. Ask if they'd like to continue our conversation." }) await user_idle.push_frame(LLMMessagesFrame(messages)) return True else : # Third attempt: End conversation await user_idle.push_frame( TTSSpeakFrame( "It seems like you're busy right now. Have a nice day!" ) ) await task.queue_frame(EndFrame()) return False # Stop monitoring # Create the processor user_idle = UserIdleProcessor( callback = handle_user_idle, timeout = 5.0 ) # Add to pipeline pipeline = Pipeline([ transport.input(), user_idle, # Add the processor to monitor user activity context_aggregator.user(), # ... rest of pipeline ]) ​ Frame Handling The processor handles the following frame types: UserStartedSpeakingFrame : Marks user as active, resets idle timer and retry count UserStoppedSpeakingFrame : Starts idle monitoring BotSpeakingFrame : Resets idle timer EndFrame / CancelFrame : Stops idle monitoring ​ Notes The idle callback won’t be triggered while the user or bot is actively speaking The processor automatically cleans up its resources when the pipeline ends Basic callbacks are supported for backward compatibility Producer & Consumer Processors AudioBufferProcessor On this page Constructor Parameters Behavior Properties Example Implementations Frame Handling Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/utilities_watchdog-timers_95cea877.txt b/utilities_watchdog-timers_95cea877.txt
new file mode 100644
index 0000000000000000000000000000000000000000..cff9e137c51a1debda8c44b540080bdf48f6a610
--- /dev/null
+++ b/utilities_watchdog-timers_95cea877.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/watchdog-timers#param-enable-watchdog-logging
+Title: Watchdog Timers - Pipecat
+==================================================
+
+Watchdog Timers - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Task Handling and Monitoring Watchdog Timers Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Watchdog Timers Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview Watchdog timers are used to detect if a Pipecat task is taking longer than expected. By default, if watchdog timers are enabled and a task takes longer than 5 seconds to reset the timer, a warning will be logged. Watchdog timers are very common in real-time applications to detect if things are frozen or taking too much time. Usually, you start the watchdog and you need to keep resetting it before the watchdog timer timeout expires. In Pipecat, watchdog timers are available if you create a task using the FrameProcessor.create_task() method. If you use asyncio.create_task() or loop.create_task() , watchdog timers will not work. ​ Configuration Watchdog timers are disabled by default. You can enable and configure them using the following PipelineTask constructor arguments: ​ watchdog_timeout_secs float default: "5.0" Watchdog timer timeout. ​ enable_watchdog_logging bool default: "False" Whether to print task processing times. ​ enable_watchdog_timers bool default: "False" Whether to enable watchdog timers. It is possible to configure watchdog timers individually for each FrameProcessor , using the same argument names in the constructor, or even per task when creating them with FrameProcessor.create_task() . ​ How It Works Watchdog timers are always available for every created Pipecat task: We enable watchdog timers using enable_watchdog_timers . A task is created with FrameProcessor.create_task() and its watchdog timer is started. The task needs to periodically call self.reset_watchdog() to prevent the watchdog timer to expire If the watchdog timer is not reset a warning will be logged ​ Usage Examples Copy Ask AI class MyFrameProcessor ( FrameProcessor ): async def process_frame ( self , frame : Frame, direction : FrameDirection): await super ().process_frame(frame, direction) ... self ._task = self .create_task( self ._task_handler()) ... async def _task_handler ( self ): while True : item = await self ._queue.get() ... # Perform any processing ... self .reset_watchdog() ​ Notes Watchdog timers are disabled by default Watchdog timers only work with Pipecat tasks (use FrameProcessor.create_task() ) Watchdog timers can be enabled and configure globally with PipelineTask , per FrameProcessor or per task Local CoreML Smart Turn DTMFAggregator On this page Overview Configuration How It Works Usage Examples Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/utilities_watchdog-timers_9da76cb9.txt b/utilities_watchdog-timers_9da76cb9.txt
new file mode 100644
index 0000000000000000000000000000000000000000..07541fc8ae0339bbf95735d9503661578892d8bd
--- /dev/null
+++ b/utilities_watchdog-timers_9da76cb9.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/utilities/watchdog-timers#configuration
+Title: Watchdog Timers - Pipecat
+==================================================
+
+Watchdog Timers - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Task Handling and Monitoring Watchdog Timers Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Watchdog Timers Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview Watchdog timers are used to detect if a Pipecat task is taking longer than expected. By default, if watchdog timers are enabled and a task takes longer than 5 seconds to reset the timer, a warning will be logged. Watchdog timers are very common in real-time applications to detect if things are frozen or taking too much time. Usually, you start the watchdog and you need to keep resetting it before the watchdog timer timeout expires. In Pipecat, watchdog timers are available if you create a task using the FrameProcessor.create_task() method. If you use asyncio.create_task() or loop.create_task() , watchdog timers will not work. ​ Configuration Watchdog timers are disabled by default. You can enable and configure them using the following PipelineTask constructor arguments: ​ watchdog_timeout_secs float default: "5.0" Watchdog timer timeout. ​ enable_watchdog_logging bool default: "False" Whether to print task processing times. ​ enable_watchdog_timers bool default: "False" Whether to enable watchdog timers. It is possible to configure watchdog timers individually for each FrameProcessor , using the same argument names in the constructor, or even per task when creating them with FrameProcessor.create_task() . ​ How It Works Watchdog timers are always available for every created Pipecat task: We enable watchdog timers using enable_watchdog_timers . A task is created with FrameProcessor.create_task() and its watchdog timer is started. The task needs to periodically call self.reset_watchdog() to prevent the watchdog timer to expire If the watchdog timer is not reset a warning will be logged ​ Usage Examples Copy Ask AI class MyFrameProcessor ( FrameProcessor ): async def process_frame ( self , frame : Frame, direction : FrameDirection): await super ().process_frame(frame, direction) ... self ._task = self .create_task( self ._task_handler()) ... async def _task_handler ( self ): while True : item = await self ._queue.get() ... # Perform any processing ... self .reset_watchdog() ​ Notes Watchdog timers are disabled by default Watchdog timers only work with Pipecat tasks (use FrameProcessor.create_task() ) Watchdog timers can be enabled and configure globally with PipelineTask , per FrameProcessor or per task Local CoreML Smart Turn DTMFAggregator On this page Overview Configuration How It Works Usage Examples Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/video_simli_01c4aa8c.txt b/video_simli_01c4aa8c.txt
new file mode 100644
index 0000000000000000000000000000000000000000..121022837799085ec934aaed661fa11355e1e2e0
--- /dev/null
+++ b/video_simli_01c4aa8c.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/video/simli#constructor-parameters-for-simliconfig
+Title: Simli - Pipecat
+==================================================
+
+Simli - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Video Simli Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Simli Tavus Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview SimliVideoService creates AI avatar video responses by converting audio input into synchronized video and audio output through Simli’s WebRTC platform. It handles real-time audio streaming, video generation, and automatic audio resampling. ​ Installation Install the required dependencies: Copy Ask AI pip install "pipecat-ai[simli]" ​ Configuration ​ Required Environment Variables Copy Ask AI SIMLI_API_KEY = your_api_key SIMLI_FACE_ID = your_face_id Get your API key and Face ID by signing up at Simli ​ Configuration Copy Ask AI SimliVideoService( SimliConfig( SIMLI_API_KEY , SIMLI_FACE_ID ), useTurnServer = False , latencyInterval = 60 ) ​ Constructor Parameters for SimliConfig ​ apiKey str required Your Simli API key. This key is required for authenticating API requests. ​ faceId str required The face identifier for Simli. This is used to associate API interactions with a specific face or persona. ​ syncAudio bool default: "True" Indicates whether to synchronize audio streams. Defaults to True . Set to False to disable audio synchronization. ​ handleSilence bool default: "True" Determines if silence in audio streams should be handled automatically. Defaults to True . ​ maxSessionLength int default: "600" The maximum length of a session in seconds. Defaults to 600 (10 minutes). ​ maxIdleTime int default: "30" The maximum idle time (in seconds) allowed during a session before it is automatically terminated. Defaults to 30 seconds. ​ Constructor Parameters for SimliVideoService ​ simli_config SimliConfig required The configuration object for Simli. This must be an instance of simli_config and contains essential settings such as API key, face ID, and other session-related configurations. ​ use_turn_server bool default: "False" Determines whether a TURN server should be used for establishing connections. Defaults to False . Set to True if your network requires TURN for WebRTC connections. ​ latency_interval int default: "0" Delay (in seconds) between ping calls to calculate latency between your machine and simli server. Set to 0 to disable. ​ Input Frames ​ Audio Input ​ TTSAudioRawFrame Frame Raw audio data for avatar speech ​ Control Frames ​ TTSStartedFrame Frame Signals start of speech synthesis ​ TTSStoppedFrame Frame Signals end of speech synthesis ​ StartInterruptionFrame Frame Signals conversation interruption ​ EndFrame Frame Signals end of conversation ​ CancelFrame Frame Signals conversation cancellation ​ Usage Example Copy Ask AI from pipecat.pipeline.pipeline import Pipeline from pipecat.services.simli.video import SimliVideoService from simli import SimliConfig import os async def create_avatar_pipeline (): # Initialize Simli service simli = SimliVideoService( SimliConfig( api_key = os.getenv( "SIMLI_API_KEY" ), face_id = os.getenv( "SIMLI_FACE_ID" ) ) ) # Create pipeline with Simli pipeline = Pipeline([ transport.input(), # Your audio input llm, # Language model service tts, # Text-to-speech service simli, # Simli video generation transport.output(), # Your video output handler ]) return pipeline ​ Frame Flow ​ Metrics Support The service collects processing metrics: Processing duration Time to First Byte (TTFB) API response times Audio processing metrics ​ Common Use Cases AI Video Avatars Virtual assistants Customer service Educational content Interactive Presentations Product demonstrations Training materials Marketing content Real-time Communication Video conferencing Virtual meetings Interactive broadcasts ​ Notes Handles real-time audio streaming Supports conversation interruptions Manages conversation lifecycle Automatic audio resampling Thread-safe processing WebRTC integration through Daily Includes comprehensive error handling OpenAI Tavus On this page Overview Installation Configuration Required Environment Variables Configuration Constructor Parameters for SimliConfig Constructor Parameters for SimliVideoService Input Frames Audio Input Control Frames Usage Example Frame Flow Metrics Support Common Use Cases Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/video_simli_27943253.txt b/video_simli_27943253.txt
new file mode 100644
index 0000000000000000000000000000000000000000..e6f664166c15e43490a0c6fcaf42a7e11e6a8bd2
--- /dev/null
+++ b/video_simli_27943253.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/video/simli#param-start-interruption-frame
+Title: Simli - Pipecat
+==================================================
+
+Simli - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Video Simli Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Simli Tavus Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview SimliVideoService creates AI avatar video responses by converting audio input into synchronized video and audio output through Simli’s WebRTC platform. It handles real-time audio streaming, video generation, and automatic audio resampling. ​ Installation Install the required dependencies: Copy Ask AI pip install "pipecat-ai[simli]" ​ Configuration ​ Required Environment Variables Copy Ask AI SIMLI_API_KEY = your_api_key SIMLI_FACE_ID = your_face_id Get your API key and Face ID by signing up at Simli ​ Configuration Copy Ask AI SimliVideoService( SimliConfig( SIMLI_API_KEY , SIMLI_FACE_ID ), useTurnServer = False , latencyInterval = 60 ) ​ Constructor Parameters for SimliConfig ​ apiKey str required Your Simli API key. This key is required for authenticating API requests. ​ faceId str required The face identifier for Simli. This is used to associate API interactions with a specific face or persona. ​ syncAudio bool default: "True" Indicates whether to synchronize audio streams. Defaults to True . Set to False to disable audio synchronization. ​ handleSilence bool default: "True" Determines if silence in audio streams should be handled automatically. Defaults to True . ​ maxSessionLength int default: "600" The maximum length of a session in seconds. Defaults to 600 (10 minutes). ​ maxIdleTime int default: "30" The maximum idle time (in seconds) allowed during a session before it is automatically terminated. Defaults to 30 seconds. ​ Constructor Parameters for SimliVideoService ​ simli_config SimliConfig required The configuration object for Simli. This must be an instance of simli_config and contains essential settings such as API key, face ID, and other session-related configurations. ​ use_turn_server bool default: "False" Determines whether a TURN server should be used for establishing connections. Defaults to False . Set to True if your network requires TURN for WebRTC connections. ​ latency_interval int default: "0" Delay (in seconds) between ping calls to calculate latency between your machine and simli server. Set to 0 to disable. ​ Input Frames ​ Audio Input ​ TTSAudioRawFrame Frame Raw audio data for avatar speech ​ Control Frames ​ TTSStartedFrame Frame Signals start of speech synthesis ​ TTSStoppedFrame Frame Signals end of speech synthesis ​ StartInterruptionFrame Frame Signals conversation interruption ​ EndFrame Frame Signals end of conversation ​ CancelFrame Frame Signals conversation cancellation ​ Usage Example Copy Ask AI from pipecat.pipeline.pipeline import Pipeline from pipecat.services.simli.video import SimliVideoService from simli import SimliConfig import os async def create_avatar_pipeline (): # Initialize Simli service simli = SimliVideoService( SimliConfig( api_key = os.getenv( "SIMLI_API_KEY" ), face_id = os.getenv( "SIMLI_FACE_ID" ) ) ) # Create pipeline with Simli pipeline = Pipeline([ transport.input(), # Your audio input llm, # Language model service tts, # Text-to-speech service simli, # Simli video generation transport.output(), # Your video output handler ]) return pipeline ​ Frame Flow ​ Metrics Support The service collects processing metrics: Processing duration Time to First Byte (TTFB) API response times Audio processing metrics ​ Common Use Cases AI Video Avatars Virtual assistants Customer service Educational content Interactive Presentations Product demonstrations Training materials Marketing content Real-time Communication Video conferencing Virtual meetings Interactive broadcasts ​ Notes Handles real-time audio streaming Supports conversation interruptions Manages conversation lifecycle Automatic audio resampling Thread-safe processing WebRTC integration through Daily Includes comprehensive error handling OpenAI Tavus On this page Overview Installation Configuration Required Environment Variables Configuration Constructor Parameters for SimliConfig Constructor Parameters for SimliVideoService Input Frames Audio Input Control Frames Usage Example Frame Flow Metrics Support Common Use Cases Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/video_simli_408415be.txt b/video_simli_408415be.txt
new file mode 100644
index 0000000000000000000000000000000000000000..90fcf021adde26eedf52504340249731c2589a39
--- /dev/null
+++ b/video_simli_408415be.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/video/simli#param-face-id
+Title: Simli - Pipecat
+==================================================
+
+Simli - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Video Simli Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Simli Tavus Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview SimliVideoService creates AI avatar video responses by converting audio input into synchronized video and audio output through Simli’s WebRTC platform. It handles real-time audio streaming, video generation, and automatic audio resampling. ​ Installation Install the required dependencies: Copy Ask AI pip install "pipecat-ai[simli]" ​ Configuration ​ Required Environment Variables Copy Ask AI SIMLI_API_KEY = your_api_key SIMLI_FACE_ID = your_face_id Get your API key and Face ID by signing up at Simli ​ Configuration Copy Ask AI SimliVideoService( SimliConfig( SIMLI_API_KEY , SIMLI_FACE_ID ), useTurnServer = False , latencyInterval = 60 ) ​ Constructor Parameters for SimliConfig ​ apiKey str required Your Simli API key. This key is required for authenticating API requests. ​ faceId str required The face identifier for Simli. This is used to associate API interactions with a specific face or persona. ​ syncAudio bool default: "True" Indicates whether to synchronize audio streams. Defaults to True . Set to False to disable audio synchronization. ​ handleSilence bool default: "True" Determines if silence in audio streams should be handled automatically. Defaults to True . ​ maxSessionLength int default: "600" The maximum length of a session in seconds. Defaults to 600 (10 minutes). ​ maxIdleTime int default: "30" The maximum idle time (in seconds) allowed during a session before it is automatically terminated. Defaults to 30 seconds. ​ Constructor Parameters for SimliVideoService ​ simli_config SimliConfig required The configuration object for Simli. This must be an instance of simli_config and contains essential settings such as API key, face ID, and other session-related configurations. ​ use_turn_server bool default: "False" Determines whether a TURN server should be used for establishing connections. Defaults to False . Set to True if your network requires TURN for WebRTC connections. ​ latency_interval int default: "0" Delay (in seconds) between ping calls to calculate latency between your machine and simli server. Set to 0 to disable. ​ Input Frames ​ Audio Input ​ TTSAudioRawFrame Frame Raw audio data for avatar speech ​ Control Frames ​ TTSStartedFrame Frame Signals start of speech synthesis ​ TTSStoppedFrame Frame Signals end of speech synthesis ​ StartInterruptionFrame Frame Signals conversation interruption ​ EndFrame Frame Signals end of conversation ​ CancelFrame Frame Signals conversation cancellation ​ Usage Example Copy Ask AI from pipecat.pipeline.pipeline import Pipeline from pipecat.services.simli.video import SimliVideoService from simli import SimliConfig import os async def create_avatar_pipeline (): # Initialize Simli service simli = SimliVideoService( SimliConfig( api_key = os.getenv( "SIMLI_API_KEY" ), face_id = os.getenv( "SIMLI_FACE_ID" ) ) ) # Create pipeline with Simli pipeline = Pipeline([ transport.input(), # Your audio input llm, # Language model service tts, # Text-to-speech service simli, # Simli video generation transport.output(), # Your video output handler ]) return pipeline ​ Frame Flow ​ Metrics Support The service collects processing metrics: Processing duration Time to First Byte (TTFB) API response times Audio processing metrics ​ Common Use Cases AI Video Avatars Virtual assistants Customer service Educational content Interactive Presentations Product demonstrations Training materials Marketing content Real-time Communication Video conferencing Virtual meetings Interactive broadcasts ​ Notes Handles real-time audio streaming Supports conversation interruptions Manages conversation lifecycle Automatic audio resampling Thread-safe processing WebRTC integration through Daily Includes comprehensive error handling OpenAI Tavus On this page Overview Installation Configuration Required Environment Variables Configuration Constructor Parameters for SimliConfig Constructor Parameters for SimliVideoService Input Frames Audio Input Control Frames Usage Example Frame Flow Metrics Support Common Use Cases Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/video_simli_7c06fb68.txt b/video_simli_7c06fb68.txt
new file mode 100644
index 0000000000000000000000000000000000000000..59464bc56577ba6da4f8a3f10c37eb327fcef209
--- /dev/null
+++ b/video_simli_7c06fb68.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/video/simli#param-latency-interval
+Title: Simli - Pipecat
+==================================================
+
+Simli - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Video Simli Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Simli Tavus Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview SimliVideoService creates AI avatar video responses by converting audio input into synchronized video and audio output through Simli’s WebRTC platform. It handles real-time audio streaming, video generation, and automatic audio resampling. ​ Installation Install the required dependencies: Copy Ask AI pip install "pipecat-ai[simli]" ​ Configuration ​ Required Environment Variables Copy Ask AI SIMLI_API_KEY = your_api_key SIMLI_FACE_ID = your_face_id Get your API key and Face ID by signing up at Simli ​ Configuration Copy Ask AI SimliVideoService( SimliConfig( SIMLI_API_KEY , SIMLI_FACE_ID ), useTurnServer = False , latencyInterval = 60 ) ​ Constructor Parameters for SimliConfig ​ apiKey str required Your Simli API key. This key is required for authenticating API requests. ​ faceId str required The face identifier for Simli. This is used to associate API interactions with a specific face or persona. ​ syncAudio bool default: "True" Indicates whether to synchronize audio streams. Defaults to True . Set to False to disable audio synchronization. ​ handleSilence bool default: "True" Determines if silence in audio streams should be handled automatically. Defaults to True . ​ maxSessionLength int default: "600" The maximum length of a session in seconds. Defaults to 600 (10 minutes). ​ maxIdleTime int default: "30" The maximum idle time (in seconds) allowed during a session before it is automatically terminated. Defaults to 30 seconds. ​ Constructor Parameters for SimliVideoService ​ simli_config SimliConfig required The configuration object for Simli. This must be an instance of simli_config and contains essential settings such as API key, face ID, and other session-related configurations. ​ use_turn_server bool default: "False" Determines whether a TURN server should be used for establishing connections. Defaults to False . Set to True if your network requires TURN for WebRTC connections. ​ latency_interval int default: "0" Delay (in seconds) between ping calls to calculate latency between your machine and simli server. Set to 0 to disable. ​ Input Frames ​ Audio Input ​ TTSAudioRawFrame Frame Raw audio data for avatar speech ​ Control Frames ​ TTSStartedFrame Frame Signals start of speech synthesis ​ TTSStoppedFrame Frame Signals end of speech synthesis ​ StartInterruptionFrame Frame Signals conversation interruption ​ EndFrame Frame Signals end of conversation ​ CancelFrame Frame Signals conversation cancellation ​ Usage Example Copy Ask AI from pipecat.pipeline.pipeline import Pipeline from pipecat.services.simli.video import SimliVideoService from simli import SimliConfig import os async def create_avatar_pipeline (): # Initialize Simli service simli = SimliVideoService( SimliConfig( api_key = os.getenv( "SIMLI_API_KEY" ), face_id = os.getenv( "SIMLI_FACE_ID" ) ) ) # Create pipeline with Simli pipeline = Pipeline([ transport.input(), # Your audio input llm, # Language model service tts, # Text-to-speech service simli, # Simli video generation transport.output(), # Your video output handler ]) return pipeline ​ Frame Flow ​ Metrics Support The service collects processing metrics: Processing duration Time to First Byte (TTFB) API response times Audio processing metrics ​ Common Use Cases AI Video Avatars Virtual assistants Customer service Educational content Interactive Presentations Product demonstrations Training materials Marketing content Real-time Communication Video conferencing Virtual meetings Interactive broadcasts ​ Notes Handles real-time audio streaming Supports conversation interruptions Manages conversation lifecycle Automatic audio resampling Thread-safe processing WebRTC integration through Daily Includes comprehensive error handling OpenAI Tavus On this page Overview Installation Configuration Required Environment Variables Configuration Constructor Parameters for SimliConfig Constructor Parameters for SimliVideoService Input Frames Audio Input Control Frames Usage Example Frame Flow Metrics Support Common Use Cases Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/video_simli_94d962e9.txt b/video_simli_94d962e9.txt
new file mode 100644
index 0000000000000000000000000000000000000000..c195e8cb27c535eeff72301551bd1cf6e1f9bade
--- /dev/null
+++ b/video_simli_94d962e9.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/video/simli#param-end-frame
+Title: Simli - Pipecat
+==================================================
+
+Simli - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Video Simli Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Simli Tavus Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview SimliVideoService creates AI avatar video responses by converting audio input into synchronized video and audio output through Simli’s WebRTC platform. It handles real-time audio streaming, video generation, and automatic audio resampling. ​ Installation Install the required dependencies: Copy Ask AI pip install "pipecat-ai[simli]" ​ Configuration ​ Required Environment Variables Copy Ask AI SIMLI_API_KEY = your_api_key SIMLI_FACE_ID = your_face_id Get your API key and Face ID by signing up at Simli ​ Configuration Copy Ask AI SimliVideoService( SimliConfig( SIMLI_API_KEY , SIMLI_FACE_ID ), useTurnServer = False , latencyInterval = 60 ) ​ Constructor Parameters for SimliConfig ​ apiKey str required Your Simli API key. This key is required for authenticating API requests. ​ faceId str required The face identifier for Simli. This is used to associate API interactions with a specific face or persona. ​ syncAudio bool default: "True" Indicates whether to synchronize audio streams. Defaults to True . Set to False to disable audio synchronization. ​ handleSilence bool default: "True" Determines if silence in audio streams should be handled automatically. Defaults to True . ​ maxSessionLength int default: "600" The maximum length of a session in seconds. Defaults to 600 (10 minutes). ​ maxIdleTime int default: "30" The maximum idle time (in seconds) allowed during a session before it is automatically terminated. Defaults to 30 seconds. ​ Constructor Parameters for SimliVideoService ​ simli_config SimliConfig required The configuration object for Simli. This must be an instance of simli_config and contains essential settings such as API key, face ID, and other session-related configurations. ​ use_turn_server bool default: "False" Determines whether a TURN server should be used for establishing connections. Defaults to False . Set to True if your network requires TURN for WebRTC connections. ​ latency_interval int default: "0" Delay (in seconds) between ping calls to calculate latency between your machine and simli server. Set to 0 to disable. ​ Input Frames ​ Audio Input ​ TTSAudioRawFrame Frame Raw audio data for avatar speech ​ Control Frames ​ TTSStartedFrame Frame Signals start of speech synthesis ​ TTSStoppedFrame Frame Signals end of speech synthesis ​ StartInterruptionFrame Frame Signals conversation interruption ​ EndFrame Frame Signals end of conversation ​ CancelFrame Frame Signals conversation cancellation ​ Usage Example Copy Ask AI from pipecat.pipeline.pipeline import Pipeline from pipecat.services.simli.video import SimliVideoService from simli import SimliConfig import os async def create_avatar_pipeline (): # Initialize Simli service simli = SimliVideoService( SimliConfig( api_key = os.getenv( "SIMLI_API_KEY" ), face_id = os.getenv( "SIMLI_FACE_ID" ) ) ) # Create pipeline with Simli pipeline = Pipeline([ transport.input(), # Your audio input llm, # Language model service tts, # Text-to-speech service simli, # Simli video generation transport.output(), # Your video output handler ]) return pipeline ​ Frame Flow ​ Metrics Support The service collects processing metrics: Processing duration Time to First Byte (TTFB) API response times Audio processing metrics ​ Common Use Cases AI Video Avatars Virtual assistants Customer service Educational content Interactive Presentations Product demonstrations Training materials Marketing content Real-time Communication Video conferencing Virtual meetings Interactive broadcasts ​ Notes Handles real-time audio streaming Supports conversation interruptions Manages conversation lifecycle Automatic audio resampling Thread-safe processing WebRTC integration through Daily Includes comprehensive error handling OpenAI Tavus On this page Overview Installation Configuration Required Environment Variables Configuration Constructor Parameters for SimliConfig Constructor Parameters for SimliVideoService Input Frames Audio Input Control Frames Usage Example Frame Flow Metrics Support Common Use Cases Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/video_simli_9edfc283.txt b/video_simli_9edfc283.txt
new file mode 100644
index 0000000000000000000000000000000000000000..ecdb79ebf970001d5136e2749da74fac18168cf7
--- /dev/null
+++ b/video_simli_9edfc283.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/video/simli#metrics-support
+Title: Simli - Pipecat
+==================================================
+
+Simli - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Video Simli Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Simli Tavus Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview SimliVideoService creates AI avatar video responses by converting audio input into synchronized video and audio output through Simli’s WebRTC platform. It handles real-time audio streaming, video generation, and automatic audio resampling. ​ Installation Install the required dependencies: Copy Ask AI pip install "pipecat-ai[simli]" ​ Configuration ​ Required Environment Variables Copy Ask AI SIMLI_API_KEY = your_api_key SIMLI_FACE_ID = your_face_id Get your API key and Face ID by signing up at Simli ​ Configuration Copy Ask AI SimliVideoService( SimliConfig( SIMLI_API_KEY , SIMLI_FACE_ID ), useTurnServer = False , latencyInterval = 60 ) ​ Constructor Parameters for SimliConfig ​ apiKey str required Your Simli API key. This key is required for authenticating API requests. ​ faceId str required The face identifier for Simli. This is used to associate API interactions with a specific face or persona. ​ syncAudio bool default: "True" Indicates whether to synchronize audio streams. Defaults to True . Set to False to disable audio synchronization. ​ handleSilence bool default: "True" Determines if silence in audio streams should be handled automatically. Defaults to True . ​ maxSessionLength int default: "600" The maximum length of a session in seconds. Defaults to 600 (10 minutes). ​ maxIdleTime int default: "30" The maximum idle time (in seconds) allowed during a session before it is automatically terminated. Defaults to 30 seconds. ​ Constructor Parameters for SimliVideoService ​ simli_config SimliConfig required The configuration object for Simli. This must be an instance of simli_config and contains essential settings such as API key, face ID, and other session-related configurations. ​ use_turn_server bool default: "False" Determines whether a TURN server should be used for establishing connections. Defaults to False . Set to True if your network requires TURN for WebRTC connections. ​ latency_interval int default: "0" Delay (in seconds) between ping calls to calculate latency between your machine and simli server. Set to 0 to disable. ​ Input Frames ​ Audio Input ​ TTSAudioRawFrame Frame Raw audio data for avatar speech ​ Control Frames ​ TTSStartedFrame Frame Signals start of speech synthesis ​ TTSStoppedFrame Frame Signals end of speech synthesis ​ StartInterruptionFrame Frame Signals conversation interruption ​ EndFrame Frame Signals end of conversation ​ CancelFrame Frame Signals conversation cancellation ​ Usage Example Copy Ask AI from pipecat.pipeline.pipeline import Pipeline from pipecat.services.simli.video import SimliVideoService from simli import SimliConfig import os async def create_avatar_pipeline (): # Initialize Simli service simli = SimliVideoService( SimliConfig( api_key = os.getenv( "SIMLI_API_KEY" ), face_id = os.getenv( "SIMLI_FACE_ID" ) ) ) # Create pipeline with Simli pipeline = Pipeline([ transport.input(), # Your audio input llm, # Language model service tts, # Text-to-speech service simli, # Simli video generation transport.output(), # Your video output handler ]) return pipeline ​ Frame Flow ​ Metrics Support The service collects processing metrics: Processing duration Time to First Byte (TTFB) API response times Audio processing metrics ​ Common Use Cases AI Video Avatars Virtual assistants Customer service Educational content Interactive Presentations Product demonstrations Training materials Marketing content Real-time Communication Video conferencing Virtual meetings Interactive broadcasts ​ Notes Handles real-time audio streaming Supports conversation interruptions Manages conversation lifecycle Automatic audio resampling Thread-safe processing WebRTC integration through Daily Includes comprehensive error handling OpenAI Tavus On this page Overview Installation Configuration Required Environment Variables Configuration Constructor Parameters for SimliConfig Constructor Parameters for SimliVideoService Input Frames Audio Input Control Frames Usage Example Frame Flow Metrics Support Common Use Cases Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/video_simli_bfee767a.txt b/video_simli_bfee767a.txt
new file mode 100644
index 0000000000000000000000000000000000000000..00d772392edb390b190ae837d20465774250d5b8
--- /dev/null
+++ b/video_simli_bfee767a.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/video/simli#frame-flow
+Title: Simli - Pipecat
+==================================================
+
+Simli - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Video Simli Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Simli Tavus Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview SimliVideoService creates AI avatar video responses by converting audio input into synchronized video and audio output through Simli’s WebRTC platform. It handles real-time audio streaming, video generation, and automatic audio resampling. ​ Installation Install the required dependencies: Copy Ask AI pip install "pipecat-ai[simli]" ​ Configuration ​ Required Environment Variables Copy Ask AI SIMLI_API_KEY = your_api_key SIMLI_FACE_ID = your_face_id Get your API key and Face ID by signing up at Simli ​ Configuration Copy Ask AI SimliVideoService( SimliConfig( SIMLI_API_KEY , SIMLI_FACE_ID ), useTurnServer = False , latencyInterval = 60 ) ​ Constructor Parameters for SimliConfig ​ apiKey str required Your Simli API key. This key is required for authenticating API requests. ​ faceId str required The face identifier for Simli. This is used to associate API interactions with a specific face or persona. ​ syncAudio bool default: "True" Indicates whether to synchronize audio streams. Defaults to True . Set to False to disable audio synchronization. ​ handleSilence bool default: "True" Determines if silence in audio streams should be handled automatically. Defaults to True . ​ maxSessionLength int default: "600" The maximum length of a session in seconds. Defaults to 600 (10 minutes). ​ maxIdleTime int default: "30" The maximum idle time (in seconds) allowed during a session before it is automatically terminated. Defaults to 30 seconds. ​ Constructor Parameters for SimliVideoService ​ simli_config SimliConfig required The configuration object for Simli. This must be an instance of simli_config and contains essential settings such as API key, face ID, and other session-related configurations. ​ use_turn_server bool default: "False" Determines whether a TURN server should be used for establishing connections. Defaults to False . Set to True if your network requires TURN for WebRTC connections. ​ latency_interval int default: "0" Delay (in seconds) between ping calls to calculate latency between your machine and simli server. Set to 0 to disable. ​ Input Frames ​ Audio Input ​ TTSAudioRawFrame Frame Raw audio data for avatar speech ​ Control Frames ​ TTSStartedFrame Frame Signals start of speech synthesis ​ TTSStoppedFrame Frame Signals end of speech synthesis ​ StartInterruptionFrame Frame Signals conversation interruption ​ EndFrame Frame Signals end of conversation ​ CancelFrame Frame Signals conversation cancellation ​ Usage Example Copy Ask AI from pipecat.pipeline.pipeline import Pipeline from pipecat.services.simli.video import SimliVideoService from simli import SimliConfig import os async def create_avatar_pipeline (): # Initialize Simli service simli = SimliVideoService( SimliConfig( api_key = os.getenv( "SIMLI_API_KEY" ), face_id = os.getenv( "SIMLI_FACE_ID" ) ) ) # Create pipeline with Simli pipeline = Pipeline([ transport.input(), # Your audio input llm, # Language model service tts, # Text-to-speech service simli, # Simli video generation transport.output(), # Your video output handler ]) return pipeline ​ Frame Flow ​ Metrics Support The service collects processing metrics: Processing duration Time to First Byte (TTFB) API response times Audio processing metrics ​ Common Use Cases AI Video Avatars Virtual assistants Customer service Educational content Interactive Presentations Product demonstrations Training materials Marketing content Real-time Communication Video conferencing Virtual meetings Interactive broadcasts ​ Notes Handles real-time audio streaming Supports conversation interruptions Manages conversation lifecycle Automatic audio resampling Thread-safe processing WebRTC integration through Daily Includes comprehensive error handling OpenAI Tavus On this page Overview Installation Configuration Required Environment Variables Configuration Constructor Parameters for SimliConfig Constructor Parameters for SimliVideoService Input Frames Audio Input Control Frames Usage Example Frame Flow Metrics Support Common Use Cases Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/video_simli_e296dc65.txt b/video_simli_e296dc65.txt
new file mode 100644
index 0000000000000000000000000000000000000000..8468f61eaf54543b751a133589153258aed403b3
--- /dev/null
+++ b/video_simli_e296dc65.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/video/simli#param-max-idle-time
+Title: Simli - Pipecat
+==================================================
+
+Simli - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Video Simli Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Simli Tavus Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview SimliVideoService creates AI avatar video responses by converting audio input into synchronized video and audio output through Simli’s WebRTC platform. It handles real-time audio streaming, video generation, and automatic audio resampling. ​ Installation Install the required dependencies: Copy Ask AI pip install "pipecat-ai[simli]" ​ Configuration ​ Required Environment Variables Copy Ask AI SIMLI_API_KEY = your_api_key SIMLI_FACE_ID = your_face_id Get your API key and Face ID by signing up at Simli ​ Configuration Copy Ask AI SimliVideoService( SimliConfig( SIMLI_API_KEY , SIMLI_FACE_ID ), useTurnServer = False , latencyInterval = 60 ) ​ Constructor Parameters for SimliConfig ​ apiKey str required Your Simli API key. This key is required for authenticating API requests. ​ faceId str required The face identifier for Simli. This is used to associate API interactions with a specific face or persona. ​ syncAudio bool default: "True" Indicates whether to synchronize audio streams. Defaults to True . Set to False to disable audio synchronization. ​ handleSilence bool default: "True" Determines if silence in audio streams should be handled automatically. Defaults to True . ​ maxSessionLength int default: "600" The maximum length of a session in seconds. Defaults to 600 (10 minutes). ​ maxIdleTime int default: "30" The maximum idle time (in seconds) allowed during a session before it is automatically terminated. Defaults to 30 seconds. ​ Constructor Parameters for SimliVideoService ​ simli_config SimliConfig required The configuration object for Simli. This must be an instance of simli_config and contains essential settings such as API key, face ID, and other session-related configurations. ​ use_turn_server bool default: "False" Determines whether a TURN server should be used for establishing connections. Defaults to False . Set to True if your network requires TURN for WebRTC connections. ​ latency_interval int default: "0" Delay (in seconds) between ping calls to calculate latency between your machine and simli server. Set to 0 to disable. ​ Input Frames ​ Audio Input ​ TTSAudioRawFrame Frame Raw audio data for avatar speech ​ Control Frames ​ TTSStartedFrame Frame Signals start of speech synthesis ​ TTSStoppedFrame Frame Signals end of speech synthesis ​ StartInterruptionFrame Frame Signals conversation interruption ​ EndFrame Frame Signals end of conversation ​ CancelFrame Frame Signals conversation cancellation ​ Usage Example Copy Ask AI from pipecat.pipeline.pipeline import Pipeline from pipecat.services.simli.video import SimliVideoService from simli import SimliConfig import os async def create_avatar_pipeline (): # Initialize Simli service simli = SimliVideoService( SimliConfig( api_key = os.getenv( "SIMLI_API_KEY" ), face_id = os.getenv( "SIMLI_FACE_ID" ) ) ) # Create pipeline with Simli pipeline = Pipeline([ transport.input(), # Your audio input llm, # Language model service tts, # Text-to-speech service simli, # Simli video generation transport.output(), # Your video output handler ]) return pipeline ​ Frame Flow ​ Metrics Support The service collects processing metrics: Processing duration Time to First Byte (TTFB) API response times Audio processing metrics ​ Common Use Cases AI Video Avatars Virtual assistants Customer service Educational content Interactive Presentations Product demonstrations Training materials Marketing content Real-time Communication Video conferencing Virtual meetings Interactive broadcasts ​ Notes Handles real-time audio streaming Supports conversation interruptions Manages conversation lifecycle Automatic audio resampling Thread-safe processing WebRTC integration through Daily Includes comprehensive error handling OpenAI Tavus On this page Overview Installation Configuration Required Environment Variables Configuration Constructor Parameters for SimliConfig Constructor Parameters for SimliVideoService Input Frames Audio Input Control Frames Usage Example Frame Flow Metrics Support Common Use Cases Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/video_tavus_051444da.txt b/video_tavus_051444da.txt
new file mode 100644
index 0000000000000000000000000000000000000000..03972219fd096c8d23e9d5720f1cba41284ce613
--- /dev/null
+++ b/video_tavus_051444da.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/video/tavus#overview
+Title: Tavus - Pipecat
+==================================================
+
+Tavus - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Video Tavus Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Simli Tavus Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview TavusVideoService enables the creation of AI avatar video responses by sending audio to Tavus’s API. It handles real-time audio streaming, conversation management, and video generation through Tavus’s platform. ​ Installation To use TavusVideoService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[tavus]" You’ll need to set up the following environment variables: TAVUS_API_KEY - Your Tavus API key TAVUS_REPLICA_ID - Your Tavus replica identifier You can obtain a Tavus API key by signing up at Tavus . ​ Configuration ​ Constructor Parameters ​ api_key str required Your Tavus API key ​ replica_id str required Tavus replica identifier ​ persona_id str default: "pipecat0" Tavus persona identifier The persona ID is optional and defaults to pipecat0 . To use the LLM output and TTS voice, do not set the persona_id. Instead, leave it set to the default value, pipecat0 . ​ session aiohttp.ClientSession required HTTP client session for API communication ​ Input Frames ​ Audio Input ​ TTSAudioRawFrame Frame Raw audio data for avatar speech ​ Control Frames ​ TTSStartedFrame Frame Signals start of speech synthesis ​ TTSStoppedFrame Frame Signals end of speech synthesis ​ StartInterruptionFrame Frame Signals conversation interruption ​ EndFrame Frame Signals end of conversation ​ CancelFrame Frame Signals conversation cancellation ​ Usage Example Copy Ask AI from pipecat.services.tavus.video import TavusVideoService import aiohttp async def main (): async with aiohttp.ClientSession() as session: # Configure service tavus = TavusVideoService( api_key = "your-tavus-api-key" , replica_id = "your-replica-id" , session = session ) # Initialize conversation room_url = await tavus.initialize() # Get persona name persona_name = await tavus.get_persona_name() transport = DailyTransport( room_url = room_url, # Your Tavus room URL token = None , bot_name = "Pipecat bot" , params = DailyParams( audio_in_enabled = True , vad_analyzer = SileroVADAnalyzer(), ), ) # Use in pipeline pipeline = Pipeline( [ transport.input(), # Transport user input stt, # STT context_aggregator.user(), # User responses llm, # LLM tts, # TTS tavus, # Tavus output layer transport.output(), # Transport bot output context_aggregator.assistant(), # Assistant spoken responses ] ) ​ API Methods ​ initialize Copy Ask AI async def initialize ( self ) -> str : Initializes a new conversation and returns the conversation URL. ​ get_persona_name Copy Ask AI async def get_persona_name ( self ) -> str : Retrieves the name of the configured persona. ​ Frame Flow ​ Metrics Support The service collects processing metrics: Processing duration Time to First Byte (TTFB) API response times Audio processing metrics ​ Common Use Cases AI Video Avatars Virtual assistants Customer service Educational content Interactive Presentations Product demonstrations Training materials Marketing content Real-time Communication Video conferencing Virtual meetings Interactive broadcasts ​ Notes Handles real-time audio streaming Supports conversation interruptions Manages conversation lifecycle Automatic audio resampling Thread-safe processing WebRTC integration through Daily Includes comprehensive error handling Simli Mem0 On this page Overview Installation Configuration Constructor Parameters Input Frames Audio Input Control Frames Usage Example API Methods initialize get_persona_name Frame Flow Metrics Support Common Use Cases Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/video_tavus_1432f7d5.txt b/video_tavus_1432f7d5.txt
new file mode 100644
index 0000000000000000000000000000000000000000..0903ac2eaf5c043e1a12208b1f6bdbc7af148b43
--- /dev/null
+++ b/video_tavus_1432f7d5.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/video/tavus#param-persona-id
+Title: Tavus - Pipecat
+==================================================
+
+Tavus - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Video Tavus Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Simli Tavus Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview TavusVideoService enables the creation of AI avatar video responses by sending audio to Tavus’s API. It handles real-time audio streaming, conversation management, and video generation through Tavus’s platform. ​ Installation To use TavusVideoService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[tavus]" You’ll need to set up the following environment variables: TAVUS_API_KEY - Your Tavus API key TAVUS_REPLICA_ID - Your Tavus replica identifier You can obtain a Tavus API key by signing up at Tavus . ​ Configuration ​ Constructor Parameters ​ api_key str required Your Tavus API key ​ replica_id str required Tavus replica identifier ​ persona_id str default: "pipecat0" Tavus persona identifier The persona ID is optional and defaults to pipecat0 . To use the LLM output and TTS voice, do not set the persona_id. Instead, leave it set to the default value, pipecat0 . ​ session aiohttp.ClientSession required HTTP client session for API communication ​ Input Frames ​ Audio Input ​ TTSAudioRawFrame Frame Raw audio data for avatar speech ​ Control Frames ​ TTSStartedFrame Frame Signals start of speech synthesis ​ TTSStoppedFrame Frame Signals end of speech synthesis ​ StartInterruptionFrame Frame Signals conversation interruption ​ EndFrame Frame Signals end of conversation ​ CancelFrame Frame Signals conversation cancellation ​ Usage Example Copy Ask AI from pipecat.services.tavus.video import TavusVideoService import aiohttp async def main (): async with aiohttp.ClientSession() as session: # Configure service tavus = TavusVideoService( api_key = "your-tavus-api-key" , replica_id = "your-replica-id" , session = session ) # Initialize conversation room_url = await tavus.initialize() # Get persona name persona_name = await tavus.get_persona_name() transport = DailyTransport( room_url = room_url, # Your Tavus room URL token = None , bot_name = "Pipecat bot" , params = DailyParams( audio_in_enabled = True , vad_analyzer = SileroVADAnalyzer(), ), ) # Use in pipeline pipeline = Pipeline( [ transport.input(), # Transport user input stt, # STT context_aggregator.user(), # User responses llm, # LLM tts, # TTS tavus, # Tavus output layer transport.output(), # Transport bot output context_aggregator.assistant(), # Assistant spoken responses ] ) ​ API Methods ​ initialize Copy Ask AI async def initialize ( self ) -> str : Initializes a new conversation and returns the conversation URL. ​ get_persona_name Copy Ask AI async def get_persona_name ( self ) -> str : Retrieves the name of the configured persona. ​ Frame Flow ​ Metrics Support The service collects processing metrics: Processing duration Time to First Byte (TTFB) API response times Audio processing metrics ​ Common Use Cases AI Video Avatars Virtual assistants Customer service Educational content Interactive Presentations Product demonstrations Training materials Marketing content Real-time Communication Video conferencing Virtual meetings Interactive broadcasts ​ Notes Handles real-time audio streaming Supports conversation interruptions Manages conversation lifecycle Automatic audio resampling Thread-safe processing WebRTC integration through Daily Includes comprehensive error handling Simli Mem0 On this page Overview Installation Configuration Constructor Parameters Input Frames Audio Input Control Frames Usage Example API Methods initialize get_persona_name Frame Flow Metrics Support Common Use Cases Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/video_tavus_2ca1e798.txt b/video_tavus_2ca1e798.txt
new file mode 100644
index 0000000000000000000000000000000000000000..ace30cc406aab76ada59ee7dbd9757b172cd6108
--- /dev/null
+++ b/video_tavus_2ca1e798.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/video/tavus#param-end-frame
+Title: Tavus - Pipecat
+==================================================
+
+Tavus - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Video Tavus Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Simli Tavus Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview TavusVideoService enables the creation of AI avatar video responses by sending audio to Tavus’s API. It handles real-time audio streaming, conversation management, and video generation through Tavus’s platform. ​ Installation To use TavusVideoService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[tavus]" You’ll need to set up the following environment variables: TAVUS_API_KEY - Your Tavus API key TAVUS_REPLICA_ID - Your Tavus replica identifier You can obtain a Tavus API key by signing up at Tavus . ​ Configuration ​ Constructor Parameters ​ api_key str required Your Tavus API key ​ replica_id str required Tavus replica identifier ​ persona_id str default: "pipecat0" Tavus persona identifier The persona ID is optional and defaults to pipecat0 . To use the LLM output and TTS voice, do not set the persona_id. Instead, leave it set to the default value, pipecat0 . ​ session aiohttp.ClientSession required HTTP client session for API communication ​ Input Frames ​ Audio Input ​ TTSAudioRawFrame Frame Raw audio data for avatar speech ​ Control Frames ​ TTSStartedFrame Frame Signals start of speech synthesis ​ TTSStoppedFrame Frame Signals end of speech synthesis ​ StartInterruptionFrame Frame Signals conversation interruption ​ EndFrame Frame Signals end of conversation ​ CancelFrame Frame Signals conversation cancellation ​ Usage Example Copy Ask AI from pipecat.services.tavus.video import TavusVideoService import aiohttp async def main (): async with aiohttp.ClientSession() as session: # Configure service tavus = TavusVideoService( api_key = "your-tavus-api-key" , replica_id = "your-replica-id" , session = session ) # Initialize conversation room_url = await tavus.initialize() # Get persona name persona_name = await tavus.get_persona_name() transport = DailyTransport( room_url = room_url, # Your Tavus room URL token = None , bot_name = "Pipecat bot" , params = DailyParams( audio_in_enabled = True , vad_analyzer = SileroVADAnalyzer(), ), ) # Use in pipeline pipeline = Pipeline( [ transport.input(), # Transport user input stt, # STT context_aggregator.user(), # User responses llm, # LLM tts, # TTS tavus, # Tavus output layer transport.output(), # Transport bot output context_aggregator.assistant(), # Assistant spoken responses ] ) ​ API Methods ​ initialize Copy Ask AI async def initialize ( self ) -> str : Initializes a new conversation and returns the conversation URL. ​ get_persona_name Copy Ask AI async def get_persona_name ( self ) -> str : Retrieves the name of the configured persona. ​ Frame Flow ​ Metrics Support The service collects processing metrics: Processing duration Time to First Byte (TTFB) API response times Audio processing metrics ​ Common Use Cases AI Video Avatars Virtual assistants Customer service Educational content Interactive Presentations Product demonstrations Training materials Marketing content Real-time Communication Video conferencing Virtual meetings Interactive broadcasts ​ Notes Handles real-time audio streaming Supports conversation interruptions Manages conversation lifecycle Automatic audio resampling Thread-safe processing WebRTC integration through Daily Includes comprehensive error handling Simli Mem0 On this page Overview Installation Configuration Constructor Parameters Input Frames Audio Input Control Frames Usage Example API Methods initialize get_persona_name Frame Flow Metrics Support Common Use Cases Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/video_tavus_3bf3c9d5.txt b/video_tavus_3bf3c9d5.txt
new file mode 100644
index 0000000000000000000000000000000000000000..5fbca2478c08146c1362230fa1ff97c023928de4
--- /dev/null
+++ b/video_tavus_3bf3c9d5.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/video/tavus#input-frames
+Title: Tavus - Pipecat
+==================================================
+
+Tavus - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Video Tavus Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Simli Tavus Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview TavusVideoService enables the creation of AI avatar video responses by sending audio to Tavus’s API. It handles real-time audio streaming, conversation management, and video generation through Tavus’s platform. ​ Installation To use TavusVideoService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[tavus]" You’ll need to set up the following environment variables: TAVUS_API_KEY - Your Tavus API key TAVUS_REPLICA_ID - Your Tavus replica identifier You can obtain a Tavus API key by signing up at Tavus . ​ Configuration ​ Constructor Parameters ​ api_key str required Your Tavus API key ​ replica_id str required Tavus replica identifier ​ persona_id str default: "pipecat0" Tavus persona identifier The persona ID is optional and defaults to pipecat0 . To use the LLM output and TTS voice, do not set the persona_id. Instead, leave it set to the default value, pipecat0 . ​ session aiohttp.ClientSession required HTTP client session for API communication ​ Input Frames ​ Audio Input ​ TTSAudioRawFrame Frame Raw audio data for avatar speech ​ Control Frames ​ TTSStartedFrame Frame Signals start of speech synthesis ​ TTSStoppedFrame Frame Signals end of speech synthesis ​ StartInterruptionFrame Frame Signals conversation interruption ​ EndFrame Frame Signals end of conversation ​ CancelFrame Frame Signals conversation cancellation ​ Usage Example Copy Ask AI from pipecat.services.tavus.video import TavusVideoService import aiohttp async def main (): async with aiohttp.ClientSession() as session: # Configure service tavus = TavusVideoService( api_key = "your-tavus-api-key" , replica_id = "your-replica-id" , session = session ) # Initialize conversation room_url = await tavus.initialize() # Get persona name persona_name = await tavus.get_persona_name() transport = DailyTransport( room_url = room_url, # Your Tavus room URL token = None , bot_name = "Pipecat bot" , params = DailyParams( audio_in_enabled = True , vad_analyzer = SileroVADAnalyzer(), ), ) # Use in pipeline pipeline = Pipeline( [ transport.input(), # Transport user input stt, # STT context_aggregator.user(), # User responses llm, # LLM tts, # TTS tavus, # Tavus output layer transport.output(), # Transport bot output context_aggregator.assistant(), # Assistant spoken responses ] ) ​ API Methods ​ initialize Copy Ask AI async def initialize ( self ) -> str : Initializes a new conversation and returns the conversation URL. ​ get_persona_name Copy Ask AI async def get_persona_name ( self ) -> str : Retrieves the name of the configured persona. ​ Frame Flow ​ Metrics Support The service collects processing metrics: Processing duration Time to First Byte (TTFB) API response times Audio processing metrics ​ Common Use Cases AI Video Avatars Virtual assistants Customer service Educational content Interactive Presentations Product demonstrations Training materials Marketing content Real-time Communication Video conferencing Virtual meetings Interactive broadcasts ​ Notes Handles real-time audio streaming Supports conversation interruptions Manages conversation lifecycle Automatic audio resampling Thread-safe processing WebRTC integration through Daily Includes comprehensive error handling Simli Mem0 On this page Overview Installation Configuration Constructor Parameters Input Frames Audio Input Control Frames Usage Example API Methods initialize get_persona_name Frame Flow Metrics Support Common Use Cases Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/video_tavus_674e9dad.txt b/video_tavus_674e9dad.txt
new file mode 100644
index 0000000000000000000000000000000000000000..11e95317f5a209809a33ca0a23c8c491a735c466
--- /dev/null
+++ b/video_tavus_674e9dad.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/video/tavus#param-start-interruption-frame
+Title: Tavus - Pipecat
+==================================================
+
+Tavus - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Video Tavus Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Simli Tavus Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview TavusVideoService enables the creation of AI avatar video responses by sending audio to Tavus’s API. It handles real-time audio streaming, conversation management, and video generation through Tavus’s platform. ​ Installation To use TavusVideoService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[tavus]" You’ll need to set up the following environment variables: TAVUS_API_KEY - Your Tavus API key TAVUS_REPLICA_ID - Your Tavus replica identifier You can obtain a Tavus API key by signing up at Tavus . ​ Configuration ​ Constructor Parameters ​ api_key str required Your Tavus API key ​ replica_id str required Tavus replica identifier ​ persona_id str default: "pipecat0" Tavus persona identifier The persona ID is optional and defaults to pipecat0 . To use the LLM output and TTS voice, do not set the persona_id. Instead, leave it set to the default value, pipecat0 . ​ session aiohttp.ClientSession required HTTP client session for API communication ​ Input Frames ​ Audio Input ​ TTSAudioRawFrame Frame Raw audio data for avatar speech ​ Control Frames ​ TTSStartedFrame Frame Signals start of speech synthesis ​ TTSStoppedFrame Frame Signals end of speech synthesis ​ StartInterruptionFrame Frame Signals conversation interruption ​ EndFrame Frame Signals end of conversation ​ CancelFrame Frame Signals conversation cancellation ​ Usage Example Copy Ask AI from pipecat.services.tavus.video import TavusVideoService import aiohttp async def main (): async with aiohttp.ClientSession() as session: # Configure service tavus = TavusVideoService( api_key = "your-tavus-api-key" , replica_id = "your-replica-id" , session = session ) # Initialize conversation room_url = await tavus.initialize() # Get persona name persona_name = await tavus.get_persona_name() transport = DailyTransport( room_url = room_url, # Your Tavus room URL token = None , bot_name = "Pipecat bot" , params = DailyParams( audio_in_enabled = True , vad_analyzer = SileroVADAnalyzer(), ), ) # Use in pipeline pipeline = Pipeline( [ transport.input(), # Transport user input stt, # STT context_aggregator.user(), # User responses llm, # LLM tts, # TTS tavus, # Tavus output layer transport.output(), # Transport bot output context_aggregator.assistant(), # Assistant spoken responses ] ) ​ API Methods ​ initialize Copy Ask AI async def initialize ( self ) -> str : Initializes a new conversation and returns the conversation URL. ​ get_persona_name Copy Ask AI async def get_persona_name ( self ) -> str : Retrieves the name of the configured persona. ​ Frame Flow ​ Metrics Support The service collects processing metrics: Processing duration Time to First Byte (TTFB) API response times Audio processing metrics ​ Common Use Cases AI Video Avatars Virtual assistants Customer service Educational content Interactive Presentations Product demonstrations Training materials Marketing content Real-time Communication Video conferencing Virtual meetings Interactive broadcasts ​ Notes Handles real-time audio streaming Supports conversation interruptions Manages conversation lifecycle Automatic audio resampling Thread-safe processing WebRTC integration through Daily Includes comprehensive error handling Simli Mem0 On this page Overview Installation Configuration Constructor Parameters Input Frames Audio Input Control Frames Usage Example API Methods initialize get_persona_name Frame Flow Metrics Support Common Use Cases Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/video_tavus_a3cec1c5.txt b/video_tavus_a3cec1c5.txt
new file mode 100644
index 0000000000000000000000000000000000000000..fe8e758362e1078e97d70f7aaf8cb15cc57d7334
--- /dev/null
+++ b/video_tavus_a3cec1c5.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/video/tavus#initialize
+Title: Tavus - Pipecat
+==================================================
+
+Tavus - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Video Tavus Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Simli Tavus Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview TavusVideoService enables the creation of AI avatar video responses by sending audio to Tavus’s API. It handles real-time audio streaming, conversation management, and video generation through Tavus’s platform. ​ Installation To use TavusVideoService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[tavus]" You’ll need to set up the following environment variables: TAVUS_API_KEY - Your Tavus API key TAVUS_REPLICA_ID - Your Tavus replica identifier You can obtain a Tavus API key by signing up at Tavus . ​ Configuration ​ Constructor Parameters ​ api_key str required Your Tavus API key ​ replica_id str required Tavus replica identifier ​ persona_id str default: "pipecat0" Tavus persona identifier The persona ID is optional and defaults to pipecat0 . To use the LLM output and TTS voice, do not set the persona_id. Instead, leave it set to the default value, pipecat0 . ​ session aiohttp.ClientSession required HTTP client session for API communication ​ Input Frames ​ Audio Input ​ TTSAudioRawFrame Frame Raw audio data for avatar speech ​ Control Frames ​ TTSStartedFrame Frame Signals start of speech synthesis ​ TTSStoppedFrame Frame Signals end of speech synthesis ​ StartInterruptionFrame Frame Signals conversation interruption ​ EndFrame Frame Signals end of conversation ​ CancelFrame Frame Signals conversation cancellation ​ Usage Example Copy Ask AI from pipecat.services.tavus.video import TavusVideoService import aiohttp async def main (): async with aiohttp.ClientSession() as session: # Configure service tavus = TavusVideoService( api_key = "your-tavus-api-key" , replica_id = "your-replica-id" , session = session ) # Initialize conversation room_url = await tavus.initialize() # Get persona name persona_name = await tavus.get_persona_name() transport = DailyTransport( room_url = room_url, # Your Tavus room URL token = None , bot_name = "Pipecat bot" , params = DailyParams( audio_in_enabled = True , vad_analyzer = SileroVADAnalyzer(), ), ) # Use in pipeline pipeline = Pipeline( [ transport.input(), # Transport user input stt, # STT context_aggregator.user(), # User responses llm, # LLM tts, # TTS tavus, # Tavus output layer transport.output(), # Transport bot output context_aggregator.assistant(), # Assistant spoken responses ] ) ​ API Methods ​ initialize Copy Ask AI async def initialize ( self ) -> str : Initializes a new conversation and returns the conversation URL. ​ get_persona_name Copy Ask AI async def get_persona_name ( self ) -> str : Retrieves the name of the configured persona. ​ Frame Flow ​ Metrics Support The service collects processing metrics: Processing duration Time to First Byte (TTFB) API response times Audio processing metrics ​ Common Use Cases AI Video Avatars Virtual assistants Customer service Educational content Interactive Presentations Product demonstrations Training materials Marketing content Real-time Communication Video conferencing Virtual meetings Interactive broadcasts ​ Notes Handles real-time audio streaming Supports conversation interruptions Manages conversation lifecycle Automatic audio resampling Thread-safe processing WebRTC integration through Daily Includes comprehensive error handling Simli Mem0 On this page Overview Installation Configuration Constructor Parameters Input Frames Audio Input Control Frames Usage Example API Methods initialize get_persona_name Frame Flow Metrics Support Common Use Cases Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/video_tavus_a8fa9315.txt b/video_tavus_a8fa9315.txt
new file mode 100644
index 0000000000000000000000000000000000000000..144603a2e018a9004536c87deeb2d9e52ba67fe2
--- /dev/null
+++ b/video_tavus_a8fa9315.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/video/tavus#api-methods
+Title: Tavus - Pipecat
+==================================================
+
+Tavus - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Video Tavus Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Simli Tavus Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview TavusVideoService enables the creation of AI avatar video responses by sending audio to Tavus’s API. It handles real-time audio streaming, conversation management, and video generation through Tavus’s platform. ​ Installation To use TavusVideoService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[tavus]" You’ll need to set up the following environment variables: TAVUS_API_KEY - Your Tavus API key TAVUS_REPLICA_ID - Your Tavus replica identifier You can obtain a Tavus API key by signing up at Tavus . ​ Configuration ​ Constructor Parameters ​ api_key str required Your Tavus API key ​ replica_id str required Tavus replica identifier ​ persona_id str default: "pipecat0" Tavus persona identifier The persona ID is optional and defaults to pipecat0 . To use the LLM output and TTS voice, do not set the persona_id. Instead, leave it set to the default value, pipecat0 . ​ session aiohttp.ClientSession required HTTP client session for API communication ​ Input Frames ​ Audio Input ​ TTSAudioRawFrame Frame Raw audio data for avatar speech ​ Control Frames ​ TTSStartedFrame Frame Signals start of speech synthesis ​ TTSStoppedFrame Frame Signals end of speech synthesis ​ StartInterruptionFrame Frame Signals conversation interruption ​ EndFrame Frame Signals end of conversation ​ CancelFrame Frame Signals conversation cancellation ​ Usage Example Copy Ask AI from pipecat.services.tavus.video import TavusVideoService import aiohttp async def main (): async with aiohttp.ClientSession() as session: # Configure service tavus = TavusVideoService( api_key = "your-tavus-api-key" , replica_id = "your-replica-id" , session = session ) # Initialize conversation room_url = await tavus.initialize() # Get persona name persona_name = await tavus.get_persona_name() transport = DailyTransport( room_url = room_url, # Your Tavus room URL token = None , bot_name = "Pipecat bot" , params = DailyParams( audio_in_enabled = True , vad_analyzer = SileroVADAnalyzer(), ), ) # Use in pipeline pipeline = Pipeline( [ transport.input(), # Transport user input stt, # STT context_aggregator.user(), # User responses llm, # LLM tts, # TTS tavus, # Tavus output layer transport.output(), # Transport bot output context_aggregator.assistant(), # Assistant spoken responses ] ) ​ API Methods ​ initialize Copy Ask AI async def initialize ( self ) -> str : Initializes a new conversation and returns the conversation URL. ​ get_persona_name Copy Ask AI async def get_persona_name ( self ) -> str : Retrieves the name of the configured persona. ​ Frame Flow ​ Metrics Support The service collects processing metrics: Processing duration Time to First Byte (TTFB) API response times Audio processing metrics ​ Common Use Cases AI Video Avatars Virtual assistants Customer service Educational content Interactive Presentations Product demonstrations Training materials Marketing content Real-time Communication Video conferencing Virtual meetings Interactive broadcasts ​ Notes Handles real-time audio streaming Supports conversation interruptions Manages conversation lifecycle Automatic audio resampling Thread-safe processing WebRTC integration through Daily Includes comprehensive error handling Simli Mem0 On this page Overview Installation Configuration Constructor Parameters Input Frames Audio Input Control Frames Usage Example API Methods initialize get_persona_name Frame Flow Metrics Support Common Use Cases Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/video_tavus_d78b9975.txt b/video_tavus_d78b9975.txt
new file mode 100644
index 0000000000000000000000000000000000000000..817da83f2919a6f6158fae7186ddd63288f48e55
--- /dev/null
+++ b/video_tavus_d78b9975.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/video/tavus#installation
+Title: Tavus - Pipecat
+==================================================
+
+Tavus - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Video Tavus Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Simli Tavus Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview TavusVideoService enables the creation of AI avatar video responses by sending audio to Tavus’s API. It handles real-time audio streaming, conversation management, and video generation through Tavus’s platform. ​ Installation To use TavusVideoService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[tavus]" You’ll need to set up the following environment variables: TAVUS_API_KEY - Your Tavus API key TAVUS_REPLICA_ID - Your Tavus replica identifier You can obtain a Tavus API key by signing up at Tavus . ​ Configuration ​ Constructor Parameters ​ api_key str required Your Tavus API key ​ replica_id str required Tavus replica identifier ​ persona_id str default: "pipecat0" Tavus persona identifier The persona ID is optional and defaults to pipecat0 . To use the LLM output and TTS voice, do not set the persona_id. Instead, leave it set to the default value, pipecat0 . ​ session aiohttp.ClientSession required HTTP client session for API communication ​ Input Frames ​ Audio Input ​ TTSAudioRawFrame Frame Raw audio data for avatar speech ​ Control Frames ​ TTSStartedFrame Frame Signals start of speech synthesis ​ TTSStoppedFrame Frame Signals end of speech synthesis ​ StartInterruptionFrame Frame Signals conversation interruption ​ EndFrame Frame Signals end of conversation ​ CancelFrame Frame Signals conversation cancellation ​ Usage Example Copy Ask AI from pipecat.services.tavus.video import TavusVideoService import aiohttp async def main (): async with aiohttp.ClientSession() as session: # Configure service tavus = TavusVideoService( api_key = "your-tavus-api-key" , replica_id = "your-replica-id" , session = session ) # Initialize conversation room_url = await tavus.initialize() # Get persona name persona_name = await tavus.get_persona_name() transport = DailyTransport( room_url = room_url, # Your Tavus room URL token = None , bot_name = "Pipecat bot" , params = DailyParams( audio_in_enabled = True , vad_analyzer = SileroVADAnalyzer(), ), ) # Use in pipeline pipeline = Pipeline( [ transport.input(), # Transport user input stt, # STT context_aggregator.user(), # User responses llm, # LLM tts, # TTS tavus, # Tavus output layer transport.output(), # Transport bot output context_aggregator.assistant(), # Assistant spoken responses ] ) ​ API Methods ​ initialize Copy Ask AI async def initialize ( self ) -> str : Initializes a new conversation and returns the conversation URL. ​ get_persona_name Copy Ask AI async def get_persona_name ( self ) -> str : Retrieves the name of the configured persona. ​ Frame Flow ​ Metrics Support The service collects processing metrics: Processing duration Time to First Byte (TTFB) API response times Audio processing metrics ​ Common Use Cases AI Video Avatars Virtual assistants Customer service Educational content Interactive Presentations Product demonstrations Training materials Marketing content Real-time Communication Video conferencing Virtual meetings Interactive broadcasts ​ Notes Handles real-time audio streaming Supports conversation interruptions Manages conversation lifecycle Automatic audio resampling Thread-safe processing WebRTC integration through Daily Includes comprehensive error handling Simli Mem0 On this page Overview Installation Configuration Constructor Parameters Input Frames Audio Input Control Frames Usage Example API Methods initialize get_persona_name Frame Flow Metrics Support Common Use Cases Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/video_tavus_f895f785.txt b/video_tavus_f895f785.txt
new file mode 100644
index 0000000000000000000000000000000000000000..c5504f27e00db2dd6752a34f04ec42ec8bff532f
--- /dev/null
+++ b/video_tavus_f895f785.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/video/tavus#param-tts-audio-raw-frame
+Title: Tavus - Pipecat
+==================================================
+
+Tavus - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Video Tavus Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Simli Tavus Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview TavusVideoService enables the creation of AI avatar video responses by sending audio to Tavus’s API. It handles real-time audio streaming, conversation management, and video generation through Tavus’s platform. ​ Installation To use TavusVideoService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[tavus]" You’ll need to set up the following environment variables: TAVUS_API_KEY - Your Tavus API key TAVUS_REPLICA_ID - Your Tavus replica identifier You can obtain a Tavus API key by signing up at Tavus . ​ Configuration ​ Constructor Parameters ​ api_key str required Your Tavus API key ​ replica_id str required Tavus replica identifier ​ persona_id str default: "pipecat0" Tavus persona identifier The persona ID is optional and defaults to pipecat0 . To use the LLM output and TTS voice, do not set the persona_id. Instead, leave it set to the default value, pipecat0 . ​ session aiohttp.ClientSession required HTTP client session for API communication ​ Input Frames ​ Audio Input ​ TTSAudioRawFrame Frame Raw audio data for avatar speech ​ Control Frames ​ TTSStartedFrame Frame Signals start of speech synthesis ​ TTSStoppedFrame Frame Signals end of speech synthesis ​ StartInterruptionFrame Frame Signals conversation interruption ​ EndFrame Frame Signals end of conversation ​ CancelFrame Frame Signals conversation cancellation ​ Usage Example Copy Ask AI from pipecat.services.tavus.video import TavusVideoService import aiohttp async def main (): async with aiohttp.ClientSession() as session: # Configure service tavus = TavusVideoService( api_key = "your-tavus-api-key" , replica_id = "your-replica-id" , session = session ) # Initialize conversation room_url = await tavus.initialize() # Get persona name persona_name = await tavus.get_persona_name() transport = DailyTransport( room_url = room_url, # Your Tavus room URL token = None , bot_name = "Pipecat bot" , params = DailyParams( audio_in_enabled = True , vad_analyzer = SileroVADAnalyzer(), ), ) # Use in pipeline pipeline = Pipeline( [ transport.input(), # Transport user input stt, # STT context_aggregator.user(), # User responses llm, # LLM tts, # TTS tavus, # Tavus output layer transport.output(), # Transport bot output context_aggregator.assistant(), # Assistant spoken responses ] ) ​ API Methods ​ initialize Copy Ask AI async def initialize ( self ) -> str : Initializes a new conversation and returns the conversation URL. ​ get_persona_name Copy Ask AI async def get_persona_name ( self ) -> str : Retrieves the name of the configured persona. ​ Frame Flow ​ Metrics Support The service collects processing metrics: Processing duration Time to First Byte (TTFB) API response times Audio processing metrics ​ Common Use Cases AI Video Avatars Virtual assistants Customer service Educational content Interactive Presentations Product demonstrations Training materials Marketing content Real-time Communication Video conferencing Virtual meetings Interactive broadcasts ​ Notes Handles real-time audio streaming Supports conversation interruptions Manages conversation lifecycle Automatic audio resampling Thread-safe processing WebRTC integration through Daily Includes comprehensive error handling Simli Mem0 On this page Overview Installation Configuration Constructor Parameters Input Frames Audio Input Control Frames Usage Example API Methods initialize get_persona_name Frame Flow Metrics Support Common Use Cases Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/video_tavus_fe043c9f.txt b/video_tavus_fe043c9f.txt
new file mode 100644
index 0000000000000000000000000000000000000000..8f72b2025bd57c47effc0a3fc5d90d61e1f05901
--- /dev/null
+++ b/video_tavus_fe043c9f.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/video/tavus#param-api-key
+Title: Tavus - Pipecat
+==================================================
+
+Tavus - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Video Tavus Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Simli Tavus Memory Vision Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview TavusVideoService enables the creation of AI avatar video responses by sending audio to Tavus’s API. It handles real-time audio streaming, conversation management, and video generation through Tavus’s platform. ​ Installation To use TavusVideoService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[tavus]" You’ll need to set up the following environment variables: TAVUS_API_KEY - Your Tavus API key TAVUS_REPLICA_ID - Your Tavus replica identifier You can obtain a Tavus API key by signing up at Tavus . ​ Configuration ​ Constructor Parameters ​ api_key str required Your Tavus API key ​ replica_id str required Tavus replica identifier ​ persona_id str default: "pipecat0" Tavus persona identifier The persona ID is optional and defaults to pipecat0 . To use the LLM output and TTS voice, do not set the persona_id. Instead, leave it set to the default value, pipecat0 . ​ session aiohttp.ClientSession required HTTP client session for API communication ​ Input Frames ​ Audio Input ​ TTSAudioRawFrame Frame Raw audio data for avatar speech ​ Control Frames ​ TTSStartedFrame Frame Signals start of speech synthesis ​ TTSStoppedFrame Frame Signals end of speech synthesis ​ StartInterruptionFrame Frame Signals conversation interruption ​ EndFrame Frame Signals end of conversation ​ CancelFrame Frame Signals conversation cancellation ​ Usage Example Copy Ask AI from pipecat.services.tavus.video import TavusVideoService import aiohttp async def main (): async with aiohttp.ClientSession() as session: # Configure service tavus = TavusVideoService( api_key = "your-tavus-api-key" , replica_id = "your-replica-id" , session = session ) # Initialize conversation room_url = await tavus.initialize() # Get persona name persona_name = await tavus.get_persona_name() transport = DailyTransport( room_url = room_url, # Your Tavus room URL token = None , bot_name = "Pipecat bot" , params = DailyParams( audio_in_enabled = True , vad_analyzer = SileroVADAnalyzer(), ), ) # Use in pipeline pipeline = Pipeline( [ transport.input(), # Transport user input stt, # STT context_aggregator.user(), # User responses llm, # LLM tts, # TTS tavus, # Tavus output layer transport.output(), # Transport bot output context_aggregator.assistant(), # Assistant spoken responses ] ) ​ API Methods ​ initialize Copy Ask AI async def initialize ( self ) -> str : Initializes a new conversation and returns the conversation URL. ​ get_persona_name Copy Ask AI async def get_persona_name ( self ) -> str : Retrieves the name of the configured persona. ​ Frame Flow ​ Metrics Support The service collects processing metrics: Processing duration Time to First Byte (TTFB) API response times Audio processing metrics ​ Common Use Cases AI Video Avatars Virtual assistants Customer service Educational content Interactive Presentations Product demonstrations Training materials Marketing content Real-time Communication Video conferencing Virtual meetings Interactive broadcasts ​ Notes Handles real-time audio streaming Supports conversation interruptions Manages conversation lifecycle Automatic audio resampling Thread-safe processing WebRTC integration through Daily Includes comprehensive error handling Simli Mem0 On this page Overview Installation Configuration Constructor Parameters Input Frames Audio Input Control Frames Usage Example API Methods initialize get_persona_name Frame Flow Metrics Support Common Use Cases Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/vision_moondream_0aff4193.txt b/vision_moondream_0aff4193.txt
new file mode 100644
index 0000000000000000000000000000000000000000..ce8b62c4925cf6d61346fdd7a00bbf44850d965f
--- /dev/null
+++ b/vision_moondream_0aff4193.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/vision/moondream#usage-example
+Title: Moondream - Pipecat
+==================================================
+
+Moondream - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Vision Moondream Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Moondream Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview MoondreamService provides local image analysis and question-answering capabilities using the Moondream model. It runs entirely on your local machine, supporting various hardware acceleration options including CUDA, Intel XPU, and Apple MPS. ​ Installation To use MoondreamService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[moondream]" You can obtain a Moondream API key by signing up at Moondream . ​ Configuration ​ Constructor Parameters ​ model str default: "vikhyatk/moondream2" Hugging Face model identifier ​ revision str default: "2024-08-26" Model revision/version ​ use_cpu bool default: "False" Force CPU usage instead of available accelerators ​ Hardware Acceleration The service automatically detects and uses the best available hardware: Intel XPU (if intel_extension_for_pytorch is installed) NVIDIA CUDA Apple Metal (MPS) CPU (fallback) ​ Input ​ VisionImageRawFrame ​ format str Image format (e.g., ‘RGB’, ‘RGBA’) ​ size tuple Image dimensions (width, height) ​ image bytes Raw image data ​ text str Question about the image ​ Output Frames ​ TextFrame ​ text str Generated description or answer about the image ​ ErrorFrame ​ error str Error information if processing fails ​ Methods See the Vision base class methods for additional functionality. ​ Usage Example Copy Ask AI from pipecat.services.moondream.vision import MoondreamService from pipecat.frames.frames import VisionImageRawFrame from PIL import Image # Configure service service = MoondreamService( model = "vikhyatk/moondream2" , revision = "2024-08-26" ) # Create pipeline pipeline = Pipeline([ image_input, # Produces VisionImageRawFrame service, # Analyzes images text_handler # Handles text responses ]) # Example frame processing image = Image.open( "example.jpg" ) frame = VisionImageRawFrame( format = image.mode, size = image.size, image = image.tobytes(), text = "What objects are in this image?" ) ​ Hardware Configuration Examples ​ CUDA (NVIDIA GPU) Copy Ask AI # Automatically uses CUDA if available service = MoondreamService() ​ Intel XPU Copy Ask AI # Requires intel_extension_for_pytorch import intel_extension_for_pytorch service = MoondreamService() ​ Force CPU Usage Copy Ask AI service = MoondreamService( use_cpu = True ) ​ Frame Flow ​ Metrics Support The service collects processing metrics: Processing duration Model loading time Inference time ​ Performance Considerations ​ Memory Usage Model size varies by version GPU memory requirements depend on image size CPU mode uses more system memory ​ Processing Speed Relative performance by hardware: NVIDIA GPU (fastest) Intel XPU Apple MPS CPU (slowest) ​ Best Practices ​ 1. Image Preparation Copy Ask AI # Optimize image before processing def prepare_image ( image_path ): image = Image.open(image_path) # Resize if needed if max (image.size) > 1024 : image.thumbnail(( 1024 , 1024 )) return image ​ 2. Error Handling Copy Ask AI try : async for frame in service.run_vision(vision_frame): if isinstance (frame, ErrorFrame): logger.error( f "Vision processing error: { frame.error } " ) elif isinstance (frame, TextFrame): process_result(frame.text) except Exception as e: logger.error( f "Unexpected error: { e } " ) ​ 3. Resource Management Copy Ask AI # Initialize once, reuse for multiple images service = MoondreamService() try : # Process multiple images for image in images: await process_image(service, image) finally : # Cleanup if needed await service.cleanup() ​ Notes Runs completely offline after model download First run requires model download Supports multiple hardware acceleration options Thread-safe processing Automatic error handling Manages model lifecycle Supports various image formats Mem0 Sentry Metrics On this page Overview Installation Configuration Constructor Parameters Hardware Acceleration Input VisionImageRawFrame Output Frames TextFrame ErrorFrame Methods Usage Example Hardware Configuration Examples CUDA (NVIDIA GPU) Intel XPU Force CPU Usage Frame Flow Metrics Support Performance Considerations Memory Usage Processing Speed Best Practices 1. Image Preparation 2. Error Handling 3. Resource Management Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/vision_moondream_3e038f37.txt b/vision_moondream_3e038f37.txt
new file mode 100644
index 0000000000000000000000000000000000000000..a13e575e18c171a3640625a2e5cb8cc38c40c6ed
--- /dev/null
+++ b/vision_moondream_3e038f37.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/vision/moondream#1-image-preparation
+Title: Moondream - Pipecat
+==================================================
+
+Moondream - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Vision Moondream Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Moondream Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview MoondreamService provides local image analysis and question-answering capabilities using the Moondream model. It runs entirely on your local machine, supporting various hardware acceleration options including CUDA, Intel XPU, and Apple MPS. ​ Installation To use MoondreamService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[moondream]" You can obtain a Moondream API key by signing up at Moondream . ​ Configuration ​ Constructor Parameters ​ model str default: "vikhyatk/moondream2" Hugging Face model identifier ​ revision str default: "2024-08-26" Model revision/version ​ use_cpu bool default: "False" Force CPU usage instead of available accelerators ​ Hardware Acceleration The service automatically detects and uses the best available hardware: Intel XPU (if intel_extension_for_pytorch is installed) NVIDIA CUDA Apple Metal (MPS) CPU (fallback) ​ Input ​ VisionImageRawFrame ​ format str Image format (e.g., ‘RGB’, ‘RGBA’) ​ size tuple Image dimensions (width, height) ​ image bytes Raw image data ​ text str Question about the image ​ Output Frames ​ TextFrame ​ text str Generated description or answer about the image ​ ErrorFrame ​ error str Error information if processing fails ​ Methods See the Vision base class methods for additional functionality. ​ Usage Example Copy Ask AI from pipecat.services.moondream.vision import MoondreamService from pipecat.frames.frames import VisionImageRawFrame from PIL import Image # Configure service service = MoondreamService( model = "vikhyatk/moondream2" , revision = "2024-08-26" ) # Create pipeline pipeline = Pipeline([ image_input, # Produces VisionImageRawFrame service, # Analyzes images text_handler # Handles text responses ]) # Example frame processing image = Image.open( "example.jpg" ) frame = VisionImageRawFrame( format = image.mode, size = image.size, image = image.tobytes(), text = "What objects are in this image?" ) ​ Hardware Configuration Examples ​ CUDA (NVIDIA GPU) Copy Ask AI # Automatically uses CUDA if available service = MoondreamService() ​ Intel XPU Copy Ask AI # Requires intel_extension_for_pytorch import intel_extension_for_pytorch service = MoondreamService() ​ Force CPU Usage Copy Ask AI service = MoondreamService( use_cpu = True ) ​ Frame Flow ​ Metrics Support The service collects processing metrics: Processing duration Model loading time Inference time ​ Performance Considerations ​ Memory Usage Model size varies by version GPU memory requirements depend on image size CPU mode uses more system memory ​ Processing Speed Relative performance by hardware: NVIDIA GPU (fastest) Intel XPU Apple MPS CPU (slowest) ​ Best Practices ​ 1. Image Preparation Copy Ask AI # Optimize image before processing def prepare_image ( image_path ): image = Image.open(image_path) # Resize if needed if max (image.size) > 1024 : image.thumbnail(( 1024 , 1024 )) return image ​ 2. Error Handling Copy Ask AI try : async for frame in service.run_vision(vision_frame): if isinstance (frame, ErrorFrame): logger.error( f "Vision processing error: { frame.error } " ) elif isinstance (frame, TextFrame): process_result(frame.text) except Exception as e: logger.error( f "Unexpected error: { e } " ) ​ 3. Resource Management Copy Ask AI # Initialize once, reuse for multiple images service = MoondreamService() try : # Process multiple images for image in images: await process_image(service, image) finally : # Cleanup if needed await service.cleanup() ​ Notes Runs completely offline after model download First run requires model download Supports multiple hardware acceleration options Thread-safe processing Automatic error handling Manages model lifecycle Supports various image formats Mem0 Sentry Metrics On this page Overview Installation Configuration Constructor Parameters Hardware Acceleration Input VisionImageRawFrame Output Frames TextFrame ErrorFrame Methods Usage Example Hardware Configuration Examples CUDA (NVIDIA GPU) Intel XPU Force CPU Usage Frame Flow Metrics Support Performance Considerations Memory Usage Processing Speed Best Practices 1. Image Preparation 2. Error Handling 3. Resource Management Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/vision_moondream_41700b2c.txt b/vision_moondream_41700b2c.txt
new file mode 100644
index 0000000000000000000000000000000000000000..2de0e4c78e09d8729bbe026380a967583222119b
--- /dev/null
+++ b/vision_moondream_41700b2c.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/vision/moondream#constructor-parameters
+Title: Moondream - Pipecat
+==================================================
+
+Moondream - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Vision Moondream Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Moondream Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview MoondreamService provides local image analysis and question-answering capabilities using the Moondream model. It runs entirely on your local machine, supporting various hardware acceleration options including CUDA, Intel XPU, and Apple MPS. ​ Installation To use MoondreamService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[moondream]" You can obtain a Moondream API key by signing up at Moondream . ​ Configuration ​ Constructor Parameters ​ model str default: "vikhyatk/moondream2" Hugging Face model identifier ​ revision str default: "2024-08-26" Model revision/version ​ use_cpu bool default: "False" Force CPU usage instead of available accelerators ​ Hardware Acceleration The service automatically detects and uses the best available hardware: Intel XPU (if intel_extension_for_pytorch is installed) NVIDIA CUDA Apple Metal (MPS) CPU (fallback) ​ Input ​ VisionImageRawFrame ​ format str Image format (e.g., ‘RGB’, ‘RGBA’) ​ size tuple Image dimensions (width, height) ​ image bytes Raw image data ​ text str Question about the image ​ Output Frames ​ TextFrame ​ text str Generated description or answer about the image ​ ErrorFrame ​ error str Error information if processing fails ​ Methods See the Vision base class methods for additional functionality. ​ Usage Example Copy Ask AI from pipecat.services.moondream.vision import MoondreamService from pipecat.frames.frames import VisionImageRawFrame from PIL import Image # Configure service service = MoondreamService( model = "vikhyatk/moondream2" , revision = "2024-08-26" ) # Create pipeline pipeline = Pipeline([ image_input, # Produces VisionImageRawFrame service, # Analyzes images text_handler # Handles text responses ]) # Example frame processing image = Image.open( "example.jpg" ) frame = VisionImageRawFrame( format = image.mode, size = image.size, image = image.tobytes(), text = "What objects are in this image?" ) ​ Hardware Configuration Examples ​ CUDA (NVIDIA GPU) Copy Ask AI # Automatically uses CUDA if available service = MoondreamService() ​ Intel XPU Copy Ask AI # Requires intel_extension_for_pytorch import intel_extension_for_pytorch service = MoondreamService() ​ Force CPU Usage Copy Ask AI service = MoondreamService( use_cpu = True ) ​ Frame Flow ​ Metrics Support The service collects processing metrics: Processing duration Model loading time Inference time ​ Performance Considerations ​ Memory Usage Model size varies by version GPU memory requirements depend on image size CPU mode uses more system memory ​ Processing Speed Relative performance by hardware: NVIDIA GPU (fastest) Intel XPU Apple MPS CPU (slowest) ​ Best Practices ​ 1. Image Preparation Copy Ask AI # Optimize image before processing def prepare_image ( image_path ): image = Image.open(image_path) # Resize if needed if max (image.size) > 1024 : image.thumbnail(( 1024 , 1024 )) return image ​ 2. Error Handling Copy Ask AI try : async for frame in service.run_vision(vision_frame): if isinstance (frame, ErrorFrame): logger.error( f "Vision processing error: { frame.error } " ) elif isinstance (frame, TextFrame): process_result(frame.text) except Exception as e: logger.error( f "Unexpected error: { e } " ) ​ 3. Resource Management Copy Ask AI # Initialize once, reuse for multiple images service = MoondreamService() try : # Process multiple images for image in images: await process_image(service, image) finally : # Cleanup if needed await service.cleanup() ​ Notes Runs completely offline after model download First run requires model download Supports multiple hardware acceleration options Thread-safe processing Automatic error handling Manages model lifecycle Supports various image formats Mem0 Sentry Metrics On this page Overview Installation Configuration Constructor Parameters Hardware Acceleration Input VisionImageRawFrame Output Frames TextFrame ErrorFrame Methods Usage Example Hardware Configuration Examples CUDA (NVIDIA GPU) Intel XPU Force CPU Usage Frame Flow Metrics Support Performance Considerations Memory Usage Processing Speed Best Practices 1. Image Preparation 2. Error Handling 3. Resource Management Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/vision_moondream_6eadf229.txt b/vision_moondream_6eadf229.txt
new file mode 100644
index 0000000000000000000000000000000000000000..2454da132cdd72d0ecec48005c1af753d37b35f5
--- /dev/null
+++ b/vision_moondream_6eadf229.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/vision/moondream#output-frames
+Title: Moondream - Pipecat
+==================================================
+
+Moondream - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Vision Moondream Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Moondream Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview MoondreamService provides local image analysis and question-answering capabilities using the Moondream model. It runs entirely on your local machine, supporting various hardware acceleration options including CUDA, Intel XPU, and Apple MPS. ​ Installation To use MoondreamService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[moondream]" You can obtain a Moondream API key by signing up at Moondream . ​ Configuration ​ Constructor Parameters ​ model str default: "vikhyatk/moondream2" Hugging Face model identifier ​ revision str default: "2024-08-26" Model revision/version ​ use_cpu bool default: "False" Force CPU usage instead of available accelerators ​ Hardware Acceleration The service automatically detects and uses the best available hardware: Intel XPU (if intel_extension_for_pytorch is installed) NVIDIA CUDA Apple Metal (MPS) CPU (fallback) ​ Input ​ VisionImageRawFrame ​ format str Image format (e.g., ‘RGB’, ‘RGBA’) ​ size tuple Image dimensions (width, height) ​ image bytes Raw image data ​ text str Question about the image ​ Output Frames ​ TextFrame ​ text str Generated description or answer about the image ​ ErrorFrame ​ error str Error information if processing fails ​ Methods See the Vision base class methods for additional functionality. ​ Usage Example Copy Ask AI from pipecat.services.moondream.vision import MoondreamService from pipecat.frames.frames import VisionImageRawFrame from PIL import Image # Configure service service = MoondreamService( model = "vikhyatk/moondream2" , revision = "2024-08-26" ) # Create pipeline pipeline = Pipeline([ image_input, # Produces VisionImageRawFrame service, # Analyzes images text_handler # Handles text responses ]) # Example frame processing image = Image.open( "example.jpg" ) frame = VisionImageRawFrame( format = image.mode, size = image.size, image = image.tobytes(), text = "What objects are in this image?" ) ​ Hardware Configuration Examples ​ CUDA (NVIDIA GPU) Copy Ask AI # Automatically uses CUDA if available service = MoondreamService() ​ Intel XPU Copy Ask AI # Requires intel_extension_for_pytorch import intel_extension_for_pytorch service = MoondreamService() ​ Force CPU Usage Copy Ask AI service = MoondreamService( use_cpu = True ) ​ Frame Flow ​ Metrics Support The service collects processing metrics: Processing duration Model loading time Inference time ​ Performance Considerations ​ Memory Usage Model size varies by version GPU memory requirements depend on image size CPU mode uses more system memory ​ Processing Speed Relative performance by hardware: NVIDIA GPU (fastest) Intel XPU Apple MPS CPU (slowest) ​ Best Practices ​ 1. Image Preparation Copy Ask AI # Optimize image before processing def prepare_image ( image_path ): image = Image.open(image_path) # Resize if needed if max (image.size) > 1024 : image.thumbnail(( 1024 , 1024 )) return image ​ 2. Error Handling Copy Ask AI try : async for frame in service.run_vision(vision_frame): if isinstance (frame, ErrorFrame): logger.error( f "Vision processing error: { frame.error } " ) elif isinstance (frame, TextFrame): process_result(frame.text) except Exception as e: logger.error( f "Unexpected error: { e } " ) ​ 3. Resource Management Copy Ask AI # Initialize once, reuse for multiple images service = MoondreamService() try : # Process multiple images for image in images: await process_image(service, image) finally : # Cleanup if needed await service.cleanup() ​ Notes Runs completely offline after model download First run requires model download Supports multiple hardware acceleration options Thread-safe processing Automatic error handling Manages model lifecycle Supports various image formats Mem0 Sentry Metrics On this page Overview Installation Configuration Constructor Parameters Hardware Acceleration Input VisionImageRawFrame Output Frames TextFrame ErrorFrame Methods Usage Example Hardware Configuration Examples CUDA (NVIDIA GPU) Intel XPU Force CPU Usage Frame Flow Metrics Support Performance Considerations Memory Usage Processing Speed Best Practices 1. Image Preparation 2. Error Handling 3. Resource Management Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/vision_moondream_6ff3a12a.txt b/vision_moondream_6ff3a12a.txt
new file mode 100644
index 0000000000000000000000000000000000000000..d9151855d9f8d41f7ad2cc18a2e2468f4bac1e00
--- /dev/null
+++ b/vision_moondream_6ff3a12a.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/vision/moondream#intel-xpu
+Title: Moondream - Pipecat
+==================================================
+
+Moondream - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Vision Moondream Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Moondream Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview MoondreamService provides local image analysis and question-answering capabilities using the Moondream model. It runs entirely on your local machine, supporting various hardware acceleration options including CUDA, Intel XPU, and Apple MPS. ​ Installation To use MoondreamService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[moondream]" You can obtain a Moondream API key by signing up at Moondream . ​ Configuration ​ Constructor Parameters ​ model str default: "vikhyatk/moondream2" Hugging Face model identifier ​ revision str default: "2024-08-26" Model revision/version ​ use_cpu bool default: "False" Force CPU usage instead of available accelerators ​ Hardware Acceleration The service automatically detects and uses the best available hardware: Intel XPU (if intel_extension_for_pytorch is installed) NVIDIA CUDA Apple Metal (MPS) CPU (fallback) ​ Input ​ VisionImageRawFrame ​ format str Image format (e.g., ‘RGB’, ‘RGBA’) ​ size tuple Image dimensions (width, height) ​ image bytes Raw image data ​ text str Question about the image ​ Output Frames ​ TextFrame ​ text str Generated description or answer about the image ​ ErrorFrame ​ error str Error information if processing fails ​ Methods See the Vision base class methods for additional functionality. ​ Usage Example Copy Ask AI from pipecat.services.moondream.vision import MoondreamService from pipecat.frames.frames import VisionImageRawFrame from PIL import Image # Configure service service = MoondreamService( model = "vikhyatk/moondream2" , revision = "2024-08-26" ) # Create pipeline pipeline = Pipeline([ image_input, # Produces VisionImageRawFrame service, # Analyzes images text_handler # Handles text responses ]) # Example frame processing image = Image.open( "example.jpg" ) frame = VisionImageRawFrame( format = image.mode, size = image.size, image = image.tobytes(), text = "What objects are in this image?" ) ​ Hardware Configuration Examples ​ CUDA (NVIDIA GPU) Copy Ask AI # Automatically uses CUDA if available service = MoondreamService() ​ Intel XPU Copy Ask AI # Requires intel_extension_for_pytorch import intel_extension_for_pytorch service = MoondreamService() ​ Force CPU Usage Copy Ask AI service = MoondreamService( use_cpu = True ) ​ Frame Flow ​ Metrics Support The service collects processing metrics: Processing duration Model loading time Inference time ​ Performance Considerations ​ Memory Usage Model size varies by version GPU memory requirements depend on image size CPU mode uses more system memory ​ Processing Speed Relative performance by hardware: NVIDIA GPU (fastest) Intel XPU Apple MPS CPU (slowest) ​ Best Practices ​ 1. Image Preparation Copy Ask AI # Optimize image before processing def prepare_image ( image_path ): image = Image.open(image_path) # Resize if needed if max (image.size) > 1024 : image.thumbnail(( 1024 , 1024 )) return image ​ 2. Error Handling Copy Ask AI try : async for frame in service.run_vision(vision_frame): if isinstance (frame, ErrorFrame): logger.error( f "Vision processing error: { frame.error } " ) elif isinstance (frame, TextFrame): process_result(frame.text) except Exception as e: logger.error( f "Unexpected error: { e } " ) ​ 3. Resource Management Copy Ask AI # Initialize once, reuse for multiple images service = MoondreamService() try : # Process multiple images for image in images: await process_image(service, image) finally : # Cleanup if needed await service.cleanup() ​ Notes Runs completely offline after model download First run requires model download Supports multiple hardware acceleration options Thread-safe processing Automatic error handling Manages model lifecycle Supports various image formats Mem0 Sentry Metrics On this page Overview Installation Configuration Constructor Parameters Hardware Acceleration Input VisionImageRawFrame Output Frames TextFrame ErrorFrame Methods Usage Example Hardware Configuration Examples CUDA (NVIDIA GPU) Intel XPU Force CPU Usage Frame Flow Metrics Support Performance Considerations Memory Usage Processing Speed Best Practices 1. Image Preparation 2. Error Handling 3. Resource Management Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/vision_moondream_7924a8b1.txt b/vision_moondream_7924a8b1.txt
new file mode 100644
index 0000000000000000000000000000000000000000..0da649322fa77f636958a70ab287128cd128b953
--- /dev/null
+++ b/vision_moondream_7924a8b1.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/vision/moondream#param-size
+Title: Moondream - Pipecat
+==================================================
+
+Moondream - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Vision Moondream Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Moondream Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview MoondreamService provides local image analysis and question-answering capabilities using the Moondream model. It runs entirely on your local machine, supporting various hardware acceleration options including CUDA, Intel XPU, and Apple MPS. ​ Installation To use MoondreamService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[moondream]" You can obtain a Moondream API key by signing up at Moondream . ​ Configuration ​ Constructor Parameters ​ model str default: "vikhyatk/moondream2" Hugging Face model identifier ​ revision str default: "2024-08-26" Model revision/version ​ use_cpu bool default: "False" Force CPU usage instead of available accelerators ​ Hardware Acceleration The service automatically detects and uses the best available hardware: Intel XPU (if intel_extension_for_pytorch is installed) NVIDIA CUDA Apple Metal (MPS) CPU (fallback) ​ Input ​ VisionImageRawFrame ​ format str Image format (e.g., ‘RGB’, ‘RGBA’) ​ size tuple Image dimensions (width, height) ​ image bytes Raw image data ​ text str Question about the image ​ Output Frames ​ TextFrame ​ text str Generated description or answer about the image ​ ErrorFrame ​ error str Error information if processing fails ​ Methods See the Vision base class methods for additional functionality. ​ Usage Example Copy Ask AI from pipecat.services.moondream.vision import MoondreamService from pipecat.frames.frames import VisionImageRawFrame from PIL import Image # Configure service service = MoondreamService( model = "vikhyatk/moondream2" , revision = "2024-08-26" ) # Create pipeline pipeline = Pipeline([ image_input, # Produces VisionImageRawFrame service, # Analyzes images text_handler # Handles text responses ]) # Example frame processing image = Image.open( "example.jpg" ) frame = VisionImageRawFrame( format = image.mode, size = image.size, image = image.tobytes(), text = "What objects are in this image?" ) ​ Hardware Configuration Examples ​ CUDA (NVIDIA GPU) Copy Ask AI # Automatically uses CUDA if available service = MoondreamService() ​ Intel XPU Copy Ask AI # Requires intel_extension_for_pytorch import intel_extension_for_pytorch service = MoondreamService() ​ Force CPU Usage Copy Ask AI service = MoondreamService( use_cpu = True ) ​ Frame Flow ​ Metrics Support The service collects processing metrics: Processing duration Model loading time Inference time ​ Performance Considerations ​ Memory Usage Model size varies by version GPU memory requirements depend on image size CPU mode uses more system memory ​ Processing Speed Relative performance by hardware: NVIDIA GPU (fastest) Intel XPU Apple MPS CPU (slowest) ​ Best Practices ​ 1. Image Preparation Copy Ask AI # Optimize image before processing def prepare_image ( image_path ): image = Image.open(image_path) # Resize if needed if max (image.size) > 1024 : image.thumbnail(( 1024 , 1024 )) return image ​ 2. Error Handling Copy Ask AI try : async for frame in service.run_vision(vision_frame): if isinstance (frame, ErrorFrame): logger.error( f "Vision processing error: { frame.error } " ) elif isinstance (frame, TextFrame): process_result(frame.text) except Exception as e: logger.error( f "Unexpected error: { e } " ) ​ 3. Resource Management Copy Ask AI # Initialize once, reuse for multiple images service = MoondreamService() try : # Process multiple images for image in images: await process_image(service, image) finally : # Cleanup if needed await service.cleanup() ​ Notes Runs completely offline after model download First run requires model download Supports multiple hardware acceleration options Thread-safe processing Automatic error handling Manages model lifecycle Supports various image formats Mem0 Sentry Metrics On this page Overview Installation Configuration Constructor Parameters Hardware Acceleration Input VisionImageRawFrame Output Frames TextFrame ErrorFrame Methods Usage Example Hardware Configuration Examples CUDA (NVIDIA GPU) Intel XPU Force CPU Usage Frame Flow Metrics Support Performance Considerations Memory Usage Processing Speed Best Practices 1. Image Preparation 2. Error Handling 3. Resource Management Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/vision_moondream_8c89d079.txt b/vision_moondream_8c89d079.txt
new file mode 100644
index 0000000000000000000000000000000000000000..a197a943be3ceb4d73c6fe6c6bb404e4ab047d0f
--- /dev/null
+++ b/vision_moondream_8c89d079.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/vision/moondream#methods
+Title: Moondream - Pipecat
+==================================================
+
+Moondream - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Vision Moondream Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Moondream Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview MoondreamService provides local image analysis and question-answering capabilities using the Moondream model. It runs entirely on your local machine, supporting various hardware acceleration options including CUDA, Intel XPU, and Apple MPS. ​ Installation To use MoondreamService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[moondream]" You can obtain a Moondream API key by signing up at Moondream . ​ Configuration ​ Constructor Parameters ​ model str default: "vikhyatk/moondream2" Hugging Face model identifier ​ revision str default: "2024-08-26" Model revision/version ​ use_cpu bool default: "False" Force CPU usage instead of available accelerators ​ Hardware Acceleration The service automatically detects and uses the best available hardware: Intel XPU (if intel_extension_for_pytorch is installed) NVIDIA CUDA Apple Metal (MPS) CPU (fallback) ​ Input ​ VisionImageRawFrame ​ format str Image format (e.g., ‘RGB’, ‘RGBA’) ​ size tuple Image dimensions (width, height) ​ image bytes Raw image data ​ text str Question about the image ​ Output Frames ​ TextFrame ​ text str Generated description or answer about the image ​ ErrorFrame ​ error str Error information if processing fails ​ Methods See the Vision base class methods for additional functionality. ​ Usage Example Copy Ask AI from pipecat.services.moondream.vision import MoondreamService from pipecat.frames.frames import VisionImageRawFrame from PIL import Image # Configure service service = MoondreamService( model = "vikhyatk/moondream2" , revision = "2024-08-26" ) # Create pipeline pipeline = Pipeline([ image_input, # Produces VisionImageRawFrame service, # Analyzes images text_handler # Handles text responses ]) # Example frame processing image = Image.open( "example.jpg" ) frame = VisionImageRawFrame( format = image.mode, size = image.size, image = image.tobytes(), text = "What objects are in this image?" ) ​ Hardware Configuration Examples ​ CUDA (NVIDIA GPU) Copy Ask AI # Automatically uses CUDA if available service = MoondreamService() ​ Intel XPU Copy Ask AI # Requires intel_extension_for_pytorch import intel_extension_for_pytorch service = MoondreamService() ​ Force CPU Usage Copy Ask AI service = MoondreamService( use_cpu = True ) ​ Frame Flow ​ Metrics Support The service collects processing metrics: Processing duration Model loading time Inference time ​ Performance Considerations ​ Memory Usage Model size varies by version GPU memory requirements depend on image size CPU mode uses more system memory ​ Processing Speed Relative performance by hardware: NVIDIA GPU (fastest) Intel XPU Apple MPS CPU (slowest) ​ Best Practices ​ 1. Image Preparation Copy Ask AI # Optimize image before processing def prepare_image ( image_path ): image = Image.open(image_path) # Resize if needed if max (image.size) > 1024 : image.thumbnail(( 1024 , 1024 )) return image ​ 2. Error Handling Copy Ask AI try : async for frame in service.run_vision(vision_frame): if isinstance (frame, ErrorFrame): logger.error( f "Vision processing error: { frame.error } " ) elif isinstance (frame, TextFrame): process_result(frame.text) except Exception as e: logger.error( f "Unexpected error: { e } " ) ​ 3. Resource Management Copy Ask AI # Initialize once, reuse for multiple images service = MoondreamService() try : # Process multiple images for image in images: await process_image(service, image) finally : # Cleanup if needed await service.cleanup() ​ Notes Runs completely offline after model download First run requires model download Supports multiple hardware acceleration options Thread-safe processing Automatic error handling Manages model lifecycle Supports various image formats Mem0 Sentry Metrics On this page Overview Installation Configuration Constructor Parameters Hardware Acceleration Input VisionImageRawFrame Output Frames TextFrame ErrorFrame Methods Usage Example Hardware Configuration Examples CUDA (NVIDIA GPU) Intel XPU Force CPU Usage Frame Flow Metrics Support Performance Considerations Memory Usage Processing Speed Best Practices 1. Image Preparation 2. Error Handling 3. Resource Management Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/vision_moondream_9233ffc0.txt b/vision_moondream_9233ffc0.txt
new file mode 100644
index 0000000000000000000000000000000000000000..abbba67fbeb281e8a37720e73a09b5d0a33d04a9
--- /dev/null
+++ b/vision_moondream_9233ffc0.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/vision/moondream#hardware-configuration-examples
+Title: Moondream - Pipecat
+==================================================
+
+Moondream - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Vision Moondream Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Moondream Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview MoondreamService provides local image analysis and question-answering capabilities using the Moondream model. It runs entirely on your local machine, supporting various hardware acceleration options including CUDA, Intel XPU, and Apple MPS. ​ Installation To use MoondreamService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[moondream]" You can obtain a Moondream API key by signing up at Moondream . ​ Configuration ​ Constructor Parameters ​ model str default: "vikhyatk/moondream2" Hugging Face model identifier ​ revision str default: "2024-08-26" Model revision/version ​ use_cpu bool default: "False" Force CPU usage instead of available accelerators ​ Hardware Acceleration The service automatically detects and uses the best available hardware: Intel XPU (if intel_extension_for_pytorch is installed) NVIDIA CUDA Apple Metal (MPS) CPU (fallback) ​ Input ​ VisionImageRawFrame ​ format str Image format (e.g., ‘RGB’, ‘RGBA’) ​ size tuple Image dimensions (width, height) ​ image bytes Raw image data ​ text str Question about the image ​ Output Frames ​ TextFrame ​ text str Generated description or answer about the image ​ ErrorFrame ​ error str Error information if processing fails ​ Methods See the Vision base class methods for additional functionality. ​ Usage Example Copy Ask AI from pipecat.services.moondream.vision import MoondreamService from pipecat.frames.frames import VisionImageRawFrame from PIL import Image # Configure service service = MoondreamService( model = "vikhyatk/moondream2" , revision = "2024-08-26" ) # Create pipeline pipeline = Pipeline([ image_input, # Produces VisionImageRawFrame service, # Analyzes images text_handler # Handles text responses ]) # Example frame processing image = Image.open( "example.jpg" ) frame = VisionImageRawFrame( format = image.mode, size = image.size, image = image.tobytes(), text = "What objects are in this image?" ) ​ Hardware Configuration Examples ​ CUDA (NVIDIA GPU) Copy Ask AI # Automatically uses CUDA if available service = MoondreamService() ​ Intel XPU Copy Ask AI # Requires intel_extension_for_pytorch import intel_extension_for_pytorch service = MoondreamService() ​ Force CPU Usage Copy Ask AI service = MoondreamService( use_cpu = True ) ​ Frame Flow ​ Metrics Support The service collects processing metrics: Processing duration Model loading time Inference time ​ Performance Considerations ​ Memory Usage Model size varies by version GPU memory requirements depend on image size CPU mode uses more system memory ​ Processing Speed Relative performance by hardware: NVIDIA GPU (fastest) Intel XPU Apple MPS CPU (slowest) ​ Best Practices ​ 1. Image Preparation Copy Ask AI # Optimize image before processing def prepare_image ( image_path ): image = Image.open(image_path) # Resize if needed if max (image.size) > 1024 : image.thumbnail(( 1024 , 1024 )) return image ​ 2. Error Handling Copy Ask AI try : async for frame in service.run_vision(vision_frame): if isinstance (frame, ErrorFrame): logger.error( f "Vision processing error: { frame.error } " ) elif isinstance (frame, TextFrame): process_result(frame.text) except Exception as e: logger.error( f "Unexpected error: { e } " ) ​ 3. Resource Management Copy Ask AI # Initialize once, reuse for multiple images service = MoondreamService() try : # Process multiple images for image in images: await process_image(service, image) finally : # Cleanup if needed await service.cleanup() ​ Notes Runs completely offline after model download First run requires model download Supports multiple hardware acceleration options Thread-safe processing Automatic error handling Manages model lifecycle Supports various image formats Mem0 Sentry Metrics On this page Overview Installation Configuration Constructor Parameters Hardware Acceleration Input VisionImageRawFrame Output Frames TextFrame ErrorFrame Methods Usage Example Hardware Configuration Examples CUDA (NVIDIA GPU) Intel XPU Force CPU Usage Frame Flow Metrics Support Performance Considerations Memory Usage Processing Speed Best Practices 1. Image Preparation 2. Error Handling 3. Resource Management Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/vision_moondream_9f5e7fbf.txt b/vision_moondream_9f5e7fbf.txt
new file mode 100644
index 0000000000000000000000000000000000000000..c52c4baa1f9533db79462af24aa3d837aedbbbac
--- /dev/null
+++ b/vision_moondream_9f5e7fbf.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/vision/moondream#frame-flow
+Title: Moondream - Pipecat
+==================================================
+
+Moondream - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Vision Moondream Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Moondream Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview MoondreamService provides local image analysis and question-answering capabilities using the Moondream model. It runs entirely on your local machine, supporting various hardware acceleration options including CUDA, Intel XPU, and Apple MPS. ​ Installation To use MoondreamService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[moondream]" You can obtain a Moondream API key by signing up at Moondream . ​ Configuration ​ Constructor Parameters ​ model str default: "vikhyatk/moondream2" Hugging Face model identifier ​ revision str default: "2024-08-26" Model revision/version ​ use_cpu bool default: "False" Force CPU usage instead of available accelerators ​ Hardware Acceleration The service automatically detects and uses the best available hardware: Intel XPU (if intel_extension_for_pytorch is installed) NVIDIA CUDA Apple Metal (MPS) CPU (fallback) ​ Input ​ VisionImageRawFrame ​ format str Image format (e.g., ‘RGB’, ‘RGBA’) ​ size tuple Image dimensions (width, height) ​ image bytes Raw image data ​ text str Question about the image ​ Output Frames ​ TextFrame ​ text str Generated description or answer about the image ​ ErrorFrame ​ error str Error information if processing fails ​ Methods See the Vision base class methods for additional functionality. ​ Usage Example Copy Ask AI from pipecat.services.moondream.vision import MoondreamService from pipecat.frames.frames import VisionImageRawFrame from PIL import Image # Configure service service = MoondreamService( model = "vikhyatk/moondream2" , revision = "2024-08-26" ) # Create pipeline pipeline = Pipeline([ image_input, # Produces VisionImageRawFrame service, # Analyzes images text_handler # Handles text responses ]) # Example frame processing image = Image.open( "example.jpg" ) frame = VisionImageRawFrame( format = image.mode, size = image.size, image = image.tobytes(), text = "What objects are in this image?" ) ​ Hardware Configuration Examples ​ CUDA (NVIDIA GPU) Copy Ask AI # Automatically uses CUDA if available service = MoondreamService() ​ Intel XPU Copy Ask AI # Requires intel_extension_for_pytorch import intel_extension_for_pytorch service = MoondreamService() ​ Force CPU Usage Copy Ask AI service = MoondreamService( use_cpu = True ) ​ Frame Flow ​ Metrics Support The service collects processing metrics: Processing duration Model loading time Inference time ​ Performance Considerations ​ Memory Usage Model size varies by version GPU memory requirements depend on image size CPU mode uses more system memory ​ Processing Speed Relative performance by hardware: NVIDIA GPU (fastest) Intel XPU Apple MPS CPU (slowest) ​ Best Practices ​ 1. Image Preparation Copy Ask AI # Optimize image before processing def prepare_image ( image_path ): image = Image.open(image_path) # Resize if needed if max (image.size) > 1024 : image.thumbnail(( 1024 , 1024 )) return image ​ 2. Error Handling Copy Ask AI try : async for frame in service.run_vision(vision_frame): if isinstance (frame, ErrorFrame): logger.error( f "Vision processing error: { frame.error } " ) elif isinstance (frame, TextFrame): process_result(frame.text) except Exception as e: logger.error( f "Unexpected error: { e } " ) ​ 3. Resource Management Copy Ask AI # Initialize once, reuse for multiple images service = MoondreamService() try : # Process multiple images for image in images: await process_image(service, image) finally : # Cleanup if needed await service.cleanup() ​ Notes Runs completely offline after model download First run requires model download Supports multiple hardware acceleration options Thread-safe processing Automatic error handling Manages model lifecycle Supports various image formats Mem0 Sentry Metrics On this page Overview Installation Configuration Constructor Parameters Hardware Acceleration Input VisionImageRawFrame Output Frames TextFrame ErrorFrame Methods Usage Example Hardware Configuration Examples CUDA (NVIDIA GPU) Intel XPU Force CPU Usage Frame Flow Metrics Support Performance Considerations Memory Usage Processing Speed Best Practices 1. Image Preparation 2. Error Handling 3. Resource Management Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/vision_moondream_ece08521.txt b/vision_moondream_ece08521.txt
new file mode 100644
index 0000000000000000000000000000000000000000..107d5d9ea966548ce9c3dd9e27d25f3f8c8dd602
--- /dev/null
+++ b/vision_moondream_ece08521.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/vision/moondream#param-error
+Title: Moondream - Pipecat
+==================================================
+
+Moondream - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Vision Moondream Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Moondream Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview MoondreamService provides local image analysis and question-answering capabilities using the Moondream model. It runs entirely on your local machine, supporting various hardware acceleration options including CUDA, Intel XPU, and Apple MPS. ​ Installation To use MoondreamService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[moondream]" You can obtain a Moondream API key by signing up at Moondream . ​ Configuration ​ Constructor Parameters ​ model str default: "vikhyatk/moondream2" Hugging Face model identifier ​ revision str default: "2024-08-26" Model revision/version ​ use_cpu bool default: "False" Force CPU usage instead of available accelerators ​ Hardware Acceleration The service automatically detects and uses the best available hardware: Intel XPU (if intel_extension_for_pytorch is installed) NVIDIA CUDA Apple Metal (MPS) CPU (fallback) ​ Input ​ VisionImageRawFrame ​ format str Image format (e.g., ‘RGB’, ‘RGBA’) ​ size tuple Image dimensions (width, height) ​ image bytes Raw image data ​ text str Question about the image ​ Output Frames ​ TextFrame ​ text str Generated description or answer about the image ​ ErrorFrame ​ error str Error information if processing fails ​ Methods See the Vision base class methods for additional functionality. ​ Usage Example Copy Ask AI from pipecat.services.moondream.vision import MoondreamService from pipecat.frames.frames import VisionImageRawFrame from PIL import Image # Configure service service = MoondreamService( model = "vikhyatk/moondream2" , revision = "2024-08-26" ) # Create pipeline pipeline = Pipeline([ image_input, # Produces VisionImageRawFrame service, # Analyzes images text_handler # Handles text responses ]) # Example frame processing image = Image.open( "example.jpg" ) frame = VisionImageRawFrame( format = image.mode, size = image.size, image = image.tobytes(), text = "What objects are in this image?" ) ​ Hardware Configuration Examples ​ CUDA (NVIDIA GPU) Copy Ask AI # Automatically uses CUDA if available service = MoondreamService() ​ Intel XPU Copy Ask AI # Requires intel_extension_for_pytorch import intel_extension_for_pytorch service = MoondreamService() ​ Force CPU Usage Copy Ask AI service = MoondreamService( use_cpu = True ) ​ Frame Flow ​ Metrics Support The service collects processing metrics: Processing duration Model loading time Inference time ​ Performance Considerations ​ Memory Usage Model size varies by version GPU memory requirements depend on image size CPU mode uses more system memory ​ Processing Speed Relative performance by hardware: NVIDIA GPU (fastest) Intel XPU Apple MPS CPU (slowest) ​ Best Practices ​ 1. Image Preparation Copy Ask AI # Optimize image before processing def prepare_image ( image_path ): image = Image.open(image_path) # Resize if needed if max (image.size) > 1024 : image.thumbnail(( 1024 , 1024 )) return image ​ 2. Error Handling Copy Ask AI try : async for frame in service.run_vision(vision_frame): if isinstance (frame, ErrorFrame): logger.error( f "Vision processing error: { frame.error } " ) elif isinstance (frame, TextFrame): process_result(frame.text) except Exception as e: logger.error( f "Unexpected error: { e } " ) ​ 3. Resource Management Copy Ask AI # Initialize once, reuse for multiple images service = MoondreamService() try : # Process multiple images for image in images: await process_image(service, image) finally : # Cleanup if needed await service.cleanup() ​ Notes Runs completely offline after model download First run requires model download Supports multiple hardware acceleration options Thread-safe processing Automatic error handling Manages model lifecycle Supports various image formats Mem0 Sentry Metrics On this page Overview Installation Configuration Constructor Parameters Hardware Acceleration Input VisionImageRawFrame Output Frames TextFrame ErrorFrame Methods Usage Example Hardware Configuration Examples CUDA (NVIDIA GPU) Intel XPU Force CPU Usage Frame Flow Metrics Support Performance Considerations Memory Usage Processing Speed Best Practices 1. Image Preparation 2. Error Handling 3. Resource Management Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/vision_moondream_f9fb22dd.txt b/vision_moondream_f9fb22dd.txt
new file mode 100644
index 0000000000000000000000000000000000000000..581c066714af623c5a9753de2161154485af6f0a
--- /dev/null
+++ b/vision_moondream_f9fb22dd.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/vision/moondream#memory-usage
+Title: Moondream - Pipecat
+==================================================
+
+Moondream - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Vision Moondream Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Moondream Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview MoondreamService provides local image analysis and question-answering capabilities using the Moondream model. It runs entirely on your local machine, supporting various hardware acceleration options including CUDA, Intel XPU, and Apple MPS. ​ Installation To use MoondreamService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[moondream]" You can obtain a Moondream API key by signing up at Moondream . ​ Configuration ​ Constructor Parameters ​ model str default: "vikhyatk/moondream2" Hugging Face model identifier ​ revision str default: "2024-08-26" Model revision/version ​ use_cpu bool default: "False" Force CPU usage instead of available accelerators ​ Hardware Acceleration The service automatically detects and uses the best available hardware: Intel XPU (if intel_extension_for_pytorch is installed) NVIDIA CUDA Apple Metal (MPS) CPU (fallback) ​ Input ​ VisionImageRawFrame ​ format str Image format (e.g., ‘RGB’, ‘RGBA’) ​ size tuple Image dimensions (width, height) ​ image bytes Raw image data ​ text str Question about the image ​ Output Frames ​ TextFrame ​ text str Generated description or answer about the image ​ ErrorFrame ​ error str Error information if processing fails ​ Methods See the Vision base class methods for additional functionality. ​ Usage Example Copy Ask AI from pipecat.services.moondream.vision import MoondreamService from pipecat.frames.frames import VisionImageRawFrame from PIL import Image # Configure service service = MoondreamService( model = "vikhyatk/moondream2" , revision = "2024-08-26" ) # Create pipeline pipeline = Pipeline([ image_input, # Produces VisionImageRawFrame service, # Analyzes images text_handler # Handles text responses ]) # Example frame processing image = Image.open( "example.jpg" ) frame = VisionImageRawFrame( format = image.mode, size = image.size, image = image.tobytes(), text = "What objects are in this image?" ) ​ Hardware Configuration Examples ​ CUDA (NVIDIA GPU) Copy Ask AI # Automatically uses CUDA if available service = MoondreamService() ​ Intel XPU Copy Ask AI # Requires intel_extension_for_pytorch import intel_extension_for_pytorch service = MoondreamService() ​ Force CPU Usage Copy Ask AI service = MoondreamService( use_cpu = True ) ​ Frame Flow ​ Metrics Support The service collects processing metrics: Processing duration Model loading time Inference time ​ Performance Considerations ​ Memory Usage Model size varies by version GPU memory requirements depend on image size CPU mode uses more system memory ​ Processing Speed Relative performance by hardware: NVIDIA GPU (fastest) Intel XPU Apple MPS CPU (slowest) ​ Best Practices ​ 1. Image Preparation Copy Ask AI # Optimize image before processing def prepare_image ( image_path ): image = Image.open(image_path) # Resize if needed if max (image.size) > 1024 : image.thumbnail(( 1024 , 1024 )) return image ​ 2. Error Handling Copy Ask AI try : async for frame in service.run_vision(vision_frame): if isinstance (frame, ErrorFrame): logger.error( f "Vision processing error: { frame.error } " ) elif isinstance (frame, TextFrame): process_result(frame.text) except Exception as e: logger.error( f "Unexpected error: { e } " ) ​ 3. Resource Management Copy Ask AI # Initialize once, reuse for multiple images service = MoondreamService() try : # Process multiple images for image in images: await process_image(service, image) finally : # Cleanup if needed await service.cleanup() ​ Notes Runs completely offline after model download First run requires model download Supports multiple hardware acceleration options Thread-safe processing Automatic error handling Manages model lifecycle Supports various image formats Mem0 Sentry Metrics On this page Overview Installation Configuration Constructor Parameters Hardware Acceleration Input VisionImageRawFrame Output Frames TextFrame ErrorFrame Methods Usage Example Hardware Configuration Examples CUDA (NVIDIA GPU) Intel XPU Force CPU Usage Frame Flow Metrics Support Performance Considerations Memory Usage Processing Speed Best Practices 1. Image Preparation 2. Error Handling 3. Resource Management Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file
diff --git a/vision_moondream_fc81c319.txt b/vision_moondream_fc81c319.txt
new file mode 100644
index 0000000000000000000000000000000000000000..620e9fcaeacc1a42f4260c998724135775b6e58d
--- /dev/null
+++ b/vision_moondream_fc81c319.txt
@@ -0,0 +1,5 @@
+URL: https://docs.pipecat.ai/server/services/vision/moondream#textframe
+Title: Moondream - Pipecat
+==================================================
+
+Moondream - Pipecat Pipecat home page Search... ⌘ K Ask AI Search... Navigation Vision Moondream Getting Started Guides Server APIs Client SDKs Community GitHub Examples Changelog Server API Reference API Reference Reference docs Services Supported Services Transport Serializers Speech-to-Text LLM Text-to-Speech Speech-to-Speech Image Generation Video Memory Vision Moondream Analytics & Monitoring Utilities Advanced Frame Processors Audio Processing Frame Filters Metrics and Telemetry MCP Observers Service Utilities Smart Turn Detection Task Handling and Monitoring Telephony Text Aggregators and Filters User and Bot Transcriptions User Interruptions Frameworks RTVI Pipecat Flows Pipeline PipelineParams PipelineTask Pipeline Idle Detection Pipeline Heartbeats ParallelPipeline ​ Overview MoondreamService provides local image analysis and question-answering capabilities using the Moondream model. It runs entirely on your local machine, supporting various hardware acceleration options including CUDA, Intel XPU, and Apple MPS. ​ Installation To use MoondreamService , install the required dependencies: Copy Ask AI pip install "pipecat-ai[moondream]" You can obtain a Moondream API key by signing up at Moondream . ​ Configuration ​ Constructor Parameters ​ model str default: "vikhyatk/moondream2" Hugging Face model identifier ​ revision str default: "2024-08-26" Model revision/version ​ use_cpu bool default: "False" Force CPU usage instead of available accelerators ​ Hardware Acceleration The service automatically detects and uses the best available hardware: Intel XPU (if intel_extension_for_pytorch is installed) NVIDIA CUDA Apple Metal (MPS) CPU (fallback) ​ Input ​ VisionImageRawFrame ​ format str Image format (e.g., ‘RGB’, ‘RGBA’) ​ size tuple Image dimensions (width, height) ​ image bytes Raw image data ​ text str Question about the image ​ Output Frames ​ TextFrame ​ text str Generated description or answer about the image ​ ErrorFrame ​ error str Error information if processing fails ​ Methods See the Vision base class methods for additional functionality. ​ Usage Example Copy Ask AI from pipecat.services.moondream.vision import MoondreamService from pipecat.frames.frames import VisionImageRawFrame from PIL import Image # Configure service service = MoondreamService( model = "vikhyatk/moondream2" , revision = "2024-08-26" ) # Create pipeline pipeline = Pipeline([ image_input, # Produces VisionImageRawFrame service, # Analyzes images text_handler # Handles text responses ]) # Example frame processing image = Image.open( "example.jpg" ) frame = VisionImageRawFrame( format = image.mode, size = image.size, image = image.tobytes(), text = "What objects are in this image?" ) ​ Hardware Configuration Examples ​ CUDA (NVIDIA GPU) Copy Ask AI # Automatically uses CUDA if available service = MoondreamService() ​ Intel XPU Copy Ask AI # Requires intel_extension_for_pytorch import intel_extension_for_pytorch service = MoondreamService() ​ Force CPU Usage Copy Ask AI service = MoondreamService( use_cpu = True ) ​ Frame Flow ​ Metrics Support The service collects processing metrics: Processing duration Model loading time Inference time ​ Performance Considerations ​ Memory Usage Model size varies by version GPU memory requirements depend on image size CPU mode uses more system memory ​ Processing Speed Relative performance by hardware: NVIDIA GPU (fastest) Intel XPU Apple MPS CPU (slowest) ​ Best Practices ​ 1. Image Preparation Copy Ask AI # Optimize image before processing def prepare_image ( image_path ): image = Image.open(image_path) # Resize if needed if max (image.size) > 1024 : image.thumbnail(( 1024 , 1024 )) return image ​ 2. Error Handling Copy Ask AI try : async for frame in service.run_vision(vision_frame): if isinstance (frame, ErrorFrame): logger.error( f "Vision processing error: { frame.error } " ) elif isinstance (frame, TextFrame): process_result(frame.text) except Exception as e: logger.error( f "Unexpected error: { e } " ) ​ 3. Resource Management Copy Ask AI # Initialize once, reuse for multiple images service = MoondreamService() try : # Process multiple images for image in images: await process_image(service, image) finally : # Cleanup if needed await service.cleanup() ​ Notes Runs completely offline after model download First run requires model download Supports multiple hardware acceleration options Thread-safe processing Automatic error handling Manages model lifecycle Supports various image formats Mem0 Sentry Metrics On this page Overview Installation Configuration Constructor Parameters Hardware Acceleration Input VisionImageRawFrame Output Frames TextFrame ErrorFrame Methods Usage Example Hardware Configuration Examples CUDA (NVIDIA GPU) Intel XPU Force CPU Usage Frame Flow Metrics Support Performance Considerations Memory Usage Processing Speed Best Practices 1. Image Preparation 2. Error Handling 3. Resource Management Notes Assistant Responses are generated using AI and may contain mistakes.
\ No newline at end of file